rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_red
icon_green
icon_green
icon_red
icon_red
icon_red
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_green
icon_orange
 

FS#752 — FS#4791 — asr-g1-a9

Attached to Project— Network
Incident
the whole network
CLOSED
100%
We are setting the ASR 9010 in the backbone.
And we are connnecting London on the router.

We have a problem on the routes that the BGP
is retaking on ASR and that are not seen
on the router. Not all the routes, some of them.

We have cut ASR for the time being.
Date:  Thursday, 11 November 2010, 17:10PM
Reason for closing:  Done
Comment by OVH - Friday, 05 November 2010, 14:00PM

RP/0/RSP1/CPU0:Nov 5 12:45:05 UTC: bgp[1046]:
%ROUTING-BGP-3-INITSYNC_FAILED : Nbr: VRF: default AFI: IPv4 Unicast
Pfx received: 83392 expected: 71163 - will retry once more


Comment by OVH - Friday, 05 November 2010, 14:00PM

%ROUTING-BGP-3-INITSYNC_FAILED [chars]

Explanation The number of prefixes synced by active doesn't match
the expected count.

Recommended Action Copy the error message exactly as it appears on
the console or in the system log, call your Cisco technical support
representative and provide the representative with the gathered
information.


Comment by OVH - Friday, 05 November 2010, 20:08PM

Weird. This morning OSPF is mounted between ASR and LDN-1 but this has provoked
the crash on certain destinations to LDN and BRU.

This afternoon, we have done several tests and it turns out that OSPF does not
mount between LDN-1 and ASR even though it was working normally with 10 other
routers at Roubaix.

The only difference is the IOS version. So we have always the same problem that
we had yesterday on the MTU
http://status.ovh.net/?do=details&id=748

In brief, we must update the IOS of LDN (it's expected this night), then we will
see if it mounts well. Moreover, we must upgrade the IOS of all RF because
the RF do not announce anything neither.


That's a real mess in bricks and reinforced. The interoperability of the equipments
within one manufacturer. If we had to make a cisco work with a Jinuper they would have told
me "It's Cisco Fault", "It's Juniper fault". We have only Cisco .... but it seems
that the 4 equipments are developped by different teams that do not work together :
6K, ASR 1000, ASR 9000, Nexus 7000.

... as though they were 4 manufacturers ...


Comment by OVH - Sunday, 07 November 2010, 12:11PM

We have opened a TAC in order to understand what is wrong.
Apparently, all of the links towards the distant POP (Bruxelles
and London for the moment) do not work well. All that is
internal at the datacentre is functional.

We have also another problem on the CEF:

RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail
Sun Nov 7 10:32:18.941 UTC
CEF resource availability summary state: GREEN
CEF will work normally
ipv4 shared memory resource:
CurrMode GREEN, CurrAvail 961508351 bytes, MaxAvail 1064480768 bytes
ipv6 shared memory resource:
CurrMode GREEN, CurrAvail 961508351 bytes, MaxAvail 1064480768 bytes
mpls shared memory resource:
CurrMode GREEN, CurrAvail 961508351 bytes, MaxAvail 1064480768 bytes
common shared memory resource:
CurrMode GREEN, CurrAvail 961508351 bytes, MaxAvail 1064480768 bytes
DATA_TYPE_TABLE_SET hardware resource: GREEN
DATA_TYPE_TABLE hardware resource: GREEN
DATA_TYPE_IDB hardware resource: GREEN
DATA_TYPE_IDB_EXT hardware resource: GREEN
DATA_TYPE_LEAF hardware resource: GREEN
DATA_TYPE_LOADINFO hardware resource: GREEN
DATA_TYPE_PATH_LIST hardware resource: GREEN
DATA_TYPE_NHINFO hardware resource: GREEN
DATA_TYPE_LABEL_INFO hardware resource: GREEN
DATA_TYPE_FRR_NHINFO hardware resource: GREEN
DATA_TYPE_ECD hardware resource: GREEN
DATA_TYPE_RECURSIVE_NH hardware resource: GREEN
DATA_TYPE_TUNNEL_ENDPOINT hardware resource: GREEN
DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: GREEN
DATA_TYPE_ECD_TRACKER hardware resource: GREEN
DATA_TYPE_ECD_V2 hardware resource: GREEN
DATA_TYPE_ATTRIBUTE hardware resource: GREEN
DATA_TYPE_LSPA hardware resource: GREEN
DATA_TYPE_LDI_LW hardware resource: GREEN
DATA_TYPE_LDSH_ARRAY hardware resource: GREEN
DATA_TYPE_TE_TUN_INFO hardware resource: GREEN
DATA_TYPE_DUMMY hardware resource: GREEN
DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: GREEN
DATA_TYPE_TABLE_UNRESOLVED hardware resource: GREEN
DATA_TYPE_MOL hardware resource: GREEN
DATA_TYPE_MPI hardware resource: GREEN
DATA_TYPE_SUBS_INFO hardware resource: GREEN
DATA_TYPE_GRE_TUNNEL_INFO hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9#

RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0
Sun Nov 7 10:31:58.905 UTC
CEF resource availability summary state: YELLOW
CEF will drop route updates
No. of times HW caused oor: 26
CEF entered oor at : Nov 7 10:30:16.992
CEF came out of oor at : Nov 7 10:30:06.628
ipv4 shared memory resource:
CurrMode GREEN, CurrAvail 787427327 bytes, MaxAvail 872878080 bytes
ipv6 shared memory resource:
CurrMode GREEN, CurrAvail 787427327 bytes, MaxAvail 872878080 bytes
mpls shared memory resource:
CurrMode GREEN, CurrAvail 787427327 bytes, MaxAvail 872878080 bytes
common shared memory resource:
CurrMode GREEN, CurrAvail 787427327 bytes, MaxAvail 872878080 bytes
DATA_TYPE_TABLE_SET hardware resource: YELLOW
DATA_TYPE_TABLE hardware resource: YELLOW
DATA_TYPE_IDB hardware resource: YELLOW
DATA_TYPE_IDB_EXT hardware resource: YELLOW
DATA_TYPE_LEAF hardware resource: YELLOW
DATA_TYPE_LOADINFO hardware resource: YELLOW
DATA_TYPE_PATH_LIST hardware resource: YELLOW
DATA_TYPE_NHINFO hardware resource: YELLOW
DATA_TYPE_LABEL_INFO hardware resource: YELLOW
DATA_TYPE_FRR_NHINFO hardware resource: YELLOW
DATA_TYPE_ECD hardware resource: YELLOW
DATA_TYPE_RECURSIVE_NH hardware resource: YELLOW
DATA_TYPE_TUNNEL_ENDPOINT hardware resource: YELLOW
DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: YELLOW
DATA_TYPE_ECD_TRACKER hardware resource: YELLOW
DATA_TYPE_ECD_V2 hardware resource: YELLOW
DATA_TYPE_ATTRIBUTE hardware resource: YELLOW
DATA_TYPE_LSPA hardware resource: YELLOW
DATA_TYPE_LDI_LW hardware resource: YELLOW
DATA_TYPE_LDSH_ARRAY hardware resource: YELLOW
DATA_TYPE_TE_TUN_INFO hardware resource: YELLOW
DATA_TYPE_DUMMY hardware resource: YELLOW
DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: YELLOW
DATA_TYPE_TABLE_UNRESOLVED hardware resource: YELLOW
DATA_TYPE_MOL hardware resource: YELLOW
DATA_TYPE_MPI hardware resource: YELLOW
DATA_TYPE_SUBS_INFO hardware resource: YELLOW
DATA_TYPE_GRE_TUNNEL_INFO hardware resource: YELLOW


Comment by OVH - Thursday, 11 November 2010, 11:52AM

found :) apparently the ASR cannot hold more than 380/390K routes.
As soon as we surpass the limit, it starts dropping the routes in the CEF

weird ... the TAC was informed ... we will see how they would respond.

RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0
Tue Nov 9 23:18:52.874 UTC
CEF resource availability summary state: GREEN
CEF will work normally
No. of times HW caused oor: 7
CEF entered oor at : Nov 9 23:09:37.462
CEF came out of oor at : Nov 9 23:13:12.535
ipv4 shared memory resource:
CurrMode GREEN, CurrAvail 783738455 bytes, MaxAvail 872878080 bytes
ipv6 shared memory resource:
CurrMode GREEN, CurrAvail 783738455 bytes, MaxAvail 872878080 bytes
mpls shared memory resource:
CurrMode GREEN, CurrAvail 783738455 bytes, MaxAvail 872878080 bytes
common shared memory resource:
CurrMode GREEN, CurrAvail 783738455 bytes, MaxAvail 872878080 bytes
DATA_TYPE_TABLE_SET hardware resource: GREEN
DATA_TYPE_TABLE hardware resource: GREEN
DATA_TYPE_IDB hardware resource: GREEN
DATA_TYPE_IDB_EXT hardware resource: GREEN
DATA_TYPE_LEAF hardware resource: GREEN
DATA_TYPE_LOADINFO hardware resource: GREEN
DATA_TYPE_PATH_LIST hardware resource: GREEN
DATA_TYPE_NHINFO hardware resource: GREEN
DATA_TYPE_LABEL_INFO hardware resource: GREEN
DATA_TYPE_FRR_NHINFO hardware resource: GREEN
DATA_TYPE_ECD hardware resource: GREEN
DATA_TYPE_RECURSIVE_NH hardware resource: GREEN
DATA_TYPE_TUNNEL_ENDPOINT hardware resource: GREEN
DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: GREEN
DATA_TYPE_ECD_TRACKER hardware resource: GREEN
DATA_TYPE_ECD_V2 hardware resource: GREEN
DATA_TYPE_ATTRIBUTE hardware resource: GREEN
DATA_TYPE_LSPA hardware resource: GREEN
DATA_TYPE_LDI_LW hardware resource: GREEN
DATA_TYPE_LDSH_ARRAY hardware resource: GREEN
DATA_TYPE_TE_TUN_INFO hardware resource: GREEN
DATA_TYPE_DUMMY hardware resource: GREEN
DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: GREEN
DATA_TYPE_TABLE_UNRESOLVED hardware resource: GREEN
DATA_TYPE_MOL hardware resource: GREEN
DATA_TYPE_MPI hardware resource: GREEN
DATA_TYPE_SUBS_INFO hardware resource: GREEN
DATA_TYPE_GRE_TUNNEL_INFO hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:18:58.974 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum
Tue Nov 9 23:19:01.674 UTC
BGP router identifier 94.23.2.19, local AS number 16276
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000
BGP main routing table version 5644641
BGP NSR Initial initsync version 366203 (Reached)
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 5644641 5644641 5644641 5644641 5643206 5643206

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
94.23.2.11 0 16276 156436 884 0 0 0 00:05:52 Idle (Admin)
94.23.2.12 0 16276 289588 886 0 0 0 00:01:48 Idle (Admin)
94.23.2.13 0 16276 512257 884 5643206 0 0 00:01:43 366061

RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:07.989 UTC
94.23.2.13 0 16276 512257 884 5643206 0 0 00:01:50 366061
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:10.588 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:13.438 UTC
94.23.2.13 0 16276 512257 884 5644641 0 0 00:01:55 366061
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:17.482 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:19.353 UTC
94.23.2.13 0 16276 513527 884 5644641 0 0 00:02:01 369044
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:22.170 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:24.326 UTC
94.23.2.13 0 16276 513527 885 5644641 0 0 00:02:06 369044
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:27.012 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:34.647 UTC
94.23.2.13 0 16276 513527 885 5647628 0 0 00:02:16 369044
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:19:52.753 UTC
94.23.2.13 0 16276 514756 885 5649167 0 0 00:02:34 370583
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:19:54.493 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:20:14.067 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:20:22.080 UTC
94.23.2.13 0 16276 517393 886 5650818 0 0 00:03:05 374009
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:20:27.528 UTC
DATA_TYPE_TABLE_SET hardware resource: GREEN
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:21:08.738 UTC
DATA_TYPE_TABLE_SET hardware resource: YELLOW
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:21:10.284 UTC
94.23.2.13 0 16276 528959 886 5662076 0 0 00:03:52 412381
RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0 | i DATA_TYPE_TABLE_SET
Tue Nov 9 23:21:15.491 UTC
DATA_TYPE_TABLE_SET hardware resource: YELLOW
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum | i 94.23.2.13
Tue Nov 9 23:21:17.509 UTC
94.23.2.13 0 16276 532198 886 5662076 0 0 00:03:59 420119
RP/0/RSP0/CPU0:rbx-g1-a9# sh ip bgp sum
Tue Nov 9 23:21:21.154 UTC
BGP router identifier 94.23.2.19, local AS number 16276
BGP generic scan interval 60 secs
Non-stop routing is enabled
BGP table state: Active
Table ID: 0xe0000000
BGP main routing table version 5702662
BGP NSR Initial initsync version 366203 (Reached)
BGP scan interval 60 secs

BGP is operating in STANDALONE mode.


Process RcvTblVer bRIB/RIB LabelVer ImportVer SendTblVer StandbyVer
Speaker 5702707 5701978 5701978 5701978 5662076 5662076

Neighbor Spk AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down St/PfxRcd
94.23.2.11 0 16276 156436 884 0 0 0 00:08:12 Idle (Admin)
94.23.2.12 0 16276 289588 886 0 0 0 00:04:07 Idle (Admin)
94.23.2.13 0 16276 534312 886 5662076 118 0 00:04:03 423709


Comment by OVH - Thursday, 11 November 2010, 11:57AM

Apparently the problem is due to the fact that we have a BGP configuration
which pushes the traffic towards the ip of the server directly. This way, with
the IP failover, we have many nexthop in the BGP tables.
Too much for the ASR 9000

RP/0/RSP0/CPU0:rbx-g1-a9#show bgp nexthops statistics
Wed Nov 10 16:47:00.089 UTC
Total Nexthop Processing
Time Spent: 9.428 secs

Maximum Nexthop Processing
Received: 01:46:32
Bestpaths Deleted: 0
Bestpaths Changed: 0
Time Spent: 2.029 secs

Last Notification Processing
Received: 00:03:28
Time Spent: 1.324 secs

Gateway Address Family: IPv4 Unicast
Table ID: 0xe0000000
Nexthop Count: 29848
Critical Trigger Delay: 3000msec
Non-critical Trigger Delay: 10000msec

Nexthop Version: 1, RIB version: 1

Total Critical Notifications Received: 6256
Total Non-critical Notifications Received: 33749
Bestpaths Deleted After Last Walk: 0
Bestpaths Changed After Last Walk: 0
Nexthop register:
Sync calls: 30834, last sync call: 00:00:38
Async calls: 3, last async call: 03:13:23
Nexthop unregister:
Async calls: 989, last async call: 00:00:51
Nexthop batch finish:
Calls: 31856, last finish call: 00:00:37
Nexthop flush timer:
Times started: 31828, last time flush timer started: 00:00:38
RIB update: 0 rib update runs, last update: 00:00:00
0 prefixes installed, 0 modified, 0 removed


RP/0/RSP0/CPU0:rbx-g1-a9#show controller np struct 6 summary location 0/0/cpu0
Wed Nov 10 19:04:51.590 UTC

Node: 0/0/CPU0:
----------------------------------------------------------------
NP: 0 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 1 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 2 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 3 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 4 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 5 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 6 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)
NP: 7 Struct 6: R_LDI
65476 of 65536 entries in use (65476 reserved)


Comment by OVH - Thursday, 11 November 2010, 11:59AM

We have just changed the configuration of the route reflector
by forcing "next-hop-self" towards the RF
http://status.ovh.net/?do=details&id=782


Comment by OVH - Thursday, 11 November 2010, 12:00PM

RP/0/RSP0/CPU0:rbx-g1-a9#clear bgp nexthop performance-statistics
Thu Nov 11 02:36:38.806 UTC


RP/0/RSP0/CPU0:rbx-g1-a9#show bgp nexthops statistics
Thu Nov 11 02:37:15.110 UTC
Total Nexthop Processing
Time Spent: 0.000 secs

Maximum Nexthop Processing
Received: 00:00:00
Bestpaths Deleted: 0
Bestpaths Changed: 0
Time Spent: 0.000 secs

Last Notification Processing
Received: 42y43w
Time Spent: 0.000 secs

Gateway Address Family: IPv4 Unicast
Table ID: 0xe0000000
Nexthop Count: 155
Critical Trigger Delay: 3000msec
Non-critical Trigger Delay: 10000msec

Nexthop Version: 1, RIB version: 1

Total Critical Notifications Received: 0
Total Non-critical Notifications Received: 0
Bestpaths Deleted After Last Walk: 0
Bestpaths Changed After Last Walk: 0
Nexthop register:
Sync calls: 2, last sync call: 00:00:05
Async calls: 0, last async call: 00:00:00
Nexthop unregister:
Async calls: 1, last async call: 00:00:22
Nexthop batch finish:
Calls: 37097, last finish call: 00:00:04
Nexthop flush timer:
Times started: 66463, last time flush timer started: 00:00:05
RIB update: 0 rib update runs, last update: 00:00:00
0 prefixes installed, 0 modified, 0 removed


RP/0/RSP0/CPU0:rbx-g1-a9#show controller np struct 6 summary location 0/0/cpu0
Thu Nov 11 02:39:21.139 UTC

Node: 0/0/CPU0:
----------------------------------------------------------------
NP: 0 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 1 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 2 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 3 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 4 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 5 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 6 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)
NP: 7 Struct 6: R_LDI
596 of 65536 entries in use (596 reserved)


RP/0/RSP0/CPU0:rbx-g1-a9#sh cef resource detail location 0/0/cpu0
Thu Nov 11 02:41:36.395 UTC
CEF resource availability summary state: GREEN
CEF will work normally
No. of times HW caused oor: 3
CEF entered oor at : Nov 11 01:28:50.907
CEF came out of oor at : Nov 11 02:36:21.139
ipv4 shared memory resource:
CurrMode GREEN, CurrAvail 773712739 bytes, MaxAvail 872878080 bytes
ipv6 shared memory resource:
CurrMode GREEN, CurrAvail 773712739 bytes, MaxAvail 872878080 bytes
mpls shared memory resource:
CurrMode GREEN, CurrAvail 773712739 bytes, MaxAvail 872878080 bytes
common shared memory resource:
CurrMode GREEN, CurrAvail 773712739 bytes, MaxAvail 872878080 bytes
DATA_TYPE_TABLE_SET hardware resource: GREEN
DATA_TYPE_TABLE hardware resource: GREEN
DATA_TYPE_IDB hardware resource: GREEN
DATA_TYPE_IDB_EXT hardware resource: GREEN
DATA_TYPE_LEAF hardware resource: GREEN
DATA_TYPE_LOADINFO hardware resource: GREEN
DATA_TYPE_PATH_LIST hardware resource: GREEN
DATA_TYPE_NHINFO hardware resource: GREEN
DATA_TYPE_LABEL_INFO hardware resource: GREEN
DATA_TYPE_FRR_NHINFO hardware resource: GREEN
DATA_TYPE_ECD hardware resource: GREEN
DATA_TYPE_RECURSIVE_NH hardware resource: GREEN
DATA_TYPE_TUNNEL_ENDPOINT hardware resource: GREEN
DATA_TYPE_LOCAL_TUNNEL_INTF hardware resource: GREEN
DATA_TYPE_ECD_TRACKER hardware resource: GREEN
DATA_TYPE_ECD_V2 hardware resource: GREEN
DATA_TYPE_ATTRIBUTE hardware resource: GREEN
DATA_TYPE_LSPA hardware resource: GREEN
DATA_TYPE_LDI_LW hardware resource: GREEN
DATA_TYPE_LDSH_ARRAY hardware resource: GREEN
DATA_TYPE_TE_TUN_INFO hardware resource: GREEN
DATA_TYPE_DUMMY hardware resource: GREEN
DATA_TYPE_IDB_VRF_LCL_CEF hardware resource: GREEN
DATA_TYPE_TABLE_UNRESOLVED hardware resource: GREEN
DATA_TYPE_MOL hardware resource: GREEN
DATA_TYPE_MPI hardware resource: GREEN
DATA_TYPE_SUBS_INFO hardware resource: GREEN
DATA_TYPE_GRE_TUNNEL_INFO hardware resource: GREEN