OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#5274 — rbx-1-6k/rbx-2-6k
Scheduled Maintenance
Report for Network & Infrastructure
Completed
We are going to update the 2 routers in RBX1 rbx-1-6k and rbx-2-6k.
The aim is to have a stabilized IPv6.
We are dealing with rbx-1-6k
rbx-1-6k uptime is 32 weeks, 5 days, 12 minutes
rbx-2-6k uptime is 1 year, 9 weeks, 4 days, 11 hours, 4 minutes
Update(s):
Date: 2011-03-28 10:00:26 UTC
Deactivating the net-flow fixes the problem. this
will give us a few days to prepare the hardware
replacements.
Date: 2011-03-28 09:42:52 UTC
rbx-2-6k(config)#no power enable module 8
%Power admin state updated
rbx-2-6k(config)# power enable module 8
%Power admin state updated
rbx-2-6k(config)#
Date: 2011-03-28 09:42:39 UTC
rbx-1-6k(config)#no mls flow ipv
rbx-1-6k(config)#no mls flow ipv6
rbx-2-6k(config)#no mls flow ip
rbx-2-6k(config)#no mls flow ipv
Date: 2011-03-28 09:42:29 UTC
rbx-2-6k(config)#no power enable module 6
%Power admin state updated
rbx-2-6k(config)# power enable module 6
%Power admin state updated
Date: 2011-03-28 09:42:15 UTC
Mar 28 09:49:10 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:49:28 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:49:39 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:49:44 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:50:02 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:50:13 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:50:17 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:50:36 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:50:50 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:50:50 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:51:09 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:51:21 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:51:25 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:51:43 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:51:59 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:52:03 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:52:26 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:52:31 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:52:36 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:52:58 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:53:07 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:53:07 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:53:37 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:53:37 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:53:37 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:54:13 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:54:13 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:54:17 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Date: 2011-03-28 06:37:26 UTC
well we have 5 DCF to change ... we did well today ...
Meanwhile, we are going to disable this monitoring. Anyway, cards need to be changed.
A/I - Monitoring is active / Monitoring is inactive
13) TestMacNotification -------------> M**N****A*** 000 00:00:15.00 10
rbx-2-6k(config)#no diagnostic monitor module 6 test 13
rbx-2-6k(config)#no diagnostic monitor module 8 test 13
rbx-2-6k(config)#no diagnostic monitor module 9 test 13
rbx-1-6k(config)#no diagnostic monitor module 8 test 13
rbx-1-6k(config)#no diagnostic monitor module 9 test 13
13) TestMacNotification -------------> M**N****I*** 000 00:00:15.00 10
Date: 2011-03-28 06:31:44 UTC
thereafter the update, we have detected errors on some cards DCF in these 2 routers.
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/diagtest.html#wp1014355
rbx-2
Mar 28 05:53:55 GMT: %DIAG-SP-3-TEST_FAIL: Module 8: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 05:59:04 GMT: %DIAG-SP-3-TEST_FAIL: Module 6: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:04:11 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:09:23 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:14:31 GMT: %DIAG-SP-3-TEST_FAIL: Module 8: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:19:34 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
rbx-1
Mar 28 06:08:26 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:13:58 GMT: %DIAG-SP-3-TEST_FAIL: Module 8: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:19:23 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Date: 2011-03-28 06:30:02 UTC
done
Date: 2011-03-28 06:29:53 UTC
We're dealing with rbx-2-6k
Date: 2011-03-28 06:29:30 UTC
rbx-1-6k done
The aim is to have a stabilized IPv6.
We are dealing with rbx-1-6k
rbx-1-6k uptime is 32 weeks, 5 days, 12 minutes
rbx-2-6k uptime is 1 year, 9 weeks, 4 days, 11 hours, 4 minutes
Update(s):
Date: 2011-03-28 10:00:26 UTC
Deactivating the net-flow fixes the problem. this
will give us a few days to prepare the hardware
replacements.
Date: 2011-03-28 09:42:52 UTC
rbx-2-6k(config)#no power enable module 8
%Power admin state updated
rbx-2-6k(config)# power enable module 8
%Power admin state updated
rbx-2-6k(config)#
Date: 2011-03-28 09:42:39 UTC
rbx-1-6k(config)#no mls flow ipv
rbx-1-6k(config)#no mls flow ipv6
rbx-2-6k(config)#no mls flow ip
rbx-2-6k(config)#no mls flow ipv
Date: 2011-03-28 09:42:29 UTC
rbx-2-6k(config)#no power enable module 6
%Power admin state updated
rbx-2-6k(config)# power enable module 6
%Power admin state updated
Date: 2011-03-28 09:42:15 UTC
Mar 28 09:49:10 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:49:28 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:49:39 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:49:44 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:50:02 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:50:13 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:50:17 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:50:36 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:50:50 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:50:50 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:51:09 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:51:21 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:51:25 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:51:43 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:51:59 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:52:03 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:52:26 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:52:31 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:52:36 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:52:58 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:53:07 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:53:07 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:53:37 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:53:37 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Mar 28 09:53:37 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:54:13 GMT: %IPC-DFC8-5-WATERMARK: 4621 messages pending in rcv for the port Card8/0:Request(2080000.7) seat 2080000
Mar 28 09:54:13 GMT: %IPC-DFC9-5-WATERMARK: 4621 messages pending in rcv for the port Card9/0:Request(2090000.7) seat 2090000
Mar 28 09:54:17 GMT: %IPC-DFC6-5-WATERMARK: 4621 messages pending in rcv for the port Card6/0:Request(2060000.7) seat 2060000
Date: 2011-03-28 06:37:26 UTC
well we have 5 DCF to change ... we did well today ...
Meanwhile, we are going to disable this monitoring. Anyway, cards need to be changed.
A/I - Monitoring is active / Monitoring is inactive
13) TestMacNotification -------------> M**N****A*** 000 00:00:15.00 10
rbx-2-6k(config)#no diagnostic monitor module 6 test 13
rbx-2-6k(config)#no diagnostic monitor module 8 test 13
rbx-2-6k(config)#no diagnostic monitor module 9 test 13
rbx-1-6k(config)#no diagnostic monitor module 8 test 13
rbx-1-6k(config)#no diagnostic monitor module 9 test 13
13) TestMacNotification -------------> M**N****I*** 000 00:00:15.00 10
Date: 2011-03-28 06:31:44 UTC
thereafter the update, we have detected errors on some cards DCF in these 2 routers.
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SXF/native/configuration/guide/diagtest.html#wp1014355
rbx-2
Mar 28 05:53:55 GMT: %DIAG-SP-3-TEST_FAIL: Module 8: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 05:59:04 GMT: %DIAG-SP-3-TEST_FAIL: Module 6: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:04:11 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:09:23 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:14:31 GMT: %DIAG-SP-3-TEST_FAIL: Module 8: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:19:34 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
rbx-1
Mar 28 06:08:26 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:13:58 GMT: %DIAG-SP-3-TEST_FAIL: Module 8: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Mar 28 06:19:23 GMT: %DIAG-SP-3-TEST_FAIL: Module 9: TestMacNotification{ID=13} has failed. Error code = 0x3C (DIAG_L2_ADD_ENTRY_ERROR)
Date: 2011-03-28 06:30:02 UTC
done
Date: 2011-03-28 06:29:53 UTC
We're dealing with rbx-2-6k
Date: 2011-03-28 06:29:30 UTC
rbx-1-6k done