OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#15398 — vac
Incident Report for Network & Infrastructure
Resolved
We are experiencing a major issue on the vac platform. We are investigating.

Update(s):

Date: 2015-11-19 13:06:09 UTC
We've recovered the traces on Vac2 for Cisco.
Vac2 is up again.

Date: 2015-11-19 13:05:43 UTC
vac1 and 3 are up.

Date: 2015-11-19 13:05:32 UTC
The 3 Vacs have crashed again.

Date: 2015-11-19 13:05:06 UTC
Vac1 is up again, the 2 linecards are replaced.

Date: 2015-11-19 13:04:48 UTC
We're isolating VAC1 to replace linecards M2.
The protection will be managed by vac2 and 3 during the maintenance.

Date: 2015-11-19 13:04:18 UTC
Vac2 is now up to date.

Date: 2015-11-19 13:04:05 UTC
We're isolating vac2 to update it.

Date: 2015-11-19 13:03:50 UTC
Vac2 is UP.

Date: 2015-11-19 13:03:40 UTC
Vac 1 and 3 are online, we keep vac2 off for the troubleshoot with Cisco.

At the moment, it's a hard bug on cards M2.
We doubt that the 3 chassis at the same time is caused by a hardware issue.

However, we launch the RMA for vac1 while keeping troubleshooting.

Date: 2015-11-19 13:01:31 UTC
We've reloaded the cards on vac1 and 3, it worked for 20 minutes and then broke down again. We're currently with the TAC Cisco to troobleshoot.

Date: 2015-11-17 03:46:04 UTC
The malfunction reoccurred,our teams are on field investigating the situation.

Date: 2015-11-16 15:11:18 UTC
We are in the process of troubleshooting with Cisco.
The failing of loopback logs is not the root-cause of default on linecards, but its consequence.

We continue the investigations to determine if the cause is hard or soft

Date: 2015-11-16 15:09:20 UTC
At 3:45 GMT+1, we had a simultaneous crash on linecards from 3 vacs RBX, SBG and BHS

2015 Nov 15 05:04:11 admin %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:4 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Pack
ets lost on the SUP in the transmit direction
2015 Nov 15 05:04:11 admin %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.31048
2015 Nov 15 05:06:31 admin %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:3 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Pack
ets lost on the SUP in the transmit direction
2015 Nov 15 05:06:31 admin %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.32607
2015 Nov 15 05:07:01 admin %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:4 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Pack
ets lost on the SUP in the transmit direction
2015 Nov 15 05:07:02 admin %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on vsh.468
2015 Nov 15 05:20:43 admin %AUTHPRIV-3-SYSTEM_MSG: pam_aaa:Authentication failed from console - login


2015 Nov 15 05:06:09 admin-vac2 %$ VDC-1 %$ %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:4 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Packets lost on the SUP in the transmit direction
2015 Nov 15 05:06:12 admin-vac2 %$ VDC-1 %$ %DIAG_PORT_LB-2-REWRITE_ENGINE_LOOPBACK_TEST_FAIL: Module:3 Test:RewriteEngine Loopback failed 10 consecutive times. Faulty module:Module 1 Error:Loopback test failed. Packets lost on the SUP in the transmit direction


We have reloaded the linecards.

4:30 GMT+1 , VAC1 was operational.
5:00 GMT+1, the service was totally restored.

We will work with the equipment supplier in order to find the cause of this crash and how it could happen again.
Posted Nov 15, 2015 - 04:45 UTC