OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
vRack EU
Incident Report for Network & Infrastructure
Resolved
We have BPG/links that are flapping on some of our g5-a9. It is also affecting the vRack.

Update(s):

Date: 2016-11-22 13:50:25 UTC
Hello,

ACLs allow for blocking/accepting traffic to and from switches/routers.

When reconfiguring the ACLs on the switches which manage the vRacks, we had a debug of
deployment of the ACLs on some switches in the scripts that manage this update.
The lines are not correctly configured on 50% of the switches, which has provoked a
blockage of access to these switches.
So the switches stopped communicating with the rest of the infrastructure.
We had to manually intervene on these switches, with a serial cable, to put
back the lines of the ACL configurations.
This explains the (very) long time that was needed to correct the issue.

We will modify the scripts to verify the status of the configuration before
applying the modification in order to make sure that the configuration in place
corresponds to the expected state prior to the update.
Only then will we apply the modifications.
This verification should have already been coded.
Overconfidence is deadly when it comes to the network.

We apologize for the disruption of service.

Best Regards,
Octave

Date: 2016-11-21 15:50:31 UTC
All switches are back now.

Date: 2016-11-21 15:42:49 UTC
Some additional details:

This failure was caused by a bug in the ACL update system of some switches.

We have identified the specific causes of the problem that are found in the configuration of these switches and a problem in the code checking.

Date: 2016-11-21 15:26:54 UTC
There are 6 switch that remains affected by this issue

Date: 2016-11-21 15:17:53 UTC
We found the source, it seems to be isolating to the following switches:

gra1-102a-a70-vrack
gra1-102a-n3-vrack
gra1-102-n56-vrack
rbx2-102a-a70-vrack
rbx2-102a-n3-vrack
rbx2-102-n6-vrack
rbx2-104a-a70-vrack
rbx2-104-n6-vrack
rbx2-105a-a70-vrack
rbx2-105-n6-vrack
rbx3-101a-a70-vrack
rbx3-101-n6-vrack
rbx3-102a-a70-vrack
rbx3-102-n6-vrack
rbx4-101a-a70-vrack
rbx4-101-n6-vrack
rbx4-110a-a70-vrack
rbx4-110-n6-vrack
rbx4-116a-a70-vrack
rbx4-116-n6-vrack
rbx6-103a-a70-vrack
rbx6-103-n56-vrack
sbg2-102a-a70-vrack
sbg2-102a-n3-vrack
sbg2-102-n56-vrack
Posted Nov 21, 2016 - 14:55 UTC