OVHcloud Network Status
Current status
Legend
- Operational
- Degraded performance
- Partial Outage
- Major Outage
- Under maintenance
FS#5807 — Moving th1 towards th2
Scheduled Maintenance
Report for Network & Infrastructure
Completed
We will leave the POP of th1 and move the set of our equipments towards a new room at th2 especially the th1-1-6k router and our Infinera DTN. The th1-1 router will be replaced with a new th2-5-6k and the DTN will be moved during the night of Thursday till Friday (from 22 to 23).
We will therefore move the arrival points of the fibres towards Roubaix, Francfort and p19 in addition to the intercos which we had with different operators on th1: Tata, Orange, OTI, Bouygues, Seabone.
The first stage of this migration is already done, the th2-5-6k router being ready to connect the different services which will be migrated. We will go at the big part tomorrow:
- morning: cut off of paris<>francfort (back up link francfort), the p19 traffic towards FRA and Eastern Europe will be temporarily routed via Roubaix
- morning: activation of 4x10G between th1-1 and th2-5
- 14:00: migration of TATA ports of th1-1 towards th2-5
- afternoon: installation of a new empty Infinera chassis on th2 and preparation of the maintenance tasks for the night.
- 00:00 migration of the optical links long distance of th1 towards th2: Roubaix<>Paris and Francfort<>Paris. The Paris<>Roubaix traffic continues to move via our second link Roubaix<>Globalswitch. Reactivation of all the 10G links towards Roubaix and Francfort on the new th2-5-6k router.
Update(s):
Date: 2011-09-23 13:02:59 UTC
The traffic is restablished on th2-1-6k to TATA, Orange, Seabone and Bouygues.
We still have a 4-5 working phases on Paris which we are going to do
in the next few days and before the end of the month.
- We must empty th1
- add ASR 9010 to th2
- add ASR 9010 to gsw
- connect all the 6k on the 2 ASR
- connect all NRA on the ASR
Date: 2011-09-23 12:58:56 UTC
End of the workings for tonight
Date: 2011-09-23 12:58:25 UTC
This 4 links are now reconnected to the th2-5-6k.
Date: 2011-09-23 12:57:49 UTC
Done.
We migrate now the circuits to Orange, Seabone and Bouygues.
Date: 2011-09-23 12:57:05 UTC
The traffic flows now from Roubaix to Paris on the 2 links.
The connexions between th1-1-6k, th2-1-6k and th2-5-6k are
in triangle and active. The traffic flows normally.
Still remains to lenghten the peering with Orange from th1 to
th2 (2x10G), the peering with Bouygues and the transit seabone
until the links migrations are done by these operators from th1 to th2.
However the OTI transit stays on th1.
Date: 2011-09-23 12:50:27 UTC
Done. We tackle the last steps of the working : migration of the link th2-1 <> th1-1 to th2-5 <> th1-1.
Date: 2011-09-23 12:49:05 UTC
We reactivate the whole 10G circuits.
Date: 2011-09-23 12:48:36 UTC
Interoute had a problem from its side on the cross-co toward us. it's fixed and the link Paris<>Francfort is up but with a too high signal in reception. We are
checking the reduction in input on an 5dB attenuator instead of 10 to be in order.
Date: 2011-09-23 12:45:44 UTC
The link Paris<>Roubaix is operational. We remount the 10G . We have a problem at the level of Paris<>Francfort (best reduction in one sens, worse in the other).
We are investigating at the level of the cables.
Date: 2011-09-23 12:43:54 UTC
We are now on the th2 site. The installation of the equipments are in progress.
Date: 2011-09-23 12:43:11 UTC
The infinera Chassis on th1 is shut down. The cards are extracted and prepared for the transport.
Date: 2011-09-23 12:42:27 UTC
It was not expected tonight. We will do it later.
Date: 2011-09-23 12:41:37 UTC
We cut the trafic th1-1<>p19-7 to migrate the link.
Date: 2011-09-23 12:41:02 UTC
The auto gain adjustment algorithm is operationnal again in the direction Paris>Roubaix. We added an offset of +2 on Etricourt to be in the range of normal working of the amplis on Willerval and Roubaix (which is 32dB max) . We should have an ulterior intervention to fix the high span loss problem
between Etricourt and Willerval.
Date: 2011-09-23 12:36:06 UTC
The setting on the offsets were done but are still not taken in consideration. We are doing a warm reset on the BMM of Telehouse in order to force the update
of the gains on the OAMs in the direction Paris>Roubaix.
Date: 2011-09-23 12:34:46 UTC
After the reboot of the OAM card, the links between Paris/Roubaix
were interrupted during 15 seconds. We took advantage of that
to make them out of the routing and then see how the trafic is
going to flow now.
We are going to reactivate a 40G between rbx-1/gsw-1 to add a little bit of capacity.
w8
Date: 2011-09-23 12:31:35 UTC
70% of the data of pcc-000659 were copied.
Date: 2011-09-23 12:28:56 UTC
After the reboot of the OAM card on Nogent, the alarm disappeared. The card is back in its normal functionning.
Date: 2011-09-23 12:21:21 UTC
We will prepare the cut down of the link Roubaix / Paris to th1-1-6k.
Traffic will flow naturally (or not) through the rest of the network.
According to this flow we will help the network to evacuate the
traffic to the Internet without packet loss.
Otherwise,do not hesitate to send us email with your IP oles@ovh.net
Date: 2011-09-23 12:12:24 UTC
We are making modifications on the optical link between Paris<>Roubaix
Date: 2011-09-23 12:11:17 UTC
We have redirected the TATA trafic from London to Amesterdam
Date: 2011-09-23 10:18:05 UTC
The 3x10G TATA have been raised on the new router th2-5-6k.We have also made a temporary link 4x10G th1-1 <> th2-5.
From the trans point of view,The new Infinera chassis is in place empty and waiting for the cards that we will bring this night of th1.
The sequence of events this night:
11PM:Corrective actions on the link Paris <> Roubaix according to the Infinera recommendations (offset settings,warm reboot of an OAM card on the Nogent site )
- 0:00 if everything went well and the link has remained stable, powering down the chassis to th1, transfer cards on th2
- 01:00 fiber routes migration to Roubaix and Francfort pn th2 in coordination with Interoute
- 1:30 Infinera DTN reactivation on th2 then reactivation of 10G circuits to Roubaix and Frankfurt
- 2:00 activation of p19 2x10G-7 <> th2-5 (DWDM)
- 02:30 check the balancing of traffic on the backbone
Date: 2011-09-22 15:22:22 UTC
The TATA traffic has moved to GBLX at Paris. It has been just adjusted in order to
be out on TATA on London. We are working with TATA in order to reactivate the 3 10G ports
at Paris on TH2 instead of TH1. We will not move again from there till tomorrow in the
morning.
Date: 2011-09-22 15:18:19 UTC
link th1-1 <> fra-5 cut off, we will cut off now the TATA traffic on th1-1
Date: 2011-09-22 15:05:49 UTC
We will cut off the Paris<>Francfort link then the 3x10G TATA on th1
We will therefore move the arrival points of the fibres towards Roubaix, Francfort and p19 in addition to the intercos which we had with different operators on th1: Tata, Orange, OTI, Bouygues, Seabone.
The first stage of this migration is already done, the th2-5-6k router being ready to connect the different services which will be migrated. We will go at the big part tomorrow:
- morning: cut off of paris<>francfort (back up link francfort), the p19 traffic towards FRA and Eastern Europe will be temporarily routed via Roubaix
- morning: activation of 4x10G between th1-1 and th2-5
- 14:00: migration of TATA ports of th1-1 towards th2-5
- afternoon: installation of a new empty Infinera chassis on th2 and preparation of the maintenance tasks for the night.
- 00:00 migration of the optical links long distance of th1 towards th2: Roubaix<>Paris and Francfort<>Paris. The Paris<>Roubaix traffic continues to move via our second link Roubaix<>Globalswitch. Reactivation of all the 10G links towards Roubaix and Francfort on the new th2-5-6k router.
Update(s):
Date: 2011-09-23 13:02:59 UTC
The traffic is restablished on th2-1-6k to TATA, Orange, Seabone and Bouygues.
We still have a 4-5 working phases on Paris which we are going to do
in the next few days and before the end of the month.
- We must empty th1
- add ASR 9010 to th2
- add ASR 9010 to gsw
- connect all the 6k on the 2 ASR
- connect all NRA on the ASR
Date: 2011-09-23 12:58:56 UTC
End of the workings for tonight
Date: 2011-09-23 12:58:25 UTC
This 4 links are now reconnected to the th2-5-6k.
Date: 2011-09-23 12:57:49 UTC
Done.
We migrate now the circuits to Orange, Seabone and Bouygues.
Date: 2011-09-23 12:57:05 UTC
The traffic flows now from Roubaix to Paris on the 2 links.
The connexions between th1-1-6k, th2-1-6k and th2-5-6k are
in triangle and active. The traffic flows normally.
Still remains to lenghten the peering with Orange from th1 to
th2 (2x10G), the peering with Bouygues and the transit seabone
until the links migrations are done by these operators from th1 to th2.
However the OTI transit stays on th1.
Date: 2011-09-23 12:50:27 UTC
Done. We tackle the last steps of the working : migration of the link th2-1 <> th1-1 to th2-5 <> th1-1.
Date: 2011-09-23 12:49:05 UTC
We reactivate the whole 10G circuits.
Date: 2011-09-23 12:48:36 UTC
Interoute had a problem from its side on the cross-co toward us. it's fixed and the link Paris<>Francfort is up but with a too high signal in reception. We are
checking the reduction in input on an 5dB attenuator instead of 10 to be in order.
Date: 2011-09-23 12:45:44 UTC
The link Paris<>Roubaix is operational. We remount the 10G . We have a problem at the level of Paris<>Francfort (best reduction in one sens, worse in the other).
We are investigating at the level of the cables.
Date: 2011-09-23 12:43:54 UTC
We are now on the th2 site. The installation of the equipments are in progress.
Date: 2011-09-23 12:43:11 UTC
The infinera Chassis on th1 is shut down. The cards are extracted and prepared for the transport.
Date: 2011-09-23 12:42:27 UTC
It was not expected tonight. We will do it later.
Date: 2011-09-23 12:41:37 UTC
We cut the trafic th1-1<>p19-7 to migrate the link.
Date: 2011-09-23 12:41:02 UTC
The auto gain adjustment algorithm is operationnal again in the direction Paris>Roubaix. We added an offset of +2 on Etricourt to be in the range of normal working of the amplis on Willerval and Roubaix (which is 32dB max) . We should have an ulterior intervention to fix the high span loss problem
between Etricourt and Willerval.
Date: 2011-09-23 12:36:06 UTC
The setting on the offsets were done but are still not taken in consideration. We are doing a warm reset on the BMM of Telehouse in order to force the update
of the gains on the OAMs in the direction Paris>Roubaix.
Date: 2011-09-23 12:34:46 UTC
After the reboot of the OAM card, the links between Paris/Roubaix
were interrupted during 15 seconds. We took advantage of that
to make them out of the routing and then see how the trafic is
going to flow now.
We are going to reactivate a 40G between rbx-1/gsw-1 to add a little bit of capacity.
w8
Date: 2011-09-23 12:31:35 UTC
70% of the data of pcc-000659 were copied.
Date: 2011-09-23 12:28:56 UTC
After the reboot of the OAM card on Nogent, the alarm disappeared. The card is back in its normal functionning.
Date: 2011-09-23 12:21:21 UTC
We will prepare the cut down of the link Roubaix / Paris to th1-1-6k.
Traffic will flow naturally (or not) through the rest of the network.
According to this flow we will help the network to evacuate the
traffic to the Internet without packet loss.
Otherwise,do not hesitate to send us email with your IP oles@ovh.net
Date: 2011-09-23 12:12:24 UTC
We are making modifications on the optical link between Paris<>Roubaix
Date: 2011-09-23 12:11:17 UTC
We have redirected the TATA trafic from London to Amesterdam
Date: 2011-09-23 10:18:05 UTC
The 3x10G TATA have been raised on the new router th2-5-6k.We have also made a temporary link 4x10G th1-1 <> th2-5.
From the trans point of view,The new Infinera chassis is in place empty and waiting for the cards that we will bring this night of th1.
The sequence of events this night:
11PM:Corrective actions on the link Paris <> Roubaix according to the Infinera recommendations (offset settings,warm reboot of an OAM card on the Nogent site )
- 0:00 if everything went well and the link has remained stable, powering down the chassis to th1, transfer cards on th2
- 01:00 fiber routes migration to Roubaix and Francfort pn th2 in coordination with Interoute
- 1:30 Infinera DTN reactivation on th2 then reactivation of 10G circuits to Roubaix and Frankfurt
- 2:00 activation of p19 2x10G-7 <> th2-5 (DWDM)
- 02:30 check the balancing of traffic on the backbone
Date: 2011-09-22 15:22:22 UTC
The TATA traffic has moved to GBLX at Paris. It has been just adjusted in order to
be out on TATA on London. We are working with TATA in order to reactivate the 3 10G ports
at Paris on TH2 instead of TH1. We will not move again from there till tomorrow in the
morning.
Date: 2011-09-22 15:18:19 UTC
link th1-1 <> fra-5 cut off, we will cut off now the TATA traffic on th1-1
Date: 2011-09-22 15:05:49 UTC
We will cut off the Paris<>Francfort link then the 3x10G TATA on th1