OVHcloud Network Status

Current status
Legend
  • Operational
  • Degraded performance
  • Partial Outage
  • Major Outage
  • Under maintenance
FS#5443 — protections against UDP attacks
Scheduled Maintenance Report for Network & Infrastructure
Completed
Last month we have noticed a very very strong increase of the attacks number in our network.
We came now to about 30 attacks per day with 3 to 5 attacks which they have many consequences on the quality of services for our clients.

It is therefore, necessary for OVH to fix these last attacks problems.

We have placed protections against the attacks on the UDP layer.
The incoming traffic is limited to 50Mbps per IP source on UDP.


Update(s):

Date: 2011-08-07 21:23:49 UTC
We are generalising the limitation to 20Mbps per IP to UDP on the routers managing OVH incoming traffic.

Date: 2011-06-03 01:40:49 UTC
We have decreased the incoming traffic in UDP by IP from to 20Mbps.
We'll see whether it will prevent remaining attacks.

Date: 2011-05-19 22:22:33 UTC

The settings are very effective. 1.2Gbps from 120 IP are purged by 10Mbps :)

We sucked the attack on a router apart to analyse it. UDP packet of 1 byte.

The list of the IP performing attack:
217.172.172.246 89.188.109.158 202.101.26.194 200.188.178.131 184.172.164.235 202.169.54.69 175.136.235.180 208.115.199.162 202.134.4.74 202.66.159.22 193.184.64.18 208.115.192.78
212.68.85.36 213.229.106.136 202.117.3.81 46.166.128.236 182.48.57.191 110.4.107.207 110.164.59.211 208.115.205.183 121.15.213.209 200.188.178.132 210.94.214.198
81.29.176.70 202.28.24.203 184.82.37.12 74.81.173.152 195.191.169.254 202.89.26.4 202.166.200.59 202.91.25.135 202.119.46.151 202.60.7.34 219.95.150.3
85.25.64.41 201.20.22.76 202.5.202.2 212.154.211.174 202.45.134.100 121.15.129.65 184.82.33.4 94.20.30.58 202.159.29.138 202.52.7.118 202.173.248.246
202.28.46.63 212.76.85.20 84.246.226.241 202.46.15.109 113.53.236.90 62.189.143.103 79.98.27.155 74.63.200.195 193.137.79.77 202.67.159.241 202.176.89.134
202.104.188.88 202.194.15.193 193.204.79.147 195.24.77.77 208.115.199.58 122.117.153.112 202.62.63.11 208.115.199.168 120.50.2.22 200.195.197.217 202.9.76.137
202.194.15.192 84.22.33.10 69.162.154.10 202.168.228.134 210.112.121.16 121.14.38.60 210.243.188.138 202.166.193.43 208.115.202.147 202.44.53.64 218.28.238.153
61.147.124.3 173.224.215.71 202.75.6.111 212.154.211.19 74.63.239.209 123.108.97.73 202.120.94.194 202.64.79.115 202.123.234.165 202.226.241.65 113.36.152.23
72.18.205.208 118.217.181.148 202.102.240.66 202.147.203.11 193.226.98.10 202.71.111.87 74.63.228.59 208.115.200.59 213.251.133.55 202.59.162.133 202.80.119.227
204.227.183.239 222.66.117.194 59.186.116.198 202.104.147.38 217.6.23.156 81.176.236.132 121.12.253.132 74.63.228.48 221.4.198.238 200.103.106.189 113.36.152.18
173.224.212.91 87.117.236.206 202.53.64.153 202.96.186.199 119.145.149.239 202.55.224.2 208.115.224.247 213.186.113.80 85.235.195.174 202.120.108.15 208.100.9.121

00:05:41.025106 ip 217.172.172.246.60802 > 178.32.174.7.52829: udp, length 1
00:05:41.025113 ip 217.172.172.246.60802 > 178.32.174.7.52092: udp, length 1
00:05:41.025117 ip 217.172.172.246.60802 > 178.32.174.7.57685: udp, length 1
00:05:41.025125 ip 217.172.172.246.60802 > 178.32.174.7.19995: udp, length 1
00:05:41.025132 ip 217.172.172.246.60802 > 178.32.174.7.62144: udp, length 1
00:05:41.029163 ip 217.172.172.246.60802 > 178.32.174.7.26174: udp, length 1
00:05:41.033086 ip 217.172.172.246.60802 > 178.32.174.7.51982: udp, length 1
00:05:41.033101 ip 217.172.172.246.60802 > 178.32.174.7.19547: udp, length 1
00:05:41.040862 ip 217.172.172.246.60802 > 178.32.174.7.43119: udp, length 1
00:05:41.040883 ip 217.172.172.246.60802 > 178.32.174.7.60090: udp, length 1

We played well and refined the rules. We will therefore block who performed the attack by blocking the destination IP.

Date: 2011-05-19 15:12:45 UTC
Following to an ongoing attack on an IP, we
have tuned settings and we have decreased the authorized
burst during an attack from 10000 to 8000.
The attack increased from 70Mbps to 10Mbps. It continues
but no longer has any impact on the server.

#sh inter f0/15 | i 30 sec
30 second input rate 2822000 bits/sec, 303 packets/sec
30 second output rate 62419000 bits/sec, 121785 packets/sec
[...]
#sh inter f0/15 | i 30 sec
30 second input rate 5422000 bits/sec, 585 packets/sec
30 second output rate 10334000 bits/sec, 20076 packets/sec

Do not hesitate to forward problems if it exists.





Date: 2011-05-13 08:10:07 UTC
Hello,
Following to the protections update against attacks
on the UDP layer, from 24h we have not had to intervene
to protect the infrastructure. We received tens of
usual attack that did not have an impact on our
customers.

We can than estimate that the settings into place are correct
and sufficient. Fast, well done:)

Yes! Let's hope it lasts:)

The summary:
- we set up protection on the input of
our network: we limit the UDP traffic to 50Mbps by
IP source. ie that a specific IP on the Internet
can not send to ovh network more than 50Mbps by
UDP.

- we have put in place protection on routers
of datacenters: we limit the UDP traffic to 50Mbps by
IP destination. ie a specific IP at OVH
can not receive more than 50Mbps from the Internet by
UDP.

The recall of protections is already into place (since 1-2 years):
- we have a restriction to 32Kbps by IP source to
OVH on ICMP layer and TCP/SYN (with some exceptions).

VPS and mC have the following protections:
- 100Mbps by IP on TCP
- 5Mbps by IP on UDP
- 32Kbps by IP on ICMP

There are no other limitations and are not foreseen.

We had a nice welcome for the update of
these protections. 1 client was not happy and we
've got lots of feedback with an \"Ouff. I think
these protections creates a nice added value of our

offers because they strengthen the services security that
our customers propose. Whether it's a game server,
a website or a DSL connection, receive a DoS from
a competitor is very unpleasant. With OVH you
are now protected against your moods of your competitors.

Kind regards,
Octave.

Date: 2011-05-13 07:48:31 UTC
we will activate protections on the routers
of RBX2, and RBX3 RBX4

vss-1/1b: done
vss-2/2b: done
vss-3/3b/4: done
vss-5a/5b: done



Date: 2011-05-12 14:02:44 UTC
It will activate the protections on the routers in
datacenters:

vrack: done
HG 2010/2011: already done
pCC: done





Date: 2011-05-12 13:58:51 UTC
Hello,
At the entrance to the backbone, we have just change
settings. We removed the filter on the whole
IP layer to keep no more than the UDP.

Thus, an IP on the Internet is limited to 50Mbps in UDP
to the entire OVH network.

If you have problems, it's necessary to forward them.
This is not because we must manage an emergency that
can not refine after. It's always the same
email in case of the man's death's risk: oles@ovh.net

Early afternoon, we'll continue to refine
settings to reach the 3 new rules at the end:

- limitation in UDP on IP source to OVH,
currently it is limited to 50Mbps and we will try
to descend to 20Mbps around 14:00

- limitation in UDP on IP destination to OVH
is currently implemented on the network HG
to 50Mbps. we do not yet know if it's useful and
whether it's necessary to refine, to put that on all routers.

- limitation on UDP on IP source with OVH to the Internet
not yet established. the aim is to prevent that an
OVH server sends an attack towards the Internet.

Kind regards
Octave.


Date: 2011-05-12 13:35:56 UTC
Entering the backbone, we removed
protection over the IP layer to
keep more than UDP.



Date: 2011-05-11 23:36:54 UTC
Regarding the attacks quantity we are dealing with everyday, we decided to dig up the hatchet :( it isn't possible.
Nothing today, we are in up to 30 attacks and there's 5 of our customers' networks that are impacted with the temporarily deterioration of the service.

therefore:

An IP source (Internet) could not send towards the OVH network, no more than 50Mbps on all the IP layer. We are looking forward to apply it on the UDP layer.

We have also added a limitation on the HG network on the destination IP in UDP for all IP out of OVH to 50Mbps.

If you have any further question, please do not hesitate to contact oles@ovh.net, noc@ovh.net.


Date: 2011-05-11 23:27:32 UTC
We have temporarily added limitations on all the IP layer. We are limiting the bandwidth to 50Mbps of input per IP source.

Meanwhile, we have added a limitation on the HG network in the destination's IP to 50Mbps on UDP out of OVH's IP.
Posted May 11, 2011 - 23:24 UTC