rssLink RSS for all categories
 
icon_red
icon_green
icon_red
icon_red
icon_blue
icon_green
icon_green
icon_red
icon_red
icon_green
icon_orange
icon_green
icon_green
icon_green
icon_red
icon_blue
icon_red
icon_orange
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_red
icon_orange
icon_green
 

FS#10862 — FS#14762 — vac1

Attached to Project— Anti-DDoS
Incident
CLOSED
100%
The VAC1 not working properly. Arbor VAC1
seems to have a problem. We just cut it.
The anti-DDoS is made on VAC2 and VAC3.
Date:  Monday, 21 September 2015, 16:47PM
Reason for closing:  Done
Comment by OVH - Monday, 21 September 2015, 16:37PM

TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=ACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161654 Alert 2161654 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=ACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=INACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161653 Alert 2161653 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=ACTIVE tmsx1.2=INACTIVE tmsx1.4=ACTIVE tmsx1.6=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161652 Alert 2161652 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=INACTIVE tmsx1.2=INACTIVE tmsx1.4=INACTIVE tmsx1.6=ACTIVE) Sep 20 19:58 - 20:00
(0:02) None Show all annotations
2161616 Alert 2161616 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=INACTIVE tmsx1.3=ACTIVE tmsx1.5=ACTIVE tmsx1.7=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161615 Alert 2161615 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Degraded' (Logical port DEGRADED, members: tmsx1.1=INACTIVE tmsx1.3=ACTIVE tmsx1.5=ACTIVE tmsx1.7=ACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161614 Alert 2161614 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=INACTIVE tmsx1.2=ACTIVE tmsx1.4=INACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161613 Alert 2161613 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Degraded' (Logical port DEGRADED, members: tmsx1.0=ACTIVE tmsx1.2=ACTIVE tmsx1.4=ACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161612 Alert 2161612 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical1' is 'Down' (Logical port INACTIVE, members: tmsx1.1=INACTIVE tmsx1.3=INACTIVE tmsx1.5=INACTIVE tmsx1.7=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations
2161611 Alert 2161611 Medium TMS Fault
Appliance: vac1-10-tms
Interface Link 'logical0' is 'Down' (Logical port INACTIVE, members: tmsx1.0=INACTIVE tmsx1.2=INACTIVE tmsx1.4=INACTIVE tmsx1.6=INACTIVE) Sep 20 19:49 - 19:51
(0:02) None Show all annotations


Comment by OVH - Monday, 21 September 2015, 16:38PM

The Cisco router does not see these DOWN. It must
be an internal problem in Arbor.


Comment by OVH - Monday, 21 September 2015, 16:38PM

Sep 20 09:40:01 apm-2-1 apm-1 pktengine[23602]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-1-0 apm-0 pktengine[3009]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-2-0 apm-0 pktengine[23610]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-2-1 apm-1 pktengine[23610]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-1 apm-1 pktengine[23583]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-0 apm-0 pktengine[23608]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable
Sep 20 09:40:01 apm-0-1 apm-1 pktengine[23618]: [W] #SEND-REPLY-FAILED -1 Resource temporarily unavailable


Comment by OVH - Monday, 21 September 2015, 16:39PM

We will reboot.

admin@vac1-10-tms:/# services tms stop
Stopping Peakflow TMS services....................................................done.
admin@vac1-10-tms:/# re
now Reload without confirmation
<cr> Reload with confirmation
admin@vac1-10-tms:/# re now
094: Rebooting the system..
Broadcast message from root (pts/8) (Sun Sep 20 18:17:35 2015):

The system is going down for reboot NOW!
Connection to vac1-10-tms closed by remote host.
Connection to vac1-10-tms closed.


Comment by OVH - Monday, 21 September 2015, 16:39PM

Sep 20 18:25:55 (none) python[6468]: [S] #SUBHOSTS-REBOOT found 3 apms
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-0-ipmc
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-1-ipmc
Sep 20 18:26:51 (none) python[6468]: [S] #SUBHOSTS-REBOOT verified apm-2-ipmc


Comment by OVH - Monday, 21 September 2015, 16:39PM

admin@vac1-10-tms:/# services tms show
Peakflow TMS state: stopped
admin@vac1-10-tms:/# services tms start
Starting Peakflow TMS services..done.
admin@vac1-10-tms:/#


Comment by OVH - Monday, 21 September 2015, 16:39PM

Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:8' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:19' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-0:4' failed on [Errno 111] Connection refused host:apm-0-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:15' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:17' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:19' failed on [Errno 111] Connection refused host:apm-0-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:20' failed on [Errno 111] Connection refused host:apm-0-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-0:6' failed on [Errno 111] Connection refused host:apm-1-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-0:20' failed on [Errno 111] Connection refused host:apm-0-0
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-1:0' failed on [Errno 111] Connection refused host:apm-1-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-1-1:3' failed on [Errno 111] Connection refused host:apm-1-1
Sep 20 18:38:31 vac1-10-tms blinky[30476]: [W] 'get_pktengine_config_version' to 'apm-0-1:18' failed on [Errno 111] Connection refused host:apm-0-1


Comment by OVH - Monday, 21 September 2015, 16:45PM

This does not look good at all. we started to manage
a hardware failure with arbor.

We have performed different manipulations, we removed cards one by one and it's still not working.
Every time we made the reload of the chassis.
We noticed that during the reload 10G ports did not go DOWN, so we think the software was not properly reloading the chassis.
We torn very violament the chassis power cables (3 cables!!) and plugged them over carefully.
It's better. The chassis has restarted all the cards. We are monitoring.


Comment by OVH - Monday, 21 September 2015, 16:46PM

Everything is in order. We are getting VAC1 on.