Blog

AT&T Network Engineers will be performing a planned maintenance on
Birmingham-Alabama Gigabit Access Router 3. The work will be performed on 6/18/2010,
during the maintenance window of 12:00AM-6:00AM, local time.
AT&T's goal is to successfully complete the task and restore service within the
stated time frame. 

We apologize for any inconvenience that you may experience.
We expect service to be impacted somewhat during the maintenance window.

AT&T Network Engineers will be performing an IOS upgrade.

We launched a new version of our web site!  Much of the information and functionality is the same, but we are updating the “look”.   Also, the new site is built on WordPress technology, which allows easy and quick updates of the site.   We are also tying the site in with Twitter.  You can follow us @netsolinc.   We will post updates of new products, specials, or events that may affect your service.  We hope you enjoy the new look and your feedback is welcomed.  If you have any issues with the site, please let us know!

We have one host down in our offsite datacenter.  It looks like a connectivity problem with the datacenter rather than a host issue.  We are working to resolve the issue.

Update 6:48pm – We have confirmed that it is yet another problem with a Cisco switch.  Datacenter engineers are working with Cisco now to resolve the issue.  Network connectivity has been intermittent since 6:37pm.

Update 8:40pm – Switching has been relatively stable for the last hour or so, but the datacenter engineers are continuing to research and diagnose the issue with Cisco TAC.

Update – Cisco identified a software bug.  They have applied updates to the equipment and the network is stable now.

Post Incident Report
Date / Time of Incident: 6/1/2010 6:44PM ET
Duration of Incident: 1hour 13minutes
Scope of Incident: Newark, Delaware Data Center Dedicated, Colocated and Cloud Customers

Description of Incident:
At 6:44PM ET one of the two redundant Cisco 6509 switches supporting the Dedicated, Colocated and Cloud Customers failed.  The second switch failed in the same manner when load from the first switch was re-routed to the second.  The end result was loss of connectivity for approximately 40 minutes.  As the switches recovered, latency lasted another 33 minutes.

Actions During Incident:
6:44 PM  - Network Operations and Network Engineering were immediately dispatched to investigate and resolve the issue.  A ticket was opened with the vendor of the switches (Cisco) and efforts to restore service were coordinated between the datacenter and Cisco.  

6:55 PM - After exhausting normal troubleshooting avenues, the Switch 1 is reloaded at console by Network Engineers

6:58 PM – Switch 2 primary supervisor engine fails to boot, secondary supervisor engine becomes active

7:03 PM – Switch 1 completes rebooting , routing protocols established, datacenter team monitors performance of switch

7:08 PM – Switch 2 Primary supervisor engine is manually reset, secondary supervisor engine becomes fully active

7:12 PM – Switch 1 primary supervisor engine fails, secondary supervisor begins to take over

7:20 PM – Switch 1 secondary supervisor takes over

7:37 PM – Switch 1 Primary supervisor engine pulled, engineers suspect faulty hardware

7:48 PM – Switch 2 Secondary Supervisor engine completes booting, begins forwarding traffic.  CPU on both switches remain elevated for 10 minutes while routing protocols are re-established

7:58 PM – Forwarding returns to normal, data delivered to Cisco for further analysis

Root Cause of Incident:
Cisco has identified a specific bug as the likely root cause that created a memory leak which triggered the event.  Still awaiting final confirmation from Cisco on the bug, but initial findings indicate software issues as the root.

Further Actions:
Based on an expected confirmation from Cisco on the bug, a software patch will be applied in a lab environment.  After successful application of the patch, Network Engineers will be applying the patch during a scheduled and announced maintenance window.

The datacenter Network Engineering team is looking into other architectural changes to add additional diversity to the customers that are currently supported by the current set of Cisco 6509s to further insolate customers from any future type of single core switching failure.

Our phone lines have all been ported over to voice over IP.   Preliminary testing looks good.  If you are interested in this type of system at your own office, please contact us.  It can save you a lot of money on your phone lines.

This morning, our phone lines are switching from traditional copper to voice over internet protocol (VOIP).   We are increasing the number of lines coming into our phone system to decrease busy signals.  The system will also allow us more flexibility with our system to allow us to serve you better.  During the switch, our office may be unreachable via phone for a short period of time.  If you have an issue, our technicians will be closely monitoring our support@internetpro.net email address. The window for this switch is scheduled from 8:30am to 10:30am CDT.

We have setup a new Twitter account for Network Solutions, Inc. You can follow it @netsolinc. Following that account will allow you to receive updates about new products and services as well as up to the minute network status updates. These updates will also be tied in to our new website.