September 21, 2017 – From approx. 11:53 PM on 9/20 until 01:17 AM 9/21 we experienced an outage with an upstream fiber optic service provider. We were able to troubleshoot the issue with the provider and restore services. We will continue to work with the provider to try to correct the root cause of the outage which was a service provider router failure.
Update September 26, 2017 – Apparently, the router supplied to us by AT&T is a hunk of junk and it caused an issue again with our network. The issue lasted from about 7:55pm to 8:20pm. We are in the process of requesting a replacement router.
Update September 28, 2017 – AT&T believes the issue is a “known problem” with the software running on the router. They will be performing an emergency maintenance to upgrade the firmware at 12:00 midnight tonight. We expect minimal impact to services as the router reboots.
Update September 29, 2017 – AT&T failed to do the firmware upgrade last night. Router failed again at approximately 8:58AM and we experienced a services outage until about 9:10AM.
Update 2 September 29, 2017 – AT&T has rescheduled the update for tonight 12:00 midnight – 3:00am. Again, this should only cause an interruption for a few minutes as the router reboots. (If they actually perform the update this time).
Update September 30, 2017 – We are 1.5 hours into the maintenance window (1:30am) and AT&T has not started the maintenance. We have been attempting to reach their support services for 45 minutes without success but are determined to have this maintenance performed tonight.
Update 2 September 30, 2017 – It looks like this maintenance is going to be performed outside our original window because AT&T “is short staffed”. Due to the importance of this issue, we are going to allow the work to be done outside the window and we are doing everything possible to get it done as soon as possible (2:50am).
Update 3 September 30, 2017 – 3:35am and it looks like AT&T’s incompetence may win out over our determination to fix this issue tonight. Still on hold with technical support without an answer as to why the work has not been completed. We are going to have this work completed as soon as we are able to get engineers to complete it. Unfortunately, it looks like we are not going to be able to give a window of time or even an estimate of when that will happen. We will post any updates we get here as we get more information.
Update 4 September 30, 2017 – 11:00am – We’ve finally gotten word that although AT&T promised updates during our windows, they had no intention of doing it because no engineers capable of performing the upgrade work until Monday. We’ve literally been told that they can’t do a simple firmware upgrade until Monday and it is just unbelievable.
Update October 4, 2017 – 12:45pm – AT&T has finally applied the firmware update. They could/would not do it during a planned maintenance window and we had to allow them to do it whenever they could get to it. This happened (without notification) at 12:21pm and the router came back online at 12:32pm. We have been assured that this will fix the issue, and we do hope to put an end to this saga.
Update October 9, 2017 – 8:15am – The event has reoccurred. As we initially thought, we think the router is bad. We are again contacting AT&T about this issue.
Update October 9, 2017 – 10:45pm – We finally got our replacement router. We will have a brief interruption while it is replaced. Unfortunately, it is the same model as the old one. We are hoping it is not a design flaw.
Update October 10, 2017 – 12:15am – The comedy of errors that *IS* AT&T sent a router with a bad port. We’re back on the original router and are awaiting a new replacement.
Update October 13, 2017 – 8:35am – Router failure. We have one being shipped to us which should arrive today. We plan on a maintenance window tonight or tomorrow night to replace the garbage we were given by AT&T with a new Cisco and a spare to replace that one if there is an issue.
Update October 17, 2017 – 4:00pm – We are up and running on our new router. We were able to make the swap with only about 10 seconds of downtime (unlike AT&T). This should now be closed.
If you have any questions or concerns, please email support@internetpro.net or call 256-547-6817.