March 4, 2022 – 10:00am – One of our web servers has suffered a failure that is affecting some of our shared hosting sites. We are working to address the problem. Originally, a RAID array lost a drive and it was replaced as normal, but apparently the rebuild process failed and we are seeing degradation in service. We will be posting updates here as we have them.
Update 1:45pm – The server with the failed RAID array is currently offline as we work to repair the issue. Some sites will be offline until this process is completed. We will update this post with further information as we continue the repairs.
Update 6:45pm – We’ve recovered the filesystem, but unfortunately have some corruption in the database system of the affected server. We’re working on restoring from backups but are running into some corruption issues.
Update 10:30pm – We have successfully restored a few sites and have a procedure for restoring the rest. Sites will be coming up as we restore them.
Update 11:15pm – We believe all sites except for one have been restored to full functionality and we have informed the owners of the affected site. If you are still experiencing an issue with your website, please email us at support@internetpro.net to let us know and we will investigate it further. We do apologize for this issue.
March 5, 2022 – 8:30am – We’re going to attempt one last effort to get the one website still having issues recovered. The effort will affect all other sites briefly. We’ll update here as to the success or failure and the conclusion of the event.
Update 8:50am – Failure. The site in question was in the midst of a redesign, so we’re pressing forward with that. There should be no more service interruptions related to this outage.