tnolen

December 15, 2020 – We began receiving calls about Gmail email not going through starting at about 5:15pm tonight.   Failure notices are being returned stating that the server does not like the recipient and that the account does not exist.    An example can be found below:

–BEGIN

—–Original Message—–
From: MAILER-DAEMON@mail.internetpro.net
[mailto:MAILER-DAEMON@mail.internetpro.net]
Sent: Tuesday, December 15, 2020 5:01 PM
To: REDACTED
Subject: failure notice

Hi. This is the qmail-send program at mail.internetpro.net.
I’m afraid I wasn’t able to deliver your message to the following addresses.
This is a permanent error; I’ve given up. Sorry it didn’t work out.

<REDACTED>:
Remote host 173.194.219.27 does not like recipient REDACTED
Remote host said: 550-5.1.1 The email account that you tried to reach does
not exist. Please try
550-5.1.1 double-checking the recipient’s email address for typos or
550-5.1.1 unnecessary spaces. Learn more at
550 5.1.1 https://support.google.com/mail/?p=NoSuchUser d202si206486ybh.359
– gsmtp

–END

This seems to be Google having another issue after their earlier problems on Monday.   Many users are experiencing issues again.  Please note that these problems are not related to our email system and that messages which are bounced in the manner described above will NOT go through.  They will have to be sent again when Google resolves their issues.

Update – Google reports that this has been resolved.  Again, any emails bounced during this outage will need to be resent.  This is not an issue with our delivery system and is entirely on Google’s side.  We cannot control the delivery of those bounced emails.   For a status report, please see this status page.

Monday December 7, 2020 – We will be performing some scheduled maintenance tonight, starting at approximately 11:00pm.  This will be a short maintenance item and is expected to last around 30 minutes.  Email services will be affected.   We recommend closing your email programs before the end of business day so that your email program doesn’t lose connection to the servers.

Update 10:30pm – Maintenance started a bit early.  Will update as things progress.

Update 11:10pm – Maintenance is complete.  All systems running normally. It does look like some did not heed the advice to shut down email clients, but there should be no adverse side effects.

If you have any questions about this maintenance, please call 256-547-6817 during normal business hours.

November 16, 2020 4:10pm – This post comes after the resolution of the issue as the server failure also affected this web site.  At approximately 3:20pm, one of our web servers experienced a hardware issue.  The equipment was pulled from the rack and repaired.  However, when the server was brought back online, the database server did not restart resulting in many users seeing messages like “Unable to establish database connection”.   Initially, we thought that there had been some corruption of the database during the failure.  However, we found that a repair previously made on a database on the server was not properly cleaned up by the technician.  The server was trying to load the corrupted backup of the database which was repaired (months ago) and was failing.  We managed to locate the bad database and remove it and service was restored at approximately 4:05pm.

This server will be monitored on our test bench for a few days before being returned to the production rack during a scheduled maintenance window.

If you are still experiencing problems which might be related to this issue, please email us at support@internetpro.net or call 256-547-6817.  We do apologize for any inconvenience this may have caused.

October 30, 2020 – We’ve been seeing an uptick in phishing attempts in the last couple of days and wanted to remind users that Network Solutions, Inc. will never reach out to you through email to confirm your password.  These emails are attempts for spammers to get your password so they can use your account to send spam through it.  Overnight, our support email box received at least 8 of these email attempts. Fortunately, our spam filtering system sent all of these to the SPAM folder and that should be your first clue  to not trust these emails.   The most common right now is a “security alert” which states that a new device has accessed your account.  It looks like the screenshot below:

Another attempt is asking for verification on your webmail.  Notice that it looks nothing like our webmail.  That version looks like this:

Please be vigilant when checking your email.  It is important to make sure links look legitimate and if someone is asking for passwords, it is most likely a scam. Our technical support team is available to answer questions you might have about suspicious emails, so please don’t hesitate to reach out to us via phone at 256-547-6817, email support@internetpro.net or chat using the chat bubble on the lower right of your screen.

Monday October 12, 2020 11:15am – At 11:09, we detected an issue with the email filesystem related to the issues we experienced up until September 25th (see: https://www.netsolinc.com/email-issues-3/).  We  jwere able to restore service at 11:24.   We are disappointed that our earlier fix was not permanent and are looking into options.

Update – Thursday November 19, 2020  1:45pm – Again we experienced an issue with the mail server after over 1 month of uptime.  Problems of this nature which can take weeks to manifest are very difficult to fix and/or be sure they are actually fixed.   We’ve put a new switch in place and are working to restore service.

Update 2:30pm  – Service restoration is proving difficult.  We are working with vendor support now as well.

Update 3:25pm – Service has been restored.  We are monitoring systems for stability.

Update 3:35pm – Backed up traffic has come in and been processed.  Server load went up as expected and all systems performed well.  Everything should be back to normal now.

If you are still experiencing issues, please give us a call at  256-547-6817 or use the chat window at the lower right of your screen.

September 14, 2020 3:50pm – We are currently experiencing issues with our email system.  Users are reporting that the server is not responding or is asking for passwords.  We are investigating the problem and will post updates here.

Update 3:58pm – The issue has been tracked to our storage area network (SAN).

Update 4:05pm – We have implemented a fix.  It will take a few minutes to resolve.

Update 4:31pm – Services appear to be restored.  We’ll be monitoring closely for the next few minutes.

Update 4:34pm – For those wondering, no email was lost.  It should be filtering into your inbox shortly.  Services may be a bit slower for a few minutes as the system processes all of the requests from the last 45 minutes.

September 15, 11:40am – The issue has reoccurred and we are investigating.  We believe we may have lost a network card on our SAN.

Update 12:22pm – The failure actually happened at 11:28am.  We believe we have a hardware fault in the SAN.  We’re currently doing a filesystem check to insure data integrity and then services will be brought back online.  We are investigating the root cause.

Update 12:27pm – All services have been restored but we continue to investigate the cause so we don’t have a repeat of the issue.

Update 12:32pm – We’ve identified another issue and are pulling services offline again.

Update 12:42pm – Services are back online.  Hopefully for real this time.

Update September 16th 5:30pm – The issue has happened yet again.  Since it is slightly after business hours, we are going to do some major maintenance, not only on the email system but others as well.  Web services as well as virtual private servers will be affected.  We’ll be posting updates here.

Update 6:45pm –  We were able to do some of the major maintenance without affecting virtual private servers and most web servers. We are continuing to work on the mail systems.

Update 7:20pm – We’ve implemented a fix as suggested by our SAN vendor.  All services have returned to normal.  We’ll have to wait to see if their fix actually takes care of the problem. Unfortunately, issues which occur every 24 hours or so are hard to troubleshoot.  We do understand your frustration and definitely share it.   We do appreciate your understanding.

Update – September 18th 9:35am – The issue has recurred.  The good news is that we are getting good at recovering more quickly.   We are in the process of getting hardware replacements.

Update – 1:35pm – Issue returned.  We’re bringing it back up and expect recovery in 10-15 minutes.

Update – 1:45pm – Services have returned to normal.   We have Saturday delivery equipment on the way.  There will be maintenance tomorrow, but it should fix the issue.

Update – September 19th 5:00pm – Apparently our Saturday delivery did not get on the truck.   We’re waiting to find out the status of our replacement equipment.  Updates to follow as we get them.

Update – September 21st 12:10pm – The good news: Our equipment has been delivered to our technician’s home and he is in route with it.  The bad news:  The issue has recurred and we are int he process of recovery. Updates to follow.

Update 12:20pm – Services have been restored.

Update 7:55pm – We are starting replacement of hardware.  Our hardware vendor maintains that there is a software problem, but the software configuration did not change in the 290 days of uptime the equipment had before failure, so we disagree.  There is also the fact that their recommended reset did not work.  We are replacing the hardware as our best troubleshooting step.  There will be a bit of downtime until we can replace the existing hardware. We will be posting updates here as usual.

Update 8:25pm – All systems have been restored and we are operating on the new hardware.  We’ll now see if the vendor is correct. Please bear with us as we troubleshoot these mission critical systems.  We’ll soon be able to upgrade some systems with this new hardware.  Stay tuned for that.

Update September 22, 2020 5:15am – Well, it looks like the vendor was correct in that they have a software error in their systems.  We have reverted to the initial hardware after a failure of the system once again which occurred at approximately 4:15am.   All systems have returned to normal operation and we can now concentrate on the software fix for the issue.

Update 11:25am – The system stopped responding again at 11:15am.  We’ve restarted the troubled equipment and it should be online momentarily.
Update 11:35am – System came online at 11:31.

Update September 25th 5:30pm – The system failed again.  We have been waiting for that and are implementing a recommended fix.  We’ll post an update as soon as the system is back online.

Update 6:05pm – Services are back up.  We took a bit longer on this one to make sure all file system checks were done and the implemented “fix” is complete.  We’ll see if it actually is a fix.

FINAL UPDATE (hopefully) – October 3rd, 10:30am – The system has now been stable for over a week.   We believe that it is truly fixed this time. We deeply regret any inconvenience this series of outages caused, and are glad to once again be able to count on the reliability of our email system.   Thank you all for your patience as we worked through this difficult issue!

If you still have problems – We are seeing some people with Outlook having trouble getting reconnected.   Our recommendation is to close Outlook for 2 minutes and then reopen. If it still doesn’t work, most likely a password has been input incorrectly.  The newer versions of Outlook won’t ask for the password again, so you have to go into the server settings to put in the correct one. If you need help with that, please call us.

If you are still having issues related to this, please contact us via 256-547-6817 or by chat using the box to the lower right of your browser.

August 30, 2020 – Our preferred VOIP provider, Siptrunk, seems to be having some issues this morning.  Problems started at approximately 5:00 am CDT.  The problem stems from the fact that many subdomains for siptrunk.com are not resolving.  Phone systems register to gw.siptrunk.com, which is not working at the moment.  We have submitted a ticket and are awaiting reply from Siptrunk for a status and estimated time to repair (ETR).   Updates will follow.

Update 7:55am – Siptrunk reports that their servers are not having issues but that there is “a widespread internet impairment affecting many online services at this time.” And that their “engineering teams are fully engaged and investigating.”

Update 8:00am – The page at downdetector.com is definitely showing widespread simultaneous outages across multiple providers, so this looks like more than just a VOIP issue.  We recommend that customers check Downdetector for updates on their ISP (Comcast, Charter, etc).  Right now, reports or outages are increasing.

Update 8:15am – Cloudflare, a large content delivery network (CDN) seems to be a common provider in this failure.  Their status page is at https://www.cloudflarestatus.com  Please note that times listed on that page are UTC which is 5 hours ahead of CDT (local time).   Reports are also stating that issues at Centurylink may be a root cause of the problems being seen on the internet and with Cloudflare. Their status page is at https://status.ctl.io/.  Currently, all involved providers seem to be “investigating” and are not listing any repair time estimates.

Update 10:15am – The network issues have been fixed at Centurylink and other providers have re-peered with them.  Routing has stabilized and things have returned to normal for the most part.

We are monitoring support@internetpro.net and Chat (use the window at the lower right of your screen).  Our phone service is also affected, so we recommend one of those two contact methods if you have questions.

**Issues with phones are resolved.  This incident is over.

August 25, 2020 – We are seeing an outbreak of viruses spreading through .DOC file attachments very similar to the outbreak we saw in November of last year (see: https://www.netsolinc.com/virus-trojan-spreading).  We have again made the decision to try to limit this activity by blocking files with a .DOC extension from going through the email system.   Within the first few seconds of initiating the block, we had already stopped 25 messages.  We do understand that people sometimes still legitimately use .DOC files, but this is a very old file type which was last used in Word version 2003.  We recommend that if you are using some old files and having trouble sending them through the email system, that you simply do a “Save As” and save them as the newer .DOCX format.

We will try to lift this restriction again as soon as we can, but for now, we are trying our best to help protect our customers and prevent the spread of malicious software.  We thank you for your understanding.  If you have any questions, please email us at support@internetpro.net or take advantage of our chat box at the lower right of your screen.

Update September 28, 2020 – It has been roughly 1 month since we blocked the .DOC files.  We’ve checked the logs and we’re still seeing some .DOC files come through, but at a *much* lower rate. Today, 7 hours of logs were examined (morning on a business day) and we found 44 blocks.  Of those, about 8 looked like bogus names, so we’re see about one an hour.  Considering that the other 36 or so (or 5 an hour) seem legitimate, we are removing the block to see if the virus starts spreading again.  Please be vigilant about your attachment viewing!

July 21, 2020 – We’ll be conducting some planned maintenance on a couple of servers tonight starting at about 10:30pm CDT.  The maintenance will include an upgrade to server operating systems which will also include an upgrade to certain software packages including PHP.

We utilize LTS (Long Term Support) versions of our operating systems so that we can upgrade less often resulting in more stability for users.  Currently, the PHP version being used by an older LTS release is showing in some software (including WordPress) as insecure and out of date.  Since this version of PHP is supported by the operating system producer and is part of the distribution, it is patched and security is maintained.  However, there are some improvements that have come with the newer versions.   Since we are receiving an increased volume of support requests due to the notices being shown in WordPress, we’ve decided to go ahead and move to the new LTS version of the OS.

During the maintenance time, some web sites will be unavailable as the servers upgrade.  Our technicians will do some testing of sites after the update, but are unable to test every page on every site.  We do not expect any compatibility issues as the versions we are upgrading from and to are very similar.  However, it would be prudent for customers to check over their sites to make sure all functionality is intact.    We will be posting updates here as the maintenance is performed.

Update 9:45pm – Beginning the process on a Virtual Private Server which affects fewer users than the next server we are doing.  We expect the shared server to start about 10:30pm.

Update 11:45pm – VPS is updated and we are now starting on shared server.  The VPS took a bit longer than anticipated, but we hope that the shared server goes more quickly.

Update July 22 1:15am – Shared server is complete.  We are doing some random testing.

If you have any questions or concerns about this maintenance, please contact us via email at support@internetpro.net, chat using the window to your right, or by phone at 256-547-6817.

June 23, 2020 – 6:30am –  Our virtual server infrastructure has been impacted due to an automatic update of a critical security issue.   A critical vulnerability in NFS was patched and an upgrade sent out by the OS vendor.  This was unplanned and unattended.  Unfortunately, the update took VPS systems offline.   We are in the process of restoring all functionality.  Updates will be posted here.

Update 7:15am – We’ve been able to restore service to many of the machines.  We do have a VPS host machine which is giving us some issues and we have an engineer physically working on it now.  Email has not been impacted and neither has shared hosting services.  The vast majority of our customers are either no longer impacted or were not impacted at all by this event.

Update 7:45am – We’ve identified a major hardware issue that did not present itself until the host was rebooted.  Work is underway to replace the affected hardware.  Unfortunately, one of the affected services is our phone system, but we do have backups in place.

Update 8:30am – Repairs have been completed.  VPS machines are being powered up.  Services should all be restored within minutes.  Our phone services have also been fully restored.

If you have any questions about this issue, please email us at support@internetpro.net.