Blog

October 30, 2020 – We’ve been seeing an uptick in phishing attempts in the last couple of days and wanted to remind users that Network Solutions, Inc. will never reach out to you through email to confirm your password.  These emails are attempts for spammers to get your password so they can use your account to send spam through it.  Overnight, our support email box received at least 8 of these email attempts. Fortunately, our spam filtering system sent all of these to the SPAM folder and that should be your first clue  to not trust these emails.   The most common right now is a “security alert” which states that a new device has accessed your account.  It looks like the screenshot below:

Another attempt is asking for verification on your webmail.  Notice that it looks nothing like our webmail.  That version looks like this:

Please be vigilant when checking your email.  It is important to make sure links look legitimate and if someone is asking for passwords, it is most likely a scam. Our technical support team is available to answer questions you might have about suspicious emails, so please don’t hesitate to reach out to us via phone at 256-547-6817, email support@internetpro.net or chat using the chat bubble on the lower right of your screen.

Monday October 12, 2020 11:15am – At 11:09, we detected an issue with the email filesystem related to the issues we experienced up until September 25th (see: https://www.netsolinc.com/email-issues-3/).  We  jwere able to restore service at 11:24.   We are disappointed that our earlier fix was not permanent and are looking into options.

Update – Thursday November 19, 2020  1:45pm – Again we experienced an issue with the mail server after over 1 month of uptime.  Problems of this nature which can take weeks to manifest are very difficult to fix and/or be sure they are actually fixed.   We’ve put a new switch in place and are working to restore service.

Update 2:30pm  – Service restoration is proving difficult.  We are working with vendor support now as well.

Update 3:25pm – Service has been restored.  We are monitoring systems for stability.

Update 3:35pm – Backed up traffic has come in and been processed.  Server load went up as expected and all systems performed well.  Everything should be back to normal now.

If you are still experiencing issues, please give us a call at  256-547-6817 or use the chat window at the lower right of your screen.

September 14, 2020 3:50pm – We are currently experiencing issues with our email system.  Users are reporting that the server is not responding or is asking for passwords.  We are investigating the problem and will post updates here.

Update 3:58pm – The issue has been tracked to our storage area network (SAN).

Update 4:05pm – We have implemented a fix.  It will take a few minutes to resolve.

Update 4:31pm – Services appear to be restored.  We’ll be monitoring closely for the next few minutes.

Update 4:34pm – For those wondering, no email was lost.  It should be filtering into your inbox shortly.  Services may be a bit slower for a few minutes as the system processes all of the requests from the last 45 minutes.

September 15, 11:40am – The issue has reoccurred and we are investigating.  We believe we may have lost a network card on our SAN.

Update 12:22pm – The failure actually happened at 11:28am.  We believe we have a hardware fault in the SAN.  We’re currently doing a filesystem check to insure data integrity and then services will be brought back online.  We are investigating the root cause.

Update 12:27pm – All services have been restored but we continue to investigate the cause so we don’t have a repeat of the issue.

Update 12:32pm – We’ve identified another issue and are pulling services offline again.

Update 12:42pm – Services are back online.  Hopefully for real this time.

Update September 16th 5:30pm – The issue has happened yet again.  Since it is slightly after business hours, we are going to do some major maintenance, not only on the email system but others as well.  Web services as well as virtual private servers will be affected.  We’ll be posting updates here.

Update 6:45pm –  We were able to do some of the major maintenance without affecting virtual private servers and most web servers. We are continuing to work on the mail systems.

Update 7:20pm – We’ve implemented a fix as suggested by our SAN vendor.  All services have returned to normal.  We’ll have to wait to see if their fix actually takes care of the problem. Unfortunately, issues which occur every 24 hours or so are hard to troubleshoot.  We do understand your frustration and definitely share it.   We do appreciate your understanding.

Update – September 18th 9:35am – The issue has recurred.  The good news is that we are getting good at recovering more quickly.   We are in the process of getting hardware replacements.

Update – 1:35pm – Issue returned.  We’re bringing it back up and expect recovery in 10-15 minutes.

Update – 1:45pm – Services have returned to normal.   We have Saturday delivery equipment on the way.  There will be maintenance tomorrow, but it should fix the issue.

Update – September 19th 5:00pm – Apparently our Saturday delivery did not get on the truck.   We’re waiting to find out the status of our replacement equipment.  Updates to follow as we get them.

Update – September 21st 12:10pm – The good news: Our equipment has been delivered to our technician’s home and he is in route with it.  The bad news:  The issue has recurred and we are int he process of recovery. Updates to follow.

Update 12:20pm – Services have been restored.

Update 7:55pm – We are starting replacement of hardware.  Our hardware vendor maintains that there is a software problem, but the software configuration did not change in the 290 days of uptime the equipment had before failure, so we disagree.  There is also the fact that their recommended reset did not work.  We are replacing the hardware as our best troubleshooting step.  There will be a bit of downtime until we can replace the existing hardware. We will be posting updates here as usual.

Update 8:25pm – All systems have been restored and we are operating on the new hardware.  We’ll now see if the vendor is correct. Please bear with us as we troubleshoot these mission critical systems.  We’ll soon be able to upgrade some systems with this new hardware.  Stay tuned for that.

Update September 22, 2020 5:15am – Well, it looks like the vendor was correct in that they have a software error in their systems.  We have reverted to the initial hardware after a failure of the system once again which occurred at approximately 4:15am.   All systems have returned to normal operation and we can now concentrate on the software fix for the issue.

Update 11:25am – The system stopped responding again at 11:15am.  We’ve restarted the troubled equipment and it should be online momentarily.
Update 11:35am – System came online at 11:31.

Update September 25th 5:30pm – The system failed again.  We have been waiting for that and are implementing a recommended fix.  We’ll post an update as soon as the system is back online.

Update 6:05pm – Services are back up.  We took a bit longer on this one to make sure all file system checks were done and the implemented “fix” is complete.  We’ll see if it actually is a fix.

FINAL UPDATE (hopefully) – October 3rd, 10:30am – The system has now been stable for over a week.   We believe that it is truly fixed this time. We deeply regret any inconvenience this series of outages caused, and are glad to once again be able to count on the reliability of our email system.   Thank you all for your patience as we worked through this difficult issue!

If you still have problems – We are seeing some people with Outlook having trouble getting reconnected.   Our recommendation is to close Outlook for 2 minutes and then reopen. If it still doesn’t work, most likely a password has been input incorrectly.  The newer versions of Outlook won’t ask for the password again, so you have to go into the server settings to put in the correct one. If you need help with that, please call us.

If you are still having issues related to this, please contact us via 256-547-6817 or by chat using the box to the lower right of your browser.

August 30, 2020 – Our preferred VOIP provider, Siptrunk, seems to be having some issues this morning.  Problems started at approximately 5:00 am CDT.  The problem stems from the fact that many subdomains for siptrunk.com are not resolving.  Phone systems register to gw.siptrunk.com, which is not working at the moment.  We have submitted a ticket and are awaiting reply from Siptrunk for a status and estimated time to repair (ETR).   Updates will follow.

Update 7:55am – Siptrunk reports that their servers are not having issues but that there is “a widespread internet impairment affecting many online services at this time.” And that their “engineering teams are fully engaged and investigating.”

Update 8:00am – The page at downdetector.com is definitely showing widespread simultaneous outages across multiple providers, so this looks like more than just a VOIP issue.  We recommend that customers check Downdetector for updates on their ISP (Comcast, Charter, etc).  Right now, reports or outages are increasing.

Update 8:15am – Cloudflare, a large content delivery network (CDN) seems to be a common provider in this failure.  Their status page is at https://www.cloudflarestatus.com  Please note that times listed on that page are UTC which is 5 hours ahead of CDT (local time).   Reports are also stating that issues at Centurylink may be a root cause of the problems being seen on the internet and with Cloudflare. Their status page is at https://status.ctl.io/.  Currently, all involved providers seem to be “investigating” and are not listing any repair time estimates.

Update 10:15am – The network issues have been fixed at Centurylink and other providers have re-peered with them.  Routing has stabilized and things have returned to normal for the most part.

We are monitoring support@internetpro.net and Chat (use the window at the lower right of your screen).  Our phone service is also affected, so we recommend one of those two contact methods if you have questions.

**Issues with phones are resolved.  This incident is over.

August 25, 2020 – We are seeing an outbreak of viruses spreading through .DOC file attachments very similar to the outbreak we saw in November of last year (see: https://www.netsolinc.com/virus-trojan-spreading).  We have again made the decision to try to limit this activity by blocking files with a .DOC extension from going through the email system.   Within the first few seconds of initiating the block, we had already stopped 25 messages.  We do understand that people sometimes still legitimately use .DOC files, but this is a very old file type which was last used in Word version 2003.  We recommend that if you are using some old files and having trouble sending them through the email system, that you simply do a “Save As” and save them as the newer .DOCX format.

We will try to lift this restriction again as soon as we can, but for now, we are trying our best to help protect our customers and prevent the spread of malicious software.  We thank you for your understanding.  If you have any questions, please email us at support@internetpro.net or take advantage of our chat box at the lower right of your screen.

Update September 28, 2020 – It has been roughly 1 month since we blocked the .DOC files.  We’ve checked the logs and we’re still seeing some .DOC files come through, but at a *much* lower rate. Today, 7 hours of logs were examined (morning on a business day) and we found 44 blocks.  Of those, about 8 looked like bogus names, so we’re see about one an hour.  Considering that the other 36 or so (or 5 an hour) seem legitimate, we are removing the block to see if the virus starts spreading again.  Please be vigilant about your attachment viewing!

July 21, 2020 – We’ll be conducting some planned maintenance on a couple of servers tonight starting at about 10:30pm CDT.  The maintenance will include an upgrade to server operating systems which will also include an upgrade to certain software packages including PHP.

We utilize LTS (Long Term Support) versions of our operating systems so that we can upgrade less often resulting in more stability for users.  Currently, the PHP version being used by an older LTS release is showing in some software (including WordPress) as insecure and out of date.  Since this version of PHP is supported by the operating system producer and is part of the distribution, it is patched and security is maintained.  However, there are some improvements that have come with the newer versions.   Since we are receiving an increased volume of support requests due to the notices being shown in WordPress, we’ve decided to go ahead and move to the new LTS version of the OS.

During the maintenance time, some web sites will be unavailable as the servers upgrade.  Our technicians will do some testing of sites after the update, but are unable to test every page on every site.  We do not expect any compatibility issues as the versions we are upgrading from and to are very similar.  However, it would be prudent for customers to check over their sites to make sure all functionality is intact.    We will be posting updates here as the maintenance is performed.

Update 9:45pm – Beginning the process on a Virtual Private Server which affects fewer users than the next server we are doing.  We expect the shared server to start about 10:30pm.

Update 11:45pm – VPS is updated and we are now starting on shared server.  The VPS took a bit longer than anticipated, but we hope that the shared server goes more quickly.

Update July 22 1:15am – Shared server is complete.  We are doing some random testing.

If you have any questions or concerns about this maintenance, please contact us via email at support@internetpro.net, chat using the window to your right, or by phone at 256-547-6817.

June 23, 2020 – 6:30am –  Our virtual server infrastructure has been impacted due to an automatic update of a critical security issue.   A critical vulnerability in NFS was patched and an upgrade sent out by the OS vendor.  This was unplanned and unattended.  Unfortunately, the update took VPS systems offline.   We are in the process of restoring all functionality.  Updates will be posted here.

Update 7:15am – We’ve been able to restore service to many of the machines.  We do have a VPS host machine which is giving us some issues and we have an engineer physically working on it now.  Email has not been impacted and neither has shared hosting services.  The vast majority of our customers are either no longer impacted or were not impacted at all by this event.

Update 7:45am – We’ve identified a major hardware issue that did not present itself until the host was rebooted.  Work is underway to replace the affected hardware.  Unfortunately, one of the affected services is our phone system, but we do have backups in place.

Update 8:30am – Repairs have been completed.  VPS machines are being powered up.  Services should all be restored within minutes.  Our phone services have also been fully restored.

If you have any questions about this issue, please email us at support@internetpro.net.

Monday June 1, 2020 – Since everything else in the world is blowing up recently, we thought we’d get in on the fun and cause some issues with our mail server.   Every couple of years, we have to redo the secure certificates used on all connections.  We’re in the process of doing that, and normally, it goes unnoticed.  This time, however, some issues have arisen.  We’re implementing some “best practices” with regards to the type of security protocols used, and they have caused some clients to get error messages and/or not be able to connect.  We are working through these problems and apologize for any inconvenience this may cause.  No emails will be lost…but you may get an error when sending and perhaps have to resend an email.

We’ll be posting updates here as we complete the maintenance.

Update 3:45pm – Apparently, we’re running the best security we can for the version of the software currently being used.  We’ll be working on upgrading that software in the near future.

Update 4:09pm – We’re seeing reports of some old Apple devices which are not trusting the certificate.  It is OK to trust the certificate because it is legitimate.  It should be valid until June 2022.  Some older devices don’t “know about” the new certificates, so they have to be told to trust them.

Update 4:20pm-  The webmail is having issues related to the certificate problem.  We are investigating it.

Update 4:40pm – The webmail issue was related to IMAP not authenticating.  All issues relating to authentication should be resolved.  Trust issues with older devices will not change as that is inherent to the devices.

Update 5:40pm – Apple devices have a history of not liking certificate changes.  We’re finding that for most Apple devices, it is necessary to delete the account and set it up again to get the new certificate.  If you need our email settings for setting up again, please see our page here:  https://www.netsolinc.com/email-settings-2

This issue is now closed.  If you have any questions not answered here, please contact us at 256-547-6817, email support@internetpro.net, or chat with us using the window to the lower right of your screen.  Please note that the phone numbers listed on our site go to a corporate phone system and do not receive text messages.  If you like to text, please use the chat window below which also works on mobile.

May 17, 2020 – 1:30pm – We started experiencing some intermittent connectivity issues at around 12:49pm today.  We are investigating and currently all network services are up.   We believe it is an upstream issue, but will post updates here as we have them.

Update 1:50pm – The issues are upstream routing problems which are out of our hands.  The upstream carriers are aware of their problems and are working on it.   Connectivity is fairly stable at the moment with only short interruptions.  We expect the problems will be completely worked out shortly.

Update 8:20pm – After being fairly stable since the last update, we’ve begun to see packet loss again.  It is very intermittent and seems to be upstream BGP routing changes.   Hopefully, the root issue has been fixed and routing is updating to follow primary routes again.   Internally, our systems are responding, and again, this is on the digital highway outside of our domain.

Update May 18, 6:45am – The issue seems to have been related to one or more core routers in Georgia having problems.  Much of the traffic for the southeast runs through that area, so the routing problems were far reaching.  We are pretty sure we were correct in our assessment of the short interruption around 8:15pm last night.  Everything has been stable since the routes were adjusted at that time.

If you have any questions, please call 256-547-6817, email support@internetpro.net, or chat using the window at the lower right of your screen.  Please be aware that if services are currently impacted, email and chat may will be affected.