tnolen

December 19, 2024 – 7:30am – This morning we had a failure of one of our power distribution units.   It affected a lot of systems which caused most of our services to be offline.  This affected email, web sites, phone systems (including ours), and other systems.  We have identified the issue and implemented a workaround.  Most systems have come back online, but we are still working to bring a few up.  We will be posting more later.

Update – 7:40am – We had reports that the email system was not authenticating.   The backend part of the system failed to come online, but we have brought it online now and all services regarding mail seem to have recovered.  Some email may be delayed in reaching your mailbox.

Update – 7:55am – We believe that all affected systems are online and functioning normally.  If you have a system that is still not working for you, please contact us at support@internetpro.net via the chat in the lower right of this screen, or call us at 256-547-6817.

Update – 8:28am – We were made aware of a few shared websites that did not come back online.  This was because of a web server misconfiguration that would have been caught during normal operation, but the reboot due to the power failure caused the bad config to be loaded and kept some sites from working properly.   This has been resolved.

This issue was caused by a failure of one of our network-controlled power distribution systems.  We use that type of system to be able to automatically and remotely reboot systems as well as to check power loads, battery conditions, etc.  The entire bank of outlets failed, and caused some systems without redundant power supplies to go down. Unfortunately, there were a couple of critical bottlenecks there which caused other systems that had not gone offline to be unreachable.   We  will be scheduling maintenance in the near future to replace the unit and that will be done during off-peak hours.  We will also be evaluating the bottlenecks made relevant in this failure and will be working to implement design changes to prevent a recurrence.

June 22, 2024 – In what seems to be a regular thing now, we had an equipment failure of a battery backup system this morning at about 7:15am.    It affected a few servers including a host for some shared websites, a virtual server host affecting a few cloud  phone system instances (including ours),  a couple of network switches, a firewall, and a couple of backup servers.     Our technicians responded quickly and brought the affected systems back online within just a few minutes.   We are investigating the root cause, as the UPS (Uninterruptible Power Supply) is reporting 100% battery and not reporting a loss of mains power.

Hopefully, as they say, things come in threes and this is the last issue for a while.   If you have any questions or concerns about this, please email us at support@internetpro.net, or call us at 256-547-6817.

Update July 28  3:45pm – We’re seeing some strange issues which may be related to this UPS problem.   Some VPS machines and at least one shared host server are being affected.  We’re also seeing some performance issues.   Technicians are investigating.

Update 4:33pm –  It was, in fact, the UPS issue.   We had replaced the batteries in the unit, but it looks like the entire unit will need to be replaced.  We’ll be ordering a new one and we’ll schedule a maintenance window for its replacement.

Update 4:39pm – We believe all affected systems are back online.

February 6, 2024 – 12:45pm –  We’re seeing email from the @internetpro.net domain being blocked as spam by Google’s servers (gmail.com and others hosted with Google).   The message says something as follows:

—BEGIN

74.125.138.26 failed after I sent the message.
Remote host said: 550-5.7.1 [74.252.14.252       7] Gmail has detected that this message is likely
550-5.7.1 unsolicited mail. To reduce the amount of spam sent to Gmail, this
550-5.7.1 message has been blocked. For more information, go to
550 5.7.1  https://support.google.com/mail/?p=UnsolicitedMessageError o20-20020a81de54000000b006041f08d2dfsi1515206ywl.160 - gsmtp

—END

We have done some testing and found that not all domains we are sending are being blocked.  Some are going to the Spam folder instead of being blocked outright and some emails are getting through.   This seems to be happening because of some new policies being enforced by Google.   There are some articles online such as this one.  However, we have been authenticating our emails as Google is requiring for a very long time, so we don’t know why our domains would be affected.  Here is a report of our SPF record that Google is now requiring.

If you are seeing errors on your domain and it is not @internetpro.net, please reach out to us by chat, call, or email at support@internetpro.net.   We are working to resolve the issue as soon as possible.

Update 4:25pm – This issue is ongoing where @internetpro.net emails are being “hard bounced” to the sender.   Other accounts on our servers are being delivered but are winding up in the recipient’s “spam” folder more times than not.    We are going to institute a new policy against remote forwards and will be making a new post about that.  We will do our best to reach all customers who are utilizing these types of forwards.

March 11, 2024 – We seemed to finally fully emerge from Google’s penalty box over the weekend and continue to see normal deliveries today.  We are truly sorry for the issues and the fact that we can no longer forward email to Gmail accounts for customer convenience.   This is a direct result of spammers and the efforts email providers are making to make delivering spam harder.  Unfortunately, it also takes away some of the conveniences and workflows that legitimate people have grown accustomed to using.   Some of the things we are doing going forward:

• We are making every effort to fix websites and other services whose email deliveries have been affected.
• Deploying DKIM and DMARC for domains which we host.  We are doing this in an order based on the volume of email being produced, but are also responding to direct inquiries.
• We are adjusting our normal SPF record policies fro Hard Fail to Soft Fail in an effort to help delivery.  Of course, we will deploy any policy requested by the customer on this.
• We are now scanning OUTGOING email for spam content.  If it looks like spam to our filters, it will most likely look like spam to the receiving email provider.  We don’t want them thinking we are sending spam, so this is an effort to prevent that.
• We have changed our policies on how websites authenticate web forms and are developing updated documentation to aid web developers in their deployment.  However, going forward our default recommendation for this is “don’t do web forms”.
• We have updated our policies on the forwarding of emails and now do not allow “remote” forwards – those that forward email to an address outside of our systems.
• We have updated our policies on authentication and now the email must be sent FROM the same email account which authenticates it.

We believe these updated policies will keep us safe from being blocked in the future and realize that they impact many users.   That is why we are here to help get any issues which customers find regarding these changes corrected.   If you have any ongoing problems with email, please contact us by phone at 256-547-6817 or email support@internetpro.net.   Forwarding bounce messages as an attachment is always preferred.  Screenshots from mobile devices are rarely helpful in tech support situations.

January 30, 2024 – 2:00pm – We started receiving reports of some users having trouble reaching email, web, and/or cloud services about 1:45pm.   All servers/services in our datacenter are responding normally, but we are detecting a lot of route changes upstream.   This is especially affecting Spectrum, Comcast, and AT&T customers.  We have seen improvement, but there are a LOT of routes on the internet, and it does take time for those to settle out.   When repairs occur, the routes are changed again which can cause additional connectivity issues.

Customers are currently seeing  slow page loads and possible timeout errors with sending or receiving email.   This is an issue “upstream” of us, so we have no control over it. It is very similar to a traffic jam on the interstate and the authorities need time to clear it out.

We will be posting updates here as we find out more information.

Update 4:15pm – This does not seem to be improving as we had earlier thought.   It seems to be an ongoing hardware issue between “peers” upstream.    We are working with the providers to get information about an estimated time to repair.

Update 5:30pm – Things seem to have cleared up.  If you are still having issues, please let us know by emailing support@internetpro.net.

Update February 4, 2024 -10:00am – This problem has cropped up again and seems to be a systemic issue with the interconnections of providers being overloaded.  We’re working on getting updates (again).

Update 10:50am – A fix has been implemented and we are monitoring the results.

January 16, 2024 – Due to inclement weather and hazardous road conditions in our county and surrounding counties, our office will remain closed today.   Technical support will still be available via email at support@internetpro.net, by phone or text at 256-547-6817 by chat by using the chat window on the support page of our website.    We do apologize for the inconvenience but we need to insure the safety of our staff members.      We expect to return to the office for normal business hours tomorrow.

October 11, 2023 9:45am – We detected an increase in emails stuck in our queues that have not been successfully delivered.   After investigating, we’ve found that emails going to Microsoft 365 hosted emails are being delayed with a “Server Busy” message from the other side.   Downdetector is showing that users are reporting problems with Microsoft 365 services.

These emails are still “in line” to be delivered and once the issues at Microsoft are resolved, they should go through.   In the event that the emails are stuck for more than 4 hours, senders will receive a notification message indicating that there has been a delay but the message delivery will be retried for up to 5 days.   We assume that Microsoft will have the problem fixed by then.    We’ll post updates here as we have them.

Update  11:45am – Bleeping Computer has posted a story about the issue, saying it has to do with some new spam rules imposed by Microsoft.  Find the article here: https://www.bleepingcomputer.com/news/microsoft/exchange-online-mail-delivery-issues-caused-by-anti-spam-rules/   We can confirm that the error message mentioned in the article is what we are seeing in our logs.

Update 2:00pm – It looks like things have returned to normal.   We will continue to keep an eye on things, but all indications are that all backed up email messages have been delivered.