News and Updates

This category contains information about events that may affect your service such as maintenance or outages. It also contains announcements on new products releases and enhancements.

Power Distribution Failure

December 19, 2024 – 7:30am – This morning we had a failure of one of our power distribution units.   It affected a lot of systems which caused most of our services to be offline.  This affected email, web sites, phone systems (including ours), and other systems.  We have identified the issue and implemented a workaround.  Most systems have come back online, but we are still working to bring a few up.  We will be posting more later.

Update – 7:40am – We had reports that the email system was not authenticating.   The backend part of the system failed to come online, but we have brought it online now and all services regarding mail seem to have recovered.  Some email may be delayed in reaching your mailbox.

Update – 7:55am – We believe that all affected systems are online and functioning normally.  If you have a system that is still not working for you, please contact us at support@internetpro.net via the chat in the lower right of this screen, or call us at 256-547-6817.

Update – 8:28am – We were made aware of a few shared websites that did not come back online.  This was because of a web server misconfiguration that would have been caught during normal operation, but the reboot due to the power failure caused the bad config to be loaded and kept some sites from working properly.   This has been resolved.

 

This issue was caused by a failure of one of our network-controlled power distribution systems.  We use that type of system to be able to automatically and remotely reboot systems as well as to check power loads, battery conditions, etc.  The entire bank of outlets failed, and caused some systems without redundant power supplies to go down. Unfortunately, there were a couple of critical bottlenecks there which caused other systems that had not gone offline to be unreachable.   We  will be scheduling maintenance in the near future to replace the unit and that will be done during off-peak hours.  We will also be evaluating the bottlenecks made relevant in this failure and will be working to implement design changes to prevent a recurrence.

 

 

 

 

 

 

 

2024-12-19T14:32:48-06:00December 19, 2024|News and Updates|Comments Off on Power Distribution Failure

Email Account Compromises

December 2, 2024 – Today, we noticed a number of email accounts which had been compromised and accessed from a residential Spectrum account in Pennsylvania.   This was very concerning because these accounts seemed unrelated and so we were worried that attackers had somehow been able to access the password database on our email system.    However, after further investigation, we do not believe this to be the case.   All of the affected accounts were using plain text username and passwords rather than logging in over secure channels (TLS/SSL).   With plain text, the data can be “snooped” over the network and we believe these users had/have compromised routers which are gathering this data.

 

We have already blocked the IP address which was accessing the compromised accounts.  We are currently in the process of reaching out to customers with these accounts to advise them to update their passwords and change any accounts which may have used the same password.   We are also investigating customer’s networks to see if they may have a compromised router.

 

Important Settings Change Coming – For years, we have offered authentication over secure channels (TLS/SSL).  This type of connection is very much like the HTTPS connection you use on a website which takes credit cards.  The connection encrypts the data so that it cannot be read by a third-party.   In the past, this has been optional but recommended.    Because of this incident, we are going to enforce the use of SSL/TLS on ALL client devices on January 14, 2024.  If you are currently connecting with a device that is not using SSL/TLS, it will ask for a password and even if you enter the correct password, it will not authenticate successfully.    Our settings can be found here:  https://netsolinc.com/email-settings     It is important to update those settings before January 14, 2025 to make sure your email client does not get interrupted.   Note: this will NOT affect emails being delivered to our servers.

If you need assistance with updating your settings, please contact us at 256-547-6817 or via the chat box in the lower right of this page.

 

2024-12-02T13:29:29-06:00December 2, 2024|Internet, News and Updates|Comments Off on Email Account Compromises

UPS Failure (Update July 28)

June 22, 2024 – In what seems to be a regular thing now, we had an equipment failure of a battery backup system this morning at about 7:15am.    It affected a few servers including a host for some shared websites, a virtual server host affecting a few cloud  phone system instances (including ours),  a couple of network switches, a firewall, and a couple of backup servers.     Our technicians responded quickly and brought the affected systems back online within just a few minutes.   We are investigating the root cause, as the UPS (Uninterruptible Power Supply) is reporting 100% battery and not reporting a loss of mains power.

Hopefully, as they say, things come in threes and this is the last issue for a while.   If you have any questions or concerns about this, please email us at support@internetpro.net, or call us at 256-547-6817.

 

Update July 28  3:45pm – We’re seeing some strange issues which may be related to this UPS problem.   Some VPS machines and at least one shared host server are being affected.  We’re also seeing some performance issues.   Technicians are investigating.

Update 4:33pm –  It was, in fact, the UPS issue.   We had replaced the batteries in the unit, but it looks like the entire unit will need to be replaced.  We’ll be ordering a new one and we’ll schedule a maintenance window for its replacement.

Update 4:39pm – We believe all affected systems are back online.

 

 

 

 

2024-07-28T16:42:40-05:00June 22, 2024|News and Updates|Comments Off on UPS Failure (Update July 28)

Data Center Internet Outage

06/14/2024 2:58pm – At approximately 1:30pm we were notified of service outages with our mail system, upon further investigation we were able to that routing from our upstream provider was experiencing issues causing our servers to go offline. The service was restored at approximately  2:30pm and all servers are back online. If you are experiencing any issues please let us know by calling our office at (256) 547-6817 or sending us a message at support@internetpro.net.

This outage was not related to the outage we experienced earlier in the week.

2024-06-14T15:08:25-05:00June 14, 2024|News and Updates|Comments Off on Data Center Internet Outage

Data Center Power Failure

06/13/2024 1:49am – We have experienced a power failure at our data center and our generator system did not power up automatically as expected. This allowed some of our systems to use up the available battery capacity and crash. We are currently in the process of running file system checks for our storage systems and restoring services. We will update once all services have been restored. If you have any issues or questions please submit them to us at support@internetpro.net.

06/13/2024 3:20am – All of services have been restored and we will be investigating further into why our backup power failed. If you have any issues please send them to us at support@internetpro.net.

Affected systems include:  VPS, hosted phone systems including our corporate system, some web services, DNS services, mail, and backups.

06/14/2024 3:00pm – The fault with our emergency backup generator has been located and repaired.

 

 

2024-06-14T15:19:56-05:00June 13, 2024|Documentation, News and Updates|Comments Off on Data Center Power Failure

New Remote Email Forward Policy

February 6, 2024 – Beginning today, we will be ending the ability to do “remote forwards” on email accounts.   A remote forward is setup to take an email address which is local to our servers and forward it to another service.   For example, if we forwarded support@internetpro.net (which is located on our servers) to someone@yahoo.com, that would be a “remote forward”.    The problem with this is that when Yahoo! sees the email, they think it is being sent by our servers.  So, if a spammer sends an email to support@internetpro.net and we forward it to Yahoo!, then Yahoo! thinks we are spamming them.

This month, Google has put much more strict policies in place dealing with spam and as a result, emails coming through our systems are being rejected or flagged as spam by Google’s servers.   This is due to the number of remote forwards we have in place, and so we’ve had to make the decision to stop these forwards and institute a policy of not allowing them in the future.

Since Google has implemented this, the rest of the industry who haven’t already done so will quickly line up to do so as well.

Some more information about remote forwards follows for the technically curious:

Why remote forwards are bad

With the anti-spam protections in place in all modern mail servers, forwarding emails to remote servers or addresses is not recommended. The whole industry is moving away from remote forwards. As Best Practices for email server operators, remote forwarding should be disabled due to the potentially system-wide negative effects they could cause, such as backscatter and IP reputation issues.

Backscatter can stem from the remote addresses being no longer valid,  servers not being authorized to send emails for the sender’s domain, or emails being rejected by remote servers due to anti-spam policies in place at the remote site.

As more backscatter events accumulate, they will not only cause higher system load and resource usage, but will also have a negative effect on the IP reputation of the sending servers. A poor IP reputation will cause system-wide delivery issues affecting accounts on those servers. For example, if our server’s IPs are blocked or rate-limited by Yahoo! or Gmail servers, all accounts will experience delivery issues when sending emails to Yahoo! and Gmail addresses. (This is happening now. See:  https://www.netsolinc.com/gmail-blocking-email/)

Aside from the backscatter and IP reputation issues, having remote forwarding available provides a way for bad actors secretly forwarding a compromised account’s emails to a remote address without the account/mailbox owner’s knowledge. For example, if third-party site’s account verification or password-reset emails are being forwarded to a remote address, the bad actor could gain access to the compromised user’s other online accounts.

 

2024-02-06T17:30:54-06:00February 6, 2024|Documentation, FAQ, News and Updates|Comments Off on New Remote Email Forward Policy

Gmail Blocking Email (Updated)

February 6, 2024 – 12:45pm –  We’re seeing email from the @internetpro.net domain being blocked as spam by Google’s servers (gmail.com and others hosted with Google).   The message says something as follows:

—BEGIN

74.125.138.26 failed after I sent the message.
Remote host said: 550-5.7.1 [74.252.14.252       7] Gmail has detected that this message is likely
550-5.7.1 unsolicited mail. To reduce the amount of spam sent to Gmail, this
550-5.7.1 message has been blocked. For more information, go to
550 5.7.1  https://support.google.com/mail/?p=UnsolicitedMessageError o20-20020a81de54000000b006041f08d2dfsi1515206ywl.160 - gsmtp

—END

We have done some testing and found that not all domains we are sending are being blocked.  Some are going to the Spam folder instead of being blocked outright and some emails are getting through.   This seems to be happening because of some new policies being enforced by Google.   There are some articles online such as this one.  However, we have been authenticating our emails as Google is requiring for a very long time, so we don’t know why our domains would be affected.  Here is a report of our SPF record that Google is now requiring.

If you are seeing errors on your domain and it is not @internetpro.net, please reach out to us by chat, call, or email at support@internetpro.net.   We are working to resolve the issue as soon as possible.

Update 4:25pm – This issue is ongoing where @internetpro.net emails are being “hard bounced” to the sender.   Other accounts on our servers are being delivered but are winding up in the recipient’s “spam” folder more times than not.    We are going to institute a new policy against remote forwards and will be making a new post about that.  We will do our best to reach all customers who are utilizing these types of forwards.

 

March 11, 2024 – We seemed to finally fully emerge from Google’s penalty box over the weekend and continue to see normal deliveries today.  We are truly sorry for the issues and the fact that we can no longer forward email to Gmail accounts for customer convenience.   This is a direct result of spammers and the efforts email providers are making to make delivering spam harder.  Unfortunately, it also takes away some of the conveniences and workflows that legitimate people have grown accustomed to using.   Some of the things we are doing going forward:

  • We are making every effort to fix websites and other services whose email deliveries have been affected.
  • Deploying DKIM and DMARC for domains which we host.  We are doing this in an order based on the volume of email being produced, but are also responding to direct inquiries.
  • We are adjusting our normal SPF record policies fro Hard Fail to Soft Fail in an effort to help delivery.  Of course, we will deploy any policy requested by the customer on this.
  • We are now scanning OUTGOING email for spam content.  If it looks like spam to our filters, it will most likely look like spam to the receiving email provider.  We don’t want them thinking we are sending spam, so this is an effort to prevent that.
  • We have changed our policies on how websites authenticate web forms and are developing updated documentation to aid web developers in their deployment.  However, going forward our default recommendation for this is “don’t do web forms”.
  • We have updated our policies on the forwarding of emails and now do not allow “remote” forwards – those that forward email to an address outside of our systems.
  • We have updated our policies on authentication and now the email must be sent FROM the same email account which authenticates it.

We believe these updated policies will keep us safe from being blocked in the future and realize that they impact many users.   That is why we are here to help get any issues which customers find regarding these changes corrected.   If you have any ongoing problems with email, please contact us by phone at 256-547-6817 or email support@internetpro.net.   Forwarding bounce messages as an attachment is always preferred.  Screenshots from mobile devices are rarely helpful in tech support situations.

 

2024-03-11T16:36:28-05:00February 6, 2024|News and Updates|Comments Off on Gmail Blocking Email (Updated)

Connectivity Issues – Updated 2/4/2024

January 30, 2024 – 2:00pm – We started receiving reports of some users having trouble reaching email, web, and/or cloud services about 1:45pm.   All servers/services in our datacenter are responding normally, but we are detecting a lot of route changes upstream.   This is especially affecting Spectrum, Comcast, and AT&T customers.  We have seen improvement, but there are a LOT of routes on the internet, and it does take time for those to settle out.   When repairs occur, the routes are changed again which can cause additional connectivity issues.

Customers are currently seeing  slow page loads and possible timeout errors with sending or receiving email.   This is an issue “upstream” of us, so we have no control over it. It is very similar to a traffic jam on the interstate and the authorities need time to clear it out.

We will be posting updates here as we find out more information.

Update 4:15pm – This does not seem to be improving as we had earlier thought.   It seems to be an ongoing hardware issue between “peers” upstream.    We are working with the providers to get information about an estimated time to repair.

 

Update 5:30pm – Things seem to have cleared up.  If you are still having issues, please let us know by emailing support@internetpro.net.

 

Update February 4, 2024 -10:00am – This problem has cropped up again and seems to be a systemic issue with the interconnections of providers being overloaded.  We’re working on getting updates (again).

Update 10:50am – A fix has been implemented and we are monitoring the results.

 

2024-02-04T10:56:12-06:00January 30, 2024|News and Updates|Comments Off on Connectivity Issues – Updated 2/4/2024

Inclement Weather

January 16, 2024 – Due to inclement weather and hazardous road conditions in our county and surrounding counties, our office will remain closed today.   Technical support will still be available via email at support@internetpro.net, by phone or text at 256-547-6817 by chat by using the chat window on the support page of our website.    We do apologize for the inconvenience but we need to insure the safety of our staff members.      We expect to return to the office for normal business hours tomorrow.

 

 

2024-01-16T07:10:52-06:00January 16, 2024|News and Updates|Comments Off on Inclement Weather

Email Delays to Microsoft

October 11, 2023 9:45am – We detected an increase in emails stuck in our queues that have not been successfully delivered.   After investigating, we’ve found that emails going to Microsoft 365 hosted emails are being delayed with a “Server Busy” message from the other side.   Downdetector is showing that users are reporting problems with Microsoft 365 services.

These emails are still “in line” to be delivered and once the issues at Microsoft are resolved, they should go through.   In the event that the emails are stuck for more than 4 hours, senders will receive a notification message indicating that there has been a delay but the message delivery will be retried for up to 5 days.   We assume that Microsoft will have the problem fixed by then.    We’ll post updates here as we have them.

Update  11:45am – Bleeping Computer has posted a story about the issue, saying it has to do with some new spam rules imposed by Microsoft.  Find the article here: https://www.bleepingcomputer.com/news/microsoft/exchange-online-mail-delivery-issues-caused-by-anti-spam-rules/   We can confirm that the error message mentioned in the article is what we are seeing in our logs.

Update 2:00pm – It looks like things have returned to normal.   We will continue to keep an eye on things, but all indications are that all backed up email messages have been delivered.

 

2023-10-11T14:13:57-05:00October 11, 2023|News and Updates|Comments Off on Email Delays to Microsoft
Go to Top