News and Updates

This category contains information about events that may affect your service such as maintenance or outages. It also contains announcements on new products releases and enhancements.

UPS Failure (Update July 28)

June 22, 2024 – In what seems to be a regular thing now, we had an equipment failure of a battery backup system this morning at about 7:15am.    It affected a few servers including a host for some shared websites, a virtual server host affecting a few cloud  phone system instances (including ours),  a couple of network switches, a firewall, and a couple of backup servers.     Our technicians responded quickly and brought the affected systems back online within just a few minutes.   We are investigating the root cause, as the UPS (Uninterruptible Power Supply) is reporting 100% battery and not reporting a loss of mains power.

Hopefully, as they say, things come in threes and this is the last issue for a while.   If you have any questions or concerns about this, please email us at support@internetpro.net, or call us at 256-547-6817.

 

Update July 28  3:45pm – We’re seeing some strange issues which may be related to this UPS problem.   Some VPS machines and at least one shared host server are being affected.  We’re also seeing some performance issues.   Technicians are investigating.

Update 4:33pm –  It was, in fact, the UPS issue.   We had replaced the batteries in the unit, but it looks like the entire unit will need to be replaced.  We’ll be ordering a new one and we’ll schedule a maintenance window for its replacement.

Update 4:39pm – We believe all affected systems are back online.

 

 

 

 

2024-07-28T16:42:40-05:00June 22, 2024|News and Updates|Comments Off on UPS Failure (Update July 28)

Data Center Internet Outage

06/14/2024 2:58pm – At approximately 1:30pm we were notified of service outages with our mail system, upon further investigation we were able to that routing from our upstream provider was experiencing issues causing our servers to go offline. The service was restored at approximately  2:30pm and all servers are back online. If you are experiencing any issues please let us know by calling our office at (256) 547-6817 or sending us a message at support@internetpro.net.

This outage was not related to the outage we experienced earlier in the week.

2024-06-14T15:08:25-05:00June 14, 2024|News and Updates|Comments Off on Data Center Internet Outage

Data Center Power Failure

06/13/2024 1:49am – We have experienced a power failure at our data center and our generator system did not power up automatically as expected. This allowed some of our systems to use up the available battery capacity and crash. We are currently in the process of running file system checks for our storage systems and restoring services. We will update once all services have been restored. If you have any issues or questions please submit them to us at support@internetpro.net.

06/13/2024 3:20am – All of services have been restored and we will be investigating further into why our backup power failed. If you have any issues please send them to us at support@internetpro.net.

Affected systems include:  VPS, hosted phone systems including our corporate system, some web services, DNS services, mail, and backups.

06/14/2024 3:00pm – The fault with our emergency backup generator has been located and repaired.

 

 

2024-06-14T15:19:56-05:00June 13, 2024|Documentation, News and Updates|Comments Off on Data Center Power Failure

New Remote Email Forward Policy

February 6, 2024 – Beginning today, we will be ending the ability to do “remote forwards” on email accounts.   A remote forward is setup to take an email address which is local to our servers and forward it to another service.   For example, if we forwarded support@internetpro.net (which is located on our servers) to someone@yahoo.com, that would be a “remote forward”.    The problem with this is that when Yahoo! sees the email, they think it is being sent by our servers.  So, if a spammer sends an email to support@internetpro.net and we forward it to Yahoo!, then Yahoo! thinks we are spamming them.

This month, Google has put much more strict policies in place dealing with spam and as a result, emails coming through our systems are being rejected or flagged as spam by Google’s servers.   This is due to the number of remote forwards we have in place, and so we’ve had to make the decision to stop these forwards and institute a policy of not allowing them in the future.

Since Google has implemented this, the rest of the industry who haven’t already done so will quickly line up to do so as well.

Some more information about remote forwards follows for the technically curious:

Why remote forwards are bad

With the anti-spam protections in place in all modern mail servers, forwarding emails to remote servers or addresses is not recommended. The whole industry is moving away from remote forwards. As Best Practices for email server operators, remote forwarding should be disabled due to the potentially system-wide negative effects they could cause, such as backscatter and IP reputation issues.

Backscatter can stem from the remote addresses being no longer valid,  servers not being authorized to send emails for the sender’s domain, or emails being rejected by remote servers due to anti-spam policies in place at the remote site.

As more backscatter events accumulate, they will not only cause higher system load and resource usage, but will also have a negative effect on the IP reputation of the sending servers. A poor IP reputation will cause system-wide delivery issues affecting accounts on those servers. For example, if our server’s IPs are blocked or rate-limited by Yahoo! or Gmail servers, all accounts will experience delivery issues when sending emails to Yahoo! and Gmail addresses. (This is happening now. See:  https://www.netsolinc.com/gmail-blocking-email/)

Aside from the backscatter and IP reputation issues, having remote forwarding available provides a way for bad actors secretly forwarding a compromised account’s emails to a remote address without the account/mailbox owner’s knowledge. For example, if third-party site’s account verification or password-reset emails are being forwarded to a remote address, the bad actor could gain access to the compromised user’s other online accounts.

 

2024-02-06T17:30:54-06:00February 6, 2024|Documentation, FAQ, News and Updates|Comments Off on New Remote Email Forward Policy

Gmail Blocking Email (Updated)

February 6, 2024 – 12:45pm –  We’re seeing email from the @internetpro.net domain being blocked as spam by Google’s servers (gmail.com and others hosted with Google).   The message says something as follows:

—BEGIN

74.125.138.26 failed after I sent the message.
Remote host said: 550-5.7.1 [74.252.14.252       7] Gmail has detected that this message is likely
550-5.7.1 unsolicited mail. To reduce the amount of spam sent to Gmail, this
550-5.7.1 message has been blocked. For more information, go to
550 5.7.1  https://support.google.com/mail/?p=UnsolicitedMessageError o20-20020a81de54000000b006041f08d2dfsi1515206ywl.160 - gsmtp

—END

We have done some testing and found that not all domains we are sending are being blocked.  Some are going to the Spam folder instead of being blocked outright and some emails are getting through.   This seems to be happening because of some new policies being enforced by Google.   There are some articles online such as this one.  However, we have been authenticating our emails as Google is requiring for a very long time, so we don’t know why our domains would be affected.  Here is a report of our SPF record that Google is now requiring.

If you are seeing errors on your domain and it is not @internetpro.net, please reach out to us by chat, call, or email at support@internetpro.net.   We are working to resolve the issue as soon as possible.

Update 4:25pm – This issue is ongoing where @internetpro.net emails are being “hard bounced” to the sender.   Other accounts on our servers are being delivered but are winding up in the recipient’s “spam” folder more times than not.    We are going to institute a new policy against remote forwards and will be making a new post about that.  We will do our best to reach all customers who are utilizing these types of forwards.

 

March 11, 2024 – We seemed to finally fully emerge from Google’s penalty box over the weekend and continue to see normal deliveries today.  We are truly sorry for the issues and the fact that we can no longer forward email to Gmail accounts for customer convenience.   This is a direct result of spammers and the efforts email providers are making to make delivering spam harder.  Unfortunately, it also takes away some of the conveniences and workflows that legitimate people have grown accustomed to using.   Some of the things we are doing going forward:

  • We are making every effort to fix websites and other services whose email deliveries have been affected.
  • Deploying DKIM and DMARC for domains which we host.  We are doing this in an order based on the volume of email being produced, but are also responding to direct inquiries.
  • We are adjusting our normal SPF record policies fro Hard Fail to Soft Fail in an effort to help delivery.  Of course, we will deploy any policy requested by the customer on this.
  • We are now scanning OUTGOING email for spam content.  If it looks like spam to our filters, it will most likely look like spam to the receiving email provider.  We don’t want them thinking we are sending spam, so this is an effort to prevent that.
  • We have changed our policies on how websites authenticate web forms and are developing updated documentation to aid web developers in their deployment.  However, going forward our default recommendation for this is “don’t do web forms”.
  • We have updated our policies on the forwarding of emails and now do not allow “remote” forwards – those that forward email to an address outside of our systems.
  • We have updated our policies on authentication and now the email must be sent FROM the same email account which authenticates it.

We believe these updated policies will keep us safe from being blocked in the future and realize that they impact many users.   That is why we are here to help get any issues which customers find regarding these changes corrected.   If you have any ongoing problems with email, please contact us by phone at 256-547-6817 or email support@internetpro.net.   Forwarding bounce messages as an attachment is always preferred.  Screenshots from mobile devices are rarely helpful in tech support situations.

 

2024-03-11T16:36:28-05:00February 6, 2024|News and Updates|Comments Off on Gmail Blocking Email (Updated)

Connectivity Issues – Updated 2/4/2024

January 30, 2024 – 2:00pm – We started receiving reports of some users having trouble reaching email, web, and/or cloud services about 1:45pm.   All servers/services in our datacenter are responding normally, but we are detecting a lot of route changes upstream.   This is especially affecting Spectrum, Comcast, and AT&T customers.  We have seen improvement, but there are a LOT of routes on the internet, and it does take time for those to settle out.   When repairs occur, the routes are changed again which can cause additional connectivity issues.

Customers are currently seeing  slow page loads and possible timeout errors with sending or receiving email.   This is an issue “upstream” of us, so we have no control over it. It is very similar to a traffic jam on the interstate and the authorities need time to clear it out.

We will be posting updates here as we find out more information.

Update 4:15pm – This does not seem to be improving as we had earlier thought.   It seems to be an ongoing hardware issue between “peers” upstream.    We are working with the providers to get information about an estimated time to repair.

 

Update 5:30pm – Things seem to have cleared up.  If you are still having issues, please let us know by emailing support@internetpro.net.

 

Update February 4, 2024 -10:00am – This problem has cropped up again and seems to be a systemic issue with the interconnections of providers being overloaded.  We’re working on getting updates (again).

Update 10:50am – A fix has been implemented and we are monitoring the results.

 

2024-02-04T10:56:12-06:00January 30, 2024|News and Updates|Comments Off on Connectivity Issues – Updated 2/4/2024

Inclement Weather

January 16, 2024 – Due to inclement weather and hazardous road conditions in our county and surrounding counties, our office will remain closed today.   Technical support will still be available via email at support@internetpro.net, by phone or text at 256-547-6817 by chat by using the chat window on the support page of our website.    We do apologize for the inconvenience but we need to insure the safety of our staff members.      We expect to return to the office for normal business hours tomorrow.

 

 

2024-01-16T07:10:52-06:00January 16, 2024|News and Updates|Comments Off on Inclement Weather

Email Delays to Microsoft

October 11, 2023 9:45am – We detected an increase in emails stuck in our queues that have not been successfully delivered.   After investigating, we’ve found that emails going to Microsoft 365 hosted emails are being delayed with a “Server Busy” message from the other side.   Downdetector is showing that users are reporting problems with Microsoft 365 services.

These emails are still “in line” to be delivered and once the issues at Microsoft are resolved, they should go through.   In the event that the emails are stuck for more than 4 hours, senders will receive a notification message indicating that there has been a delay but the message delivery will be retried for up to 5 days.   We assume that Microsoft will have the problem fixed by then.    We’ll post updates here as we have them.

Update  11:45am – Bleeping Computer has posted a story about the issue, saying it has to do with some new spam rules imposed by Microsoft.  Find the article here: https://www.bleepingcomputer.com/news/microsoft/exchange-online-mail-delivery-issues-caused-by-anti-spam-rules/   We can confirm that the error message mentioned in the article is what we are seeing in our logs.

Update 2:00pm – It looks like things have returned to normal.   We will continue to keep an eye on things, but all indications are that all backed up email messages have been delivered.

 

2023-10-11T14:13:57-05:00October 11, 2023|News and Updates|Comments Off on Email Delays to Microsoft

3CX Updates

August 7, 2023 – The next 3CX update is beginning to roll out and will be applied to all customers by the end of the week.  The new version is 18.8 and will likely be the last in the version 18 series.   Key features for this release include:

  • Simpler and better integration with Office 365
  • Search 365 contacts from within 3CX
  • Improved CRM integration – Search CRM contacts from within 3CX
  • New reports interface from within Web Client
  • Yealink T4 phones can now be used as router phone

 

Coming with the next release will be a requirement for a new user permission level called “System Owner”.   We believe this has been applied to all customers already, but version 18.8 will give a popup in the web client if this is not the case. Please let us know if you see this popup.

If you have any questions or concerns about the new version, please email support@internetpro.net, use the Chat link below, or call us at 256-547-6817.

2023-08-07T08:58:48-05:00August 7, 2023|News and Updates, VOIP|Comments Off on 3CX Updates

Voice Quality Issues

July 31, 2023 5:00pm – We have gotten reports of customers having call quality problems.  This is affecting some of our colocated and cloud hosted VOIP customers, including our internal phone system.  Voice quality for incoming sound is very degraded.   Our engineers have discovered an underlying networking symptom, but the root cause has not yet been determined.  We believe that the issue is upstream of our datacenter, and we have reached out to our preferred SIP provider for assistance.   We will be providing updates here as we get them.

Update August 1 7:00am – We have not yet received any news from upstream providers, but voice quality does seem somewhat improved this morning.  It does not seem to be completely fixed, but communication is possible.   Please note that *users* of the phone systems are experiencing the issue and hearing quality issues from people calling into their system.  The callers do not hear any issues.

 

2023-08-01T06:58:49-05:00July 31, 2023|News and Updates, VOIP|Comments Off on Voice Quality Issues
Go to Top