News and Updates

March 4, 2022 – 10:00am – One of our web servers has suffered a failure that is affecting some of our shared hosting sites.  We are working to address the problem.  Originally, a RAID array lost a drive and it was replaced as normal, but apparently the rebuild process failed and we are seeing degradation in service.  We will be posting updates here as we have them.

Update 1:45pm – The server with the failed RAID array is currently offline as we work to repair the issue. Some sites will be offline until this process is completed. We will update this post with further information as we continue the repairs.

Update 6:45pm – We’ve recovered the filesystem, but unfortunately have some corruption in the database system of the affected server.  We’re working on restoring from backups but are running into some corruption issues.

Update 10:30pm  – We have successfully restored a few sites and have a procedure for restoring the rest.  Sites will be coming up as we restore them.

Update 11:15pm – We believe all sites except for one have been restored to full functionality and we have informed the owners of the affected site.   If you are still experiencing an issue with your website, please email us at support@internetpro.net to let us know and we will investigate it further.  We do apologize for this issue.

March 5, 2022 – 8:30am – We’re going to attempt one last effort to get the one website still having issues recovered.  The effort will affect all other sites briefly.  We’ll update here as to the success or failure and the conclusion of the event.

Update 8:50am – Failure.   The site in question was in the midst of a redesign, so we’re pressing forward with that.  There should be no more service interruptions related to this outage.

February 15, 2022 – We’re seeing a new batch of phishing emails seeking to harvest usernames and passwords from the email system. Please remember that we will never send out emails asking you to verify your passwords.  It is important to stay diligent regarding these types of communications – not just with our email but with things like your bank account, taxes, social media, or any other login you have.  Here is an example of what is being sent today:

If you have any questions or have already interacted with this email, please email us at support@internetpro.net, call 256-547-6817, or use the chat window in the lower right of your screen.

January 17, 2022 – Well, we made it through 2 years of the pandemic before Covid hit our office, but it has come with a vengeance.  For the next few days, we may not be able to respond as quickly as normal for service calls due to a large portion of our staff being sick and/or quarantined.   We will be doing as much as we can using our remote tools such as iFoye to reduce further exposure.   Please bear with us as we work through the process of getting our staff better.

Some staff working remotely are also having a bit of trouble talking on the phone due to coughing, so if possible, we recommend the chat box at the lower right of your browser or email support@internetpro.net if possible.

Normally, the virus icon is reserved for computer viruses, but I thought it was appropriate today…

Thank you for your understanding and patience.

Update January 18, 2022 – We have yet another staff member out today, so our ability to respond continues to be limited.

Update January 21, 2022 – More staff members are out.  We will most likely be closed today for lunch from 1:00 to 2:00pm.  We apologize for the inconvenience.

January 1, 2022  12:00pm – Happy New Year!  The new year brings more of the same old problems.  In this case, a bug in the anti-malware engine of Microsoft Exchange servers has caused email delivery to stop.  Customers who have their own on premise Exchange server will be affected by this.   There is no official patch just yet.   The bug is in the anti-malware service on these servers and the current work around is to stop this service.  You can find an article with more information about this bug here.

Email delivery will be delayed and senders who send to people using these Exchange servers may/will receive a “delayed delivery notification”.   As long as the issue has been addressed within a few days, those emails will eventually be delivered.  If any sender has a short retry time setup on their email, they will receive a second “failure notice” indicating that the email did not go through and will not be tried any longer.   The industry standard is for email delivery to be retried 5 days, but this can be adjusted by individual email operators.

If you have any questions or concerns, please email us at support@internetpro.net. Please keep in mind that our office is closed for the holiday, so you may not receive an immediate response.   We are working on implementing the workaround for customers with proactive support contracts and those who report the issue to us and wish it fixed at holiday/after hours rates. This will allow the email to be delivered.  For others, we will be rolling out the workaround or fix if it has been rolled out during normal business hours Monday.

Update – Microsoft has released an emergency fix which does require manual application.   We are working to put this fix in place at normal rates for affected customers.

December 8, 2021 – The 3CX company has updated the Chrome extension many of our customers were using.  They actually have replaced the old extension with a new “Click2Call” extension.  The new extension does not have the same functionality as the old one with the small window showing phone controls, presence, etc.   The new extension only allows you to click phone numbers in your browser to initiate a call.   The full web client is still available, but for those who want the smaller popup window, you will need to install the new Windows desktop app.

For customers under maintenance with Network Solutions, Inc., we’ll be glad to provide free assistance and instruction for updating to the new desktop app for Windows users.  Users of Mac or Linux will need to use the web client as the desktop app is not available.  It is possible to load the old extension for Mac and Linux, but it will stop working completely in June 2022 when it is removed from the Chrome app store.

To install the desktop app yourself, simply navigate to the web client (most of our customers have this bookmarked).  It is something along the lines of https://companyname.al.3cx.us:5001/webclient usually.    Once logged in, there will be a Microsoft logo in the upper left.  Click that to pop up an instruction windows.  There are two steps: 1) Install and 2) Provision (activate).

Installing will download a program to run. Once that is run and installed, click the second step and the app will provision itself.

If you have any questions or need assistance, please contact our office at 256-547-6817 or use the Chat window below.   Please note that this was not a  scheduled update, so we have many unexpected requests for help.  We may need to schedule a return call as a result.

December 7, 2021 –  AWS is the underlying infrastructure for a good portion of the web.  As such, the degradation of their services is affecting MANY sites and services.  These include, but are not limited to Control4’s 4Sight platform, social media websites, cloud software providers including practice management and accounting programs, and much more.  Directly affected are Amazon’s own services including Ring, Alexa, and their shopping/shipping services.  If you are having an issue online, chances are it is related to this ongoing issue.

Here are some links which provide more information, including details about today’s outage:  https://apnews.com/article/amazon-web-services-outage-978394126305

The AWS services dashboard:  https://status.aws.amazon.com/

Downdetector is a good source showing many sites which are reporting related issues:  https://downdetector.com/

Here at Network Solutions, Inc., we don’t rely on any AWS platforms, so our services are unaffected.

September 27, 2021 – 9:30am – We are experiencing an issue with a shared web server which is affecting some customers.   We are currently investigating the issue as it is taking longer to recover than we initially thought.  We’ll be posting updates here as we get them.

Update 10:03am – Services have been restored.  A combination of DOS (denial of service) attack and hardware failure was responsible.  We are working on a hardware fix.

If you have any questions or concerns about this, please email us at support@internetpro.net, text or call 2565476817, or use the chat box in the lower right of your screen.

September 25, 2021  3:30pm  – We are getting reports of our customers not being able to receive calls from Verizon.  Right now, only Verizon has been tested.   We’re working on other tests to see if we can narrow down the issue.

We will post updates here as we have them.

Update 3:40pm – We have confirmed that this is a Verizon issue.   It is affecting Verizon wireless customers and customers of cellular services which use their networks such as Spectrum and Straight Talk.   Downdetector is also reporting Verizon issues here: https://downdetector.com/status/verizon/.

Update  3:55pm – It looks like the worst has passed.   Downdetector is showing that the reports of issues are starting to drop and we have had reports of Verizon calls going through to our customers now.

September 27, 2021 – 10:00am – We are not sure the issues are related to this weekend’s events, but our carrier has posted a bulletin stating this:
“NOTICE: Currently a large underlying carrier is experiencing intermittent service disruption. This outage is causing inbound calls of some numbers to not be delivered to SIPTRUNK. We apologize for any service interruption that these issues may cause. “

Update 11:30am – We have discovered that DTMF tones are no longer working on digital receptionists.  DTMF tones are the tone made by your phone when you dial on the keypad and they are used to choose options on the menus in phone systems (for instance, push 1 for sales, push 2 for support, etc.).   Because of this, people are not able to make choices on those menus.  Some customers have their systems set to hang up if no choice is made to weed out telemarketers.  Right now, those systems may be weeding out everyone.  If this seems to be the case on your system, please contact us and we can change the system so that “no choice made” goes somewhere. We have alerted our carrier to the issue in case they are unaware.

Update 11:48am – This seems to be a widespread issue as we have now received a report from a customer using AT&T as the carrier who is having similar issues.

Update 12:00pm from SIPTRUNK – “Our carrier is experiencing an intermittent issue with inbound and outbound calls. This includes audio & DTMF tones. But static is usually only introduced at analog equipment, so we suggest rebooting any physical devices you might have, including router, modems, firewalls, PBXs, etc. But we have multiple tickets open with the carrier. All teams are actively engaged at this time. Currently no ETA. We will provide an update as soon as one is available.”

Update 3:45pm – SIPTRUNK reports that the issue is still under investigation.

Update 6:35pm – Our carrier reports that the network behavior has returned to normal.  We are changing phone system programming that was changed because of the DTMF issues.   Please let us know if you are still experiencing abnormal phone system behavior.

Update September 28 8:45am – Although the carrier reported that things were back to normal, we are still receiving reports or issues (and experiencing them ourselves).   We have reported this and are awaiting news.

Update 4:30pm – Updates have been slow to come today.   Our carrier has given us a bit more information now though.  They are seeing about 1/7 the tickets today as yesterday, so things are improving.   The issue is not a misconfiguration, but rather is a DDOS attack directed at VOIP carriers.  The affected carriers have not released that it is a ransomware attack, but that happened recently to a large Canadian carrier last week and the rumor is that this is related and may be the same attackers.   Earlier, Verizon was the suspected source of the issue and even Verizon themselves thought they were the problem.  However, the underlying carrier currently being attacked is Bandwidth. Because of the interconnections between companies, it seemed to be a Verizon problem at first because they had the most end users affected.   Bandwidth is a large “behind the scenes” carrier.   You can check their Down Detector information here: https://downdetector.com/status/bandwidth/   and their own status page here: https://status.bandwidth.com/.   We expect them to post “everything normal” later today as off-hours voice traffic drops a lot and reduces the impact of the attacks, but we may see the same issues return tomorrow if they have not been able to mitigate the attacks.

Update September 29  8:45am – Currently, it looks like a repeat of yesterday, where daytime business traffic is starting to have an impact on performance of the network.  Downdetector is showing an increase of reported issues again: https://downdetector.com/status/bandwidth/.   Our carrier is working on diversifying their routes to other upstream carriers, but this involves a lot of manual porting of hundreds of thousands of numbers, so this will take some time.   Mitigation of the attack at the carrier level is improving, so we expect less problems as time goes on, but there *are* still issues happening.

Update 10:45am – Carrier has posted an issue this morning but are currently saying it only affects inbound calls from the Verizon network.  Of course, other providers using the Verizon network such as Spectrum Mobile, Straight Talk, etc. will also be affected.  Here is another recent news article with a little more information:  https://www.zdnet.com/article/bandwidth-ceo-confirms-outages-caused-by-ddos-attack/

Update 4:00pm – Siptrunk sent out a letter regarding the issue.  No real extra news, but some clarifications.   Here is an excerpt from the letter we received:

–BEGIN

As you may be aware, one of the US’s largest underlying carriers (ULC) has been under a Distributed Denial-of-Service (DDOS) attack this week. This attack is similar but larger in scope than the attack on VoIP.MS that started on September 16th.    Since this is a ULC we can easily route around them on outbound calls, but this is not the case for inbound calls. The ULCs serve as the “carrier of record” for the DIDs or telephone numbers used by our customers. The current design of the US telephone network means that all calls destined to a particular telephone number have to be handled by the carrier of record. The only way to change this is to port or move the number to a new carrier of record and for landlines, this process can take 5 to 7 days under normal circumstances.

SIPTRUNK is investigating all options to address intermittent inbound call issues caused by the DDOS on the ULC, but please know this is affecting large and small customers the likes of which including Microsoft Teams, Google voice, RingCentral, etc… to just name a few.

We have seen significant improvement on the ULC that is being attacked, and we have confidence they will get their network restored to 100% capacity as quickly as possible.

However, with that said SIPTRUNK has been working with other ULCs to arrange porting of DIDs to alternative ULCs. As you can imagine there are a number of carriers and customers attempting mass ports and this is causing congestion in the porting process. We will continue to work to have inbound calls delivered via alternative ULCs.

–END

September 30 – 7:15am – For the technically curious, here is a good video between a well-known MSP (Managed Service Provider) and a VOIP provider on Youtube discussing the issue.  It uses a lot of jargon, but may be interesting to some of you.  https://www.youtube.com/watch?v=CeCd9zW-R5o.  It is from yesterday, so when they say “yesterday”, they are talking about the 28th.  However, not much has changed.

Update 1:45pm – Upstream providers report much better stability of the network today.  There are still some users reporting issues, but the number and severity seems to be diminished.

Update 4:15pm – SIPTRUNK has released another statement.  Here are the relevant parts:
–BEGIN
All SIPTRUNK Network Services remain operational. We continue to monitor the underlying carrier (ULC) that has been impacted over this week. At this time, they remain under a DDoS attack however they are taking the necessary steps to limit service disruptions. We continue to see significant improvements as they fine-tune their DDOS mitigation efforts, and we have confidence they will get their network restored to 100% capacity as quickly as possible.

We continue to collaborate with other ULCs to organize emergency ports from the impaired carrier to their services. It has become clear that the DDoS attacks are impacting the ability of Providers to port numbers. Providers are experiencing significant porting delays and we are also aware of the potential disruptions that could occur when moving numbers. We will continue to test and try alternative solutions to move beyond the control of the attackers.

Please understand this is not just an attack on a ULC but an attack on the United States Infrastructure and it has visibility at all levels of the federal government.
–END

Update October 1 4:15pm – All ULC services have been stabilized and calls are processing normally.  We performed a scheduled port today and it went through without issue. We believe this issue has now been resolved.

If you have any questions, please text 256-547-6817 or email support@internetpro.net.

09/01/21 8:09 PM: We have been informed of a widespread Comcast outage affecting a large number of our customers. Currently we are being told that they estimate services to be restored around 11:34 PM.

We will update this post if more information becomes available to us.

Update:  Services were restored pretty close to the expected time frame.  If you are still experiencing Comcast issues, please call or text us at 256-547-6817 or email support@internetpro.net.

Update 09/02/2021 2:00pm – The outage seems to be happening again.  It is affecting the same customers as well. Current ETR (estimated time to repair) is 5:51pm although we do see some devices coming bak online as of 2:12pm.

Update 3:25pm – It looks like most customers have come back online now.

Updated 9:30pm –  There was another outage from about 5:30pm to around 9:15pm affecting the same node.  We’ve been told that the issues are fiber related and have to do with the roll out of a new fiber product