tnolen

November 22, 2019 2:30pm– We’ve been seeing the spread of a virus/trojan in the Trickbot family.  These are spread using the older Word document format (files ending with .doc).  They are not being spread using the newer Word document format (ending in .docx).   We are temporarily blocking all .doc attachments to try to limit the spread of this trojan.  If you need to send someone a .doc file and you can’t convert it to .docx, we recommend that you put it into a .zip archive first and then send it.    Normally, .doc files can be converted to .docx by just using a version of Word newer than 2003 and saving it as the new format. We apologize for any inconvenience this may cause.

Update 4:00pm  –  It seems that we have inadvertently blocked .docx files as well even though we explicitly told the system not to.  We are investigating the issue.
Update 4:15pm – We’ve corrected the issue so only .doc files are being blocked as we originally intended.

November 25 – 2:30pm – We do understand that the blocking of .doc files is an inconvenience for some customers and we do apologize.  However, since 6:30am today, we have blocked very close to 3000 of these viruses and have been able to keep at least one customer from spreading this infection.

November 26 – 9:00am – We’ve seen a drastic reduction in the number of .doc files being blocked, so we believe the majority of infected computers have been found/cleaned/quarantined.  Therefore, we have removed the restriction for .doc files.

Update 11:30am – It seems that we were a bit premature at lifting this restriction.  We’ve gotten complaints of this virus continuing to spread, so we’ve applied the restriction to .doc files once again.

December 9, 2019 – 5:00pm – We’re stopping much fewer .doc files now and we feel that many of the ones we are stopping are legitimate (based on tech support contacts).   Therefore, we’ve again lifted the ban on the .doc files.  If we see an uptick of the trojan spreading again, we’ll block them and post updates here.

If you have any questions about this temporary policy, please email us at support@internetpro.net or utilize the chat window in the lower right part of your screen.

November 22, 2019 – We will be performing some planned maintenance on our mail server system starting at 11:00pm Thursday the 28th (yes, Thanksgiving).   The maintenance window is 3 hours ending a 2:00am on Friday the 29th.   During part of the maintenance window, email services will be inaccessible.   We’ll do our best to keep the time to a minimum.   We are implementing a new storage system which will improve performance and allow future growth of our email offerings.   This will be “phase 1” and after it is complete, we’ll be moving data to the new system.  Once the data has been synchronized, we’ll have another brief maintenance window to finalize the sync and make the storage system live.  We will announce this second window when we have more information on the timing.

During these windows, no email will be lost.  While the mail server system is inaccessible, any incoming email will “queue up” and wait for the system to come back online.  After the system is online, all delayed email will be delivered.

Update:  November 28 – 11:00pm – Maintenance has started.  Mail services going offline starting now.  Updates to follow when back online.

Update: 11:25pm – All services are back online.  We don’t expect any further interruptions for “phase 1”.

If you have any questions or concerns, please email us at support@internetpro.net, call 256-547-6817, or utilize the online chat window in the lower right corner of  your screen.

November 20, 2019 1:00pm – We are seeing city wide issues with Comcast today (busy day…).   These are showing up as internet down completely or very slow speeds.  For most customers, powering down the modem, waiting for about 10 seconds, and powering it back up has been resolving the issue.  Please note:  It is taking longer than usual after the power up before internet is restored.  It may take up to 10 full minutes.

We have seen these issues very widespread in Rainbow City and reaching to downtown Gadsden.

November 20, 2019 – 11:00am – We have received reports indicating that we have been blacklisted by Barracuda’s spam system.  Barracuda’s spam filters are in wide use, so this affects many recipients.   We are working to find out the reason for the blacklisting and are asking to be removed.   These type of listings take a bit of time as not all internet service providers and mail server operators synchronize their lists at the same time or in the same interval.   In the mean time, if you have sent email to a recipient and received a bounce, you can ask the recipient to put you on their whitelist so that your email will flow until this is resolved.

A sample bounce message will contain this information:

–BEGIN

Remote host <IP address> does not like recipient <recipient email address>
Remote host said: 554-Service unavailable; Client host
[mail.internetpro.net] blocked using 554-Barracuda Reputation;
554 http://www.barracudanetworks.com/reputation/?r=1&ip=74.252.14.252

–END

We do apologize for the inconvenience and will post updates here as we have them.

Update 11:22am – It seems that Barracuda Networks is having some issues.  This may be the reason for the apparent blacklisting.  Their website and reporting tools are almost unresponsive and their monitoring service is unreachable completely.

Update 11:40am – We’ve been able to submit a removal request and are awaiting the results of that.  In addition, we believe we’ve tracked the cause of the listing.  Surprisingly, the source sends very little spam, which is probably why only one blacklist has listed us.  We’ve blocked the possible source and are continuing to look for any other issues.

Update 1:15pm –  We’ve confirmed removal of our servers from Baracuda’s blacklist.  However, some ISPs and spam appliances will take some time to update their local lists.  In our tests, some email is still being rejected, but we expect this to clear over the next few hours.

November 12, 2019 –  Starting at approximately 1:15am CST, our systems detected an area wide outage of Comcast. This is affecting many of our customer phone systems, VPNs, and other services.  We are investigating.

Update 2:15am – Comcast seems to have cleared up the issue.  It lasted approximately 50 minutes, but systems are online now in the Gadsden area.  We’ve detected a few down systems outside of Gadsden but expect that they are due to the inclement weather.

October 22, 2019 – 8:00pm – AT&T is experiencing a *major* outage which is affecting many of our systems.  Phones are currently offline but are being redirected to tech cell phones as a failover.  Eventually, we’ll have time to actually implement real failover on our phone system so that doesn’t happen, but not today. 🙁    The outage is outside of our network, and we have not been able to get an estimated time to repair (ETR) from AT&T.   We are seeing reports online of the outages, and it seems to be widespread.  https://downdetector.com/status/att/

Update 9:45pm – After being on hold with AT&T, we’ve finally been issued a ticket to track the issue.  There is still no ETR, but it seems that AT&T is at least aware of the issue.

Update 10:00pm  – Service seems to be restored at least in this area.  We are seeing increased connectivity on our systems.

September 9, 2019 – We are experiencing some technical difficulties with our email server. The webmail interface is currently down.

Update: 12:37pm – Service is restored.

August 15, 2019 – We will be performing some upgrades on our email system which will briefly interrupt email connections.   We will be performing the maintenance in a window tonight between 6:00pm and 7:00pm.   We are doing this outside of our normal “late night” maintenance window so that software support services will be available in case of an error.

Most of the upgrades are going to be “under the hood” including an operating system upgrade.  There are going to be some changes to the web interfaces and to some system functionality, but most of the changes other than the web interface will go unnoticed by end users.  One small change that you may notice is that the calendar in Portal and Tuxedo will now fast forward or scroll to a time block relevant to ‘now’ and in addition, users will be able to set “work hours” which will also be displayed differently on week/day view.

We’ll be posting updates here as the work progresses.   If you have any questions, please contact us at support@internetpro.net, chat using the Online Support window in the lower right of your browser, or call us at 256-547-6817.

Update: 6:15pm – Maintenance has started and we’re waiting on a filesystem check to complete.   As soon as it does, services should be restored for the most part, with only brief interruptions expected.  The filesystem check has bene going for about 15 minutes now.

Update: 6:16pm – Services are back up.

Update 6:34pm – The upgrade ran into an issue.   Services are still online, but the upgrade has not completed on the server.   We are working to resolve the issues.

Update 6:43pm – Issues have been corrected.  Upgrade is continuing.  Services are still online.

Update 6:45pm – Services going offline for one final time.  We expect less than 5 minute interruption and the process will be complete.

Update 6:48pm – Services back online and the maintenance is complete.   We are doing “backend” checks, but there should be no more customer impact.

Update 6:57pm – It seems that there are some small cosmetic issues which we need to resolve, especially in the “Mail Settings” interface.  We’ll be working on those shortly, but the functionality is not impacted – just the look.

July 27, 2019 – We’ve had a couple of reports of email going to our users being delayed.  The sender may receive a message from the mail server with a message similar to the one below:

—BEGIN

The response was:

DNS Error: 49391460 DNS type ‘mx’ lookup of <domain hosted with us> responded with code NOERROR 49391460 DNS type ‘aaaa’ lookup of mail.internetpro.net. responded with code SERVFAIL 49391460 DNS type ‘a’ lookup of mail.internetpro.net. responded with code SERVFAIL

—END

We checked name lookups and initially found no problem.  However, after a few reports came in, we did find that our tertiary name server (ns3.internetpro.net) was not responding properly.   This seems to be only affecting IPv6 requests, and Google seems to prioritize those as they seem to be the only email provider having issues with our server.    So, users whose email is serviced by Google may receive the above message.  This is not a permanent error and the email will go through.

We are currently troubleshooting the issue affecting the name server and will post updates here.

Update 2:45pm – The server seems to be answering IPv6 queries correctly now.  We’re not entirely sure what caused the issue, but will be checking it periodically.   If you receive a delay like the one described above and it is time stamped *after 2:45pm today*, please let us know by forwarding it to support@internetpro.net.  We understand that you may receive one of these messages if you check your email after this time, but we are specifically looking for messages which reached your Inbox after this time.   Please make sure that the person receiving the delay message send it to support@internetpro.net with a different account.   They will most likely receive the same message sending to support@internetpro.net.   If needed, please call 256-547-6817 for help in getting one of these messages to us.

Update 9:00pm – We’ve had a report of this still happening.  As a preventative measure, we’ve taken the tertiary server and IPv6 offline.   This should force all queries to the other servers which do not seem to be giving this error.
Update 9:45pm – Our workaround has definitely worked as we are seeing delayed deliveries coming in now.  Even our support mailbox has received a few.
Update July 30 11:00am –  We are still getting reports of some affected domains.  This is mostly domains which are using our name services but not our email services.  We are addressing workarounds as we receive reports.   The update above regarding delivery to our servers still seems true as we haven’t had any further reports of that issue.
Update 2:25pm – We received a report from a customer with G-Suite services from Google.  They were able to chat with Google support.  Below is a partial transcript:

–BEGIN

Google Cloud Support, Iva: Upon checking it seems the MX records are not resolving for all servers
Google Cloud Support, Iva: That explains why you are able to get some emails but others are not delivered
Google Cloud Support, Iva: The MX records are there but they are not recognized by all servers
Google Cloud Support, Iva: It would be best to check with the domain host to see why exactly this is happening
<customer>: ok do I need to recheck the MX records for the domain?
Google Cloud Support, Iva: From what I see the records themselves are correct. The issue is with the fact some servers do not detect them

–END

The way we are reading this is that the support agent confirmed that our servers are giving correct answers but Google’s servers are not reading them properly.  Unfortunately, getting Google to fix the problem may be tough.

Update 7:15pm – We’ve gotten a Google reseller (Giorgio Torregrosa of PC-911 ) to open a ticket on our behalf!  We’re hoping for a call from someone at Google soon.  Thanks Giorgio!!
Update 8:15pm –  Google is able to reproduce the issue on an internal lookup tool and they see that it doesn’t happen on *any* tests outside of their system.  They have escalated the issue to find the problem on their end.

July 31 7:00am – We are still awaiting an update from Google
12:00pm – We reached out to Google via email for an update but have not heard back.
8:30pm – Google reports that their engineers were not able to reproduce or see a problem.  We have relayed that the issue is still ongoing and that they need to look into it some more.

August 1, 7:15am – Google has requested that we send them some “recent examples with timestamp and Msg-ID”.  This is pretty tough to do for us because almost everyone sends a screenshot from their phone which do not include these items.  If you know someone who has received a delay message, please have them forward the original *as an attachment* to support@internetpro.net.  Forwarding it as an attachment will allow the message to retain the items requested by Google.  Note, this message will need to come from the original sender.  Our customers can’t forward an example which may have been forwarded to them by the sender.

10:30am – We’re receiving reports from customers that email deliveries seem to be improved for them.  Since the delays sometimes last 24-48hrs, it is tough to say the issue is fixed, but we are getting some anecdotal evidence that things are better.

August 2, 7:00am – Google has reported that they are no longer able to reproduce the problem with their internal tools.  We have also received more reports that customers are successfully getting timely deliveries and have not received any reports of delays since late yesterday.   We *believe* the problem may be cleared up, but the nature of these delays is that they sometimes take 24-48 hours to be reported by Google, so right now we are crossing our fingers.   If anyone has a delay for a message sent starting in the afternoon of August 1 or later, please let us know by emailing support@internetpro.net.

August 3, 10:00am – We have not received a report of this happening for over 48 hours, so we do believe this issue has been resolved by Google.