July 27, 2019 – We’ve had a couple of reports of email going to our users being delayed. The sender may receive a message from the mail server with a message similar to the one below:
—BEGIN
The response was:
DNS Error: 49391460 DNS type ‘mx’ lookup of <domain hosted with us> responded with code NOERROR 49391460 DNS type ‘aaaa’ lookup of mail.internetpro.net. responded with code SERVFAIL 49391460 DNS type ‘a’ lookup of mail.internetpro.net. responded with code SERVFAIL
—END
We checked name lookups and initially found no problem. However, after a few reports came in, we did find that our tertiary name server (ns3.internetpro.net) was not responding properly. This seems to be only affecting IPv6 requests, and Google seems to prioritize those as they seem to be the only email provider having issues with our server. So, users whose email is serviced by Google may receive the above message. This is not a permanent error and the email will go through.
We are currently troubleshooting the issue affecting the name server and will post updates here.
Update 2:45pm – The server seems to be answering IPv6 queries correctly now. We’re not entirely sure what caused the issue, but will be checking it periodically. If you receive a delay like the one described above and it is time stamped *after 2:45pm today*, please let us know by forwarding it to support@internetpro.net. We understand that you may receive one of these messages if you check your email after this time, but we are specifically looking for messages which reached your Inbox after this time. Please make sure that the person receiving the delay message send it to support@internetpro.net with a different account. They will most likely receive the same message sending to support@internetpro.net. If needed, please call 256-547-6817 for help in getting one of these messages to us.
Update 9:00pm – We’ve had a report of this still happening. As a preventative measure, we’ve taken the tertiary server and IPv6 offline. This should force all queries to the other servers which do not seem to be giving this error.
Update 9:45pm – Our workaround has definitely worked as we are seeing delayed deliveries coming in now. Even our support mailbox has received a few.
Update July 30 11:00am – We are still getting reports of some affected domains. This is mostly domains which are using our name services but not our email services. We are addressing workarounds as we receive reports. The update above regarding delivery to our servers still seems true as we haven’t had any further reports of that issue.
Update 2:25pm – We received a report from a customer with G-Suite services from Google. They were able to chat with Google support. Below is a partial transcript:
–BEGIN
Google Cloud Support, Iva: Upon checking it seems the MX records are not resolving for all servers
Google Cloud Support, Iva: That explains why you are able to get some emails but others are not delivered
Google Cloud Support, Iva: The MX records are there but they are not recognized by all servers
Google Cloud Support, Iva: It would be best to check with the domain host to see why exactly this is happening
<customer>: ok do I need to recheck the MX records for the domain?
Google Cloud Support, Iva: From what I see the records themselves are correct. The issue is with the fact some servers do not detect them
–END
The way we are reading this is that the support agent confirmed that our servers are giving correct answers but Google’s servers are not reading them properly. Unfortunately, getting Google to fix the problem may be tough.
Update 7:15pm – We’ve gotten a Google reseller (Giorgio Torregrosa of PC-911 ) to open a ticket on our behalf! We’re hoping for a call from someone at Google soon. Thanks Giorgio!!
Update 8:15pm – Google is able to reproduce the issue on an internal lookup tool and they see that it doesn’t happen on *any* tests outside of their system. They have escalated the issue to find the problem on their end.
July 31 7:00am – We are still awaiting an update from Google
12:00pm – We reached out to Google via email for an update but have not heard back.
8:30pm – Google reports that their engineers were not able to reproduce or see a problem. We have relayed that the issue is still ongoing and that they need to look into it some more.
August 1, 7:15am – Google has requested that we send them some “recent examples with timestamp and Msg-ID”. This is pretty tough to do for us because almost everyone sends a screenshot from their phone which do not include these items. If you know someone who has received a delay message, please have them forward the original *as an attachment* to support@internetpro.net. Forwarding it as an attachment will allow the message to retain the items requested by Google. Note, this message will need to come from the original sender. Our customers can’t forward an example which may have been forwarded to them by the sender.
10:30am – We’re receiving reports from customers that email deliveries seem to be improved for them. Since the delays sometimes last 24-48hrs, it is tough to say the issue is fixed, but we are getting some anecdotal evidence that things are better.
August 2, 7:00am – Google has reported that they are no longer able to reproduce the problem with their internal tools. We have also received more reports that customers are successfully getting timely deliveries and have not received any reports of delays since late yesterday. We *believe* the problem may be cleared up, but the nature of these delays is that they sometimes take 24-48 hours to be reported by Google, so right now we are crossing our fingers. If anyone has a delay for a message sent starting in the afternoon of August 1 or later, please let us know by emailing support@internetpro.net.
August 3, 10:00am – We have not received a report of this happening for over 48 hours, so we do believe this issue has been resolved by Google.