Thursday 13th July 2017

Network Upstream DNS authority outage, scheduled 6 years ago

At approximately 1 AM EDT (-0400 GMT) monitoring picked up incorrect reverse DNS results upstream from our delegative authority, ns1/2/3/ The issue has been traced back to inconsistent DNS records for as well as incorrect PTR delegation. This has not had an immediate impact on email, because PTR records are cached for up to 24 hours. By 1 PM EDT the problem still persists waiting on a solution upstream. At this time there is limited impact to email deliverability as a mail server checks the IP address matches has a hostname and the hostname matches the IP.

3:18 PM EDT: further digging has yielded that DNS delegation on is incorrect in's zone definition. Waiting to hear back upstream.  86400   IN      NS  86400   IN      NS  86400   IN      NS  86400   IN      NS
;; Received 131 bytes from in 227 ms 86400 IN NS 86400 IN NS
;; Received 117 bytes from in 129 ms

7:02 PM EDT: still working with upstream vendor waiting on response. Confirmed with data center vendor reported an "outage" last night. Whatever that entails is unknown; however, likely to be related to the zone misconfiguration that also affects other class B IP addresses in the neighborhood.

July 15, 12:22 AM EDT: no response from upstream NOC (NetDepot) that manages DNS. Spoke with data center, NS records to mail servers were converted to PTR records around 10:30 PM. No change reported when querying ns1/2/3/ If this problem persists into tomorrow, servers may begin relaying mail through an external network, which will cause issues with SPF records. Hope for the best.

# dig @ -x

; <<>> DiG 9.9.4-RedHat-9.9.4-38.el7_3.3 <<>> @ -x
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 41113
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 0

;       IN      PTR


12:56 AM EDT: reporting PTR records on with a dangling NS authority record for ns2 still present (which usurps authority). Hopefully a step in the right direction and just a matter of cache.

# dig +norec -x

;       IN      PTR

;; ANSWER SECTION: 86400  IN      PTR


1:06 AM EDT: rDNS confirmed in place for mail servers. Mail will continue to flow over the next 2-4 hours to its intended recipients. This fix will remain in place until the NOC corrects its PTR record delegation. Updates will be posted as they come through.

July 17, 3:09 PM EDT: No update yet from NOC. Mail is continuing to flow through as expected; however, reverse DNS assignment is still invalid which has minimal impact.

July 19, 2:52 PM EDT: DNS still not restored. Escalated incident with Zayo supervisor. Supervisor has informed us that through confirmation with engineering that NetDepot changed DNS backends on Friday. The transfer has not gone over as smoothly as expected. Furthermore, NetDepot will not address tickets from Zayo unless they come from Zayo's Accounts Payable department. Supervisor in process of working with engineering team to open a ticket with NetDepot from Zayo's AP department.

Incidentally, Zayo now owns the data center that NetDepot once operated from and NetDepot still has a presence there. NetDepot owns the IP address space. What fun!

3:21 PM EDT: rDNS coming back online. Issue appears to have been resolved.

July 20, 2:20 PM EDT Periodic timeouts reported on Escalated issue with NOC, awaiting response. Remaining nameservers properly forward rDNS to the correct authoritative nameservers (ns1/

3:58 PM EDT rDNS appears to be operational once again across all 4 nameservers.