Issues with an Upstream DNS Provider

This report is a post-mortem of the incident that occurred on the 22nd of August, 2018. This incident impacted access to the `now.sh` domain and all subsequent deployments under it. This outage did not affect custom domains.

Timeline

At 08:00 UTC our on-call engineers were alerted that the `now.sh` domain stopped responding to our checks. These engineers found that the cause was due to our upstream registrar which had shut down the domain due to a phishing report.

By 08:39 UTC, our engineers were in contact with the upstream registrar and working on re-activating our domain.

At 10:57 UTC our engineers had resolved the issues with the registrar and began restoring availability to all deployments under the `now.sh` domain.

At 11:01 UTC all availability to deployments under the `now.sh` domain was restored.

Conclusion

Although deployments under the `now.sh` domain we inaccessible, those aliased to custom domains were not affected.

Post Actions

We maintain very active communication channels with all our major providers, including our registrars.

The next step for us is to work with the upstream registrar to gain clientUpdateProhibited status to ensure this incident is never replicated. We will also work on a fallback for `now.sh` as a precaution so that there will be no possibility of users being affected in the future.