Caching issue for Now 1.0 deployment in BRU1

This report is a post-mortem of the incident that occurred on the 19th of December, 2018. This incident impacted incoming traffic, via the BRU region, to Now 1.0 deployments that had aliases which were updated between the dates of February 12th and February 15th 2018.

Timeline

At 10:30 UTC an update to our internal caching logic at a routing level exposed an issue that affected aliases routed by Now 1.0 in the BRU region that were updated between the dates of February 12th and February 15th, 2018.

At 13:50 UTC reports alerted us to this issue and users started noticing 404s for their Now 1.0 routed aliases in BRU1. Investigation began immediately.

At 14:00 UTC a fix was deployed to fix the caching issue.

At 14:35 UTC the fix was confirmed to have resolved the caching issue and all aliases affected were accessible again.

Conclusion

This issue affected only 0.05955694% of existing aliases, those of which were updated between the dates of February 12th and February 15th, 2018 routed by Now 1.0 in the BRU region.