Background job processing (including ordering jobs) down
Incident Report for Subscribe Pro
Resolved
Background job processing went down on 10/1 or us-east-1 region and 10/4 for eu-west-1 region. Both incidents were due to the same root cause:
* A key scheduling service failed.
* This was triggered by our core transactional database recycling after a minor version upgrade.
* This happened around 3am eastern time on 10/4 in us-east-1 region.
* Happened around 3am eastern time on 10/1 in eu-west-1 region.
* We should have noticed this, however there was an error in some of our alerting / notification scripts.
* We have since updated and thoroughly tested the notification scripts.
Posted Oct 04, 2022 - 19:05 EDT