Degraded Performance
Incident Report for Subscribe Pro
Postmortem

We experienced downtime today between 1:30 and 1:52 eastern today. First, I'd like to apologize on behalf of our whole team. We understand that our platform is important and that you, our customers are relying on our platform to be up and running so you can accept orders and payments from your customers.

The root cause was a test we ran on our database, to prepare for an upcoming infrastructure migration. We were running a test on our database to prepare for migration of our platform to a new hosting provider. We simply didn't anticipate the risk of running this particular test. We clearly should have anticipated the risk in this case.

Currently we're rethinking our process of risk assessment for any actions we undertake on our production infrastructure. We plan to formalize this process this week. By making the risk assessment process explicit and formal, we expect to avoid issues like this one in the future.

Posted Aug 04, 2016 - 15:25 EDT

Resolved
This incident has been resolved.
Posted Aug 04, 2016 - 14:00 EDT
Investigating
We're currently experiencing degraded performance. We are investigating the issue and will report when full service is restored.
Posted Aug 04, 2016 - 13:41 EDT