Partial Outage of the Merchant UI and API
Incident Report for Subscribe Pro
Postmortem

Today our API and web UI were down for close to 15 minutes.

The root cause was a long running database update which was necessary as part of an upgrade. When we estimated the length of time the update would need we were off by 1 order of a magnitude. Consequently the update took close to 15 minutes instead of a little over a minute and translated into downtime for the API and UI.

These sort of database updates are something we do rarely. Never-the-less, we have identified methods we can use to do more extensive testing in the future, for similar updates. This would allow us to plan alternate, incremental ways to update the database and schedule those at off times.

Posted Sep 24, 2019 - 14:17 EDT

Resolved
This incident has been resolved.
Posted Sep 24, 2019 - 13:10 EDT
Monitoring
A fix is in place and we are monitoring.
Posted Sep 24, 2019 - 12:00 EDT
Investigating
We are investigating a partial outage of our Merchant UI and API
Posted Sep 24, 2019 - 11:51 EDT
This incident affected: Subscribe Pro Application (API | api.subscribepro.com, Management Portal | platform.subscribepro.com).