API and rendering infrastructure issues

Incident Report for Percy

Resolved

Our job processing queue is fully caught up and the backlog has been cleared. Thank you again for your patience.
Posted Oct 21, 2019 - 19:48 PDT

Update

We are increasing capacity to get through the processing backlog. Thank you for your patience.
Posted Oct 21, 2019 - 17:50 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

We have determined that a configuration change we made caused problems connecting to our redis infrastructure. We were able to quickly identify the problem and have reverted the change. We apologies for the inconvenience and will work on new procedures to improve our processes.
Posted Oct 21, 2019 - 16:27 PDT

Investigating

We are currently investigating this issue.
Posted Oct 21, 2019 - 16:11 PDT
This incident affected: API and Rendering infrastructure.