This incident has been resolved as of last week. We have improved internal systems and monitoring for this issue, and will continue to watch it closely.
Oct 6, 13:32 PDT
After root cause analysis and some infrastructure changes to address the issue, we have not seen any similar high latency issues in >12 hours. We will continue to monitor closely.
Sep 29, 18:35 PDT
We have identified an issue with redis latency, which is causing bursts of 503/504s from Percy's API.
We do not have a fix at this time but are investigating further and will post updates as we have them.
We appreciate your patience and apologize for the inconvenience. This ops issue has been a tricky issue to resolve and we are coordinating with our redis provider on a resolution. We will leave this ticket open until we are sure the problem has been fully resolved.
Sep 29, 09:32 PDT