Summary

On June 1, 2024 PDT (June 2, UTC), Rye services experienced elevated latencies and error rates. This report details the impact, root cause analysis, and steps we are taking to prevent future occurrences.

Customer Impact

From 0:03 PM to 8:41 PM PDT on June 1 (0:03-3:41, June 2, UTC), developers submitting carts through the Rye GraphQL API received “504 Timeout” responses after a 1-minute delay. Despite these errors, orders continued processing in the background. Specific impacts included:

• Elevated latency for most cart creation and submission requests.
• Multiple customers assumed initial request failures, leading to re-creation of carts and duplicate orders.
• Affiliate commissions were not applied to orders.
  • Update: After further exploration we figured that affiliate commissions were applied correctly

Root Cause Analysis

1. Hypertune Outage:
   1. Hypertune, our feature-flagging platform, was completely down.
   2. The Hypertune client SDK retried requests up to 10 times, blocking execution and causing a 1-minute delay for each feature flag retrieval.
   3. After giving up, the SDK defaulted to incorrect business logic, such as failing to apply affiliate commissions.
2. Google Cloud Run Timeout:
   1. Google Cloud Run, our service platform, returned 504 Timeout responses while continuing to process requests in the background.
   2. Lack of clear documentation on handling specific API failures led developers to retry failed cart submissions, causing duplicate orders.

Action Items

To prevent future issues and improve incident management processes, Rye will undertake the following actions:

1. Improve Documentation:
   1. Enhance documentation on API failure modes, error handling, and safe cart submission retries. Specifically concerning around idempotency of ordering and recommended retry mechanisms for carts. This will likely require some changes in how retries are handled by our customers to ensure orders are not duplicated.
2. Enable Offline Mode in Hypertune SDK:
   1. Ensure a recent offline snapshot of feature flags is used if the Hypertune server is unavailable.
3. Create a Status Page:
   1. Implement a status page for faster, and clearer communication around outage metrics and impact.
4. Implement On-Call Rotation:
   1. Establish a sufficient 24/7 on-call rotation for incident management.
5. Deploy Better Incident Management Tools:
   1. Utilize tools such as Slack bots and automated Zoom call setups for incident response to allow developers to receive more granular and in-depth updates around incident response.
6. System Review and Improvements:
   1. Conduct a review of critical system parts and implement stability and reliability improvements.

We deeply regret the inconvenience caused and are committed to ensuring the reliability of our services through these actions.