Cloudflare WARP connection instability
Incident Report for Fluid Attacks
Postmortem

Impact

At least one organization experienced issues with repository cloning within their groups due to ZTNA and egress IP problems caused by DNS resolution issues with Cloudflare Warp. The issue started on UTC-5 24-08-20 10:50 and was reactively discovered 2.9 days (TTD) later by a client who reported the problem to one of our engagement managers [1]. This led to constant notifications being sent to the client, mistakenly indicating it was their issue, when in fact, it was related to Warp. The problem was resolved in 14.2 days (TTF) resulting in a total impact of 17.2 days (TTR) [2].

Cause

Warp’s readiness to navigate the client’s network cannot be determined consistently, leading to unreliable behavior where cloning sometimes succeeds and sometimes fails. Log reviews revealed that most errors stemmed from Warp’s inability to resolve DNS.

Solution

A specific check was implemented to ensure that DNS works correctly before starting any cloning processes. While this doesn't fully resolve the underlying issue of not knowing if Warp is operating properly before initiating a clone, it allows us to confirm that DNS resolves correctly before proceeding [3].

Conclusion

The core issue with Warp persists and will continue as long as it's in use. We need to keep identifying errors and adding targeted checks. Over time, this approach should lead to greater stability, ensuring all necessary validations are done before starting any cloning processes. THIRD_PARTY_ERROR < LACK_OF_TRACEABILITY

Posted Sep 11, 2024 - 15:46 GMT-05:00

Resolved
The incident has been resolved, and Git Roots are now cloning correctly.
Posted Sep 09, 2024 - 11:17 GMT-05:00
Identified
Instability in the Cloudflare WARP connection has led to errors in cloning certain Git Roots.

Workaround: It's recommended to close the related events from the Events tab and synchronize the repository from the Scope section.
Posted Aug 28, 2024 - 14:31 GMT-05:00
This incident affected: Dependencies (Cloudflare WARP).