Impact
At least three users experienced problems when trying to generate reports on the platform. The issue started on UTC-5 25-12-18 16:47 and was reactively discovered 15.3 hours (TTD) later by a client who reported through our help desk [1] that reports could not be generated correctly. During the investigation, we found that several background processes related to actions such as disabling roots, updating attack surfaces, and applying organization policies were not being processed as expected. The problem was resolved in 2.8 hours (TTF), resulting in a total window of exposure of 18.2 hours (WOE) [2].
Cause
The problem was caused by a recent change where some code was removed, but other parts of the system were still trying to use it. Additionally, the component responsible for handling background tasks was being checked with very basic validation rules, which were not strong enough to detect this type of issue before it reached production [3].
Solution
All remaining references to the removed code were cleaned up, allowing the affected background processes to run correctly again and restoring normal report generation [4].
Conclusion
As a follow-up action, we will prioritize strengthening the validation rules used for this part of the system so that similar issues can be detected earlier and prevented in the future. INCOMPLETE_PERSPECTIVE < MISSING_TEST