Some users experienced an outage of the AI-powered services on our platform: the AI agent chat failed with "Failed to connect to chat service", sessions from external MCP clients could not start, and some pull request reviews from the PR Assistant did not finish processing. The issue started on UTC-5 26-10-26 11:35 and was reactively discovered 5.8 hours (TTD) later by a staff member who noticed that the AI agent had stopped responding. The problem was resolved in 1.2 hours (TTF), resulting in a total window of exposure of 7 hours (WOE).
The services inside our infrastructure locate each other through an internal name resolution service. A change that enabled a new network security layer on some groups of servers modified how name resolution works on those servers. On those servers the internal name resolution service ended up forwarding requests to itself, detected the loop, and stopped. Nothing restricted this service to the servers where it works correctly, and since servers are replaced every day, one of its two instances landed on an affected server on October 19 and stopped working. The service kept running on a single instance for a week without any alert. On October 26, an unrelated infrastructure change automatically updated this service to a newer version. The restart moved the last healthy instance onto an affected server, leaving no instances available, so every AI service that depends on internal name resolution stopped responding. The new version itself was not the cause; both versions behave the same way on the affected servers.
The service was restored by restricting the internal name resolution service to servers without the new network layer, after which both instances started correctly. The permanent fix keeps this service on a dedicated group of base servers, the same placement already used by other core components, and raises that group to a minimum of two servers. A complementary change fixes the versions of the base components of our infrastructure, so updates are deliberate and reviewed instead of a side effect of unrelated changes. The team will also add an alert on the availability of this service.
This incident showed that the change to the network configuration of our servers was validated for the services it targeted but not for the base components of our infrastructure that could run on those servers, so the defect stayed hidden until the daily server rotation exposed it. Without an alert on the availability of name resolution and with component versions not fixed, a routine infrastructure update turned a silent degradation into an outage. Placing base components on dedicated servers, fixing their versions and monitoring their availability would have prevented any user from being affected. INFRASTRUCTURE_ERROR < MISSING_ALERT < INCOMPLETE_PERSPECTIVE