Root cause analysis of a recurring widespread system failure after weeks of troubleshooting the network, firewall, and client had yielded no results.
Symptom
Every Monday, an energy provider experienced major problems with its local Outlook mailboxes. More than 1,000 employees were affected, some for hours at a time.
During a conference call with over 60 participants, the firewall, network, and client configuration were discussed as possible causes. There was no definitive conclusion. The issue was reproducible, but no one could explain it.
Analysis
Ceterion was commissioned to conduct a standalone analysis and began by examining the log files of the affected end devices, rather than the network components discussed earlier. The decisive factor was the temporal pattern: a disruption that occurs exclusively on Mondays has its root cause over the weekend.
The investigation pointed to the locally installed antivirus software. Over the weekend, it scanned all mailbox files and reset the modification flag for every message it checked. Outlook interpreted this as a modification and, upon its first launch on Monday, triggered a rescan of all affected items. With mailbox files sometimes exceeding two gigabytes and more than 1,000 workstations logging in simultaneously on Monday morning, this led to system overload.
The analysis took 14 days. The reason lies in the pattern of the error itself: Since the effect occurred only on Mondays, each hypothesis could be tested only once a week.
In the joint support case, Ceterion provided Microsoft with guidance on where to look. The findings were subsequently confirmed.
Outcome
- more than 1,000 jobs saved
- 14 days from the time the assignment is placed to the identification of the definitive cause
- More than 60 people were involved in the previous investigation into the cause, which yielded no results
After adjusting the antivirus configuration, the problem did not recur. The changes to the network and firewall that had previously been considered were no longer necessary.