Google recently experienced a significant service disruption, causing inconvenience for users of Google Workspace. In this incident report, we provide an overview of the disruption, its impact on users, and the steps taken to resolve the issue. Please note that the information presented here is based on our current understanding and may be subject to change as our investigation progresses.
The incident began on June 2, 2023, at 12:25 (US/Pacific) and lasted for approximately 16 hours and 32 minutes. Services were fully restored on June 3, 2023, at 04:57.
The disruption primarily affected the Google Workspace Admin Console, causing issues with performing List, Create, Update, or Delete operations for users and organizational units. These issues were experienced by some Google Workspace Administrators globally.
Preliminary analysis indicates that the incident was caused by a configuration change that hindered the synchronization of resources with a backend system responsible for authorizing requests. As a result, attempts to modify resources, including new or existing ones, were met with 403 Forbidden errors. A comprehensive investigation is underway to determine the exact root cause and prevent similar incidents in the future.
Google Workspace Administrators encountered difficulties when performing various operations related to users and organizational units via the Admin Console or Directory APIs. Notably, the issue only affected resources created after June 2, 2023, at 12:25. Users experienced 403 Forbidden errors during List, Create, Update, or Delete operations.
To mitigate further impact, engineers promptly rolled back the configuration change on June 3, 2023, at 00:45. Subsequently, at 04:57, engineers successfully restored full functionality by backfilling data to the backend system, resolving the issue for all resources.
Google is committed to conducting a thorough Incident Review (IR) in the coming days to identify the exact cause of the disruption. The IR will provide a detailed analysis of the incident, helping to implement necessary measures to prevent similar incidents and ensure a more reliable experience for Google Workspace users.