Incident report 02-07-2022

Tuesday, February 7, 2023

Situation: Awaiting RCA 

Description of the incident: 

The uContact solution had a complete outage on February 6, 2023, for around 50 minutes. Once logged into the system, the incident may have given you issues with the platform’s telephony capabilities. 

At 5:30 p.m. (GMT-3) that day, the issue had been completely addressed.

Incident review:

The primary symptom of the failure was an impact on the Google Cloud Platform's (GCP) firewall and router. The source of the issue was at the Google level, and a case has been escalated to the Google Cloud Platform (GCP) support team in order to obtain a diagnosis and remedy for future occurrences. As soon as we have new information to share, we will provide an official statement. 

Solution: 

To reboot the system, the uContact team followed these steps: 

1. Restart all the instances. 

2. Verify the impacted firewall rules. 

It should be noted that the information used to generate the firewall rules was taken from our backups of the GCP setups. The time spent on analyzing and manually setting up the routes makes up the total amount of time required to implement the solution. 

For your peace of mind, all settings, data, and recordings have been adequately backed up so that, even if a service disruption occurs in the future, it does not imply a total loss.  

We sincerely apologize for the inconvenience and thank you for your understanding.

Open