Global outage on all the productions and sandboxes

Incident Report for Hivebrite Status page

Postmortem

Dear customer,

Incident Overview

Our platform experienced an unexpected incident that prevented you from accessing Hivebrite for several minutes. We sincerely apologize for the inconvenience and here are more details on the incident.

INCIDENT PROPERTIES
Started	Dec 23, 2024 08:26 pm UTC
Impact Ended	Dec 23, 2024 08:49 pm UTC
Impact Duration	23 minutes
Environment	prod

What Happened

Our traffic handler that manages redirection to our marketing website URLs was not configured properly. Because of that, in case of worldwide high traffic, it loses the connection with our server. We had identified the issue a few weeks ago and were planning a maintenance window in January to correct it. Everything was ready for the resolution. We thought there would be no issue to wait as much as this configuration existed for one year and a half and it had never led to any issue until now. However, the probabilities went against us.

Impact on You

There was no data loss and there is no long lasting impact after this incident.

What was done

Thanks to everything being already ready, the team could react and correct the behaviour straightaway. The infrastructure has been corrected during the incident. It will not reoccur.

We sincerely appreciate your patience and understanding during this time. We are committed to continually improving our services to deliver a more resilient and reliable experience.
If you have any lingering concerns or questions, please do not hesitate to reach out to your CSM or our support team.

Sincerely,

Hivebrite

Posted Dec 24, 2024 - 15:44 CET

Resolved

The incident has been resolved.
Please reach out to your CSM if you have any questions.

Posted Dec 24, 2024 - 11:30 CET

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Dec 23, 2024 - 21:49 CET

Update

We are continuing to work on a fix for this issue.

Posted Dec 23, 2024 - 21:46 CET

Identified

The issue has been identified and a fix is being implemented.

Posted Dec 23, 2024 - 21:45 CET

Investigating

Our platform is currently experiencing technical issues that are preventing users and admins from accessing the platform. There is no impact to existing customer data.
We apologize for the inconvenience and are working to resolve this as soon as possible.

Posted Dec 23, 2024 - 21:42 CET

This incident affected: European servers (European production servers, European sandbox servers), US servers (US production servers, US sandbox servers), APAC servers (Apac production servers), and Switzerland production servers.