Clients unable to make new connections to Resources

Incident Report for Firezone Production

Postmortem

On 4/3/24 around 8:30a pacific, a new version of the Firezone Relays was deployed to production which contained a logic bug that caused them to fail to be used by Clients after a certain period of time.

This logic bug was not caught in automated testing or our staging environment because it only occurred in Relays with a certain access pattern.

We rolled back the afflicted Relays around 5pm pacific time which resolved the issue.

We’ve since added better load testing to catch issues like this in the future.

Posted Apr 04, 2024 - 15:27 UTC

Resolved

This incident has been resolved.

Posted Apr 04, 2024 - 02:48 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Apr 04, 2024 - 01:40 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Apr 04, 2024 - 01:39 UTC

Update

We are continuing to investigate the issue. No new information to report at this time.

Posted Apr 03, 2024 - 21:09 UTC

Investigating

We're investigating an incident with the latest release of the Firezone Gateway (1.0.0-pre.12). We advise all customers to not upgrade their Gateways at this time if they have not done so. We will give a status updates here as we discover more information.

Posted Apr 03, 2024 - 17:54 UTC

This incident affected: Control Plane API, Admin Portal, and Relay Servers.