Production deployed without matching published Gateways
Incident Report for Firezone Production
Postmortem

On 2/29/23 at 2:05pm, a production hotfix was deployed to fix an issue with IdP sync. This deployment contained backwards-incompatible changes to the Control Plane API. Typically, new Gateways are published whenever this is the case so that customers can upgrade their Gateways to avoid outages.

In this case, a bug in our CI pipeline logic prevented new Gateways from being published, causing an outage that prevented existing Gateways from being able to serve Resources.

This outage was resolved on 3/3/24 at 11:48pm by publishing new Gateways that are compatible with the version of the Control Plane API that was deployed.

All customers are advised to upgrade their Gateways ASAP using the instructions appropriate for their deployment method, which can found at https://www.firezone.dev/kb/administer/upgrading#upgrading-gateways

Posted Mar 03, 2024 - 20:20 UTC

Resolved
We've received reports of Gateway outages
Posted Mar 02, 2024 - 13:30 UTC