Infastructure Migration

Scheduled Maintenance Report for CxAlloy

Postmortem

Summary

We did not migrate the infrastructure during this maintenance period. Although we scheduled this maintenance with the expectation to perform the migration, we identified a blocking issue the day before.

We instead used the maintenance period to resolve that blocking issue. We successfully resolved the issue during the maintenance period and we will now schedule a future maintenance period to perform the migration.

Blocking Issue

The blocking issue was that one of our replicas between our current environment and the new environment got out of sync. This was due to a combination of an IP address change and an overly short retention period for the binary logs, which are what makes replication possible.

The IP address change created a connection error on the replica. By the time we identified and resolved it the necessary binary logs had been purged, leaving that replica permanently out of sync.

During the maintenance we stood up a new replica and ensured it was in sync. Separately, we also resolved the underlying issues that caused the out of sync issue by switching from IP-based addressing to DNS-based addressing and extending the retention period for binary logs in the new environment.

Posted Aug 26, 2024 - 13:59 EDT

Completed

The scheduled maintenance has been completed.
Posted Aug 24, 2024 - 23:39 EDT

In progress

Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted Aug 24, 2024 - 22:00 EDT

Scheduled

We will be migrating our infrastructure during this time. For details on last week's attempted migration please read that maintenance's postmortem.
Posted Aug 20, 2024 - 18:11 EDT
This scheduled maintenance affected: CxAlloy TQ (Application, Servers).