Cloud & DevOps

Cross-Cloud MySQL Replication, and the Write-Ownership Rule That Keeps It Alive

A managed MySQL primary in one cloud, a self-managed replica in another, replicating over the public internet at zero seconds of lag. The replication was the easy part. Deciding who owns each table was not.

August 19, 2026 · 7 min read · Cartolinks Engineering

Photo by Brett Sayles on Pexels

Migrating a database between cloud providers has two well-known options. Dump and import, which is simple and involves downtime proportional to your data size. Or replicate and promote, which is near-zero downtime and considerably more interesting.

We're running the second one. A managed MySQL 8.0 primary in one provider streams binlogs to a self-managed MySQL replica in another, across the public internet, filtered to a specific set of schemas. Both replication threads running, zero seconds behind.

Setting that up is genuinely straightforward — GTID-based replication, a replication user, network access to the source binlogs, and a seed. The part nobody warns you about is what happens when the replica needs to do work of its own.

The setup

• Topology: managed MySQL primary → self-managed replica, cross-provider, over the public internet to the source's endpoint.
• Filtered replication. The replica only subscribes to the reporting schemas it actually needs, not the whole instance. This keeps the replica small and the replication stream narrow.
• Seeded with a parallel logical dump rather than a physical backup — which matters more than it sounds, for reasons that show up in the next section.
• Monitored with a MySQL exporter on a least-privilege account (PROCESS, REPLICATION CLIENT, and SELECT on performance_schema, capped at three connections).

The alerting is deliberately small. Replication stopped — either thread down for two minutes — pages immediately. Lag above 300 seconds sustained for five minutes pages. That's the whole replication SLO, and it's been enough.

- alert: MysqlReplicationStopped
  expr: mysql_slave_status_slave_io_running == 0
     or mysql_slave_status_slave_sql_running == 0
  for: 2m
  labels: { severity: critical }

- alert: MysqlReplicationLagHigh
  expr: mysql_slave_status_seconds_behind_master > 300
  for: 5m
  labels: { severity: critical }

One design detail in those rules is worth stealing: these metrics only exist on replica hosts. On a primary, the series simply aren't produced — so the rules cannot false-fire on the wrong box. Writing alerts that produce no series where they don't apply is much more robust than writing alerts you then have to suppress.

The problem: the replica has a day job

A read replica that only ever reads is easy. Ours isn't one. It also builds nightly aggregate tables — daily summaries per market — using scheduled MySQL events that run locally on the replica.

So this box is simultaneously receiving replicated rows from the primary *and* writing its own rows locally. Both into the same schemas. And here is the failure mode that creates:

If a table is written at the primary and written locally on the replica, a replicated row event eventually arrives for a primary key the replica already wrote itself. The applier hits a conflict it has no way to resolve, throws an error, and replication stops. Not degrades — stops, and stays stopped until a human intervenes.

The seeding method makes this sharper than it would otherwise be. Because the replica was seeded with a logical dump, the dump recreated the source's scheduled events *as local events on the replica*. They didn't have to be deliberately created there — they arrived with the data, enabled, ready to write.

The rule: every table gets exactly one owner

There's no clever technical fix here. The fix is a decision, made per table, and written down where the next person will find it — which for us means a comment block at the top of every mart-building SQL file:

-- !! WHERE TO RUN THIS !!
-- Run this at the SOURCE database, NOT on the replica.
-- The replica is an ACTIVE REPLICA of that source. Creating the
-- table/rows locally risks colliding with anything that later
-- replicates down and breaking replication.
-- Applied at the source, it replicates down on its own.

And for the tables that went the other way, the mirror-image warning:

-- REPLICA-OWNED. Runs on the replica, NOT at the source.
-- !! NEVER also run this at the source !! — source-side row events
-- for locally-written primary keys can break replication.

Both rules exist in the same estate, for different tables, and neither is more correct than the other. What matters is that the ownership is explicit, recorded next to the code that would violate it, and phrased as a warning rather than a description.

Long backfills will kill your appliers

The other rule we learned the hard way: a long backfill on a replica takes locks, and the replication appliers waiting behind those locks eventually give up and die. Wrap backfills so the SQL thread isn't fighting your migration:

STOP REPLICA SQL_THREAD;
-- run the long backfill
START REPLICA SQL_THREAD;

The IO thread keeps pulling binlogs into the relay log the whole time, so nothing is lost — the replica just applies them after you're done, and catches up.

What we haven't built, and why we say so

Being straight about the gaps is more useful than pretending the setup is finished. Right now this estate has no automatic failover. There's replication, and there's a documented manual promotion procedure, and that's it. Recovery from losing a primary means provisioning a box, restoring, recreating users, and repointing applications — with an undefined recovery time, because nobody has timed it.

The backup story has a similar honest edge. Nightly logical dumps mean a recovery point objective of roughly 24 hours. What makes that mildly infuriating is that seven days of binlogs sit on each box, unarchived — the raw material for near-zero data loss is right there, and it's useless the moment the box itself dies, because nothing ships it anywhere.

Both are known, both are written down as ranked gaps rather than discovered later by an incident. If you take one thing from this: a documented gap is a completely different object from an unknown one. The first is a prioritisation decision. The second is an outage you haven't had yet.

If you're doing this yourself

• Decide write ownership per table, before the replica does any local work, and record it next to the code — not in a wiki nobody opens.
• Remember that a logical dump seeds your scheduled events too. Check what's enabled on the replica immediately after seeding.
• Alert on both replication threads and on lag, and write the rules so they produce no series on hosts where they don't apply.
• Stop the SQL thread around long backfills.
• Replication is not a backup. It faithfully replicates your DROP TABLE in under a second.