Multi-cloud performance · 2026
Fixing a cross-cloud slowdown with batched lookups
When our catalog datastore moved from GCP to AWS while the services reading it stayed on GCP, pricing-feed runs slowed 4–9×. Scaling the database would not have helped—the bottleneck was cross-cloud network round trips, not query execution.
Context
The commerce platform ingests price and offer feeds from affiliate networks and retailers and matches each offer to a product in a multi-locale catalog of millions of products. Feed ingestion runs on Google Cloud. As part of a wider migration, the team moved the catalog's document store from MongoDB on GKE to Amazon DocumentDB on AWS. Before the move, I screened six services for DocumentDB compatibility; none used unsupported operators.
The problem
After the move, feed runs that had taken two to three hours took nine to eleven. Some runs failed outright, and feeds that missed their windows went stale.
Diagnosis
I profiled a single record end to end instead of relying on service-wide dashboards. Nearly all the time per record went to network round trips between GCP and AWS, dwarfing the time spent inside the database. The matching code looked up one record at a time—harmless on the same network, but expensive across clouds where every lookup paid a full cross-cloud round trip.
That finding ruled out the obvious fixes. A bigger database instance, more indexes or more connections would have left the round-trip count unchanged.
The fix
- Batched lookups. I redesigned SKU matching to collect a chunk of records and resolve them in one query, so each round trip carries many lookups instead of one.
- Behind a feature flag. The batched path shipped dark so I could enable it per environment while keeping the legacy path available.
- Same-row A/B test. Before turning it on in production, I ran both paths on identical inputs in a test environment and compared the outputs. The matching step was 31–62× faster, and match counts were identical.
Results
On the first production cycle with the flag enabled, 12 of the 13 stalled feeds completed, and feed runs now finish about 70% faster than before the migration.
One trade-off remains: a single feed's match rate sits slightly below its earlier baseline, which we track separately.
My part: I ran the pre-migration compatibility screen, diagnosed the cross-cloud latency, redesigned the batched lookups behind a feature flag, ran the A/B verification, and led the production rollout. Teammates migrated the applications to DocumentDB.
What I carry forward
Treat network round trips as a requirement of any cross-cloud move. A compatibility check confirms that the queries run; a short load test from the calling cloud shows how often each one crosses the network. Together they give an early, measurable signal, and the profile-one-record and A/B method from this fix is a ready template for that test.