Search platform · 2021–2026
Re-platforming product search from Solr 7 to Solr 9
Over five years on CNET's Solr search, I went from shipping schema changes on VM clusters to writing the replacement design and leading the re-platform of product search from a 2021 Solr 7 stack to Solr 9 on Kubernetes (AWS EKS). The hardest problems weren't in the cutover—they were silent failures that reported success.
Context
Solr serves exact product lookups for the editorial sites and older consumers, while Vertex AI Search handles faceted search. The Solr side had grown to nine cores on two legacy clusters. At the same time, the source data was moving to Amazon DocumentDB, so indexing had to move with it.
What I did
- Scoped and designed it. Asked to scope the upgrade, I partnered with a teammate to map what the nine legacy cores held and which services called them, built an OpenSearch proof of concept behind a search-backend flag to evaluate alternatives, and wrote and presented the design consolidating the nine cores into three.
- Moved indexing to DocumentDB. I moved the import jobs to read from the new source over TLS.
- Cut over production. I moved the production cores to Solr 9 and onto the new source with no planned downtime.
- Kept changes self-service. On the new cluster, schema, config and image changes ship as pull requests that GitOps applies, and I wrote runbooks for which changes are config-only and which need a new image. One trap went into them: Solr 9 caches a schema by configset name, so a schema change needs a new name, not a reload.
Testing the upgrade before production
Local testing had been the weak point in 2021, so this time I proved the upgrade in stages, each closer to production:
- A local version lab. Solr 7, 8, 9 and 10 ran side by side with the real configs and nonprod data. Solr 9 needed six config fixes and matched the old imports; Solr 10 was blocked, which ruled it out.
- A sandbox cluster. Before the dev clusters existed, I proved a core at production shape on a Kubernetes sandbox. Later, comparing its working configset with dev's isolated the cause of a broken deploy.
- Old and new side by side. Solr 7 and Solr 9 ran in parallel in nonprod, and I checked both against the source data, not just against each other. Solr 9 matched the source within a fraction of a percent. Solr 7 was missing more than a third of its live products, so "match the old system" would have been the wrong test.
- A gate suite and a one-command smoke test. In the team's Bruno API test repository (which I started), I added a gate suite covering schema parity, plugins, reads, the write path, and import jobs—verifying that the index actually changes after a write. Because the API hides a dead core behind an HTTP 200 with empty data, I also built a one-command smoke test that queries every search core and returns a single pass/fail verdict; during AWS testing it caught a dead core, which I rebuilt with the deploy tooling. Both suites now run as my—and my coding agents'—verify step before opening a pull request (how that works).
- Proof at cutover. Instead of trusting a document count, I wrote a fresh value and read it back through the consuming application. Rollback was switching the import schedule back.
The failures that hid their cause
Much of the risk lay in verification. Four failures either reported success or pointed at the wrong cause:
- An import that dropped a quarter of the data. A full rebuild reported success while about 24% of its documents had failed. Job status alone would have passed an index missing hundreds of thousands of products. A clean rerun completed with zero failures.
- A TLS trust gap that looked like a network outage. Connections to the managed database hung for about two minutes and then timed out, which looked like a firewall or routing problem. The cause was a missing certificate authority in the Solr image. Once the image trusted it, connections opened in about a second and a half.
- An import handler that built an empty index. The stock data-import library produced an empty product index against the new source. A patched build fixed it.
- A replication deadlock after cutover. A scheduled full import pushed one replica into a full-copy recovery on low disk and hit a known Solr bug. The recovery thread held the write lock, so every write to that shard failed for about four days while reads kept serving stale data and imports kept reporting success. Once a pod restart released the lock, I backfilled the index, replaced the replica whose disk had filled, parked the scheduled job that triggered it, and brought every replica back to full consistency.
Each of those failures became part of a 15-point health check that runs on demand. It checks that every replica is active and agrees on its document count, that each node has the disk for a full replica copy, that no thread is holding the write lock, that the import schedule is what it should be, and that recent changes in the source have reached the index.
Background: five years on Solr
My Solr work started in 2021, on the search cores behind the content API that served the live editorial sites: CNET, ZDNET, Metacritic and their sister brands. A schema change there landed in the read path of the live sites, not in a back-office tool. Those cores ran as SolrCloud on Google Cloud VMs that the infrastructure team built with Terraform: three nodes per cluster across zones, ZooKeeper on each node, and an internal load balancer in front. I operated and deployed to it. Schemas lived in git and shipped through a Jenkins pipeline that uploaded them to ZooKeeper, so my roughly 150 schema commits went out without infrastructure changes. The same year I handled the search side of relaunching ZDNET on the shared CMS: new index configuration, snapshot loads and load tests before go-live. Local Solr held only a few months of content, so I moved test loads into Kubernetes batch jobs that ran against the shared QA environment, and I tested the schemas on newer Solr releases. Product search ran on a separate Solr 7 stack until the re-platform above. The editorial cores didn't need to move: when the sites moved to WordPress, we retired the content API they served.
My part: In 2021 I shipped the schema commits to the live sites' search cores, did the search work for the ZDNET relaunch and built the Kubernetes test-load jobs; the infrastructure team built the VM clusters and the deploy pipeline. In 2026 I led the Solr re-platform: the scoping, the design, the staged upgrade testing, the core consolidation and the cutover. I moved the search imports onto DocumentDB, ran the recovery and built the health checks afterwards, and started the Bruno test repository. Teammates migrated the applications to DocumentDB, and the platform team built the Kubernetes (EKS) infrastructure underneath it.
What I carry forward
Answer "how will we know it's right?" before the cutover, and check against the source, not the old system. Every failure here was invisible in job status and plain in a direct check of the index. The 15 health checks and the smoke test from this project are re-runnable, so the next re-platform can start with verification in place from day one.