Why we rebuilt our busiest webhook endpoint in Rust

In January 2026, a provider sent roughly three times its normal volume to our webhook servers. All 13 Python pods hit 100% CPU. Some stopped answering health checks and were replaced, leaving about five pods to work through the queue. Failed requests came back through the retry system, adding roughly 2.5 times more load. Webhooks for all of our customers slowed down. We scaled from 13 pods to 20, then 30.

After this first incident, a few of us started talking seriously about pulling the webhook service out of the Python megalith and rebuilding it in something faster. Coding agents made the idea feel much more practical. I remember playing with Opus 4.5 in Claude Code over Christmas break and thinking, “My life will never be the same.”

Six weeks later, while still considering what to do with the webhook service, it happened again. A large retailer released about 21,000 orders in 60 seconds, compared with a normal peak of about 4,000. At the same time, status updates that Nash sent through the same webhook fleet spiked. Combined traffic reached roughly 1,000 requests per second. All 30 pods hit 100% CPU. Average response time went from 0.13 seconds to 56.8 seconds, and the load balancer returned roughly 51,000 server errors in seven minutes. Fun! The retailer reported more than 600 delivery requests timing out; many end customers would get the dreaded “your delivery has been delayed” emails.

The next message I wrote in Slack was that we needed a separate, isolated service. We had a customer problem and a shared infrastructure problem. Adding more Python pods would buy time, but it would leave the same failure mode in place and keep raising our AWS bill. I had built systems in Rust before, so I got to work doing what I now do best: engineering prompting. We ran the benchmark the next day.

Why the Python pool saturated

When a delivery status changed, Nash sent an internal webhook through Svix back to our own webhook servers. The handler converted it to the retailer's format, then made a synchronous HTTP call. A failed send was retried.

The status endpoint was Flask code running in Gunicorn. A downstream call normally took 100 to 900 milliseconds. During that time, the thread handling it did nothing else. If the downstream service stopped responding, that thread could wait 30 seconds before the call timed out.

Those status updates shared workers with inbound requests from the retailer, including requests to create new deliveries. Once status traffic filled the pool, create requests waited at the load balancer. A problem on one path could take capacity from every webhook endpoint on those servers.

We doubled the autoscaling ceiling from 30 pods to 60 after the February incident. That was the right immediate fix. It did not keep status traffic from starving new delivery requests the next time volume jumped.

We pulled one endpoint into its own service

The new service used Axum and Tokio. The endpoint spends most of its time waiting on network calls, so an async server could keep many requests in flight without tying each one to a worker.

Rust was not the only way to build an isolated async service. It was the option we knew how to own. Separating this traffic from the shared worker pool mattered more than the language by itself.

Rust used the same Postgres tables and Redis keys as Python. We moved traffic at the load balancer. If the new service broke, we could send requests back to Python without moving data. We also parsed signing keys once at startup, used Rust enums for every status, and checked SQL queries against a saved schema during the build.

The code still had to match Python

Coding agents wrote most of the first pass. That part was quick. The real work was proving that the new service behaved like the Python code already running in production.

The first pass looked right. It could still emit the wrong status string or treat a nullable database column as required.

We made unmapped status variants fail the build. SQLx checked queries against the saved database schema. A dry-run path sent production-shaped requests through Rust without writing data or calling the retailer, then compared the result with Python.

The compiler covered missing cases and bad queries. For behavior the type system could not see, we gave the agents a way to send traffic through both systems and compare the outputs. That gave us confidence the Rust implementation matched Python.

Rust kept scaling after Python stopped

In February, one day after the second incident, we simulated traffic to both services at 10, 50, 100, 200, and 500 concurrent users. The test endpoints simulated the 100 to 900 milliseconds we normally waited on each outbound call. At 10 users, the services looked alike. At 50, Python began to fall behind. Rust kept scaling.

At 500 users, Rust served 439 requests per second and Python served 28.4. Python's p99 reached the test's 30-second timeout; Rust's was 1.3 seconds. Peak memory was 5,068 MB for Python and 242 MB for Rust. These were load-test results.

Production gave us the result we cared about. During the first week, three 1-vCPU, 2 GB Rust pods carried almost all traffic on the endpoint. They routinely handled more than 200 requests per second and bursts above 500. The same traffic had taken 10 to 15 larger Python pods. Each Python pod also had roughly two to four times the CPU and eight times the memory.

We moved 90% of the load off the shared Python servers. A flood of status updates no longer had to consume the same workers that accepted new deliveries. That isolation mattered more than the benchmark by itself.

We moved traffic in stages

About 40 million of the endpoint's 44 million weekly requests were courier location updates, so we started there. We began with dry-run comparisons, moved location updates, watched production for several days, then routed the remaining event types.

A 30-day snapshot from June showed 107 requests per second on average, 150 to 265 during business hours, and bursts above 400. On the same endpoint, p99 latency fell from 799 milliseconds on Python to 524 on Rust. Those numbers describe that 30-day window, not a live dashboard.

What we fixed after launch

The Rust port was not all hunky-dory. Just over two weeks after launch, the Rust service stopped forwarding updates. About 150 orders were affected. Our monitoring missed it, and we found it through support.

We could not prove one clean root cause. Slow connection failures could pile up because the service had no concurrency limit and used a 15-second outbound timeout. A deleted container image also kept the platform from replacing the stuck pod, which made the outage last longer.

We added a circuit breaker that opens after five upstream failures and probes the downstream service again after 30 seconds. We changed our image retention policy and added a separate health server, though the infrastructure probes still needed to be wired to it when we wrote this.

The outage changed what we fixed next. It did not recreate the failure mode that started this project: status traffic was still isolated from the shared Python fleet.

Every new service needs a human owner (for now)

We talk about this tradeoff a lot at Nash. Pulling a module into its own service can isolate load and failures, but it also creates another thing someone has to own in production. We have tried this with other modules and decided the ownership cost was too high.

Writing the replacement is getting cheaper, especially with coding agents. The limiting factor for us is still people. Someone has to understand the service after it ships and take the page when it fails. This was our first Rust service in production. We still had to define what matching Python meant, build the rollback path, run the load test, and work out how the new service could fail.

This one was a no-brainer. It moved roughly 90% of the load off the shared Python fleet and reduced the compute footprint. Today it takes maybe 30 minutes a month to maintain, and it has far more headroom than the traffic we expect.

I would make the same migration again. I would not use Rust for every service at Nash, but this endpoint spends most of its time waiting on network calls under bursty, high-volume traffic.

The endpoint has been quiet for months. For a webhook forwarder, that is about as good as it gets.

I hope I didn't just jinx it.