- Written by
gronxb
- Published
Revisiting the Codemagic Stress Test with Hot Updater v1.0
We reran the cached update-check path that Codemagic's stress test highlighted, and wrote down where Hot Updater and Codemagic Patch each fit.
Codemagic published a stress-test video comparing an earlier Hot Updater deployment with Codemagic Patch. Hot Updater slowed down at around 250 requests per second, while Patch handled 1,000. Codemagic traced the gap to Hot Updater querying a database on each check, where Patch serves a pre-generated manifest through a CDN.
That was a fair point. In Hot Updater v1.0, Release Catalogs serve shared update metadata that a shared cache can answer, so a cache hit doesn't touch the database.
We wanted to see how the two cached paths compare today, so we measured them at the same 1,000 requests per second.
Measuring the cached read path
We maintain Hot Updater, so take this as our own measurement. We ran Hot Updater v1.0 on Cloudflare, with real D1 and the preset's client API-key policy.
We didn't run a Patch deployment. Instead, we built a control: a synthetic manifest made with Patch's own serializer, served from Cloudflare Workers Static Assets. It stands in for Patch's CDN delivery, not for Patch's storage, publishing, or SDK. Both paths shared a hostname and connection pool to keep transport differences small.
We scheduled 1,000 requests per second for each path in three 12-second runs. Each path answered all 36,000 of its requests with HTTP 200 and the expected Bundle ID or package hash.
| Metric at 1,000 requests/s | Hot Updater v1.0 on Cloudflare | Static manifest (Patch format, our hosting) |
|---|---|---|
| HTTP 200 + expected identifier | 36,000 / 36,000 | 36,000 / 36,000 |
| HTTP, timeout, or payload failures | 0 | 0 |
| p50, median of the three runs | 51.6 ms | 50.9 ms |
| p95, all three runs pooled | 83.4 ms | 79.1 ms |
| p99, all three runs pooled | 184.2 ms | 95.4 ms |
| Served from cache | 36,000 (100%) | 36,000 (100%) |
The right column is a manifest in Patch's format, with Patch's own cache headers, that we hosted ourselves. It is not a Patch deployment.
All 81,000 measured requests passed, including the 250 and 500 requests/s stages, and every response on both paths came from cache. Median and p95 latency were within a few milliseconds of each other.
The p99 differs. Hot Updater keeps a Catalog fresh for five seconds, so the edge refreshes it from the Worker every five seconds, while the static manifest stayed cached for the whole run. At 250 and 500 requests/s that refresh did not show up in the tail: Hot Updater's p99 was 82 and 81 ms, against 86 and 87 ms for the static control. At 1,000 requests/s, one edge location (Hong Kong) still saw short bursts of slower responses, mostly around a refresh, which put Hot Updater's pooled p99 at 184 ms. The Tokyo edge, which served the other sixth of the traffic, had a 91 ms p99, against 119 ms for the static control there.
The five-second window is a deliberate trade. A new release or a rollback reaches devices within about 15 seconds without any CDN purge integration. Patch's manifest is cached for up to 300 seconds and relies on a CDN purge when you publish to show a new release sooner.
Both paths served one warmed URL, which is the best case for any cache. Real traffic spreads across more apps, versions, and keys, and these short runs don't measure maximum capacity or either product end to end. Within that scope, the cached check no longer shows the gap from the video. The bigger difference is how each one fits the platform you already use.
Use the database your platform already has
Hot Updater is built to fit the provider you're on. Its provider presets use each platform's own database, storage, and functions, and the setup tools deploy them, so there's no server or VM to run. A custom server mounts the same handlers in your existing backend.
Patch takes an integrated approach: its API and release worker run as a Node.js service with PostgreSQL. The default installation runs that stack with Docker Compose on one host, together with Caddy and MinIO. Bundles can move to S3-compatible storage or Google Cloud Storage, and the database can be an external PostgreSQL. Either way, the service runs on a host you provide.
Here is what that means on each platform:
| You're on | Hot Updater uses | Running Patch there |
|---|---|---|
| Firebase | Firestore, Storage, Cloud Functions | PostgreSQL and a host for the service; bundles can go to Cloud Storage |
| Cloudflare | D1, R2, a Worker with a shared cache | PostgreSQL and a host for the service; bundles can go to R2 |
| AWS | DynamoDB, S3, Lambda@Edge behind CloudFront | PostgreSQL and a host for the service; bundles can go to S3 |
| Supabase | PostgreSQL, Storage, an Edge Function | A host for its API and worker, with Supabase's PostgreSQL as its external database |
| Your own backend | Handlers in Hono, Express, or Elysia, with Kysely, Drizzle, Prisma, or MongoDB | Its own service alongside your backend, using PostgreSQL (yours or a new one) |
For a Firebase team, Hot Updater keeps OTA metadata in Firestore next to the rest of the app's data and answers devices from Cloud Functions. For an Express and Prisma team, its schema goes through the existing migrations and its admin routes sit behind the existing authentication. A team on MongoDB or SQLite keeps using it. Storage and database are separate adapters, and a provider we don't support yet can implement the database contract.
We measured the Cloudflare preset above. The AWS preset caches Catalogs in CloudFront, while Supabase and Firebase run the function on every check.
Patch's stack has its own advantages: one known setup to install, published manifests and bundles that Codemagic has shown staying available with the API offline, and commercial support and optional hosting from Codemagic.
What comes with the server
Beyond delivering updates, Hot Updater v1.0's server runs plugins, and the managed presets include the official ones:
- Insights shows downloads, launches, crashes, and failed updates for each bundle, with rollback from the Console.
- Remote Config changes app values without a new bundle.
- Client API keys protect the app's routes.
Bundle diffing reuses unchanged files on the device and sends a binary patch for a changed Hermes bundle, and bundle signing works with a local key, AWS KMS, Google Cloud KMS, or a remote signer. The Console can be deployed for your team on Cloudflare, Docker, Netlify, or Vercel.
Licenses
Most Hot Updater packages use the
MIT license.
@hot-updater/server and @hot-updater/console use
MIT with one condition:
if you offer them to others as a hosted update service, you show "Powered by
hot-updater" with a link. Running them for your own apps, or for apps you
build or operate for clients, needs nothing extra.
Patch's SDK and CLI use Apache 2.0. Its server uses the Codemagic Server License, which is free up to 1,000,000 monthly active users (unique devices that request an update) and needs a commercial license above that. It doesn't allow offering a competing service, and each version becomes Apache 2.0 two years after it's published.
Choosing
Hot Updater fits when you want OTA to use the database, storage, and functions of the platform you're already on, whether that's Cloudflare, AWS, Supabase, Firebase, or your own backend, with Insights, Remote Config, bundle diffing, and signing on the same server. Patch fits a team that wants one self-contained stack on its own host, with commercial support from Codemagic.
Thanks to the Codemagic team for the original video. It pushed us to measure the read path more carefully than we would have on our own.
Choose a provider preset to set up OTA in your cloud account, or follow the custom-server guide to add it to your backend.
Measurement record
Download the summary,
request-level results (gzip),
and test harness with reproduction instructions.
A same-deployment control without CDN-Cache-Control
is summarized under control in the summary. We removed deployment
identifiers and credentials from the public files.
They keep the request timings, statuses, cache observations, and source hashes.
Read path and comparison limits
Why v1.0 behaves differently
Release Catalogs are compiled when release settings change. Devices fetch a shared Catalog and evaluate their own state locally. The managed Cloudflare template enables Workers Cache. A Catalog response uses:
Cache-Control: public, max-age=0, s-maxage=5
CDN-Cache-Control: public, max-age=5, stale-while-revalidate=5, stale-if-error=0
Vary: Accept-Encoding, x-api-keyOn a Workers Cache hit,
Cloudflare returns the response without running the Worker or querying D1.
Workers Cache reads CDN-Cache-Control first: when a Catalog expires, it
keeps answering from cache (CF-Cache-Status: UPDATING) while the Worker
revalidates it in the background. A revalidation reads the client API key
and the Catalog from D1, and the server keeps its own five-second Catalog
cache and merges concurrent loads of the same Catalog.
Scope
We tested one warmed URL per path, one API key, and one synthetic release
for a short period, which gives both paths plenty of cache reuse. Hot
Updater had a five-second freshness window. The static manifest used the
300-second window from Patch's own manifest cache policy
(CDN_MANIFEST_CACHE_CONTROL). The 12-second stages crossed Hot Updater's TTL but
never reached the manifest's expiry. With a CDN adapter configured, Patch
purges manifests when publishing.
We did not measure how long either system takes to get a published update
to a device.
The test also left out traffic across many apps or versions, cache purges after publishing, rollout changes, origin failures, Insights reporting, artifact resolution, downloads, and patch generation. A complete Hot Updater update may need a separate artifact request, and the two response bodies carry different information. These results don't show SDK equivalence, hosting costs, or outage behavior.
v1.0 changed the read path after the version shown in the video, so this test can't explain the earlier slowdown or measure a speedup between versions. It does show that v1.0 on Cloudflare serves every Catalog request from cache, without database work on each request.
Environment, fixtures, and methodology
Code and deployment
| Setting | Recorded value |
|---|---|
| Measurement interval | October 9, 2026, 07:01:25.390–07:03:11.862 UTC, including warmup and drain |
| Hot Updater packages | 1.0.0-rc.43; protocol, plugin-core, the official plugins, and server rebuilt from the checkout |
| Hot Updater commit | becff8b37d6f95b5edb85d5159ea16c114ae32d0 |
| Codemagic commit | 3d4ad8b7a7d4ebedbe7cd0513742c1418625221c |
| Load generator | One Node.js v24.15.0 process on a Mac mini in South Korea; Apple M4, 10 CPU cores, 24 GiB RAM; Darwin 27.0.0 arm64 |
| Tooling | pnpm 11.6.0, Wrangler 4.99.0 |
| Hosting | One temporary Cloudflare Worker with D1 and Static Assets in the same account, using one workers.dev hostname |
| D1 placement | APAC location hint; initial import reported SIN |
| Worker compatibility | 2026-08-13, nodejs_compat, the same as the managed template |
| Worker configuration | cache.enabled=true; observability enabled with head_sampling_rate=0.01; no explicit CPU override |
| Account plan and infrastructure size | Account billing tier and managed Worker hardware were not recorded; no dedicated VM was used |
| Edge distribution, each target | HKG 83.33%, NRT 16.67%, from CF-Ray suffixes |
The Hot Updater handler used the actual createHotUpdater, D1 adapter,
apiKeys() plugin, and Hono request path. We added a D1 wrapper and diagnostic response headers to
observe each request, leaving authentication, Catalog selection, and cache
logic unchanged. We omitted storage because the Catalog GET doesn't use it.
We applied the repository's D1 migration and inserted one channel, one
compiled Catalog, and one client API-key record. We compiled the Catalog from one
enabled iOS Release with 100% rollout, no custom cohorts, and
targetAppVersion: "*". We left Bundle and Release rows empty because the
fixture only needed to read persisted metadata. Publishing and artifact
resolution were outside the test.
| Request or fixture | Hot Updater v1.0 on Cloudflare | Static manifest (Patch format, our hosting) |
|---|---|---|
| Request path | /release-catalogs/app-version/ios/cHJvZHVjdGlvbg/1.0.0 | /patch/production/1.0.0/manifest.json |
| App target | iOS, production, app version 1.0.0 | Synthetic manifest for the same version and channel |
| Client authentication | One registered x-api-key shared by all requests | Public static object |
| Stored document | 426-byte compiled Catalog; 724-byte HTTP body including forward and rollback candidates | 289-byte JSON from Codemagic's manifestSerializer.ts |
| Shared cache TTL | 5 seconds | 300 seconds |
| Cache headers | Cache-Control: public, max-age=0, s-maxage=5; CDN-Cache-Control: public, max-age=5, stale-while-revalidate=5, stale-if-error=0 | Cache-Control: public, max-age=0, s-maxage=300, must-revalidate |
Cloudflare served the static asset before the Worker handler. The test ran without a Codemagic API, PostgreSQL, release worker, R2 bucket, or VM. The manifest's bundle URL was a placeholder; we never downloaded it. After measurement, we removed the temporary Worker, D1 database, and local client-key file.
Request scheduling and statistics
- Both paths shared twelve persistent HTTP/2 sessions. Each stage started with twelve sequential warmup requests, which we excluded from the results.
- Each target received 250 requests/s for six seconds, 500 requests/s for six seconds, then three runs of 1,000 requests/s for twelve seconds each: 40,500 measured requests per target. Stages ran sequentially. The order alternated between the two targets, starting with Hot Updater.
- A clock-based scheduler sent due requests approximately every two milliseconds, without waiting for earlier responses. After scheduling ended, it waited for pending responses. The harness used a ten-second HTTP/2 stream inactivity timeout, rather than an absolute request deadline.
- Requests sent
Accept-Encoding: identity, with no query-string cache busters or clientIf-None-Matchheaders. We kept existing cache entries. The load generator made HTTP requests without running a React Native SDK. - We measured latency from just before opening the HTTP/2 stream until the full response body arrived, before JSON validation. That includes network transit and transport queuing, but excludes dispatch lag. The raw data also records dispatch lag and scheduled-to-completion latency.
- Percentiles use nearest rank. We calculated the headline p95 from all 36,000 request timings per target, rather than averaging the three p95 values.
- At 1,000 requests/s, the largest per-stage p99 dispatch lag was 6.01 ms. The final response arrived 70–97 ms after the nominal twelve-second window. Including that drain, completed throughput was 992–994 requests/s.
- We counted a sample as successful if it returned HTTP 200, parseable JSON, and the expected first Catalog Bundle ID or manifest target package hash. That check doesn't validate full SDK selection, rollback, or installation.
- Hot Updater answered 35,512 responses as
HITand 488 asUPDATING(served from cache while the Worker revalidated), with noREVALIDATED,EXPIRED, orMISS. The static control answered all 36,000 asHIT. Cached responses can repeat an origin run's diagnostic headers, so we don't sum their D1 counters into totals. - 510 Hot Updater responses took longer than 150 ms, all at HKG, against 29 for the static control.
Both targets had the same POP proportions, though network conditions could still vary between stages. We didn't calculate confidence intervals or run an equivalence test, so these observations don't establish identical latency distributions.
Every measured stage
All times are milliseconds. Rows follow execution order; warmup is excluded. "Static control" is the manifest in Patch's format that we hosted ourselves.
| Offered requests/s | Run | Target | Correct / sent | p50 | p95 | p99 | HIT + UPDATING |
|---|---|---|---|---|---|---|---|
| 250 | Ramp | Hot Updater v1.0 | 1,500 / 1,500 | 50.4 | 77.9 | 81.9 | 1,452 + 48 |
| 250 | Ramp | Static control | 1,500 / 1,500 | 51.3 | 79.3 | 86.3 | 1,500 |
| 500 | Ramp | Static control | 3,000 / 3,000 | 50.9 | 79.4 | 86.5 | 3,000 |
| 500 | Ramp | Hot Updater v1.0 | 3,000 / 3,000 | 51.0 | 76.8 | 80.7 | 2,906 + 94 |
| 1,000 | 1 | Hot Updater v1.0 | 12,000 / 12,000 | 51.6 | 81.4 | 180.3 | 11,763 + 237 |
| 1,000 | 1 | Static control | 12,000 / 12,000 | 52.2 | 79.7 | 93.8 | 12,000 |
| 1,000 | 2 | Static control | 12,000 / 12,000 | 50.6 | 78.9 | 93.2 | 12,000 |
| 1,000 | 2 | Hot Updater v1.0 | 12,000 / 12,000 | 50.8 | 80.3 | 133.3 | 11,893 + 107 |
| 1,000 | 3 | Hot Updater v1.0 | 12,000 / 12,000 | 51.7 | 92.0 | 221.5 | 11,856 + 144 |
| 1,000 | 3 | Static control | 12,000 / 12,000 | 50.9 | 78.5 | 98.2 | 12,000 |