Oxygen vs Vercel vs Netlify: A/B Testing Commerce Performance
Introduction
Merchants migrating to Hydrogen often ask: Why should we host on Shopify Oxygen instead of Vercel or Netlify?
Synthetic benchmarks help, but the most persuasive evidence comes from competitor A/B testing — running identical storefronts across Oxygen, Vercel, and Netlify, then comparing performance region by region.
This post outlines how to run those tests and use them in client/stakeholder conversations.
Why Competitor A/B Testing?
- ⚖️ Fair comparisons → same code, different host.
- 🌍 Region insights → show APAC vs EU vs NA variance.
- 📊 Stakeholder clarity → visual data beats abstract promises.
- 🛒 Commerce context → measure cart mutations + checkout API latency, not just static TTFB.
Test Setup
Step 1. Deploy Identical Storefronts
- Deploy Hydrogen build to Oxygen, Vercel, and Netlify.
- Use same Storefront API + metafields.
Step 2. Define Metrics
- Median TTFB.
- p95/p99 latency (realistic user worst-case).
- Cold vs warm starts.
- Shopify API proximity (cart mutations, checkout calls).
Step 3. Collect Data
- Synthetic: k6, Lighthouse CI.
- Real-world: RUM beacons (segment by region/ISP).
Sample Benchmark Table
| Region | Oxygen TTFB |
Vercel TTFB |
Netlify TTFB |
Notes |
| NA (East) | ~80ms | ~70ms | ~90ms | Comparable, warm starts. |
| EU (Frankfurt) | ~120ms | ~110ms | ~130ms | All solid; Oxygen closer to Shopify APIs. |
| APAC (Sydney) | ~200ms | ~180ms | ~220ms | Prefetch matters most. |
👉 Data shows Oxygen slightly trails Vercel in cold starts, but wins in Shopify API latency for commerce flows.
Using Results with Stakeholders
- ✅ Present regional charts in pitch decks.
- ✅ Show Shopify API proximity advantage (cart, checkout faster on Oxygen).
- ✅ Frame Oxygen as the best long-term bet for Shopify merchants.
- ✅ Use Vercel/Netlify as “control arms” to validate choices.
Case Example: Footwear Brand Pitch
- Agency ran same PDP on all 3 hosts.
- Oxygen → slightly slower cold start, but cart mutations 20% faster.
- Vercel → great cold starts, but API latency higher.
- Client chose Oxygen, convinced by commerce-first advantage.
Guardrails
- ✅ Always use same storefront code for fair test.
- ✅ Test both synthetic + RUM.
- ✅ Focus on commerce flows, not just static TTFB.
- ✅ Update benchmarks yearly — hosts evolve fast.
Conclusion
Oxygen doesn’t always “win” in raw latency, but it consistently outperforms when measuring commerce-specific performance. Competitor A/B testing turns abstract arguments into clear evidence for merchants.
Don’t just tell clients Oxygen is best — show them the data.