Oxygen vs Vercel vs Netlify: A/B Testing Commerce Performance

Introduction

Merchants migrating to Hydrogen often ask: Why should we host on Shopify Oxygen instead of Vercel or Netlify?

Synthetic benchmarks help, but the most persuasive evidence comes from competitor A/B testing — running identical storefronts across Oxygen, Vercel, and Netlify, then comparing performance region by region.

This post outlines how to run those tests and use them in client/stakeholder conversations.

Why Competitor A/B Testing?

• ⚖️ Fair comparisons → same code, different host.
• 🌍 Region insights → show APAC vs EU vs NA variance.
• 📊 Stakeholder clarity → visual data beats abstract promises.
• 🛒 Commerce context → measure cart mutations + checkout API latency, not just static TTFB.

Test Setup

Step 1. Deploy Identical Storefronts

• Deploy Hydrogen build to Oxygen, Vercel, and Netlify.
• Use same Storefront API + metafields.

Step 2. Define Metrics

• Median TTFB.
• p95/p99 latency (realistic user worst-case).
• Cold vs warm starts.
• Shopify API proximity (cart mutations, checkout calls).

Step 3. Collect Data

• Synthetic: k6, Lighthouse CI.
• Real-world: RUM beacons (segment by region/ISP).

Sample Benchmark Table

Region	Oxygen TTFB	Vercel TTFB	Netlify TTFB	Notes
NA (East)	~80ms	~70ms	~90ms	Comparable, warm starts.
EU (Frankfurt)	~120ms	~110ms	~130ms	All solid; Oxygen closer to Shopify APIs.
APAC (Sydney)	~200ms	~180ms	~220ms	Prefetch matters most.

👉 Data shows Oxygen slightly trails Vercel in cold starts, but wins in Shopify API latency for commerce flows.

Using Results with Stakeholders

• ✅ Present regional charts in pitch decks.
• ✅ Show Shopify API proximity advantage (cart, checkout faster on Oxygen).
• ✅ Frame Oxygen as the best long-term bet for Shopify merchants.
• ✅ Use Vercel/Netlify as “control arms” to validate choices.

Case Example: Footwear Brand Pitch

• Agency ran same PDP on all 3 hosts.
• Oxygen → slightly slower cold start, but cart mutations 20% faster.
• Vercel → great cold starts, but API latency higher.
• Client chose Oxygen, convinced by commerce-first advantage.

Guardrails

• ✅ Always use same storefront code for fair test.
• ✅ Test both synthetic + RUM.
• ✅ Focus on commerce flows, not just static TTFB.
• ✅ Update benchmarks yearly — hosts evolve fast.

Conclusion

Oxygen doesn’t always “win” in raw latency, but it consistently outperforms when measuring commerce-specific performance. Competitor A/B testing turns abstract arguments into clear evidence for merchants.

Don’t just tell clients Oxygen is best — show them the data.