From 3.7 to 11.5 test builds a month, without hiring a developer
- Experiment velocity
- 3.1×
- from 3.7 to 11.5 builds a month
- Median build turnaround
- 2.1 days
- from 16 business days
- Builds bounced by client QA
- 2%
- from 31% — zero in the last five months
- Live rollbacks
- 0
- four in the year before us
The brief
What they came to us with
Ravenline sells CRO strategy to mid-market ecommerce brands — research, hypothesis roadmaps, the analysis that decides what is worth testing. What it does not sell is code. Every build went out to two freelance developers, and by 2025 that pipe had become the business bottleneck: the consultancy could diagnose faster than it could ship.
The numbers were blunt. Across the roster, 3.7 experiment builds went live in an average month. Median turnaround from approved brief to client-ready variant was 16 business days. Worse, 31% of builds came back from the end client’s QA with defects — each bounce another fortnight, another awkward status call. And once, a variant with no error handling took a client’s checkout down for 40 minutes on a Tuesday afternoon. Ravenline’s strategy was never the problem. Its delivery was undoing it.
The founder priced the obvious fix — in-house developers at £65k-plus each before management overhead, for a workload that spikes and dips with the roadmap — and chose the other route: retain Optyv as a white-label build arm, invisible to her clients, on a monthly capacity retainer. We build; Ravenline stays the only name anyone sees.
Constraints
- Total invisibility. Preview links unbranded, no Optyv fingerprints in code comments or commit history, all communication inside Ravenline’s own Slack and Jira.
- Three testing platforms across the roster — VWO, Optimizely and Convert — because each end client had already bought its own tool. One build standard, three targets.
- No access to end-client stakeholders. Every brief had to arrive complete: a Figma file, a hypothesis doc, and nothing else, with questions answered asynchronously by Ravenline.
- Per-client QA standards. One retailer demands WCAG AA on every variant; another supports a browser matrix going back to iOS 14. "Passes QA" means something different on every account.
The approach
How we worked the problem
Velocity problems are almost never typing-speed problems. We rebuilt the handover, the build standard and the QA gate before we accelerated anything.
- 01
Fix the intake before the output
The freelancer era ran on Slack threads and screenshots, and half of every three-week turnaround was clarification ping-pong. We replaced it with a definition of ready: a brief enters the queue only as a Figma file plus a one-page hypothesis doc — audience, metric, platform, QA tier. Incomplete briefs bounce back in an hour, not on day nine.
Everything runs inside Ravenline’s own Jira and Slack, so from the outside the consultancy simply got faster. A monthly capacity retainer replaced per-build quoting, which killed the estimate-negotiate-wait loop that had been adding days before any code existed.
- Definition of ready: Figma + hypothesis doc, or it bounces same day
- All delivery inside Ravenline’s Jira — no parallel tracker
- Capacity retainer, drawn down per build, reconciled monthly
- 02
One build standard, three platforms
Ravenline’s clients had already bought VWO, Optimizely or Convert, and nobody was migrating for our convenience. So we wrote one internal build standard — anti-flicker, targeting, activation, analytics events — and mapped it onto each platform’s quirks once, in a shared playbook, instead of rediscovering them per test.
Every variant ships with fallback-to-control error handling: if anything in the variant throws, the visitor silently gets the original page and the event is logged. The 40-minute checkout outage was the ghost in every conversation, and this is what exorcised it. Ten months, zero rollbacks.
- Shared platform playbook: VWO, Optimizely Web, Convert
- Fallback-to-control wrapper on every variant, no exceptions
- Standard event schema so results read identically across tools
- 03
QA to the end client’s bar, before Ravenline ever sees it
A 31% bounce rate is not a talent problem, it is a standards problem: nobody had written down what each end client’s QA actually checks. We did — a per-client matrix covering browsers, devices, accessibility tier and edge states — and made passing it the gate for handover. Nothing reaches Ravenline that has not already survived a harder review than the client will give it.
Builds come back as unbranded preview links with the completed matrix attached, so Ravenline forwards them to clients without touching, rebadging or re-testing anything.
- Written QA matrix per end client, versioned as their stack changes
- Internal gate stricter than any client’s own review
- Unbranded preview link + signed-off matrix as the delivery unit
- 04
Report the number that could embarrass us
Velocity is a vanity metric if quality slips while it climbs — twelve sloppy builds a month is worse than four sound ones. So the monthly report Ravenline receives leads with builds shipped, but always alongside client-QA bounce rate and live rollbacks, and one internal miss is written up in it below. If those two lines ever move the wrong way, the velocity number does not count.
- Monthly report: velocity, bounce rate and rollbacks together
- Internal QA failures written up, not buried
- Retainer scope reviewed quarterly against the roster, not assumed
The log
The build log
Around eighty builds in ten months. Six that tell the story — early milestones and representative tests. The uplift column is the relative change in each test’s primary metric, as called by Ravenline’s analysts: we build, they run the numbers.
- 01
Pilot build, day four: sticky basket bar (VWO)
Because 57% of mobile PDP scrolls carried past the add-to-basket button, we believe a sticky basket bar will raise mobile add-to-basket.
- 02
Same test, two stacks: delivery-promise banner (VWO + Optimizely)
Because exit surveys on both accounts named delivery uncertainty first, we believe a dated delivery promise in the buy box will raise add-to-basket.
- 03
Bundle builder PDP — failed our own QA, rebuilt in three days
Because bundle buyers show 2.3× the average order value, we believe a three-step bundle builder on the PDP will raise revenue per visitor.
- 04
Checkout trust block (Convert)
Because payment doubt was the top exit-survey theme at checkout, we believe security and returns signals beside the pay button will raise checkout completion.
- 05
Plan-tier reorder, server-side (Optimizely)
Because the mid tier converts best when framed against the top tier, we believe leading with the premium plan will raise mid-tier take-up.
- 06
Collection filter simplification (VWO)
Because usage of 10 of the 14 filter options rounded to zero, we believe collapsing to four primary filters will raise collection-to-PDP click-through.
- Pilot build, day four: sticky basket bar (VWO)+5.6%97%shipped
- Same test, two stacks: delivery-promise banner (VWO + Optimizely)+4.2%96%shipped
- Bundle builder PDP — failed our own QA, rebuilt in three days-1.8%94%not shipped
- Checkout trust block (Convert)+6.3%98%shipped
- Plan-tier reorder, server-side (Optimizely)+3.9%95%shipped
- Collection filter simplification (VWO)+0.7%flatnot shipped
| Experiment | Relative change | Confidence | Shipped |
|---|---|---|---|
| Pilot build, day four: sticky basket bar (VWO) | +5.6% | 97% | Yes |
| Same test, two stacks: delivery-promise banner (VWO + Optimizely) | +4.2% | 96% | Yes |
| Bundle builder PDP — failed our own QA, rebuilt in three days | -1.8% | 94% | No |
| Checkout trust block (Convert) | +6.3% | 98% | Yes |
| Plan-tier reorder, server-side (Optimizely) | +3.9% | 95% | Yes |
| Collection filter simplification (VWO) | +0.7% | flat | No |
The results
What it moved
Ten months in, Ravenline ships 11.5 builds a month against 3.7 before — 3.1× the velocity — and median turnaround from approved brief to client-ready link has fallen from 16 business days to 2.1. The bundle builder in the log is the honest asterisk: it failed our own matrix on iOS Safari, was rebuilt in three days, and never reached the client broken. That is the system working, not failing.
The quality lines held while the volume tripled, which is the only reason the velocity number means anything. Client-side QA bounces fell from 31% of builds to 2% across the engagement — two builds, both in the first two months — and the last five months have been clean. Live rollbacks went from four in the prior year to none, because fallback-to-control means a broken variant now degrades to the original page instead of taking a checkout with it.
The commercial result is the one the founder cares about. Delivery margin is up because a retainer costs less than the two hires it replaced, and the roster has grown from six retained clients to nine without a single developer joining payroll. Ravenline sells strategy again. The builds are just there.
- Builds shipped per month+211%3.711.5
- Median build turnaround-87%lower is betterdays16days2.1
- Builds bounced by client QA-94%lower is better31%2%
- Retained clients on roster+50%69
| Metric | Before | After |
|---|---|---|
| Builds shipped per month | 3.7 | 11.5 |
| Median build turnaround | days16 | days2.1 |
| Builds bounced by client QA | 31% | 2% |
| Retained clients on roster | 6 | 9 |
| Period | Experiment velocity |
|---|---|
| M1 | builds3.7 |
| M2 | builds4.5 |
| M3 | builds5.8 |
| M4 | builds6.4 |
| M5 | builds7.9 |
| M6 | builds8.6 |
| M7 | builds9.3 |
| M8 | builds10.1 |
| M9 | builds10.8 |
| M10 | builds11.5 |
“We stopped hiring test developers and started sending Optyv a brief instead. Twelve builds a month, none of them bounced by QA, and my team never touches the code.”
