Ten months of testing, +24% subscription starts
- Subscription starts
- +24%
- from 2.9% to 3.6% of sessions
- Second-box retention
- +19%
- from 58% to 69%
- Experiments run
- 17
- 9 winners, 4 flat, 4 losses
- Added subscription revenue
- £286k
- annualised from tested lift
The brief
What they came to us with
Grano & Sale imports pantry staples from small Italian producers — single-estate oil, bronze-die pasta, tinned San Marzano, salt from Trapani — and ships them as a recurring box. The catalogue is beautiful and the food is genuinely good. The commercial problem had nothing to do with either.
The brand was buying first boxes efficiently and then losing them. Subscription starts sat at 2.9% of sessions, which was liveable; second-box retention sat at 58%, which was not. Two in five people who paid for a box never took another one, and 14.2% cancelled before the first box had even been picked. The economics only work from box three onwards, so the whole business was running on a leaky first month.
We were hired to raise subscription starts. Within a fortnight it was obvious that raising starts alone would make things worse — more people entering a funnel that dropped 42% of them is just a more expensive way to lose money. So the brief we actually worked to was: sell the subscription honestly enough at the start that the second box survives.
Constraints
- Perishable and seasonal stock. Oil is pressed once a year, some producers ship twice; any test that promised choice had to be honest about what was actually in the warehouse that week.
- Fixed delivery windows. Boxes leave on Tuesdays and Thursdays from one Bologna depot, so "pick your day" was never on the table — the test had to make the real window feel controlled, not invent a fake one.
- Low order value. £31.60 average first box against a £4.90 shipped cost per parcel. A test that lifted starts but dropped basket size was a net loss, so every result was read against contribution, not conversion.
- Around 74,000 sessions a month. Enough for one clean test at a time on the subscription funnel, and nowhere near enough to test the checkout micro-copy anyone wanted to argue about.
The approach
How we worked the problem
The first four weeks produced no tests and one uncomfortable finding: almost everything people wanted to know before subscribing was only answerable after subscribing. The roadmap was ordered by how much of that gap each idea closed.
- 01
Interview the churned, not the happy
We pulled 340 cancellations from the previous quarter and got 96 of them on email or a ten-minute call. The reasons clustered hard: 38% could not predict what would arrive next month, 24% had a box land while they still had unopened pasta from the last one, and 19% could not find the pause control and cancelled instead because it was faster.
Only 6% mentioned price. That single number redirected the entire roadmap away from discounting, which is where the team had been planning to spend the budget.
- 340 cancellations sampled, 96 responses
- Reasons coded into 7 themes, weighted by revenue lost
- Pause vs cancel behaviour traced in Recharge for 12 months
- 02
Write hypotheses that can be wrong
Everything entered the roadmap in one shape: because [evidence], we believe [change] will move [metric] by [amount] for [audience]. Anything that could not be falsified — "make the brand feel more Italian" — went in a separate document and stayed there.
We also pre-registered the losing condition. For the restock-timing test, a lift in starts with a drop in second-box retention counted as a loss, not a win, and we agreed that in writing before it launched.
- 29 hypotheses written, 17 shipped
- Scored on evidence strength, reach and build cost
- Minimum detectable effect calculated before build, not after
- 03
Test on contribution, not on the checkout button
With a £31.60 basket and a £4.90 parcel, a 5% lift in starts that costs 3% of basket value is roughly break-even and nobody notices until the quarter closes. Every variant was instrumented to report first-box value and 60-day retained value alongside the start rate.
Two ideas died on that instrumentation rather than on the primary metric — including a heavily discounted trial box that converted beautifully and retained appallingly.
- Contribution per subscriber tracked as a guardrail on every test
- 60-day retained value read before any ship decision
- Stock feed wired into variants so no test promised a sold-out producer
- 04
Call tests honestly, including the four we lost
Fixed sample size agreed before launch, a minimum of two full delivery cycles, and no calls made on a Friday afternoon because the graph looked good. Four tests lost and four came back flat. All eight were written up at the same length as the winners.
The discounted trial box was the expensive one to get wrong, and finding out for £0 in test traffic rather than a full quarter of margin is the whole argument for running the programme this way.
- Pre-registered stopping rules, no interim peeking
- 95% confidence floor for a ship decision
- Every result written up in a shared decision log
The log
The experiment log
Seventeen tests over ten months. The six that decided the direction, with the relative change in subscription-start rate for the tested audience.
- 01
Show the next two boxes, by producer
Because 38% of churned subscribers could not predict what would arrive next, showing the actual contents of the next two boxes on the plan page will raise subscription starts and second-box retention.
- 02
Choose your restock rhythm: 4, 6 or 8 weeks
Because 24% of cancellations cite a box arriving before the last one was finished, offering three restock intervals at sign-up will raise starts without diluting contribution.
- 03
Delivery-window strip above the price
Because delivery uncertainty is the second-highest exit-survey theme, stating the exact Tuesday or Thursday a first box lands will raise subscription starts on mobile.
- 04
"Pause, do not cancel" in the buy box
Because 19% of cancellations were people who could not find the pause control, promising pause-anytime at the point of sign-up will raise starts and cut first-order cancellation.
- 05
Regional provenance map on the plan page
Because producer-story pages have the highest assisted-conversion rate on site, a map of sourcing regions on the plan page will raise subscription starts.
- 06
£12 discounted trial box
Because only 6% of churned subscribers cite price, we believe a discounted trial box will raise starts but fail its retained-value guardrail.
- Show the next two boxes, by producer+7.1%97%shipped
- Choose your restock rhythm: 4, 6 or 8 weeks+5.4%96%shipped
- Delivery-window strip above the price+3.8%95%shipped
- "Pause, do not cancel" in the buy box+2.6%95%shipped
- Regional provenance map on the plan page+0.3%flatnot shipped
- £12 discounted trial box-4.7%93%not shipped
| Experiment | Relative change | Confidence | Shipped |
|---|---|---|---|
| Show the next two boxes, by producer | +7.1% | 97% | Yes |
| Choose your restock rhythm: 4, 6 or 8 weeks | +5.4% | 96% | Yes |
| Delivery-window strip above the price | +3.8% | 95% | Yes |
| "Pause, do not cancel" in the buy box | +2.6% | 95% | Yes |
| Regional provenance map on the plan page | +0.3% | flat | No |
| £12 discounted trial box | -4.7% | 93% | No |
The results
What it moved
Ten months in, subscription starts have moved from 2.9% to 3.6% of sessions — up 24% — on the same media spend and the same catalogue. Second-box retention went from 58% to 69%, and first-order cancellation more than halved, from 14.2% to 8.6%.
The four winners in the log above compound to roughly 20%. The remaining four points came from five small winners nobody would put in a deck: a clearer allergen line, a shorter address step, a producer photograph on the confirmation screen. Individually forgettable, collectively worth about a fifth of the result.
Average first box rose too, from £31.60 to £38.90, mostly because the restock-rhythm choice let people take a larger box less often instead of a smaller one they resented. That was not the hypothesis. It is the kind of thing you only find by measuring basket value on a test about timing.
The programme now runs two concurrent tests with the internal team writing their own hypotheses, and the £12 trial box has stayed dead through two separate attempts to resurrect it.
- Subscription starts+24%2.9%3.6%
- Second-box retention+19%58%69%
- First-order cancellation-39%lower is better14.2%8.6%
- Average first box+23%£31.6£38.9
| Metric | Before | After |
|---|---|---|
| Subscription starts | 2.9% | 3.6% |
| Second-box retention | 58% | 69% |
| First-order cancellation | 14.2% | 8.6% |
| Average first box | £31.6 | £38.9 |
| Period | Subscription starts |
|---|---|
| M1 | 2.9% |
| M2 | 2.92% |
| M3 | 3.01% |
| M4 | 3.05% |
| M5 | 3.14% |
| M6 | 3.22% |
| M7 | 3.28% |
| M8 | 3.41% |
| M9 | 3.52% |
| M10 | 3.6% |
“They told us not to redesign our navigation — the test they ran killed our own pet idea. Saved us six figures.”
