The hardest thing about running a CRO agency is not finding the insight. It is that a test which misses Thursday makes you look slow to the client, while the vendor who caused it never hears about it. That asymmetry is the whole problem with buying development, and it is why agencies churn through build partners at a rate they would find alarming in any other supplier.
We sell Shopify development, so read this the way you would read any supplier writing about its own category. What follows is the buying criteria we would use if we were sitting on your side of the table, including the parts that make us easier to say no to.
What Shopify specifically makes hard, the five questions worth asking before you send anybody a real brief, how to run a pilot that tells you something, what the market charges, and the two contract terms that matter more than the rate.
The build queue is your problem
You can hire strategists. You can win retainers. What breaks agencies is throughput, and throughput is the one part of the operation most agencies have outsourced to someone with no exposure to the consequences of missing.
Notice the shape of it. A variation that flickers gets escalated to your Slack, not theirs. A build that slips gets explained by you on the client call. Every new client theme resets your vendor’s learning curve and you pay for the ramp in missed dates. None of that appears in their invoice or their retrospective.
Most agencies with in-house developers still hit this, because the internal team gets pulled onto whichever client is loudest that fortnight. The realistic question is not whether to have your own developers. It is what you send outside, and how you choose who catches it.
What Shopify makes hard
A generalist Shopify developer and an experiment developer are not the same hire, and the difference only shows up on somebody else’s store. Client-side testing tools inject after the theme has rendered, which on Shopify means after the apps have too, and the apps are where the surprises live.
Five things reliably catch out a team that has built Shopify stores but has not run experiments on them. Ask about the last one first: on a slow template you are measuring the load rather than the layout, and theme weight and third-party scripts are the usual reason a design test comes back flat.
- App collisions. A client installs a review app or an upsell app mid-test and the variation silently stops matching the control. A team that has lived here will name that failure mode before you do.
- Online Store 2.0 sections. Merchants reorder sections without telling anyone, so variation code written against a fixed DOM position breaks the moment somebody drags a block in the theme editor.
- Checkout is not yours to edit. Outside Plus, checkout is off limits, and half the tests an agency wants to run on it need Functions or extensibility rather than JavaScript.
- Page-builder themes. Anything built in a drag-and-drop app generates markup that changes between saves, which turns a two-hour build into a two-day one and is rarely priced for.
- Template speed. Heavy themes and stacked third-party scripts add enough delay that the variation is judged partly on how fast it arrives.
Five questions before you send a brief
These are diagnostic rather than technical. You are not testing whether they know the answer, you are testing whether they have been in the situation. A vendor who has answers with a specific failure mode; a generalist answers with a reassurance.
- What happens to a running variation when the client installs a new app? The useful answer describes a monitoring habit, not a promise that it will be fine.
- Show me a QA document from a real build. If it does not name browsers, devices and the goals validated, it is not a QA document, it is a note.
- How do you stop the visitor seeing the control first? Listen for injection timing and a performance budget. Listen for whether they mention it costing anything, because it does.
- Who ships the winner into the theme when the test wins, and is that quoted separately? Winners left in the testing tool become permanent weight on every page.
- Which of the tests in my last backlog would you have refused to build? Anybody who says none of them has not thought about whether the pages could produce a readable result.
Four of the five are really one question asked from different angles: has this team lived in the engineering half of conversion work, or in web development generally? Both are legitimate trades. Only one of them is the one you are buying.
Run a paid pilot, and read the paperwork
Send one real brief, the way you would brief a live client test, and pay for it. Free pilots select for vendors with idle capacity, and they give you no information about how somebody behaves when the work is chargeable and the queue is full.
Then judge the scope response as hard as you judge the build. What comes back before any code is written tells you more than the finished variation does: whether they identified the risk in your client’s theme, whether the price is fixed or indicative, and whether the date is a commitment or an aspiration.
A scope that names a risk you had not spotted is worth more than one that comes back cheaper. The second kind tends to rediscover the risk halfway through the build, at which point it is your problem and your client’s call.
Keep your own scorecard from the first pilot onwards: builds delivered, dates missed, QA bounces, rollbacks. Vendors quote their own numbers and nobody audits them. Yours is the only version anybody can check, and it makes the renewal conversation short.
What the market charges
These are quotes we see, not our rate card. Individual client-side test builds on Shopify sit somewhere between £150 and £900 depending on how much of the page moves and how hostile the theme is. Reserved sprint capacity is more often quoted between £2,500 and £6,500 a month.
Two to five working days for a standard client-side build is the normal quoted range, which is worth knowing mainly so you can tell an outlier in either direction. Much slower usually means you are in a queue behind product work. Much faster usually means QA is the thing being compressed.
The rate is the least interesting variable. A build that arrives on time, flicker-free and correctly tracked at the top of that range costs less than a cheap one you have to re-run, because re-running it costs you three weeks of client patience as well as the second invoice.
The two terms worth more than the rate
The first is non-solicitation, in writing. Every vendor selling white-label test development will say yes to it on a call. Only some will put it in the agreement, and the ones who hesitate are telling you something you should listen to.
The second is code ownership. Everything written should land in a repository you or your client controls, documented well enough that somebody else could pick it up. If parting ways means losing the work, you have not hired a supplier, you have acquired a dependency.
Beyond those two, structure the agreement so the vendor learns about a missed date before your client does. That single condition does more for a partnership than any service level in the document. If you want a second opinion on a partner you are already using, or a pilot brief scoped without a sales call attached to it, get in touch.
