A/B testing has traditionally required Shopify agencies to add another layer to the merchant's technology stack. Testing platforms, custom scripts, duplicated themes, analytics integrations, and deployment processes were often necessary just to answer a relatively simple question: does version B perform better than version A?
Shopify Rollouts changes that model by bringing controlled releases and experimentation directly into the platform. Agencies can now schedule changes, release them gradually, run temporary experiences, or compare a variation against a control across the online store theme, checkout, and Customer Accounts. More importantly, Shopify handles traffic allocation and provides experiment analytics within the same environment where those experiences are managed.
The significance for agencies goes beyond replacing an external A/B testing tool. Rollouts introduces experimentation into the Shopify deployment workflow itself, creating a closer connection between development, release management, analytics, and CRO.
How Rollouts actually works
Rollouts is managed from Markets > Rollouts in the Shopify admin. Rather than being exclusively an experimentation feature, it acts as a deployment layer where merchants and agencies can define a set of changes and control when, where, and to whom those changes are released.
Shopify currently defines three primary rollout types:
• Launch. A set of changes is scheduled and ultimately merged permanently into the store. This can be used for something as straightforward as publishing a redesigned theme at a specific date and time.
• Event. Changes are published temporarily and automatically reverted when the event ends. A merchant could, for example, activate a dedicated Black Friday experience for a defined period without manually restoring the previous configuration afterward.
• Experiment. A variation is tested against the current experience as a control. Traffic is divided between both versions so their performance can be compared before deciding whether the variation should become permanent.
This makes Rollouts useful well beyond CRO. It can become part of everyday release management, seasonal merchandising, migrations, and risk controlled deployments. In fact, some of the most immediate benefits agencies can obtain from Rollouts have nothing to do with A/B testing at all.
From experimentation to everyday release management
The value of Rollouts is not limited to sophisticated A/B testing programs. Some of its most practical applications involve solving much simpler operational problems around how and when storefront changes go live.
We are already using Rollouts with Tanglefree, where promotional banners need to be activated on specific dates as campaigns begin. Rather than relying on someone to manually publish those changes at the exact moment a promotion goes live, the team can prepare the experience in advance and use Rollouts to control when it becomes available to customers.
The implementation itself is intentionally simple, but the operational benefit is meaningful. Promotional launches often involve multiple moving parts, and manually coordinating storefront changes introduces unnecessary dependency on team availability and increases the possibility of timing errors. Scheduling those changes in advance makes the release process more predictable and allows the team to focus on the campaign rather than the mechanics of publishing it.
It is also a useful example of how agencies can begin incorporating Rollouts into existing workflows without immediately building a full experimentation program. Scheduled promotional changes can be the starting point. From there, the same infrastructure can support temporary campaign experiences, gradual releases, and eventually controlled experiments comparing different storefront experiences.
For agencies, this progression is important. Rollouts does not need to enter the workflow as an advanced CRO initiative from day one. It can first solve straightforward release management problems and then become part of a broader experimentation strategy as the merchant's needs evolve.
Traffic allocation is more flexible than a simple 50/50 test
For experiments, Shopify uses a 50 percent control and 50 percent variation split by default, but agencies can change that distribution before launching the experiment. Experiments also default to an end date 90 days after launch, which can also be changed.
There is an important distinction between traffic allocation and traffic split.
Traffic allocation determines what percentage of eligible visitors participates in the rollout. Traffic split determines how participating visitors are distributed between the control and variation.
For example, an agency could configure an experiment to reach only 50 percent of eligible visitors and then divide those participants equally between control and variation. The remaining eligible traffic would remain outside the experiment.
Eligibility can also depend on market. This allows agencies to limit a rollout to particular geographic or commercial markets instead of exposing every visitor to the same experiment.
The distinction becomes particularly useful when managing risk. A significant theme or checkout change can initially be exposed to a relatively small audience instead of immediately becoming an all traffic experiment.
Shopify's current Rollouts interface also surfaces the percentage of traffic reached and how that traffic is distributed between versions, giving teams greater visibility into the actual audience participating in the test.
What does Shopify measure?
Native experimentation becomes much more interesting when measurement is handled within the same system. Rollouts automatically collects analytics after launch, although the available metrics depend on the type of resource being changed and whether the rollout is a launch or an experiment.
For an online store theme experiment, Shopify can report metrics including:
• Conversion rate over time
• Bounce rate
• Reached checkout rate
• Add to cart rate
For checkout and Customer Accounts experiments, the available metrics focus more directly on the transaction itself, including checkout conversion rate.
Combined experiments can report storefront and checkout metrics in separate sections. This matters when an experiment modifies multiple parts of the journey because agencies can distinguish storefront behavior from checkout performance rather than reducing the entire test to a single number.
There are also limitations agencies need to understand. Rollouts does not currently allow merchants to define arbitrary custom experiment metrics. Shopify determines which metrics are available according to the resources and changes included in the rollout.
Rollout analytics also measure visitors across the Online Store, cart, and checkout. Traffic on other surfaces where a rollout might apply, including headless storefronts and Shopify POS, is not included in the experiment results.
For technical and CRO teams, that distinction is important. Rollouts provides native measurement, but it does not eliminate the need for a broader analytics stack when an experiment needs to be evaluated against merchant specific KPIs, downstream retention, contribution margin, or behavior outside Shopify's measured surfaces.
Experiments become part of the theme development workflow
One of the most interesting aspects of Rollouts for agencies is how it changes the relationship between experimentation and theme development.
Historically, testing a substantial Shopify theme change could involve creating a duplicate theme, modifying it, connecting an experimentation layer, controlling visitor assignment, collecting data, and then manually determining how the winning implementation should be merged back into production.
Rollouts brings much of that process closer to Shopify's native theme architecture.
Agencies can test entirely different themes, not simply small visual variations. A merchant could compare its existing production theme against a substantially redesigned version, test a different navigation architecture, evaluate a new PDP structure, or gradually introduce a new brand experience.
Shopify also creates a copy of the published theme or checkout and accounts configuration involved in the rollout. This is important operationally because the live resource can continue to evolve independently while the experiment remains isolated.
That addresses a practical problem agencies regularly face: development does not stop simply because an experiment is running.
The production team may need to fix bugs, deploy unrelated features, update merchandising, or continue development while CRO experiments remain active. Keeping the rollout resource separate reduces the risk that ongoing production changes unintentionally alter the experiment.
Multiple experiments can run without contaminating each other
Another technically relevant capability is mutually exclusive experimentation.
Shopify allows multiple live experiments to run simultaneously while preventing the same visitors from participating in conflicting experiments. This is particularly useful for agencies managing continuous optimization programs where waiting for every individual test to finish before starting another would dramatically reduce testing velocity.
This does not remove the need for careful experiment design. Agencies still need to think about interactions between changes, sample size, test duration, seasonality, and whether two experiments could influence the same business metric.
But Shopify now provides infrastructure for running multiple experiments without requiring agencies to build their own traffic exclusion logic.
For mature CRO programs, that can significantly simplify operations.
Themes, checkout, and Customer Accounts can be tested together
Rollouts also expands the definition of what constitutes an ecommerce experiment.
A change does not always live entirely within a theme. A redesign might include storefront modifications, a different checkout configuration, and changes to Customer Accounts. Shopify now allows these resources to be combined into a single rollout.
That opens the door to testing more complete customer journeys.
An agency could create a variation containing a redesigned storefront experience together with checkout changes and then evaluate the combined impact rather than testing each component in isolation.
Alternatively, individual surfaces can still be tested independently when the objective is to isolate a specific variable.
This flexibility is important because not every optimization question should be answered through a narrowly scoped UI test. Sometimes the hypothesis concerns an entire journey rather than a button, headline, or individual component.
Market specific experimentation adds another layer
Shopify Markets is also connected to Rollouts.
Agencies can schedule or test localized theme content for specific markets, which creates opportunities to evaluate whether different experiences perform better across regions rather than assuming that one storefront configuration should work globally.
This can include localized messaging, calls to action, merchandising decisions, or broader theme changes.
For agencies working with international Shopify merchants, this creates a more sophisticated optimization model. Instead of asking whether variation B performs better globally, the agency can begin asking whether it performs better for the audience where the hypothesis actually applies.
That distinction becomes increasingly important as Shopify pushes more configuration into Markets.
Plan availability matters
Rollouts should not be presented to clients as universally available experimentation functionality without considering plan requirements.
Experiments require the merchant to be on Shopify Grow or higher.
That means agencies need to consider the merchant's plan when designing an experimentation strategy. The feature may reduce dependence on external testing tools for eligible merchants, but it does not automatically make native A/B testing available to every Shopify store.
It is also worth distinguishing experimentation from the broader Rollouts feature set. Rollouts covers scheduled launches and temporary events in addition to experiments, while specific capabilities and analytics depend on rollout type and merchant configuration.
Rollouts does not eliminate experimentation expertise
Native infrastructure removes a considerable amount of technical friction, but it does not answer the most difficult questions.
Shopify can split traffic. It can maintain a control and a variation. It can collect supported metrics. It can help isolate experiments and coordinate deployments. What it cannot decide for an agency is whether the hypothesis is commercially meaningful.
A poorly designed test remains a poorly designed test regardless of how sophisticated the experimentation infrastructure becomes. Agencies still need to determine what should be tested, which metric represents success, how long a test should run, whether the observed result is actionable, and whether the potential improvement justifies the implementation cost.
They also need to recognize when Rollouts' native metrics are insufficient and deeper analytics are required.
This shifts the agency's value from operating experimentation software toward designing an experimentation program.
What happens to third party testing tools?
Rollouts inevitably changes the conversation around Shopify A/B testing apps, but it would be premature to describe it as the end of the category.
For many merchants, native testing of themes, checkout configurations, and Customer Accounts may remove the need for an external platform in relatively straightforward experiments. Native traffic management, deployment, and measurement can reduce technical complexity and eliminate another script or application from the stack.
More sophisticated experimentation platforms can still provide capabilities beyond Rollouts, particularly around advanced statistics, custom goals, personalization, deeper audience segmentation, cross channel measurement, and specialized experimentation workflows.
The practical question for agencies is therefore not whether third party testing platforms disappear.
It is when Shopify Rollouts is sufficient and when the merchant's experimentation requirements justify something more specialized.
That assessment itself can become part of the agency's strategic role.
A new development model for Shopify agencies
Rollouts ultimately brings experimentation closer to the standard Shopify development lifecycle.
Instead of treating CRO as a separate discipline that begins after developers finish building something, agencies can increasingly design projects around a cycle of hypothesis, development, controlled deployment, measurement, and iteration.
A developer can build a new theme experience. The agency can expose it to a controlled percentage of eligible traffic. Shopify can collect native performance metrics. The CRO or analytics team can evaluate the results. The winning implementation can then move toward broader deployment.
That creates a much tighter relationship between development and optimization.
For agencies, this may be the most important consequence of Rollouts. Native A/B testing does not simply make experimentation easier. It creates the possibility of making experimentation part of how Shopify development itself is performed.
The Tanglefree example also illustrates that agencies do not need to wait until they have a sophisticated experimentation framework to begin adopting this model. Using Rollouts for predictable promotional deployments can establish the operational discipline and familiarity with the tool that later makes gradual releases and controlled experiments easier to introduce.
As that model becomes more common, clients may increasingly expect agencies not only to deliver changes but also to demonstrate whether those changes improve the business.
The agencies that adapt will therefore need a combination of development expertise, analytics knowledge, CRO methodology, and commercial judgment. Shopify is providing more of the infrastructure required to run the experiment. The agency's job increasingly becomes deciding which experiments are worth running and what to do with the results.
If your agency is evaluating how Rollouts could fit into its development workflow, CRO services, or client optimization programs, now is the right time to determine where native experimentation can simplify the stack and where more advanced testing capabilities are still necessary.
Want to explore how Shopify Rollouts can create new opportunities for your agency and your clients?
