Experiments
Last updated
Ploy Experiments lets you A/B test one section of a live marketing page, such as a hero or pricing table. You choose a goal event, describe the challenger to Ploy's Agent in a chat, and send between 10% and 90% of visitors to it. Then you promote the better version and publish it.
An experiment shows your visitors two versions of one page section: the current section, called the champion, and a new version, called the challenger. Ploy measures how each version performs on a conversion event you choose. When you have seen enough, you choose which version becomes the next champion.
Limits and plan requirements
- Experiments is rolling out. It is turned on per workspace. If you don't see Experiments in your site navigation, your workspace doesn't have it yet.
- Each experiment tests one section of one page on a Ploy site that has been published at least once.
- Your goal and guardrails must be conversion events your site already tracks. The form takes one guardrail, and Ploy's Agent can set up to three. See Analytics for how events are tracked.
- Your plan sets how many experiments can be running or paused at once across your organization. See Plan limits.
- Promoting a winner does not publish it. Visitors see the new version after you publish the site to Production.
Create an experiment
- Open your site and select Experiments in the site navigation.
- Choose to create a new experiment. The Create an experiment dialog opens.
- Fill in the fields described below.
- Click Create and edit challenger.
| Field | What to enter |
|---|---|
| Experiment name | A name your team will recognize. |
| Page URL | The page that has the section you want to test, for example /pricing. |
| Section key | The section on that page to test. |
| Hypothesis | What you expect the challenger to change, and why. |
| Goal | The conversion event that decides which version performs better. |
| Guardrail (optional) | A conversion event that should not get worse while you test. |
The form takes one guardrail. To watch up to three guardrails, ask Ploy's Agent to set them up in a chat.
Edit the challenger
After you create the experiment, you edit the challenger in a Ploy chat with Ploy's Agent. You can compare the drafts before you start the experiment.
Start the experiment
Your site needs at least one publish before an experiment can start. Until then, you see the notice Publish before starting an experiment.
- Set the Traffic split. The challenger can get from 10% to 90% of visitors.
- Click Start experiment. Starting an experiment publishes and verifies the live page.
Use Pause and Resume as needed. A draft that never started can be removed with Delete draft.
Statuses
As an experiment moves along, it can show one of these statuses:
- Not started
- Publish required
- Needs proof
- Gathering visitors
- Completed
- Archived
- Needs manual repair
Read the results
While the experiment gathers visitors, it shows how the champion and the challenger perform on your goal and any guardrails.
Guardrail warnings
Once every version has enough visitors, Ploy checks each guardrail event. If the challenger is very likely doing worse on a guardrail, the experiment shows Guardrail at risk: <event>, with the name of that event. You can still promote early, but Ploy shows a warning first.
Choose a champion and publish
- Click Choose champion. The Choose the next section champion dialog opens.
- Review the Check before you promote step, then promote the version you want to keep.
- Publish the site to Production. The dialog reminds you: Publish the site to Production afterwards.
Promoting is not publishing. Visitors do not see the new champion until you publish the site to Production. See Publishing & Deploys.
Plan limits
Your plan sets how many experiments can be running or paused at the same time. The count covers your whole organization, not one workspace. When you reach the limit, Ploy shows an upgrade window titled Upgrade for more than N experiment(s) at a time, where N is your plan's limit.
See Credits for how plans work, or the pricing page for current plans.
FAQ
I do not see Experiments in my site navigation
Experiments is enabled per workspace during rollout. If your workspace does not have it yet, the Experiments item does not appear.
Why can't I start my experiment?
Check that the site has been published at least once. If you see Publish before starting an experiment, publish the site, then start the experiment. Also check whether your organization has reached its plan limit for experiments.
I promoted a champion, but visitors still see the old section
Promoting a champion does not reach visitors until you publish the site to Production.
What's Next?
- Track the right events: Read Analytics to set up the conversion events you use as goals and guardrails.
- Go live: Read Publishing & Deploys to publish your new champion.
- Check your plan: See Credits for plan allowances.