How to activate headline A/B testing

Marfeel offers headline and image A/B testing as an add-on that lets publishers compare headline and image variants to find what resonates best with readers. This article assumes you have the feature enabled on your account. If you don’t, contact your Account Manager to activate a trial.

Marfeel offers two different ways to implement A/B testing: a no-code setup managed from hub.marfeel.com, and a JavaScript-based instrumentation that loads the runtime directly on the page.

Note that if you have articles with very few clicks, the AB test is unlikely to give statically relevant results. In such cases, the data may not provide enough evidence to confidently declare a variant as the winner.

Therefore, it is recommended to only use AB testing on pages with significant traffic that will generate clicks on the articles. This will ensure that the test produces meaningful and reliable insights for decision-making purposes.

No-code A/B testing activation

Marfeel allows publishers to enable headline and image A/B testing without having to add any code or extra instrumentation. The Marfeel SDK lazy loads the required A/B testing runtime, so you can get started within seconds.

This option can be easily enabled from hub.marfeel.com and is ideal for getting started without any friction or cost.

To activate headline and image A/B testing:

1. Create a new Editorial A/B testing Tag Experience:
2. Add a title (this is for internal purposes only).
3. Under Where do you want to set A/B tests?, specify the url of the page where you want to have A/B testing enabled:
4. Admins have control over how they want to proceed once the system has identified a statistically relevant winner of an A/B test. There are two options:
   • Have Marfeel automatically replace the article: Keep winner variant
   • Have the system report the results and make the permanent change on the CMS: Keep original variant
5. Experiments end whenever there is a clear winner with statistical significance or, in case variants are too similar, whenever the max duration is met. Max duration’s default value is 30 minutes, but it can be configured to be up to 2 hours for pages with low traffic.

Regardless of which option you choose, the best practice for replacing the current headline or image with the new winner is to permanently change it in the CMS.

Activate notifications

Subscribe to notifications to receive an alert when experiments are complete. This way, you won’t have to check manually to verify if a given test has finished.

Go to the A/B test experience set up above and click Subscribe.

Select from the menu options to receive notifications via email, Slack, or Microsoft Teams. The best way to use these notifications is to subscribe a Microsoft Teams or Slack channel where your homepage editors are.

Once activation is complete, editors can run headline and image A/B tests directly from the HUD.

JavaScript-based instrumentation

Marfeel allows publishers to load the A/B testing runtime directly on the page using JavaScript. This method avoids a round-trip to Marfeel servers and helps prevent visual flickers when swapping headlines.

Once you’ve decided to fully adopt A/B testing, this activation method is strongly recommended.

This implementation does not remove the need of creating the no-code experience. Without one, AB tests will not run. If JavaScript implementation is present, the experience does not need to be active to work, but it's still recommended to keep it active.

To activate it, add these 2 extra lines of code to the pages where you want to enable A/B testing, oftentimes just the homepage.

• Remember to replace the site_id=0 with your account ID.
• The url=https://example.com/ parameter must contain the same URL used in the rel=canonical of the page.

<script type="module" src="https://marfeelexperimentsexperienceengine.mrf.io/experimentsexperience/render?siteId=0&url=https://example.com/&experimentType=HeadlineAB&lang=es&version=esnext" async></script>
<script type="application/javascript" src="https://marfeelexperimentsexperienceengine.mrf.io/experimentsexperience/render?siteId=0&url=https://example.com/&experimentType=HeadlineAB&lang=es&version=legacy" nomodule="" async></script>

The nomodule attribute on the script tag is a browser hint to only load one or the other depending on its supported features. It’s not supposed to load both of them. If nomodule is not present, both JS files will load incorrectly. You can read more about nomodule on Mozilla docs.

You can also set up image A/B testing using the same activation process to test both headlines and images.

What is the difference between no-code and JavaScript A/B testing activation?

No-code activation is enabled from hub.marfeel.com and lazy loads the A/B testing runtime via the Marfeel SDK. JavaScript instrumentation loads the runtime directly on the page, avoiding a round-trip to Marfeel servers and preventing visual flickers when swapping headlines.

What happens when a headline A/B test finds a winner?

Admins can choose between two options: have Marfeel automatically replace the article with the winning variant (Keep winner variant), or have the system report the results so the editor can make the permanent change in the CMS (Keep original variant).

How long does an A/B test run before it ends?

Experiments end when there is a clear winner with statistical significance. If variants are too similar, the test ends when the max duration is met. The default max duration is 30 minutes, but it can be configured up to 2 hours for pages with low traffic.