DPG Media - Experimentation Framework

Working agile within a corporate environment. Something everybody seems to talk about. For VTM GO we were fortunate enough to put this into practice. By establishing the right mindset and setting up multiple processes and workflows it turned out to be very possible. In the following case, you will read more about the buy-in, processes we initiated, and some real-world examples. We believe this case will be insightful for everyone who either currently finds themselves in similar circumstances or is transforming into such a situation.

Implement an experimentation framework and mindset, increasing monthly active users by 20%, year-on-year.

Client

For DPG Media we have been working on several projects, including VTM and VTM GO. In the following case we present focus areas for both VTM as well as VTM GO.

VTM GO is a video-on-demand platform owned by DPG Media, which is the biggest commercial TV station in Benelux. With this streaming service, people inhabitants of Belgium can watch live and delayed TV via the internet. VTM GO is completely free, so you don't even need a cable subscription. The platform combines all Belgian DPG Media channels: VTM, Q2, Vitaya, CAZ, VTM Kids, VTM Kids Jr and Qmusic.

One Metric That Matters (OMTM)

Our client’s OMTM resolved around monthly active users on VTM GO. Monthly active users have been defined as users who watch at least one long-form, for example, an episode or movie, per month. It should be no surprise, there are multiple KPI’s that could potentially influence this metric that was tracked and analysed as well. More information about the One Metric That Matters can be found here.

Within the project, we were responsible for the entire user journey. To do so we used the infamous AAARRR-funnel, or pirate-funnel, to serve as a framework to gain more insights into which funnel-steps were most important.

Using this as a framework ensured we did not scatter our ideas across the user journey but instead had a clear focus, such as phases like the retention and activation of users.

Alongside the overarching goal, we had created some additional goals which supported the OMTM. We wanted to add more value by:

Implementing an experimentation mindset across departments, ranging from the creative department to development, to make sure everybody was aligned and willing to cooperate throughout the process.
Proposing new channels and implement it in a way that internal stakeholders, such as the marketing or the internal media-buying team, could incorporate this in their ongoing-layer / always-on marketing layer.
Sharing our expert opinion and improving current channels by demonstrating the growth framework of ideation, prioritization, testing, and analysing.

Methodology

Through a clear and structural methodology we wanted to ensure the quality of the data collection, data analysis and project deliverables. Additionally, it also shows which data insights were critical to come to the final results. We believe that the journey towards positive results is just as important as the final results. We implemented the following methodological approach in consultation with DPG-Media:

Multidisciplinary approach: all stakeholders represented to make swift decisions.
Impact: Delivering high-quality, impactful experiments focused on business value*.
Traffic: That can be completed in one sprint of two weeks to limit the scope of experiments.
Speed: While keeping the number of relevant experiments (per sprint) as high as possible.
Planning: Considering planning, priority and workload from each department.
Industrialization: Roll-out and scale of proven experiments by development.

Processes

When the methodology and requirements were clear to all stakeholders, the next critical element was to create a clear process and workflow that fits with DPG Media’s as well our own organizational structure. Together we designed the following processes that aimed to create a situation where the time window between ideation and experimentation is as small as possible:

We ensured we had a team consisting of a product-owner, marketing manager, tracking squad member, developers, and creatives to be as efficient as possible.
Working in sprints of two weeks.
Following our Growth Hacking process consistently.
Having one important sprint-meeting per week to present proposals, pivot, and share insights / results from current experiments.
This meeting is used to protect that the decision-making process could remain at a fast pace as all stakeholders are present.
Working on location is a prerequisite to make sure the speed of communication is as high as possible. Especially in a highly dynamic corporate environment.
Creating a structured way of communicating experiments, including:
Experiment design
Context
Results
Next steps

Put it into practice 1 – Browser notifications

One of the new channels we recently tapped into was browser notifications. We’ve been using PushEngage to send browser notifications with the goal to bring users back to the platform. After multiple sprints with different experiments, we’ve noticed promising results. However, each campaign had to be done manually and personalisation possibilities were very limited.

That’s why our goal was to increase the relevance and frequency of browser-notifications by sending automated, hyper-relevant episode-level browser notifications when a user stopped watching a show/program.

Results

 Significant results (vs ad-hoc campaigns)
Higher campaign effectiveness (CTR * LF-session): +25.02%
80% of users who clicked on notifications resumed watching Long Form videos where they left off.
The above metrics verify that our hypothesis was correct. Sending hyper-relevant (episode-level) browser-notifications works better than (current) ad-hoc notifications.
Do not forget the cost- and time-efficiency associated with this setup. As the notifications were automated and needed to be set up only once.

Put it into practice 2 – Registration flow

Through funnel analysis, we found that the registration flow/funnel was an important area of improvement for VTM Go. As soft data indicated multiple misclicks/frustration clicks and hard data showed drop-off rates we found there could be some interesting growth through-out the registration flow. In collaboration with tracking, development, and business we came up with several ideas, such as:

Testing conversational versus single registration form:

Hypothesis: An interactive registration process leads to more registrations (& less drop off during the process)

Frequency: AB test of two different form types, tested on different devices

50% of the users see current conversational form
50% of the users see single form, non-conversational form

KPI: % registration submitted; measured by conversion rate (registration / users)

Impact - KPI: We found that this experiment lead to +7.76% more registrations on Desktop

Step-visualisation

Hypothesis: If we show a stepper (step-visualisation) how many steps the user is going to make before he/she finishes the registration, we expect to see lower drop-offs at each step.

Frequency: A/B test showing a stepper for users logging in / register.

50% does not see a stepper - AS IS
50% does see a stepper (text) - Variant 1

KPI: % registration submitted; measured by conversion rate (registration / users)

Impact – KPI: Lift in registrations with the form

+5,52% more registrations (across devices)
732 extra registrations per month

Let's sum up!

All the discussed topics resulted in key-advantages that form the foundation of the current and future success of this collaboration:

Closing the gap between business and IT focussed stakeholders who decide together on next priorities (experiments/learnings)
A data-driven customer-centric approach to test and optimize the complete customer journey.
Scaling and rolling out validated learnings by the development teams (cost & time saving).
Transforming the organisation mindset by increasing awareness & importance of experimentation to reach business goals.