Forget Culture: Here’s How to Build an Experimentation Ecosystem

Welcome, Manuel from Effective Experiments!

Manuel da Costa:

Thanks, Divyansh. Sure. Thanks for having me on this VWO webinar. I’m really excited, to talk about this topic. So thank you for that introduction.

D:

Absolutely. Manuel, before we begin, and start with the actual discussion, I want to let attendees know that you can participate in this discussion. Go to Webinar does not allow me to switch on your cameras, but I can switch on your mic So do share your thoughts on the questions being discussed. Send me a request using a chat or a questions box from the control panel, and I’ll be happy to unmute you.

Manuel, take it away.

M:

Thanks again. Cool. Thank you to everyone, attending, this webinar. It’ll be great to get some interaction going as well. So it’s not just me talking to thinner. If you could drop a line in the chat to tell me, you know, who you are, and where you’re attending from, that’d be great to get a sense of, where this webinar is being watched in real-time. And, also, as Divyansh said, if you have any questions, please feel free to ask. I’m happy to answer those live on this webinar as well. Great. So the topic of this webinar is ‘Forget culture and start building an experimentation ecosystem’.

Now over the past few years, we’ve been hearing about this culture of experimentation, and what I’ve seen through our in-depth research of the industry is it’s not really what it’s cut out to be. In this webinar, I’ll be exploring that, and I’ll be showing you some of the challenges that I’ve seen come up over and over again that I’ve seen, upfront, close-up front. Some of them have been discussed with me privately, and some of them which we’ve, helped our clients through as well. A little bit about myself, first. I’m the founder of, a company called Effective Experiments. It’s a platform and a service that is helping companies centralize their experimentation workflow and insights with the aim of scaling and improving visibility. I’ve been in the industry for a long time now. It’s about 11 years now, so that’s quite a long time in a relatively new industry. So I’ve been working closely with enterprise companies on helping them with scaling experimentation programs and adding effective experiments to develop something called experimentation ops that is really driving that change. But enough about me, you’re here to learn about what is this webinar all about.

But firstly, who is this, who is this webinar actually designed for? This is primarily designed for organizations looking to drive innovation through experimentation and optimization. It’s aimed at the company, but I’m fully aware that the people attending this webinar are the practitioners, the CRO specialists, and the experimentation experts. So, if this resonates with you, take this and share it with your senior stakeholders as well, because driving this change, as you will see later on, is, an effort that goes organization-wide. So you have heard your last statement. Right? We need to build a culture of experimentation to improve our success to make sure that we get, buy in to get, to make sure that we get more resources, and so on and so forth. And this talk of a culture of experimentation kept coming up everywhere. It was in blog posts. It was in LinkedIn posts. It was in videos. 

Someone’s launched a course about it. It’s been talked about in conferences over and over and over again. And initially, I thought that this was also something that was really interesting and needed for CRO specialists and experimentation specialists to do. They need to build this culture of experimentation, but the more I dug into it, I kept asking this question, what exactly is a culture of experimentation?

The same answers kept cropping up, and there were answers which I called TBU. They’re true, but they’re ultimately useless, and you’ll see why. These were the types of answers that I’ve got. Everyone should test get them excited, share information, and show them how many wins we have. Right?

And it’s all on the surface level. It sounds good. It sounds, yeah, true. But there was never any depth to it. There was never any, strategy to it. There was never any purpose to it. It was just like these are statements, and you would see people talking about, yeah, we shared, you know, PowerPoints. We shared all our experiments, and everyone was super excited and super pumped about it. The funny thing is, we’ve done a lot of research into the industry where we started interviewing the stakeholder level and C level, within those organizations to understand the awareness of experimentation and this so-called thing called culture.

And it is surprising and shocking at the same time, and I’ll be releasing that research in due course, but the way I look at CROs and experimentation, is best told through the scale of Sisyphus. In Greek mythology, Sisyphus was someone who really annoyed, Zeus and Hades, the god of the underworld, and he was punished for this. And his punishment was he had to take this big boulder, this big rock, and take it to the top of a mountain. but every time you go really close to the top, the rock of the boulder would roll all the way back down. So you have to go all the way back down again and pick it up and again, log it up the hill only for it to fall down yet again, and so on and so forth.

The way I look at how experimentation is growing within organizations is best described with this scenario because I’ve been watching companies. I’ve been watching, CRO programs, and teams have seen them kind of go through this cycle of starting, trying to develop it, and then almost going back to the start because a new person has joined, or, they’re not really focused on the the important things. And the reality of a lot of CRO programs right now and you know, you might relate to this privately or publicly, is that there is a lot of lack of awareness within the wider organization about what is CRO. What is experimentation? What is A/B testing?

There’s a lack of buying, there is a lack of engagement, and there’s low adoption. And when I talk about these things, lack of awareness and lack of buying, lack of engagement, adoption. Sometimes I’ll have a CRO specialist or experimentation specialist come to that. So actually, no, that’s absolutely wrong. We have, everyone is involved in the test thing.

Everyone, you know, comes to the meetings, and we have, stakeholders who, get all our power points. And that’s great. That’s good. But a lot of it is surface level where it’s a box dipping exercise. Like, people were present in the meeting, for, you know, tracking engagement later.

So, the reality of it is quite stark for a lot of CROs right now. And then comes this whole concept of we need to build a culture of experimentation, which is now the CRO is not only doing the test planning and, prioritizing and hypothesizing and, and execution analysis and report writing and, sending out reports and pre-presenting those reports. Now they’ve got to also do building this culture of experimentation. And it’s just another task in the never-ending list of tasks that a CRO has to do or a CRO team has to do. And I don’t think that this should be the case because a lot of CROs are kind of becoming disillusioned in conversations of how they’ve said, you know, it’s so hard to build a culture in this company.

It’s so hard to get people excited. It’s so hard to get people to do to run their own tests. And there’s a fundamental issue with this because when we talk about culture, when we talk about what we can influence culture, you’re putting pressure on the CROs to become these change-makers when really they’re barely trying, getting by with the work they have to do already. So here are some really harsh truths I wanna start with right now, and it might sting, but this is hard this is the nature of the beast right now. 

The first one is CRO sits in a silo in the organization with no authority or influence to make change beyond the silo it operates in.

The second thing is cultural change only happens when there’s a top-down initiative. It could also happen with some grassroots movements, but generally, the bigger changes in the company happen because of a top-down initiative. Companies, claiming to have a culture of experimentation are mostly lying or presenting a false reality. and this is where the disconnect happens. You have people presenting at conferences and talking about this big, culture of experimentation that they have.

But, really, that puts pressure on people attending or watching that because they now have to try and achieve something that is really not achievable on their own. And organizations are still treating CRO or experimentation as an optional, a nice-to-have, like, an add-on, right? And they’re not really clear on the value. It provides, yes, it increases conversion and sometimes it increases revenue, but really what is this CRO? So there’s a lot of lack of awareness still happening. So these are some of the truths, of the industry as it stands right now. And, again, from observing the industry, from an outside view, looking into these organizations, and, teams, these patterns have emerged over and over and over again. And it got me thinking, how do companies not change? Right? If you have, if they’ve struggled to get momentum, why isn’t that happening after a few years?

So what is the difference? What’s different about an ecosystem? So I just wanna make sure that you’re not just thinking, changing the word up here, right, because a few years ago, we had this thing where people said, let’s not call a CRO or let’s call it experimentation. So I’m not doing the same thing over here. I’m not saying, yep, let’s not call it culture. Let’s call it an ecosystem. 

What I am talking about is something different in its approach rather than just bold statements with no, structure underneath them. So an ecosystem is essentially a complex interconnected system where one impacts the other. So if you look at it this way, every person in the organization is important in this. It’s not just, the CROs, being given this hero complex where they have to go, and it’s an us versus them mentality, and they’re going up against the hippos because the hippos are on, you know, on, educated about the wonders of experimentation and the CROs have to go out and build all that. I’m not interested in all that here. I wanna present you with something where structurally it’s sound, and it gives you, ideas on how to improve that. So this ecosystem is essentially a system whereby all the different parts fit together, all the different pieces fit together, and you will see the bigger picture and the impact it makes.

So how do you build an ecosystem? Right. Firstly, obviously, you need to review where you’re at right now. You need to research, you need to see what’s the current scenario as it stands, then you need to engineer the conditions for the change. The next bit is to monitor it and iterate on it.

That sounds actually quite familiar. Right? It is actually your optimization process. If you think about research, ID, high-cost size, tests, analysis, right, same cycle. It’s literally that you’re looking at this through a lens of what you’re already familiar with, which is the optimization flow or the experimentation flow.

But here’s the thing. You cannot do this alone. For any change to happen, there has to be an element of a top-down approach because, no matter what do you what your opinion is, I’ve seen this firsthand where CROs think that they can go bottom up and make that change. It’s a futile effort. At best, this kind of change takes a lot of time and a lot of effort. So it’s something that has to come from the top down. And this is where the building blocks of this experimentation ecosystem are, I won’t show you the building blocks of this experimentation ecosystem. and there are 3 of them. The first one is priority, then comes people, and then comes the process. And I want to talk to you through, some aspects of this.

Obviously, we have, know, we don’t have a lot of time to go through every single aspect, in detail. So I’m gonna give you, you know, a high-level view of some of these with some more in-depth strategies as we go along as well. So what is the priority? Now priority is really, really the most important aspect when it comes to building an experimentation ecosystem. Let me ask you this, right? Does this sound familiar to you?

When a CRO says, I don’t think our stakeholders understand what we’re doing. I don’t think that they see the value of experimentation. I don’t know whether they see that we’re helping the business so much with the testing and the research that we do. So here’s the challenge for a lot of experimentation and CRO teams when CRO or experimentation was brought into the company, chances are, and it may not be the case, but chances are, it was something that was brought on fairly late. It was something that they heard about at our conference, or they heard about it through another competitor doing something similar.

And they said we need to have the CRO effort as well. We need to also do it. So they hired one person to do everything or they hired a small team over time. And that team was kind of sat in the corner doing their own stuff, running their own tests, and presenting the results. So I’m doing good work, but nothing has changed. It was like there was no awareness in Obayan. You still haven’t approved the point. I was talking to a CRO recently. We’ve just joined the company, and we’re talking about the investment into Sierra, and she was like, well, it’s not a lot right now. I’ve basically been being brought in to prove that it works for the company.

And that to me is an interesting take. Like, a company that doesn’t fully bind to this mindset of experiment or CRO, they are hiring people with the hope that maybe they will be able to prove it. Right? So here’s the first challenge – what is the current section and awareness of CRO slash experimentation in your organization? Go and find out. Right? This is an exercise for you. I don’t have the answer for your session, but you can go and find out. And the best way to do it is to grab a coffee slash virtual copies since we’re all remote now, with some senior stakeholders in the business.

If you can get them on, you know, to talk to you. With no agenda, there’s no agenda in this call. There’s no agenda to present a case or, to show anything. It’s just to talk to them and understand a bit more about their initiatives and throughout the course of that conversation, find out what is their awareness of experimentation or CRO. And we’ve done this exercise with a few of our clients as well where they’ve gone out and they’ve asked their stakeholders And the answer is kind of all almost shocked them because some of them said, oh, yeah.

You know, you’re that guy that runs some tests and you throw out some numbers and some improvements, but the awareness, they cannot articulate what exactly they do. Right? So but that’s that gives you the baseline. That gives you an understanding of where your unit, the CRO unit, or you stand within that organization. Because that’s your research. That’s gonna be waived on your improved form. And then you need to do something, which is probably not being done a lot or it’s not being done a lot, which is changing the narrative about experimentation. Right? If chances are, like, when you speak to your stakeholders and or your or senior management, they may perceive CRO as this thing that, you know, you run test and we get some wins and we get more revenue, more money, and all that kind of stuff. but I think it is time to change the narrative about experimentation and CRO.

It’s not just about winning tests, right? It’s not just winning tests because a lot of people shoot themselves in the foot when it comes to this. The first thing is when they go in and they pick up on all the low-hanging fruits. Everyone wants to win. Let me say that. Right? Everyone wants to win. Everyone wants to show that they’re proving value, but an experiment is essentially an exercise in learning. That’s the key thing. So an experiment regardless of whether it wins or loses it’s still able to provide you with something either more revenue or more insights.

But the challenge is if we’ve gone in and kind of hooked up the, the C level and the senior management to this drug of, like, we’re gonna run a test, and we’re gonna print money. They get used to that idea. They get their baseline is only that. And so what happens when the wind’s dryer, what happens when the test results plateau and you’re not able to show the value because the only value you show is in winning tests? So if you’re able to change the narrative about experimentation to providing insights and providing value that way, then regardless of whether you win or lose, you’re still providing that value to your stakeholders.

And that’s what you need to look at because the priority of experimentation has to change first and foremost. The priority for experimentation is really low in a lot of companies right now, even the more mature ones. So unless you change the narrative where you’re going out and talking to stakeholders and understanding the baseline and then using that baseline to shift the narrative to what it really should be, that priority is not gonna change. So that’s something to think about. There’s an exercise for you to do.

So you know what your baseline is in your organization, and then you can use that to shift the narrative a little by little. The next thing is people. And this is really important because, over the, you know, the decade that I’ve been involved in the industry, there’s we’ve gone from, like, CRO being, part of marketing, like, a digital marketing specialist doing that as part of their work to, a CRO themselves doing it. Just one person then small teams. Now it’s growing and growing to a point where product owners are now given the remit of running tests, and it is growing.

But the challenge is experimentation teams even now just comprised mainly of technicians and practitioners. You have analysts, you have UX designers, you have CRO specialists, researchers, the development support. You really have these people who are really good at that crap. Right? This is not a bad thing. It’s you have people who know the technical aspects inside out, but to drive change, to drive this change of attitude towards experimentation, the awareness, it requires a completely different skill set. This is where I’ve found that a lot of CROs struggle because to do that, it has a lot of boring work to be done. There is a lot of boring work that doesn’t yield any tangible results, like, really quickly. And a lot of change, change management, the change itself has a strong focus on people, rather than technology. You can’t just I mean, with the technology, you can easily run a test, right, you can create a test.

You can run a test and so on and so forth, but people, that’s a different ballgame altogether. And this is where it be, like, the CROs themselves, they struggle with that because they don’t, it’s a completely different, skill set for them. Because the current setup is so, technician and practitioner focus, the language spoken about experimentation insights is quite jargon-heavy. Right? So a lot of people you’ll see in our little, bubble will talk you know, stats, p values, and, everything about experimentation that we all understand.

But then we try and talk in the same language to stakeholders, and they don’t understand all of that. So you’re providing all these stats, these numbers. Great. You are doing exactly what you’re reporting on, but that report doesn’t translate to your stakeholders. So the information goes on upstream to them.

Whether it’s in the presentation or whether it’s bundled up with 100 other insights from other departments, these stakeholders glance at these numbers, and then that’s it. But they don’t understand the impact of it immediately. So here’s one exercise for you to do if, you need to change that, whether you’re a changemaker or whether you’re still, a technician or a practitioner, start doing this. Look at the experimentation results that you want to present or send upstream, but rather than send screenshot after screenshot with the results and tables and stuff like that. Start looking at it this way. What is the insight that you gained?

So maybe look, pick out 2, 3 to 5 insights. What are the top insights you’ve on you that you have found from all of these experiments? What does it mean? Right? Translated. Like, what’s the insight? What have you found? What does it mean? And this is the next one. And most importantly, who does it impact?

If you can clearly articulate that, and then show the supporting evidence. What you’re gonna get is much greater engagement from your stakeholders. Because now you’re telling them exactly what that means to them. Yeah. The conversion rates are up great, so what?

But if you can tell them specifically what decisions can be made as a result of those insights and who needs to make those decisions, then you’re in a much better position. You become an adviser and not a reporter. Right? A lot of CROs have this provision where the stakeholders are coming to them and asking them for help on how to improve the product or the service of the business. This is how you get there.

Because now, it’s a two-way street. It’s not just about you firing off power points or talking to a room full of people who are just there because it’s an all-hands meeting. You’re giving them practical advice. From all the data that you found, you’ve summarized it. All that made it easy for them.

And then if they want to learn more, you’ve given them the supporting evidence to engage with. And that’s how you drive awareness and engagement step by step on that one. But, really, as a whole, the experimentation setup needs to change as well. And right now, we’re quite a practitioner skewed, which means that a lot of technicians, a lot of practitioners, as I said earlier, but we need to have bolt on top of that. The 2 main roles that every experimentation team, or center of excellence, or center of enablement needs are orchestrating, are focused on managing teams with a view of improving quality and onboarding members, monitoring, and coaching. That’s really important, and I’ll show you why in a second. Ambassadors are people-oriented. They’re extroverted. They’re able to build relationships with the C-suite. They talk their language.

They can transfer insights and knowledge really easily. So the orchestrators and the ambassadors support the practitioner in taking the information and, translating it to the stakeholders. But none of this is possible without the support of what I call an executive sponsor. An executive sponsor is not just someone that you know, gives you a budget, or gives you, resources. But it’s the person that has your back because as you will see, in the next slide, when CROs try to get other teams or other people to coordinate and collaborate with them, they don’t get that.

And having an executive sponsor who oversees those people is gonna become really important because they can set the remit they can set the rules and the guidelines for everyone to form. And now we go into the process, which is a nice segue because right now, it’s almost too easy, to launch a test. Right? But we haven’t paused to think about the process. We aren’t opposed to thinking about the governance and guardrails to ensure quality.

I review and do quite a few audits of experimentation programs. Sometimes there’s an external, consultant. And one of the things I find is how experiments are launched and they are, run without any process appearance sometimes corners are cut. Sometimes the reporting is not sound. There are a lot of, quality issues.

So when someone says, oh, we’ve run 100 tests this quarter. Great. But I’m more interested in what is the quality of those 100 experiments. Because right now, there’s a hidden epidemic that’s growing in experimentation. A lot of companies now want product owners to test. They want other teams to test, and you have a CRO or CRO team trying to support them and give them access to tools, give them workshops, and show them how it’s done. And then they leave them to their own devices. But as these teams start adopting experimentation, there’s a lot of uncertainty about the quality of their work and the processes that they do. So everyone’s doing like, different things. And it’s really hard to figure out whether the experiments being run have any quality or not.

So the need for speed of getting everyone to test, and everyone to launch their own experiments is prioritized over a slow strategic approach. Basic if you think of how people are onboarded these days and I’ve researched this as well where teams are onboarded, and they are given basic training where they’re shown here’s how you run your, how we run tests. And here’s access to their tools that you need to run right now, and then they left their own devices. That doesn’t cut it because what happens then is you get problems in this process, which is process adherence issues.

You get teams only reporting winning tests and hiding others. Right? The random ID has been tested without proper prioritization. There’s this, I think called Harking where a hypothesizing after the results are known. Quality analysis and reporting new involvement from stakeholders.

But all of this is happening because there isn’t any oversight on all of this. So with all the teams kind of spread out doing their own thing. It’s hard to get oversight, and the CROs themselves are struggling because they don’t have enough bandwidth or time to properly manage this. They lack the authority to enforce any change, and they lack the expertise, as I’ve mentioned earlier, in that change management process, and stakeholders are not involved. So you know, CROs will be telling a product team.

Can you start running some tests and can you, report on it, and then they find that you know, the product teams are not even bothering with it, or they’re not even reporting on it correctly? And that’s because of the lack of involvement of stakeholders. So the way to look at growing the experimentation capability, in a company, it’s not like launching a rocket. Right? It’s more like teaching people how to ride a bike.

First, you have stabilizers where they can safely ride it and you can watch them, and then you’re helping them, and then you let them go because you’re safe in the knowledge that they know how to ride the bike and they’re not gonna fall. Change is hard. Right. Even with an ecosystem of experimentation, it’s not gonna go from nothing to some to everything in one go. You are going to go, you are going to get ups and downs.

And it’ll be, a case of two steps forward, three steps back. But this is a really interesting, concept Kubler Ross change curve, but I wanna show you over here. And any change that happens, whether it’s in life, whether it’s in business, whether it’s in your experimentation program, it’s always the case, like this. People will always be in denial about it, and they’ll be frustrated because there’s a change, and they’ll be sad about it. and then slowly they’ll accept it, and they will integrate it into their workflow.

So you have to accept that as you start making the change whether that is presenting the news reality to your stakeholders, whether it is about getting teams to work together, collaborate together, there will be these ups and downs, and you have to work through this to get to the point where it is integrated into the business as usual. where do tools fit into all of this? And when I, first presented this, people, priority people, and, processes elements. People say the tools are also important. They are, but not as important as you think. Right?

Tools are there to facilitate and help. They can amplify a good process, but they can also amplify a really bad process. Now, that’s not to say that if you use the wrong tool or a bad tool, it’s okay. You still need to use a tool that’s fit for purpose to help with that. But tools on a whole are kind of secondary to those 3 P’s that I showed earlier, the priority, the people, and the processes.

Those are the most important. So the last bit of this, conversation now is how do you put it all together? How do you, I’ve talked about people, the priority I’ve talked about, people and processes. How are we gonna piece this together in reality, at least? Right? So this only works if there is a top-down initiative. So the way this think of it as a sandwich. You got top-down initiative facilitated and supported by the teams from the bottom up. And that’s when this change shift will work. So first things first. Focus on senior management. Focus on that perception of experimentation and value because that’ll give you an understanding of where their mind is at when it comes to experimentation. Then you know how to ship that attitude, ship that mindset by showing them the possibilities of it. This will help establish the need to build and orchestrate a sustainable experimentation program. You get all the support that you need, but more importantly, the remit. Right?

If a senior manager says, things have to be done in a certain way, it will get done. So you want that level of support and not so much buying from, like, in terms of monetary, obviously, you want that as well, but you want someone that has your back and is setting the remit. This will get you all the key executive sponsors and the ability to set that remit and the guardrails for your processes. 2nd, look at the experimentation setup. As I told you, if your experimentation team is full of practitioners who are begrudgingly the presentations and report writing and trying to share and keep everyone informed, higher orchestrators, upscale people to become orchestrators to focus on the quality of experimentation.

Where do you find them? Well, project managers are a good one. They are they’re fine in terms of going through the mundane work of checking people’s work, reporting on it, and so on and so forth, training, monitoring, etc. Get your ambassadors in place. The plans are most people-oriented and will talk in the language of the stakeholders, refining processes and creating those guardrails to ensure that everyone follows the same process. You cannot grow an experimentation program without having strong foundations. the strong foundations are, as a result of standardization to ensure that everyone works in a certain way. So your output is reliable and trustworthy and that will improve the confidence of, your organization towards experimentation. So let’s look at it in one example over here.

And in this, we have a center of excellence. I know typically, as at this one time, the center of excellence teams are also running their tests and also kind of being almost an internal agency. The way I look at a center of excellence team is they’re more of an enablement team. They’re there as orchestrators and ambassadors to support, train, share insights, and govern the different teams that are running tests. Now, obviously, if you’re just one person, it may be different, but this example is for an organization that is growing experimentation where different teams that are responsible for running tests your centric, enablement is overseeing those tests.

But if you look at each side of those columns, you have those executive sponsors. In this example, fictional example, you have Team A and Team B that directly report to the executive sponsor above them, and Team C and Team D that report to the executive sponsor above that the center of excellence team is in communication and has the support of both executive sponsors which means that team C and team D and A and B are going to follow the processes as has been set by the center enable, the center of enablement, and the stakeholder awareness sits above all of them because the awareness of experimentation is value and what brings to the businesses has been established. The executive sponsors ensure that the teams follow the processes set by the orchestrators and the ambassadors, and it runs like a machine after that. 

This is again one way of doing it. In other organizations, it might depend on every organization as they have their own nuances. So, it may vary from, company to company, just saying that as a caveat. And this is just a small sample of, the experimentation ops framework and strategies that we’ve developed here at effective experiments. If you’re interested in learning more, you can check out our website or connect with me on LinkedIn as well, I post a lot about improving experimentation programs but, this is the end of my presentation. I hope I provided you with some value and if you have any questions, I’ll be on standby. and I’ll call Divyansh now if he can take over and facilitate that. Thank you, everyone.

D:

Hey, everyone. I’m back from my attic, but, yeah, thank you so much, Manuel for such a great presentation. A few of the insights we can take back and implement ourselves, because, that is what, like, the bottlenecks that you shared can be seen and be felt in a lot of enterprise organizations across because building teams around experimentation is a big, big challenge but, a systemic process, specifically a 3 step process that you shared would be a great way to, you know, kind of navigate the buy-in that experimenters, a lot of the times struggle with. Guys, I’ll request all of the attendees who ask questions, to clarify any doubt, in case, you’d want to you’d want to go ahead. I’m seeing a few questions. Lewis, says, Hi, man.

And then okay. Timo, can I unmute you? You can go ahead and ask the questions yourself. So that’d be a great interactive way to do so. I’ll just unmute you, Timo. Let me know if there’s an issue. Timo, you can go ahead and ask the question yourself to Manuel.

M:

I can’t hear it.

D:

Yeah. Probably. It isn’t available. but I’ll read out the question. Would you say that experimentation can be applied across all departments in an organization? How to find out if there is actually a need? So Timo from Hamburg wants an answer to this question.

M:

This is a really good question. And I think, earlier I mentioned at this point that, you know, Sierra went through a phase where, like, we were thinking about redesigning it. And we should rename it as experimentation. My current view about experimentation is it’s, a discipline that is regardless of the channel. It’s not I know VWO does it for web and apps and stuff like that, but experimentation can happen in multiple different teams, departments, and even business experiments.

Right? There are different types of ways of experimenting, to find answers, to explore, and to optimize. So chances are if you have teams. If you look at digital, it is the easiest, to get going. Right? So you have your web test, you have your email test, PTC, social, SEO, and so on and so forth. But what about offline? If you have offline, stores, I mean, you know, sometimes you can optimize how people move through the store or find certain things. I know supermarkets do that a lot where they’re testing product positioning to see, footfall and, and, purchasing. Yes.

You know, experimentation should be happening across the business because that drives that constant need to learn and constant need to optimize. So the question about how do you find the need for that? Well, you look at the opportunities that are there. You can position the opportune fees that might be relevant to those departments or those teams. If we always advocate for our senior vice president or vice president of experimentation sitting right on top of the overseas experimentation across all the different departments and be channel agnostic.

So they will get information about the web. They’ll get information about email and so on and so forth. Because, ultimately, to orchestrate and unify all those teams together so they can learn from one another and drive the insights forward. Right? So the best way to go by finding the need is really building that need or seeing what opportunities they may not be thinking of and educating them about that.

D:

Definitely. Thank you for answering that question, Manuel. There’s one more, question from Felipe. Felipe, should I go ahead and unmute you as well? But I tried doing it for Timo.

It seems it did not work out. But, yeah, what’s the harm? Let me give it a shot. Okay. Philippe, if you’d want to ask your question to Manuel directly, am I audible, and can you go ahead? 

I think that’s some problem with the plan. I just read out I just read out the question. Sure. I think a problem happens when people confuse CRO with experimentation. CRO was born to find small or medium discoveries to increase revenue and adjust knobs but experimentation has an even greater purpose of detecting opportunities and risks and qualifying their impact. Definitely.

M:

Yeah. Absolutely. That’s exactly the point I was trying to make earlier. Like, we conflate CRO and experimentation is the same thing, but the way I look at CRO is if it’s a subset of the wider experimentation spectrum where you can either, you know, learn you want to learn something new. So you’re experimenting to learn. You want to explore. You can use experimentation techniques as a way of exploring, what you want to optimize. You’ve got A/B testing as a methodology for optimizing and learning. Right? So that’s the whole point of it.

And there is a wider thing about experimentation that businesses need to be doing. This is why that culture of experimentation doesn’t sit right with me because there’s such a bigger potential out there. And for a lot of CROs, we are still, you know, you might still be doing web A/B testing. That’s your remit. So how do you build that mindset when you’re only working on the optimization part of it?

So there’s still a long way to go, but that’s why in my opinion, it starts from the top down. That initiative starts from the top down, but the awareness might not be there. If people work towards their awareness, then the initiative starts, and that’s where the change and all the shift will start happening.

D:

And I think we all can agree that it is a seismic shift to, take place, actually, to change the mindset around experimentation, in an organization. It’s a seismic shift, right, for, we’ll be sharing this knowledge base, with, all our attendees. you’ll have this presentation. You’ll have this recording. You can always revisit and rewatch and connect with Manuel for any blockers or bottlenecks around experimentation that you might be facing at your place. Right? And, I think he’ll be as hap helpful as he’s been today. So thank you so much, was an incredible session, and, in short, we keep doing most sessions going forward. Thank you all.

M:

Thanks for attending, and see you all soon. Bye bye.