If you like what you heard on this week’s episode, be sure to give us a Review, Like, or Share to help support us!
Did you know that mastering Marketing Experiments doesn’t have to be complex?
To show you, we’ve interviewed three Marketing Experiments experts to give you their opinion and viewpoint on how to be successful with Marketing Experiments.
From scaling to fine tuning, we hope you enjoy this deep dive.
Subscribe to BoostSauce to learn even more marketing and sales knowledge, faster…
When it comes to Marketing Experiments, there are only a few people we turn to for amazing advice that works across the board for different goals.
Whether you’re in SaaS, eCommerce, or lead gen, you’ll be excited to learn that the recipes these experts will share will all help you hit your goals faster.
In Order Of The Guests Below:
Neil Patel – CEO @ Neil Patel Digital
Neil Patel is the co-founder of Crazy Egg, Hello Bar and KISSmetrics. He helps companies like Amazon, NBC, GM, HP and Viacom grow their revenue. The Wall Street Journal calls him a top influencer on the web, Forbes says he is one of the top 10 online marketers, and Entrepreneur Magazine says he created one of the 100 most brilliant companies in the world. He was recognized as a top 100 entrepreneur under the age of 30 by President Obama and one of the top 100 entrepreneurs under the age of 35 by the United Nations. Neil has also been awarded Congressional Recognition from the United States House of Representatives
Nuggets Dropped x 29
“People want choices now because they’re more sensitive about data”
Eric Siu – CEO of Single Grain & Co Founder of ClickFlow
Eric Siu is the CEO of digital marketing agency Single Grain and Co-Founder of ClickFlow. Apart from running his multiple businesses, Eric hosts two podcasts: Marketing School with Neil Patel and Leveling Up, an entrepreneurial podcast where he dissects growth levers that help businesses scale.
Nuggets Dropped x 34
“Using customer data platforms can be an ROI-effective experiment”
Pedro Clivati – Head of Growth @ GrowthHackers
Pedro Clivati is the Head of Growth at GrowthHackers.com – where he’s running the experimentation process around the biggest online growth community in the world and its growth management software. Previously, he was working as head of acquisition for Contentools, mentored startups on Founders Club and participated in #500Startups, #GrowthX, #Launch.
Nuggets Dropped x 49
“Companies should run experiments for 2 weeks max to stay agile”
Marketing Experiments Mastery With Neil Patel
Johnathan: All right everybody! Super excited to have you here. I got a big dog in the house, from a marketing perspective, somebody who I looked up to for a very long time, Neil Patel. Thanks for having, or thanks for being here!
Neil: Oh no, thanks for having me! So appreciate it. It’s my pleasure.
Johnathan: Awesome, awesome. Now you guys, you and Eric Siu, have a podcast too that I’ve been listening to for a while. What’s it called again?
Neil: Marketing School.
Johnathan: Marketing School, yeah. You guys drop a ton of knowledge. A lot of the tools that you guys talk about there too, it’s doing really well. You guys been at it for a while.
Neil: Yes, doing well. We’ve been on it for a few years now.
Johnathan: Yeah.
Neil: So, just continue cranking away.
Johnathan: Awesome. Well yeah. Anybody who’s listening to this podcast, check their podcast out. Lots of great value.
Today Neil, you and I are gonna be talking about your marketing experiments, what you’ve learned from in the past, what you are doing today, and also some of like, the unexpected outcomes of that. So, like where’s the best place to start?
Neil: Wherever you want to end up starting. I’ve done so much, so–
Johnathan: All right.
Neil: Up to you where you wanna start.
Johnathan: Well let’s start in the past man. Let’s look back at some of the things that were really eye opening to you that you can remember, and like how did you run that experiment? What were the outcomes? And stuff like that. Anything that comes to mind right now?
Neil: Yeah, so there’s a lot of simple ones. I’ll start with the basics then go from there.
The simplest one that most people don’t end up doing, whatever lead forms you have, whatever check out pages you have, I don’t care if it’s B2C or B2B, just turn it into two steps. I always see like a nine to 11 percent increase in conversion. 
Johnathan: Okay, I like that.
Neil: Affiliate works. And if you’re Amazon, you don’t need to do it, but for the rest of the world who’s and we don’t have as strong of a bribe, two step works out really well.
Johnathan: Yeah.
Neil: Another one that’s done extremely well for me is instead of making people register, they already have a Facebook email or Google email, just do a sign-in authentication with like Google or Facebook.
I’m usually seeing a 30 to 40 percent lift. I used to, before, see a 90 plus percent lift.
Johnathan: Wow.
Neil: But not many people are doing it.
Johnathan: Yeah.
Neil: And instead of saying “Sign up”, change button to Log in, you’ll see usually another single digit increase from changing the button to Log in.
Johnathan: Nice . I remember hearing about that authentication thing that you did too.
I liked you talked about the two step. We call that the breadcrumb technique ourselves, where you know we ask for personal information on the last step and then everything that’s easier to fill out on the first step.
Is that the same way you go about it? Or do you have any idea of the order of fields and forms?
Neil: I do. So, the key with form fields is, the easiest information first, the most customized personal like phone number, last, And you’ll see the most surge that way. If you ask phone number first you’ll get screwed.
For example, my form fields ask for like name or URL first, even before I ask for email.
Johnathan: Good. I like that. That’s super smart. What from your own stuff that you guys are doing, cause you now have an agency, you’re doing a lot of things there too, obviously you’re massive on the content side.
I know that you bought like a keyword tool a while back too. I don’t know if we can classify that as an experiment, but I think it was a brilliant idea. How did you go about thinking of that?
Neil: Yeah, I bought a keyword tool called Ubersuggest. And what I did was, as a marketer, the SEOs, even me, we try to build all this content and bump up these links and I’m like, you know what?
I know you’re not supposed to buy links, but just think about how much energy you put into building links, how much time you put into writing content, how much energy you put into getting those rankings, and just see how much it would cost to buy a site that already has all of it, and they don’t generate any revenue, and I bet you it’s cheaper. 
Johnathan: I like that. I’m imagining that there’s not a lot of those sites out there but it sounds like you got a good ROI from it so far.
Neil: Well, technically not yet. But hopefully in another year or two.
Johnathan: Long term, cool. How do you… so I don’t know, like what does your day to day involve, like what does it look like for you? Are you like the CMO? Or what do you spend most of your time on today?
Neil: Generating leads and helping customers with strategy and making sure our customers are growing.
Johnathan: Okay cool. Now, how do you prioritize when it comes to generating leads for yourself?
Neil: So the way I prioritize is, I used to believe in, and I still do, I used to prioritize through marketing and CRO.
Johnathan: Okay.
Neil: And what I found is, tools generate more traffic than content marketing.
Johnathan: Okay.
Neil: So I spent more time building up tools, than generating content, cause with content marketing, to generate a lot of traffic over years, what most people don’t know is yes you continually update and rewrite your content.
If you look at your content, you can use that like revive animalz, to see what of your contents decay. And what you wanna do is rewrite and edit the ones that are dying down, but if you look at how much time you spend on that, it’s cheaper to just build a tool.
Johnathan: Yeah. I completely agree. We’ve started updating our old blog posts, things that have been like three plus years old. And we’re seeing great results from it.
From tools when it comes to getting leads at the end of the day, Those free tools, like that’s the main goal. Right? Like I guess with everything content, it’s the same thing. At the end of the day if its not producing a business and back for you, you’re not really keen on doing more of it.
How do you decide on which tools to create? Let’s say that you’re, if you wanna give me an example of an industry.
Neil: Go to… What is it called? CodeCanyon?
Johnathan: Okay.
Neil: There’s a tool on. And for like 10 to 20 dollars, you can buy a tool. And you’ll find that it can generate a lot of traffic.
Now here’s the trick. You go and you look at whatever search is popular in the tool space within your region. And you can use, you know, the similar webs of the world to see are you driving traffic. Go find similar tools from there within your competitors, then you’re off into the races!
Johnathan: This is freaking amazing. What are you working on today? What’s the biggest things in front of you?
Neil: Ubersuggest is one of the biggest things. My ad agency, Neil Patel Digital–
Johnathan: Yeah.
Neil: We spend a ton of time on that. But in general it’s just helping people grow, and finding out more creative ways, so–
Johnathan: Yeah. For sure. Have there been experiments that you run in the past or currently when you’re like this is gonna take off, this is gonna be awesome, and it was a complete dud? Did that ever happen?
Neil: Many times, so I’ll give you an example what this is. I assumed that if you go and update all your content, even if it decays, that you will get more traffic. And that was the wrong approach.
What was easier for me to do is, yeah sure, you want to look at content that has decayed, but more importantly you wanna look at, in Google Search Console, or Webmaster Tools, whatever you wanna end up calling it, the content that has a lot of impressions, low click-through rate. You ideally wanna look for under four percent click-through rate.
And then you wanna look at the keywords that have under a five percent click-through rate, take those keywords, cause you can click on the page right, your overall page is under four percent.
You wanna look for the keywords that have under five percent once you click into the page level. See which ones that aren’t mentioned in the article, add them. That’s the first thing that’s a quick boost. Second thing is, add those keywords to the title tag by the description.
Third thing that you wanna end up doing is, Googling those terms, see what everyone in the top 10 includes, and make sure that your content is more thorough than them.
Johnathan: Yeah.
Neil: And that’s the easier way to gain.
Johnathan: Gain! So, like… We might have to start doing that. We have seen increases in traffic from just updating posts, but not gone to the detail like actually solving the problem, like you just mentioned with the three steps.
What, anything else that comes to mind for you in regards to failed marketing experiments?
Neil: Uh, so one thing that I’ve learned from a failed marketing experiment is, time changes what works. So for example, remember how I said that before I was getting, I used to get like 90 plus percent increases in conversion.
Johnathan: Yeah.
Neil: From using Google, etc. Nowadays, I receive the data, not everyone wants to do that.
Johnathan: Right.
Neil: So I had one business that was the only option. We started adding in an email, or like putting your name, email, password, or sign in with Google.
Johnathan: Yeah.
Neil: We saw increase in conversion. People want choices now because they’re more sensitive about their data.
Johnathan: Yeah, that makes sense. Anything else, you mentioned CodeCanyon, and figuring out how to like, get tools on the lower end cost scale.
What do you think, in regards to things that are gonna work in the future, is it still gonna be tools? Is it gonna be other things that you see that are gonna be valuable? What are your thoughts?
Neil: Can you repeat the question?
Johnathan: Yeah. I was gonna say right now you’re mentioning that going to CodeCanyon, potentially buying a tool that you can re-skin for your own industry, and your own value that you can bring.
In the future, obviously tools will still be valuable, but is there anything else that you think that’s happening that’ll be more valuable from a lead gen perspective for you?
Neil: From a lead generation, video’s gonna be huge. YouTube, Instagram, LinkedIn video.
Johnathan: Yeah.
Neil: But here’s the thing. Just having a video, you’re not gonna get leads or sales from it. Within your videos you’ve gotta a) talk about your products and service and where people can go to sign up works really well.
And b) you can have a call to action at the end of the video that’s clickable, where they can go back to whatever landing page you want click the lead or purchase.
Johnathan: Yeah.
Neil: That works really well. Another tool that I’ve seen is, you do a video on Wistia, and you show ’em the first 30 seconds to a minute, if they wanna watch more they gotta put in their information so you can collect a lead from that.
Johnathan: Yeah.
Neil: That’s a quick hack that works pretty well. And then the other thing that I love doing is live video crushes it! From not just an engagement standpoint, but if you tell people to spend somewhere, they tend to listen.
And they tend to … He does extremely well just from live video.
Johnathan: Yeah. That makes sense. Cool. Well anything else you wanna cap off with? Otherwise I have no more questions.
Neil: That’s it on my end.
Johnathan: Awesome man. Well, thank you so much for your time. We’ll bring you back another time and share some knowledge. Appreciate you.
Neil: Thanks for having me.
Johnathan: All right. Thanks Neil, bye.
Marketing Experiments Mastery With Eric Siu
Johnathan: All right everyone, I’m pumped. I’ve been respecting this guy from afar. It’s actually rare that we get to meet in person, but we have met in person before. We have Eric Siu, who is the owner of Single Grain and the CEO of ClickFlow. How are you, Eric?
Eric: I am doing well man, great to connect again.
Johnathan: Yeah, yeah, thank you and for people listening, he was saying that it was the best Caucasian pronunciation he’s had of his last name, so I will take that and start off with the first nugget for the show.
So we’re gonna be talking about marketing experiments, and we also had Neil on the show in regards to his point of view, and I know you guys work pretty closely. So it’s a really, really fun dynamic to see what you put in the bucket of marketing experience and what he does, too.
So, when your mindset, you have your own podcast, Marketing School, which is amazing. People listening here need to go subscribe because I remember listening awhile back and you guy dropped tons of tool knowledge, I remember. Maybe it was just a couple of episodes I listened to, but it was really, really rad.
So I couldn’t think of anybody better than you to have for this segment in our episode. How do you go about your own marketing experiments and also those for your clients?
Eric: Yeah, super appreciate that. So I guess I could speak to the experiment that we’ve been running for ourselves, and what’s interesting is Neil and I, we just had an event. I hate to call it a mastermind, but we had this event where Neil–
Johnathan: Do you say that because that’s the thing that marketers wanna avoid?
Eric: Dude, it’s so internet markety, man
Johnathan: It totally is, I agree.
Eric: I try to stay away from that. Yeah, yeah.
Johnathan: All right, we’ll call it something else.
Eric: We call it the Growth Accelerator. But anyway, Neil spoke, he went first, and he dropped 21 different tactics, you know, loaded it up. I went the day after and I loaded up with different tactics.
It’s very complementary and I think, one of the experiments that’s been working really well for us right now, is the use of customer data platforms. Have you had anybody talk about that on this podcast?
Johnathan: No, I don’t even know where you’re going with this. I’m excited.
Eric: Uh-oh. Okay, great. Dude, you can totally use this. Anybody that’s producing content right now, and then getting organic traffic, you have a lot of unknown IP addresses that are landing on your website.
So a customer data platform, what we use is hull.io, that’s h-u-l-l.io we use that, thank you for that, that’s my first nugget. But we use that for, it’s our single source of truth, right?
Johnathan: Right.
Eric: It combines data from intercom, from hubspot, it combines with our sales enablement tool, which is outreach. It combines all the tools, right?
Johnathan: Okay.
Eric: Basically, it allows you to weaponize your data.
So, if I’m Coca Cola and I visit your website and I have visited, let’s say, your services page because you guys have your PPG services as an example. If I can uncover four people from the Coca Cola team and because I know they visited that page I can drop them into a sequence inside of outreach, which is sales enablement, and it happens automatically, right?
Johnathan: Yeah.
Eric: And so, from there like it takes away the new feed to have to constantly have a sales person on it. Right, it’s basically automated sales. That’s one aspect that you can do with a customer data platform.
Johnathan: Wait, wait, wait. Let me…
Eric: Do you want me to keep going? Or do you want to follow up with me?
Johnathan: Well, lets slow down a bit because I just like got super tingly and excited. And it’s like, I don’t know if you get this feeling, but when you listen to really cool music, you’ll get like goosebumps. You literally just get goosebumps…
Eric: I have a nice voice.
Johnathan: Like from what you just said because let me unpack that slowly really quick for everybody listening. So, you’re basically saying that outreach.io is your sales name on that platform that actually has both like email sequences in it and also customer data.
Like, so if it is Coca Cola, you basically gave it filter to say people from the marketing team at Coca Cola, because you’re guessing, right? You don’t know the person that actually went on your site. You know the company that went on your site. Is that correct?
Eric: That’s correct. And what I do to make that a little easier, the customer data platform, hull.io or anything, is there’s a bunch of different tools. But, you could set filters. You can say I’m only going to reach out, this roll will only be triggered if two unique IP addresses from Coca Cola visit our website.
So you’re not gonna just do it for every single person. Does that make sense?
Johnathan: It does. And so can you set filters around like, the amount of visits and within a certain time frame? Or like them hitting a certain page because that shows higher intent? Like, those can be triggers as well?
Eric: Yeah, so you can set up different segments. So for example, we rank for a lot of different terms, right?
So, let’s say you guys rank for, I don’t know, podcast advertising or something and you guys sell podcast advertising. Very high intent, makes sense, right?
And you could just say hey, Jonathan. I noticed you visit our page on podcast advertising. Would you like to have a chat with us? Because we actually have this report on podcast advertising trends. It’ll take five to 10 minutes and we just wanna show it to you. Is that cool?
But yeah, something like that.
Johnathan: All right, wow. So, I’m just thinking. So, we use Clearbit, we use Proof, we use Segment, and I’m thinking there’s a lot of similarities. But, we still do parts of this manually.
And so, frick, I’m pumped. I just wanna end this conversation right now and just go start working on this. Let’s go to the next thing. This is exciting.
Eric: Well, there’s more you can do with it. Do you wanna make it easier?
Johnathan: Yeah, yeah.
Eric: Do you want me to give you more that you can do with it?
Johnathan: Yes.
Eric: Okay, so Segment is something, they’re kinda going down that route, too. So you might wanna talk to their sales team to see if you can go a little deeper. But yeah, I mean, you know they’ll combine with Clearbit and all these other integrations.
So, what you can also do is, again, let’s go back to the Coca Cola example. You do not know who visited. But, you uncover four VP of marketing’s from their team. You can actually do a form of dynamic retargeting based on those IP addresses, right?
You can say hey like, you know, we help companies like Coca Cola. Obviously, this has to be, you know, above water with ad policies. But, anyway, you can retarget base and make it, you know, customized towards them and you can do that automatically, right?
So, you can, you know, upload automatically to Facebook. That’s one additional thing that you can do with OCDPs. And the other thing, too, you talked about Proof and you know Lovedave and…
Johnathan: Rad people over there.
Eric: Love what they’re doing with personalization. But also, guess what? Hull.io lets you do personalization as well.
Johnathan: Okay.
Eric: So, yeah. That’s another thing. There’s just a lot of flexibility you get because you know, there’s, it’s like looking at a Facebook feed but it’s looking at all these people that are off the entry list.
So, lets say someone opts into your list. You could actually look at every single action they perform. Like, they engage with you on intercom. Then they did something on your sales page. Then they visited another page, opt in for a resource.
It’s all combined into one source. So, you have like a timeline you can look at without having to log into disparate sources.
Johnathan: That’s amazing. That’s so cool. So yeah, anybody listening that does anything B2B related you just got an early Christmas present. Congratulations. That’s amazing.
What, I can’t even, what other things do you have in that amazing mind of yours?
Eric: Well, so, thank you for that. But let’s see what else we got going on.
So, what I think and I’m glad you guys are doing a podcast. Because, hull.io, by the way, we signed almost [Cuts Out] a larger deal just because that initiation happened through our sales enablement tool to get us a podcast advertising deal.
But what I recommend for people is, you know, and you guys might be doing this already, but I think most people that are producing content, if you’re really good at video, you automatically have audio files. So, like this podcast can probably be chopped into five separate audio files.
Johnathan: Yeah.
Eric: And then from there, they can be published into like, you know, a LinkedIn carousel or Instagram carousel, like a bunch of stories, right?
Johnathan: Yeah.
Eric: So, you’re kind of , the whole thing now is that like, you know, people used to say in marketing oh, it’s the rule of seven. You have to see something seven times. I actually think it’s probably like the rule of 20 or 30, now. People need to see you a lot more. Get what I mean?
Johnathan: It’s crowded, yeah. That makes sense. Okay.
Eric: Yeah, so we’ve been producing a lot more content. Like, I’ve been just kinda testing this out the last couple weeks where I’m like writing a tweet storm everyday and I just kinda repurpose it to LinkedIn. And then the Facebook stories, Instagram stories, whatever.
And I’m getting a lot more engagement and that has actually led to a call with a really well respected university later this week. And my team’s gonna take that call. But it literally happened because the guy DMs me and said, we just started engaging a little bit.
So, I think most people don’t take advantage of their social platforms because they don’t post natively to those platforms. They don’t make the most of the esteemed piece of content that they’re producing.
So, I think you gotta kind of, you know, we call it content sprouting, where you take one piece and you make it into like, you know, 30 or 40 pieces.
Johnathan: Yeah, yeah. I think I saw like the Gary Vaynerchuk like contra-distribution model. Is that something similar? Have you seen that?
Eric: Yeah, I’ve seen it. I mean, one of the guys that used to work with me, work for me as a copywriter, he now works for Gary V. He’s always talking, I’m still texting with him all the time. So, I get the lowdown on kinda the inner workings there. But it is very much that.
Johnathan: That’s so cool. Yeah, no and I think, I can’t even imagine. So like you have all these, let’s say I call them like octopus arms, like content. And it’s valuable content. Like you’re coming out with really, really cool stuff.
And then you also have the hull.io example of like you had this like safety net that’s catching a lot of things because it’s just such a perfect machine that you guys are building right now. It’s incredible.
Eric: Thanks for that.
Johnathan: Cool, cool. How do you guys go about deciding and how do you prioritize these experiments? Is it more so, oh this is a new tool that allows us to do something that we weren’t able to do before now we’re gonna try it out?
Or are there like certain things that you’re trying to crack the code on? Because it sounds like you guys are pretty solid and I can see that from the outside, too. You guys are. But what’s like your biggest pain point from a marketing and maybe acquisition perspective right now?
Eric: I think there’s always, so I guess in terms of how we run the experiment, I mean, we usually use the ICE framework: impact, confidence, I think it’s ease maybe.
Johnathan: Yeah, it is, it is.
Eric: I forgot what the E was.
Johnathan: I only know that because I had somebody else tell me that on the other show. So, now I’m an expert.
Eric: Nice, well, so there’s that, right? And then intercom came out with something called RICE. I think it was reach and then it multiplies the other ones. And it started to get all complicated.
So, what we basically do now is we just use like, we follow a book called The Entrepreneur’s Operating System. The book’s called Traction. And we have a marketing meeting every single week on it.
We have a leadership meeting, 90 minute meeting. 90 minute marketing meeting. And you know, we brainstorm topics. We brainstorm kind of experiments. And then we kind of just choose from there. That’s actually made life a lot easier and less stressful.
Johnathan: Do you guys have Rocks as well?
Eric: We do, we have Rocks. But we’ve kind of made it into a hybrid. So we’ve taken the OKR model from Measure What Matters. And we’ve kind of you know, mixed it up a little bit.
Johnathan: Nice, yeah. Do what works for you. That’s awesome.
We had Traction then we had What The Heck Is EOS? Which is the entrepreneurial operating system. Like there’s only really two books that we follow at this company. It is the What The Heck Is EOS? Which is like the simplified version of Traction.
Eric: The truncated version, yeah.
Johnathan: Yeah, exactly. And then we have The One Minute Manager which is the other way that we kinda like operate based of servant leadership.
Eric: I love it, dude.
Johnathan: That’s awesome. That’s so funny that you guys do that, too.
Eric: Yeah, required reading.
Johnathan: Yeah, exactly. Cool, so you guys do this brainstorm. In regards to that mastermind thing that you just had, which we won’t call it mastermind, what were some of the things that you guys, that you shared there that you’re okay sharing with us, maybe?
Eric: Yeah, so the other one is this, I mean, this all comes from my talk but, when I think about, going to that mastermind, the ticket was $25,000 to a 10, right? So it’s a very high ticket and it’s very high touch.
So, what was going on was people would put in their phone number and we would automatically text them, right? So you can use like Zapier or something. We use basically, once they put in their phone number, we’ll hook it in using Zapier to Twilio. And it will automatically send them a text message saying hey, yes another one. But we have…
Johnathan: By the way, you’ve gotten way more clucks that are silent that you’ve just been dropping in my head. So I haven’t been like interrupting your speech pattern. So, just know that you’re like really further ahead than like the seven you’ve gotten so far.
Eric: Yes, all right. Well, I appreciate that. I only came on this show for the clucks.
But we, basically what happens is we text them automatically, right? So we’re like hey, do you have any questions around the mastermind? And I’m actually crowdfunding for Clickflow right now, which is my software. And we raised a couple hundred grand already. And same deal, right?
Basically, they will put in their phone number, their email address, whatever. And then they’ll get a text message saying hey, do you have any questions regarding the investment? Do you have any questions regarding, you know, whatever? Whatever it is, right?
And the response rate, not the open rate, the response rate is 60%. So what we do there is we basically sync it up with, we use Front, that’s kinda the email tool we use. And we can message people back and forth as if we’re texting them through Front.
Because it hooks in with the Twilio API. And you know, it’s a shared inbox system, too. So we can have our sales people jump on it, too. So, again, it’s all about, you know, messaging and interacting with people. And that’s one aspect of it. And I could go a little further if you want.
Johnathan: Yeah, we got some time. Let’s say we have like five more minutes and we’ll wrap it up. But yeah, anything you want. This is awesome.
Eric: Okay, yeah. So going a little deeper into that, I mean, maybe like two hours ago I had the ManyChat founder, Mikael Yang, for the second round of podcast interviews on the Leveling Out podcast, which is my other podcast.
But I was just getting more data from him. You know, when I last spoke to him, a couple years ago, he’s like, yeah we have about 1 million messages sent per day. Guess how many messages they have sent per day, now that they’re managing.
Johnathan: Like 10 million?
Eric: No, like 266 million per day. So, 100 to 266, right. It’s crazy. I think that’s 8 billion a month. But the key point here is massive.
Like Twilio as a company, their stock price continues to sky rocket, right? SMS messaging, it’s not going anywhere. So I look at ManyChat. They’re not just a Facebook Messenger platform anymore. But they have SMS, they have email. So if you could somehow integrate SMS with Facebook Messenger, you could do really ninja stuff.
And I’m glad I’m talking to you about this. I’m happy to share all this with you because I think you can do this. Not many other agencies can actually weaponize all the stuff that I’m talking about.
Johnathan: I appreciate it.
Eric: So from a B2B perspective.. Yeah, yeah, you’re welcome. And from a Facebook Messenger standpoint, sure you could use ManyChat. This works for B2C, too. So it works for consumers, works for B2B as well. People just like this stuff, right.
Johnathan: Yeah.
Eric: So, yeah.
Johnathan: That’s amazing, that’s amazing. What are you guys trying to tackle next? Like what’s the next thing? Because you guys, you’re doing really well. You’re a very, very popular marketer. You know your shit. Like this is why I have you on the show.
What, let’s just, completely side note, Eric. Like now that I’ve got you. Like what are your goals as a marketer and as a person like career wise or financially? What’s your end thing?
Eric: Yeah, I think you and I, I think we’re fortunate enough to be at a point where it’s kind of like house money. Like we probably have more than we ever dreamed that we would have. So that’s a very good position to be in, right?
So, after that it’s kinda like okay, so how do I help the world even more? So, my whole thing, and this is one I’ve been working on for a while and it’s taken a couple years but I have a book coming out called Leveling Up, which is why my podcast has been renamed to Leveling Up from Growth Everywhere.
It’s literally about, you know, how can I help not just a certain number of human beings, but how can I help every single human being in the world level up, right? Whether that’s through personal or kinda professional.
Now I set the number that high because I want my goal to be unrealistic so I can, you know, shoot for the moon or shoot for the stars, land on the moon, right?
Johnathan: That’s amazing, man. Well I, mine’s just about planting trees right now. So I need to level mine up quite a bit compared to yours. Like that is a much better goal.
Eric: I love that goal, just make it much bigger. I don’t know how many you want to plant but maybe like make it way bigger, yeah.
Johnathan: Yeah, you’re awesome, man. I really appreciate you being on the show. The people like that are listening, I hope you guys got a tease of what knowledge drop could come in the future, too, if we have you back on the show, Eric. Really appreciate you.
Eric: Thank you.
Johnathan: All right, talk soon.
Eric: All right, see ya, bye.
Johnathan: All right, bye.
Marketing Experiments Mastery With Pedro Clivati
Johnathan: Alright everybody, I have my good friend Pedro Clivati, he is the Head of Growth over at GrowthHackers and we’re gonna be talking about marketing experiments today. How are you Pedro?
Pedro: I’m good, I’m good, thanks for having me Jonathan!
Johnathan: Of course man, super pumped! Somebody who lives and sees everything that the GrowthHackers community has to share, plus gets to work on so many fun and exciting things I want the people here listening to understand you know, the foundation of marketing experiments.
Now we could call this growth hacking, and I’m completely okay using the term interchangeably, you might have some different thoughts, but when it comes to marketing experiments, can you kinda tell how your mindset approaches that for the listeners?
Pedro: Sure, sure, yeah, so there’s a lot of buzz around like, should that be talked about as growth hacking, growth marketing, marketing experiments, but I believe it all comes down to the same thing: actually running experiments all around, figuring things out, see if we’re actually going all-in.
Right, so regardless of how you name it, I believe it all comes down to the framework of coming up with new ideas, testing them out, and validating it for rolling out.
Johnathan: Yeah.
Pedro: So yeah, that’s pretty much like, I don’t think there’s much difference between them, there’s difference between the name but the framework it’s the same.
Johnathan: Perfect, that makes sense, that’s what I was thinking about too. We may have to change our episodes, because we also have a growth hacking episode that we planned on launching, but we might change that now, who knows.
So, let’s say that you approach, I mean you do marketing experiments and growth marketing for GrowthHackers, is that correct to assume?
Pedro: Yeah, that’s correct and we do have a Marketing team and a Growth team, and the main difference between both of them is our Marketing team is focused on acquisition part of our funnel, right?
So they are always making change, they also run experiments themselves, they have their own operations so their daily routine, etc. But it’s always connected with attracting visitors, with converting this virtually, that’s the part of the customer journey they’re responsible for.
And I’ve deferred them from the Growth team, it’s precisely because the Growth team has the opportunity or the flexibility to apply their knowledge base and their experiments and their knowledge across any department of the company.
Johnathan: Got it.
Pedro: So that’s why we talk a lot about breaking silos, it doesn’t belong to one department or another, it actually varies, like where is the biggest bottleneck in the company right now that our Growth team should be acting upon, regardless if it’s acquisition, activation, retention, revenue, referrals, or any other growth library that you might have.
Johnathan: Got it, that makes sense, so I just, now that you define it that way I can think of myself as a one man Head of Growth at our agency KlientBoost, ’cause I just finished some proposal stuff that was a bottleneck, so that’s awesome, that’s cool to hear. Now, if we talk about
Pedro: Yeah, it’s more in that–
Johnathan: I was gonna say if we talk about you know, marketing experiments and focus in on that for you, can you kind of help us when you either do it at GrowthHackers or let’s say you did it at a previous company, or one of your clients, for example.
How do you decide on what to prioritize? Where does that initial research start?
Pedro: Awesome, so we basically have a three-layered cadence of KPI that we are always looking at. So the main, like the broadest one we used to call that our North Star Metric.
Some people define it as as the one single metric that matters, or the metric that matters, there’s a lot of different names to it.
But it’s the one metric that actually, regardless of the department that we’re talking about, it’s marketing, it’s sales, it’s product, it’s technology, it’s customer service, whatever the department is, if they’re all doing well, that’s the measure that we are improving, right?
So it’s always connecting with both the growth of the company to improve revenue, but also the value that you’re trying to provide. And the reason why we have this metric is precisely because of why we mentioned earlier, right?
It is something that gives you flexibility to run across departments so the mark of your message should be something that engulfs them all, right?
And right below the North Star Metric, we got our focus for the time being. So let’s say that like one real example here, Uber’s North Star Metric is actually rides completed. So if you complete a ride of Uber, that means that the rider is getting value, the person who is actually riding is also getting value, and Uber is making money. Right?
Johnathan: Makes sense.
Pedro: So all of the stakeholders involved are getting value out of it, so that would be our North Star Metric. So the objectives are the focus value of what you’re going to be doing over the course of the year.
Could be something like helping bombarding next year, let’s increase the number of drivers that we have, let’s increase the number of riders we have, so those are the major objectives,
Johnathan: Sorry, Pedro real quick, ’cause I didn’t hear you the first time, can you hear me okay? So I was asking the second thing,
Pedro: I can hear you perfectly, yeah.
Johnathan: So the first thing was the North Star Metric, the second thing were the objectives, is that correct?
Pedro: Yeah, so the second thing is the objectives. While the third and the lowest level is the cadence of the API. Are they staffed, the experiment, the ideas of things that are gonna be right.
So ideally those ideas are complete networks, calling them maps technology it’s strictly following the methodology would give everyone in the company a part. That’s the growth idea, the Growth team is still responsible and should be why they are gonna growth hack, so when the company can pick up and suggest the growth ideal, they’re giving else ideas.
Johnathan: Everybody’s part of the team, yeah.
Pedro: Those are like the ideas of the experiments that you’re gonna be running, right? So you could be recapping the North Star Metric like you’re presenting all the departments, everything that we are building, the value that your parts provide.
Secondly, we’ve got objectives that are macro-focused for the time being and last but not least, we’ve got the ideas that are the experiments that are actually gonna be running.
And to choose which experiments are actually going to be running, we personally use the ICE Score, so that stands for Impact, Confidence, and Easy. So every single idea that is suggested to our backlog, they are classified among those three criteria: Importance, Confidence, and Easy.
So we, depending on the resources that we have, the time that we have available, and the urgency of delivering results, we prioritize according to that.
Johnathan: Awesome, so to recap, sometimes the connection might not be that great. We have number one is your North Star Metric as a company, the second thing are the objectives or what’s included to be able to influence that North Star Metric, and then third are the ideas to actually be part of those objectives.
The ideas are split up in three categories known as the ICE Framework, right, so you said, what was the first one, Impact,
Pedro: Correct.
Johnathan: And the C was what?
Pedro: Yeah, Impact.
Confidence, so how confident that you are that the idea is actually gonna reach the high point. And then Easy, so how many people would you need to actually run the experiment, or how much money do you need. So it’s pretty much like how much resources are you gonna require to run that experiment.
Johnathan: Got it, so we have Impact, Confidence, and then Ease, like the resources you basically mentioned. And is that scored, are those things scored on a one to 10 basis, or a one to five, how do you score those?
Pedro: So each of them are scored between one to 10, right? A lot of people are looking at the average score across those three criteria, but I would strongly recommend you to look at each criteria separately.
So, let me give you a real example, we are, let’s go back to your in the beginning of the barter, there is like a backlog full of ideas for you to run, your Product team is available to you, so you might just filter them by the highest ease, or the ones that you are more confident that are going to work out, instead of looking at the average.
Because the average might have like one idea has a 1.4 on Confidence, and 10 on Easy, and another idea has a one on Easy and a 10 on Confidence, they are gonna have the same average score, but if you look at each criteria it gives you a different perspective.
Johnathan: Right, that makes sense.
Pedro: So let’s say that we are in the end of the Quarter and the Product team is focused on the launch of a new feature. Everyone is looking at that, so you don’t have much money, you don’t have much resources, you don’t have much time.
So sort all of the ideas in the backlog based on the highest Easy, or the easiest one for you to implement. That gives you a better perspective, like which ideas you should be running now.
Johnathan: That makes a ton of sense, so cool.
So basically for people listening, you were saying that like if something has a higher Confidence score than an Ease score, obviously considering your resources and what you have available depending on the time of year you might decide to prioritize things differently, considering outside circumstances beyond the ICE Score that you have, like the average, which is really really smart to think about that way.
So Pedro, once you have these ideas figured out, what are some of the common challenges that may happen when you actually roll with that and everything else is defined, is there anything that happens?
Pedro: Uh, yeah, so a few challenges that I see like with Growth teams actually having is like, I guess the most common ones is not having total support from the higher-ups. So the basis of a Growth team is actually to run experiments on things that have never been done before, so based on that, it’s likely that most of those experiments are not gonna work out.
Feel how you like, the Growth team needs to have a tolerance higher than any other department. Some successful Growth teams that work with us, they have up to 70% of their experiments actually don’t play out as they were expecting. And that’s okay, because the 30% that actually plays out well, pays off for almost all the rest.
But for them to keep failing over time, they need to have an acceptance from the higher-ups, from their VPs, and then the understanding from the rest of the company that the Growth team is going to fail. Because they are going to be taking higher risks than any other department.
But once they actually reach the target, it pays a lot of the rest, so one like really common struggle that I see Growth teams facing is precisely this misunderstanding from the C-levels and the Directors that the Growth team is treat them like a department is taking out a lot more risk than any other department, and that’s okay, that’s okay.
Johnathan: It’s so, it’s funny you say that, there’s this comic strip that’s really famous for a lot of marketers and business people in general. It’s called like the Marketoonist I think, and it’s funny like literally to the point of what you were saying.
There was this one frame where the boss was saying to his people, “We need to do something so innovative that it’s gonna like crush all our goals and we’re gonna be like the best company in our industry.”
And the second one, when the actual team presents it to him which is, I’m assuming, like the boss, the CEO, he’s like, “Hmm, this seems risky, can you show me examples of other companies who’ve done this too and succeeded?”
And so it’s to your point, you need the buy-in of the CEO or the C-suite people because otherwise you’re gonna keep having issues.
Pedro: Exactly, and I think like on last year when Amazon’s shareholder’s letter, Jeff Bezos mentioned something really similar, like most of the companies are really want to innovate, but they don’t, they are not willing to actually accept the fact that for you to be innovative, you’re gonna fail a lot!
Johnathan: I believe that.
Pedro: So, you know what, you just want the upsides, but you’re not up for the downsides. I think that’s the most common challenge that I see Growth teams face.
Johnathan: Okay, so we had the lack of buy-in and understanding from C-suite, is there anything in regards to like ownership of task, or like the execution of these experiments that become an issue that you’ve seen?
Pedro: So normally like when Growth teams are just starting out, they don’t start out with all of the resources that they will eventually need. So they usually keep things soft with what we call like a mixed structure team, where like one person responsible for doing up the growth strategy, and prioritizing ideas, choosing what goes into production or not.
But when the time comes for actually executing those steps, they need resources from other areas, right? They communicate with the Product team, or the Marketing team, or the Sales team to actually execute those experiments, and that’s really common.
But once they start to drain resources, and the Growth team starts to evolve, then they usually go into an independent structure, where you have someone on the VP of Growth who has his own team.
And they still have, they normally have on the Growth team is someone who has a skillset on the data science, a marketer, a developer or programmer, and a designer or a UI or UX back study. That’s like the perfect gang, right? They all have the skillset are available for one person to allocate.
Yeah, so one other struggle that I could mention is when they don’t have this understanding from other areas or other Directors it also becomes harder for them to actually execute those experiments if they don’t have the buy-in from other areas.
And when we think about that on a startup reality, it’s pretty easy for us to figure it out, we are usually like, if it is a smaller startup we are all seated in the same room, or on the same floor at least, so you know who’s working on the Product team, and you know who’s working on the Marketing team, it’s pretty easy to run after them.
When we think about that on scale, then sometimes they might be in a completely different building, or in a completely different city or in completely different state, and someone from the Marketing team might have never actually met someone from the Product team.
So those gaps between areas, that flow of information in these areas, just don’t exist. Each department are needing their insider-own wards, and it becomes pretty hard to actually just reiterate something that you have worked. So that’s pretty much when we talk about breaking silos between teams.
That’s one of the responsibilities of a Growth team, is actually like to being able to run across departments and not like, it doesn’t belong to any specific area, so that allows to have perspectives from everyone, and the flow of information from the Product team, to the Marketing team, to the Sales team actually happens and it molds well.
Johnathan: That makes a ton of sense, and obviously see the main benefit.
I had a question for you, because thinking of the tolerance of risk from higher-ups if you’re trying to run marketing experiments or even you know, focusing on growth for your company, is there a certain time limit that you have on experiments?
‘Cause let’s say an experiment could take six months, but if you spent all the resources within your organization, and it doesn’t work out, that’s gonna hurt more than an experiment that was a lot shorter. Do you have any rules of thumb for that?
Pedro: That’s a good question, now let me answer that in two parts. So one is that the reason why we have a cadence of three layers of KPI is precisely because as a company everyone is gonna be looking at either your objectives or your North Star Metric.
So we are never, like the rest of the company will never be looking at if you are actually improving the conversion rate of your landing page. They’re gonna be looking at the bottom line. So how many leads have you delivered?
So the expectation of the rest of the company is not connected with the experiment itself, but with the impact that the experiment is bringing to the company. So that’s a really common piece of approach.
Like, I love to talk about the improvement of the conversion rate that I did on the landing page, or how I did raise the customer acquisition through an experiment on a paid media acquisition channel, but the bottom line, like what the company actually cares about, is the bottom line, it’s the whole number. It’s not the conversion rate improvement, right?
So that’s why we have objectives, that’s why you have layers of KPIs, so we’re not looking at the experiment itself, but the impact of that experiment is bringing to the whole company, that’s the first point.
Second point is, Sean talks a lot about that, so he always says that growth hacking is about like high-tempo tasks and on a small scale. So what does that actually mean? Right? We recommend companies should be running experiments for as long as two weeks.
Johnathan: As long as, like, max two weeks?
Pedro: Max two weeks. And the reason behind that is like it takes a while. So if you have an experiment that needs to be like, live for like six months, try to break that up, like into small experiments, instead of keeping one experiment live for six months.
Johnathan: That makes sense.
Pedro: That allows you to decrease the influence of external factors in your media output of your experiment. So stay live for like two weeks, maybe like some political instability is going to affect your bottom line, but overall it’s easier to decrease the influence of external factors.
And we also talk a lot about smaller scale. So let’s say that you are running like your newcomers, and you are changing like I wanna improve the checkout experience, and instead of running an experiment for like 100% of the users that go into that page, just run for like 10 or 20% of that amount.
If things worked out, then you can bring that number and say hey, I’ve found a way to actually improve the conversion rate 50%, and if we scale that up that means x-percent improvement on the bottom line, or the number of checkouts, right?
But if it doesn’t work out well, your experiment, then you don’t need to roll that out into a larger scale. So you’re minimizing the downside while maximizing the upside if things worked out.
Johnathan: Yeah that makes like, it makes it so, so clear and simple too. Anything that I’m not asking? I know we’re coming up here on time, anything that I should be asking you that I haven’t asked you yet?
Pedro: I don’t think so! I think that we have covered a lot, yeah.
Johnathan: It was a super, super well put together, oh go ahead!
Pedro: I would say one thing and a lot of things tend to look at growth hacking for only from the hacking perspective, right?
So for some that brings like a negative connotation, and people tend to look at that blog post like, “How I increase my traffic in three days” or “How the percent my traffic in three days doing x or doing y.” I’m not saying that this doesn’t work, I’m saying that this is not growth hacking.
So growth hacking or marketing experiments or growth marketing whatever you want to call it, is actually the process of constant experimentation, constant and thoughtful experimentation, right?
So I although we always read about that single change, and that change in startup life. That didn’t come out from one single experiment. They probably had hundreds of experiments that went wrong before advertising that.
So it’s more about like constant small improvements, that when you send, when you sum that all together, they make a huge difference, than one super bullet that’s gonna change all of our lives.
Johnathan: I agree, I agree. Pedro, so, so cool, thank you for giving us a foundation. I am actually, thought that I knew about growth marketing and marketing experiments, even growth hacking in general.
I met with Sean Ellis many times in person, I didn’t know it as clearly as you just explained it. So I’m super thankful, I’ll let him know that he sucks.
Pedro: Yeah, awesome.
Johnathan: I’m just kidding, but I appreciate your time!
Pedro: Thank you Jonathan, thanks for having me!
Johnathan: Alright, we’ll talk soon, bye!
Pedro: Bye!