How Fantasy Football Makes
You A Better Split Tester

Art Anthony

This year, almost 75 million Americans are playing fantasy football, not to mention all of those international folks (myself included) who are also joining in.

Those among you who play the game are now over halfway through your season and probably have some idea of whether or not you’re playoff bound. At 5-4 and tied for third place, things are looking promising for me.

It’s not uncommon for fantasy football fanatics to spend hours every week tinkering with their lineup and agonizing over whether to start that running back who burned them last week.

So wouldn’t it be great if some of those skills transferred outside of what, even if you take it as seriously as me, is just a game?

Well, they can, and do.

Which is good news if you spend as much time playing fantasy football as I do.

The game has a few significant comparisons with several different aspects of business and marketing but, for the purposes of this post, I’ll be sticking primarily with lessons you can take away from fantasy football to make you a better split tester.


Trust Your Gut (Sometimes!)


split testing

This comes with some limitations though – Image Source


Switching someone out of your lineup for another player who ends up having a great week is one of the most satisfying parts of fantasy football.

In the same way, learning to trust your gut can be valuable when it comes to setting up a new split test or PPC campaign. If instinct is telling you to run a particular experiment or try including something new in your ads, it’s usually worth listening to that hunch.

That doesn’t mean you should blindly follow your instincts and over pursue a test by devoting too much time to it.

Sometimes a player you’re convinced is going to do well on any given Sunday is just a dud, and the same is true of split testing.

Noah Kagan of AppSumo has previously written that only 1 out of every 8 split tests he runs has a significant impact, and you have to assume that he wouldn’t have run them in the first place unless he had an inkling that they could have a positive impact.

Many of us have a tendency to stick with things for too long, be it a failing test or an underperforming fantasy player, because “I’m sure it’s going to turn around if we just give it another week.”

Unfortunately, things don’t always turn out the way we want them to.

So, don’t be afraid to stick with the gut, but don’t put the blinkers on when your gut’s wrong. But how do you know when your gut is wrong? That’s when it’s time to get your hands dirty and dive into the data.


Look At Data, Not Projections


Every week ESPN, Yahoo and assorted other fantasy sites publish projections for each player. In most cases, those predictions turn out to be way off.

Again, this is much like the world of A/B testing: sometimes you’re sure that certain campaigns, techniques or experiments that you run are going to have a huge impact, whether it’s because of case studies you’ve read or just a hunch you have, but it turns out that it’s just not meant to be.

You might predict a certain % improvement in conversion based on past experience or a killer hypothesis, but it’s never a good idea to ignore the results you get in favour of that initial prediction.

As well as the raw data, you also need to look at the context. Let’s say a player scores three touchdowns one week. It’s tempting to play them next week but, when you look a little closer, you see that they’re playing against a much tougher run defense and an all round better team.

Even tests that seem to be big winners are often subject to validity threats that can undermine their performance if/when they go live. A test could considerably outperform a control but, if you only run it on weekdays, then the results you’re seeing aren’t a true comparison with the status quo.

Take data from this site as an example:



If a tester were to run an experiment that didn’t include weekends – which many do, since they know that they won’t be there to repair things or disable to the test if it breaks something on the site – they would almost certainly see an improvement as these days have the two lowest conversion rates.

Similarly, not letting a test run for long enough could also skew results. Let’s say the owner of the site above starts a test on Thursday and checks the results 8 days later.

Again, the test is bound to appear as if it’s performing better because Thursdays have such a high conversion rate. You have to give these things time to develop.

Same goes for fantasy. If a player scores 20 points one week, but has been on the field for the past six and racked up 0 points every time, I’m going to be hesitant to start him until I see whether or not he can demonstrate consistent improvement.



Don’t Be Afraid To Hit Up The Waiver Wire

Fantasy football seasons are often won and lost by the extent of a player’s involvement with the waiver wire, which allows players to pick up players who’ve had a particularly good week or are set to return after injury.

Digging deep for seemingly insignificant players is great practice for finding testable aspects of a site that may appear unimportant but actually have a huge impact when tested.

That involves a significant element of prioritization: what should appear really important to users on this site? What do you want them to do next?

These are the things you should run tests on. A previous post on the KlientBoost blog deals with how to do this, but I’ll quickly summarize a few of the points here:


  • Start where the money is
  • Define the goals you want to achieve
  • Create hypotheses about what (you hope) will happen
  • Address big traffic drops and other issues ASAP

Substitute a few terms in the above (money –> points, traffic drops –> fumbles) and they could just as easily apply to managing a fantasy football team*.

* That’s a process that works both ways, incidentally, which you’ll know if you’ve ever seen The Automatic Faucet episode of The League, in which Pete uses his fantasy football skills to help his boss manage a project at work.

An ability to identify hidden value (or potential value) is a trait that the best split testers have, because significant changes aren’t always big ones but small changes aren’t always significant.

The key to effective prioritizing is to get as close to the data as you possibly can, with the aim of understanding how and why a test has improved on what was there before it. For example, previous tests with big uplift that involved changing a headline or CTA may demonstrate a wider issue with tone of voice or how easily content on a site is understood.

Without this skill, it’s easy to get trapped in a cycle of testing more and more elements without any clear end goal, (or picking up useless player after useless player on the waiver wire!), as previously discussed in this post on our blog.



Understand That Sometimes It’s Boom Or Bust



Consider the following two situations:


  1. Several of your best fantasy players are on bye weeks, so you can’t use them in this week’s lineup. Any obvious alternatives have already been snapped up from the waiver wire, and it looks like slim pickings.
  2. Your CRO testing seems to have hit a plateau. You’ve run a number of tests with positive results, and implemented them permanently, but you’re struggling to improve any further on what you have.

These situations are analogous in that, in both cases, you need to take a risk on trying something completely new and unproven.

In fantasy, this is what we call boom or bust.

There’s comes a point when you WILL run out of “safe” tests like button colors and changing CTAs (Calls To Action) from “Try It Now” to “Give It A Try”.

Eventually, you have to step back and try to throw one down the field by changing headlines, stripping out some of the body copy etc.

This can be scary, and risky in that it might have no impact or even a negative one, but it’s also where split testing has its biggest impact; like in fantasy, put up with a few busts and you will eventually find a boom.

To be clear, I’m only suggesting above that it’s ok to take measured risks when previous steps you’ve taken haven’t yielded much in the way of results, not that you should try all sorts of crazy things in the hope of eventually getting a win out of it!



And The Kick Is…Good!

Ok, I admit it: this article started its life as a pitch to my significant other on why fantasy football isn’t the waste of time she says it is.

But, as I thought a little harder and did more research, I came to realize that it really does develop some skills that can be pretty valuable for marketers and split testers.

However you feel about the game, there’s no denying that fantasy football success hinges on several different things:


  • Good organizational skills and an exceptional approach to long term strategy
  • The ability to dig deep to find players who can get you points
  • Making calls based on results rather than instincts, balanced with…
  • …a willingness to trust your gut every now and then
  • Luck (I’m talking about some good fortune here, not Andrew)

There are those who’ll argue that the final bullet point listed above is far and away the most important one and, in some respects, the same is true of CRO and marketing.

There’s always the chance that good timing or a total shot in the dark will lead to hitting a home run – yikes, now I’m really mixing sports metaphors – thanks to some beginners’ luck.

But, more often than not, it’s the person who takes a logical, measured approach to things – whether fantasy football or split testing – who will come out on top.

P.S. Did you like what you read here? Gained some insight? :) Tweet and post this to your peeps to share the wealth.