seojocktoberfest - testing for understanding seo - michael briggs

Testing for Understanding SEO

Michael Briggs

My Mate Nick

My Mate Nick

“We used to Be Mavericks”

Not Like This!!!

We used to Spam at Scale

We used to Spam Creatively

Loser

Liar

Fake

Phoney

How we solve this?

• Scientific Method

• Build -> Measure -> Learn

• The accidental test

Scientific Method

Lets see how accurately I can get to an answer

Is Google Flights Hiving Off Traffic

It Must Be

Problem

• Obviously isn’t good enough

• I work for a billion dollar plus company

• Your clients probably need more as they work in the same way

• Need to get to a definitive answer– Or die trying

The Scientific Method

Google Flights Problems

• I hadn’t chosen to do the test

Google Flights Problems

• I hadn’t chosen to do the test

• Google was running the test

Google Flights Problems

• I hadn’t chosen to do the test

• Google was running the test

• Need to then fill in the blanks

Google Flights Problems

• I hadn’t chosen to do the test

• Google was running the test

• Need to then fill in the blanks

• Was looking backwards

The Real World Hypothesis

Google Flights launched and it’s changed the search landscape and is affecting my CTR

from Organic

My Real Thoughts on the Matter

Google Flights launched and it’s hiving off traffic because Google are a new

technological overlord.

If we don’t stop them they will take over the world.

Meh

• Problem

– Search Volumes are growing (and inaccurate)

–Google Webmaster Tools (I am not ever calling it a “search console”) is awful

– There is a wealth of data

–Where can I find something useful

–Where should I be looking

Hypothesis Development

• New Hypothesis– Google flights launched and it is hiving off traffic. If click through rate changes and

positions stay stable then there is a correlation between the launch of Google flights

Hypothesis Development

• New Hypothesis– Google flights launched and it is hiving off traffic. If click through rate changes and

positions stay stable then there is a correlation between the launch of Google flights

– If I have stable variables I can remove these from analysis

• I know on term set X my positions haven’t changed

– I’m lucky we’ve been number one for a range of things for a while

– And find the variables that change

• Traffic by device

• Search volume by device (error factor)

• Implied variable of CTR

The Graph

Problems

• Not enough data on the keyword set– Longtail

– Google Keyword Tool is crap

– Data gap

• There were multiple changes on the historical data set– Hummingbird

– SEM Changes - Ad extensions

– Listing had go site links added to it

– Site in position two changed

• Positional Data from rank tracking isn’t accurate– Personalisation

Conclusion

Forward Looking Test

I think, therefore I’ll test, so I know

Build Measure Learn

Build

Measure Learn

Forward Looking

• I think this to be true

• I will create a hypothesis

• I will test it

• I will make a conclusion (hopefully)

Accepted Truth

Unique Content

• Hypothesis

–If I add unique content to a page it will make me rank better in Google because Google loves unique content therefore I need unique content.

How do I build this as a significant test

• Take two sets of pages

–Test

–Control

• Change content on the test

• Measure the difference in the normalised traffic

Worlds Most Boring Graph

Test Results

Conclusion

Why

• Content writers weren’t good enough?

• Google is a liar and we’ve been lied to?

• The signal balance had changed in a way we didn’t understand?

• Something else?

Why

• Content writers weren’t good enough?

–Build new hypothesis

• Google is a liar and we’ve been lied to?

• The signal balance had changed in a way we didn’t understand?

• Something else?

We Rebuilt The Content

Test Outcome

Unique Content only helps if someone who knows wtf they are doing actually read the content and makes sure it is in the form it is meant to be in and is properly optimised.

Unique content isn’t enough.

The Accidental Test

Fail Forward

Web Engineering

What it really can look like

Sometimes it looks like this!

SEO’s Regress to This

Because Overnight

What this meant was some days of hell!

Failure is Always an Option

Direct got Similar Hit

90% of Direct Traffic on pages examined was really Organic

• Fail Hard

• Fail Fast

• Fail Frequently

• Learn from every failure

#failForward

Thanks

michael.ke.briggs@gmail.com