robots: txt, meta & x - the snog, marry & avoid of the web crawling world - brighton seo sep...

#BrightonSEO15th September 2017

@chrisgreen87www.strategiq.co

StrategiQChris Green

@chrisgreen87http://bit.ly/snog-marry-avoid

Robots: Txt, Meta & X The Snog, Marry & Avoid of the Webcrawling World

http://bit.ly/snog-marry-avoid





How do we knowthe best way to manage

Googlebot’s crawl/indexing?



There are many methods(we’re spoilt for choice really)



But the two most commonly misused are Robots.txt vs Meta Robots directives





Why?



To the casual observer they’re very similar ways of doing the same thing…





To block Google



But that’s not a helpful way of thinking of them





In many

it can stop you getting the most out of your site

circumstances





I’m going to run through a framework

to help to change this thinking





I’m going to run through a framework

to help to you make the right choices





But some words of warning



This is advanced stuffOne foot wrong & you could cause serious damage



They’re not always the first-choice

These are only part of your toolkit



There are so many “ifs” & “buts”





If we can finish today

with slightly more understanding





And a different approach

then we’re onto a winner!





Time to introduce the robots



Robots.txt



Meta Robots



X-Robots



Possibly the most important SEO tools





But which do you...





Snog





Marry





Or avoid?





But what does a slightly s**t

BBC 3 show have to do with SEO?





One site’s “snog” is





One site’s “snog” is another’s “marry”





One site’s “snog” is another’s “marry”

Or perhaps even “avoid”



There are lots of thoughts on how to use these



There are lots of thoughts on how to use these - many are wrong





I’m going to show you a way of simplifying

things





To pick the right tool for the job



Know the problem you’re trying to fix!





Is it a crawl problem?

Google isn’t seeing enough of your site





Or an index problem?

Google’s indexing too much of it



We fix crawl problems with Robots.txt



User-agent: *

Disallow: /*mad-spider-trap*



And we fix index problems with Meta Robots



www.domain.com/crap-page

<meta name=“robots” content=“NOINDEX, FOLLOW”>

http://www.domain.com/home



Identifying the problem





Index problems are simple to spot





Does your site look too big?

(in Google’s eyes)





But ID’ing a crawl problem...

Can be trickier





Look for spider traps

https://www.portent.com/blog/seo/field-guide-to-spider-traps-an-seo-companion.htm







Where does this “cost” you on crawl budget?



A word on crawl budget.





It’s “a thing”

http://searchengineland.com/google-explains-crawl-budget-means-webmasters-267597







But Google doesn’t publicise a site’s crawl

budget





You can work out a version of it yourself

Thanks to Yoast for this - https://yoast.com/crawl-budget-optimization/

https://yoast.com/crawl-budget-optimization/

https://yoast.com/crawl-budget-optimization/



Look at GSC Crawl stats average pages crawled per day



Look at GSC, how big Google sees your site as





Pages / Avg crawled per day =

Crawl Score





9,781 / 1,458 = 6.7

x 6.7 more pages than are getting crawled each day





If you have 10

You have 10x the pages that Google is crawling daily



A pretty big crawl problem!



But, how big is big?





< 1,000 pages, crawl budget is less of a problem

https://webmasters.googleblog.com/2017/01/what-crawl-budget-means-for-googlebot.html?m=1







1,000 - 10,000 is moderate





10,000+ pages… things start to get “fun”



Crawl & Index problems aren’t mutually exclusive





Index bloat at scale can hurt crawl





Crawl issues can stop or slow the repair of

index issues



Some example scenarios





eCommerce filters which are getting indexed

(badly)



www.domain.com/shop/mens/trainers/size-12/red/





User-agent: *

Disallow: /*size*Disallow: /*red*





Not until index issue is cleared up*





*Unless...



User-agent: *

Noindex: /*size*Disallow: /*size*Noindex: /*red*Disallow: /*red*





Google isn’t “cool” with this

https://www.seroundtable.com/google-do-not-use-noindex-in-robots-txt-20873.html







But it’s proved to work

https://www.deepcrawl.com/blog/best-practice/robots-txt-noindex-the-best-kept-secret-in-seo/

http://ohgm.co.uk/de-index-pages-blocked-robots-txt/



http://ohgm.co.uk/de-index-pages-blocked-robots-txt/



Only ~0.3% of the Majestic Million use this method





Don’t be too aggressive though!





Be aware that some filtered pages can be

worth indexing





Blog taxonomies



www.domain.com/blog/blog-category/www.domain.com/blog/tags-bloody-tags/








eCommerce site without indexed filters but x6+ crawl

score



User-agent: *

Disallow: /*filters*

(& meta robots just in case)





Other misc pages?

Just noindex





Anything else?



Back to my original premise





Meta Robots is my“marry”





Robots.txt is my “snog”





It really can make the difference



Meta robots replaced with robots.txt disallow:





But it is easy to screwit up





What about x-robots?





For when meta robots isn’t possible…





For when meta robots isn’t possible…

… assuming you can edit htaccess





It’s not my “avoid” though





What is?





The lazy option!



User-agent: *

Disallow: /*pointless*Disallow: /*disallow-rules*Disallow: /*instead-of*Disallow: /*fixing-the-problem.html



www.domain.com/200000-filtered-combos

<meta name=“robots” content=“NOINDEX, NOFOLLOW”>




User-agent: *

Disallow: /we-should-write-better-content/

#but don’t want to prioritise



The “best choice” depends onyour limitations





Do you have all the access you need?





Or enough buy-in?





Otherwise, some workarounds are better than doing nothing



Meta Robots via GTM

https://moz.com/blog/seo-changes-using-google-tag-manager

https://moz.com/blog/seo-changes-using-google-tag-manager



Robots.txt when no other option

User-agent: *

Disallow: /better-than-doing-nothing/



Key takeaways



Is it a crawl or index problem?



Check what you can change



Check what you can changeAnd what you can’t…



Make the “best case” fix



Implement, crawl & check again!



Use this flowchart to help

http://bit.ly/bseo-flow

http://bit.ly/bseo-flow



Thank you.http://bit.ly/snog-marry-avoid@chrisgreen87

http://bit.ly/snog-marry-avoid

robots: txt, meta & x - the snog, marry & avoid of the web crawling world - brighton seo sep...

Marketing