combatting crawl bloat & pruning your content effectively
TRANSCRIPT
![Page 1: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/1.jpg)
Charlie WhitworthSEO Director
@WhitworthSEO http://www.slideshare.net/whitworthseo
Banc
![Page 2: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/2.jpg)
what will we cover?
bancmedia.-com
Crawl Bloat & Pruning
@Whitworth-SEO
![Page 3: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/3.jpg)
beat the bloat
Crawl Bloat
Making search engines work too hard
and not taking full advantage of your
crawl budget@Whitworth-
SEObancmedia.-
com
![Page 4: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/4.jpg)
crawl definitions
=Crawl Budget
bancmedia.-com
@Whitworth-SEO
Crawl Rate & Crawl Demand
![Page 5: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/5.jpg)
crawl rate (limit) : maximum fetching rate
Search Console Limits
bancmedia.-com@WhitworthSEO
Website/Server Health
![Page 6: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/6.jpg)
crawl demand
Popularity (Traffic) Staleness
@Whitworth-SEO
bancmedia.-com
How much does Google want to see your URLs?
![Page 7: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/7.jpg)
Crawl budget – determined by rate & de-mand
What Google can & wants to crawl
@WhitworthSEO bancmedia.-com
• Empty pages• No valued added• Dupe
• Soft 404• SPAM• Hacked content
![Page 8: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/8.jpg)
crawl budget
“Prioritizing what to crawl, when, and how much resource the server hosting the site can allocate to crawling is more important for bigger sites, or those that auto-
generate pages based on URL parameters”
Gary Ilyes
bancmedia.-com
@Whitworth-SEO
![Page 9: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/9.jpg)
how to identify crawl bloat
bancmedia.-com
@Whitworth-SEO
![Page 10: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/10.jpg)
how to identify crawl bloat
bancmedia.-com
@Whitworth-SEO
![Page 11: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/11.jpg)
why is this a problem?
bancmedia.-com@WhitworthSEO
Don’t work them too hard
Search engine bots hate crawling rubbish
![Page 12: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/12.jpg)
why is this a problem?
Engagement metrics are key
especially now we’re “mobile first”
@Whitworth-SEO
bancmedia.-com
![Page 13: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/13.jpg)
examples of rubbish: search result URLs
/search/bancmedia.-
com@WhitworthSEO
/sessionkey=
/search-results?q=
![Page 14: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/14.jpg)
examples of rubbish: paginated URLs
bancmedia.-com
rel=next/prevnoindex
@Whitworth-SEO
![Page 15: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/15.jpg)
examples of rubbish: faceted navigations
json-ld
disallow parametersbancmedia.-
com@Whitworth-
SEO
noindex
![Page 16: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/16.jpg)
who’s seen this?
![Page 17: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/17.jpg)
cut the crap
Get Rid Of It
@Whitworth-SEO
bancmedia.-com
![Page 18: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/18.jpg)
plan
![Page 19: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/19.jpg)
noindex
<meta name=“robots" content=“noindex">Won’t deal with crawl bloat in the short term
Still an effective tacticbancmedia.-
com@WhitworthSEO
![Page 20: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/20.jpg)
url parameters
@WhitworthSEO
Powerful tool when used properly
No need for developer sup-port
bancmedia.-com
![Page 21: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/21.jpg)
xml sitemaps
Redirected URLs
Noindexed URLsPoor quality pages
@WhitworthSEO bancmedia.-com
![Page 22: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/22.jpg)
robots.txt
bancmedia.-com@WhitworthSEO
Exercise Caution!
Powerful crawl bloat tool when used effect-ively
![Page 23: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/23.jpg)
robots.txt
Blocking Crawl Paths – Stems Authority Flow
bancmedia.-com@WhitworthSEO
Google needs to see tags to honour them!
![Page 24: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/24.jpg)
google’s hint
Google gives you all you need to monitor your results
All these features are your friends
@WhitworthSEO bancmedia.-com
![Page 25: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/25.jpg)
crawl stats
Great way to gauge your efforts
@Whitworth-SEO
bancmedia.-com
![Page 26: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/26.jpg)
bancmedia.-com@WhitworthSEO
Pruning
the prune
![Page 27: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/27.jpg)
pruning – the fun bit
SEO Housekeep-ing
Now you have your crawl under control, show off your best bits
bancmedia.-com@WhitworthSEO
![Page 28: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/28.jpg)
pruning – why?
• Spinning
• Excessive blog-ging
• Sparse articles@WhitworthSEO bancmedia.-
com
• Over optimisa-tion
• Saturation
![Page 29: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/29.jpg)
stage one – content audit/content is king
Content auditing still essential nonethe-less
bancmedia.-com@WhitworthSEO
SEO’s most annoying cliché
![Page 30: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/30.jpg)
stage two: the prune
410 – Gone
@WhitworthSEO bancmedia.-com
For pages you want gone forever
![Page 31: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/31.jpg)
stage two: the prune
bancmedia.-com@WhitworthSEO
Won’t help crawl bloat short term
Will disappear eventu-ally*
![Page 32: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/32.jpg)
stage two: the prune
Crawl Bloat
Authority
Indexa-tion
@WhitworthSEO bancmedia.-com
![Page 33: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/33.jpg)
stage two: the prune
NoIndex
Will be crawled less over time ensures a high qual-ity SERP
For duplicate content, cannibalisation and API style pages
@WhitworthSEO bancmedia.-com
![Page 34: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/34.jpg)
stage two: canonical
For duplicate content, cannibalisation and API style pages
@WhitworthSEO bancmedia.-com
canonical
![Page 35: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/35.jpg)
stage three: content repurposing/enhance-ment
@WhitworthSEO bancmedia.-com
EvergreenSeasonalLong formResearch
![Page 36: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/36.jpg)
search quality guidelines
@WhitworthSEO bancmedia.-com
Search Quality Evaluator Guidelines
![Page 37: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/37.jpg)
stage four: the finishing touch
Link Profile Ana-lysis
Links pointing at ex-cluded pagesUsual disavow re-
quest
@WhitworthSEO bancmedia.-com
Don’t ruin all your hard work with a crap link profile
![Page 38: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/38.jpg)
crawl bloat and pruning checklist
Does your site have crawl bloat?Choose the appropriate fixes for your sitePrune your remaining content
Continually improve and enhance@WhitworthSEO bancmedia.-
com
![Page 39: Combatting Crawl Bloat & Pruning Your Content Effectively](https://reader035.vdocument.in/reader035/viewer/2022081605/58f9c216760da32f4b8b63c8/html5/thumbnails/39.jpg)
final tips: thanks for coming
@WhitworthSEO bancmedia.-com
Thanks for Listen-ing!
2 Tips