searchlove boston 2016 | mike king | developer thinking for seos

DEVELOPER THINKING FOR

SEOS

MICHAEL KING

MANAGING DIRECTOR, IPULLRANK

@IPULLRANK

Baby Sign Language

…and how it relates to technical SEO

IPULLRANK.COM @ IPULLRANK

So, I’m Having a Baby (She is too)…

It’s a girl!


Last Night I Learned Infants Can Learn Sign Language

…


Which is Great Because They Act like Drunks Otherwise

…


Which is Astonishingly Similar to the Perception of

Marketers

SEOs generally know what they want, just not how to effectively ask for it.

As of Late People Started Saying You Don’t Need to Be Technical


Yet, the Web is More Technical than Ever Before

The modern SEO needs to think like a developer because we are in a technical SEO renaissance.


Lucky for You This Is The Point Where I have My Own Agency

Shameless plug.

Changes in Web Technologies

What is pushing this renaissance?


JavaScript is Not Going Away

Nearly every site on the web is using JavaScript to perform some type of transformation on the page. It’s obvious that Google needed to solve for this.


Growth of AngularJS

AngularJS, the single page application, MVW framework continues to grow dramatically in the top million websites.


AngularJS is Popular in Most Verticals

It’s likely that you’ll run into an AngularJS site soon enough in your optimization duties


ReactJS is Growing Too

Similarly, Facebook’s MVW framework is growing as well. Although, it has a built in feature for making sure that it renders proper HTML, but it’s still a framework you should know how to optimize for.

Check out this post https://builtvisible.com/react-js-seo/


Ironically, React (Facebook) is SEO-ready, Angular (Google) is Not

Angular 1 did not have anything to make it easier to crawl. React has ReactDOMServer.renderToString to allow React to serve fully rendered HTML from the server.

View Source is Dead

Inspect Element Only


HTTP/2 – Perhaps the Most Important Thing

All you need to know right now is that it’s fast as hell.

Adoption is Super Low For Now Though

We Need More from Our SEO Tools

Many of them are far behind search engines

What are rankings in 2016?

So, You’re #1? Cool Story, Bro.

Where they at tho?


On What Device, OS and Location?

Different browser and OS configurations will show you different results. So what truly is a mobile ranking?

+ + = Ranking #3+Nexus 5 Android NYCChrome

Safari

+ + = Ranking #1+iPhone 6 iOS NYC

Are we measuring for contextand actionability or just vanity?

Please Give Me Position 0 in My Rankings

Please Give Me My Results in Context of Paid

#1 is really #8 in this SERP

What is cloaking in 2016?


304 Response Codes

The Not-Modified response code tells clients that they do not need to download a page again if it has already been downloaded.


Googlebot Respects the 304

What is cloaking when Google accepts 304 response codes and does not revisit that content after indexation?


What is Cloaking in the Adaptive/Responsive Era?

Many people show less or more content based on the context of the browser/user, so what is cloaking now?

Google accepts directives in HTTP headers.


X-Robots-Tag HTTP Header

Many of the popular crawling tools do not look for this in the response headers.


Hreflang HTTP Header

I have not seen a tool that looks for this HTTP header. However, this implementation is preferred over the many lines of hreflang tags on every page.


Rel-Canonical HTTP Header

Many tools do not look for the rel-canonical HTTP header.

All of these HTTP headers are better than clogging up the <head> of the code.


JavaScript Redirects for Mobile

Google allows client side redirects for automation redirection of mobile to desktop and vice versa. Many crawling tools do not account for this.

Many SEO tools do not account for these things


Chrome DevTools Does

Natively in Chrome’s Developer Tools you can see all of the HTTP Request and Response Headers in the Network tab section.


Geolocation Emulation is Built-in to Chrome

If you’re running into issues with finding local rankings now that Google has gotten rid of nearEquals you can just change your location on the browser level.


Remote Debugging

You can also see this for a specific mobile devices with the remote debugging features of Chrome.

Crawlability

What can be crawled at this point?


Google is Crawling Headless

Make no mistake that Google has not been crawling with just text driven crawlers for a long time.


Adam Audette Said It So You KNOW It’s True

Adam ran a few tests a little under a year ago further proving out these capabilities


Also an Ex-Googler Confirmed It

A former Google developer said the headless crawling was his primary project from 2006 – 2010.


I Said it in 2011


JsCrawlability.com

Eric Wu has been doing tests to get a better determination of what it is that Google can actually see. http://www.jscrawlability.com

But Mike, what does it matter?


Well I Captured Video of Headless Googlebot

It’s a question of what does Googlebot do on your site. Using LuckyOrange I captured Googlebot sessions from Search Console’s Fetch and Render and then I submitted it to the index. They load the page and leave. The mouse never moves, the headless browser never came back.


Get a Free Trial of LuckyOrange and Try it Yourself

This code will only place LuckyOrange on the page if a JS-enabled Googlebot comes to your site. My test did not show what Googlebot from the wild does when it’s crawling.


Log Files Went out of Style?

No, log file analysis has not gone out of style. In fact, it’s more valuable than ever before.

LOOK AT YOUR LOG FILES!


Log File Analysis

We parse the logs and query them in MySQL, but you can use any number of tools for this.


Screaming Frog Log File Analyzer

The Screaming Frog team rolled out an awesome new tool recently for log file analysis.


Search Console’s Crawl Stats are Mediocre

I personally find what Google gives us to be pretty much useless. Don’t less this take the place of going through your logs.


Log Files Help You Spot What Really Happens

I had a client come last year that was unsure whether it was Penguin or changes in their offline media buying habits had led to the downturn in traffic. Layering the Googlebot logs made it clear.

0

50000000

100000000

150000000

200000000

250000000

0

500000

1000000

1500000

2000000

2500000

Unknown organic organic home organic dyor

organic er organic home,dyor,er organic cushion organic net of home

organic ritani organic diamonds direct home paid brand sessions

tv spend web sessions (as reported by adaptv) Top 800 Monthly SV impressions

Googlebot Visits

Penguin Update


Social Shares Influence Crawl Patterns

We find that there is a higher correlation with crawling and social shares than crawling and links. Commonly held SEO knowledge would make you believe this is entirely dictated by links.

0

1000000

2000000

3000000

4000000

5000000

6000000

0

100000

200000

300000

400000

500000

600000

700000

Googlebot Visits Links Social Shares FB Twitter G+


Prerendering is the Best Practice

Using a solution like prerender.io is the best practice.


Prerender Can Cause Problems

However, we’ve seen Prerender setups misidentify Googlebot as a human users and serve the wrong the AngularJS versions.

0

20000

40000

60000

80000

100000

120000

Googlebot Perceived as Human


Prerender Setup Serving Non-200

Prepping the prerender cache can cause outages and lead to pages falling in and out of the index.

0

2000

4000

6000

8000

10000

12000

14000

16000

20

6

30

1

30

2

30

4

40

3

40

4

40

5

41

0

50

0

50

3

50

4

20

6

30

1

30

2

30

4

40

3

40

4

41

0

50

3

50

4

20

6

30

1

30

2

30

4

40

4

40

5

41

0

50

0

50

3

50

4

20

6

30

1

30

2

30

4

40

1

40

4

40

5

41

0

50

0

50

3

50

4

20

6

30

1

30

2

30

4

40

3

40

4

40

5

41

0

50

2

50

3

50

4

1 2 3 4 5

-

hit

miss

pass


Also Prerendering is Not a Requirement

We got rid of prerender on 2 sections of the site and saw the traffic from Organic Search increase YoY. Google can index it just fine.


AJAX Crawling Deprecated

Google has gotten rid of the AJAX crawling scheme because they don’t need it anymore. They have perfected their ability to crawl headless at reasonable scale. They say don’t block the CSS or JS because it’s a browser.

They Say Don’t Prerender Just for Google


Prerender Does Speed Up Crawling

There is certainly value in prerendering for speed, but then that makes me wonder, why not just not use Angular?


A Better Idea Is a Library that Doesn’t Need Prerender

Let’s stop going after the shiny object and go with the things that are built to work, shall we? Check out MeteorJS – http://www.meteor.com


Or go with Angular2

Brad Green is the Google engineering product manager for AngularJS. He says Angular2 is better in every way.


Don’t Forget Google Wants Progressive Enhancement

Thinking with the IoT in mind, progressive enhancement still makes sense. However, frameworks like AngularJS are completely contradictory in that they don’t offer graceful degradation.

Scraping

The backbone of a lot of our analysis as SEOs


Most Tools use cURL or Similar

There are many similar libraries, but many SEO tools are built on library called cURL for managing HTTP requests. This is a text based crawler.

This is a fundamental flaw of many SEO tools because they are not constructing the DOM and its CSS/JS transformations. Search Engines are.


PhantomJS – For Scraping the Modern Web

There are many headless browsers out there such as Selenium, but PhantomJS is the go to. http://www.phantomjs.org


HorsemanJS – The Easy PhantomJS

I prefer HorsemanJS which is a NodeJS module that is incredibly simple. http://www.horsemanjs.org


In-Browser Scraping

I wrote a post on how you can scrape using your browser’s Console.


Scrape the Unscrapable

Nothing is in unscrapable when you’re using a browser.


Multi-Page In-browser Scraping

If you want to scrape multiple pages check out ArtooJS https://medialab.github.io/artoo/


WebScraper.io

WebScraper is another high-powered in-browser scraping tool. No code required. http://www.webscraper.io


Or You Can Make Screaming Frog Headless

You can make Screaming Frog headless by placing a proxy with a headless browser in front of it. From my initial findings, there is not much different aside from the order it identifies URLs and its ability to scrape content from pages.


Here’s the Code

Here’s the node code for how I did it. HorsemanJS is not the best tool for the proxy, but it’s great for a proof of concept.

Screaming Frog pings for robots.txt a lot. Make sure you handle it with a cURL request rather than a headless one.

For speed, don’t load images

Make sure you’re setting the right headers before your return the HTML. HorsemanJS does not return headers.


PHP Wrapper for PhantomJS

For those of you that prefer PHP, the PHP PhantomJS wrapper is pretty decent as well. http://jonnnnyw.github.io/php-phantomjs/

For speed, don’t load images

Content & Linking

You knew I wasn’t going to get up here and not talk about these things, right?


Google Looks at Entities First

According to Paul Haahr, Google looks at entities first and it appears that we as an industry have talked about this in theory, but not truly worked it into the optimization workflow.


AlchemyAPI

I encourage you to work AlchemyAPI into your keyword research and content optimization process.


Term Relevance

There’s also been a lot of talk about co-relevance, topic modeling and TF*IDF. This is a great opportunity for technical elements to influence content creation.


In Fact Just Do Everything Cyrus Says

https://moz.com/blog/7-advanced-seo-concepts


Moz’s New Tool Will Help

Moz has made the related keywords concept more accessible by adding it to Moz Pro. Searchmetrics also has a similar solution.


Onpage.Org’s TF-IDF Tool is Also Great

http://onpage.org


John Mueller Says 302s Pass PageRank

Don’t let Google mislead you about 302s vs. 301s.


These Rankings Say He’s Wrong

We had a client with millions for 302s and all we did was change them to 301s

June August September

HyperTraffic (1-3) 1272 1,283 1,473

Traffic (4-10) 979 983 837

Striking Distance (11-20) 167 161 110

Emerging (21-40) 106 103 98

Developmental (41-100) 156 90 87

Unranked 0 60 75

0

200

400

600

800

1000

1200

1400

1600301 redirects


This Traffic Improvement Says He’s Wrong

Traffic went up pretty dramatically despite seasonality.

August 27th301 redirects

11,039,18810,455,517

11,348,902

12,049,481

11,250,148

12,902,847

11,040,192

13,158,296

11,287,790 12,203,99812,508,164

13,158,296

14,041,313

15,609,390

0

2,000,000

4,000,000

6,000,000

8,000,000

10,000,000

12,000,000

14,000,000

16,000,000

18,000,000

April May Jun Jul Aug Sep Oct

Organic Visits

2014 2015

But. We need to stop saying ‘just do 301 redirects.’


Instead Do Optimal 301 Redirects

Pattern matching on the .htaccess level is always faster than a list of redirects and always better than doing it on the code/page level. Developers will respect that you understand what you’re talking about.

ReWriteRule url_pattern

file_reference [R=301, L]

redirect 301 /relative/path/to/file.php

http://www.yoursite.com/path/to/new/file.php





>





Internal Linking Structures

Internal Linking Structures are still an incredibly powerful asset. I’d like to see more tools like what Portent has built to visualize them and understand where there is opportunity. https://www.portent.com/blog/seo/seo-force-directed-diagrams.htm


Searchmetrics Does a Great Job of This

I wish all the link indices had a stronger focus on internal linking structure and visualizing where there are opportunities to flow more link equity throughout the site. Searchmetrics’ Site Structure tool does this well.


Paul’s Method is Good for Rolling Your Own

http://searchengineland.com/improve-internal-linking-calculate-internal-pagerank-r-246883

If Possible, Always Update your internal links on the database level

WordPress Example

UPDATE wp_posts SET post_content = REPLACE (post_content, 'http://www.oldsiteurl.com', 'http://www.newsiteurl.com');

Google is becoming the presentation layer of the web. Help them do it.


Schema.org / JSON-LD

Schema.org is a lot more reasonable to implement now that they are allowing JSON-LD. Being in knowledge boxes and such yields a lot more traffic than it takes away.

Page Speed

Google’s next big play


Google Expects Ludicrously Speedy

Google has an expectation that the above the fold content on the page will load within 1 second. You basically get 400 ms to make that happen.


The Critical Rendering Path


DevTools Is Your Friend

In the Timeline section, DevTools gives you a point by point break down of each operation and how it impacts the site’s load time. This is how you can determine what exactly is slowing down your pages.


How Your Code Is Laid Out Matters A LOT


This Is what PageSpeed Insights Helps With


External Resources Also Slow Down the Page

Chartbeat is typically a huge culprit and using rel-dns-prefetch can speed this up considerably.


AMP is a Spec to Enforce the Critical Rendering Path

Sound familiar?


Responsive Sites are SLOOOOOOOW

One of the key things you can do, aside from optimizing for the critical rendering path, is conditional loading.


PreBrowsing Directives

Browsers have preloading directives which allow the page to download elements in the background for later viewing or within the same session.


Rel-Preconnect

Rel-preconnect will resolve the DNS, start the TCP handshake and negotiate the TLS tunnel beforehand, effectively shaving hundreds of milliseconds to several seconds in page load.


When to Use Rel-Preconnect

When you see a long time for connections to happen or a lot of idle time, it’s a good idea to use rel-preconnect.


How Rel-Prerender Works

It loads the page in an invisible tab to make the page appear instantly.


Not to Be Confused with Prerender.io


Google Uses It in the SERPs

This could potentially be a way to definitively identify navigational queries. h/t @yoast


I Sent Thousands of Headless Visits


Rel-Prerender’d Pages Generally Perform Better


Use it With the GA API

Get the pageviews of the ga:pagePath based on the ga:previousPagePath set to the current page that you’re on.

You’ll get a list of the pages your users are most likely to visit next. Set one of these as the rel-prerender.


Code.

Inject rel-prerender into the page programmatically.

You could also session the URLs the user is going to toensure you are always prerendering a new page.


It Improved Our Site Speed 68.35%

One line of code did this magic.

Don’t use it on mobile.

Analytics packages aside from Google Analytics may show fake sessions.


Defer JS Load with Page Visibility API

…


Rel-Preload for Same Page Preloading

I haven’t played with it yet, but Ian says they’re using it on the Portent site. https://www.smashingmagazine.com/2016/02/preload-what-is-it-good-for/

How to Go Forward

Things you should know and do to be prepared for the technical renaissance


Understand the Document Object Model

https://developer.mozilla.org/en-US/docs/Web/API/Document_Object_Model/Introduction


Critical Rendering Path

https://developers.google.com/web/fundamentals/performance/critical-rendering-path/analyzing-crp?hl=en


Page Speed

https://www.portent.com/blog/design-dev/ultimate-site-speed-guide-what-impacts-site-speed.htm


Log File Analysis

https://builtvisible.com/log-file-analysis/


Understand SEO for JavaScript Frameworks

https://builtvisible.com/javascript-framework-seo/


Learn DevTools

http://ipullrank.com/6-things-you-should-know-about-in-chrome-devtools/


Accelerated Mobile & Instant Pages

Even if you’re not a publisher, I encourage you to watch these projects as I expect that Google and Facebook will grow the specsdramatically once they’re adopted.


Read Up On HTTP/2

https://moz.com/blog/http2-a-fast-secure-bedrock-for-the-future-of-seo


Get Back to Testing All The Things

Let’s stop chasing the content train and get back to making experiences that perform.

That’s All I’ve Got

THANK YOU

Michael King

Managing Director, iPullRank

[email protected] http://ipullrank.com

@iPullRank

searchlove boston 2016 | mike king | developer thinking for seos

Marketing