thousands of pages missing from google serps...and how to prevent the problem! by christine smith

#SMX #25A Christine Smith @websmithc

...and how to prevent the problem!

Thousands of Pages Missing from Google SERPs


§ >1M technical documents

§ Self-support for server and software admins

§ Searchable by error codes, etc.

IBM’s self-support site

http://www-01.ibm.com/support/docview.wss?uid=swg21363866


Thousands of pages missing from SERPs!


Google monthly referrals dropped 28%


ü Pages/URLs displayed correctly

ü Redirects were working normally §  302 www.ibm.com -> www-01.ibm.com (multiple infrastructures)

ü Canonical URLs were correct <link href=“http://www.ibm.com/support/docview.wss?uid=swg21363866" rel="canonical"/>

ü Robots.txt was not blocking anything

The Good News: The pages were fine


§ Only 10% of Sitemap URLs were indexed § Regenerated sitemaps

ü Corrected URLs to match canonical URLs ü Improved to 60% of Sitemap URLs indexed

ü  Today, 88% are indexed

§ But, still no Google referral improvement

Sitemaps were not optimal


§ Opened Google Site Search support ticket § Google findings: § A sampling of the missing URLs were:

1.  Marked as a duplicates of the Support Registration page and

2.  Were last crawled five months before!

Engaged Google


§  Since Google’s index marked the pages as duplicates…

Thousands of Support pages were effectively

deindexed

In other words…


§ There was no way to get a list of all URLs affected

§ Panda was updated and Data Refreshed within a few weeks

Meanwhile, according to Google


§ Submitted a Manual Actions Reconsideration Request §  Some technical docs were incorrectly flagged

§ Requested increased crawl rate for the domain §  Hoping the pages would be revisited

We kept pulling every lever available…


Google referrals increased 22% the following month


§ Traffic bounced back.. Almost overnight. § GSS ticket was closed, but not resolved § Was it.. Ø Increase crawl rate? Ø Better sitemaps? Ø Panda and data refresh? Ø Normal re-crawl of the pages?

What was the fix?


§ Likely culprit: §  Faulty redirect or §  A bad site maintenance redirect

§ Typical server response during outage: §  404 or 500 or 504 HTTP response §  Or 302 redirect to a maintenance page

More importantly: What was the cause?


§ Give a 503 Service Unavailable HTTP response!

§  Retry-after time in the header helpful §  Tells Google to come back later

§ Do not set all 5xx responses to 503 §  Google will ignore and assume site is down

Site maintenance – the right way

Reference: http://googlewebmastercentral.blogspot.com/2011/01/how-to-deal-with-planned-site-downtime.html


§ Some web platforms automatically give 503 HTTP response during an upgrade §  e.g. Wordpress

§ Others require workarounds, like §  Apache, IHS (IBM), IIS (Microsoft) rewrite rules §  Akamai logic

How to indicate site maintenance


RewriteEngine on RewriteCond %{REQUEST_URI} !^(/maintenance.html)$

#RewriteRule ^(.*)$ - [R=503]

#Planned outage for /services/ path

RewriteRule ^(/services.*)$ - [R=503]

#Planned outage for /support/ path

RewriteRule ^(/support.*)$ - [R=503]

ErrorDocument 503 /maintenance.html

Update httpd.conf during maintenance

•  Requires Apache / IBM HTTP Server restart to update httpd.conf before and after maintenance •  Remember to both port 80 and port 443 •  If the entire site is down, return a 503 response on robots.txt


Akamai flow for 503 HTTP response During planned outage, upload a filename matching the application path to a NetStorage /maintenance/ directory Custom Flow •  If Origin gives 500 or 504 response,

then •  If the maintenance file exists,

then •  Serve maintenance page

with 503 response •  Set retry-after to 1 day

•  Otherwise, serve error page matching Origin response (500/504)

•  Otherwise, serve Origin response


ü  Never give 200 or 301 HTTP response when site is down! ü  Confuses the crawler/indexer ü  May cause pages to be “deindexed” as duplicates

ü  Configure 503 HTTP responses only during planned outages and…. ü  Don’t forget to remove when maintenance is over!

Lessons Learned


THANK YOU!

SEE YOU @SMX WEST SAN JOSE, CA

MARCH 1-3, 2016


§  Apache: §  https://gist.github.com/jjulian/1889874

§  IBM HTTP Server (IHS): §  http://www.ibm.com/support/docview.wss?

uid=swg21397422 §  Microsoft IIS:

§  http://serverfault.com/questions/483145/how-to-add-a-site-wide-downtime-error-message-in-iis-with-a-custom-503-error-co

References: Rewrite Rules for 503 Response

thousands of pages missing from google serps...and how to prevent the problem! by christine smith

Internet