How Search Engine Works

The Main Parts of a Search Engine

• Spider (or “web crawler”)

• Indexer

• Search software (an algorithm)

Lets us Start with SPIDERS

• A web crawler (also known as a web spider or web robot) is a program or automated script which browses (crawls) the World Wide Web in a methodical, automated manner.This process is called Web crawling or spidering.

• In short, All websites are codes and these codes are read by spiders.

• So basically a Spider or Spiders are software programs that crawl throughout the internet and select relative data to be INDEXED.

• When we Search in a search engine is actually searching the index of that search engine and not the whole internet.

• Spiders start by fetching few web pages then they follow the links on those pages and fetch the pages they point to, and follow all the links on those pages and fetch the pages they link to and so on.

• Until they've indexed a pretty big chunk of the web, many billions of pages stored across thousands of machines.

Let US SEE what is

INDEXING ?

Is It Easy to Search here?

Or Here?

• This is the equivalent form of the information that our spiders acquire after they finish with crawling.

• Random organization and NO STRUCTURE.

• The information available throughout the world wide web is in all kinds of shapes sizes and FORMATS.

• AND This is what indexing does.

• Makes data accessible in a Structured format, easily accessible through search.

Indexer

• Search engine indexing is the process of a search engine collecting, parses and stores data for use by the search engine.

• The actual search engine index is the place where all the data the search engine has collected is stored.

• It is the search engine index that provides the results for search queries, and pages that are stored within the search engine index that appear on the search engine results page.

What is a Search Algorithms ?

• Basically, a search engine algorithm is a set of rules, or a unique formula, that the search engine uses to determine the significance or rankings of a web page, and each search engine has its own set of rules.

• The algorithms, as they are different for each search engine, are also closely guarded secrets.

• Search algorithm sorts on the basis of many things like location of keyword, synonyms, adjacent words, etc

• But there are certain things that all search engine algorithms have in common.

• Relevancy

• Individual Factors

• Off-Page Factors

Relevancy• This is the First thing every search engine

checks.• The algorithm will determine whether this web

page has any relevancy at all for the particular keyword.

• Location of keywords in that page is also important for the relevancy of that website.

• Web pages that have the keywords in the title, as well as within the headline or the first few lines of the text will rank better for that keyword than websites that do not have these features

Individual Factors •  A second part of search engine algorithms are

the individual factors that make that particular search engine different from every other search engine out there.

• Each search engine has unique algorithms, and the individual factors of these algorithms are why a search query turns up different results on Google than MSN or Yahoo!.

• One of the most common individual factors is the number of pages a search engine indexes.

• They may just have more pages indexed, or index them more frequently, but this can give different results for each search engine.

• Some search engines also penalize for spamming, while others do not.

Off-Page Factors • Another part of algorithms that is still individual

to each search engine are off-page factors. • Off-page factors are such things as click-through

measurement and linking. • The frequency of click-through rates and linking

can be an indicator of how relevant a web page is to actual users and visitors, and this can cause an algorithm to rank the web page higher.

• Off-page factors are harder for web masters to craft, but can have an enormous effect on page rank depending on thesearch engine algorithm.

• Search engine algorithms are the mystery behind search engines, sometimes even amusingly called the search engine’s

“Secret Sauce”. • Beyond the basic functions of a search

engine, the relevancy of a web page, the off-page factors, and the unique factors of each search engine help make the algorithms of each engine an important part of the search engine optimization design.