Stop Searching and Start Stop Searching and Start FINDING: Strategies for FINDING: Strategies for Effective Web ResearchEffective Web Research
a presentation bya presentation by
Ken Wiseman Ken Wiseman
& IMSA& IMSA
[email protected]@wisemantech.com
Our goals today ...Our goals today ...
Discover the biggest mistakes made by most Discover the biggest mistakes made by most Internet users: Internet users: Typing search terms in the wrong box.Typing search terms in the wrong box. Using the wrong tool at the wrong time.Using the wrong tool at the wrong time.
Talk about the differences between Talk about the differences between directories and search engines (and when to directories and search engines (and when to use each.)use each.)
Learn some advanced Google searching Learn some advanced Google searching techniques.techniques.
DO ALL OF THIS IN ENGLISH!DO ALL OF THIS IN ENGLISH!
To Start To Start
Put the right request in the right Put the right request in the right placeplace
Put Web addresses in the address box (this is the URL stuff that begins with http://)
Put search terms (the stuff you are looking for) in the search box
The Biggest MistakeThe Biggest Mistake
Thinking that search tools are Thinking that search tools are card catalogs of the webcard catalogs of the web
9 Billion pages reside on the Web 9 Billion pages reside on the Web (4/02)(4/02)
• No search tool indexes all of the web.• The largest -Google-indexes less than 30% of the total (3 billion).• Each engine indexes a different set of web pages.
Search Engine Size - Search Engine Size -
0
500
1000
1500
2000
2500
3000
3500
SE 3033 2106 1689 1453 1147 1018 1015 733 275
AlltheWe
AltaVista
Wisenut
Hotbot
MSN Teoma
NL Gigablast
Search Engine Showdown - 12/02
The Second Biggest The Second Biggest MistakeMistake
Using the wrong tool at the wrong Using the wrong tool at the wrong timetime
Three questionsThree questions
Where would you find the telephone number or Where would you find the telephone number or address of the Woodfield theatre?address of the Woodfield theatre? A telephone bookA telephone book
Where you would find the definition of the word Where you would find the definition of the word “pestilence?”“pestilence?” A dictionary (or in your school yearbook)A dictionary (or in your school yearbook)
Where would you find the name of the war that Where would you find the name of the war that the Treaty of Westphalia ended?the Treaty of Westphalia ended? An encyclopediaAn encyclopedia
What would happen if What would happen if you tried to look up the you tried to look up the definition of the word definition of the word
“pestilence” in the “pestilence” in the telephone book?telephone book?
YAHOO ISN’T A YAHOO ISN’T A SEARCH SEARCH ENGINE!ENGINE!
... it is a directory.... it is a directory.
(but this maybe changing)(but this maybe changing)
DirectoriesDirectories
Usually human-Usually human-compiled guides to the compiled guides to the web, where sites are web, where sites are organized by categoryorganized by category
Major directories:Major directories: MSNMSN YahooYahoo Netscape ODPNetscape ODP
DirectoriesDirectories How Internet Directories WorkVisit Web sitesEvaluatesDirectory EmployeeAdds & CatalogsDirectory ServerDirectory’s searchable index
Directory’sSearchable
index
Directory’sBrowsableCategories
SearchesUser BrowsesUser seeking informationreceives infoInternet/Web
Pages
What directories are good for ...What directories are good for ...
““What is the Web page address for some What is the Web page address for some company, organization, or entity?” (or “who company, organization, or entity?” (or “who makes product X?”)makes product X?”)
““Where can I find a list of Web pages that Where can I find a list of Web pages that focus on a particular, ‘universal’ topic?”focus on a particular, ‘universal’ topic?”
In other words, directories are In other words, directories are GREATGREAT for for “telephone book” searches.“telephone book” searches.
What directories AREN’T good for ...What directories AREN’T good for ...
Directories are Directories are horriblehorrible for for “encyclopedia” or “dictionary” searches.“encyclopedia” or “dictionary” searches.
The only exception is if the topic is so The only exception is if the topic is so universal that the directories have no universal that the directories have no choice but to link to a page or two that choice but to link to a page or two that discuss that topic (and even then the discuss that topic (and even then the selection will be slim.)selection will be slim.)
Search Engines have three parts:Search Engines have three parts:
1.1. A spider (also called a A spider (also called a "crawler" or a "bot") that "crawler" or a "bot") that goes to every page or goes to every page or representative pages on representative pages on every Web site that wants every Web site that wants to be searchable and to be searchable and reads it, using hypertext reads it, using hypertext links on each page to links on each page to discover and read a site's discover and read a site's other pages.other pages.
Search Engines have three parts:Search Engines have three parts:
2.2. A program that creates a A program that creates a huge index (sometimes huge index (sometimes called a "catalog") from the called a "catalog") from the pages that have been read.pages that have been read.
A program that receives your search request, compares it to the entries in the index, and returns results to you.
Directories vs Search EnginesDirectories vs Search Engines
Directories are human-compiled and have a Directories are human-compiled and have a small number of pages in their databases small number of pages in their databases (usually in the low millions)(usually in the low millions)
Search engines are machine-compiled and Search engines are machine-compiled and have a have a HUGEHUGE number of pages in their number of pages in their databases (usually in the hundreds of databases (usually in the hundreds of millions or even the billions)millions or even the billions)
The Second Biggest The Second Biggest Mistake -- RestatedMistake -- Restated
Using a directory as if it was a Using a directory as if it was a search engine ... and then not search engine ... and then not
understanding why you can’t find understanding why you can’t find anything!anything!
Top search sites – January 2002Top search sites – January 2002
1.1. MSNMSN
2.2. YahooYahoo
3.3. GoogleGoogle
4.4. AOLAOL
5.5. Ask JeevesAsk Jeeves
6.6. LookSmartLookSmart
7.7. InfospaceInfospace
8.8. OvertureOverture
9.9. NetscapeNetscape
10.10. AltaVistaAltaVista
-- Courtesy Jupiter Media Metrix
Which ones are directories?Which ones are directories?
1.1. MSNMSN
2.2. YahooYahoo
3.3. GoogleGoogle
4.4. AOLAOL
5.5. Ask JeevesAsk Jeeves
6.6. LookSmartLookSmart
7.7. InfospaceInfospace
8.8. OvertureOverture
9.9. Netscape ODPNetscape ODP
10.10. AltaVistaAltaVista
Why Use a Search Engine?Why Use a Search Engine?6 Billion+
3
2
1
0
Millions
Directory Name
Open Directory LookSmart MSN Yahoo NBCi/Snap Britannica google
Web Pages Indexed by Directories
Secondary resultsSecondary results
Most directories use a Most directories use a search engine as a search engine as a backup (Yahoo and backup (Yahoo and Netscape use Google, Netscape use Google, almost everyone else almost everyone else uses Inktomi)uses Inktomi)
Why add the extra Why add the extra step?step?
How the sites stack upHow the sites stack up
Most directories (like Most directories (like MSN and AOL) link to MSN and AOL) link to 2 or 3 million pages.2 or 3 million pages.
Most search engines Most search engines (like AlltheWeb and (like AlltheWeb and Google) link to Google) link to BillionsBillions of pages. of pages.
-- Courtesy searchenginewatch.com
Why do people Why do people predominantly use predominantly use
directories when search directories when search engines have more engines have more
stuff?stuff? Because no one Because no one ever takes the time to ever takes the time to teach us how to use a teach us how to use a
search engine!search engine!
The Third Biggest The Third Biggest MistakeMistake
Not knowing how to use Not knowing how to use directories or search engines to directories or search engines to
actually actually FINDFIND stuff stuff
Search engine rule #1Search engine rule #1
Be specific ... because if Be specific ... because if you aren’t specific, you’ll you aren’t specific, you’ll end up with a bunch of end up with a bunch of
garbage!garbage!
Preparing to SearchPreparing to Search Formulation of the research questionFormulation of the research question Identification of important concepts within the Identification of important concepts within the
question question Identification of search terms to describe those Identification of search terms to describe those
concepts concepts Consideration of synonyms and variations of Consideration of synonyms and variations of
those terms those terms Take a look at Take a look at Vivisimo Vivisimo
clustered results for helpclustered results for help Preparation of the search logicPreparation of the search logic
Search engine rule #2Search engine rule #2
Use quotes to search for Use quotes to search for phrases.phrases.
““ken wiseman”ken wiseman”
Use quotes for phrasesUse quotes for phrases To search for phrases, just put your phrase in To search for phrases, just put your phrase in
quotes.quotes. For example, For example, disney fantasyland disney fantasyland “pirates of the caribbean”“pirates of the caribbean” This would show you all the pages in Google’s This would show you all the pages in Google’s
index that contain the word index that contain the word disneydisney AND the word AND the word fantasylandfantasyland AND the phrase AND the phrase pirates of pirates of the caribbeanthe caribbean
By the way, while this search is By the way, while this search is technicallytechnically OK, my OK, my choice of keywords contains a (deliberate) factual choice of keywords contains a (deliberate) factual mistake. mistake.
Can you spot it?Can you spot it?
Arr, She Blows!Arr, She Blows!
Pirates of the Caribbean isn’t Pirates of the Caribbean isn’t in Fantasyland, it’s in in Fantasyland, it’s in Adventureland in Orlando Adventureland in Orlando and New Orleans Square in and New Orleans Square in Anaheim.Anaheim.
So searching for So searching for disney disney AND AND fantasyland fantasyland AND AND “pirates of the “pirates of the caribbean” caribbean” probably isn’t probably isn’t a good idea. a good idea.
Search engine rule #3Search engine rule #3
Use the + sign to Use the + sign to require.require.
Apple+computerApple+computer
Search engine math:Search engine math:+ & And+ & And
Limits your search
Apple & Computer
Only returns pages with both of these terms on them
Search engine rule #4Search engine rule #4
Use the - sign to Use the - sign to exclude.exclude.
apple -computerapple -computer
Search Engine Math:Search Engine Math:- & not- & not
Limits your search
Women not History
Only returns pages that contain one but not the other term on them
Boolean ORBoolean OR
Sometimes the default Sometimes the default ANDAND gets in the way. gets in the way. That’s where That’s where OROR comes in. comes in.
The Boolean operator The Boolean operator OROR is is alwaysalways in caps and in caps and goes between keywords.goes between keywords.
For example, an improvement over our earlier For example, an improvement over our earlier search would be search would be disney fantasyland OR disney fantasyland OR “pirates of the caribbean”“pirates of the caribbean” This would show you all the pages in Google’s index This would show you all the pages in Google’s index
that contain the word that contain the word disneydisney AND the word AND the word fantasylandfantasyland OR the phrase OR the phrase pirates of the pirates of the caribbeancaribbean (without the quotes) (without the quotes)
Three Ways to Three Ways to OROR at Google at Google
Just type Just type OROR between keywords between keywordsdisney fantasyland OR “pirates of the disney fantasyland OR “pirates of the caribbean”caribbean”
Put your Put your OROR statement in parentheses statement in parenthesesdisney (fantasyland OR “pirates of the disney (fantasyland OR “pirates of the caribbean”)caribbean”)
Use the Use the || (“pipe”) character in place of the word (“pipe”) character in place of the word ORORdisney (fantasyland | “pirates of the disney (fantasyland | “pirates of the caribbean”)caribbean”)
All three methods yield the All three methods yield the exactexact same results. same results.
Search engine math:Search engine math:OROR
Women or History
Returns every page with either of these terms on them
Broadens your search
OR, She Blows!OR, She Blows!
Just remember, Google’s Just remember, Google’s Boolean default is Boolean default is ANDAND
Sometimes the default Sometimes the default ANDAND gets in the way. gets in the way. That’s where That’s where OROR comes comes in.in.
How Insensitive!How Insensitive!
Google is Google is notnot case sensitive. case sensitive. So, the following searches all yield exactly So, the following searches all yield exactly
the same results: the same results: disney fantasyland piratesdisney fantasyland pirates
Disney Fantasyland PiratesDisney Fantasyland Pirates
DISNEY FANTASYLAND PIRATESDISNEY FANTASYLAND PIRATES
DiSnEy FaNtAsYlAnD pIrAtEsDiSnEy FaNtAsYlAnD pIrAtEs
Search engine rule #5Search engine rule #5
Combine symbols as often as Combine symbols as often as possible possible
(see rule #1).(see rule #1).
+”Martha Washington” –george +revolution+”Martha Washington” –george +revolution
The five rulesThe five rules
1.1. Be specific ... because if you aren’t specific, Be specific ... because if you aren’t specific, you’ll end up with a bunch of garbage!you’ll end up with a bunch of garbage!
2.2. Use quotes to search for phrases.Use quotes to search for phrases.
3.3. Use the + sign to require.Use the + sign to require.
4.4. Use the - sign to exclude.Use the - sign to exclude.
5.5. Combine symbols as often as possible Combine symbols as often as possible (see rule #1).(see rule #1).
6.6. Don’t forget ORDon’t forget OR
Did You Know…Did You Know…
that large chunks of the Web are invisible to that large chunks of the Web are invisible to most search engines.most search engines.
That no one has a good handle on the That no one has a good handle on the magnitude of the invisible web*magnitude of the invisible web*
That much of the invisible web is of great That much of the invisible web is of great value to educators & studentsvalue to educators & students
So What?So What?
Would you intentionally exclude large Would you intentionally exclude large chunks of the Library of Congress’ 12 million chunks of the Library of Congress’ 12 million documents from your searches?documents from your searches?
How about the US Census Bureau?How about the US Census Bureau? How about health and medical databases?How about health and medical databases? Many newspapers?Many newspapers?
What Today’s Search Tools What Today’s Search Tools Can and Cannot findCan and Cannot find
Not search tool specific Not search tool specific Search tools were created to handle flat Search tools were created to handle flat
HTML pages. HTML pages. When confronted with a search box the When confronted with a search box the
search tool is stopped unless it has specific search tool is stopped unless it has specific instructions on how to handle that input box.instructions on how to handle that input box.
Dynamically created web pages have Dynamically created web pages have unusual URLsunusual URLs
What Today’s Search Tools What Today’s Search Tools Can and Cannot findCan and Cannot find
The LII page for Automobile (http://lii.org/search/file/automobiles) is in Google;
The LII page for Motorcycles (http://lii.org/search?title=Motorcycles; query=Motorcycles; searchtype=subject) is not.
Do you see why NOT??
Simple ExamplesSimple Examples
Ken Wiseman - 602 Hits
None contain my contact info
But…
Typical Search PagesTypical Search Pages
Finding Specialized databases in Google…Finding Specialized databases in Google…
Adding “searchable database” reduced the hits from 14,000 to 140 and added additional resources not found in the standard search!
Index of Specialized DatabasesIndex of Specialized Databases
BeaucoupsBeaucoups Listing of specialized search toolsListing of specialized search tools Most of the information is invisible to general Most of the information is invisible to general
search enginessearch engines Over 2500 search tools listed.Over 2500 search tools listed.
Other Invisible Web Recovery Other Invisible Web Recovery sitessites
Don’t Forget Don’t Forget Portals…Portals…
http://www.skewlsites.com/
http://www.homeworkspot.com/
http://www.awesomelibrary.org/
http://www.about.com/