role of text mining in search engine
DESCRIPTION
Please leave me a mail at [email protected] if you like the content of document.TRANSCRIPT
Role of Text Mining in Search Engines
IST 345 Term Project
Team 3
Fall 2008
Role of Text Mining in Search Engine
Agenda
1111 IntroductionIntroduction
2222 Current TrendsCurrent Trends
3333
4444 Future TrendFuture Trend
5555
6666 ConclusionsConclusions
Role of Text Mining in Search Engine
Text Mining in Search EngineText Mining in Search Engine
Case StudyCase Study
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Really
RESULTS
How do Search Engine Work?
Role of Text Mining in Search Engine
1. New Website is posted, Linked to, or has its content altered
Role of Text Mining in Search Engine
2. Search Engine’s Spider Crawl the page
Role of Text Mining in Search Engine
Goglebot: Google
Slurp: Yahoo
MSNbot: MSN
Skims text, image descriptions, meta data, page titles and URL
Role of Text Mining in Search Engine
Follow links, count links In and Out
Role of Text Mining in Search Engine
SearchWikiEngineSEOSEM
Web
Role of Text Mining in Search Engine
Index key terms, count word frequency
Why Mining in Search Engine?
Overwhelming information in typical user query results
Results are only partly related to each other
Many users investigate only the two or three top ranked documents
Traditional lists of ranked documents do not seem to be sufficient for the exploratory search tasks
Role of Text Mining in Search Engine
Additional techniques are needed to help the users analyze the search results efficiently and drill down to the information they are looking for.
Users need to explore the information• Discovering new patterns, • New entities, and • Knowledge they do not even realize they needed
Role of Text Mining in Search Engine
Why Mining in Search Engine?
Text Mining
Text mining applications today draw on a wide range of techniques and serve many purposes in information management and business intelligence.
TM techniques can be organized into four categories:
1) Classification techniques
2) Association analysis
3) Information extraction techniques
4) Clustering techniques
Role of Text Mining in Search Engine
Use of Text Mining in Search Engine
Text categorization (faceted search systems)
Using multi-dimensional categories to describe (groups of) documents Richer descriptions Expensive to develop the categories
Semantic Web Search
Linguistic analysis of text Addition to purely statistical techniques
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Contextualized clustering
Group the search results by topic Clustering of documents according to terms found in the documents
Use of Text Mining in Search Engine
Clustering Engines
Scatter/Gather
Grouper
The Lingo system
The Clusty/Vivisimo engine (www.clusty.com and www.vivisimo.com)
SnakeT
HOBSearch
Role of Text Mining in Search Engine
Clusty / Vivisimo Engine
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Future Trends
Anticipated to expand 1000 times
With the introduction of full-text search engines such as AltaVista, Excite, HotBot, Infoseek, Lycos, and Northern Light, the Web can be viewed as a searchable 15-billion-word encyclopedia.
Nutch engine is the future search engine
Fetch several billion pages per month Maintain an index of these pages Search the index up to 1000 times per second Provide very high quality search results Operate at minimal cost
Future Trend Introduction of full-text search engines such as AltaVista, Excite, HotBot, Infoseek, Lycos, and Northern Light
Summarization of online documents
More efficient Categorization/Clustering for the search results
Entity Extraction by using linguistics and pattern detection
Answering intelligent questions
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine
Case Study: Data Crow
Role of Text Mining in Search Engine
Case Study: Data Crow
Role of Text Mining in Search Engine
Customized Search
Music
Data Crow contains two separate modules The Music Album and Audio CD module
Use one of the online services (MusicBrainz, Amazon, Discogs and others) to find information on your CD and or music files
Parse information from your mp3, flac, ape and or ogg file and fill missing information using an online service
Role of Text Mining in Search Engine
Conclusion
Learnt the evolution of Search Engine
Efficiency of Search Engine can be increased by using mining techniques
Increased demand from customized search
Role of Text Mining in Search Engine
Which of the following is not a search engine?
• Google• Open Directory• Yahoo search• Lycos
Open Directory
Role of Text Mining in Search Engine
What search engine has the largest index of listings on the web?
• Yahoo• Google• MSN• Microsoft
Role of Text Mining in Search Engine
Search Engines and directory are both the same thing because:
• They both index information• They’re both on the internet• They’re not the same thing• They both search for information
They’re not the same thing
Role of Text Mining in Search Engine
What was the first search engine ever created?
• WWW Wanderer• Google; created by Larry and Sergey in 1995• Yahoo; created in 1994• MSN Search; created in 1982
WWW Wanderer: technically not really a search engine, but a pioneer of the crawling process
Q: Google is limited to how many search terms in one query:
• 16• 18• 13• 15
15
Role of Text Mining in Search Engine
Q: How do search engines find out about sites on the Web?
• Osmosis• Search engines automatically know everything on the Web.• Two different ways: search engine spiders index the information, or site owners submit it manually• Search engines have special features that enable them to know when your site is uploaded. It's called "crystal ball technology."
Two different ways: search engine spiders index the information, or site owners submit it manually.
Role of Text Mining in Search Engine
Role of Text Mining in Search Engine