Chapter 19

Web Crawler

Copyright © 2005 Pearson Addison-Wesley. All rights reserved. 19-2

Chapter Objectives

• Provide a case study example from problem statement through implementation

• Demonstrate how hash tables and graphs can be used to solve a problem

Copyright © 2005 Pearson Addison-Wesley. All rights reserved. 19-3

Web Crawler

• A web crawler is a system that searches the web, beginning with a user-designated we page, looking for a designated target string

• A web crawler follows all of the links on each page that it encounter until there are no more pages or until it reaches a designated limit

Copyright © 2005 Pearson Addison-Wesley. All rights reserved. 19-4

Web Crawler

• For this case study, we will create a graphical web crawler with the following requirements– Enter a designated starting web page

– Enter a target string for which to search

– Limit the search to 50 pages

– Display the results when done

Copyright © 2005 Pearson Addison-Wesley. All rights reserved. 19-5

Web Crawler - Design

• Our web crawler system consists of three high-level components:– The driver

– The graphical user interface

– The web crawler implementation• Makes use of graphs and hashtables

Copyright © 2005 Pearson Addison-Wesley. All rights reserved. 19-6

Web Crawler - Design• The algorithm for the web crawler is as follows

– Add the starting page to a HashSet of pages to be searched and to our graph

– Remove a page from the set of pages to be searched

– Search the page for the target string• If string exists, add page to list of results

– Search the page for links• If links have not already been searched, add them to set of

pages to be searched and to our graph

– Repeat the three previous steps until our limit is reached or the set is empty

Copyright © 2005 Pearson Addison-Wesley. All rights reserved. 19-7

FIGURE 19.1 User interface design

Copyright © 2005 Pearson Addison-Wesley. All rights reserved. 19-8

FIGURE 19.2 UML description