The art of Information Engineering » Crawling http://www.grok.in (ignorance killed the cat, curiosity was framed) Tue, 11 Aug 2009 06:30:18 +0000 en hourly 1 Web Crawling with Perl http://www.grok.in/blog/2008/06/09/web-crawling-with-perl/ http://www.grok.in/blog/2008/06/09/web-crawling-with-perl/#comments Sun, 08 Jun 2008 19:13:25 +0000 Siddhartha Reddy http://www.grok.in/?p=49 If you are looking to write a web crawler, Perl, with all its great CPAN modules, is one of the best platforms you can pick. There are CPAN modules for most of the common components of a web crawler. Here, I’ll point to some of the modules that you would want to start out with.

Read on…

]]> http://www.grok.in/blog/2008/06/09/web-crawling-with-perl/feed/ 3 Introduction to Web Crawling http://www.grok.in/blog/2008/06/07/introduction-to-web-crawling/ http://www.grok.in/blog/2008/06/07/introduction-to-web-crawling/#comments Sat, 07 Jun 2008 16:40:17 +0000 Siddhartha Reddy http://www.grok.in/?p=35 In the context of the World Wide Web, crawling refers to gathering web pages, by following hyperlinks, starting from a small set of web pages, for the purposes of further processing. For example, a Web search engine needs to gather as many pages as possible before it indexes and makes them available for searching.

A program which performs crawling is variably known as a crawler, a spider, a robot or simply a bot. The set of pages from which the crawler starts crawling is known as seed list.

Although it seems pretty straightforward, writing a good web crawler is not very much so. There are a good number of challenges which vary subtly depending on whether it’s a large-scale web crawler or a crawler for a handful of websites. These challenges include: ensuring politeness to the web servers (by observing the widely accepted robots exclusion protocol), URL normalization, duplicate detection, avoiding spider traps, maintaining a queue of un-fetched pages, maintaining a repository of crawled pages, re-crawling and a few more. For large-scale crawlers, one of the most important challenges is to increase the throughput by optimizing the resource utilization, because their coverage usually gets limited by this.

Read on…

]]> http://www.grok.in/blog/2008/06/07/introduction-to-web-crawling/feed/ 5