Web Crawling with Perl

If you are looking to write a web crawler, Perl, with all its great CPAN modules, is one of the best platforms you can pick. There are CPAN modules for most of the common components of a web crawler. Here, I’ll point to some of the modules that you would want to start out with.

Read on…

This entry was posted in Crawling and tagged , . Bookmark the permalink. Post a comment or leave a trackback: Trackback URL.


  1. Posted October 16, 2008 at 3:27 pm | Permalink

    check this article about web crawling,


  2. edmar
    Posted November 25, 2008 at 11:49 pm | Permalink

    the blog is nice..but it is be nice if it will be included with examples..like for WWW::Mechanize..


  3. Posted April 1, 2009 at 11:59 am | Permalink

    Thanks for the useful information.

    kiran Kumar. Bodi

  4. roboto
    Posted April 22, 2011 at 2:15 am | Permalink

    Hi Siddhartha Reddy,

    Your article is very interesting. I am currently working on my pet project that involves website crawling. My dillema is should i use Java or perl for webcrawlers. What language is better in terms of performance?

  5. Posted September 18, 2013 at 1:28 am | Permalink


  • About grok.in

    This is a blog primarily focussed on the subjects of Information Engineering—Retrieval, Extraction & Management, Machine Learning, Scalability and Cloud Computing.