Comments on: Introduction to Web Crawling

By: Michael Wolf

Michael Wolf — Sun, 01 Nov 2009 11:06:28 +0000

The “read more…” link from *this* article goes to the full text of a *different* article

By: Jack

Jack — Thu, 16 Oct 2008 09:48:24 +0000

Check this link about web crawling it’s very interesting,

http://crawltheweb.blogspot.com/

By: Siddhartha Reddy

Siddhartha Reddy — Mon, 09 Jun 2008 05:45:40 +0000

@sanket,

Converting relative URLs to absolute is necessary, but, I was referring to canonicalization of URLs — different URLs can be referring to the same resource. For example, http://www.grok.in:80/ and http://www.grok.in/ refer to the same resource. Canonicalizing URLs will ensure that these different URLs are considered as the same; in the previous example, both the URLs will be normalized to one of the two forms.

By: sanket

sanket — Mon, 09 Jun 2008 03:55:51 +0000

What is *URL Normalization*? Converting relative to absolute addresses or something like that?

By: Ujj

Ujj — Sat, 07 Jun 2008 17:22:17 +0000

good to know you write crawlers. I was doing research in Grid Computing sometime back when I dived into crawlers and fell in love with the whole concept and the math involved, is a field to spend a lifetime in. Good post.