Grokking the System Design Interview
Ask Author
Back to course home

0% completed

Vote For New Content
Designing a Web Crawler
On this page

1. What is a Web Crawler?

A web crawler is a software program that browses the World Wide Web in a methodical and automated manner. It collects documents by recursively fetching links from a set of starting pages. Many sites, particularly search engines, use web crawling as a means of providing up-to-date data. Search engines download all the pages to create an index on them to perform faster searches.

Some other uses of web crawlers are:

  • To test web pages and links for valid syntax and structure.
  • To monitor sites to see when their structure or contents change.

.....

.....

.....

Like the course? Get enrolled and start learning!

On this page