Cataloging the web was doomed from the start. In the summer of 1993, Matthew Gray created the World Wide Web Wanderer (WWWW), arguably the first internet bot and web crawler. During its first official attempt to index the web, The Wanderer returned from its expedition with 130 URLs. But even in the baby years of the internet, this list was incomplete.
To understand how a simple web crawler works, imagine making a travel itinerary that contains three cities: New York, Tokyo, Paris. While visiting each destination, listen for any mentions of other places and add those to your itinerary. Your world crawl is complete when you have visited all of the cities on your ever-growing list. Will you have seen a lot of places by the end of your journey? Undoubtedly. But will you have seen the whole world? Almost certainly not. There will always be cities, or entire webs of cities, that are effectively invisible to this process.
A web crawler similarly consults a list of URLs and recursively visits any links it sees. But the resulting index should not be confused with a comprehensive directory of the internet, which does not exist.
Please select this link to read the complete article from WIRED.