Building BobSmithPhotography.net has stopped expanding because I've been overwhelmed with spam on the comment forums. So, if you're here and have questions about web design come and visit Forums, Blogs, Wikis dot com. It has articles that I've written, some of which are also here, and a web design forum as well so you can ask any questions you may have. There is still a lot of good stuff here though, so poke around the links and take a look.
Web CrawlerA web crawler, or spider, is a program that visits web pages keeping an index of what it finds. Most web crawlers are used by search engines to build their indexes. Some crawlers are used by email harvesters that use the email addresses they find for spamming. Well behaved web crawlers follow the robots.txt standard so that you can restrict where they crawl. See Also: |
 |