26. What is a Search Engine Spider or crawler or robot?

A spider (not the wriggly 8 legged creature J), also known as a robot or a crawler is one of the most spoken about characters, when discussions hover around SEO or search engine optimization.

So what is a spider?

It is a program that follows, or "crawls", links grabbing content from websites or pages adding it to pre-defined databases. A search engine spider  does the same job throughout the Internet, and adds/updates content on large search engine indexes. Thus the search engine spider also named, robot, crawler or bot is used by search engines to find new content and provide up to date data to the users, who search for information.

While crawling through websites and web pages, spiders follow links from one page to another and from one site to another and this is the primary reason you will find optimization experts stating that links to your website (aka inbound links or backlinks) are very important. The more links the spiders find pointing towards your site, the more times they will visit your site pages and this coupled with a few other parameters helps you boost your websites organic search ranking results. Search engine behemoth Google relies on its spiders to create their vast index of web content.

While robots can find web pages by following links from other web pages, it is always suggested that web pages be submitted directly to a search engine and directories thereby generating a request for a visit by their spider for indexation. In fact, it's a good idea to manually submit your site to a human-edited directory because, usually spiders from other search engines will find them there too. Spider-based engines will usually pick up your website pages and index them regardless of whether you've submitted it to a search engine or not.

Thus, spiders, crawlers or robots are tools to discover websites and their content with an objective to index them for the purpose of showing the same to search users when they are looking for content relevant to the search terms or keywords.

Spiders have their limitations with reading and indexing scripts, flash, images but love text content. Content is what is key to the spider. If your website is up to the mark on content for the user, has been coded properly, has Meta tags for the spider and is updated regularly, you may have spiders visiting and indexing your pages often. This will help your SEO efforts.