The logs of my web-server recently showed that my site was visited by "spiders" from thunderstone, a web index service. (Spiders are automatic indexing programs that sequentially request web pages based upon links, and then sort those pages for use in search engines.) This, of course, piqued my interest in web indexing applications, such as the mighty (and ubiquitous) Google.
Google has some information about its ranking system here. They include a page in which one can enter the URL of your site, so that it will eventually be spidered through. After the crawl through the web, they analyze the links and such, and build a new index. This index is double-checked, and then rolled out to the production servers. (This rollout process is known as the "Google Dance", and seems to have spawned a whole subculture, concerned about their new rankings. See Kuro5hin, this tool for monitoring the rollout progress, and this good technical description.) This site has some good user-submitted information about Google.
I have to imagine that there are a lot of smart people working on ways of indexing the many terabyte of information out there. Google has done it better than anyone else, and seems to be able to keep up with the people who seek to fool it. Kudos.
Because of Google's importance in day-to-day web access, many people have developed a great sensitivity toward its ranking. Because of indexing algorithm strongly depends upon the number of "incoming" links to one's site, people sometimes seem to almost beg people to link to them. I find this intriguing... What kind of society does that build? I am encouraged: Sites with useful content get linked to by people who are best able to judge, and Google rewards those useful sites with higher rankings. It seems to be more than just democratic (although that is a part of it), it seems to encourage a meritocracy, the best of all possible systems.
I've toyed with submitting my site to Yahoo or Netscape's directory. I almost wonder, though: Do I really want more visitors to my site? Would I limit the content of my ramblings if I believed that a wide range of strangers was reading my stuff? Hmm... I'm not really putting anything personal on here, and nothing that I wouldn't stand up in public and proclaim. (I sometimes embed messages to specific people, and that can certainly continue. If anything, the hidden messages become more special because of public traffic.)