Google/Yahoo bot

[sewcute]

What does it mean when I see a google or Yahoo bot signed in?

[madd74]

Bot are spiders/crawlers, usually from search engines, that scan the web for information. Some of them are used to update their search engine database. To date, MWF has 52 different associated bots registered. The problem with search spiders, is that they can mes up session stats, cause problems with load time, and a bunch of other things. Also, there might be an instance where a poorly coded spider causes unneccesary load, or a search site might end up sending 6 different spiders to do a few different tasks. By assigning an IP address associated with the bot, or using certain key words that are assocaited with similar bots, you can give all of those similar ones the same user name. That is why you see the most users online ever was 11. It was not necessarily 11 different people, it could ahve been a few humans, and the rest bots.

Another thing, is that once they are registered with the board, you can assign them specific language to use, as well as a certain template. Some templates are more complex, thus causing more load to the board. You can assign them the most simple of templates to decrease load time.

[sewcute]

I thought you had fixed it so that the bots couldn't be part of the site anymore?

[madd74]

These are valid "good" bots. There is a difference between that and spam bots. A spam bot is a critter that lives simply to register to boards and post spam about going to their site and buying their crap. It is the same principle as spam email. The bots you see are actually web spiders and crawlers, even though since they are not run by humans, they are also considered bots. The difference is that these have a "useful" purpose.

[sewcute]

what is their purpose?

[madd74]

Bot are spiders/crawlers, usually from search engines, that scan the web for information. Some of them are used to update their search engine database.

This is pretty much the purpose of spiders, to search the web to look for information to send back to their "master". The more times a bot is sent out to hunt for information, the more accurate that search engine is. Ever click on a link to find that it is no longer there? A great search engine has spiders looking for that, so the information can be changed when you go to that search engine. In a bad search engine, you might do a search for "hugs.mp3" and when you click on a link, you get a 404 error (that the page cannot be found) on almost every link you click. Of course, with over a billion, if not more, possible pages of information, even the best search engines will refer to bad links. The better ones make sure when you check that search in a year, it is no longer bad

[sewcute]

interesting. I did not know that.

SO when they are on the board are they actually logging in? Or does the board just know "what/who" they are?

[madd74]

Anyone can just come onto the board. When you do, you leave a mark of who you are, your IP address, and other information. When you click the "Who's Online" link, you can see what people are doing, for when you access the board, you access the database of information where the posts are stored and such. If my IP did not ever change, like most spiders and such, then instead of seeing "guest", it would show the associated name with the number, a similar concept to something like CID. Also, some spiders show specific information from where they are, just like when you log onto any site, you tell it what browser and OS you are coming from (usually). Some bots have info about them when they read, and that is all they really do, is read. For example, there are two people on right now. The Who is online shows:

madd74 IP: 71.32.163.89 ? Whois
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.7) Gecko/20070914 Firefox/2.0.0.7 Viewing who is online 07/10/07 16:17:34

Mair IP: 12.210.13.232 ? Whois
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) Reading topic in Outer Unbound 07/10/07 16:15:45

The general user does not see this detailed of information, usually. However, as you can see, it is all rather logged.

[Mair]

Why do you allow them to check out MWF?

[madd74]

By letting bots who behave read check out the forums, in theory, I allow them to index the information from the board and store that in search engines, so people on search engines can better find where I am, maybe even join, for while the board is not yet gold, i still welcome new users to join my little world and add to it.

[Mair]

Oh that makes perfect sense. I get it.

[sewcute]

so if I understand correctly, if I were LOGGED OUT, and checking from my home computer, then the board would know it's me?

[madd74]

An excellent question, and "not really".

See, the board keeps track of the IP address of every message that is posted. If you have a broadband connection, then your IP address is usually the same for up to about two weeks, before it refreshes and is a different one. So, if you posted something, and then you logged out as a guest and attempted to post, and someone looked up "show all posts from this address", then it would show the guest post associated with your others. Now, the board itself would still register you as a "guest", because humans have to log on, where bots do not.

[sewcute]

then why is my IP the same like when I leave comments on lj?

[madd74]

Your IP address is assigned to you by your ISP. This address is "programmed" into your modem. It is how it communicates with the world. You generally have the same IP address for a while, however, it will eventaully be renewed.