[Webhosting] Search Engines, Bots and SEO

[willawake]

maybe we can have a webhosting thread?



i was checking stats and....huh? msnbot is making big transfer for a bot. so i guessed that windows live must have image search? yup it does

its quite nice as well, the way it keeps bringing more images as you scroll down. but since i am so into google i didnt even know this year old news.

image search links are pretty important for traffic. not only for image based sites for which they are very important obviously but also for any site. images are good ads for a site really. This is why it is important to name image files well, have appropriate captions under them and also to use alt text.

unfortunately image searches are also the way that bloggers search for images to rip

still msn has a long way to go to beat google.



How much bigger is google than msn on bringing traffic. er....like 10 times. yahoo is pretty crap also.
....and Google images brings in my case like an equal amount of traffic as google search...therefore its equally important to me.

[bgavin]

Re: Webhosting

My web server is attached to a 384/kb uplink speed, and the bots were clogging my bandwidth.

I had to modify my robots.txt file to exclude my library from bot searches. It doesn't get into the search engines, but I got my bandwidth back.

[Per Hansson]

Re: Webhosting

What's up with that MSN crap, can't we just force the MSN bot to not search the site?

That's way way more traffic than acceptable!

(we of course want the google bot to continue indexing)

[willawake]

Re: Webhosting

yes as Bruce said you can modify robots.txt (a file you should place in the public root directory of each site). in linux this is of course public_html (it is highly recommended to put a robots.txt otherwise the error log gets filled with 404 errors for this file each time bots visit.

i will go into this in detail for anyone interested.

a robots.txt permitting access to everything for all bots would be :

User-agent: *
Disallow:

a robots.txt disallowing access to everything for all bots would be :

User-agent: *
Disallow: /

so if you wanted to dissalow msnbot you would put instead

User-agent: msnbot
Disallow: /

apparently all the msnbots obey the above "msnbot" statement. there are several

msn Shopping bot is msnbot-products
msn News bot is msnbot-news
msn Image Search bot is msnbot-media
msn Search bot is still just plain msnbot

Some people are interested to stop google images indexing their images. I have heard it is not recommended to dissalow the google images bot cos this can discourage the googlebot also. Best way would be to put the images in a folder /images and disallow googlebot and google images access to that folder

[willawake]

Re: Webhosting

now if you have an active site like a forum or a CMS (Content Management System) then it is important to stop the bots indexing the contents of quite a few directories that do not contain content. firstly we do not want whatever is in there to be indexed but also the bots put a pointless load on the server crawling all those directories and stuff if it finds links to them.

for vbulletin something like this would be good. probably a lot of it is unnecessary but better safe than sorry.....

User-agent: *
Disallow: /forums/ajax.php
Disallow: /forums/ajax_cron.php
Disallow: /forums/attachment.php
Disallow: /forums/calendar.php
Disallow: /forums/checkspelling.php
Disallow: /forums/cron.php
Disallow: /forums/editpost.php
Disallow: /forums/external.php
Disallow: /forums/faq.php
Disallow: /forums/global.php
Disallow: /forums/image.php
Disallow: /forums/inlinemod.php
Disallow: /forums/joinrequest.php
Disallow: /forums/login.php
Disallow: /forums/member.php
Disallow: /forums/memberlist.php
Disallow: /forums/misc.php
Disallow: /forums/moderator.php
Disallow: /forums/newattatchment.php
Disallow: /forums/newreply.php
Disallow: /forums/newthread.php
Disallow: /forums/online.php
Disallow: /forums/payment_gateway.php
Disallow: /forums/payments.php
Disallow: /forums/pdfthread.php
Disallow: /forums/poll.php
Disallow: /forums/postings.php
Disallow: /forums/printthread.php
Disallow: /forums/private.php
Disallow: /forums/profile.php
Disallow: /forums/register.php
Disallow: /forums/report.php
Disallow: /forums/reputation.php
Disallow: /forums/search.php
Disallow: /forums/sendmessage.php
Disallow: /forums/showgroups.php
Disallow: /forums/subcription.php
Disallow: /forums/subcriptions.php
Disallow: /forums/threadrate.php
Disallow: /forums/usercp.php
Disallow: /forums/usernote.php
Disallow: /forums/admincp/
Disallow: /forums/articlebot/
Disallow: /forums/clientscript/
Disallow: /forums/cpstyles/
Disallow: /forums/customavatars/
Disallow: /forums/customprofilepics/
Disallow: /forums/modcp/
Disallow: /forums/images/
Disallow: /forums/install/
Disallow: /forums/includes/
Disallow: /forums/signaturepics/
Disallow: /forums/subscriptions/

[willawake]

Re: [Webhosting] Search Engines, Bots and SEO

Google is keeping a close eye on a small, suburban Melbourne start-up that claims to be developing a search engine that improves on the world leader.

MyLiveSearch, which plans to go live as a public beta in a few weeks, is the brainchild of self-taught software developer Rob Gabriel.full article

MyLiveSearch turns your own computer into a 'super-spider' to search it in real time.

oh great thats all webmasters need, millions of userbots

[willawake]

Re: [Webhosting] Search Engines, Bots and SEO

this bot is always hanging around crs040.goo.ne.jp

the name is rather funny crs (cant remember shit)
maybe thats why it keeps coming back