Find Affordable Web Hosting

mrapache · 02-07-2007, 11:35 AM

I am curious to see some example uses and implementations of robots.txt.. specifically implementations and the reasons behind them for increasing SEO. The robots.txt I am using is based on the example: SEO with robots.txt

WordPress 2.1 robots.txt

Code:

User-agent: * 

# disallow all files in these directories
Disallow: /cgi-bin/
Disallow: /admin/
Disallow: /comments/
Disallow: /z/j/
Disallow: /z/c/
Disallow: /about/legal-notice/
Disallow: /about/copyright-policy/
Disallow: /about/terms-and-conditions/
Disallow: /about/feed/
Disallow: /about/trackback/
Disallow: /contact/
Disallow: /stats*
Disallow: /tag
Disallow: /category/uncategorized*

# disallow all files ending with these extensions
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.txt$

# disallow all files in /wp- directorys
Disallow: /wp-*/

# disallow all files with? in url
Disallow: /*?

I also use a custom robots.txt for phpBB

phpBB robots.txt

Code:

User-agent: * 
# disallow all files with a? in url 
Disallow: /*?* 
  
# disallow all files ending in specific extension 
Disallow: /*.php$ 
Disallow: /*.js$ 
Disallow: /*.inc$ 
Disallow: /*.css$ 
Disallow: /*.txt$ 

# disallow these dirs
Disallow: /js/
Disallow: /css/
Disallow: /cgi-bin/
Disallow: /db/
Disallow: /admin/
Disallow: /cache/
Disallow: /includes/
Disallow: /templates/

# disallow these files and dirs
Disallow: /V
Disallow: /stats*
Disallow: /post
Disallow: /member
Disallow: /mx_

# disallow these urls
Disallow: /rss.php
Disallow: /viewtopic.php
Disallow: /viewforum.php
Disallow: /index.php?
Disallow: /posting.php
Disallow: /groupcp.php
Disallow: /search.php
Disallow: /login.php
Disallow: /profile.php
Disallow: /memberlist.php
Disallow: /faq.php
Disallow: /common.php
Disallow: /index.php
Disallow: /memberlist.php
Disallow: /modcp.php
Disallow: /privmsg.php
Disallow: /viewonline.php

# disallow urls starting with quote
Disallow: /"

but this phpBB forum is different than the default because it has special optimizations already.

Basically this helps get rid of duplicate content, low-quality content, css, javascript, php, etc.. but does allow search engines to read the articles, find images, find pdfs, etc.
Anyone else have improvements or other robots.txt examples?

casitecenter · 03-30-2007, 06:09 AM

you can also use robot meta tag inside html for pages
good luck

mrapache · 03-30-2007, 11:43 PM

Yeah you are correct.

Heres what I'm doing now to SEO WordPress

Code:

<?php if(is_single() || is_home()) { ?>
<meta name="googlebot" content="index,follow,noodp,noarchive" />
<meta name="robots" content="all,index,follow" />
<meta name="msnbot" content="all,index,follow" />
<?php } else { ?>
<meta name="googlebot" content="follow,noodp,noindex,noarchive" />
<meta name="robots" content="follow,noindex,noarchive,nocache" />
<meta name="msnbot" content="follow,noindex,noarchive,nocache" />
<?php }?>

And then my robots.txt is less restrictive than initially because we still want google to follow our pages, and they won't if its disallowed in robots.txt.

casitecenter · 05-15-2007, 05:19 AM

thank you so much
good luck

best regards

mrapache · 05-16-2007, 05:42 PM

no problem bro.. implementing this into my blog has given huge seo results. g'luck

casitecenter · 05-29-2007, 01:00 PM

Quote:

Originally Posted by mrapache

no problem bro.. implementing this into my blog has given huge seo results. g'luck

Yes this is your right when you share a idea with others you will have huge visitors
any way ,i have the best wishes for you and continue it