2015-12-03, 06:44 AM
User-agent: *
Disallow: /private.php
Disallow: /printthread.php
Disallow: /search.php
Disallow: /modcp.php
Disallow: /reputation.php
Disallow: /calendar.php
Disallow: /online.php
Disallow: /contact.php
Disallow: /captcha.php
Disallow: /memberlist.php
Disallow: /task.php
Disallow: /usercp.php
Disallow: /ratethread.php
When I search the forums on google, first one is normal website mysite/forums and second search result is printable version of the website and it has a tree like structure with other threads that are also printthread.php. However when I search using the thread title then normal thread comes up in the search.
I created the site map using this site freesitemapgenerator.com I do see printthread.php links in the sitemap.
Is robots.txt getting ignored?