Information Architecture

Useful Tools

SEO Browser
See your website like a search engine sees it. Using this free online tool you can browse as a spider might, following links to other pages to see how they are interpreted by the search engines too.

Page Rank Recovery Tool
The basic version is free. Enter a URL to report back on up to 200 404 error pages (i.e. pages that could be losing PageRank), then indicate where these should 301 redirect to before creating the snippets to copy & paste into your .htaccess file. Brilliant!

W3C Link Checker
Enter a URL to check links and anchors in Web pages or full Web sites.

Google Sitemap Generator
A free tool that you install on your web server, allowing you to create and then automatically update your XML sitemap.

GSiteCrawler
Free downloadable Google (and Yahoo!) XML sitemap generator for Windows.

Reference

Web Robots FAQs
A list of frequently asked questions about web robots. Select the question to go to the corresponding answer page.

How To Guides

XML Sitemap Structure
Guidance on the correct protocols to structure an XML sitemap, showing essential and optional tags to include.

Robots.txt
A comprehensive tutorial on the robots.txt file from SEO Book.