Rent-A-Crawler
Posted: 08/15/07 11:20 PM
A few months ago I drastically changed how the urls on my site were built. I moved to using the ASP.NET 2.0 virtual path provider to make more friendly urls. Because I changed the "directory" structure, havoc was wreaked upon my unsuspecting relative urls. I had to go through my posts and look for bad links. I figured I might had missed some.
So I decided to write a crawler, to crawl my site and look for bad urls. And sure enough...I did miss some. Just in case you want to use it yourself, I have it available for download, along with some notes on how you implement a crawler in C#.
Enjoy!