PuerkitoBio / fetchbot
A simple and flexible web crawler that follows the robots.txt policies and crawl delays.
Last push 5 years ago
crawler
robots-txt
Created
Jan 2014
Total stars
791
Last 7 days
0
Last 30 days
0
Last 12 months
+3
README badge
[](https://bestofgo.dev/PuerkitoBio/fetchbot)