A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Last push 5 years ago

crawler robots-txt

Created

Jan 2014

Total stars

791

7 days

0

30 days

0

12 months

+3

Star trends

README badge

Best of Go badge for PuerkitoBio/fetchbot
[![Best of Go](https://bestofgo.dev/badge/PuerkitoBio/fetchbot.svg)](https://bestofgo.dev/PuerkitoBio/fetchbot)