GitHub topics

crawling

Mostly in Text Processing 22%, Security Tools 16%

Repositories

18

Total stars

45,695

Trending

gocolly

Elegant Scraper and Crawler Framework for Golang

Created Sep 2017

crawler crawling framework go golang
Total 25,630
30 days +66
StJudeWasHere

Open source SEO audit tool.

Created Mar 2022

crawler crawling docker docker-compose go
Total 874
30 days +62
go-rod

A Chrome DevTools Protocol driver for web automation and scraping.

Created Jan 2020

automation cdp chrome-devtools chrome-devtools-protocol chrome-headless
Total 7,165
30 days +29
edoardottt

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Created Apr 2021

bugbounty crawler crawling endpoint-discovery endpoints
Total 3,819
30 days +28
hakluke

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

Created Dec 2019

bugbounty crawling hacking osint pentesting
Total 5,176
30 days +27
shurco

🌱 goClone - clone websites in seconds

Created Jul 2023

cloner cloning crawler crawling go
Total 132
30 days +5
XORbit01

🕸️ Crawl in the web network

Created Apr 2023

crawler crawling data data-science datamining
Total 385
30 days +2
slotix

Extract structured data from web sites. Web sites scraping.

Created Feb 2017

cdp chrome-fetcher crawling extract-data go
Total 716
30 days +2
creekorful

Fast, highly configurable, cloud native dark web crawler.

Created Mar 2020

architecture crawler crawling elasticsearch golang
Total 97
30 days +1
rookmoot

A fast, modern and intelligent proxy rotator perfect for crawling and scraping public data.

Created Jan 2017

crawler crawling proxy proxy-server rotating-proxy
Total 156
30 days +1

Top

gocolly

Elegant Scraper and Crawler Framework for Golang

Created Sep 2017

crawler crawling framework go golang
Total 25,630
go-rod

A Chrome DevTools Protocol driver for web automation and scraping.

Created Jan 2020

automation cdp chrome-devtools chrome-devtools-protocol chrome-headless
Total 7,165
hakluke

Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application

Created Dec 2019

bugbounty crawling hacking osint pentesting
Total 5,176
edoardottt

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Created Apr 2021

bugbounty crawler crawling endpoint-discovery endpoints
Total 3,819
StJudeWasHere

Open source SEO audit tool.

Created Mar 2022

crawler crawling docker docker-compose go
Total 874
slotix

Extract structured data from web sites. Web sites scraping.

Created Feb 2017

cdp chrome-fetcher crawling extract-data go
Total 716
mhmdiaa

Second-order subdomain takeover scanner

Created Jun 2017

crawler crawling infosec mapping penetration-testing
Total 408
XORbit01

🕸️ Crawl in the web network

Created Apr 2023

crawler crawling data data-science datamining
Total 385
infinilabs

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

Created Jul 2017

crawler crawling elasticsearch lightweight scraping
Total 311
antchfx

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Created Sep 2017

crawler crawling framework golang scraping
Total 265
dimkouv

Download a large list of files concurrently

Created Feb 2019

crawling download-manager downloader golang
Total 165
rookmoot

A fast, modern and intelligent proxy rotator perfect for crawling and scraping public data.

Created Jan 2017

crawler crawling proxy proxy-server rotating-proxy
Total 156
trandoshan-io

Go process used to crawl websites

Created Aug 2019

crawler crawling docker go golang
Total 149
shurco

🌱 goClone - clone websites in seconds

Created Jul 2023

cloner cloning crawler crawling go
Total 132
alash3al

Scraply a simple dom scraper to fetch information from any html based website

Created May 2019

crawler crawling dom golang scraper
Total 130
creekorful

Fast, highly configurable, cloud native dark web crawler.

Created Mar 2020

architecture crawler crawling elasticsearch golang
Total 97
pzaino

an open, self-hosted framework for discovering, collecting, understanding, correlating, transforming, and acting on information from heterogeneous environments.

Created Nov 2023

automation blue-team-tool content-detection content-discovery crawler
Total 62
supacrawler

Supacrawler's ultralight engine for scraping and crawling the web. Written in go for maximum performance and concurrency.

Created Aug 2025

crawling playwright rag rag-chatbot webscraping
Total 55

Related

All topics →