GitHub topics

web-crawler

Mostly in Text Processing 14%

Repositories

14

Total stars

19,128

Trending

originaleric

Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.

Created May 2026

agent-skills ai-agents cli content-extraction golang
Total 301
30 days +285
crawlab-team

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

Created Feb 2019

crawlab crawler crawling-tasks docker go
Total 12,312
30 days +31
Anakin-Inc

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Created Mar 2026

ai anti-detect golang markdown open-source
Total 3,818
30 days +28
gosom

Golang Crawling and scraping framework

Created Mar 2023

crawler go go-framework golang scraper
Total 219
30 days +5
tech-engine

GoScrapy: High perfomance webscraping framework in Go, inspired by Python's Scrapy.

Created Jul 2023

data-extraction go-scrapy golang goscraper scrapy
Total 373
30 days +3
s0rg

The unix-way web crawler

Created Oct 2021

cli crawler go golang golang-application
Total 342
30 days +3
internetarchive

State-of-the-art web crawler 🔱

Created Aug 2020

archiving web-crawler zeno
Total 436
30 days +3
Sriram-PR

Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)

Created Apr 2025

data-preparation documentation golang-cli llm mcp
Total 110
30 days +2
saeeddhqan

Interactive CLI Web Crawler

Created Jul 2020

cli crawler data-mining fuzzing go
Total 183
30 days +2
yields

A web crawler for Go

Created Sep 2020

go golang scraper spider web-crawler
Total 282
30 days +1

Top

crawlab-team

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

Created Feb 2019

crawlab crawler crawling-tasks docker go
Total 12,312
Anakin-Inc

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Created Mar 2026

ai anti-detect golang markdown open-source
Total 3,818
internetarchive

State-of-the-art web crawler 🔱

Created Aug 2020

archiving web-crawler zeno
Total 436
tech-engine

GoScrapy: High perfomance webscraping framework in Go, inspired by Python's Scrapy.

Created Jul 2023

data-extraction go-scrapy golang goscraper scrapy
Total 373
s0rg

The unix-way web crawler

Created Oct 2021

cli crawler go golang golang-application
Total 342
infinilabs

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

Created Jul 2017

crawler crawling elasticsearch lightweight scraping
Total 311
originaleric

Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.

Created May 2026

agent-skills ai-agents cli content-extraction golang
Total 301
yields

A web crawler for Go

Created Sep 2020

go golang scraper spider web-crawler
Total 282
antchfx

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Created Sep 2017

crawler crawling framework golang scraping
Total 265
gosom

Golang Crawling and scraping framework

Created Mar 2023

crawler go go-framework golang scraper
Total 219
saeeddhqan

Interactive CLI Web Crawler

Created Jul 2020

cli crawler data-mining fuzzing go
Total 183
Sriram-PR

Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)

Created Apr 2025

data-preparation documentation golang-cli llm mcp
Total 110
creekorful

Fast, highly configurable, cloud native dark web crawler.

Created Mar 2020

architecture crawler crawling elasticsearch golang
Total 97
redcode-labs

A tool to find redirection chains in multiple URLs

Created Mar 2021

golang reconnaissance redirection url url-redirection
Total 79

Related

All topics →