Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.
Created May 2026
Repositories
14
Total stars
19,128
Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.
Created May 2026
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Created Feb 2019
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Created Mar 2026
Golang Crawling and scraping framework
Created Mar 2023
GoScrapy: High perfomance webscraping framework in Go, inspired by Python's Scrapy.
Created Jul 2023
Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
Created Apr 2025
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
Created Feb 2019
Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up
Created Mar 2026
GoScrapy: High perfomance webscraping framework in Go, inspired by Python's Scrapy.
Created Jul 2023
🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)
Created Jul 2017
Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.
Created May 2026
Golang Crawling and scraping framework
Created Mar 2023
Go web crawler to scrape documentation sites and convert content to clean Markdown for LLM ingestion (RAG, training data)
Created Apr 2025
Fast, highly configurable, cloud native dark web crawler.
Created Mar 2020
A tool to find redirection chains in multiple URLs
Created Mar 2021