GitHub topics

web-scraping

Repositories

25

Total stars

46,395

Trending

gosom

scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

Created Apr 2023

agent-skills claude-skills codex-skills distributed-scraper distributed-scraping
Total 8,266
30 days +1,634
originaleric

Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.

Created May 2026

agent-skills ai-agents cli content-extraction golang
Total 301
30 days +285
pinchtab

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

Created Feb 2026

browser-automation cdp golang headless-chrome orchestrator
Total 10,598
30 days +209
karust

Self-hosted web search for AI agents and SEO. Google, Bing, Yandex, Baidu, DuckDuckGo & Ecosia, rendered in a real browser: ranked results and clean page Markdown. API, CLI, MCP 🎉

Created Jun 2023

ai-agents baidu bing duckduckgo google
Total 1,607
30 days +122
sardanioss

Go HTTP client with browser-identical TLS/HTTP2 fingerprinting. Bypass bot detection by perfectly mimicking Chrome, Firefox, and Safari at the cryptographic level (JA3/JA4, Akamai fingerprint, header order). Supports HTTP/1.1, HTTP/2, HTTP/3, sessions, cookies, and proxies.

Created Dec 2025

anti-bot bot-detection browser-fingerprint browser-fingerprinting cloudflare
Total 1,388
30 days +44
1broseidon

Fast, stateless CLI for web search and scrape. Built for AI agents.

Created Mar 2026

ai-agents cli golang scraping search
Total 754
30 days +31
go-rod

A Chrome DevTools Protocol driver for web automation and scraping.

Created Jan 2020

automation cdp chrome-devtools chrome-devtools-protocol chrome-headless
Total 7,165
30 days +30
Anakin-Inc

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Created Mar 2026

ai anti-detect golang markdown open-source
Total 3,818
30 days +28
edoardottt

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Created Apr 2021

bugbounty crawler crawling endpoint-discovery endpoints
Total 3,818
30 days +28
MontFerret

Declarative data automation language and Go runtime for structured extraction workflows.

Created Aug 2018

browser-automation chrome-devtools-protocol data-automation data-extraction dsl
Total 6,031
30 days +16

Top

pinchtab

High-performance browser automation bridge and multi-instance orchestrator with advanced stealth injection and real-time dashboard.

Created Feb 2026

browser-automation cdp golang headless-chrome orchestrator
Total 10,598
gosom

scrape data from Google Maps. Extracts data such as the name, address, phone number, website URL, rating, reviews number, latitude and longitude, reviews,email and more for each place

Created Apr 2023

agent-skills claude-skills codex-skills distributed-scraper distributed-scraping
Total 8,266
go-rod

A Chrome DevTools Protocol driver for web automation and scraping.

Created Jan 2020

automation cdp chrome-devtools chrome-devtools-protocol chrome-headless
Total 7,165
MontFerret

Declarative data automation language and Go runtime for structured extraction workflows.

Created Aug 2018

browser-automation chrome-devtools-protocol data-automation data-extraction dsl
Total 6,031
Anakin-Inc

Open-source web scraping API. Turn any website into clean markdown or structured JSON. Anti-detect browser, proxy auto-selection, self-hosted. One command: make up

Created Mar 2026

ai anti-detect golang markdown open-source
Total 3,818
edoardottt

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Created Apr 2021

bugbounty crawler crawling endpoint-discovery endpoints
Total 3,818
karust

Self-hosted web search for AI agents and SEO. Google, Bing, Yandex, Baidu, DuckDuckGo & Ecosia, rendered in a real browser: ranked results and clean page Markdown. API, CLI, MCP 🎉

Created Jun 2023

ai-agents baidu bing duckduckgo google
Total 1,607
sardanioss

Go HTTP client with browser-identical TLS/HTTP2 fingerprinting. Bypass bot detection by perfectly mimicking Chrome, Firefox, and Safari at the cryptographic level (JA3/JA4, Akamai fingerprint, header order). Supports HTTP/1.1, HTTP/2, HTTP/3, sessions, cookies, and proxies.

Created Dec 2025

anti-bot bot-detection browser-fingerprint browser-fingerprinting cloudflare
Total 1,388
DataHenHQ

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

Created Feb 2021

crawler man-in-the-middle mitm proxy-server scraper
Total 815
1broseidon

Fast, stateless CLI for web search and scrape. Built for AI agents.

Created Mar 2026

ai-agents cli golang scraping search
Total 754
s0rg

The unix-way web crawler

Created Oct 2021

cli crawler go golang golang-application
Total 342
infinilabs

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

Created Jul 2017

crawler crawling elasticsearch lightweight scraping
Total 311
originaleric

Built by Dig: A local-first content crawler for AI agents. Retrieve web pages, posts, images, and videos from public and user-authorized signed-in sources via Agent Skills, MCP, and CLI.

Created May 2026

agent-skills ai-agents cli content-extraction golang
Total 301
gosom

Golang Crawling and scraping framework

Created Mar 2023

crawler go go-framework golang scraper
Total 219
semanser

Get structured JSON data from any page.

Created Oct 2023

api go golang gpt gpt-3
Total 178
suntong

Go cascadia package command line CSS selector

Created Dec 2016

cascadia command-line command-line-tool css-selector csv-table
Total 146
siongui

Download/access photos, videos, stories, story highlights, postlives, following and followers of Instagram

Created Feb 2018

downloader go golang gopherjs instagram
Total 115
anxuanzi

AI-powered browser automation for Go — describe tasks in plain English, let the agent handle the clicks.

Created Dec 2025

agent-framework ai ai-agent automation browser-automation
Total 86
zoharbabin

Search the web and primary sources, read full pages, and check every citation. MCP server for Claude, Cursor, Codex and more. Runs on your machine.

Created May 2026

ai ai-agent anti-hallucination bibliography citation-verification
Total 79
Dhairya3391

Kari just a tui for getting media from providers and playing it in your media player with nice to have features.

Created Jun 2026

anilist anime cartoons cli go
Total 72
soulteary

Rule-driven RSS generator in Go that converts web pages into feeds, with JavaScript rules, headless-browser rendering, and memory or Redis caching.

Created Mar 2023

docker golang headless-browser javascript rss
Total 69
tednaleid

fast cmd-line app that quickly requests millions of urls and can save/echo the results

Created Jan 2017

command-line curl golang http http-client
Total 63
jonfriesen

🥷 A simple puppeteer evasions shim for playwright-go projects.

Created Jun 2024

anti-bot anti-bot-detection playwright playwright-go stealth
Total 53
temoto

Heroshi – open source web crawler.

Created Oct 2008

http-client status-inactive web-scraping
Total 51
combo23

Go API that generates DataDome payloads to bypass bot protection on sites like Footlocker. Built with Gin + MongoDB, supports DataDome up to v4.19.0

Created Dec 2023

antibot automation gin go golang
Total 50

Related

All topics →