GitHub topics

spider

Repositories

48

Total stars

75,217

Trending

gocolly

Elegant Scraper and Crawler Framework for Golang

Created Sep 2017

crawler crawling framework go golang
Total 25,630
30 days +66
wujunwei928

Golang短视频去水印:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...

Created Jul 2022

crawler douyin kuaishou spider video
Total 1,107
30 days +33
crawlab-team

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

Created Feb 2019

crawlab crawler crawling-tasks docker go
Total 12,312
30 days +31
ma6254

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

Created Mar 2019

biquge crawler fiction golang novels
Total 1,069
30 days +25
jaeles-project

Gospider - Fast web spider written in Go

Created Jan 2020

bugbounty crawler go gospider spider
Total 3,024
30 days +16
fireinrain

A tool for download asmr media from asmr.one(Thanks for the asmr.one)

Created Feb 2023

asmr downloader go golang spider
Total 239
30 days +14
Nhoya

OSINT Swiss Army Knife

Created Aug 2017

axfr bitbucket crawler forensics git
Total 680
30 days +6
andeya

Pholcus is a distributed high-concurrency crawler software written in pure golang

Created Jul 2015

crowler spider
Total 7,590
30 days +6
gosom

Golang Crawling and scraping framework

Created Mar 2023

crawler go go-framework golang scraper
Total 219
30 days +5
tophubs

今日热榜,一个获取各大热门网站热门头条的聚合网站,使用Go语言编写,多协程异步快速抓取信息,预览:https://mo.fish

Created Jul 2019

golang hot hotlist spider today-s-hot-list
Total 4,731
30 days +5

Top

gocolly

Elegant Scraper and Crawler Framework for Golang

Created Sep 2017

crawler crawling framework go golang
Total 25,630
crawlab-team

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

Created Feb 2019

crawlab crawler crawling-tasks docker go
Total 12,312
andeya

Pholcus is a distributed high-concurrency crawler software written in pure golang

Created Jul 2015

crowler spider
Total 7,590
tophubs

今日热榜,一个获取各大热门网站热门头条的聚合网站,使用Go语言编写,多协程异步快速抓取信息,预览:https://mo.fish

Created Jul 2019

golang hot hotlist spider today-s-hot-list
Total 4,731
jaeles-project

Gospider - Fast web spider written in Go

Created Jan 2020

bugbounty crawler go gospider spider
Total 3,024
geziyor

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Created Jun 2019

crawler go scraper scraping spider
Total 2,784
yann-shi

BitTorrent DHT Protocol && DHT Spider.

Created Aug 2016

bittorrent-dht-protocol dht go spider
Total 2,771
pibigstar

Go语言实例教程从入门到进阶,包括基础库使用、设计模式、面试易错点、工具类、对接第三方等

Created Aug 2018

blockchain design go go-demo go-design
Total 2,711
hu17889

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Created May 2014

crawler go pipeline schedule spider
Total 1,815
bookstairs

A download tools for clawing the ebooks from internets.

Created Apr 2022

epub golang spider
Total 1,305
wujunwei928

Golang短视频去水印:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...

Created Jul 2022

crawler douyin kuaishou spider video
Total 1,107
ma6254

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

Created Mar 2019

biquge crawler fiction golang novels
Total 1,069
wspl

🐾 Creeper - The Next Generation Crawler Framework (Go)

Created Feb 2017

crawler cross-platform framework golang language
Total 778
signal18

Signal 18 repman - Replication Manager for MySQL / MariaDB / Percona Server

Created Oct 2015

alerting backups configuration-management failover group-replication
Total 748
Nhoya

OSINT Swiss Army Knife

Created Aug 2017

axfr bitbucket crawler forensics git
Total 680
go-crawler

带你了解一下Golang的市场行情

Created Apr 2018

crawler go golang lagou spider
Total 617
nange

golang实现的爬虫框架,使用者只需关心页面规则,提供web管理界面。基于colly开发。

Created Mar 2018

crawler go golang gospider spider
Total 415
XORbit01

🕸️ Crawl in the web network

Created Apr 2023

crawler crawling data data-science datamining
Total 385
tech-engine

GoScrapy: High perfomance webscraping framework in Go, inspired by Python's Scrapy.

Created Jul 2023

data-extraction go-scrapy golang goscraper scrapy
Total 373
IAmStoxe

A golang utility to spider through a website searching for additional links.

Created Jul 2020

spider
Total 348
infinilabs

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

Created Jul 2017

crawler crawling elasticsearch lightweight scraping
Total 311
yields

A web crawler for Go

Created Sep 2020

go golang scraper spider web-crawler
Total 282
hezhizheng

golang spider Crawler 爬虫 电影

Created Dec 2019

colly crawler docker fasthttp go
Total 275
dwisiswant0

A fast tool to fetch URLs from HTML attributes by crawl-in.

Created Nov 2020

crawler devtool extractor galer go
Total 263
fireinrain

A tool for download asmr media from asmr.one(Thanks for the asmr.one)

Created Feb 2023

asmr downloader go golang spider
Total 239
gosom

Golang Crawling and scraping framework

Created Mar 2023

crawler go go-framework golang scraper
Total 219
suosi-inc

一个 Golang 实现的相对智能、无需规则维护的通用新闻网站数据提取工具库。含域名探测、网页编码语种识别、网页链接分类提取、网页新闻要素抽取以及新闻正文抽取等组件。

Created Jul 2022

extractor langdetect spider webspider
Total 206
wangshibiaoFlytiger

[https://www.sofineday.com], golang项目开发脚手架,集成最佳实践(gin+gorm+go-redis+mongo+cors+jwt+json日志库zap(支持日志收集到kafka或mongo)+消息队列kafka+微信支付宝支付gopay+api加密+api反向代理+go modules依赖管理+headless爬虫chromedp+makefile+二进制压缩+livereload热加载)

Created Aug 2019

alipay api-server compress cors gin-framework
Total 178
twiny

Domain names collector - Crawl websites and collect domain names along with their availability status.

Created Jul 2020

backlinks crawler domain expired-domain golang
Total 165
2young2simple

一款分布式爬虫平台,帮助你更好的管理和开发爬虫。 内置一套爬虫定义规则(模版),可使用模版快速定义爬虫,也可当作框架手动开发爬虫。(兴趣使然的项目,用的不爽了就更新)

Created Oct 2017

crawler golang spider
Total 161
go-crawler

Golang爬虫 爬取豆瓣电影Top250

Created Mar 2018

crawler douban go golang movie
Total 159
lizongying

A web crawling framework implemented in Golang, it is simple to write and delivers powerful performance. It comes with a wide range of practical middleware and supports various parsing and storage methods. Additionally, it supports distributed deployment. 基于golang实现的爬虫框架,编写简单,性能强劲。内置了丰富的实用中间件,支持多种解析、保存方式,支持分布式部署。

Created Apr 2023

crawler golang scraping scrapy spider
Total 150
WShihan

一个用于下载哔哩哔哩/Bilibili(B站) 视频评论的超简单命令行工具。

Created Jun 2024

bilibili comment download spider
Total 149
PeterYangs

文章采集工具 Article collection tool

Created Feb 2021

article-spider go golang scraper spider
Total 143
luckylittle

Grabs all of the audio files from all of the Blinkist books

Created Jan 2019

audiobooks blinkist books crawler data-archiving
Total 139
superjcd

gocrawler, go分布式爬虫框架

Created Feb 2024

crawler distributed gocrawler golang spider
Total 105
jinzhongjia

Go编写的影视资源采集器

Created May 2022

golang gorm movies spider sqlite3
Total 94
clindet

Portable command-line tool to query bioinformatics APIs, data, databases and files.

Created Aug 2019

bioinformatics database spider
Total 92
medcl

GOPA, a spider written in Go.(NOTE: this project moved to https://github.com/infinitbyte/gopa )

Created Jul 2013

crawler golang lightweight spider
Total 92
PhiFever

爱发电爬虫(afdian.com)

Created Mar 2024

afdian crawler mcp mcp-server spider
Total 80
go-crawler

Golang爬虫 爬取汽车之家 二手车产品库

Created Apr 2018

crawler go golang spider
Total 79
zhoutk

create some tools use go lang.

Created Jun 2018

golang goroutine mysql spider tools
Total 75
x-way

Golang module to detect bots and crawlers via the user agent

Created Aug 2020

bot-detection crawler crawler-detection detect go
Total 73
sb-child

python 爬虫, golang 爬虫: 涩涩网站 学习资料 视频爬取下载工具 完美适配 miya(蜜芽)和 yysp(夜夜视频, 夜夜視頻)

Created Mar 2021

beautifulsoup4 colly decrypt downloader ffmpeg
Total 66
neoql

Some toolkits implements part of BT Protocol, like DHT spider.

Created Jan 2018

bt dht golang spider
Total 59
bajins

基于go-gin框架建立减少冗余动作项目,如:下载一些工具

Created Jun 2019

crawler gin gin-gonic golang key
Total 58
yusp998

微信公众号爬虫,以API方式提供公众号文章获取,包括阅读量、点赞等

Created Dec 2023

api crawler spider wechat
Total 57
mangenotwork

gathertool是golang脚本化开发库,目的是提高对应场景程序开发的效率;轻量级爬虫库,接口测试&压力测试库,DB操作库等。

Created Apr 2021

crawl gathertool golang grpc-go proxy-server
Total 55

Related

All topics →