GitHub topics

crawler

Mostly in Security Tools 16%

Repositories

83

Total stars

134,989

Trending

Autumn-27

ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes

Created Feb 2024

bug-bounty bugbounty bugbounty-tool crawler dirscanner
Total 2,560
30 days +588
projectdiscovery

A next-generation crawling and spidering framework.

Created Jan 2021

cli crawler gocrawler hacktoberfest headless
Total 17,890
30 days +213
iawia002

👾 Fast and simple video download library and CLI tool written in Go

Created Feb 2018

bilibili crawler download downloader go
Total 31,878
30 days +116
gocolly

Elegant Scraper and Crawler Framework for Golang

Created Sep 2017

crawler crawling framework go golang
Total 25,630
30 days +66
StJudeWasHere

Open source SEO audit tool.

Created Mar 2022

crawler crawling docker docker-compose go
Total 874
30 days +62
wujunwei928

Golang短视频去水印:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...

Created Jul 2022

crawler douyin kuaishou spider video
Total 1,107
30 days +33
crawlab-team

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

Created Feb 2019

crawlab crawler crawling-tasks docker go
Total 12,312
30 days +31
edoardottt

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Created Apr 2021

bugbounty crawler crawling endpoint-discovery endpoints
Total 3,819
30 days +28
ma6254

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

Created Mar 2019

biquge crawler fiction golang novels
Total 1,069
30 days +25
goclone-dev

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

Created Mar 2020

cloning crawler go golang website-cloner
Total 2,305
30 days +23

Top

Top 50 of 83

iawia002

👾 Fast and simple video download library and CLI tool written in Go

Created Feb 2018

bilibili crawler download downloader go
Total 31,878
gocolly

Elegant Scraper and Crawler Framework for Golang

Created Sep 2017

crawler crawling framework go golang
Total 25,630
projectdiscovery

A next-generation crawling and spidering framework.

Created Jan 2021

cli crawler gocrawler hacktoberfest headless
Total 17,890
crawlab-team

Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架

Created Feb 2019

crawlab crawler crawling-tasks docker go
Total 12,312
zu1k

Automatically crawls proxy nodes on the public internet, de-duplicates and tests for usability and then provides a list of nodes

Created Aug 2020

crawler proxypool
Total 4,045
edoardottt

Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more

Created Apr 2021

bugbounty crawler crawling endpoint-discovery endpoints
Total 3,819
Qianlitp

A powerful browser crawler for web vulnerability scanners

Created Dec 2019

arsenal blackhat chrome-devtools chromedp crawler
Total 3,051
jaeles-project

Gospider - Fast web spider written in Go

Created Jan 2020

bugbounty crawler go gospider spider
Total 3,024
geziyor

Geziyor, blazing fast web crawling & scraping framework for Go. Supports JS rendering.

Created Jun 2019

crawler go scraper scraping spider
Total 2,784
Autumn-27

ScopeSentry-Cyberspace mapping, subdomain enumeration, port scanning, sensitive information discovery, vulnerability scanning, distributed nodes

Created Feb 2024

bug-bounty bugbounty bugbounty-tool crawler dirscanner
Total 2,560
goclone-dev

Website Cloner - Utilizes powerful Go routines to clone websites to your computer within seconds.

Created Mar 2020

cloning crawler go golang website-cloner
Total 2,305
PuerkitoBio

Polite, slim and concurrent web crawler.

Created Sep 2012

crawler robots-txt
Total 2,052
rendora

Dynamic server-side rendering using headless Chrome

Created Nov 2018

angular chrome-devtools chrome-headless crawler dynamic-rendering
Total 1,989
hu17889

[爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only.

Created May 2014

crawler go pipeline schedule spider
Total 1,815
dwisiswant0

The fastest dork scanner written in Go.

Created Jul 2020

bing-dorks bugbounty bugbounty-tool crawler dork-scanner
Total 1,313
wujunwei928

Golang短视频去水印:抖音,皮皮虾,火山,微视,最右,快手,全民小视频,皮皮搞笑,西瓜视频,虎牙,梨视频,acfun,好看视频...

Created Jul 2022

crawler douyin kuaishou spider video
Total 1,107
ma6254

小说下载|小说爬取|起点|笔趣阁|导出Markdown|导出txt|转换epub|广告过滤|自动校对

Created Mar 2019

biquge crawler fiction golang novels
Total 1,069
Le0nsec

一个方便安全研究人员获取每日安全日报的爬虫和推送程序,目前爬取范围包括先知社区、安全客、Seebug Paper、跳跳糖、奇安信攻防社区、棱角社区以及绿盟、腾讯玄武、天融信、360等实验室博客,持续更新中。

Created Dec 2021

anquanke bot crawler security seebug
Total 939
StJudeWasHere

Open source SEO audit tool.

Created Mar 2022

crawler crawling docker docker-compose go
Total 874
DataHenHQ

DataHen Till is a companion tool to your existing web scraper that instantly makes it scalable, maintainable, and more unblockable, with minimal code changes on your scraper. Integrates with any scraper in 5 minutes.

Created Feb 2021

crawler man-in-the-middle mitm proxy-server scraper
Total 816
PuerkitoBio

A simple and flexible web crawler that follows the robots.txt policies and crawl delays.

Created Jan 2014

crawler robots-txt
Total 791
wspl

🐾 Creeper - The Next Generation Crawler Framework (Go)

Created Feb 2017

crawler cross-platform framework golang language
Total 778
chushuai

Wscan is a web security scanner that focuses on web security, dedicated to making web security accessible to everyone.

Created May 2022

cel-go chromedp crawler headless martian
Total 719
Nhoya

OSINT Swiss Army Knife

Created Aug 2017

axfr bitbucket crawler forensics git
Total 680
iudicium

Prying Deep - An OSINT tool to collect intelligence on the dark web.

Created Oct 2023

crawler darkweb go gocolly golang-osint
Total 628
go-crawler

带你了解一下Golang的市场行情

Created Apr 2018

crawler go golang lagou spider
Total 617
yhy0
yhy0 / Jie

Jie stands out as a comprehensive security assessment and exploitation tool meticulously crafted for web applications. Its robust suite of features encompasses vulnerability scanning, information gathering, and exploitation, elevating it to an indispensable toolkit for both security professionals and penetration testers. 挖洞辅助工具(漏洞扫描、信息收集)

Created Jan 2023

apollo-exp bugcrowd crawler hackerone jie
Total 612
nange

golang实现的爬虫框架,使用者只需关心页面规则,提供web管理界面。基于colly开发。

Created Mar 2018

crawler go golang gospider spider
Total 415
mhmdiaa

Second-order subdomain takeover scanner

Created Jun 2017

crawler crawling infosec mapping penetration-testing
Total 408
XORbit01

🕸️ Crawl in the web network

Created Apr 2023

crawler crawling data data-science datamining
Total 385
dennis-tra

🌌 An agnostic network crawler exposing comprehensive peer information and network topology information.

Created Jun 2021

cid crawler filecoin golang hacktoberfest
Total 382
s0rg

The unix-way web crawler

Created Oct 2021

cli crawler go golang golang-application
Total 342
infinilabs

🕷️ An easy-to-use spider written in Golang. (previous named GOPA.)

Created Jul 2017

crawler crawling elasticsearch lightweight scraping
Total 311
devanshbatham

Gorecon is a All in one Reconnaissance Tool , a.k.a swiss knife for Reconnaissance , A tool that every pentester/bughunter might wanna consider into their arsenal

Created Feb 2019

admin-panel-finder backups-finder cmsdetecter configurationfiles crawler
Total 288
hezhizheng

golang spider Crawler 爬虫 电影

Created Dec 2019

colly crawler docker fasthttp go
Total 275
Ptt-Alertor

📢 Ptt 文章通知機器人!Notify Ptt Article in Realtime

Created Jan 2017

chatbot crawler linebot messenger-bot ptt
Total 266
antchfx

Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Created Sep 2017

crawler crawling framework golang scraping
Total 265
dwisiswant0

A fast tool to fetch URLs from HTML attributes by crawl-in.

Created Nov 2020

crawler devtool extractor galer go
Total 263
R4yGM

DorkScout - Golang tool to automate google dork scan against the entiere internet or specific targets

Created Jul 2021

bug-bounty crawler ghdb golang google-dorks
Total 244
gosom

Golang Crawling and scraping framework

Created Mar 2023

crawler go go-framework golang scraper
Total 219
0xsha

Rapid Smart Contract Crawler

Created Jun 2022

blockchain crawler dataset evm-bytecode geth
Total 203
songtianyi

golang light-weight image crawler

Created Dec 2016

aiss crawler douban downloader girls
Total 202
nfx
nfx / slrp

rotating open proxy multiplexer

Created Mar 2022

crawler golang proxy proxy-checker proxy-list
Total 199
karust

Extract web archive data using Wayback Machine and Common Crawl

Created Jun 2019

commoncrawl concurrency crawler golang wayback-machine
Total 187
saeeddhqan

Interactive CLI Web Crawler

Created Jul 2020

cli crawler data-mining fuzzing go
Total 183
twiny

Domain names collector - Crawl websites and collect domain names along with their availability status.

Created Jul 2020

backlinks crawler domain expired-domain golang
Total 165
2young2simple

一款分布式爬虫平台,帮助你更好的管理和开发爬虫。 内置一套爬虫定义规则(模版),可使用模版快速定义爬虫,也可当作框架手动开发爬虫。(兴趣使然的项目,用的不爽了就更新)

Created Oct 2017

crawler golang spider
Total 161
go-crawler

Golang爬虫 爬取豆瓣电影Top250

Created Mar 2018

crawler douban go golang movie
Total 159
rookmoot

A fast, modern and intelligent proxy rotator perfect for crawling and scraping public data.

Created Jan 2017

crawler crawling proxy proxy-server rotating-proxy
Total 156
wuchunfu

Golang 实现的 IP 代理池, 涉及到的技术点: go gorm proxy proxypool ip crawler 爬虫 mysql viper cobra

Created Oct 2020

crawler go ip proxy proxy-server
Total 153

Related

All topics →