GitHub topics

kafka

Mostly in Messaging 10%

Repositories

148

Total stars

147,185

Trending

datazip-inc

OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

Created Oct 2024

apache-iceberg cdc change-data-capture data-ingestion data-pipeline
Total 2,038
30 days +480
openmeterio

Metering and Billing for AI, API and DevOps. Collect and aggregate millions of usage events in real-time and enable usage-based billing.

Created Jun 2023

analytics billing clickhouse cost finops
Total 2,457
30 days +73
influxdata

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

Created Apr 2015

golang hacktoberfest influxdb json kafka
Total 17,933
30 days +69
ThreeDotsLabs

Building event-driven applications the easy way in Go.

Created Nov 2018

cqrs event-driven event-sourcing events go
Total 9,981
30 days +54
twmb

franz-go is a feature complete, pure Go library for Kafka from 0.8.0 through 4.4+. Producing, consuming, transacting, administrating, etc.

Created Mar 2019

client go golang kafka kafka-client
Total 3,162
30 days +51
warpstreamlabs

Fancy stream processing made operationally mundane. This repository is a fork of the original project before the license was changed.

Created May 2024

benthos bento connect connectors kafka
Total 2,211
30 days +46
akvorado

Flow collector, enricher and visualizer

Created Jun 2022

clickhouse collector ipfix kafka netflow
Total 2,431
30 days +45
flexprice

Usage-based pricing and billing for developers 🔓 Cloud or self-hosted ⚙️ No-code UI 💰 Realtime usage metering 🎟 Credits & top-ups 🔑 Control feature access

Created Nov 2024

billing clickhouse events go hacktoberfest
Total 6,875
30 days +35
Mikaelemmmm

🔥基于go-zero(go zero) 微服务全技术栈开发最佳实践项目。Develop best practice projects based on the full technology stack of go zero (go zero) microservices.

Created Dec 2021

asynq dtm elasticsearch filebeat gitlab
Total 5,236
30 days +25
redpanda-data

Fancy stream processing made operationally mundane

Created Mar 2016

amqp cqrs data-engineering data-ops etl
Total 8,812
30 days +24

Top

Top 50 of 148

influxdata

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.

Created Apr 2015

golang hacktoberfest influxdb json kafka
Total 17,933
IBM

Sarama is a Go library for Apache Kafka.

Created Jul 2013

go kafka kafka-client
Total 12,539
ThreeDotsLabs

Building event-driven applications the easy way in Go.

Created Nov 2018

cqrs event-driven event-sourcing events go
Total 9,981
redpanda-data

Fancy stream processing made operationally mundane

Created Mar 2016

amqp cqrs data-engineering data-ops etl
Total 8,812
segmentio

Kafka library in Go

Created May 2017

consumer go golang golang-library golang-native
Total 8,653
flexprice

Usage-based pricing and billing for developers 🔓 Cloud or self-hosted ⚙️ No-code UI 💰 Realtime usage metering 🎟 Credits & top-ups 🔑 Control feature access

Created Nov 2024

billing clickhouse events go hacktoberfest
Total 6,875
Mikaelemmmm

🔥基于go-zero(go zero) 微服务全技术栈开发最佳实践项目。Develop best practice projects based on the full technology stack of go zero (go zero) microservices.

Created Dec 2021

asynq dtm elasticsearch filebeat gitlab
Total 5,236
travisjeffery

Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)

Created Oct 2016

distributed-systems go kafka messaging queue
Total 5,009
nytimes

A Microservice Toolkit from The New York Times

Created Dec 2015

gizmo go google-pubsub googlecloud http-server
Total 3,772
PeerDB-io

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

Created Apr 2023

bigquery cdc clickhouse cloud-native distributed-systems
Total 3,325
twmb

franz-go is a feature complete, pure Go library for Kafka from 0.8.0 through 4.4+. Producing, consuming, transacting, administrating, etc.

Created Mar 2019

client go golang kafka kafka-client
Total 3,162
lovoo

Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go.

Created Mar 2017

goka golang kafka microservices processing
Total 2,552
danielqsj

Kafka exporter for Prometheus

Created Sep 2017

kafka kafka-metrics metrics prometheus prometheus-exporter
Total 2,541
openmeterio

Metering and Billing for AI, API and DevOps. Collect and aggregate millions of usage events in real-time and enable usage-based billing.

Created Jun 2023

analytics billing clickhouse cost finops
Total 2,457
akvorado

Flow collector, enricher and visualizer

Created Jun 2022

clickhouse collector ipfix kafka netflow
Total 2,431
warpstreamlabs

Fancy stream processing made operationally mundane. This repository is a fork of the original project before the license was changed.

Created May 2024

benthos bento connect connectors kafka
Total 2,211
reugn

A lightweight stream processing library for Go

Created Apr 2019

aerospike data-pipeline data-stream etl kafka
Total 2,179
streamdal

A swiss army knife CLI tool for interacting with Kafka, RabbitMQ and other messaging systems.

Created Jul 2020

event-bus event-driven golang hacktoberfest kafka
Total 2,155
datazip-inc

OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

Created Oct 2024

apache-iceberg cdc change-data-capture data-ingestion data-pipeline
Total 2,038
betty200744

This repo contains my notes on working with Go and computer systems.

Created Sep 2019

algorithms data-structures docker-compose example gin
Total 1,936
vanus-labs

Vanus is a Serverless, event streaming system with processing capabilities. It easily connects SaaS, Cloud Services, and Databases to help users build next-gen Event-driven Applications.

Created Jan 2022

cloud-native cloudevents cloudnative event-bridge event-driven
Total 1,698
looplab

Event Sourcing for Go!

Created Nov 2014

aws cqrs ddd domain-driven-design entity
Total 1,680
orlangure

Test your code without writing mocks with ephemeral Docker containers 📦 Setup popular services with just a couple lines of code ⏱️ No bash, no yaml, only code 💻

Created Jan 2020

cockroachdb docker elasticsearch end-to-end-testing go
Total 1,487
fhmq
fhmq / hmq

High performance mqtt broker

Created Aug 2017

acl bridge broker cluster go
Total 1,393
wait4x

Wait4X allows you to wait for a port or a service to enter the requested state.

Created Nov 2019

cli dns go golang http
Total 1,382
kevwan

go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.

Created Aug 2020

elasticsearch elk kafka logstash
Total 1,226
kudobuilder

Kubernetes Universal Declarative Operator (KUDO)

Created Nov 2018

cluster cncf crd hacktoberfest kafka
Total 1,216
Edgio

Enterprise Network Flow Collector (IPFIX, sFlow, Netflow)

Created Feb 2017

cisco golang ipfix ipv4 ipv6
Total 1,156
childe

使用 golang 模仿的 Logstash。用于消费 Kafka 数据,处理后写入 ES、Clickhouse 等。

Created Jan 2018

elasticsearch golang kafka logstash
Total 1,102
deviceinsight

Command Line Tool for managing Apache Kafka

Created Dec 2018

apache-kafka avro cli fish golang
Total 1,075
bsm

Cluster extensions for Sarama, the Go client library for Apache Kafka 0.9 [DEPRECATED]

Created Dec 2014

consumer deprecated go kafka sarama
Total 1,000
fgeller

Kafka command line tool that likes JSON

Created Jan 2016

apache-kafka cli golang json kafka
Total 952
cloudflare

The high-scalability sFlow/NetFlow/IPFIX collector used internally at Cloudflare.

Created Mar 2018

cisco flow go ipfix juniper
Total 917
netsampler

High performance sFlow/IPFIX/NetFlow Collector

Created Apr 2021

collector go ipfix kafka netflow
Total 832
mailgun

gRPC/REST proxy for Kafka

Created Feb 2015

grpc http json kafka proxy
Total 791
banzaicloud

Oh no! Yet another Apache Kafka operator for Kubernetes

Created May 2018

cruisecontrol kafka kafka-operator kubernetes
Total 788
team-ide

Team IDE 集成MySql、Oracle、金仓、达梦、神通等数据库、SSH、FTP、Redis、Zookeeper、Kafka、Elasticsearch、Mongodb、小工具等管理工具

Created Nov 2021

database elasticsearch file-manager kafka mongodb
Total 785
redpanda-data

KMinion is a feature-rich Prometheus exporter for Apache Kafka written in Go. It is lightweight and highly configurable so that it will meet your requirements.

Created Feb 2019

consumer consumer-group exporter group kafka
Total 711
rasadov

Modular e-commerce backend with a GraphQL gateway and gRPC microservices for accounts, products, orders, payments, and recommendations.

Created Jan 2025

backend docker docker-compose ecommerce elasticsearch
Total 697
grepplabs

Proxy connections to Kafka cluster. Connect through SOCKS Proxy, HTTP Proxy or to cluster running in Kubernetes.

Created Feb 2018

kafka kafka-gateway kafka-proxy oauthbearer proxy
Total 634
Mongey

Terraform provider for managing Apache Kafka Topics + ACLs

Created Sep 2017

kafka kafka-acl kafka-topic terraform terraform-provider
Total 628
ConduitIO

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

Created Jan 2022

conduit data-engineering data-integration data-pipeline data-stream
Total 619
dapr

Community driven, reusable components for distributed, durable, and agentic AI applications across all cloud service providers

Created Sep 2019

agent-memory agentic-storage anthropic-ai anthropic-compatible async-messaging
Total 606
apache

Based on the proxy gateway service of dubbo-go, it solves the problem that the external protocol calls the internal Dubbo cluster. At present, it supports HTTP and gRPC[developing].

Created Jan 2020

api-gateway api-rest cloud-native cloudnative dubbo
Total 577
housepower

Easily load data from kafka to ClickHouse

Created May 2018

clickhouse clickhouse-bulk clickhouse-server kafka
Total 533
ozontech

A blazing fast tool for building data pipelines: read, process and output events. Our community: https://t.me/file_d_community

Created Feb 2020

actions clickhouse elasticsearch events file
Total 508
DataDog

Kafka storage rebalancing, automated replication throttle, cluster API and more

Created Aug 2017

golang kafka kafka-kit kafka-topic-management rack-aware
Total 502
pingcap

This repo maintains DM (a data migration platform) and TiCDC (change data capture for TiDB)

Created Aug 2019

cdc dm kafka mysql ticdc
Total 466
tx7do

Unified transport and broker extension set for the Kratos microservice framework: 30+ message-queue, RPC-framework, realtime, and network-protocol transports as pluggable modules, with generic typed handlers, a unified message abstraction, publish/subscribe middleware chains, and OpenTelemetry tracing.

Created Mar 2022

activemq amqp asynq gin go-kratos
Total 458
mhausenblas

Distributed Named Pipes

Created Dec 2016

distributed-systems ipc kafka named-pipes pipe
Total 452

Related

All topics →