Repositories

39

Total stars

53,675

Trending

datazip-inc

OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

Created Oct 2024

apache-iceberg cdc change-data-capture data-ingestion data-pipeline
Total 2,038
30 days +480
galaxy-io

Pluggable data replication with checkpointing, batching, and integrity events

Created Jun 2026

change-data-capture data data-collection data-integration data-pipeline
Total 333
30 days +110
turbot

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

Created Jan 2021

aws azure cis cloud cnapp
Total 8,027
30 days +37
apache

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Created Jul 2021

dashboard-friendly data data-analysis data-engineering data-integration
Total 3,206
30 days +32
cloudquery

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Created Nov 2020

airbyte attack-surface-management aws azure bigquery
Total 6,563
30 days +27
redpanda-data

Fancy stream processing made operationally mundane

Created Mar 2016

amqp cqrs data-engineering data-ops etl
Total 8,812
30 days +24
PeerDB-io

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

Created Apr 2023

bigquery cdc clickhouse cloud-native distributed-systems
Total 3,325
30 days +20
rudderlabs

Privacy and Security focused Segment-alternative, in Golang and React

Created Jul 2019

bigquery cdp customer-data customer-data-lake customer-data-pipeline
Total 4,524
30 days +18
dexadata

Apache Airflow-compatible workflow orchestrator, re-engineered in Go: the drop-in Airflow UI and REST API on a single-binary control plane (no Python GIL), with pod-per-task isolation and GitOps-native DAGs. Self-host on one machine (Lite) or Kubernetes (Pro).

Created May 2026

airflow airflow-alternative apache-airflow container-native dag
Total 64
30 days +15
slingdata-io

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

Created Oct 2020

elt etl extract load
Total 929
30 days +12

Top

redpanda-data

Fancy stream processing made operationally mundane

Created Mar 2016

amqp cqrs data-engineering data-ops etl
Total 8,812
turbot

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.

Created Jan 2021

aws azure cis cloud cnapp
Total 8,027
cloudquery

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Created Nov 2020

airbyte attack-surface-management aws azure bigquery
Total 6,563
rudderlabs

Privacy and Security focused Segment-alternative, in Golang and React

Created Jul 2019

bigquery cdp customer-data customer-data-lake customer-data-pipeline
Total 4,524
nucleuscloud

Open Source Data Security Platform for Developers to Monitor and Detect PII, Anonymize Production Data and Sync it across environments.

Created Aug 2023

benthos docker etl faker fine-tuning
Total 4,129
PeerDB-io

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

Created Apr 2023

bigquery cdc clickhouse cloud-native distributed-systems
Total 3,325
apache

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Created Jul 2021

dashboard-friendly data data-analysis data-engineering data-integration
Total 3,206
reugn

A lightweight stream processing library for Go

Created Apr 2019

aerospike data-pipeline data-stream etl kafka
Total 2,180
datazip-inc

OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

Created Oct 2024

apache-iceberg cdc change-data-capture data-ingestion data-pipeline
Total 2,038
compose

Sync data between persistence engines, like ETL only not stodgy

Created Oct 2014

elasticsearch etl go mongodb mysql
Total 1,438
rwynn

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.

Created May 2016

change-streams connector daemon elasticsearch etl
Total 1,332
jf-tech

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

Created Aug 2020

codeless csv delimited edi edifact
Total 1,087
slingdata-io

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

Created Oct 2020

elt etl extract load
Total 929
raystack

Optimus is an easy-to-use, reliable, and performant workflow orchestrator for data transformation, data modeling, pipelines, and data quality management.

Created Mar 2021

airflow analytics analytics-engineering automation bigquery
Total 770
onepanelio

The open source, end-to-end computer vision platform. Label, build, train, tune, deploy and automate in a unified platform that runs on any cloud and on-premises.

Created Sep 2019

ai aiops annotation computer-vision deeplearning
Total 732
ConduitIO

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

Created Jan 2022

conduit data-engineering data-integration data-pipeline data-stream
Total 619
grailbio

A serverless cluster computing system for the Go programming language

Created Sep 2019

bigdata cluster computing etl go
Total 556
appbaseio

Power of appbase.io via CLI, with nifty imports from your favorite data sources

Created May 2017

appbase cli elasticsearch etl
Total 471
nao1215

loads CSV, TSV, LTSV, JSON, JSONL, Parquet, XLSX, ACH, and Fedwire files into SQLite; includes prep and frame for cleanup and in-memory transforms

Created Aug 2025

ach csv data-preprocessing dataframe etl
Total 389
uber

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

Created May 2017

avro cdc clickhouse etl hdfs
Total 364
galaxy-io

Pluggable data replication with checkpointing, batching, and integrity events

Created Jun 2026

change-data-capture data data-collection data-integration data-pipeline
Total 333
turbot

Use SQL to instantly query AWS resources across regions and accounts. Open source CLI. No DB required.

Created Jan 2021

aws aws-cli backup etl hacktoberfest
Total 232
jitsucom

Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)

Created Jun 2022

data-engineering datawarehouse etl etl-pipeline ingestion
Total 224
Breeze0806

go-etl is a toolset for data extraction, transformation and loading.

Created Nov 2020

csv dameng db2 etl go
Total 194
pbnjay

A Go native tabular data extraction package. Currently supports .xls, .xlsx, .csv, .tsv formats.

Created Feb 2021

etl xls xlsx
Total 155
hkpeaks

The Peaks Consolidation is equipped with state-of-the-art algorithms and data structures that support high-performance databending exercises. It specializes in management accounting and consolidation, with some special topics in machine learning and bioinformatics.

Created Apr 2023

accounting benchmark csv dataframe etl
Total 106
renart-data

Open source data platform for SQL and Python pipelines, notebooks and dashboards, with type checking, scheduling and definitions in Git.

Created Apr 2026

dashboards data-engineering data-pipeline data-platform developer-tools
Total 96
turbot

Use SQL to instantly query repositories, users, gists and more from GitHub. Open source CLI. No DB required.

Created Jan 2021

backup etl github github-cli github-client
Total 89
beneath-hq

Beneath is a serverless real-time data platform ⚡️

Created Apr 2020

analytics beneath data-engineering data-pipelines data-science
Total 84
hw2499

etl engine 轻量级 跨平台 流批一体ETL引擎 数据抽取-转换-装载 ETL engine lightweight cross platform batch flow integration ETL engine data extraction transformation loading

Created Apr 2022

cdc clickhouse crontab datafusion elastic
Total 78
hofstadter-io

CLI and library for diff, patch, and ETL operations on CUE, JSON, and Yaml

Created Oct 2021

configuration cue cuelang diff etl
Total 78
zeus-fyi

When no one can tell the difference between art, and an empty canvas, the meaning is lost.

Created Nov 2022

aws azure devops digitalocean ethereum
Total 75
dbtrail

HTAP Zero-ETL analytical replica for MySQL, RDS and Aurora. Reads the binlog like a replica, keeps Parquet in your bucket, opens in DuckDB. Time travel and row-level undo included.

Created Feb 2026

binlog duckdb duckdb-community duckdb-database duckdb-server
Total 72
maxim2266

csvplus extends the standard Go encoding/csv package with fluent interface, lazy stream operations, indices and joins.

Created Sep 2016

csv csv-format etl etl-framework etl-pipeline
Total 66
dexadata

Apache Airflow-compatible workflow orchestrator, re-engineered in Go: the drop-in Airflow UI and REST API on a single-binary control plane (no Python GIL), with pod-per-task isolation and GitOps-native DAGs. Self-host on one machine (Lite) or Kubernetes (Pro).

Created May 2026

airflow airflow-alternative apache-airflow container-native dag
Total 64
turbot

Use SQL to instantly query Kubernetes API resources. Open source CLI. No DB required.

Created Mar 2021

backup etl hacktoberfest k8s kubernetes
Total 64
turbot

Steampipe SQLite is a zero-ETL engine for SQLite. Virtual tables translate queries into live API calls for cloud services and APIs. Hundreds of plugins with thousands of documented examples.

Created Oct 2023

aws azure data devsecops etl
Total 58
realdatadriven

ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.

Created Dec 2024

data-engineering data-lake data-lakehouse data-quality data-quality-checks
Total 57
chararch

GoBatch is a batch processing framework in Go like Spring Batch in Java

Created Dec 2021

batch batch-processing etl go go-batch
Total 57

Related

All topics →