Categories

Data Integration Frameworks

Frameworks for performing ELT / ETL

Repos

24

Stars

26,061

30 days

+742

jf-tech

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

Created Aug 2020

codeless csv delimited edi edifact
Total 1,087
30 days +1

More

cloudquery

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Created Nov 2020

airbyte attack-surface-management aws azure bigquery
Total 6,564
30 days +27
PeerDB-io

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

Created Apr 2023

bigquery cdc clickhouse cloud-native distributed-systems
Total 3,325
30 days +20
apache

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Created Jul 2021

dashboard-friendly data data-analysis data-engineering data-integration
Total 3,206
30 days +32
brokercap

Bifrost ---- 面向生产环境的 MySQL,MariaDB,kafka 同步到Redis,MongoDB,ClickHouse,StarRocks,Doris,Kafka等服务的异构中间件

Created Apr 2018

cdc-go clickhouse datalink datax dts
Total 2,092
30 days +4
datazip-inc

OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

Created Oct 2024

apache-iceberg cdc change-data-capture data-ingestion data-pipeline
Total 2,036
30 days +478
bruin-data

Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.

Created Aug 2023

analytics bigquery data-analysis data-ingestion data-modeling
Total 1,819
30 days +37
rwynn

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.

Created May 2016

change-streams connector daemon elasticsearch etl
Total 1,333
30 days +3
slingdata-io

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

Created Oct 2020

elt etl extract load
Total 929
30 days +12
ConduitIO

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

Created Jan 2022

conduit data-engineering data-integration data-pipeline data-stream
Total 619
30 days +3
galaxy-io

Pluggable data replication with checkpointing, batching, and integrity events

Created Jun 2026

change-data-capture data data-collection data-integration data-pipeline
Total 332
30 days +109
jondot

A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop.

Created Nov 2014

Total 212
30 days +0
transferia

Open Source Cloud Native Ingestion engine

Created Jun 2024

bigdata cdc clickhouse elt go
Total 199
30 days +5
gitstliu

A Mysql data to file/elasticsearch/kafka/redis/redis-cluster/mongodb(audit and normal)/TiDB/postgresql with binlog tool.

Created May 2019

audit-log binlog elasticsearch golang kafka
Total 99
30 days +0
lawrencejones

Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)

Created Apr 2019

bigquery data google postgres postgres-change-capture
Total 91
30 days +0
estuary

Connectors for capturing data from external data sources

Created May 2021

Total 83
30 days +1
hw2499

etl engine 轻量级 跨平台 流批一体ETL引擎 数据抽取-转换-装载 ETL engine lightweight cross platform batch flow integration ETL engine data extraction transformation loading

Created Apr 2022

cdc clickhouse crontab datafusion elastic
Total 78
30 days +0
d8a-tech

🇪🇺 Warehouse-native Analytics compatible with Google Analytics and Matomo tracking protocols. Ingest into ClickHouse, BigQuery or CSV/Parquet on S3/GCS/local fs while maintaining complete control over your data.

Created Aug 2025

bigquery clickhouse data-warehouse ga4 matomo
Total 64
30 days +8
retail-ai-inc

Real-time, one-way CDC from production MongoDB, MySQL, MariaDB, PostgreSQL and Redis to secondary databases or GCS, with PII masking and encryption. Built for analytics, cold storage and disaster recovery.

Created Nov 2024

cdc change-data-capture cold-storage data-masking database-replication
Total 64
30 days +0
graveyard

Library and worker to handle transfer of data in s3 into redshift. Includes table creation and manipulation, as well as time-based insertion.

Created Jan 2015

Total 60
30 days +0
realdatadriven

ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.

Created Dec 2024

data-engineering data-lake data-lakehouse data-quality data-quality-checks
Total 57
30 days +2
fullstorydev

Service for moving your Fullstory export files to a data warehouse

Created Aug 2017

Total 50
30 days +0
compose

Sync data between persistence engines, like ETL only not stodgy

Created Oct 2014 Archived

elasticsearch etl go mongodb mysql
Total 1,438
30 days +0
jitsucom

Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)

Created Jun 2022 Archived

data-engineering datawarehouse etl etl-pipeline ingestion
Total 224
30 days +0

From awesome-go (MIT), plus repos sorted by Jev.