Categories

Data Integration Frameworks

Frameworks for performing ELT / ETL

Repos

24

Stars

26,059

30 days

+742

jf-tech

omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

Created Aug 2020

codeless csv delimited edi edifact
Total 1,086
30 days +0

More

cloudquery

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Created Nov 2020

airbyte attack-surface-management aws azure bigquery
Total 6,563
30 days +27
PeerDB-io

Fast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage

Created Apr 2023

bigquery cdc clickhouse cloud-native distributed-systems
Total 3,325
30 days +20
apache

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Created Jul 2021

dashboard-friendly data data-analysis data-engineering data-integration
Total 3,205
30 days +32
datazip-inc

OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.

Created Oct 2024

apache-iceberg cdc change-data-capture data-ingestion data-pipeline
Total 2,038
30 days +480
rwynn

a go daemon that syncs MongoDB to Elasticsearch in realtime. you know, for search.

Created May 2016

change-streams connector daemon elasticsearch etl
Total 1,332
30 days +2
slingdata-io

Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.

Created Oct 2020

elt etl extract load
Total 929
30 days +12
ConduitIO

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

Created Jan 2022

conduit data-engineering data-integration data-pipeline data-stream
Total 619
30 days +3
galaxy-io

Pluggable data replication with checkpointing, batching, and integrity events

Created Jun 2026

change-data-capture data data-collection data-integration data-pipeline
Total 333
30 days +110
hw2499

etl engine 轻量级 跨平台 流批一体ETL引擎 数据抽取-转换-装载 ETL engine lightweight cross platform batch flow integration ETL engine data extraction transformation loading

Created Apr 2022

cdc clickhouse crontab datafusion elastic
Total 78
30 days +0
realdatadriven

ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.

Created Dec 2024

data-engineering data-lake data-lakehouse data-quality data-quality-checks
Total 57
30 days +2
compose

Sync data between persistence engines, like ETL only not stodgy

Created Oct 2014 Archived

elasticsearch etl go mongodb mysql
Total 1,438
30 days +0
jitsucom

Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)

Created Jun 2022 Archived

data-engineering datawarehouse etl etl-pipeline ingestion
Total 224
30 days +0

From awesome-go (MIT), plus repos sorted by Jev.