GitHub topics

data-engineering

Mostly in Data Integration Frameworks 25%

Repositories

20

Total stars

52,929

Trending

argoproj

Workflow Engine for Kubernetes

Created Aug 2017

airflow argo argo-workflows batch-processing cloud-native
Total 17,124
30 days +78
treeverse

lakeFS - Data version control for your data lake | Git for data

Created Sep 2019

apache-spark apache-sparksql aws-s3 azure-blob-storage azure-storage
Total 5,591
30 days +34
apache

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Created Jul 2021

dashboard-friendly data data-analysis data-engineering data-integration
Total 3,206
30 days +32
cloudquery

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Created Nov 2020

airbyte attack-surface-management aws azure bigquery
Total 6,564
30 days +27
redpanda-data

Fancy stream processing made operationally mundane

Created Mar 2016

amqp cqrs data-engineering data-ops etl
Total 8,812
30 days +24
rudderlabs

Privacy and Security focused Segment-alternative, in Golang and React

Created Jul 2019

bigquery cdp customer-data customer-data-lake customer-data-pipeline
Total 4,524
30 days +18
dexadata

Apache Airflow-compatible workflow orchestrator, re-engineered in Go: the drop-in Airflow UI and REST API on a single-binary control plane (no Python GIL), with pod-per-task isolation and GitOps-native DAGs. Self-host on one machine (Lite) or Kubernetes (Pro).

Created May 2026

airflow airflow-alternative apache-airflow container-native dag
Total 64
30 days +15
nnnkkk7

A lightweight Snowflake emulator built with Go and DuckDB for local development and testing

Created Dec 2025

ci-cd cicd data-engineering database docker
Total 60
30 days +7
bacalhau-project

Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.

Created Nov 2021

ai-art ai-data-collection ai-pipeline batch-processing bioinformatics-pipeline
Total 878
30 days +4
renart-data

Open source data platform for SQL and Python pipelines, notebooks and dashboards, with type checking, scheduling and definitions in Git.

Created Apr 2026

dashboards data-engineering data-pipeline data-platform developer-tools
Total 96
30 days +3

Top

argoproj

Workflow Engine for Kubernetes

Created Aug 2017

airflow argo argo-workflows batch-processing cloud-native
Total 17,124
redpanda-data

Fancy stream processing made operationally mundane

Created Mar 2016

amqp cqrs data-engineering data-ops etl
Total 8,812
cloudquery

Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.

Created Nov 2020

airbyte attack-surface-management aws azure bigquery
Total 6,564
treeverse

lakeFS - Data version control for your data lake | Git for data

Created Sep 2019

apache-spark apache-sparksql aws-s3 azure-blob-storage azure-storage
Total 5,591
rudderlabs

Privacy and Security focused Segment-alternative, in Golang and React

Created Jul 2019

bigquery cdp customer-data customer-data-lake customer-data-pipeline
Total 4,524
superstreamlabs

Memphis.dev is a highly scalable and effortless data streaming platform

Created Feb 2022

data data-engineering data-pipeline data-stream-processing data-streaming
Total 3,446
apache

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Created Jul 2021

dashboard-friendly data data-analysis data-engineering data-integration
Total 3,206
bacalhau-project

Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.

Created Nov 2021

ai-art ai-data-collection ai-pipeline batch-processing bioinformatics-pipeline
Total 878
ConduitIO

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

Created Jan 2022

conduit data-engineering data-integration data-pipeline data-stream
Total 619
apecloud

Unified MySQL, Postgres & FlightSQL Server, Powered by DuckDB.

Created Aug 2024

analytics arrow business-analytics business-intelligence columnar-storage
Total 595
moabukar

A collection of online resources to help you on your Tech journey.

Created Oct 2021

ansible aws azure backend data-engineering
Total 435
jitsucom

Service for bulk-loading data to databases with automatic schema management (Redshift, Snowflake, BigQuery, ClickHouse, Postgres, MySQL)

Created Jun 2022

data-engineering datawarehouse etl etl-pipeline ingestion
Total 224
kevin-hanselman

A lightweight CLI tool for versioning data alongside source code and building data pipelines.

Created Feb 2020

data-engineering data-pipelines data-science dataset dvcs
Total 220
aleksandarskrbic

Kafka data generator and load testing tool - generate fake messages, simulate producers and consumers, test broker failures, and run chaos engineering scenarios

Created Dec 2025

apache-kafka apache-spark chaos-engineering consumer-lag data-engineering
Total 166
raptor-ml

Transform your pythonic research to an artifact that engineers can deploy easily.

Created Apr 2022

ai-infra data-engineering data-science dataops feature-engineering
Total 164
renart-data

Open source data platform for SQL and Python pipelines, notebooks and dashboards, with type checking, scheduling and definitions in Git.

Created Apr 2026

dashboards data-engineering data-pipeline data-platform developer-tools
Total 96
beneath-hq

Beneath is a serverless real-time data platform ⚡️

Created Apr 2020

analytics beneath data-engineering data-pipelines data-science
Total 84
dexadata

Apache Airflow-compatible workflow orchestrator, re-engineered in Go: the drop-in Airflow UI and REST API on a single-binary control plane (no Python GIL), with pod-per-task isolation and GitOps-native DAGs. Self-host on one machine (Lite) or Kubernetes (Pro).

Created May 2026

airflow airflow-alternative apache-airflow container-native dag
Total 64
nnnkkk7

A lightweight Snowflake emulator built with Go and DuckDB for local development and testing

Created Dec 2025

ci-cd cicd data-engineering database docker
Total 60
realdatadriven

ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.

Created Dec 2024

data-engineering data-lake data-lakehouse data-quality data-quality-checks
Total 57

Related

All topics →