omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
Created Aug 2020
Repos
24
Stars
26,061
Last 30 days
+742
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
Created Aug 2020
Data pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.
Created Nov 2020
Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.
Created Jul 2021
OLake Go: open-source replication from databases, Kafka and S3 to Apache Iceberg and Parquet. Sources: Postgres, MySQL, MongoDB, Oracle, MSSQL, DB2, Kafka, S3. Formerly datazip-inc/olake. Table maintenance lives in datazip-inc/olake-fusion.
Created Oct 2024
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Created Aug 2023
Sling is a CLI tool that extracts data from a source storage/database and loads it in a target storage/database.
Created Oct 2020
Open Source Cloud Native Ingestion engine
Created Jun 2024
A Mysql data to file/elasticsearch/kafka/redis/redis-cluster/mongodb(audit and normal)/TiDB/postgresql with binlog tool.
Created May 2019
Logically replicate data out of Postgres into sinks (files, Google BigQuery, etc)
Created Apr 2019
etl engine 轻量级 跨平台 流批一体ETL引擎 数据抽取-转换-装载 ETL engine lightweight cross platform batch flow integration ETL engine data extraction transformation loading
Created Apr 2022
Real-time, one-way CDC from production MongoDB, MySQL, MariaDB, PostgreSQL and Redis to secondary databases or GCS, with PII masking and encryption. Built for analytics, cold storage and disaster recovery.
Created Nov 2024
Library and worker to handle transfer of data in s3 into redshift. Includes table creation and manipulation, as well as time-based insertion.
Created Jan 2015
ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.
Created Dec 2024
Service for moving your Fullstory export files to a data warehouse
Created Aug 2017
Sync data between persistence engines, like ETL only not stodgy
Created Oct 2014 Archived
From awesome-go (MIT), plus repos sorted by Jev.