Apache Arrow DataFusion and Ballista query engines
DataFusion is an extensible query execution framework, written inRust, that uses Apache Arrow as itsin-memory format.
DataFusion supports both an SQL and a DataFrame API for buildinglogical query plans as well as a query optimizer and execution enginecapable of parallel execution against partitioned data sources (CSVand Parquet) using threads.
DataFusion also supports distributed query execution via theBallista crate.
DataFusion is used to create modern, fast and efficient datapipelines, ETL processes, and database systems, which need theperformance of Rust and Apache Arrow and want to provide their usersthe convenience of an SQL interface or a DataFrame API.
High Performance: Leveraging Rust and Arrow's memory model, DataFusion achieves very high performance
Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific usecase
High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.
Projects that adapt to or serve as plugins to DataFusion: