| Asynchronous I/O in DuckDB: Work, Thread, Work(duckdb.org) | |
| 280 points by pdet 11 days ago | 31 comments | |
tl;dr: DuckDB v2.0 (fall 2026) introduces asynchronous I/O for Parquet and CSV reads, using a separate ASYNC thread pool and a read-ahead queue with memory governance to keep remote fetches in flight while worker threads decode. Benchmarks on TPC-H Q6 against S3 show ~3x speedup for Parquet (up to 3.7x when tuned to saturate 25 Gbit/s), ~20x for CSV, and ~1.5x for cold local reads. Concurrent query workloads see CPU utilization jump from ~6 to ~48 cores on a 64-vCPU machine, with JSON and DuckDB-native format support planned next. | |
HN Discussion:
| |