System Design Cases
DuckDB
DuckDB — embedded analytical SQL engine ("SQLite for OLAP"). In-process columnar engine with vectorized execution (1024-row batches, SIMD), reads Parquet/Arrow/CSV directly with predicate+projection pushdown. Single-writer multi-reader. Compared with Pandas, ClickHouse, Snowflake. 5 scenarios: SELECT FROM parquet (zero-copy), 1B rows aggregation on laptop, JOIN parquet × csv, DuckDB-Wasm in browser, OLTP misuse anti-pattern. 2 ADRs: DuckDB vs alternatives, vectorized execution rationale.
DuckDB: in-process analytical execution and file-oriented data access
DuckDB is an embeddable analytical database with columnar-vectorized execution. It can query Parquet directly and push projections and filters into scans, including remote files when the filesystem and format support the required range access. That is selective I/O, not a guarantee of zero copy or zero network transfer.
Within one read-write process DuckDB supports multiple writer threads using MVCC and optimistic concurrency control: appends do not conflict, while concurrent changes to the same row can return a conflict. The stable native-file model does not make several independent processes concurrent writers to one file.
Модель и предпосылки
- DuckDB runs inside a host process; its memory, CPU, filesystem, credentials, and failure lifecycle are part of the application boundary.
- Parquet row groups, statistics, compression, file count, projection, and filters determine scanned bytes and parallelism.
- Larger-than-memory operators can spill to a configured temporary directory, but some operators/states can still exhaust memory.
Проверяемые утверждения
- C1. DuckDB is in-process and optimized for analytical workloads using columnar-vectorized execution.
- C2. One read-write process can run multiple writer threads; non-conflicting writes succeed, appends do not conflict, and same-row update/delete conflicts can fail.
- C3. Parquet projection/filter pushdown can skip columns and row groups; actual remote bytes depend on metadata, statistics, ranges, and predicates.
- C4. DuckDB supports disk spilling for larger-than-memory work, with documented exceptions and possible OOM.
- C5. DuckDB-Wasm runs in browsers but is single-threaded by default and constrained by WebAssembly/browser memory and CORS.
Исполняемые сценарии
Parquet projection and filter pushdown. The engine reads required columns and may skip row groups; bytes are measured rather than declared zero.
Multiple writers in one process. Non-conflicting appends/updates can commit concurrently; conflicting same-row updates fail optimistically.
Larger-than-memory spill with limits. Blocking operators can spill, while disk capacity and unsupported intermediate states remain failure modes.
DuckDB-Wasm browser boundary. Browser execution keeps data local but inherits Wasm memory, default threading, CORS, and device constraints.
OLTP misuse boundary. Many independent services do not coordinate writes by opening the same native file; keep a supported authority.
Ошибки проектирования
- Не обещайте «миллиард строк на любом ноутбуке»: data shape, operators, memory, temp disk и threads определяют результат.
- Не используйте native DuckDB file как общий multi-process OLTP server без поддерживаемого coordination layer.
- Не считайте remote Parquet бесплатным: listing, metadata, range requests, egress и poor row-group layout остаются.
- Не исполняйте untrusted SQL/files в основном процессе без sandbox и resource limits.
Границы гарантии
- DuckDB может хранить таблицы и транзакции, но его design center — analytics, не high-concurrency network OLTP.
- Pushdown зависит от выражения и доступной статистики; сложный predicate может читаться шире.
- Wasm и native имеют разные extension/filesystem/security capabilities.