v0.2: cascades optimizer, SIMD, and better storage

After 9 months' development, we are happy to announce a new version of RisingLight. Major changes in this version:

• New cascades-style binder / planner / optimizer framework powered by egg.
• New SIMD expression evaluation design with better performance.
• More efficient storage with better encoding and predicate push-downs.
• Better docs on the storage layer.

Thanks for your support!

What's Changed

• docs: migrate to new Slack community by @skyzh in #606
• docs: add architecture overview by @skyzh in #609
• feat(executor): introduce DataChunkBuilder and split chunks in TopN by @wangqiim in #610
• deps: upgrade prost 0.10 by @alissa-tung in #613
• feat(storage): record first_key in block_index_builder by @adlternative in #598
• feat(executor): introduce DataChunkBuilder and split chunks in Order … by @wangqiim in #612
• chore(ci): install protoc by direct download by @skyzh in #614
• feat(executor): implement SortMergeJoin executor by @D2Lark in #600
• chore(test): add more test case by @D2Lark in #616
• docs: add instructions to install protobuf by @skyzh in #617
• docs: add badges of dbdb.io, crates.io and docs.rs by @skyzh in #621
• feat(storage): add support composite sort key in MergeIterator by @BaymaxHWY in #622
• executor: use DataChunkbuilder to refactor ValuesExecutor by @JayiceZ in #623
• build: bump toolchain and deps, remove unused dep by @TennyZhuang in #629
• fix: make header correct when using group by by @shmiwy in #624
• refactor(planner): use AggInputRefResolver to handle group by expr by @Fedomn in #630
• feat(binder): support “primary key (a, b)” syntax by @shmiwy in #631
• feat: support multi ordered primary key by @shmiwy in #633
• community: add xxchan as maintainer by @skyzh in #634
• docs: add overview of storage by @skyzh in #635
• refactor(executor): refactor CopyFromFile using DataChunkBuilder by @noneback in #637
• fix(planner): enhance group by illegal column validation by @Fedomn in #642
• feat: use ref-cast for BlobRef cast by @Kikkon in #643
• feat(storage): support range-filter scan by sort key by @shmiwy in #644
• feat(optimizer): push the filter down below the agg by @lokax in #648
• binder: primary key should by-default be non-null by @Ted-Jiang in #649
• chore: remove Mutex in database/schema/table catalog by @chowc in #650
• feat: support sort agg by @Kikkon in #646
• fix: typo and clippy warnings by @yuzi-neko in #652
• feat(binder): support Having clause and more flexible OrderBy by @lokax in #651
• chore(sqllogictest): use libtest-mimic as test harness by @skyzh in #656
• chore(test): ensure single thread runtime for sqllogictest by @skyzh in #657
• feat(test): add sqlplannertest by @skyzh in #661
• feat(cli): support multi-line interactive mode by @Gun9niR in #662
• chore: upgrade dependencies by @wangrunji0408 in #663
• fix: 'unsupported command' everywhere by @cadl in #664
• fix: typo by @gogim1 in #665
• feat(storage): use variable size encoding to compress rle count by @xinchengxx in #666
• fix: create table with invalid primary keys constraints by @gogim1 in #667
• chore: bump dependencies by @skyzh in #672
• docs: add logo for RisingLight by @skyzh in #671
• fix: append NULLs when inserting value with incomplete columns by @gogim1 in #673
• feat(storage): support dict encoding by @Kikkon in #668
• feat(optimizer): support column pruning for some operators by @lokax in #653
• fix(storage): reserve correct size by @lokax in #678
• chore: bump dependencies, toolchain, and adapt to new sqlparser by @skyzh in #681
• chore: add github codespace devcontainer configuration by @yeya24 in #680
• refactor: refactor risinglight-sqllogictest by @xxchan in #682
• refactor: remove unnecessary current dir by @xxchan in #683
• storage: use Arc during pin and unpin by @chaixuqing in #685
• chore(storage): select compaction strategy using statistics by @chaixuqing in #689
• feat(planner): plan select distinct by @lokax in #686
• feat(executor): support perfect hash aggregate by @lokax in #690
• fix: add row_handler to schema by @yinfredyue in #692
• chore: bump toolchain and update dependencies by @wangrunji0408 in #695
• feat: introduce ordered float by @wangrunji0408 in #699
• feat(storage): support dict encoding in primitive column builder by @Kikkon in #701
• feat: make binary operations auto-vectorized by @lokax in #700
• feat: introduce next-gen planner & optimizer by @wangrunji0408 in #702
• feat(binder): introduce egg binder for queries by @wangrunji0408 in #705
• fix(storage): checksum BlockType in block header by @unconsolable in #704
• feat(planner): support distinct on by @wangrunji0408 in #706
• refactor(type): unify DataTypeKind by @wangrunji0408 in #708
• feat(storage): generic nullable block encoding by @unconsolable in #709
• feat(binder): egg binder for drop statement by @eliasyaoyc in #711
• feat(planner_v2): add type checker by @wangrunji0408 in #717
• refactor(storage): replace primitive nullable encoding with generic way by @unconsolable in #713
• feat(binder): new binder for copy, delete and create_table by @eliasyaoyc in #719
• fix(type): add NULL type and fix type analysis by @wangrunji0408 in #721
• feat(planner_v2): support explain by @wangrunji0408 in #722
• refactor(executor): simply drop the futures to cancel a query by @wangrunji0408 in #720
• feat(planner_v2): cost function and multi-stage optimization by @wangrunji0408 in #723
• feat: introduce executor v2 for the new planner by @wangrunji0408 in #725
• feat(planner_v2): support standalone VALUES clause by @wangrunji0408 in #726
• feat(storage): support NULL on blob and char columns by @unconsolable in #724
• refactor(fetch size): add fast return and compare ROWSET MAX OUTPUT by @eliasyaoyc in #716
• feat(executor_v2): migrate executors to run TPCH by @wangrunji0408 in #727
• feat(planner_v2): pass all sqllogictests by @wangrunji0408 in #728
• chore: enable query engine v2 by default by @wangrunji0408 in #729
• feat(rt): support tokio-console by @wangrun...

v0.1.3: special release for crates.io

RisingLight is finally available as a crate on crates.io! There're a lot of bug fixes and new small features in v0.1.3.

What's Changed

• feat: map filter inputref by @XieJiann in #548
• fix(storage): delete RowSets when drop table by @BaymaxHWY in #555
• chore(storage): use get_or_try_insert_with for block cache by @BaymaxHWY in #556
• feat(array): improve 15x performance by @sundy-li in #554
• chore(storage): remove TODO comment by @BaymaxHWY in #557
• fix(executor): Char and Varchar's length limit not respected by @xiaoyong-z in #558
• chore: remove header in DataChunk. by @RinChanNOWWW in #560
• fix(executor): output no chunks from dummy by @skyzh in #561
• feat(tracing): add minitrace to RisingLight by @WindowsXp-Beta in #550
• test: do not detect tests in build.rs by @xxchan in #564
• binder: plan not optimal for count(*) with filter by @xiaoyong-z in #559
• feat(ci): parallism test by @skyzh in #568
• fix(storage): ignore duplicated deletion by @skyzh in #569
• fix(ci): sqllogictest not running by @skyzh in #570
• chore(storage) improve flush logging by @kwannoel in #571
• docs: update roadmap issue by @skyzh in #573
• chore: use published crates by @xxchan in #575
• release: bump to v0.1.1 by @skyzh in #576
• feat(array) introduce DataChunkBuilder and split chunks in NestedLoop… by @BaymaxHWY in #567
• refactor(executor): refactor ArrayImpl::Bool for cheap cloning by @kwannoel in #577
• refactor(array): cheap clone for all arrays by @skyzh in #579
• feat(function): add function definition by @MingjiHan99 in #581
• chore: bump rust toolchain to 2022-03-09 by @skyzh in #583
• refactor(storage): remove Arc from Arc by @kwannoel in #585
• feat(tracing): make minitrace a cli option by @D2Lark in #586
• chore(tracing): remove enable_tracing option from db interface by @skyzh in #587
• feat(storage): report statistics after finishing table scan (#474) by @adlternative in #594
• chore: bump toolchain and adapt to new portable simd API by @Y7n05h in #595
• feat(executor):optimize NestedLoopJoin's memory usage by @BaymaxHWY in #599
• chore: switch sqlparser-rs to crates.io version by @wangrunji0408 in #602
• chore: bump to v0.1.3 by @skyzh in #603
• chore: update contributors by @skyzh in #605

New Contributors

• @XieJiann made their first contribution in #548
• @BaymaxHWY made their first contribution in #555
• @sundy-li made their first contribution in #554
• @RinChanNOWWW made their first contribution in #560
• @kwannoel made their first contribution in #571
• @D2Lark made their first contribution in #586
• @adlternative made their first contribution in #594
• @Y7n05h made their first contribution in #595

Full Changelog: v0.1...v0.1.3

Thanks all for your contributions!

> select * from pg_catalog.contributors
+----------------+
| github_id      |
+----------------+
| BaymaxHWY      |
| D2Lark         |
| Fedomn         |
| LiuYuHui       |
| MingjiHan99    |
| PsiACE         |
| RinChanNOWWW   |
| Sunt-ing       |
| TennyZhuang    |
| WindowsXp-Beta |
| Y7n05h         |
| adlternative   |
| alissa-tung    |
| arkbriar       |
| kwannoel       |
| likg227        |
| ludics         |
| nanderstabel   |
| pleiadesian    |
| rapiz1         |
| skyzh          |
| st1page        |
| sundy-li       |
| tabVersion     |
| wangrunji0408  |
| xiaoyong-z     |
| xxchan         |
| yingjunwu      |
| zehaowei       |
| zzl200012      |
+----------------+

v0.1: The Initial Release

After about half a year's development, we finally release RisingLight v0.1. RisingLight is an embed database, with an interactive shell built-in, and supports running a few TPC-H queries.

> select * from pg_catalog.contributors
+----------------+
| github_id      |
+----------------+
| Fedomn         |
| LiuYuHui       |
| MingjiHan99    |
| PsiACE         |
| Sunt-ing       |
| TennyZhuang    |
| WindowsXp-Beta |
| alissa-tung    |
| arkbriar       |
| likg227        |
| ludics         |
| nanderstabel   |
| pleiadesian    |
| rapiz1         |
| skyzh          |
| st1page        |
| tabVersion     |
| wangrunji0408  |
| xiaoyong-z     |
| xxchan         |
| yingjunwu      |
| zehaowei       |
| zzl200012      |
+----------------+