[DuckDB] Support predicate pushdown via pyarrow dataset

[eddyxu]

Problem Statement

Allow the duckdb extension to pass the filters (predicates) to lance, so it gives lance a chance to optimize I/O exec plan.

SELECT col1, col2 FROM s3://bucket/path/foo.lance WHERE a > 1 AND b < 2

will essentially calls

Dataset.scanner(columns=["col1", "col2"], filter="a > 1 AND b < 2")

[changhiskhan]

We're not limited to the pyarrow interface here right? With the extension we could push down arbitrary predicates

[eddyxu]

Duckdb seems has another set of limitations tho. For example, unlikely it will push functions down.

[wjones127]

DuckDB pushes down only a limited subset of expressions, so it wouldn't be hard to create a simple mapping from PyArrow expressions to Lance SQL just for those.

https://github.com/duckdb/duckdb/blob/239f51293c429168774c3943e96ddf2451253a07/tools/pythonpkg/src/arrow/arrow_array_stream.cpp#L294-L356