duckdb + polars performance improvements (ploomber#725)

Author: edublancas

.github/workflows/scheduled.yaml

@@ -9,6 +9,8 @@ on:
 jobs:
   broken-links:
     runs-on: ubuntu-latest
+    if: ${{ !contains(github.event.pull_request.labels.*.name, 'allow-broken-links') }}
+
     steps:
       - uses: actions/checkout@v2
       - name: Set up Python ${{ matrix.python-version }}

CHANGELOG.md

@@ -8,14 +8,15 @@
 * [Feature] Moved `%sqlrender` feature to `%sqlcmd snippets` (#647)
 * [Feature] Added tables listing stored snippets when `%sqlcmd snippets` is called (#648)
 * [Doc] Modified integrations content to ensure they're all consistent (#523)
-* [Doc] Document --persist-replace in API section (#539)
+* [Doc] Document `--persist-replace` in API section (#539)
 * [Fix] Fixed CI issue by updating `invalid_connection_string_duckdb` in `test_magic.py` (#631)
 * [Fix] Refactored `ResultSet` to lazy loading (#470)
 * [Fix] Removed `WITH` when a snippet does not have a dependency (#657)
 * [Fix] Used display module when generating CTE (#649)
 * [Doc] Re-organized sections. Adds section showing how to share notebooks via Ploomber Cloud
 * [Fix] Adding `--with` back because of issues with sqlglot query parser (#684)
-* [Fix] Improving << parsing logic (#610)
+* [Feature] Better performance when using DuckDB native connection and converting to `pandas.DataFrame` or `polars.DataFrame`
+* [Fix] Improving `<<` parsing logic (#610)
 * [Fix] Migrate user feedback to use display module (#548)
 
 ## 0.7.9 (2023-06-19)

benchmarks/duckdb.ipynb

@@ -0,0 +1,31 @@
+"""
+Sample script to profile the sql magic.
+"""
+from sql.magic import SqlMagic
+from IPython import InteractiveShell
+import duckdb
+from pandas import DataFrame
+import numpy as np
+
+num_rows = 1000_000
+
+df = DataFrame(np.random.randn(num_rows, 20))
+
+magic = SqlMagic(InteractiveShell())
+
+conn = duckdb.connect()
+magic.execute(line="conn --alias duckdb", local_ns={"conn": conn})
+magic.autopandas = True
+magic.displaycon = False
+
+
+# NOTE: you can put the @profile decorator on any internal function to profile it
+# the @profile decorator is injected by the line_profiler package at runtime, to learn
+# more, see: https://github.com/pyutils/line_profiler
+@profile  # noqa
+def run_magic():
+    magic.execute("SELECT * FROM df")
+
+
+if __name__ == "__main__":
+    run_magic()

benchmarks/profiling.py

@@ -39,6 +39,7 @@ parts:
       - file: integrations/questdb
       - file: integrations/oracle
       - file: integrations/trinodb
+      - file: integrations/duckdb-sqlalchemy
 
   - caption: API Reference
     chapters:
@@ -69,6 +70,7 @@ parts:
     - file: tutorials/etl
     - file: tutorials/excel
     - file: tutorials/product-analytics
+    - file: tutorials/duckdb-native-sqlalchemy
 
   - caption: Community
     chapters:

doc/_toc.yml

@@ -149,22 +149,22 @@ conn.execute("CREATE TABLE some_table (name, age)")
 ### Non SQLAlchemy supported engines
 
 When working with engines that are not supported by SQLAlchemy, e.g. `QuestDB`, we won't be able to use `sqlalchemy.create_engine`.
-Instead, we should initiate an engine using the native method and use the `CustomConnection` object.
+Instead, we should initiate an engine using the native method and use the `DBAPIConnection` object.
 
 ```python
 import psycopg as pg
-from sql.connection import CustomConnection
+from sql.connection import DBAPIConnection
 
 engine = pg.connect("dbname='qdb' user='admin' host='127.0.0.1' port='8812' password='quest'")
-conn = CustomConnection(engine)
+conn = DBAPIConnection(engine)
 
 plot.histogram("my_table", "column_name", bins=50, conn=conn)
 ```
 
 For a full example on how to use JupySQL with a non SQLAlchemy supported engine please see [QuestDB](./../integrations/questdb).
 
 ```{note}
-Please be advised that there may be some features/functionalities that won't be fully compatible with JupySQL when using `CustomConnection`.
+Please be advised that there may be some features/functionalities that won't be fully compatible with JupySQL when using `DBAPIConnection`.
 ```
 
 ## Unit testing