Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow user to override range requirement #342

Closed
llimllib opened this issue Nov 2, 2021 · 7 comments
Closed

Allow user to override range requirement #342

llimllib opened this issue Nov 2, 2021 · 7 comments
Labels
library @duckdb/duckdb-wasm

Comments

@llimllib
Copy link
Contributor

llimllib commented Nov 2, 2021

relates to #157

In development, I use devd, which does not support range requests, and have a small data set. It would be very useful for me to be able to disable the range support requirement while I'm in development; my production server supports them.

The inability to turn this off means if I want to use duckdb-wasm, I have to rewrite my whole development setup.

@ankoh
Copy link
Member

ankoh commented Nov 2, 2021

I understand your point. I was just not quite sure about the negative effects of a default fallback to non-ranged fetches.
The problem is, that duckdb-wasm runs in a web worker.
So if your user expects ranged reads and we would fallback to full reads for whatever reason, we might end up blocking the database until the entire file is downloaded.
This is quite problematic especially for large files.

@ankoh
Copy link
Member

ankoh commented Nov 2, 2021

It is non-trivial to abort downloading such a file once we started. We can terminate the web worker but thats about it (without SharedArrayBuffers).
This will change as soon as DuckDB gets fully async I/O but that's not possible today.

@ankoh
Copy link
Member

ankoh commented Nov 2, 2021

We could add something like allowFullHTTPReads to the DuckDBConfig that defaults to false, for now.

@llimllib
Copy link
Contributor Author

llimllib commented Nov 2, 2021

that would be the ideal solution for me, if it doesn't make sense overall feel free to close this though

@ankoh
Copy link
Member

ankoh commented Nov 3, 2021

@llimllib Please try version 0.1.12-dev26.0 , it automatically falls back to full requests.
We only log a warning to the console for now.

@ankoh
Copy link
Member

ankoh commented Nov 3, 2021

Screenshot 2021-11-03 at 23 50 03

Neat side effect:
The shell can now read files from servers that dont include range in the CORS header Access-Control-Allow-Headers.

@ankoh ankoh added the library @duckdb/duckdb-wasm label Nov 3, 2021
@llimllib
Copy link
Contributor Author

llimllib commented Nov 4, 2021

Can confirm it works, thank you very much!

image

@llimllib llimllib closed this as completed Nov 4, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
library @duckdb/duckdb-wasm
Projects
None yet
Development

No branches or pull requests

2 participants