HTTPFS - support writing?

[ozkatz]

At the moment, it appears that duckdb-wasm can write only to an S3 interface with no support for writing to an HTTP url.

Is there a specific limitation to prevent such support in browsers?
Or has this simply not been implemented yet?

[Mause]

The limitation is that the HTTP protocol has no standard way of communicating a file write, unlike the s3 protocol

[ozkatz]

Thanks @Mause
Although, wouldn't HTTP PUT be that standard way?

The HTTP PUT request method creates a new resource or replaces a representation of the target resource with the request payload.

[carlopi]

It seems that also main DuckDB implements PUT requests only on S3, I assume since it's not very common and defaulting to an error (assuming HTTP resources to be read-only basically) was to my knowledge not noticed until now.

I am not against DuckDB-Wasm moving first, probably behind a runtime flag at first while exploring, but I will have to defer to @Mause and @samansmink, they have more context and a deeper grasp of the problem.

[samansmink]

The main reasoning for only supporting writes with S3 and not any HTTP server, is that generally to write using HTTP to some server there's authentication required. For the S3 protocol this authentication is standardized and widely used. Also we currently only support uploading using the S3 multipart upload API which would not work on any http server.

If there are common use cases for more generic HTTPFS writing we could definitely consider them, though!

[ozkatz]

Thanks @carlopi @samansmink! This makes sense.

Regarding authentication - When running in a browser environment, this typically Just Works™️ (as in, the browser will take care of adding any required cookie / Authorization header to outgoing requests, assuming duckdb-wasm is included in an HTTP based application that already required authentication.

I do understand, however, that the browser is not the only deployment target for this library :)

Another alternative is a callback API, essentially allowing to "plug in" custom behaviors on read/write/... (which also reduces the surface area for bugs)