feat: Support writing full record to a single variant column

[pnadolny13]

target-bigquery provides the option to either denormalize into columns or leave as a single json column. That could be a useful feature for this target also.

target-bigquery docs:

Denormalized variants indicate data is unpacked during load with a resultant schema in BigQuery based on the tap schema. Non-denormalized means we have a fixed schema which loads all data into an unstructured JSON column. They are both useful patterns. The latter allowing BigQuery to work with schemaless or rapidly changing sources such as MongoDB instantly, while the former is more performant and convenient to start modeling quickly.

[DouweM]

@pnadolny13 I think this would be a great setting handled by the target SDK for any (database) destinations that support JSON objects!

[tayloramurphy]

@pnadolny13 or @edgarrmondragon do we have a relevant SDK issue for this? Would be good to get it in there.

We do have some interest from a prospect for this.

[edgarrmondragon]

@pnadolny13 or @edgarrmondragon do we have a relevant SDK issue for this? Would be good to get it in there.

We do have some interest from a prospect for this.

@tayloramurphy I don't think there is one. I'm logging it.

[edgarrmondragon]

@tayloramurphy I don't think there is one. I'm logging it.

See meltano/sdk#699 meltano/sdk#1883

[edgarrmondragon]

@tayloramurphy I meant to link meltano/sdk#1883 😅

[tjvananne]

+1 for loading JSON data into variant fields. We have some deeply-nested JSON fields that we prefer to not flatten at load time.

This is how the transferwise variant of target-snowflake works today, but we're actively working on moving away from that variant (it sometimes isn't compatible with non-transferwise taps).