Working thread for "pre-issues" or ideas

[z3z1ma]

Just a dumping ground where you can log thoughts prior to committing an issue. More open ended. Iteration is expected to be fast before a reasonably stable release.

[pnadolny13]

Cross posting what I said in slack in case its helpful:

My initial config wasn't correct when I was testing because I put a string where an object should have been. The discovery flow wasn't re-logging any of the helpful messages that the docker container returned so it was a bit tricky to figure out what I had misconfigued. Once I saw the log messages it was obvious but I had to hack the code a bit to relog them.

It looks like the re-logging logic already exists in https://github.com/z3z1ma/tap-airbyte/blob/e642200c8d186ae7447c70a5a74d381c3dbebe7c/tap_airbyte/tap.py#L145, if that was pulled out to a generic method that could be used by airbyte_catalog then I would have quickly known my bug. We might end up needing to implement our own checking vs using the subprocess.run check argument so we can receive the message, log them, then raise the exception.

Heres a sample of the LOG and TRACE messages:

{"type": "LOG", "log": {"level": "INFO", "message": "initialised stream with format: "}}
{"type": "LOG", "log": {"level": "FATAL", "message": "'str' object has no attribute 'get'\nTraceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 85, in run\n    raw_config = self.source.read_config(parsed_args.config)\n  File \"/airbyte/integration_code/source_s3/source.py\", line 62, in read_config\n    if config.get(\"format\", {}).get(\"delimiter\") == r\"\\t\":\nAttributeError: 'str' object has no attribute 'get'"}}
{"type": "TRACE", "trace": {"type": "ERROR", "emitted_at": 1670878200933.264, "error": {"message": "Something went wrong in the connector. See the logs for more details.", "internal_message": "'str' object has no attribute 'get'", "stack_trace": "Traceback (most recent call last):\n  File \"/airbyte/integration_code/main.py\", line 13, in <module>\n    launch(source, sys.argv[1:])\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 131, in launch\n    for message in source_entrypoint.run(parsed_args):\n  File \"/usr/local/lib/python3.9/site-packages/airbyte_cdk/entrypoint.py\", line 85, in run\n    raw_config = self.source.read_config(parsed_args.config)\n  File \"/airbyte/integration_code/source_s3/source.py\", line 62, in read_config\n    if config.get(\"format\", {}).get(\"delimiter\") == r\"\\t\":\nAttributeError: 'str' object has no attribute 'get'\n", "failure_type": "system_error"}}}

[z3z1ma]

Makes total sense,
done in this commit

[pnadolny13]

@z3z1ma I think you already implemented what I was asking for in #2. I did notice though that the output for source-s3 had duplicate keys in it likely because the format setting has many optional implementations. Even with that the output is way more helpful than before!

connector_config:
  # The name of the stream you would like this source to output. Can contain letters, numbers, or underscores.
  dataset: fixme
  # A regular expression which tells the connector which files to replicate. All files which match this pattern will be replicated. Use | to separate multiple patterns. See <a href="https://facelessuser.github.io/wcmatch/glob/" target="_blank">this page</a> to understand pattern syntax (GLOBSTAR and SPLIT flags are enabled). Use pattern <strong>**</strong> to pick up all files.
  path_pattern: fixme
  # The format of the files you'd like to replicate
  format: 
    # Option 1
    filetype: fixme
    # The character delimiting individual cells in the CSV data. This may only be a 1-character string. For tab-delimited data enter '\t'.
    delimiter: fixme
    # Configures whether a schema for the source should be inferred from the current data or not. If set to false and a custom schema is set, then the manually enforced schema is used. If a schema is not manually set, and this is set to false, then all fields will be read as strings
    infer_datatypes: fixme
    # The character used for quoting CSV values. To disallow quoting, make this field blank.
    quote_char: fixme
    # The character used for escaping special characters. To disallow escaping, leave this field blank.
    escape_char: fixme
    # The character encoding of the CSV data. Leave blank to default to <strong>UTF8</strong>. See <a href="https://docs.python.org/3/library/codecs.html#standard-encodings" target="_blank">list of python encodings</a> for allowable options.
    encoding: fixme
    # Whether two quotes in a quoted CSV value denote a single quote in the data.
    double_quote: fixme
    # Whether newline characters are allowed in CSV values. Turning this on may affect performance. Leave blank to default to False.
    newlines_in_values: fixme
    # Optionally add a valid JSON string here to provide additional options to the csv reader. Mappings must correspond to options <a href="https://arrow.apache.org/docs/python/generated/pyarrow.csv.ConvertOptions.html#pyarrow.csv.ConvertOptions" target="_blank">detailed here</a>. 'column_types' is used internally to handle schema so overriding that would likely cause problems.
    additional_reader_options: fixme
    # Optionally add a valid JSON string here to provide additional <a href="https://arrow.apache.org/docs/python/generated/pyarrow.csv.ReadOptions.html#pyarrow.csv.ReadOptions" target="_blank">Pyarrow ReadOptions</a>. Specify 'column_names' here if your CSV doesn't have header, or if you want to use custom column names. 'block_size' and 'encoding' are already used above, specify them again here will override the values above.
    advanced_options: fixme
    # The chunk size in bytes to process at a time in memory from each file. If your data is particularly wide and failing during schema detection, increasing this should solve it. Beware of raising this too high as you could hit OOM errors.
    block_size: fixme
    # Option 2
    filetype: fixme
    # If you only want to sync a subset of the columns from the file(s), add the columns you want here as a comma-delimited list. Leave it empty to sync all columns.
    columns: fixme
    # Maximum number of records per batch read from the input files. Batches may be smaller if there aren’t enough rows in the file. This option can help avoid out-of-memory errors if your data is particularly wide.
    batch_size: fixme
    # Perform read buffering when deserializing individual column chunks. By default every group column will be loaded fully to memory. This option can help avoid out-of-memory errors if your data is particularly wide.
    buffer_size: fixme
    # Option 3
    filetype: fixme
    # Option 4
    filetype: fixme
    # Whether newline characters are allowed in JSON values. Turning this on may affect performance. Leave blank to default to False.
    newlines_in_values: fixme
    # How JSON fields outside of explicit_schema (if given) are treated. Check <a href="https://arrow.apache.org/docs/python/generated/pyarrow.json.ParseOptions.html" target="_blank">PyArrow documentation</a> for details
    unexpected_field_behavior: fixme
    # The chunk size in bytes to process at a time in memory from each file. If your data is particularly wide and failing during schema detection, increasing this should solve it. Beware of raising this too high as you could hit OOM errors.
    block_size: fixme
  # Optionally provide a schema to enforce, as a valid JSON string. Ensure this is a mapping of <strong>{ "column" : "type" }</strong>, where types are valid <a href="https://json-schema.org/understanding-json-schema/reference/type.html" target="_blank">JSON Schema datatypes</a>. Leave as {} to auto-infer the schema.
  schema: fixme
  # Use this to load files from S3 or S3-compatible services
  provider: 
    # Name of the S3 bucket where the file(s) exist.
    bucket: fixme
    # In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.
    aws_access_key_id: fixme
    # In order to access private Buckets stored on AWS S3, this connector requires credentials with the proper permissions. If accessing publicly available data, this field is not necessary.
    aws_secret_access_key: fixme
    # By providing a path-like prefix (e.g. myFolder/thisTable/) under which all the relevant files sit, we can optimize finding these in S3. This is optional but recommended if your bucket contains many folders/files which you don't need to replicate.
    path_prefix: fixme
    # Endpoint to an S3 compatible service. Leave empty to use AWS.
    endpoint: fixme

If its helpful I spent some time figuring out how to parse json schemas to meltano yamls for automating adding plugin definitions to the hub based on the --about output. The function I ended up with is https://github.com/pnadolny13/hub-utils/blob/1649edae5a0f877b9bff7de694268bf4eb98732b/hub_utils/meltano_util.py#L114. I'm sure its not perfect but it has worked pretty well so far. It chooses to unnest objects using dot notation. I figured this would be something a python package existed for i.e. json schema flattening but I couldnt find anything that worked.