0.13.2

Changes:

• f79a6be Update to TileDB 2.6.3
• 711ff0a Merge pull request #421 from TileDB-Inc/backport-420-to-release-0.13
• 2a04192 Remove error when incomplete query has 0 results; Add more native libraries to spark jar (#420)
• bfa3147 Update Java/Spark versions for 0.13.2

This list of changes was auto generated.

0.13.1

Changes include:

• Upgrade to TileDB 2.6.2 #412

0.13.0

What's Changed

• Update to TileDB 2.6.0 by @Shelnutt2 in #409

Reader Performance Improvements

TileDB 2.6.0 includes additional improvements over TileDB 2.5 that provide great performance improvements when exporting data. Performance can be over 2x faster in many cases!

The options for the legacy reader algorithm (pre-TileDB 2.5) can still be be enabled with these options:

CLI
--tiledb-config sm.query.sparse_unordered_with_dups.reader=legacy

Spark
.option("tiledb.sm.query.sparse_unordered_with_dups.reader", "legacy")

Full Changelog: 0.12.3...0.13.0

0.12.3

Changes:

• 2400785 Update java/spark for version 0.12.3
• bf45228 Fix verbose toggle for apis

This list of changes was auto generated.

0.12.2

Changes include:

• Update to TileDB 2.5.3 #406

0.12.1

Changes:

• 7434314 Remove macOS-10.14 (#404)
• 1f0d8d7 remove macos-10.14 (#403)
• 9b1d5cd Update Java/Spark versions for 0.12.1 (#402)
• 6125297 Enable verbose logging and catch arrow errors in python API

This list of changes was auto generated.

0.12.0

What's Changed

• Update Doc Links in README.md by @nguyenv in #395
• Fix issue with legacy ingestion option by @gspowley in #397
• Spark warning if create time attribute does not exist in VCF headers by @gspowley in #398
• Update to TileDB 2.5.2 by @aaronwolen in #394
• Set TileDB total memory budget by @gspowley in #399
• Update Java/Spark versions for 0.12.0 by @gspowley in #400

Reader Performance Improvements

TileDB 2.5.2 includes refactored readers that provide great performance improvements when exporting data.
The new reader algorithms are enable by default.

The legacy reader algorithm can be enabled with these options:

CLI
--tiledb-config sm.query.sparse_unordered_with_dups.reader=legacy

Spark
.option("tiledb.sm.query.sparse_unordered_with_dups.reader", "legacy")

Full Changelog: 0.11.3...0.12.0

0.11.3

Changes:

• Update system_memory_mb() for macos by @gspowley in #392
• Update spark/java for 0.11.3 by @aaronwolen in #393

Full Changelog: 0.11.2...0.11.3

0.11.2

Changes:

• 98cc576 Update spark/java for 0.11.2
• 1e4df54 Add dataset create option to materialize all attributes in a VCF file (#389)
• be8e47a Throw exception when trying to ingest invalid VCF files (#390)
• f7b18c4 Merge pull request #387 from TileDB-Inc/gspowley/sc-11265/streamline-ingestion
• 78fce1e address comments; add c-api and python bindings
• 1558fb2 Merge pull request #388 from TileDB-Inc/gspowley/tsv-format-prefix
• a9ef5cf add backwards support for s:
• 5b4a056 make test really deterministic
• bf51395 make test deterministic
• 930ae51 log message changes

See More

• 9a562d1 change tsv export format prefix to f:
• 1672551 memory/perf tuning; support legacy options
• bbf7bdf modify ingestion algorithm
• e0eaa89 reset expected record count; increase open file limit
• f55289a skip record count check for v2/v3
• ac3d002 skip ingestion record check when resume is enabled
• c5f2d83 add advanced cli options
• 0c211de check expected record count
• 69114ed handle contig records < output buffer records
• 0ff5e99 limit task size; fix uint size mismatch
• 8c44e28 add more debug logs
• 6c3d35e debug ci test failure
• a03fc32 remove set::locale::global
• 19fc6f8 calculate per contig task size
• d1616e2 use percentage of total system memory
• ffc8627 add record heap overhead
• df8b56d add total memory breakdown
• f60b840 Merge branch 'master' into gspowley/sc-11265/streamline-ingestion
• a067ff9 Add tiledb-py to tiledbvcf-py's docker image
• 9bfaca7 Restore tiledbvcf-py's working directory to data
• 0c16ddb Install llvm7's dev package
• b4dba75 Set locale with lang/lc_all environment variables
• 32037ce Avoid additional packages
• 8ec178e Pin base images to ubuntu 20.04
• 2985e0a Merge pull request #384 from TileDB-Inc/gspowley/sc-11266/add-tsv-id-column
• 184cb5c Merge pull request #366 from TileDB-Inc/sethshelnutt/ch10128/initial-support-for-annotated-vcf-files
• 27d22e4 add missing id column in tsv export
• ceb97ba simplify store options and add total memory budget
• 46809fd Don't install pip packages into home directory
• 7382351 Merge pull request #380 from TileDB-Inc/gspowley/sc-11097/bgz-bed-file
• b386848 fix comment
• af674aa add support for .bgz bed file extension
• 149f614 Support annotated VCF files w/out a sample name
• c65f10a Merge pull request #374 from TileDB-Inc/gspowley/sc-10592/region-overlap
• f1ef5a5 Merge pull request #378 from TileDB-Inc/gspowley/sc-10786/cli-updates
• c339668 Update TileDB to 2.4.2
• 1ab014c update cli test to catch argument naming
• 5dbe0bc fix cli backwards compatibility
• c849785 Merge pull request #377 from TileDB-Inc/gspowley/sc-10786/cli-updates
• c9b7312 remove change to default number of writer threads
• 1bca661 print command options to debug log
• d877bae add version subcommand
• e96ad65 replace old cli tool
• 5332f90 create the smallest super regions
• 5b92700 implement super regions
• 8c79c71 improve help message formatting
• f52a18e new cli using cli11 parser
• 06c46ee find first overlapping region
• a34db71 Merge branch 'master' into gspowley/sc-10592/region-overlap
• 48cdb85 Merge pull request #375 from TileDB-Inc/gspowley/sc-10684/threading-updates
• 8393d71 update std::thread to std::async; add threading try/catch blocks
• de91d4c improve performance
• a2f727d region overlap search optimization

This list of changes was auto generated.

What's Changed

• Update std::thread to std::async and add threading try/catch blocks by @gspowley in #375
• Migrate to the CLI11 argument parsing library by @gspowley in #377
• Update TileDB to 2.4.2 by @aaronwolen in #379
• Fix CLI backwards compatibility by @gspowley in #378
• Update intersecting region search to handle overlapping regions by @gspowley in #374
• Support "bed.bgz" BED file extension by @gspowley in #380
• Update location of pip packages in tiledbvcf-py docker image by @aaronwolen in #381
• Support annotated VCF files w/out a sample name by @Shelnutt2 in #366
• Add missing ID column in TSV export by @gspowley in #384
• Docker image improvements by @aaronwolen in #383
• Change TSV export prefix for FORMAT fields by @gspowley in #388
• Ingestion updates: streamline options, limit memory usage, and improve performance by @gspowley in #387
• Throw exception when trying to ingest invalid VCF files by @gspowley in #390
• Add dataset create option to materialize all attributes in a VCF file by @gspowley in #389
• Update spark/java for 0.11.2 by @aaronwolen in #391

Full Changelog: 0.11.1...0.11.2

0.11.1

Changes include:

• Add bedfile line number handling to new spark partitioner #368
• Add libcurl to final docker image for cli #371
• Update spark/java version for 0.11.1 #372
• Update native release CI Ubuntu image to 20.04 #373