v0.3.4

[0.3.4] (14 Aug 2023)

This release of Dorado contains a few bug fixes and a hotfix for CUDA out of memory issues encountered during duplex runs with v0.3.3.

• c5c0ea0 - Introduce pipeline API functions to simplify setup for simplex/duplex basecalling.
• 9614eba - Fix potential hang in modbase calling node.
• 67f84a6 - Set the --max-reads default to unlimited for dorado aligner
• f6a0422 - Fix CUDA OOM in duplex by removing tensor caching in decode and updating memory fraction for stereo model.
• 107ebba - Account for filtered duplex reads when tagging simplex parents and calculating duplex yield.

v0.3.3

[0.3.3] (8 Aug 2023)

This release of Dorado introduces improvements to Duplex pair identification for improved duplex yields, faster basecalling on A100 and H100, improvements to modified base calling speed on Apple silicon, and major enhancements to the portability of Dorado binaries.

• 7307146 - Major reduction to required GPU memory, especially for A100/H100. Allows greater batch size and consequently improved basecalling speed
• 8073364 - Improvements to Duplex pairing algorithm for increased Duplex yield. Situations where complement read is truncated are now handled.
• 39ffb35 - Report the duplex rate percentage
• 65b8b8a - Major speed upgrade to modified base calling on Apple silicon devices
• 481438a - Improve performance of basecalling of Fast model on M silicon by 6% by inlining of function used during decode
• fe8dbf2 - Improve basecalling stability on Apple silicon by limiting run time of LSTM kernel to avoid CB submission errors
• 0abea5f - Upgrade dna_r9.4.1_e8 5mCG models to version 0.1 for improved accuracy
• 752e094 - Upgrade to OpenSSL3
• 447b559 - Switch to target-based includes CMake builds - Prevents dependent projects from copying all nclude paths and link libs
• b0d10a9 - Fix edge case where it is possible to spawn no basecaller worker threads
• 1650b83 - Fix issue with inability to find CUPTI library during compilation on some systems
• e4ba3e5 - Add missing SA:Z tag to alignments generated by Dorado
• c5a4cfc - Fix various linker errors
• bf72fdd - Dorado will error out gracefully if no POD5s are provided
• bcdeb8f - Improvements to portability of Dorado binaries via use of static linking on macOS and Linux
• e14a7e6 - Improvements to error handling on Apple Silicon devices (Metal command buffer error handling)
• ca1d191 - Improvements to read ids for non-split reads
• 5d9238a - Revamp AsyncQueue interface
• 6a3ccb6 - Removed RNA003 model which is obsolesced by RNA004
• 1a94fac - Add summary stats and progress bar for basecalling from Fast5
• 072ed96 - Add pipeline restart capability and pairing cache retention option to termination
• 01acbd5 - Stop progress bar cutting out near the end of the run
• 211968c - Add suppression for false-positive vptr issues in older Xcodes
• 1e14d2a - Reduce unnecessary startup cost by setting an upper limit on the maximum batchsize to 10,240
• 09c5b28 - Speed up the fixed cost of auto batchsize detection on mGPU systems by running in parallel

v0.3.2

[0.3.2] (13 Jul 2023)

This release of Dorado introduces basecalling models for the RNA004 chemistry, better identification of duplex read pairs and improved read Q score estimation. It also incorporates various important bug fixes which improve the stability and usability of Dorado.

• 4ce0cb0 - Add RNA004 models
• 24b6c4e - Retry basecalling on CUDA OOM after clearing allocator cache
• 3897ba5 - Add troubleshooting guide to README.md
• 9d55b44 - Fix bug with resume file header parsing
• f9289e2 - Improvements to duplex read splitting algorithm for improved accuracy
• 2869dbc - Solve memory leak during modified base calling
• 4d8ca17 - Fix race condition which was introducing nondeterministic basecalls
• ddb6f71 - Fix aligner regressions from pipeline change
• a57987f - Add R941 v3.3 5mCG 5hmCG models
• ba40e53 - Refactor of basecalling pipeline management strategy
• 26be1a0 - Query enabled Apple silicon cores, not all cores
• 185058e - Replace empty value with "Unknown" in read group tags to satisfy SAM specification
• d953f33 - Add time ordered reads pair cache strategy to PairingNode to support greater variety of ways to run duplex basecalling
• d2700dd - Fix to enable Duplex basecalling on CPU. This will be slow but functionally correct.
• f5ccd0d - Add channel/mux/start_time/start_time_ms to the duplex read
• 0ee5a9b - Fixes to host OOM issues
• 2fe609b - Exclude some non-informative bases at read start from mean qscore calculation
• 3090328 - Fix no output when stdout == stderr and both are the tty
• 7e70de7 - Add support for compute 6.1 (GTX 1080 Ti) plus handling CUDA failures
• 995d0fb - Runner creation refactor
• 9d21036 - Modbase smoke test BLAS fix
• 30658c4 - Add ScopedAutoReleasePool to prevent autorelease leaks on MacOS
• cf502e3 - Update sample rate check to allow some tolerance

v0.3.1

[0.3.1] (26 Jun 2023)

This is a minor release of Dorado. It introduces various bug fixes, as well as performance and usability enhancements. Of particular note, this release introduces the ability to resume simplex basecalling if interrupted, adds RNA002 models, improves the speed of modified base calling and duplex calling, and solves an issue whereby Dorado users were experiencing segmentation faults on version of Linux with older glibc.

• 08218cd - Added --resume-from for simplex basecalling.
• 790a002 - Mitigate simplex scaling performance regression on mGPU systems.
• e0c1beb - Turn off all logging if stdout and stderr point to the same file to avoid curruption
• ae1e5e3 - Support for http proxies in dorado download via dorado_proxy and dorado_proxy_port envvars.
• a62465c - Added warning if user tries to use duplex with fast model
• 99f7483 - Improved error reporting on OSX
• 4f61c18 - Removed source of error related to race-condition
• b1405ab - Added new RNA002 models with V3 architecture
• e75e327 - Aligner throws error when reference path does not exist
• a977916 - Refactor of progress bar and account for filtered reads
• f162705 - Fix the build when compiling with VS2017
• 84ecf32 - Update Koi to v0.3.3
• d6463b3 - Refactor aligner/hts nodes
• c46a2a3 - Fix segfault on Ubuntu 16.04
• 2a96f89 - Improve performance of mod base calling by changing encoding format
• 612ba20 - Node performance stats monitoring
• dc9ea3b - CRFModel: perform clamp on torch tensors in-place. Use torch::InferenceMode
• bf18314 - Solve various bugs in duplex pairing algorithm
• 164ca6e - Add support for running sanitizers as part of CI
• 379fc21 - Add RG tag to duplex pipeline
• 68b5b81 - Add missing dependencies for Mac
• 628722d - Add error checking to pod5 api calls
• c48ab08 - Progress bar fixes for Windows
• ce68b55 - Update metal-cpp so we can make use of NS::SharedPtr<>
• 53aec51 - Update Readme to include roadmap
• 5ea6705 - Adding smoke tests

v0.3.0

[0.3.0] (18 May 2023)

This is a major release of Dorado which introuduces: Duplex pairing and splitting for directly going from POD5 to duplex reads, major performance improvements to simplex and duplex basecalling on A100 GPUs via int8 model quantization and the output of aligned BAM from Dorado and support for producing summary tsv files from BAM.

• ddb7c1e - Improvements to modified basecalling performance
• f879af5 - Add support for CPU basecalling of modified bases
• 282a66c - Add duplex pair alignment accuracy check to reduce risk of incorrect matches
• 3bb0ffc - Add dx:i tag to Dorado output to indicate whether a read is duplex dx:i:1 or simplex dx:i:0
• 78d6bc4 - Improvements to Duplex calling performance
• 20972d4 - Added ability to filter reads by read length, default of 5
• b8ceee4 - Include simplex output in SAM when running duplex basecalling
• 724bafd - Add dorado summary command which produces a summary.tsv file from a SAM/BAM file
• da13d36 - Add splitting of live splitting of concatemer reads into duplex pairs
• 8992e67 - Fix for segfault in older glibc version
• d1377a0 - Add [email protected] simplex model
• 88b547c - Add 5kHz duplex Stereo model and duplex 5kHz support
• 2660956 - Various CPU performance improvements
• f4ea664 - All context modbase tags
• 338911d - Add v4.2 6mA 5mC modbase models
• 1684168 - Improved support for short read duplex basecalling
• 6fe6adb - Add verbose logging option for duplex basecalling.
• a035d7f - Check model sample rate agrees with raw data, add option to skip check
• 26c1112 - Add ability to perform automatic duplex pairing in dorado.
• a824a7d - Output uncompressed BAM when dorado output is a pipe.
• f27d672 - Add CPU to list of devices in help
• 3329bb5 - Fix Fast5 basecalling
• 4d91533 - Improvements to reduce possibility of out of memory issues on CUDA devices via a GPU device mutex
• 14de2e8 - Improvements to progress bar reporting
• 2095fea - Add alignment and BAM generation to duplex and simplex
• fef15ae - Improvements to stereo duplex encoding.
• c6dc187 - Add --reference option to basecalling to allow basecalls to be aligned to a reference.
• 51ca9e7 - Add v4.2.0 5kHz simplex models
• 1a215e7 - Reduce CPU load from mean_q_score_from_qstring
• d3f7320 - Handle empty read-ids file
• 25e2cd1 - Upgrade to Pod5 v0.1.20
• 98eb30d - Add Cutlass LSTM kernels for significant performance improvement on A100 GPUs
• 6aea63a - Reduce CPU load due to trimming
• 7459371 - Increase per-device ScalerNode thread count
• 758d0d9 - Minor improvements to RemoraEncoder::encode_kmer
• b2af21b - Add read filter node to filter reads by Q score, length etc.
• fb60425 - Reduce stereo duplex CPU load
• 0bca7d8 - Reduce torch indexing overhead in modified basecalling
• 4632f05 - Expose k, w comandmline options to dorado aligner
• d560661 - Improved read trimming
• 3cd1c80 - Improve performance of reverse_complement calculation
• 92ef398 - Fix segfault in modified basecalling

v0.2.4

[0.2.4] (12 Apr 2023)

• 92ef398 - Fix out of bound access when modbase calling

v0.2.3

[0.2.3] (06 Apr 2023)

• 6bf227b - Upgrade to Pod5 v0.1.16

v0.2.2

[0.2.2] (04 Apr 2023)

• e1159c4 - Add V4.1 modified base models
• dd389f1 - Add ability to load POD5/FAST5 files recursively, remove the runners flag
• 5a55416 - Fix bug whereby the last read was ignored for read lists not terminated by a newline character
• 1c7988a - Performance improvements to Stereo Basecalling with better batch size selection
• 27c8a2e - Upgrade to Pod5 v0.1.13 which allows loading large files on vmem-limited systems
• 3a9bb7a - Incorporate improvements in Koi which give more predictive QV Scores
• 6896f09 - Fixes to dorado duplex basespace
• 1079b75 - Upgrade to Torch 2.0 and Cuda 11.8
• bcfd64d - If a Stereo model is not available for duplex basecalling, download it automatically
• 8b90643 - Improve Dorado startup time when using POD5 files by getting ReadGroup information from metadata tables
• e86e970 - Use jemalloc to override allocator on Linux
• 4d6a898 - Enable Stereo Duplex on M1
• db097be - Make metal stereo conv1 kernel available
• 17d97d3 - Make MetalModel capable of handling > 1 input channel (Required for Stereo Duplex)
• 6d5f07f - Fix memory leak in getting read groups
• 3e3b21a - Remove deprecated use of FindCUDA and show real location of found toolkit
• ff80a9f - Improvements to Mk1C performance
• 35dcb65 - Fix meaning of TLEN in SAM output
• 7f13113 - Add support for arbitrary messages in Dorado pipelines
• a93ae2c - Addition of a progress bar for basecalling
• a93ae2c - Solve bug which was occuring when a read had no mod base context hits
• 22a3140 - Refactoring of Dorado pipeline system to Add AsyncQueue and use it in ReadSink/Nodes
• 824459e - Add "cuda:auto" as alternative to "cuda:all" when selecting a compute accelerator device on CLI
• d0c9387 - Store reads raw data internally in fp16 format
• 6cd8170 - Switch post-ScalerNode raw_data to float16

v0.2.1

[0.2.1] (16 Feb 2023)

• 121dddf - Fix malformed SAM header

v0.2.0

[0.2.0] (16 Feb 2023)

Dorado v0.2.0 is a major release that introduces several performance and usability improvements. In particular, we are happy to announce the inclusion and release of v4.1.0 simplex models, which improved basecalling in low-complexity genomic regions.

• 9b8ea11 - v4.1.0 with models higher accuracy basecalling in low-complexity genomic regions
• 1311cbe - Increased basecalling performance on macOS
• 74a04fc - The stereo duplex model has been updated with calibrated qscores parameters
• 1170ef6 - SAM output now include read groups
• 63fb334 - linux-arm64 builds
• 117be24 - Added --read-id filter and only basecalling paired reads in duplex for higher basecalling speed
• a8ec895 - Added --min-qscore filter
• c80bae6 - Set builds to default to Release
• e802181 - Modbase CPU fixes
• f0b9654 - Better auto-batchsize detection on macos
• 1a422db - Switch to transparent model URLs
• c2e694e - 260bps v4.0.0 5mCG_5hmCG@v2 models
• 6b9249f - Initial CUDA 12.0 support