v2.5.0-beta

This release is larger than usual because it has been a year since last release. It adds new features, bug fixes, and 8 new contributors.

All unit tests pass (98.4% coverage). Fuzz tests passed hundreds of millions of execs. Fuzzing will continue until v2.5.0 is tagged.

Notable Changes to Review Before Upgrading

These new features and bug fixes were cherry-picked to highlight for review (for projects using older version in production).

• PR 370: Add SimpleValue type to more fully support CBOR Simple Values, including values not assigned by IANA and...

• PR 376: Add ByteString type to support CBOR maps with byte string keys because Go doesn't allow []byte as map keys and...

• PR 379: Make Decoder.Decode() return io.ErrUnexpectedEOF instead of io.EOF on EOF if current CBOR data item is incomplete.

• PR 380: Make Unmarshal() and Valid() return cbor.ExtraneousDataError (instead of ignoring extraneous data if any remain).

Notable Changes Affecting Speed or Memory Use

• PR 335: Reuse underlying array if RawMessage has sufficient capacity.

• PR 382: Return buffer to pool in Encode(). It adds a bit of overhead to Encode() but NewEncoder().Encode() is a lot faster and uses less memory.

Other Notable Changes:

• PR 342: Add DecOptions.UTF8 to decode invalid UTF-8. Default is unchanged (reject invalid UTF-8 and return error).

• PR 355 Allow MaxNestedLevels setting to be configured up to 65535.

• PR 352, 377: Add EncOptions.NilContainersMode to encode nil Go maps and slices as either CBOR nil (default) or empty container.

• PR 381: Add Decoder.Skip() to skip CBOR data item in CBOR Sequences (RFC 8742).

What's Changed

Most coding changes are already mentioned. Other changes include CI, comments, and docs.

(click to expand)

All Changes to Code

• Add support for unassigned/reserved CBOR simple values by @fxamacker in #370
• Add ByteString type to support any CBOR byte string by @fxamacker and @agaffney in #376
• Make Decoder.Decode() return io.ErrUnexpectedEOF instead of io.EOF on EOF if CBOR data item is truncated by @fxamacker in #379
• Fix handling of extra data in Unmarshal() & Valid() by @fxamacker in #380
• Add Decoder.Skip() to skip CBOR data item in CBOR Sequences (RFC 8742) by @fxamacker in #381
• Reuse underlying array if RawMessage has sufficient capacity by @zensh in #335
• Return buffer to pool when using Encoder.Encode by @fxamacker in #382
• Add decoding option to allow invalid UTF-8 by @fxamacker in #342
• Allow MaxNestedLevels to be up to 65535 by @immesys in #355
• add option to enforce nil container marshaling as empty containers by @dedefer in #352
• Refactor NilContainersMode option by @fxamacker in #377

Changes to CI, Comments, and Docs

• Remove trailing whitespaces in .golangci.yml by @CodingVoid in #333

• Update ci.yml by @fxamacker in #334

• Update ci.yml by @x448 in #339

• Update ci.yml by @x448 in #340

• Remove default permissions from GitHub Actions workflows by @x448 in #341

• Fix ci for stream mode branch by @fxamacker in #344

• Create dependabot.yml by @x448 in #346

• Bump github/codeql-action from 1 to 2 by @dependabot in #347

• Update pull_request_template.md by @x448 in #350

• Fix typos by @deeglaze in #353

• Mention security assessment by NCC Group by @fxamacker in #358

• godoc.org link in Readme should point to v2 by @jdharms in #361

• Add Go 1.19 to ci.yml test matrix by @fxamacker in #363

• Bump safer-golangci-lint.yml to 1.46.2 by @x448 in #365

• Bump Go to 1.19 in ci-go-cover.yml by @x448 in #367

• Bump safer-golangci-lint.yml to 1.49.0.1 by @x448 in #368

• Mention 1276 repos depend on fxamacker/cbor/v2 by @x448 in #371

• Fix EncMode example in the package comment by @creachadair in #375

New Contributors (alphabetically)

• @agaffney made their first contribution in #376
• @CodingVoid made their first contribution in #333
• @creachadair made their first contribution in #375
• @dedefer made their first contribution in #352
• @deeglaze made their first contribution in #353
• @immesys made their first contribution in #355
• @jdharms made their first contribution in #361
• @zensh made their first contribution in #335

Non-coding contibutions were made by opening notable issues that directly or indirectly improved this release.

• @burdiyan opened issue and followups that led to Encode() returning buffer to pool as default behavior.
• @espoal opened issue and followups that led to adding Skip() feature for CBOR Sequences (RFC 8742).
• @qmuntal opened issue and @x448 added feedback that led to improved support for CBOR Simple Values.
• @x448 opened multiple issues and provided helpful feedback in addition to his merged PR.
• @zensh opened multiple issues and provided helpful feedback in addition to his merged PR.

Full Changelog: v2.4.0...v2.5.0-beta

v2.4.0 (January 3, 2022)

This release adds two user-requested features to the decoder. It passed 3+ billion execs fuzzing before being tagged.

What's Changed

• Add option to specify default Go map type when decoding CBOR map into interface{} by @fxamacker in #316
• Add support for decoding registered CBOR tag to interface type by @fxamacker in #308
• Update CBOR docs for v2.4.0 by @x448 in #318

Special Thanks

• @lukseven for opening issue #303 and proposing a solution
• @Gui-Yom for issue #301

Full Changelog: v2.3.1...v2.4.0

v2.3.1 (Dec 28, 2021)

IMPORTANT:

• This release fixes an important typo in README and omission in CONTRIBUTING.
• No changes to code outside _test.go files.
• Changes to non-test files are limited to comments.
• Next release (v2.4.0) started fuzz testing and is expected to be tagged within 1-2 weeks.

Changes to v2.3.1 include:

• Fix typo in docs (example code snippet) that can cause bugs. Thanks @herrjemand!
• Update CONTRIBUTING to mention signing requirements. Thanks @lukseven and @x448!
• Update README. Thanks @x448 and @rumpelsepp!
• Update ci.yml to use Go 1.17.x. Thanks @x448!
• Add Revive as a lint checker.
• Cleanup lint messages in _test.go files
• Cleanup lint messages in non-test files if the changes are limited to comments (no actual coding changes).

Full Changelog: v2.3.0...v2.3.1

v2.3.0 (May 30, 2021)

Upgrading is recommended: v2.3.0 has bug fixes, is faster, and passed 1+ billion execs fuzzing.

⭐ Features and Improvements

• Add built-in support for big.Int (#209)
• Add support for tag 55799 self-describing CBOR (#227)
• Export valid function (#248)
• Increase user-configurable CBOR limit for MaxArrayElements and MaxMapPairs (#207)
• Add decoding option to be more strict than encoding/json: fail on CBOR map if destination struct field is not found (#178)
• Add option for decoding CBOR pos and neg integer to interface{} to not distinguish between uint and int (#216)

🚀 Performance

fxamacker/cbor 2.3.0 is faster than 2.2.0 by up to 14% (using CWT and COSE example data from RFCs).

name                                 old time/op    new time/op    delta
DecodeCWTClaims-4                      1.34µs ± 0%    1.25µs ± 0%   -6.90%  (p=0.000 n=10+9)
DecodeCOSE/128-Bit_Symmetric_Key-4     1.01µs ± 0%    0.86µs ± 0%  -14.02%  (p=0.000 n=9+9)
DecodeCOSE/256-Bit_Symmetric_Key-4     1.02µs ± 0%    0.88µs ± 0%  -13.60%  (p=0.000 n=9+10)
DecodeCOSE/ECDSA_P256_256-Bit_Key-4    1.69µs ± 0%    1.45µs ± 0%  -14.14%  (p=0.000 n=10+10)
DecodeWebAuthn-4                       1.46µs ± 0%    1.32µs ± 0%   -9.65%  (p=0.000 n=10+10)
EncodeCWTClaims-4                       766ns ± 0%     780ns ± 0%   +1.87%  (p=0.000 n=10+10)
EncodeCOSE/128-Bit_Symmetric_Key-4      910ns ± 0%     908ns ± 0%     ~     (p=0.059 n=9+10)
EncodeCOSE/256-Bit_Symmetric_Key-4      912ns ± 0%     912ns ± 0%     ~     (p=0.909 n=10+10)
EncodeCOSE/ECDSA_P256_256-Bit_Key-4    1.13µs ± 1%    1.14µs ± 0%   +0.61%  (p=0.001 n=9+10)
EncodeWebAuthn-4                        794ns ± 2%     823ns ± 1%   +3.69%  (p=0.000 n=9+10)

fxamacker/cbor 2.3.0 vs ugorji/go 1.2.6

fxamacker/cbor 2.3.0 (not using unsafe) is faster than ugorji/go 1.2.6 (using unsafe).

name                                 old time/op    new time/op    delta
DecodeCWTClaims-4                      2.06µs ± 1%    1.25µs ± 0%  -39.57%  (p=0.000 n=10+9)
DecodeCOSE/128-Bit_Symmetric_Key-4     1.47µs ± 1%    0.86µs ± 0%  -41.25%  (p=0.000 n=9+9)
DecodeCOSE/256-Bit_Symmetric_Key-4     1.50µs ± 2%    0.88µs ± 0%  -41.63%  (p=0.000 n=10+10)
DecodeCOSE/ECDSA_P256_256-Bit_Key-4    2.22µs ± 2%    1.45µs ± 0%  -34.65%  (p=0.000 n=10+10)
DecodeWebAuthn-4                       1.55µs ± 0%    1.32µs ± 0%  -14.97%  (p=0.000 n=9+10)
EncodeCWTClaims-4                      1.46µs ± 0%    0.78µs ± 0%  -46.52%  (p=0.000 n=10+10)
EncodeCOSE/128-Bit_Symmetric_Key-4     1.79µs ± 1%    0.91µs ± 0%  -49.38%  (p=0.000 n=9+10)
EncodeCOSE/256-Bit_Symmetric_Key-4     1.79µs ± 1%    0.91µs ± 0%  -49.15%  (p=0.000 n=10+10)
EncodeCOSE/ECDSA_P256_256-Bit_Key-4    2.09µs ± 1%    1.14µs ± 0%  -45.41%  (p=0.000 n=10+10)
EncodeWebAuthn-4                        981ns ± 0%     823ns ± 1%  -16.05%  (p=0.000 n=10+10)

name                                 old alloc/op   new alloc/op   delta
DecodeCWTClaims-4                        760B ± 0%      176B ± 0%  -76.84%  (p=0.000 n=10+10)
DecodeCOSE/128-Bit_Symmetric_Key-4       800B ± 0%      240B ± 0%  -70.00%  (p=0.000 n=10+10)
DecodeCOSE/256-Bit_Symmetric_Key-4       816B ± 0%      256B ± 0%  -68.63%  (p=0.000 n=10+10)
DecodeCOSE/ECDSA_P256_256-Bit_Key-4      913B ± 0%      352B ± 0%  -61.45%  (p=0.000 n=10+10)
DecodeWebAuthn-4                       1.56kB ± 0%    0.99kB ± 0%  -36.41%  (p=0.000 n=10+10)
EncodeCWTClaims-4                      1.36kB ± 0%    0.18kB ± 0%  -87.06%  (p=0.000 n=10+10)
EncodeCOSE/128-Bit_Symmetric_Key-4     1.97kB ± 0%    0.22kB ± 0%  -88.62%  (p=0.000 n=10+10)
EncodeCOSE/256-Bit_Symmetric_Key-4     1.97kB ± 0%    0.24kB ± 0%  -87.80%  (p=0.000 n=10+10)
EncodeCOSE/ECDSA_P256_256-Bit_Key-4    1.97kB ± 0%    0.32kB ± 0%  -83.74%  (p=0.000 n=10+10)
EncodeWebAuthn-4                       1.31kB ± 0%    1.09kB ± 0%  -17.07%  (p=0.000 n=10+10)

name                                 old allocs/op  new allocs/op  delta
DecodeCWTClaims-4                        6.00 ± 0%      6.00 ± 0%     ~     (all equal)
DecodeCOSE/128-Bit_Symmetric_Key-4       4.00 ± 0%      4.00 ± 0%     ~     (all equal)
DecodeCOSE/256-Bit_Symmetric_Key-4       4.00 ± 0%      4.00 ± 0%     ~     (all equal)
DecodeCOSE/ECDSA_P256_256-Bit_Key-4      7.00 ± 0%      7.00 ± 0%     ~     (all equal)
DecodeWebAuthn-4                         5.00 ± 0%      5.00 ± 0%     ~     (all equal)
EncodeCWTClaims-4                        4.00 ± 0%      2.00 ± 0%  -50.00%  (p=0.000 n=10+10)
EncodeCOSE/128-Bit_Symmetric_Key-4       6.00 ± 0%      2.00 ± 0%  -66.67%  (p=0.000 n=10+10)
EncodeCOSE/256-Bit_Symmetric_Key-4       6.00 ± 0%      2.00 ± 0%  -66.67%  (p=0.000 n=10+10)
EncodeCOSE/ECDSA_P256_256-Bit_Key-4      6.00 ± 0%      2.00 ± 0%  -66.67%  (p=0.000 n=10+10)
EncodeWebAuthn-4                         4.00 ± 0%      2.00 ± 0%  -50.00%  (p=0.000 n=10+10)

Benchmarks used Go 1.15.12 on linux_amd64. Benchmark results are data dependent so run them using your own data.

🐞 Bug Fixes

• Allow decoding to struct field of interface type (#260, #275)
• Decoding registered tag to empty interface should return object of registered type (#223)
• Fix: encoding cbor.RawTag with empty content returns malformed CBOR data (#258)
• Fix: encoding uninitialized cbor.(Raw)Tag returns malformed CBOR data (#256)
• Decoding CBOR null to time.Time should have no effect (#254)
• Fix: decoding CBOR null to cbor.Tag shouldn't return any error (#252)
• Properly handle empty values for custom types (#232)
• Decoding should ignore CBOR tag number 55799 when it is a prefix (#228)
• Validate CBOR tag content type if tag number is 0 or 1, even when TimeTag = DecTagIgnored (#221)
• Registering tag (TagSet.Add) with already registered tag number should be rejected (#218)
• DecOptions.ExtraReturnErrors field should be typed #240

📖 Docs

• Make README more friendly to dark themes (#269)
• Private member struct tag (json:"-" and cbor:"-") are supported but not documented (#201)
• TagOptions struct is missing from README.md (#199)
• Fix go.dev and README compatibility (#173)
• Replace CBOR "draft RFC" with CBOR "RFC 8949" because it was approved by IETF (#265)

🏗️ Chores

• Audit library for any missing attribution for code snippets from sources other than Go's stdlib (#237)
• Audit library for any missing attribution for code snippets from Go's stdlib (encoding/json) (#233)
• Create pull request template for code contributions (#197)
• CI: Use safer-golangci-lint.yml GitHub Action Workflow contributed by @x448
• CI: Bump golangci-lint to 1.40.1
• CI: Use CodeQL analysis

🧪 Tests and Fuzzing

• Code coverage remains above 98%.
• Coverage-guided fuzzing reached 1+ billion execs ~2 days after v2.3.0 release.

👍 Special Thanks

• Special thanks to @kostko, @turbolent, @x448, @Yawning and others for reporting bugs, providing feedback, and more.

⛈️ Winter Storm Uri Displaced Me (still at a hotel 3+ months later)

• As of May 30, I'm still displaced due to Winter Storm Uri started causing damage on February 16. Fire sprinklers started leaking, pipes shattered in the ceilings, and water outage lasted 2+ weeks.

v2.2.0 (Feb 24, 2020)

IMPORTANT: This release fixes a bug that affects all prior versions (1.x and 2.x). Please update if you use Go arrays instead of slices with CBOR indef length arrays. Bug was detected by newer fxamacker/cbor-fuzz.

v2.2 is a small release and is the most reliable version currently available.

Changes include:

• Feat: Support decoding CBOR byte string (major type 2) to Go byte array. (commit 52db071). Thanks @ZenGround0 for requesting this feature.
• Feat: Add more decoding options (MaxNestedLevels, MaxArrayElements, MaxMapKeyPairs, IndefLength, TagsMd.) (commit cb23445)
• Feat: Add more encoding options (IndefLength, TagsMd.) (commit cb23445)
• Fix: Fix potential error when decoding shorter CBOR indef length array to Go array (slice wasn't affected). This bug affects all prior versions of 1.x and 2.x. (commit c7f2cc7)
• Docs: Replace README.md markdown tables with SVG tables to work around go.dev bugs. Less than half were replaced to avoid delaying this release. Thanks @x448 for working on this!

v2.2 passed 473+ million execs of coverage-guided fuzzing on Feb 24, 2020.
UPDATE: v2.2 passed 3.2+ billion execs of coverage-guided fuzzing on Mar 7, 2020.

v2.1.0 (Feb. 17, 2020)

This release focused on three items:

• CBOR tags (major type 6) for encoding and decoding
• Duplicate map key detection options for decoding
• Faster decoding with less memory use (to help offset overhead of new features)

Decoding got faster and memory use dropped more than expected, especially for 'keyasint' structs.

Here's how this library compares to another one using default options and test data from RFC 8392 A.1.

	fxamacker/cbor 2.1	ugorji/go 1.1.7
Encode CWT claims	457 ns/op, 176 B/op, 2 allocs/op	995 ns/op, 1424 B/op, 4 allocs/op
Decode CWT claims	796 ns/op, 176 B/op, 6 allocs/op	1105 ns/op, 568 B/op, 6 allocs/op

Changes include:

• #44 - Add support for CBOR tags (major type 6) with API for built-in and user-defined tags
• #122 - Add decoding options for duplicate map keys
• #147 - Decode "keyasint" structs 26% faster and with 57% fewer allocs (COSE, CWT, etc.)
• #125 - Add encoding option to tag or not tag time values
• #47 - Improve decoding speed (optimizations already identified during v1)
• Basic optimization for CBOR tags feature, more can be done in later releases
• #151 - Implement default encoding for uninitialized time.Time values when tag-required option is specified.
• Update README.md and benchmarks

There will be three ways to create EncMode (and similar API for DecMode):

Function (returns EncMode)	Encoding Mode
EncOptions{...}.EncMode()	immutable options, no tags
🆕 EncOptions{...}.EncModeWithTags(ts)	both options & tags are immutable
🆕 EncOptions{...}.EncModeWithSharedTags(ts)	immutable options & mutable shared tags

To minimize bloat, only tags 0 and 1 (datetime) are going to be part of the default build. The API provides a way for users to register and handle other CBOR tags for user-defined Go types.

In future releases, additional tags may be provided by the library in a modular way (possibly build flags or optional packages).

Special Thanks

@x448 for helping with v2.1 API, docs, and providing the general idea for DupMapKeyEnforcedAPF.
@kostko for providing feedback on the draft v2.1 API for CBOR tags.
@laurencelundblade for providing feedback to @x448 regarding CBOR Null & Undef interop with JSON.
@ZenGround0 for using this library in go-filecoin and requesting a useful feature that'll be in v2.2.

v2.1 fuzzing passed 380+ million execs before release, and passed 2.1+ billion execs on Feb 21, 2020.

v1.5.1 (Feb 9, 2020)

This release is outdated. Upgrading to v2.0.1 or newer is recommended.

Changes include:

• Backport v2.0 fix to v1.5 (sanitize decoded NaN and Infinity time values.) See issue #141.
• Add more unit tests for floating-point data.

Fuzzing passed 500+ million execs on Feb 9, 2020.

v2.0.1 (Feb 6, 2020)

Changes include:

• Fix: API bug introduced in v2.0.0 with (Un)Marshaler and (Un)MarshalerWithMode interface mixup
• Fix: Reject CBOR non-array data for structs decorated with "toarray"

More at #129 and #137.

Special thanks to ZenGround0 for reporting issues.

v2.0.1 passed 300+ million execs fuzzing at time of release.
v2.0.1 passed 2+ billion execs fuzzing on Feb 9, 2020.

v2.0.0 (Feb 2, 2020)

Changes in v2.0:

• Improved API.
• Faster performance.
• Reduced memory usage.
• Improved code quality using around 20 linters.
• Replaced EncOptions.TimeRFC3339 bool with TimeMode (5 settings).
• Decode NaN and Infinity time values to Go's "zero time" value.
• Removed deprecated v1 options and Valid() function.
• Many improvements to README.md.

Why 2.0?

v2 decoupled options from encoding/decoding functions so that:

• new features like CBOR tags can be added without breaking API changes.
• more encoding/decoding function signatures are identical to encoding/json.
• more function signatures can remain stable forever even as CBOR options & tags evolve.

Roadmap

• v2.1 (Feb. 9, 2020) support for CBOR tags (major type 6) and some decoder optimizations.
• v2.2 (Feb. 2020) options for handling duplicate map keys.

CBOR Tags (major type 6) was moved to milestone v2.1. and is over 50% done.

Status

v2.0 passed 1+ billion execs in coverage-guided fuzzing before release. This is the most well-tested and most reliable release of this library. Upgrading is highly recommended.

Improved API

More function signatures match encoding/json. These will never change their parameters or return types:
Marshal, Unmarshal, NewEncoder, NewDecoder, encoder.Encode, decoder.Decode.

"Mode" in this API means definite way of encoding or decoding.

EncMode and DecMode are interfaces created from EncOptions or DecOptions structs.

EncMode and DecMode use immutable options so their behavior won't accidentally change at runtime. Modes are intended to be reused and are safe for concurrent use.

// Create EncOptions, using either struct literal or a function.
opts := cbor.CanonicalEncOptions()

// If needed, modify options -- e.g. how time values should be encoded.
opts.Time = cbor.TimeUnix

// Create reusable EncMode interface with immutable options, safe for concurrent use.
em, err := opts.EncMode()   

// Use EncMode like encoding/json, with same function signatures.
b, err := em.Marshal(v)
// or
encoder := em.NewEncoder(w)
err := encoder.Encode(v)

Package Level Functions

These package level functions use default options without CBOR tags. Their API matches encoding/json. EncMode and DecMode also provide these functions with same signatures.

b, err := cbor.Marshal(v)
err := cbor.Unmarshal(b, &v)
encoder := cbor.NewEncoder(w)
decoder := cbor.NewDecoder(r)

EncOptions.Time

Replaced EncOptions.TimeRFC3339 bool with TimeMode:

• TimeUnix // secs, encodes to CBOR integer using fewest bytes
• TimeUnixMicro // μs, encodes to CBOR float with subsecs in fractional part
• TimeUnixDynamic // secs or μs, encodes to either int or float depending on empty subsecs
• TimeRFC3339 // secs, encodes to string
• TimeRFC3339Nano // ns, encodes to string with trailing zeros removed

v1.5.0 (Jan. 12, 2020)

This release adds encoding options while maintaining backward compatibility.

With EncOptions.ShortestFloat = ShortestFloat16, floating-point values (including subnormals) encode
float64 -> float32 -> float16 when values can round-trip. This is optimized by avoiding the round-trip test except for a subset of subnormal input values and using a faster test for other values.

Conversions for infinity and NaN use InfConvert and NaNConvert settings.

New functions like CanonicalEncOptions(), CTAP2EncOptions(), etc. return predefined EncOptions that can be modified to create custom configurations.

Changes include:

• Use x448/float16 (same team as this library) because all 4+ billion possible conversions are tested.
• Feature: Add funcs that return predefined EncOptions (commit b7b5733, e203b29)
  • CanonicalEncOptions() -- Canonical CBOR (RFC 7049)
  • CTAP2EncOptions() -- CTAP2 Canonical CBOR
  • CoreDetEncOptions() -- Core Deterministic Encoding (draft RFC, subject to change)
  • PreferredUnsortedEncOptions() - Preferred unsorted serialization (draft RFC, subject to change)
• Feature: Add ShortestFloat option for encoding (commit 29e78ee)
  • ShortestFloatNone (default)
  • ShortestFloat16
• Feature: Add NanConvertMode for encoding (commit 16c573c)
  • NaNConvertNone
  • NanConvert7e00
  • NanConvertQuiet
  • NaNConvertPreserveSignal
• Feature: Add InfConvertMode for encoding (commit 16c573c)
  • InfConvertNone
  • InfConvertFloat16
• Refactor and improve quality of code, docs, and tests

UPDATE: Fuzzing passed 4.75+ billion execs with Go 1.12 on Jan. 22, 2020.
Fuzzing passed 2.88+ billion execs with Go 1.13 on Jan. 31, 2020 (slower machine).