checked_add() not catching underflow/overflow when adding a very large number to a very small number

[junjunjd]

Hello and first off, thank you for this crate.

It seems that checked_add() does not catch addition overflow when adding a very large number to a very small number.
For example,

let a = dec!(79_228_162_514_264_337_593_543_950_335);
let b = dec!(0.45);
let c = dec!(0.5);
assert_eq!(a, a.checked_add(b).unwrap());  // not panic
assert_eq!(a, a.checked_add(c).unwrap());  // panic

My guess is that this has something to do with the potential loss of precision when adding a very large number to a very small number. checked_add() will try to round up dec!(0.45) before performing the addition.

Is this expected? Is there a way to prevent this from happening?

[paupino]

Hmm, this is an interesting case. The reason why this is happening is because when adding together the two numbers the resulting value has a bits in underflow. For underflow, we attempt to round the number to "make it fit" - of course, useful for scenarios where the number may be repeating (e.g. 1 / 3). In the case of adding 0.5: it tries to round the underflow up. In doing so, it of course truly overflows. Whereas for the case of adding 0.45 it rounds down the underflow and succeeds since it's rounding toward zero.

I'd need to have a think about this one, as the behavior is explicitly for underflow handling (even though it results in an overflow). Ultimately, it is expected behavior even if it's slightly ambiguous.

That being said, I do agree that it isn't immediately obvious and in some cases you may actually want underflow to be ignored. I'll need to mull on that one and perhaps do some research on how we could handle this ergonomically.

[paupino]

As a side note, something I'd like to see is delayed evaluation - retaining precision until point of evaluation back into a Decimal.

Something like this would be useful for a lot of the mathematical functions, but could also be used for features such as this. i.e. it could return an error on underflow unless an explicit call to round is made before evaluation.

[jkbpvsc]

How does this make the lib any different from just 128bit floating point numbers?
The lib describes itself as

Decimal represents a 128 bit representation of a fixed-precision decimal number.
A Decimal number implementation written in pure Rust suitable for financial calculations that require significant integral and fractional digits with no round-off errors.

Following a rudimentary description of fixed-point-arithmetic (https://en.wikipedia.org/wiki/Fixed-point_arithmetic)

Fixed-point refers to a method of representing fractional (non-integer) numbers by storing a fixed number of digits of their fractional part.

This library clearly has floating point and can result in underflow and a loss of precision. The primary use of fixed-point financial libraries is to avoid a loss of precision within the fixed bounds.

[paupino]

The major difference between a typical IEEE-754 floating point number and this library is that we store numbers as m * 10^-e whereas a floating point number will store the number as m * 2^e. The latter is a lot faster due to base 2 representation but also due to many optimizations that CPUs can make for floating point numbers. The difference, however, is that it can quite difficult to store a precise representation of a number, hence why we talk about floating point numbers storing an approximation of the number and requiring a epsilon for equality comparisons.

By storing the number with a base 10 exponent we effectively are storing the number of decimal places. For example, if the mantissa is 123 and the exponent is 2 then we effectively know that we have 2 decimal places and the number is 1.23. Likewise, because we are storing an exact representation due to only storing where the decimal point should go, it allows us to make precise comparisons instead of approximations. e.g. m = 123, e = 2 is the same as m = 1230, e = 3. This is a bit trickier with IEEE-754 FP numbers due to base 2 being difficult to store "exactly".

Of course, the issue is that we have limited space to be able to store everything. This was a design choice for this library so that we could leverage some optimizations. Due to the space limitations, the library takes some liberties to ensure that everything works as expected - that is, it will try to round underflow where it makes sense.

With the introduction of const generics there is the opportunity to extend this libraries optimizations to allow greater precision with little compromise - which is definitely something I'm looking into. Though, there will always be an underflow component somewhere which needs to be handled - which could turn into overflow as it reaches storage limitations.

I certainly don't intend to mislead here - I don't think we've used the term fixed-point anywhere but if fixed-precision is being misconstrued then I'm more than happy to clarify this in the blurb.

[schungx]

How does this make the lib any different from just 128bit floating point numbers?

Technically speaking, there is no difference. Both are represented by an integral mantissa plus an integral exponent.

For floating-point numbers, both the mantissa and the exponent are base 2.

For this library, the mantissa is base 2 while the exponent is base 10.

And therefore, precision is not really the issue here. This library has no more precision than a 128-bit floating-point. They are equivalent.

The difference is representation. This library can represent all decimal numbers within range (because it is base-10). A floating-point number cannot represent all decimal numbers because the exponent is base-2: there are numbers that cannot be exactly represented, thus requiring mapping to an alternate representation, resulting in "loss of precision". Understand that it is not precision that is lost here... it is a failure of representation that forces an artificial difference.

In a world where people have two fingers and count in base-2 (say, an apple is priced at $100.1 dollars -- $4.5 in decimal), then the IEEE floating-point format is exact and has no more "loss of precision" than this library. In such a world, the happy programmers has no need for rust_decimal... since a rust_binary is equivalent to their IEEE floating-point.