Precision and 'abs'

[Peter W A Wood]

Graham 

I think that it is a documentation error. As far as I know, LiveCode uses binary floating point numbers when performing arithmetic. Not all non-whole decimal numbers cannot be accurate represented in binary floating point numbers. I cannot see how just comparing the absolute value of the difference between two numbers will be any more accurate than comparing the numbers.

This problem isn’t by any means unique to LiveCode. Here is a Windows JavaScript example that shows the same behaviour:

var a = 1.0 + 2.0;
var b = 3.0;
if (a === b) {
  res = 'correct';
 }{
  res = 'incorrect'; 
 }
WScript.Echo('a === b ' + res);
if (Math.abs(a - b) === 0) {
  res = 'correct';
}{
  res = 'incorrect';
}
WScript.Echo('abs (a - b) === 0 ' + res);
 
It displays incorrect for both comparisons. (To run the script save it in a file with the extension .js and double-click on it - for Windows only)

The normal workaround in JavaScript is to perform all calculations using integers and then scaling the numbers down when you want to display them.

Regards

Peter

This is an enhancement request (Bug 9349) for LiveCode to use decimal floating point numbers in place of binary floating point numbers.


> On 4 Feb 2015, at 19:19, Graham Samuel <livfoss at mac.com> wrote:
> 
> I’m having some trouble comparing numbers with lots of decimal places - numbers that ought to be equal are slightly unequal. In the LC Dictionary entry for ‘numberFormat’ there’s a warning about this, and a suggested solution:
> 
>> 
>> Note: Since LiveCode does not use decimal numbers for its internal calculations (for reasons of speed), the decimal representation of a number is sometimes slightly off the correct number. For example, 10^-1 is equal to 0.1, but is calculated (to eighteen decimal places) as 0.100000000000000006. Because of this, setting the numberFormat to specify many decimal places after the decimal point may produce unexpected results in a statement that tests for an exact number. To prevent this, either avoid setting the numberFormat to a value more precise than you need, or use the abs function instead of the = operator to test equality:
>> 
>>  set the numberformat to ".##################"
>>  put 10^-1 = 0.1 -- reports false because of the decimal error
>>  put abs((10^-1) - 0.1) = zero -- reports true
> 
> Well, this sounds good, but looking up the definition of 'abs', there is no suggestion that taking the abs of a number will in any way alter its precision, so what does this advice mean? I have done a test where two numbers differ only after the tenth decimal place, but the test
> 
>  set the numberformat to "0.##########"
>  put abs(aa-bb) = 0
> 
> returns false, so it didn't help.
> 
> Is this just a documentation bug, or is there something funny going on in the LC7.0.2-rc-2?
> 
> Graham
> 
> 
> 
> 
> 
> _______________________________________________
> use-livecode mailing list
> use-livecode at lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode