TOPIC: Bug in the Number builtin class
#58
Bug in the Number builtin class 4 Years, 6 Months ago
Hi, I have discovered a very bothersome bug with Number builtin class that has struggled me for many hours.

Try to execute the following line of code (e.g. in the workspace):

1234567890123.Format("##############")

The result should be 1234567890123, but it returns:

1234567890124 :ohmy:

This bug occurs only with numbers > than 12 digits and for me it's very serious, because in the application that I'm trying to develop I have number fields of more than 12 digits.

There is a workaround?
 
 
Mauro
 
#62
Re:Bug in the Number builtin class 4 Years, 6 Months ago
This looks like an issue with Format. I will look into it.

If you do not need to do any special formatting, you can use String(n) to convert to a string, instead of .Format

You should be able to have 16 digits of precision for integers, but as few as 13 depending on where the decimal place is. Do you need more than this?

Sorry for the problem.

Post edited by: andrew, at: 2006/02/20 17:55
 
 
andrew
 
#67
Re:Bug in the Number builtin class 4 Years, 6 Months ago
The problem is that I often use the NumberControl in the Field_definitions and this control uses the Format method of Number builtin class.

For now, in the NumberControl implementation, I have commented the lines where is used the Format method (but in this way I have disabled the mask functionality of the NumberControl).
 
 
Mauro
 
#69
Re:Bug in the Number builtin class 4 Years, 6 Months ago
It looks like if you specify mask: false that will bypass Format

The problem is in the exe - Format rounds numbers to the precision of the mask. It does this by adding .5 and then taking the integer portion. In this case, adding .5 exceeds the available precision, leading to the problem.

I will fix it to only round when necessary, which should solve this specific problem.

However, this does not solve the underlying problem that there is only a limited precision.

My success with incorporating the Boehm garbage collector has renewed my interest in using other people's modules. One option would be to replace my number stuff with someone else's that supported arbitrary precision numbers. e.g. www2.hursley.ibm.com/decimal/
 
 
andrew
 
#72
Re:Bug in the Number builtin class 4 Years, 6 Months ago
For now, I'm satisfied with 16 digits precision for integer numbers.

I hope that the issue will be fixed soon.

Thank you for your support. ;)
 
 
Mauro
 
#445
Re:Bug in the Number builtin class 3 Years, 9 Months ago
andrew wrote:
[...] My success with incorporating the Boehm garbage collector has renewed my interest in using other people's modules. One option would be to replace my number stuff with someone else's that supported arbitrary precision numbers. e.g. www2.hursley.ibm.com/decimal/
Interesting idea! Currently the precision is depending on a 16 digit window, sliding 4 digits at a time, which can be inconvenient.

Regards,

Karel