TOPIC: octal and hex numbers limit?
#491
octal and hex numbers limit? 3 Years, 8 Months ago
Hi, if you enter this in the workspace:

077777777777

and then do F9 (Run), the Eval function returns this error:

Can't convert: 077777777777

Instead, the same value but without a digit, doesn't give problems:

07777777777
=> 1073741823

The same thing with an hex number with more than 8 digits:

0xfffffffff
 
 
Mauro
 
#494
Re:octal and hex numbers limit? 3 Years, 8 Months ago
Hex and octal literals are converted with strtoul (in sunumber.cpp SuNumber::literal). This is limited to 32 bit values.

Other parts of the number handling (e.g. bit operations) use 32 bit integers for processing.

This code could probably be converted to use 64 bit values now that compiler support for this is more standard.
 
 
andrew
 
#496
Re:octal and hex numbers limit? 3 Years, 8 Months ago
If there is a limit I think there are two options:

1) Raise the limit to a bigger value, if it's not too difficult.
2) Not raise the limit but report this info in the User's Manual.

;)
 
 
Mauro
 
#499
Re:octal and hex numbers limit? 3 Years, 8 Months ago
For now, I added this to the documentation.

I would suggest putting suggested changes for the documentation into the Wiki and then I can periodically add them. Don't worry about exact wording - I can edit when I add it.
 
 
andrew