Feed Sign in with OpenID OpenID

Simon Willison’s Weblog

Fixed Point Arithmetic in Python

The Python Tutorial now includes a new appendix on the limitations of floating point arithmetic. Via Simon Brunning, who also linked to the lengthier What Every Computer Scientist Should Know About Floating-Point Arithmetic almost exactly a year ago.

Posted 28th April 2003 at 8:20 pm

This is Fixed Point Arithmetic in Python by Simon Willison, posted on 28th April 2003.

View blog reactions

Next: Tim Bray on Unicode

Previous: CSS Headings

2 comments

  1. MS SQL Server has a "Money" column type. MS Access has "Currency". I would presume these store values differently than Float columns. How do these (and MySQL's "Decimal"?) circumvent this limited-accuracy problem -- i.e. how do they manage to store $1.50 as "1.5" instead of "1.500000000000001"?

    Joe Grossberg - 29th April 2003 21:00 - #

  2. Ah, never mind: "money and smallmoney datatypes. These two datatypes are accurate to one ten-thousandth of a monetary unit. For display purposes only, they round up to two decimal places." http://www.sybase.com/detail?id=20313

    Joe Grossberg - 29th April 2003 21:06 - #

Comments are closed.

Previously hosted at http://simon.incutio.com/archive/2003/04/28/fixedPoint

A django site