Problems with Precision

In Review

Computers encode numbers in binary (base 2) and have a limited number of "bits" to record both integer and floating-point values.

Note: These examples were created for Python 2.X versions. With Python 3.0, all the integer variables are of "infinite" precision and the floats are always double precision. Also, equations such as 1/3 will automatically be converted to floating-point.

1. Integer Numbers

In most languages you have to watch the size of your integer values to make sure they don't go beyond the possible bounds of the value. However, in Python you don't really have to worry about this. Python will default to an "integer" value which can be between about -2 billion to 2 billion. When your numbers go over that amount Python will convert the variable automatically to an integer that can be of virtually any size (until it runs out of memory). Try the following code to see a demonstration. Feel free to change the size of the loop to see just how big the numbers can be!

x=pow(2,30) # start at 2^30th an integer (32 bits)
type(x)
count=0
while (count<3):
 print("x="+format(x)+" Type="+format(type(x)))
 x=x*2 # when x goes from 2^30th to 2^31st the type will change to long (64 bits)
 count+=1

2. Floating Point Numbers

Floating point numbers in Python are represented by 64-bit floating point numbers. These numbers can be from -2^1023 to 2^1023 which are really big numbers so you basically don't have to worry about floating point values either!

x=float(pow(2,1023)) # start with 2^1023 (a really big number)
count=0
while (count<2):
 print("x="+format(x)+" Type="+format(type(x)))
 x=x*2 # the number will go to infinite when at 2^1024 because this is too big for a float
 count+=1

When you define a variable it will automatically become an integer unless it contains a fractional portion, then it will be defined as a float. When the number becomes too large for a 64-bit float, it will convert to infinite or "inf". You can test for infinity using the math packages "isinf(x)" function.

At rare times a floating point value can also contain a value that is "not a number" or "nan". This can happen if you divide 0 by infinity. You can test for "nan" using the math packages "isnan(x)" function.

3. Remember to use floating-point constants

Enter and run the following code:

Value=1/3
print(Value)

Why would 1/3 be 0! The answer is that 1 and 3 are integer values so the fractional portion of the values is missing. Try the following instead:

Value=1.0/3.0
print(Value)

Like most scripting languages, Python is a "variant-type" language. This means that Python will do it's best to create variables that match what you need but at times it will miss the mark. Remember to provide the ".0" after integers that you are going to use as floating point values.

4. Numbers are stored in binary

Computers store numbers in binary (base 2). This means that they can represent integer values precisely but they cannot represent floating point values exactly as they are in decimal. Try the following:

Value=1.0/3.0
print("17 digits: {0:.17}".format(Value))

Notice that the 1/3 value does not extent indefinitely as it should and it ends with a "1". This is because this is the best that a computer can do with the 64-bit floating point values (48-bit for the mantissa or significant digits).

5. Maintaining Precision

The floating point values in Python allow a little over 15 digits of precision. This allows for large numbers but there are times when you manipulate floating point values and then find they are wrong. Try the following:

Value=1.0/3.0
print("17 digits?: {0:.17}".format((Value-0.333333333333333)*1000000000000000))

Notice that you have lost precision by subtracting a value that is close to the current value.

6. Scientific Notation

Python represents large floating point numbers with scientific notation. This means the values can have the form:

 

Try putting the values from the "Python Notation" column into a variable and see what the value looks like stored in "Stack Data". You'll see that the scientific notation is only used when the values are greater than 10,000 in magnitude.

Additional Resources

Python Documentation: Floating Point Arithmetic: Issues and Limitations

Article: Perils of Floating-Point