Next: Comparing FP Values, Up: Inexactness of computations [Contents][Index]
So, before you start to write any code, you should think about what you really want and what’s really happening. Consider the two numbers in the following example:
x = 0.875 # 1/2 + 1/4 + 1/8 y = 0.425
Unlike the number in y
, the number stored in x
is exactly representable
in binary because it can be written as a finite sum of one or
more fractions whose denominators are all powers of two.
When gawk
reads a floating-point number from
program source, it automatically rounds that number to whatever
precision your machine supports. If you try to print the numeric
content of a variable using an output format string of "%.17g"
,
it may not produce the same number as you assigned to it:
$ gawk 'BEGIN { x = 0.875; y = 0.425 > printf("%0.17g, %0.17g\n", x, y) }' -| 0.875, 0.42499999999999999
Often the error is so small you do not even notice it, and if you do,
you can always specify how much precision you would like in your output.
Usually this is a format string like "%.15g"
, which, when
used in the previous example, produces an output identical to the input.