jren
Guest
Tue Sep 06, 2011 4:34 pm
Hi,
I have two questions related to precision of float:
1.
In my SKILL code:
grid1 = 0.5
fprintf(stdout "grid1 = %.20f\n" grid1)
grid2 = 0.4
fprintf(stdout "grid2 = %.20f\n" grid2)
Output in CIW:
grid1 = 0.50000000000000000000
grid2 = 0.40000000000000002220
How can I get 0.4 = 0.4000000000000000 as 0.5?
2.
I tried to use function techSetPrecision(), but I am confused by the
following test:
techSetPrecision(2)
grid3 = 0.5553567
grid3 = grid3 * 1.0
fprintf(stdout "grid3 = %.20f\n" grid3)
In the CIW:
grid3 = 0.55535670000000003643
What I assume is grid3 should become 0.5600000 since I set precision
of number of digits in float as 2. Can anyone help me better
understand function techSetPrecision?
Thanks!
Jren
Jean-Marc Bourguet
Guest
Wed Sep 07, 2011 7:34 am
jren <ddsweet_at_gmail.com> writes:
Quote:
In my SKILL code:
grid1 = 0.5
fprintf(stdout "grid1 = %.20f\n" grid1)
grid2 = 0.4
fprintf(stdout "grid2 = %.20f\n" grid2)
Output in CIW:
grid1 = 0.50000000000000000000
grid2 = 0.40000000000000002220
How can I get 0.4 = 0.4000000000000000 as 0.5?
You can't. Floating point is a pain to work with. Just like you can't
represent 1/3 exactly in decimal with a fixed number of digits, you
can't represent 0.4 exactly in common floating point format as they are
using a binary representation (they are able to represent only a subset
of fractions with a power of 2 as denominator, so 0.5 which is 1/2 can
be represented exactly, but 0.4 must be represented as an approximation)
Yours,
--
Jean-Marc
jren
Guest
Wed Sep 07, 2011 1:06 pm
On Sep 7, 3:34 am, Jean-Marc Bourguet <j...@bourguet.org> wrote:
Quote:
jren <ddsw...@gmail.com> writes:
In my SKILL code:
grid1 = 0.5
fprintf(stdout "grid1 = %.20f\n" grid1)
grid2 = 0.4
fprintf(stdout "grid2 = %.20f\n" grid2)
Output in CIW:
grid1 = 0.50000000000000000000
grid2 = 0.40000000000000002220
How can I get 0.4 = 0.4000000000000000 as 0.5?
You can't. Floating point is a pain to work with. Just like you can't
represent 1/3 exactly in decimal with a fixed number of digits, you
can't represent 0.4 exactly in common floating point format as they are
using a binary representation (they are able to represent only a subset
of fractions with a power of 2 as denominator, so 0.5 which is 1/2 can
be represented exactly, but 0.4 must be represented as an approximation)
Yours,
--
Jean-Marc
Thanks a lot for your explanation.
Regards!
Jren
Andrew Beckett
Guest
Mon Sep 26, 2011 1:37 pm
jren wrote, on 09/07/11 12:06:
Quote:
On Sep 7, 3:34 am, Jean-Marc Bourguet<j...@bourguet.org> wrote:
jren<ddsw...@gmail.com> writes:
In my SKILL code:
grid1 = 0.5
fprintf(stdout "grid1 = %.20f\n" grid1)
grid2 = 0.4
fprintf(stdout "grid2 = %.20f\n" grid2)
Output in CIW:
grid1 = 0.50000000000000000000
grid2 = 0.40000000000000002220
How can I get 0.4 = 0.4000000000000000 as 0.5?
You can't. Floating point is a pain to work with. Just like you can't
represent 1/3 exactly in decimal with a fixed number of digits, you
can't represent 0.4 exactly in common floating point format as they are
using a binary representation (they are able to represent only a subset
of fractions with a power of 2 as denominator, so 0.5 which is 1/2 can
be represented exactly, but 0.4 must be represented as an approximation)
Yours,
--
Jean-Marc
Thanks a lot for your explanation.
Regards!
Jren
Note also that this not just a SKILL issue. You'd have exactly the same problem
with most programming languages (certainly any that use IEEE floating point
representation, including C).
Andrew.
