Follow along with the video below to see how to install our site as a web app on your home screen.
Note: This feature may not be available in some browsers.
If I enter 5/3 in the Cadence calulator, I get 1 instead of 1.666667.
How do I specify how many digits to display?
aelPushSignifDigits(10) doesn't work
---
Erik
--If I enter 5/3 in the Cadence calulator, I get 1 instead of 1.666667.
How do I specify how many digits to display?
aelPushSignifDigits(10) doesn't work
---
Erik
Erik,
5/3 is an integer division, and so the correct answer is 1. It's not to do with
precision. You need to do either:
5.0/3
5/3.0
5.0/3.0
(having one of the numbers as a float will force the calculation to
be done as a float). Whilst this might seem unusual in a calculator, it is
not unusual in a programming language (the same behaviour would be seen
in C). Since the calculator is in essence a SKILL expression builder, and
SKILL has the same kind of rules as C for algebraic expression handling,
that's why this happens.
aelPushSignifDigits only has an affect on ael expressions (such as instance
parameter expressions), not SKILL. SKILL evaluates to the full double
precision floating point - provided that the expression is a floating
point expression rather than an integer expression. However, it would
normally just show up (in the default print representation) using around
6 or 7 digits - to get SKILL to display return values with greater precision
you can use:
sstatus(fullPrecision t)
Regards,
Andrew.
On 10 Jul 2003 20:09:38 -0700, erikwanta@starband.net (Erik Wanta) wrote:
If I enter 5/3 in the Cadence calulator, I get 1 instead of 1.666667.
How do I specify how many digits to display?
aelPushSignifDigits(10) doesn't work
---
Erik
--I understand what you guys are saying, but I don't like it. I would
expect that the Cadence calculator would act the same as my pocket
calculator. If I want to do integer division, then I could just use
the floor() function.
Thank you for contacting Cadence Customer Support.
The Service Request identification number is 32507128
---
Erik
Andrew Beckett <andrewb@DELETETHISBITcadence.com> wrote in message news:<7qtsgvge41mq4dhu85mseqpeh91hm6uo7q@4ax.com>...
Erik,
5/3 is an integer division, and so the correct answer is 1. It's not to do with
precision. You need to do either:
5.0/3
5/3.0
5.0/3.0
(having one of the numbers as a float will force the calculation to
be done as a float). Whilst this might seem unusual in a calculator, it is
not unusual in a programming language (the same behaviour would be seen
in C). Since the calculator is in essence a SKILL expression builder, and
SKILL has the same kind of rules as C for algebraic expression handling,
that's why this happens.
aelPushSignifDigits only has an affect on ael expressions (such as instance
parameter expressions), not SKILL. SKILL evaluates to the full double
precision floating point - provided that the expression is a floating
point expression rather than an integer expression. However, it would
normally just show up (in the default print representation) using around
6 or 7 digits - to get SKILL to display return values with greater precision
you can use:
sstatus(fullPrecision t)
Regards,
Andrew.
On 10 Jul 2003 20:09:38 -0700, erikwanta@starband.net (Erik Wanta) wrote:
If I enter 5/3 in the Cadence calulator, I get 1 instead of 1.666667.
How do I specify how many digits to display?
aelPushSignifDigits(10) doesn't work
---
Erik