METHOD: TestBench How to? Verification over "Generic" parame

[Edward Fisher]

Hello all,

To aid re-use, my VHDL is heavily parameterised using generics. For example bus, bit-widths, counter maximums, reset high/low etc.

During test benching, these can of course be passed to a UUT via generic map (), but they are treated as block instantiation constants evaluated at elaboration time and cannot be changed as part of the test bench's process.

Scenario:
- Lets say we have a 8-bit counter (BIT_WIDTH = 8) which is given a hard maximum value of 200 (COUNT_MAX = 200). Lets also say we have a block reset that is active high (RST_LEVEL = '1'). I may also have a generate statement in my design that allows the synthesis (or not) of some parity bits (INCLUDE_PARITY = TRUE).
- In my test bench, I may have a series of tests that provide some test vectors. For resets etc, these can use the defined value of RST_LEVEL. However, when it comes to the checking the full scope of the code the test vectors are increased by my use of generics. Reset may only be 0 or 1, but it can be logic high or low, i.e. 4 vectors not 2. If parity is included, I need to check these outputs, but they will be undefined or coded to zeros if INCLUDE_PARITY = FALSE. Likewise, as the counter is parameterised, perhaps I want to check that this is correct over a range of bit-widths (8, 16, 32 or say 12 or 14 etc), the count maximum could also change.

Question:
- For cases where the use of generics increases the possibility space for test vectors and automatic tests, I have previously ran the test bench twice, first for (BIT_WIDTH = 8) and then for (BIT_WIDTH = 16). However, just like my man test vectors, I'd prefer to do this in an automated manner.
- From a test methodology perspective, how do you easily test across multiple generics, not only testing over the possible physical input vectors, but also testing over a large variety of generic combinations?

For Example:
- (BIT_WIDTH = 8)(RST_LEVEL = '1') - All test vectors, then
- (BIT_WIDTH = 8)(RST_LEVEL = '0') - All test vectors, then
- (BIT_WIDTH = 16)(RST_LEVEL = '1') - All test vectors, then
- (BIT_WIDTH = 16)(RST_LEVEL = '0') - All test vectors etc.

Many thanks,
Ed

 

[gtwrek]

In article <81df8a4b-828d-4688-8abe-efd008969c2a@googlegroups.com>,
Edward Fisher <emd.fisher@gmail.com> wrote:

Hello all,

To aid re-use, my VHDL is heavily parameterised using generics. For example bus, bit-widths, counter maximums, reset high/low etc.

During test benching, these can of course be passed to a UUT via generic map (), but they are treated as block instantiation constants evaluated at elaboration time and
cannot be changed as part of the test bench's process.

Scenario:
- Lets say we have a 8-bit counter (BIT_WIDTH = 8) which is given a hard maximum value of 200 (COUNT_MAX = 200). Lets also say we have a block reset that is active high
(RST_LEVEL = '1'). I may also have a generate statement in my design that allows the synthesis (or not) of some parity bits (INCLUDE_PARITY = TRUE).
- In my test bench, I may have a series of tests that provide some test vectors. For resets etc, these can use the defined value of RST_LEVEL. However, when it comes to the
checking the full scope of the code the test vectors are increased by my use of generics. Reset may only be 0 or 1, but it can be logic high or low, i.e. 4 vectors not 2. If
parity is included, I need to check these outputs, but they will be undefined or coded to zeros if INCLUDE_PARITY = FALSE. Likewise, as the counter is parameterised, perhaps
I want to check that this is correct over a range of bit-widths (8, 16, 32 or say 12 or 14 etc), the count maximum could also change.

Question:
- For cases where the use of generics increases the possibility space for test vectors and automatic tests, I have previously ran the test bench twice, first for (BIT_WIDTH
= 8) and then for (BIT_WIDTH = 16). However, just like my man test vectors, I'd prefer to do this in an automated manner.
- From a test methodology perspective, how do you easily test across multiple generics, not only testing over the possible physical input vectors, but also testing over a
large variety of generic combinations?

For Example:
- (BIT_WIDTH = 8)(RST_LEVEL = '1') - All test vectors, then
- (BIT_WIDTH = 8)(RST_LEVEL = '0') - All test vectors, then
- (BIT_WIDTH = 16)(RST_LEVEL = '1') - All test vectors, then
- (BIT_WIDTH = 16)(RST_LEVEL = '0') - All test vectors etc.

Edward,

I can offer some advice from a verilog user's perspective.
We parameterize our designs extensively too. Some general advice first -
Dont use parameters where a (perhaps constant) input would work just as
well. In your examples, COUNT_MAX, RST_LEVEL, INCLUDE_PARITY, can
likely be changed to component inputs instead of parameters. Reserve
the use of parameter/generics to case where it's absolutely required.

This has many advantages - both from a test and use-case perspective.
From a test perspective, the advantages should be obvious - as it
addresses the specific concerns your trying to solve. These
"configuration" inputs are not run-time constants - hence your
testbenches can change them, as needed.

From a design reuse perspective, it's an advantage as the component is
well, more reusable. Just because in the one use case that parameter is
a constant doesn't mean that in other use-cases making it a variable
(perhaps with constraints on when it may be changed) isn't needed.
I can't tell you how many times I have something simple - like your
COUNT_MAX example where I originally coded it as a parameter, then
sometime down the road realize I needed it to be semi-variable, and had
to recode it as such.

From a Synthesis QoR point of view - I've done MANY examples of
comparisions between a constant parameter, and a constant input.
Synthesis may take different paths with the different coding styles,
one may take longer than the other to build. But in the end, the QoR
differences are in the noise. The synthesis optimizers are QUITE good
these days - it'a mature technology.

So - minimize your parameter/generic set first.
Then, you're left with a verification corners on the remaining parameter
sets. This is, as you've found really outside the scope of a simulator.
This just means you need to move the task up - to scripts that drive
multiple simulation runs.

When designing your reusable modules encourage your designers to list -
and even check with assertions - the limits and acceptable values for
your parameter sets. Then use these limits to either 1.) Drive the
verification over ALL corners (when achievable) or 2.) Verify
certain directed corners (i.e. your specific use cases) + random parameter
sets.

Regards,

Mark

 

[KKoorndyk]

On Tuesday, May 21, 2019 at 7:13:02 AM UTC-4, Edward Fisher wrote:

Hello all,

To aid re-use, my VHDL is heavily parameterised using generics. For example bus, bit-widths, counter maximums, reset high/low etc.

During test benching, these can of course be passed to a UUT via generic map (), but they are treated as block instantiation constants evaluated at elaboration time and cannot be changed as part of the test bench's process.

Scenario:
- Lets say we have a 8-bit counter (BIT_WIDTH = 8) which is given a hard maximum value of 200 (COUNT_MAX = 200). Lets also say we have a block reset that is active high (RST_LEVEL = '1'). I may also have a generate statement in my design that allows the synthesis (or not) of some parity bits (INCLUDE_PARITY = TRUE).
- In my test bench, I may have a series of tests that provide some test vectors. For resets etc, these can use the defined value of RST_LEVEL. However, when it comes to the checking the full scope of the code the test vectors are increased by my use of generics. Reset may only be 0 or 1, but it can be logic high or low, i.e. 4 vectors not 2. If parity is included, I need to check these outputs, but they will be undefined or coded to zeros if INCLUDE_PARITY = FALSE. Likewise, as the counter is parameterised, perhaps I want to check that this is correct over a range of bit-widths (8, 16, 32 or say 12 or 14 etc), the count maximum could also change.

Question:
- For cases where the use of generics increases the possibility space for test vectors and automatic tests, I have previously ran the test bench twice, first for (BIT_WIDTH = 8) and then for (BIT_WIDTH = 16). However, just like my man test vectors, I'd prefer to do this in an automated manner.
- From a test methodology perspective, how do you easily test across multiple generics, not only testing over the possible physical input vectors, but also testing over a large variety of generic combinations?

For Example:
- (BIT_WIDTH = 8)(RST_LEVEL = '1') - All test vectors, then
- (BIT_WIDTH = 8)(RST_LEVEL = '0') - All test vectors, then
- (BIT_WIDTH = 16)(RST_LEVEL = '1') - All test vectors, then
- (BIT_WIDTH = 16)(RST_LEVEL = '0') - All test vectors etc.

Many thanks,
Ed

I typically use VHDL Configurations to accomplish this. If you have the 3rd Edition of the Ashenden book, check out Chapter 13. Doulos has a decent overview: https://www.doulos.com/knowhow/vhdl_designers_guide/configurations_part_1/

And Sigasi has a good detailed example illustrating how to define the generic port maps: https://insights.sigasi.com/tech/advanced-vhdl-configurations-tying-component-unrelated-entity.html

I have successfully employed this methodology to develop a full suite of test cases covering all (or a specific subset) permutations of generic port values. I end up with a separate VHDL Configuration for each test case, but it makes my testbench architecture generic (meaning a single architecture for all test cases) and it makes running regressions and generating code coverage results quite easy.

It sounds like you might even have be able to employ a single test case that uses those generics, which is exactly what I did for my most recent design.

 

[Thomas Stanka]

Am Dienstag, 21. Mai 2019 13:13:02 UTC+2 schrieb Edward Fisher:

To aid re-use, my VHDL is heavily parameterised using generics. For example bus, bit-widths, counter maximums, reset high/low etc.
[..]
- From a test methodology perspective, how do you easily test across multiple generics, not only testing over the possible physical input vectors, but also testing over a large variety of generic combinations?

For Example:
- (BIT_WIDTH = 8)(RST_LEVEL = '1') - All test vectors, then
- (BIT_WIDTH = 8)(RST_LEVEL = '0') - All test vectors, then
- (BIT_WIDTH = 16)(RST_LEVEL = '1') - All test vectors, then
- (BIT_WIDTH = 16)(RST_LEVEL = '0') - All test vectors etc.

Are you aware that your testbench entity is allowed to have generics so it is possible to pass bitwidth etc to your testbench. Those generics could be set by simulator start (eg for modelsim: vsim -G BIT_WIDTH=8).

You could setup a "super testbench" eg in tcl or python to iterate over the set of generics you like to have tested.

regards,

Thomas