The best VHDL library around for basic testbench checking fu

Kevin,

Lars is talking about the VUnit procedure check_relation and not anything in Bitvis. The reason it looks like it takes a boolean but can show a string message is that the code is pre-processed by a Python script rewriting the call. Such pre-processing is an optional feature that can be enabled in VUnit. As I mentioned in my previous reply to you, since VUnit is not just a VHDL library but a complete tool it could do things like preprocessing and code generation on the fly. Code generation of to_string and check_equal on all records within a package is something that we are thinking about doing to save redundant code. We are still exploring this area.

 

Hi Kevin,

Please see comments below.

tirsdag 30. juni 2015 17.25.11 UTC+2 skrev KJ fřlgende:

On Friday, June 12, 2015 at 9:39:45 AM UTC-4, espen.t...@bitvis.no wrote:
on average 50% of FPGA designers' time is spent on verification, and
almost half of that verification time is spent on debugging. This means:

1. Good reports for unexpected design behaviour is critical.
2. Good progress reporting is also critical.
3. Good basic testbench features are required

Thus we need a library with good functionality for mismatch reporting,
progress reporting and for checks etc. that are needed for every single
testbench; like

Since one can just as easily do all of the above with straight VHDL and be just as concise or even more so, it does not really follow that what is needed is a 'library with good...'.

Anything provided with almost any library can be done with straight vhdl.
The point of a support library is
1. Improve overview, readability, maintainability and structure
2. Reduce amount of writing ONLY if all above is satisfied.

- checking value against expected
- waiting for something to happen - with a timeout
- checking stability of a signal
- waiting for a signal to be stable for a given time (with timeout)

The only free library library (to my knowledge) to provide all this
functionality is Bitvis Utility Library.

OK, but the VHDL language provides this as well.

See above comment.

If this sounds interesting, you should read the below intro.
You can download the library and PPT from
http://bitvis.no/resources/utility-library-download/ without any
registration.

Thanks for providing, that in itself is a useful service.

My read is that the library is way to low level to be an effective archive to capture the original intent of the testbench which means that a testbench written using Bitvis will be just as opaque as the testbenches you complain about now. As an example of checking register function (from slide 31 of the PowerPoint):

write(C_ADDR_ITR, x"AA", "ITR : Set interrupts");
check(C_ADDR_IRR, x"AA", ERROR, "IRR");

Some simple observations:
- The hard coded constants x"AA" are not independent. Changing one requires you to change the second one. But this is not controlled by the code in any way.

Totally agree, but this was not at all the point in this example.
The point is you write
'write(<addr>, <data>, <msg&gt rather than lots of signal wiggling and other vhdl statements.
Almost everybody make BFMs (Bus Functional Modules) these days just for that reason.
The use of VHDL constants rather than literals is of course recommended, but that was not the point here.

> - Similarly, as one works through the rest of the script, there are other hidden dependencies that occur.

Again - this is up to the user. It has nothing to do with using the library..

> - The expected response of the DUT is implicit (the reading back of data from IRR and expecting it to be the same as what was written into ITR). At first glance, one almost might thought it was an error to write to one register and expect some other register to read back that same data. The code you have is actually just an undetectable typo away from being a 'write then read back' test of just ITR (or IRR).

I'm not sure I understand what you mean here, but again the DUT has nothing to do with the library.
The functionality of the ITR register is in fact very useful to system debugging.
There are better ways of doing a write/read test, but again this is not the point here.
The examples here are also used in a course we have, where we later discuss using constants, records, overloads, procedures calling procedures, etc.

- Although this particular test is simply testing the bits somewhat independently, those bits typically have definitions from a record but here you're totally ignoring those definitions and turning on and off bits in a byte with no regard for what that bit defines. While OK for a simple read/write testing as you're showing, there is nothing in Bitvis that would let it scale it up to something more general which is what you would want once you get beyond the simple read/write tests. Consider now this could be written in vanilla VHDL:

-- Let's check operation of the 'This' and 'That' interrupt bits
Fpga_Reg.ITR :> (
This_Interrupt => "1",
That_Interrupt => "1",
Reserved => (others => '0')
);
reg_write(Fpga_Reg.ITR);
Fpga_Reg.IRR := Fpga_Reg.ITR; -- This is the DUT response that is expected

IRR is not supposed to be the same as ITR. More bits may be set in IRR from previous interrupts or ITR writes.

Fpga_Reg_Readback.IRR := reg_read;

assert (Fpga_Reg.IRR = Fpga_Reg_Readback.IRR) report
"IRR register did not read back correctly" & LF &
"Expected: Fpga_Reg.IRR=" & image(Fpga_Reg.IER) & LF &
"Actual: Fpga_Reg_Readback.IRR=" & image(Fpga_Reg_Readback.IRR)
severity ERROR;

While the code is wordier, it is also self-documenting. Using the Bitvis library, one would have to dig into a whole lot more design specific detail and documentation (that is outside the scope of the testbench itself) just to understand what the testbench is trying to accomplish. With what I've shown, that should not be the case. As a bonus, you don't have any of the shortcomings that I pointed out earlier either.

The shortcomings is up to the user. Use good constant names, records, etc.
Nothing to do with BVUL.
The BFM is in fact also not a part of BVUL, but I would definitely recommend it rather than using explicit assert statements.
A BFM is more readable, easier to maintain, etc.
Yes - you do have to understand the functionality of any procedure you use, but without procedures and functions it is difficult to handle complex testbenches. In this case one has to read somewhere that parameter 1 is addr, p2 is data and p3 is message, OR you could use explicit parameter mapping. In fact as long as you use unsigned, std_logic_vector and string respectively for these parameters you can't go wrong (other than compile error).

There is a powerpoint presentation on how to use the library and the BFMs. This can be browsed as a file, or you can even watch a 1 hour webinar. Everything should be more obvious then.

Of course the main issue is that Bitvis, since it is attempting to be generic, cannot be design specific, but in order to get good code clarity you do want the executable code to be design specific. You do want to have executable that looks like 'Control_Reg.Fire_The_Gun := "1"' to then have a response of 'Status_Reg.Off_To_The_Races := "1"', not hard coded hex constants...that then have to change as the 'Off_To_The_Races' bit gets moved from bit 5 to bit 6.

BVUL does not stop you from using records. That is your choice.


> Testbench modelling at the top design level should start by modeling the board that the design will be put into. This then naturally leads to modeling the system that the board goes into as well. Following that approach produces a library of parts that can then be reused in other testbenches because they are modeling actual parts, not just something cobbled together to control/check interface ABC of design XYZ. It will also produce XYZ design specific stuff as well.

BVUL is only one piece in the puzzle. Please see my post above - starting with 'Hi Olof, I final comment from my side'.

I have yet to have a time when 'verbosity control' was something I would want. The testbench will stop on an error and I have the complete log file that I need to debug the problem. What you don't mention at all that is useful is simply to produce multiple log files. For example, logging transaction that occur on a particular interface to a CSV file so that it can be pulled up in Excel. Using those interface log files along with the main transcript log file is a powerful debug aid.

For those who do post processing in other tools that is a good approach.
There is nothing stopping you from using BVUL that way.
For those who want to look at the transcript or a log file directly - verbosity control could be very useful.
Also ID-based verbosity control allows very simple filtering as you have both the scope and the ID available for that.

I don't want to seem to be too harsh, it's not that I think the library is 'bad'. Much of what you have is useful info, but the library while it may improve how some people today write a testbench eventually it will stunt testbench development because it does not go far enough to produce maintainable code. While I accept that you have found users that find it useful, for me it would be a big step backwards since it would produce less maintainable code and probably take just as long, or longer to develop in the first place.

There will always be different opinions and taste, but please spend the time to go through the PPT to see the purpose of some of the features.

Kevin Jennings

 

[Lars Asplund]

Hi Kevin

In addition to what Olof said

Den onsdag 1 juli 2015 kl. 05:31:28 UTC+2 skrev KJ:

On Tuesday, June 30, 2015 at 4:22:16 PM UTC-4, Lars Asplund wrote:
Hi Kevin,

One place where I think verbosity control is useful is when you have a trace
log like the one I showed in the previous comment (runner_trace_logger).. I
don't want to filter what goes to file because I don't know what will be
interesting before I have the problems that caused me to open the file.

Exactly. No filtering. At the point where the sim stops at an assertion, I have everything I need so there is no need to filter anything. I also don't necessarily need to look at everything in the file since I'm debugging a specific problem. I would look at the transcript window or file for basic information, I would look at auxiliary files if necessary but primarily I will be looking at the signals and variables at the point where the sim stopped in order to determine why the assertion condition failed.

During long simulations I may also want to get some progress from the trace
log on stdout but there's no point if I can't reduce that output to a message
pace which I can read. For that I use verbosity control on the log level.

For that I simply grab the scroll bar which effectively pauses the window.. Or, if I have the transcript set to go to an output file rather than the GUI, I simply open the file in a text editor while the sim keeps on running.

I guess how you browse for information is much of a personal preference but what I'm trying to say is that if I can't know in advance what information that will be interesting I do not filter (and save to file). If I do know in advance that I want to see that basic progress information among the thousands of debug message lines I rather set stdout verbosity than doing that browsing.

When it comes to assert statements like the one you mention I think they can
be reduced to (here I'm assuming that IRR and ITR are std_logic).

check_equal(Fpga_Reg.IRR, Fpga_Reg_Readback.IRR, "IRR register did not read
back correctly");


Not quite. Your example, which I was following, looked like ITR and IRR were both software registers. Both of them looked to be eight bits wide which implies to me, that the individual bits would be defined in a record and used the way that I was showing. So the comparison between the ITR and IRR would be between two design specific record types, not just std_logic.

This means that your example here of check_equal wouldn't work without first creating an override of check_equal that works with those specific record types. That's OK, but it means that now when you define new record types, you'll have to also create a 'check_equal' override. Right now, when I create a record type, there will typically be overridden functions of to_std_ulogic_vector, from_std_logic_vector and frequently, but not always image.. Having to add another override for 'check_equal' is more work, so I would have to be convinced of the value to do so first. Actually, of late, what I've found to be more useful is a 'diff_image' function that takes two record type arguments and returns an image only where record elements between the two are different. That way, I'm not eyeballing 10 different fields that are the same to weed out the one or two that are different when the assertion fails and prints the 'diff_image'.

while still being self-documenting. Given that a check for equality is one of
the most common ones this will save you a lot of redundant typing.

I agree that wrapping the assertion into a procedure will typically save typing. On the other hand, many times that typing is only actually done one time within a procedure that may gets called all over the place so the savings on typing isn't really there.

The idea is that check_equal should cover commonly used data types. If you miss something that you feel is "common" you can create an issue on https://github.com/LarsAsplund/vunit/issues.

For other types you can use check_relation which will get some of the work done for you.

Another good thing about standard check and log procedures, at least for us
writing tools, is that they are easier to parse such that you can add code-
related feature not available in VHDL itself. For example, if you enable the
location preprocessor in your VUnit run script
(ui.enable_location_preprocessing()) and the have a log like this

info("Some log message");

the output will be like this (verbose format)

0 ps: INFO in (tb_demo.vhd:27): Some log message

This is useful when finding things and filtering your CSV file in Excel.. A
drawback is that it is the preprocessed file you will see in the simulator
and you my be tempted to edit that one and not the original file.


We may be talking about different things. Whereas the assertion output logging and whatever one puts to the console is one thing, the CSV files I would typically generate are totally separate files that are not nearly so free form as what would go to the console/transcript window.

As an example, there might be a monitor procedure which takes address, data, read/write commands, wait/ack signals all as inputs and whenever a transaction on that bus completes, a new line is written with sim time, address, command, data. No filtering or simply using Excel's built-in filtering has always been enough, no pre-processing needed. The one drawback is that Excel locks the file when it opens so one has to either make a copy of the file if the sim is still running, or you have to stop the sim. Otherwise, the sim quickly stops because it won't be able to write out that new line. But even that isn't all that bad, because it doesn't actually crash Modelsim, it just stops the sim on the 'file_open' but the subsequent writes to the file complete normally (since I don't start the sim then until I have finished looking at the CSV file) so I haven't actually lost anything. It seems to be an odd, but fortuitous feature/bug of Modelsim.

Sorry for being a bit unclear here. What I meant was that having easy-to-parse procedures like check() instead of assert and log() instead of report makes it easier to create preprocessing features like adding location information (file name and line number) to output messages. VHDL can give you *when* something happened with "now" but not *where* it happen which is also very interesting. You can optionally have this for both logs and checks but I gave a log example (info("Some log message") because I think that's where you have the most value. With checks it can add a value if you don't stop at the first error but when you do you'll get the same and more information from the call stack.

For more details see https://github.com/LarsAsplund/vunit/blob/master/examples/vhdl/check/check_example.vhd

Unfortunately, that file does not have the source for 'check_relation', only examples (which are essentially like you've shown here) which would not allow you to separate the two things that are being compared (i.e. This = That) to report individually on 'This' and 'That'. All you can report on is the Boolean that is the result of comparing 'This' with 'That'.

As Olof mentioned the examples in the file are simplified to focus on the functionality of the checks but I wasn't pointing to that. I was pointing at some of the drawbacks of check_relation which are listed on line 215-217. If you want to see how checks are implemented you should start here https://github.com/LarsAsplund/vunit/tree/master/vhdl/check and if you want to see how the preprocessing supporting check_relation works you should go to https://github.com/LarsAsplund/vunit/blob/master/vunit/check_preprocessor.py

Regards,

Lars

 

[KJ]

On Wednesday, July 1, 2015 at 3:16:04 AM UTC-4, espen.t...@bitvis.no wrote:

Since one can just as easily do all of the above with straight VHDL and be
just as concise or even more so, it does not really follow that what is
needed is a 'library with good...'.

Anything provided with almost any library can be done with straight vhdl.
The point of a support library is
1. Improve overview, readability, maintainability and structure
2. Reduce amount of writing ONLY if all above is satisfied.

The examples in the PowerPoint compared with how I showed I would handle that same example do not agree with your statement. What I showed was self-documenting and more maintainable. What I was showing is only 'more writing' if you don't consider the additional documentation that would need to be written to explain the testbench itself.

Some simple observations:
- The hard coded constants x"AA" are not independent. Changing one
requires you to change the second one. But this is not controlled by the
code in any way.

Totally agree, but this was not at all the point in this example.
The point is you write
'write(<addr>, <data>, <msg&gt rather than lots of signal wiggling and other
vhdl statements.
Almost everybody make BFMs (Bus Functional Modules) these days just for that
reason.

What I showed in my example was even simpler, no address was required at all at the top testbench level. What I showed was:
reg_write(Fpga_Reg.ITR);
Fpga_Reg_Readback.IRR := reg_read;
This is simpler than the Bitvis approach. Now obviously an address at some point is required, but that address would be in the lower level helper procedure that takes as input an argument of the specified record type. What Bitvis demonstrated is read/write using std_logic_vectors. While this will be needed, it should not be used at the top level of the testbench as Bitvis was advocating.

> The use of VHDL constants rather than literals is of course recommended, but > that was not the point here.

The point as I got it was how to use the Bitvis library in a testbench. What was advocated was using calls to the Bitvis library at the top level of the testbench. The Bitvis library by itself does not provide the lower level signal twiddling for a particular interface, a user would need to write that. With my approach they would have to do this as well. At the testbench top level, the use of design independent std_logic_vectors as shown by Bitvis provides a less robust, less readable way than my approach which is to provide a simpler interface procedure which is explicitly tied to design specific record definitions.

- Similarly, as one works through the rest of the script, there are other
hidden dependencies that occur.

Again - this is up to the user. It has nothing to do with using the library.

No, but Bitvis provided the example to show the library. What I was demonstrating is the maintainability problems you will run into if you follow this approach. You showed how easy and clean it is to have read/write procedures. I showed a slightly easier and cleaner approach, the main advantage though is that it avoids those maintainability problems.

The Bitvis example does not show the mess that would go into computing all those hard coded constants which is where the mess and hidden dependencies would make that approach much less maintainable then you seem to think. While you can sit back and say the computation of those constants is outside of the scope of the Bitvis library (and I agree it is), by not showing what that code would look like using Bitvis and now not comparing it to how I propose it should look, you're trying to avoid the issue. It's not that I don't think that Bitvis does what it says, it's that what it does doesn't belong at the top level of the testbench and since it doesn't really provide the lowest level of the testbench, it's not clear at what level it really would be useful to me.

- The expected response of the DUT is implicit (the reading back of data
from IRR and expecting it to be the same as what was written into ITR). At
first glance, one almost might thought it was an error to write to one
register and expect some other register to read back that same data. The
code you have is actually just an undetectable typo away from being
a 'write then read back' test of just ITR (or IRR).

I'm not sure I understand what you mean here, but again the DUT has nothing
to do with the library.

The point is that how the DUT is supposed to respond to something (i.e. the write to ITR) is something that should be clear in the testbench. I made that expected response explicit. The Bitvis example did not and actually could be interpreted in a totally different way...because of the way the top level testbench was written.

The functionality of the ITR register is in fact very useful to system
debugging.

Yes it is.

There are better ways of doing a write/read test, but again this is not the
point here.

The Bitvis example was not a write/read test. It was a functionality test. You write to one register and expect a response from a different register.. A write/read test would check the response from the same register.

The examples here are also used in a course we have, where we later discuss
using constants, records, overloads, procedures calling procedures, etc.

OK

-- Let's check operation of the 'This' and 'That' interrupt bits
Fpga_Reg.ITR :> > (
This_Interrupt => "1",
That_Interrupt => "1",
Reserved => (others => '0')
);
reg_write(Fpga_Reg.ITR);
Fpga_Reg.IRR := Fpga_Reg.ITR; -- This is the DUT response that is expected

IRR is not supposed to be the same as ITR. More bits may be set in IRR from
previous interrupts or ITR writes.

But not at that point in the testbench. The 'more bits' part shows up in a totally undocumented way further down. With my approach, again that behavior would have been self-documenting. With the Bitvis example all that is provided is a slew of hard coded constants with no code backing up the computation of those constants.

While the code is wordier, it is also self-documenting. Using the Bitvis
library, one would have to dig into a whole lot more design specific detail
and documentation (that is outside the scope of the testbench itself) just
to understand what the testbench is trying to accomplish. With what I've
shown, that should not be the case. As a bonus, you don't have any of the
shortcomings that I pointed out earlier either.

The shortcomings is up to the user. Use good constant names, records, etc..
Nothing to do with BVUL.

Agreed that the shortcoming has nothing to do with the library itself. But the shortcoming will be inherent in anyone who uses that library. However, if you take a different approach as I showed, you don't have that shortcoming at all.

The BFM is in fact also not a part of BVUL, but I would definitely recommend
it rather than using explicit assert statements.
A BFM is more readable, easier to maintain, etc.

But that is not the point. My example also uses bus functional models (reg_write and reg_read). The assertions are at the higher level. After you do that read, did you get the right result.

Yes - you do have to understand the functionality of any procedure you use,
but without procedures and functions it is difficult to handle complex
testbenches. In this case one has to read somewhere that parameter 1 is addr,
p2 is data and p3 is message, OR you could use explicit parameter mapping..

If you use the Bitvis library that is...however if you use my approach there is only one parameter to the write which is the register that you want to write and there are no parameters to the read. It's also just as easy to have a 'read and compare' procedure which takes only one parameter which is the expected result. That would hide the assertion statement that you think is so objectionable. I didn't take my example that far simply because the Bitvis example didn't either.

In fact as long as you use unsigned, std_logic_vector and string respectively
for these parameters you can't go wrong (other than compile error).

Sure you can, in fact that is exactly where you will go wrong. Here are some examples of errors you will have that are completely avoidable in the first place:
- Reading or writing the wrong address (i.e. passing the value of the address of ITR rather than IRR)
- Changing the value of some hard coded constant in one place and not considering what else has to change as a result. What I showed would take that completely into account. What Bitvis showed was 'well you as the user have to find and fix those things'

There is a powerpoint presentation on how to use the library and the BFMs..
This can be browsed as a file, or you can even watch a 1 hour webinar.
Everything should be more obvious then.

Wow...did you not even notice that was where I got the example from in the first place?

Of course the main issue is that Bitvis, since it is attempting to be
generic, cannot be design specific, but in order to get good code clarity
you do want the executable code to be design specific. You do want to have
executable that looks like 'Control_Reg.Fire_The_Gun := "1"' to then have a
response of 'Status_Reg.Off_To_The_Races := "1"', not hard coded hex
constants...that then have to change as the 'Off_To_The_Races' bit gets
moved from bit 5 to bit 6.

BVUL does not stop you from using records. That is your choice.

What I described are the issues you will have if you choose to use something like the Bitvis approach in the first place.

I have yet to have a time when 'verbosity control' was something I would
want. The testbench will stop on an error and I have the complete log file
that I need to debug the problem. What you don't mention at all that is
useful is simply to produce multiple log files. For example, logging
transaction that occur on a particular interface to a CSV file so that it
can be pulled up in Excel. Using those interface log files along with the
main transcript log file is a powerful debug aid.

For those who do post processing in other tools that is a good approach.
There is nothing stopping you from using BVUL that way.
For those who want to look at the transcript or a log file directly -
verbosity control could be very useful.

I don't use any post processing tools. I use the output files directly and I said, have not found verbosity control to be anything that I would want since I don't know ahead of time what I won't end up needing to look at to debug a problem...which is the whole reason for generating those output files.

There will always be different opinions and taste, but please spend the time
to go through the PPT to see the purpose of some of the features.

I did spend the time to go through PPT, and I've posted what I see as shortcomings. They are not shortcomings of the Bitvis library itself. They are shortcomings for anyone who chooses to use the library. However, the ideas represented (i.e. BFMs, encapsulation, etc.) are good, it's just that trying to bundle that idea into a generic library like Bitvis is not such a good idea.

Kevin Jennings

 

[Lars Asplund]

Hi,

I've now added an example and made some smaller modification to simplify using BVUL check and logging functionality with VUnit. The example testbench (https://github.com/LarsAsplund/vunit/blob/bitvis-utility-library-integration/examples/vhdl/bvul_integration/test/tb_bvul_integration.vhd) has a lot of comments so you can just read it to get the basic idea. If you have any comments you can post them with the BVUL integration issue (https://github.com/LarsAsplund/vunit/issues/54) or comment the code itself (https://github.com/LarsAsplund/vunit/commit/ccd6ec83e3ff95d3518834eb9864298769621f47).

The best way is of course to test it out for real. If you're new to VUnit you can download and run the example like this

git clone https://github.com/LarsAsplund/vunit.git
cd vunit
git checkout bitvis-utility-library-integration
cd examples\vhdl\bvul_integration
python run.py -b <path to BVUL root (Bitvis_Utility_Library_vx_y_z)>

If you want to start out just using your simulator like you're used to you can compile everything with the run script.

python run.py -b <path to BVUL root (Bitvis_Utility_Library_vx_y_z)> --compile

In the bvul_integration directory you'll find vunit_out which contains the compiled output. For example, vunit_out/modelsim is the directory to go to if you're using ModelSim. The testbench is compiled into library lib.

The example is currently on its own branch, I will leave it there for a while to see if there are any comments. After that I'll merge it to the VUnit mainline.

/Lars

 

[Lars Asplund]

Note that you need a Github account to be able to create comments.

/Lars