CRC is an FPGA PITA...

  • Thread starter gnuarm.del...@gmail.com
  • Start date
The resources on CRCs are terrible and they explain things poorly. I\'ve had the opposite problem--the last one I did had to compute a CRC with 512 bits of input per cycle. In that case you need to do a matrix operation--the 512-bit input vector is summed with the shifted CRC and multiplied (over GF(2)) by a 512x32 binary matrix. I compute these generator matrices in Matlab: each row of the matrix is a shifted and reduced version of the generator poly. But the textbooks never seem to cover this stuff. The permutations of Endianness and inversions confuse things further.

For 1-bit-input operations, the matrix reduces to a row vector.

The problem with most resources is they don\'t answer some of these questions:

1. How is this doing a modulo operation? (The way they draw the diagrams, it\'s not clear that there are 2 things going on: a shift, which is a multiplication of the poly by x, and a subtraction, which is an XOR. Also, Horner\'s Method is being used, which isn\'t always clear.)
2. How is XOR a subtraction? (In a characteristic-2 field, addition and subtraction are the same.)
3. Why are we doing a modulo? (The point is to make the codeword, which is the message plus the CRC, a polynomial which has zeros (roots) at certain locations.)
4. That doesn\'t look like a polynomial! (It\'s not--it\'s the GF(2) binary coefficients of the polynomial written as a hex word. Even though the field being used might not be GF(2), the coefficients of the poly are in GF(2))..
5. Why not use a regular checksum? (A regular checksum doesn\'t have good minimum Hamming distance qualities. You can have 2 bit errors and still have a good checksum, but the CRC will show the error if the minimum Hamming distance of the CRC code is >=4.)


On Monday, December 14, 2020 at 2:02:23 PM UTC-7, gnuarm.del...@gmail.com wrote:
On Monday, December 14, 2020 at 2:59:02 AM UTC-5, David Brown wrote:
On 13/12/2020 21:06, Anssi Saari wrote:
\"gnuarm.del...@gmail.com\" <gnuarm.del...@gmail.com> writes:

Anyone else have similar problems?

I\'ve only used this one on FPGA designs: https://www.easics.com/crctool/

It generates a VHDL or Verilog implementation. Selectable polynomial
with some common ones predefined, selectable input data width.

BTW, with the standard bit order CRC32, polynomial 0x04C11DB7, the inverted output of an input \"ABCD1234\" (lsb first) would be 16#E928FF7E. Anyone care to confirm that?

Not really. I don\'t have anything I trust handy for that. From some
(software) experimentation years ago when I wanted to generate the same
CRC32 as is used in zip, rar and sfv files (and also Python\'s crc32
function in binascii package) the polynomial used was reversed from
that, 0xEDB88320.

That\'s the joy of CRC. With the same bit size and polynomial, you can
bit reverse the data coming in, byte reverse the incoming data (if it is
handled in larger lumps), bit reverse the generated crc, byte reverse
the generated crc, start with a non-zero shift register, and various
other options - in any combination. The result is equally strong
(though some people argue that starting with a non-zero shift register
is better because the crc of an all-zero input then varies by the length
of the input).

If you are handling the crc in two different ways (two different
implementations), you have to check that it really matches up, and be
prepared to fiddle the options a little until you get it right.

But the actual implementation is simple. In software, you usually do
this a byte at a time, with a lookup table. In an FPGA, a bitwise CRC
is a simple linear feedback register and should be straightforward - I\'d
expect an experienced FPGA designer will find it less effort to
implement it themselves than to use some random web page of questionable
quality. The key trick is to ignore the mathematical background for how
it works - polynomial division rings sound scary, but the implementation
is quite easy. (This means picking your polynomial from the wikipedia
page rather than guessing one yourself.)
Funny that you talk about the number of ways to permute the algorithm and then talk about how easy it is to design in an FPGA. Yeah, we had one error in the code which I found by inspection... eventually. That was only perhaps a quarter of the work though. The vast majority was trying to understand what all the references were doing since they often don\'t actually explain at the bit level detail. Hell, maybe half of them don\'t even allow hex inputs, must less talk about the bit ordering fed into the algorithm. I think this is because they are software people who are only thinking of CRC calculations on files of characters.

Like I said in the first post, there is little on the Internet to support debugging a CRC. I did finally find one site that would allow me to enter data in binary which means I could calculate on 1 bit at a time to check each stage of the computation.

That was the hard part, finding that one web site.
https://leventozturk.com/engineering/crc/

--

Rick C.

+ Get 1,000 miles of free Supercharging
+ Tesla referral code - https://ts.la/richard11209
 
On 15/12/2020 15:44:55, gnuarm.del...@gmail.com wrote:
On Tuesday, December 15, 2020 at 7:23:13 AM UTC-5, Mike Perkins
wrote:
On 12/12/2020 21:36:11, gnuarm.del...@gmail.com wrote:
I had a newbie working on a bit wise CRC32 implementation and he
could not get the results of any of the many online CRC32
generators available. So I coded up the same algorithm and tried
it myself with similar results.

I tried digging around and found little that I could use to
either evaluate an algorithm or to generate comparison data other
than the final result. What I needed was something that could
produce a result for each input bit. Most of the reference
designs are for sequential languages which are byte oriented.
Even if you present them with a single bit they treat it as a
byte and perform the logic at a byte level. Heck, some of the
calculators don\'t even allow hex data input, treating everything
as an ascii character.

Finally I found this link...

https://leventozturk.com/engineering/crc/

Not very easy to use, much of the controls and labeling is
rather cryptic. Eventually by stumbling around I found it I
didn\'t touch anything on the controls other than selecting CRC32
and inputting the data as binary, lsb first I could get an output
that agreed with my result before the final bit inversion used in
CRC32.

Wow! If I knew more about web page design maybe I would build a
page that makes this a bit easier.

The standard example code is C, written to calculate a result one
bit at a time. However it won\'t work on less than a byte of data.
In fact, it isn\'t good for comparing to a true bitwise
implementation because before the loop where they calculate the
result bitwise, the input byte is xor\'ed with the CRC register
all at once mucking the data for comparison purposes.

Anyone else have similar problems?

BTW, with the standard bit order CRC32, polynomial 0x04C11DB7,
the inverted output of an input \"ABCD1234\" (lsb first) would be
16#E928FF7E. Anyone care to confirm that?
I recall spending days trying to emulate the USB and ethernet CRCs.
The Easics website is great at providing the polynomial but that is
only half the story.

What I have done is create all the variations of the starting word,
the bit order and any other permutations in a CRC module all in
parallel. Then streaming a known good packet and seeing which
result was correct and choosing just that output.

I\'ve also breadboarded polynomials in c or c++ but that\'s tricky
in conjuction with the Easic\'s polynomials.

I see what you are referring to, but thinking there is a \"correct\"
result is the problem. There is no one correct result. Even the
online checkers don\'t have a universal one right answer, some of them
can be permuted in various combinations. In one case I had two
agreeing and a third would not provide the same result even when I
thought I had all three configured the same way.

It would appear the reason for the reverse polynomial specification
is that it is desired to generate the CRC using the lsb first which
is easier/simpler to think about in the reversed configuration. This
will result in a bit reversed remainder. Combine that with the
options of starting pattern, inverting the output and treating the
input as hex vs. ascii characters and you have a nearly intractable
number of combinations. Given all that, it\'s not really surprising I
only ever found one web site I was able to coax into providing an
agreeable result.

There is often a \"correct\" result that is accepted by peripherals such
as an ethernet port. However there are some online packet generators
that compute the CRC to compare with and Wireshark can provide the CRC
for a packet if the checksum number crunching is not offloaded in
peripheral hardware.


--
Mike Perkins
Video Solutions Ltd
www.videosolutions.ltd.uk
 
On 15/12/2020 15:44:55, gnuarm.del...@gmail.com wrote:
On Tuesday, December 15, 2020 at 7:23:13 AM UTC-5, Mike Perkins
wrote:
On 12/12/2020 21:36:11, gnuarm.del...@gmail.com wrote:
I had a newbie working on a bit wise CRC32 implementation and he
could not get the results of any of the many online CRC32
generators available. So I coded up the same algorithm and tried
it myself with similar results.

I tried digging around and found little that I could use to
either evaluate an algorithm or to generate comparison data other
than the final result. What I needed was something that could
produce a result for each input bit. Most of the reference
designs are for sequential languages which are byte oriented.
Even if you present them with a single bit they treat it as a
byte and perform the logic at a byte level. Heck, some of the
calculators don\'t even allow hex data input, treating everything
as an ascii character.

Finally I found this link...

https://leventozturk.com/engineering/crc/

Not very easy to use, much of the controls and labeling is
rather cryptic. Eventually by stumbling around I found it I
didn\'t touch anything on the controls other than selecting CRC32
and inputting the data as binary, lsb first I could get an output
that agreed with my result before the final bit inversion used in
CRC32.

Wow! If I knew more about web page design maybe I would build a
page that makes this a bit easier.

The standard example code is C, written to calculate a result one
bit at a time. However it won\'t work on less than a byte of data.
In fact, it isn\'t good for comparing to a true bitwise
implementation because before the loop where they calculate the
result bitwise, the input byte is xor\'ed with the CRC register
all at once mucking the data for comparison purposes.

Anyone else have similar problems?

BTW, with the standard bit order CRC32, polynomial 0x04C11DB7,
the inverted output of an input \"ABCD1234\" (lsb first) would be
16#E928FF7E. Anyone care to confirm that?
I recall spending days trying to emulate the USB and ethernet CRCs.
The Easics website is great at providing the polynomial but that is
only half the story.

What I have done is create all the variations of the starting word,
the bit order and any other permutations in a CRC module all in
parallel. Then streaming a known good packet and seeing which
result was correct and choosing just that output.

I\'ve also breadboarded polynomials in c or c++ but that\'s tricky
in conjuction with the Easic\'s polynomials.

I see what you are referring to, but thinking there is a \"correct\"
result is the problem. There is no one correct result. Even the
online checkers don\'t have a universal one right answer, some of them
can be permuted in various combinations. In one case I had two
agreeing and a third would not provide the same result even when I
thought I had all three configured the same way.

It would appear the reason for the reverse polynomial specification
is that it is desired to generate the CRC using the lsb first which
is easier/simpler to think about in the reversed configuration. This
will result in a bit reversed remainder. Combine that with the
options of starting pattern, inverting the output and treating the
input as hex vs. ascii characters and you have a nearly intractable
number of combinations. Given all that, it\'s not really surprising I
only ever found one web site I was able to coax into providing an
agreeable result.

There is often a \"correct\" result that is accepted by peripherals such
as an ethernet port. However there are some online packet generators
that compute the CRC to compare with and Wireshark can provide the CRC
for a packet if the checksum number crunching is not offloaded in
peripheral hardware.


--
Mike Perkins
Video Solutions Ltd
www.videosolutions.ltd.uk
 
On 15/12/2020 15:44:55, gnuarm.del...@gmail.com wrote:
On Tuesday, December 15, 2020 at 7:23:13 AM UTC-5, Mike Perkins
wrote:
On 12/12/2020 21:36:11, gnuarm.del...@gmail.com wrote:
I had a newbie working on a bit wise CRC32 implementation and he
could not get the results of any of the many online CRC32
generators available. So I coded up the same algorithm and tried
it myself with similar results.

I tried digging around and found little that I could use to
either evaluate an algorithm or to generate comparison data other
than the final result. What I needed was something that could
produce a result for each input bit. Most of the reference
designs are for sequential languages which are byte oriented.
Even if you present them with a single bit they treat it as a
byte and perform the logic at a byte level. Heck, some of the
calculators don\'t even allow hex data input, treating everything
as an ascii character.

Finally I found this link...

https://leventozturk.com/engineering/crc/

Not very easy to use, much of the controls and labeling is
rather cryptic. Eventually by stumbling around I found it I
didn\'t touch anything on the controls other than selecting CRC32
and inputting the data as binary, lsb first I could get an output
that agreed with my result before the final bit inversion used in
CRC32.

Wow! If I knew more about web page design maybe I would build a
page that makes this a bit easier.

The standard example code is C, written to calculate a result one
bit at a time. However it won\'t work on less than a byte of data.
In fact, it isn\'t good for comparing to a true bitwise
implementation because before the loop where they calculate the
result bitwise, the input byte is xor\'ed with the CRC register
all at once mucking the data for comparison purposes.

Anyone else have similar problems?

BTW, with the standard bit order CRC32, polynomial 0x04C11DB7,
the inverted output of an input \"ABCD1234\" (lsb first) would be
16#E928FF7E. Anyone care to confirm that?
I recall spending days trying to emulate the USB and ethernet CRCs.
The Easics website is great at providing the polynomial but that is
only half the story.

What I have done is create all the variations of the starting word,
the bit order and any other permutations in a CRC module all in
parallel. Then streaming a known good packet and seeing which
result was correct and choosing just that output.

I\'ve also breadboarded polynomials in c or c++ but that\'s tricky
in conjuction with the Easic\'s polynomials.

I see what you are referring to, but thinking there is a \"correct\"
result is the problem. There is no one correct result. Even the
online checkers don\'t have a universal one right answer, some of them
can be permuted in various combinations. In one case I had two
agreeing and a third would not provide the same result even when I
thought I had all three configured the same way.

It would appear the reason for the reverse polynomial specification
is that it is desired to generate the CRC using the lsb first which
is easier/simpler to think about in the reversed configuration. This
will result in a bit reversed remainder. Combine that with the
options of starting pattern, inverting the output and treating the
input as hex vs. ascii characters and you have a nearly intractable
number of combinations. Given all that, it\'s not really surprising I
only ever found one web site I was able to coax into providing an
agreeable result.

There is often a \"correct\" result that is accepted by peripherals such
as an ethernet port. However there are some online packet generators
that compute the CRC to compare with and Wireshark can provide the CRC
for a packet if the checksum number crunching is not offloaded in
peripheral hardware.


--
Mike Perkins
Video Solutions Ltd
www.videosolutions.ltd.uk
 
On 15/12/2020 15:44:55, gnuarm.del...@gmail.com wrote:
On Tuesday, December 15, 2020 at 7:23:13 AM UTC-5, Mike Perkins
wrote:
On 12/12/2020 21:36:11, gnuarm.del...@gmail.com wrote:
I had a newbie working on a bit wise CRC32 implementation and he
could not get the results of any of the many online CRC32
generators available. So I coded up the same algorithm and tried
it myself with similar results.

I tried digging around and found little that I could use to
either evaluate an algorithm or to generate comparison data other
than the final result. What I needed was something that could
produce a result for each input bit. Most of the reference
designs are for sequential languages which are byte oriented.
Even if you present them with a single bit they treat it as a
byte and perform the logic at a byte level. Heck, some of the
calculators don\'t even allow hex data input, treating everything
as an ascii character.

Finally I found this link...

https://leventozturk.com/engineering/crc/

Not very easy to use, much of the controls and labeling is
rather cryptic. Eventually by stumbling around I found it I
didn\'t touch anything on the controls other than selecting CRC32
and inputting the data as binary, lsb first I could get an output
that agreed with my result before the final bit inversion used in
CRC32.

Wow! If I knew more about web page design maybe I would build a
page that makes this a bit easier.

The standard example code is C, written to calculate a result one
bit at a time. However it won\'t work on less than a byte of data.
In fact, it isn\'t good for comparing to a true bitwise
implementation because before the loop where they calculate the
result bitwise, the input byte is xor\'ed with the CRC register
all at once mucking the data for comparison purposes.

Anyone else have similar problems?

BTW, with the standard bit order CRC32, polynomial 0x04C11DB7,
the inverted output of an input \"ABCD1234\" (lsb first) would be
16#E928FF7E. Anyone care to confirm that?
I recall spending days trying to emulate the USB and ethernet CRCs.
The Easics website is great at providing the polynomial but that is
only half the story.

What I have done is create all the variations of the starting word,
the bit order and any other permutations in a CRC module all in
parallel. Then streaming a known good packet and seeing which
result was correct and choosing just that output.

I\'ve also breadboarded polynomials in c or c++ but that\'s tricky
in conjuction with the Easic\'s polynomials.

I see what you are referring to, but thinking there is a \"correct\"
result is the problem. There is no one correct result. Even the
online checkers don\'t have a universal one right answer, some of them
can be permuted in various combinations. In one case I had two
agreeing and a third would not provide the same result even when I
thought I had all three configured the same way.

It would appear the reason for the reverse polynomial specification
is that it is desired to generate the CRC using the lsb first which
is easier/simpler to think about in the reversed configuration. This
will result in a bit reversed remainder. Combine that with the
options of starting pattern, inverting the output and treating the
input as hex vs. ascii characters and you have a nearly intractable
number of combinations. Given all that, it\'s not really surprising I
only ever found one web site I was able to coax into providing an
agreeable result.

There is often a \"correct\" result that is accepted by peripherals such
as an ethernet port. However there are some online packet generators
that compute the CRC to compare with and Wireshark can provide the CRC
for a packet if the checksum number crunching is not offloaded in
peripheral hardware.


--
Mike Perkins
Video Solutions Ltd
www.videosolutions.ltd.uk
 
On 15/12/2020 15:44:55, gnuarm.del...@gmail.com wrote:
On Tuesday, December 15, 2020 at 7:23:13 AM UTC-5, Mike Perkins
wrote:
On 12/12/2020 21:36:11, gnuarm.del...@gmail.com wrote:
I had a newbie working on a bit wise CRC32 implementation and he
could not get the results of any of the many online CRC32
generators available. So I coded up the same algorithm and tried
it myself with similar results.

I tried digging around and found little that I could use to
either evaluate an algorithm or to generate comparison data other
than the final result. What I needed was something that could
produce a result for each input bit. Most of the reference
designs are for sequential languages which are byte oriented.
Even if you present them with a single bit they treat it as a
byte and perform the logic at a byte level. Heck, some of the
calculators don\'t even allow hex data input, treating everything
as an ascii character.

Finally I found this link...

https://leventozturk.com/engineering/crc/

Not very easy to use, much of the controls and labeling is
rather cryptic. Eventually by stumbling around I found it I
didn\'t touch anything on the controls other than selecting CRC32
and inputting the data as binary, lsb first I could get an output
that agreed with my result before the final bit inversion used in
CRC32.

Wow! If I knew more about web page design maybe I would build a
page that makes this a bit easier.

The standard example code is C, written to calculate a result one
bit at a time. However it won\'t work on less than a byte of data.
In fact, it isn\'t good for comparing to a true bitwise
implementation because before the loop where they calculate the
result bitwise, the input byte is xor\'ed with the CRC register
all at once mucking the data for comparison purposes.

Anyone else have similar problems?

BTW, with the standard bit order CRC32, polynomial 0x04C11DB7,
the inverted output of an input \"ABCD1234\" (lsb first) would be
16#E928FF7E. Anyone care to confirm that?
I recall spending days trying to emulate the USB and ethernet CRCs.
The Easics website is great at providing the polynomial but that is
only half the story.

What I have done is create all the variations of the starting word,
the bit order and any other permutations in a CRC module all in
parallel. Then streaming a known good packet and seeing which
result was correct and choosing just that output.

I\'ve also breadboarded polynomials in c or c++ but that\'s tricky
in conjuction with the Easic\'s polynomials.

I see what you are referring to, but thinking there is a \"correct\"
result is the problem. There is no one correct result. Even the
online checkers don\'t have a universal one right answer, some of them
can be permuted in various combinations. In one case I had two
agreeing and a third would not provide the same result even when I
thought I had all three configured the same way.

It would appear the reason for the reverse polynomial specification
is that it is desired to generate the CRC using the lsb first which
is easier/simpler to think about in the reversed configuration. This
will result in a bit reversed remainder. Combine that with the
options of starting pattern, inverting the output and treating the
input as hex vs. ascii characters and you have a nearly intractable
number of combinations. Given all that, it\'s not really surprising I
only ever found one web site I was able to coax into providing an
agreeable result.

There is often a \"correct\" result that is accepted by peripherals such
as an ethernet port. However there are some online packet generators
that compute the CRC to compare with and Wireshark can provide the CRC
for a packet if the checksum number crunching is not offloaded in
peripheral hardware.


--
Mike Perkins
Video Solutions Ltd
www.videosolutions.ltd.uk
 

Welcome to EDABoard.com

Sponsor

Back
Top