On Saturday, April 22, 2023 at 1:55:01 PM UTC-4, David Brown wrote:
> On 22/04/2023 18:56, Rick C wrote:
> > On Saturday, April 22, 2023 at 11:13:32 AM UTC-4, David Brown wrote:
> >> On 22/04/2023 05:14, Rick C wrote:
> >>> On Friday, April 21, 2023 at 11:02:28 AM UTC-4, David Brown wrote:
> >>>> On 21/04/2023 14:12, Rick C wrote:
> >>>>>
> >>>>> This is simply to be able to say this version is unique,
> >>>>> regardless of what the version number says. Version numbers are
> >>>>> set manually and not always done correctly. I'm looking for
> >>>>> something as a backup so that if the checksums are different, I
> >>>>> can be sure the versions are not the same.
> >>>>>
> >>>>> The less work involved, the better.
> >>>>>
> >>>> Run a simple 32-bit crc over the image. The result is a hash of
> >>>> the image. Any change in the image will show up as a change in the
> >>>> crc.
> >>>
> >>> No one is trying to detect changes in the image. I'm trying to label
> >>> the image in a way that can be read in operation. I'm using the
> >>> checksum simply because that is easy to generate. I've had problems
> >>> with version numbering in the past. It will be used, but I want it
> >>> supplemented with a number that will change every time the design
> >>> changes, at least with a high probability, such as 1 in 64k.
> >>>
> >> Again - use a CRC. It will give you what you want.
> >
> > Again - as will a simple addition checksum.
> A simple addition checksum might be okay much of the time, but it
> doesn't have the resolving power of a CRC. If the source code changes
> "a = 1; b = 2;" to "a = 2; b = 1;", the addition checksum is likely to
> be exactly the same despite the change in the source. In general, you
> will have much higher chance of collisions, though I think it would be
> very hard to quantify that.
>
> Maybe it will be good enough for you. Simple checksums were popular
> once, and can still make sense if you are very short on program space.
> But there are good reasons why they fell out of favour in many uses.
> >
> >
> >> You might want to go for 32-bit CRC rather than a 16-bit CRC, depending
> >> on the kind of program, how often you build it, and what consequences a
> >> hash collision could have. With a 16-bit CRC, you have a 5% chance of a
> >> collision after 82 builds. If collisions only matter for releases, and
> >> you only release a couple of updates, fine - but if they matter during
> >> development builds, you are getting a more significant risk. Since a
> >> 32-bit CRC is quick and easy, it's worth using.
> >
> > Or, I might want to go with a simple checksum.
> >
> > Thanks for your comments.
> >
> It's your choice (obviously). I only point out the weaknesses in case
> anyone else is listening in to the thread.
>
> If you like, I can post code for a 32-bit CRC. It's a table, and a few
> lines of C code.

You know nothing of the project I am working on or those that I typically work on. But thanks for the advice.

--

Rick C.

+-- Get 1,000 miles of free Supercharging
+-- Tesla referral code - https://ts.la/richard11209

Den 2023-04-20 kl. 04:06, skrev Rick C:
> This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
>
The proper way to do this is to have a directive in the linker.
This reserves space for the CRC and defines the area where the CRC is
calculated.
I am not aware of any linker which support this.

Two months ago, I added the DIGEST directive to binutils aka the GNU
linker. It was committed, but then people realized that I had not signed
an agreement with Free Software Foundation.
Since part of the code I pushed was from a third party which released
their code under MIT, the licensing has not been resolved yet
but the patch is in binutils git, but reverted.

You would write (IIRC):
DIGEST "CRC64-ECMA", (from, to)
and the linker would reserve 8 bytes which is filled with the CRC in the
final link stage.

/Ulf

> I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.
>
> I keep thinking of using a placeholder, but that doesn't seem to work out in any useful way. Even if you try to anticipate the impact of adding the checksum, that only gives you a different checksum, that you then need to anticipate further... ad infinitum.
>
> I'm not thinking of any special checksum generator that excludes the checksum data. That would be too messy.
>
> I keep thinking there is a different way of looking at this to achieve the result I want...
>
> Maybe I can prove it is impossible. Assume the file checksums to X when the checksum data is zero. The goal would then be to include the checksum data value Y in the file, that would change X to Y. Given the properties of the module N checksum, this would appear to be impossible for the general case, unless... Add another data value, called, checksum normalizer. This data value checksums with the original checksum to give the result zero. Then, when the checksum is also added, the resulting checksum is, in fact, the checksum. Another way of looking at this is to add a value that combines with the added checksum, to be zero, leaving the original checksum intact.
>
> This might be inordinately hard for a CRC, but a simple checksum would not be an issue, I think. At least, this could work in software, where data can be included in an image file as itself. In a device like an FPGA, it might not be included in the bit stream file so directly... but that might depend on where in the device it is inserted. Memory might have data that is stored as itself. I'll need to look into that.
>

Den 2023-04-20 kl. 04:06, skrev Rick C:
> This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
>
The proper way to do this is to have a directive in the linker.
This reserves space for the CRC and defines the area where the CRC is
calculated.
I am not aware of any linker which support this.

Two months ago, I added the DIGEST directive to binutils aka the GNU
linker. It was committed, but then people realized that I had not signed
an agreement with Free Software Foundation.
Since part of the code I pushed was from a third party which released
their code under MIT, the licensing has not been resolved yet
but the patch is in binutils git, but reverted.

You would write (IIRC):
DIGEST "CRC64-ECMA", (from, to)
and the linker would reserve 8 bytes which is filled with the CRC in the
final link stage.

/Ulf

> I'm not thinking anything too fancy, like a CRC, but rather a simple modulo N addition, maybe N being 2^16.
>
> I keep thinking of using a placeholder, but that doesn't seem to work out in any useful way. Even if you try to anticipate the impact of adding the checksum, that only gives you a different checksum, that you then need to anticipate further... ad infinitum.
>
> I'm not thinking of any special checksum generator that excludes the checksum data. That would be too messy.
>
> I keep thinking there is a different way of looking at this to achieve the result I want...
>
> Maybe I can prove it is impossible. Assume the file checksums to X when the checksum data is zero. The goal would then be to include the checksum data value Y in the file, that would change X to Y. Given the properties of the module N checksum, this would appear to be impossible for the general case, unless... Add another data value, called, checksum normalizer. This data value checksums with the original checksum to give the result zero. Then, when the checksum is also added, the resulting checksum is, in fact, the checksum. Another way of looking at this is to add a value that combines with the added checksum, to be zero, leaving the original checksum intact.
>
> This might be inordinately hard for a CRC, but a simple checksum would not be an issue, I think. At least, this could work in software, where data can be included in an image file as itself. In a device like an FPGA, it might not be included in the bit stream file so directly... but that might depend on where in the device it is inserted. Memory might have data that is stored as itself. I'll need to look into that.
>

Den 2023-04-20 kl. 22:26, skrev David Brown:
> On 20/04/2023 18:45, Rick C wrote:
>> On Thursday, April 20, 2023 at 11:33:28 AM UTC-4, George Neuner
>> wrote:
>>> On Wed, 19 Apr 2023 19:06:33 -0700 (PDT), Rick C
>>> <gnuarm.del...@gmail.com> wrote:
>>>
>>>> This is a bit of the chicken and egg thing. If you want a embed
>>>> a checksum in a code module to report the checksum, is there a
>>>> way of doing this? It's a bit like being your own grandfather, I
>>>> think.
>>> Take a look at the old xmodem/ymodem CRC. It was designed such
>>> that when the CRC was sent immediately following the data, a
>>> receiver computing CRC over the whole incoming packet (data and CRC
>>> both) would get a result of zero.
>>>
>>> But AFAIK it doesn't work with CCITT equation(s) - you have to use
>>> xmodem/ymodem.
>>>> I'm not thinking anything too fancy, like a CRC, but rather a
>>>> simple modulo N addition, maybe N being 2^16.
>>> Sorry, I don't know a way to do it with a modular checksum. YMMV,
>>> but I think 16-bit CRC is pretty simple.
>>>
>>> George
>>
>> CRC is not complicated, but I would not know how to calculate an
>> inserted value to force the resulting CRC to zero.  How do you do
>> that?
>
> You "insert" the value at the end.  Anything else is insane.

In all projects I have been involved with, the application binary starts
with a header looking like this.

MAGIC WORD 1
CRC
Entry Point
Size
other info...
MAGIC WORD 2
APPLICATION_START
....
APPLICATION_END (aligned with flash sector)

The bootloader first checks the two magic words.
It then computes CRC on the header (from Entry Point) to APPLICATION_END

I ported the IAR ielftool (open source) to Linux at
https://github.com/emagii/ielftool

This can insert the CRC in the ELF file, but needs tweaks to work
with an ELF file generated by the GNU tools.

/Ulf

>
> CRC's are quite good hashes, for suitable sized data.  There are perhaps
> some special cases, but basically you'd be doing trial-and-error
> searches to find an inserted value that gives you a zero CRC overall.
> 2^16 is not an overwhelming search space, but the whole idea is pointless.
>
>>
>> Even so, I'm not trying to validate the file.  I'm trying to come up
>> with a substitute for a time stamp or version number.  I don't want
>> to have to rely on my consistency in handling the version number
>> correctly.  This would be a backup in case there was more than one
>> version released, even only within the "lab", that were different.  A
>> checksum that could be read by the controlling software would do the
>> job.
>
> A CRC is fine for that.
>
>>
>> I have run into this before, where the version number was not a 100%
>> indication of the uniqueness of an executable.  The checksum would be
>> a second indicator.
>>
>> I should mention that I'm not looking for a solution that relies on
>> any specific details of the tools.
>>
>
> A table-based CRC is easy, runs quickly, and can be quickly ported to
> pretty much any language (the C and Python code, for example, is almost
> the same).

Den 2023-04-22 kl. 05:14, skrev Rick C:
> On Friday, April 21, 2023 at 11:02:28 AM UTC-4, David Brown wrote:
>> On 21/04/2023 14:12, Rick C wrote:
>>>
>>> This is simply to be able to say this version is unique, regardless
>>> of what the version number says. Version numbers are set manually
>>> and not always done correctly. I'm looking for something as a backup
>>> so that if the checksums are different, I can be sure the versions
>>> are not the same.
>>>
>>> The less work involved, the better.
>>>
>> Run a simple 32-bit crc over the image. The result is a hash of the
>> image. Any change in the image will show up as a change in the crc.
>
> No one is trying to detect changes in the image. I'm trying to label the image in a way that can be read in operation. I'm using the checksum simply because that is easy to generate. I've had problems with version numbering in the past. It will be used, but I want it supplemented with a number that will change every time the design changes, at least with a high probability, such as 1 in 64k.
>

Another thing I added (and was later removed) was a timestamp directive.
A 64 bit integer with the number of seconds since 1970-01-01 00:00.

/Ulf

On Thursday, April 27, 2023 at 12:26:47 PM UTC-4, Ulf Samuelsson wrote:
> Den 2023-04-20 kl. 04:06, skrev Rick C:
> > This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
> >
> The proper way to do this is to have a directive in the linker.
> This reserves space for the CRC and defines the area where the CRC is
> calculated.

That assumes there is a linker. How does the application access this information?

> I am not aware of any linker which support this.
>
> Two months ago, I added the DIGEST directive to binutils aka the GNU
> linker. It was committed, but then people realized that I had not signed
> an agreement with Free Software Foundation.
> Since part of the code I pushed was from a third party which released
> their code under MIT, the licensing has not been resolved yet
> but the patch is in binutils git, but reverted.
>
> You would write (IIRC):
> DIGEST "CRC64-ECMA", (from, to)
> and the linker would reserve 8 bytes which is filled with the CRC in the
> final link stage.

You are making a lot of assumptions about the tools. I'm pretty sure they don't apply to my case. I'm not at all clear how this is workable, anyway. Adding the checksum to the file, changes the checksum, which is where this conversation started... unless I'm missing something significant.

--

Rick C.

+++ Get 1,000 miles of free Supercharging
+++ Tesla referral code - https://ts.la/richard11209

On 2023-04-27 20:09, Rick C wrote:
> On Thursday, April 27, 2023 at 12:26:47 PM UTC-4, Ulf Samuelsson wrote:
>> Den 2023-04-20 kl. 04:06, skrev Rick C:
>>> This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
>>>
>> The proper way to do this is to have a directive in the linker.
>> This reserves space for the CRC and defines the area where the CRC is
>> calculated.
>
> That assumes there is a linker.

Almost all toolchains have a linker.

> How does the application access this information?

In Ulf's suggestion, it seems the DIGEST directive emits 8 bytes of
checksum at the current point (usually the linker "." symbol). I assume
one can give that point in the image a linkage symbol, perhaps like

_checksum DIGEST "CRC64-ECMA", (from, to)

or like

_checksum EQU. .
DIGEST "CRC64-ECMA", (from, to)

(This is schematic linker code, not necessarily proper syntax.)

One can then from the application code access the "checksum" location as
an externally defined object, say:

extern uint8[8] checksum;

The linker will connect that C identifier to the actual address of the
DIGEST checksum. Here I assumed that the C compiler mangles C
identifiers into linkage symbols by prefixing an underscore; YMMV.

> You are making a lot of assumptions about the tools. I'm pretty sure
> they don't apply to my case. I'm not at all clear how this is
> workable, anyway. Adding the checksum to the file, changes the
> checksum, which is where this conversation started... unless I'm
> missing something significant.

But you have insisted that your "checksum" is for the purpose of
identifying the version of the program, not for checking the integrity
of the memory image. If so, that checksum does not have to be the
checksum of the whole memory image, as long as it is the checksum of the
part of the image that contains the actual code and constant data, and
so will change according to changes in those parts of the image.

Um, just noting some typos in my last, with apologies:

On 2023-04-27 21:29, Niklas Holsti wrote:

>   _checksum  EQU.   .

should be

_checksum EQU .

(Thunderbird inserted an extra period out of "friendliness"...)

>    extern uint8[8] checksum;

should be (my C is rusty):

extern uint8 checksum[8];

Den 2023-04-27 kl. 19:09, skrev Rick C:
> On Thursday, April 27, 2023 at 12:26:47 PM UTC-4, Ulf Samuelsson wrote:
>> Den 2023-04-20 kl. 04:06, skrev Rick C:
>>> This is a bit of the chicken and egg thing. If you want a embed a checksum in a code module to report the checksum, is there a way of doing this? It's a bit like being your own grandfather, I think.
>>>
>> The proper way to do this is to have a directive in the linker.
>> This reserves space for the CRC and defines the area where the CRC is
>> calculated.
>
> That assumes there is a linker. How does the application access this information?
>
>
Linker command file
public CRC64; start, stop
HEADER = .;
QUAD(MAGIC);
CRC64 = .;
DIGEST "CRC64-ECMA", (start, stop)
start = .;
# Your data to be protected
...
stop = .;

C source code.

extern uint64_t CRC64;
extern char* start;
extern char* stop;

uint64_t crc;

crc64 = calc_crc64_ecma(start, stop);
if (crc64 == CRC64) {
/* everything is OK */
}

>> I am not aware of any linker which support this.
>>
>> Two months ago, I added the DIGEST directive to binutils aka the GNU
>> linker. It was committed, but then people realized that I had not signed
>> an agreement with Free Software Foundation.
>> Since part of the code I pushed was from a third party which released
>> their code under MIT, the licensing has not been resolved yet
>> but the patch is in binutils git, but reverted.
>>
>> You would write (IIRC):
>> DIGEST "CRC64-ECMA", (from, to)
>> and the linker would reserve 8 bytes which is filled with the CRC in the
>> final link stage.
>
> You are making a lot of assumptions about the tools. I'm pretty sure they don't apply to my case. I'm not at all clear how this is workable, anyway. Adding the checksum to the file, changes the checksum, which is where this conversation started... unless I'm missing something significant.
>
I am assuming that no tool support this off the shelg, but the patches
are inside binutils, but reverted.

/Ulf

Den 2023-04-27 kl. 20:29, skrev Niklas Holsti:
> On 2023-04-27 20:09, Rick C wrote:
>> On Thursday, April 27, 2023 at 12:26:47 PM UTC-4, Ulf Samuelsson wrote:
>>> Den 2023-04-20 kl. 04:06, skrev Rick C:
>>>> This is a bit of the chicken and egg thing. If you want a embed a
>>>> checksum in a code module to report the checksum, is there a way of
>>>> doing this? It's a bit like being your own grandfather, I think.
>>>>
>>> The proper way to do this is to have a directive in the linker.
>>> This reserves space for the CRC and defines the area where the CRC is
>>> calculated.
>>
>> That assumes there is a linker.
>
>
> Almost all toolchains have a linker.
>
>
>> How does the application access this information?
>
>
> In Ulf's suggestion, it seems the DIGEST directive emits 8 bytes of
> checksum at the current point (usually the linker "." symbol). I assume
> one can give that point in the image a linkage symbol, perhaps like
>
>   _checksum  DIGEST "CRC64-ECMA", (from, to)
>
> or like
>
>   _checksum  EQU.   .
>              DIGEST "CRC64-ECMA", (from, to)
>
>
> (This is schematic linker code, not necessarily proper syntax.)
>
> One can then from the application code access the "checksum" location as
> an externally defined object, say:
>
>    extern uint8[8] checksum;
>
> The linker will connect that C identifier to the actual address of the
> DIGEST checksum. Here I assumed that the C compiler mangles C
> identifiers into linkage symbols by prefixing an underscore; YMMV.
>

Yes, that is more or less it.

>
>> You are making a lot of assumptions about the tools.  I'm pretty sure
>> they don't apply to my case.  I'm not at all clear how this is
>> workable, anyway.  Adding the checksum to the file, changes the
>> checksum, which is where this conversation started... unless I'm
>> missing something significant.
No, you reserve room for the checksum, but that needs to be outside
the checked area.
The address of the checksum needs to be known to the application.
Also the limits of the checked area.
That is why the application has a header in front in my projects.
The application is started by the bootloader, which checks
a number of things before the application is started.
The application can read the header as well to allow checking
the code area at runtime.

>
>
> But you have insisted that your "checksum" is for the purpose of
> identifying the version of the program, not for checking the integrity
> of the memory image. If so, that checksum does not have to be the
> checksum of the whole memory image, as long as it is the checksum of the
> part of the image that contains the actual code and constant data, and
> so will change according to changes in those parts of the image.
>

/Ulf

On 2023-04-27 23:36, Ulf Samuelsson wrote:
> Den 2023-04-27 kl. 19:09, skrev Rick C:
>> On Thursday, April 27, 2023 at 12:26:47 PM UTC-4, Ulf Samuelsson wrote:
>>> Den 2023-04-20 kl. 04:06, skrev Rick C:
>>>> This is a bit of the chicken and egg thing. If you want a embed a
>>>> checksum in a code module to report the checksum, is there a way of
>>>> doing this? It's a bit like being your own grandfather, I think.
>>>>
>>> The proper way to do this is to have a directive in the linker.
>>> This reserves space for the CRC and defines the area where the CRC is
>>> calculated.
>>
>> That assumes there is a linker.  How does the application access this
>> information?
>>
>>
> Linker command file
>         public CRC64; start, stop
>         HEADER = .;
>         QUAD(MAGIC);
>     CRC64 = .;
>         DIGEST "CRC64-ECMA", (start, stop)
>         start = .;
>         # Your data to be protected
>         ...
>         stop = .;
>
> C source code.
>
> extern uint64_t CRC64;
> extern char* start;
> extern char* stop;
>
> uint64_t crc;
>
> crc64 = calc_crc64_ecma(start, stop);
> if (crc64 == CRC64) {
>    /* everything is OK */
> }

I'm nit-picking, but that C code does not look right to me. The extern
declarations for "start" and "stop" claim them to be names of memory
locations that contain addresses, but the linker file just places them
at the starting and one-past-end locations of the block to be protected.
So the "start" variable contains the first bytes of the "data to be
protected", and the contents of the "stop" variable are not defined
because it is placed after the "data to be protected", where no code or
data is loaded (it seems).

It seems to me that the call to calc_crc64_ecma should get the addresses
of "start" and "stop" as arguments (&start, &stop), instead of their
values. But perhaps calc_crc64_ecma is not a function, but a macro that
can itself take the addresses of its parameters.

On 27/04/2023 18:36, Ulf Samuelsson wrote:
> Den 2023-04-20 kl. 22:26, skrev David Brown:
>> On 20/04/2023 18:45, Rick C wrote:
>>> On Thursday, April 20, 2023 at 11:33:28 AM UTC-4, George Neuner
>>> wrote:
>>>> On Wed, 19 Apr 2023 19:06:33 -0700 (PDT), Rick C
>>>> <gnuarm.del...@gmail.com> wrote:
>>>>
>>>>> This is a bit of the chicken and egg thing. If you want a embed
>>>>> a checksum in a code module to report the checksum, is there a
>>>>> way of doing this? It's a bit like being your own grandfather, I
>>>>> think.
>>>> Take a look at the old xmodem/ymodem CRC. It was designed such
>>>> that when the CRC was sent immediately following the data, a
>>>> receiver computing CRC over the whole incoming packet (data and CRC
>>>> both) would get a result of zero.
>>>>
>>>> But AFAIK it doesn't work with CCITT equation(s) - you have to use
>>>> xmodem/ymodem.
>>>>> I'm not thinking anything too fancy, like a CRC, but rather a
>>>>> simple modulo N addition, maybe N being 2^16.
>>>> Sorry, I don't know a way to do it with a modular checksum. YMMV,
>>>> but I think 16-bit CRC is pretty simple.
>>>>
>>>> George
>>>
>>> CRC is not complicated, but I would not know how to calculate an
>>> inserted value to force the resulting CRC to zero.  How do you do
>>> that?
>>
>> You "insert" the value at the end.  Anything else is insane.
>
> In all projects I have been involved with, the application binary starts
> with a header looking like this.
>
>
> MAGIC WORD 1
> CRC
> Entry Point
> Size
> other info...
> MAGIC WORD 2
> APPLICATION_START
> ...
> APPLICATION_END (aligned with flash sector)
>
>
> The bootloader first checks the two magic words.
> It then computes CRC on the header (from Entry Point) to APPLICATION_END
>
> I ported the IAR ielftool (open source) to Linux at
> https://github.com/emagii/ielftool
>
> This can insert the CRC in the ELF file, but needs tweaks to work
> with an ELF file generated by the GNU tools.
>
> /Ulf

That can work for some microcontrollers, but is unsuitable for others -
it depends on how the flash is organised. For an msp430, for example,
it would be fine, as the interrupt vectors (including the reset vector)
are at the end of flash. But for most ARM Cortex M devices, it would
not be suitable - they expect the reset vector and initial stack pointer
at the start of the flash image. Some devices have a boot ROM, and then
you have to match their specifics for the header - or you can have your
own boot program, and make the header how ever you like.

I am absolutely a fan of having some kind of header like this (and
sometimes even a human-readable copyright notice, identifier and version
information). And having it as near the beginning as possible is good.
But for many microcontrollers, having it at the start is not feasible.
And if you can't put the CRC at the start like you do, you have to put
it at the end of the image.

I've never really thought about trying to inject a CRC into an elf file.
I use elfs (or should that be "elves" ?) for debugging, not flash
programming. And usually the main concern for having a CRC at the end
of the image is when you have an online update of some kind, to check
that nothing has gone wrong during the transfer or in-field update.

On 27/04/2023 18:42, Ulf Samuelsson wrote:
> Den 2023-04-22 kl. 05:14, skrev Rick C:
>> On Friday, April 21, 2023 at 11:02:28 AM UTC-4, David Brown wrote:
>>> On 21/04/2023 14:12, Rick C wrote:
>>>>
>>>> This is simply to be able to say this version is unique, regardless
>>>> of what the version number says. Version numbers are set manually
>>>> and not always done correctly. I'm looking for something as a backup
>>>> so that if the checksums are different, I can be sure the versions
>>>> are not the same.
>>>>
>>>> The less work involved, the better.
>>>>
>>> Run a simple 32-bit crc over the image. The result is a hash of the
>>> image. Any change in the image will show up as a change in the crc.
>>
>> No one is trying to detect changes in the image.  I'm trying to label
>> the image in a way that can be read in operation.  I'm using the
>> checksum simply because that is easy to generate.  I've had problems
>> with version numbering in the past.  It will be used, but I want it
>> supplemented with a number that will change every time the design
>> changes, at least with a high probability, such as 1 in 64k.
>>
>
> Another thing I added (and was later removed) was a timestamp directive.
> A 64 bit integer with the number of seconds since 1970-01-01 00:00.
>

Timestamping a build in some way (as part of the "make", using __DATE__
or __TIME__ in source code, or some feature of a revision control
system) is very tempting, and can be helpful for tracking exactly what
code you have on the system.

However, IMHO having reproducible builds is much more valuable. I am
not happy with a project build until I am getting identical binaries
built on multiple hosts (Windows and Linux). That's how you can be
absolutely sure of what code went into a particular binary, even years
or decades later.

A compromise that can work is to distinguish development builds and
production builds, and have timestamping in development builds. That
also reduces the rate at which your minor version number or build number
goes up, and avoids endless changes to your "version.h" include file.

On 27/04/2023 18:26, Ulf Samuelsson wrote:
> Den 2023-04-20 kl. 04:06, skrev Rick C:
>> This is a bit of the chicken and egg thing.  If you want a embed a
>> checksum in a code module to report the checksum, is there a way of
>> doing this?  It's a bit like being your own grandfather, I think.
>>
> The proper way to do this is to have a directive in the linker.
> This reserves space for the CRC and defines the area where the CRC is
> calculated.
> I am not aware of any linker which support this.
>
> Two months ago, I added the DIGEST directive to binutils aka the GNU
> linker. It was committed, but then people realized that I had not signed
> an agreement with Free Software Foundation.
> Since part of the code I pushed was from a third party which released
> their code under MIT, the licensing has not been resolved yet
> but the patch is in binutils git, but reverted.
>
> You would write (IIRC):
>    DIGEST "CRC64-ECMA", (from, to)
> and the linker would reserve 8 bytes which is filled with the CRC in the
> final link stage.
>
> /Ulf
>

I like that. Thanks for doing that work.

Is there also a way to get the length of the final link, and insert it
near the beginning of the image? I suppose that would be another kind
of DIGEST where the algorithm is simply (to - from). (I assume that
"to" and "from" may be linker symbols.)

On 27/04/2023 20:29, Niklas Holsti wrote:
> On 2023-04-27 20:09, Rick C wrote:
>> On Thursday, April 27, 2023 at 12:26:47 PM UTC-4, Ulf Samuelsson wrote:
>>> Den 2023-04-20 kl. 04:06, skrev Rick C:
>>>> This is a bit of the chicken and egg thing. If you want a embed a
>>>> checksum in a code module to report the checksum, is there a way of
>>>> doing this? It's a bit like being your own grandfather, I think.
>>>>
>>> The proper way to do this is to have a directive in the linker.
>>> This reserves space for the CRC and defines the area where the CRC is
>>> calculated.
>>
>> That assumes there is a linker.
>
>
> Almost all toolchains have a linker.
>

It is possible that Rick is using Forth, rather than C (or other
languages traditionally compiled in a similar manner, such as C++ and
Ada). There are also some commercial C toolchains for brain-dead 8-bit
CISC devices that are monolithic and offer very little control over the
linking.

Ulf is correct that the ideal place to handle this is part of the
linking process. I do it with a post-link Python script run during the
build, because the linkers I use can't handle this at the moment. But
if Ulf's patch works its way into binutils then I'll be able to do it
directly during linking, which is neater. (I will still have post-link
scripts to handle things like renaming image files according to version,
making zips for sending to customers, etc. - linkers can't do
/everything/ !)

On 27/04/2023 22:44, Ulf Samuelsson wrote:
> Den 2023-04-27 kl. 20:29, skrev Niklas Holsti:
>> On 2023-04-27 20:09, Rick C wrote:

>
>>
>>> You are making a lot of assumptions about the tools.  I'm pretty sure
>>> they don't apply to my case.  I'm not at all clear how this is
>>> workable, anyway.  Adding the checksum to the file, changes the
>>> checksum, which is where this conversation started... unless I'm
>>> missing something significant.
> No, you reserve room for the checksum, but that needs to be outside
> the checked area.
> The address of the checksum needs to be known to the application.

The address here could have a symbol, and then declared "extern" in the
C code - it would not have to be a known numerical address. But if the
image is checked or started from another program (such as a boot
program), you need an absolute address somewhere to chain this all together.

> Also the limits of the checked area.
> That is why the application has a header in front in my projects.
> The application is started by the bootloader, which checks
> a number of things before the application is started.
> The application can read the header as well to allow checking
> the code area at runtime.
>

Or for my preferences, the CRC "DIGEST" would be put at the end of the
image, rather than near the start. Then the "from, to" range would
cover the entire image except for the final CRC. But I'd have a similar
directive for the length of the image at a specific area near the start.

Den 2023-04-28 kl. 09:12, skrev David Brown:
> On 27/04/2023 18:36, Ulf Samuelsson wrote:
>> Den 2023-04-20 kl. 22:26, skrev David Brown:
>>> On 20/04/2023 18:45, Rick C wrote:
>>>> On Thursday, April 20, 2023 at 11:33:28 AM UTC-4, George Neuner
>>>> wrote:
>>>>> On Wed, 19 Apr 2023 19:06:33 -0700 (PDT), Rick C
>>>>> <gnuarm.del...@gmail.com> wrote:
>>>>>
>>>>>> This is a bit of the chicken and egg thing. If you want a embed
>>>>>> a checksum in a code module to report the checksum, is there a
>>>>>> way of doing this? It's a bit like being your own grandfather, I
>>>>>> think.
>>>>> Take a look at the old xmodem/ymodem CRC. It was designed such
>>>>> that when the CRC was sent immediately following the data, a
>>>>> receiver computing CRC over the whole incoming packet (data and CRC
>>>>> both) would get a result of zero.
>>>>>
>>>>> But AFAIK it doesn't work with CCITT equation(s) - you have to use
>>>>> xmodem/ymodem.
>>>>>> I'm not thinking anything too fancy, like a CRC, but rather a
>>>>>> simple modulo N addition, maybe N being 2^16.
>>>>> Sorry, I don't know a way to do it with a modular checksum. YMMV,
>>>>> but I think 16-bit CRC is pretty simple.
>>>>>
>>>>> George
>>>>
>>>> CRC is not complicated, but I would not know how to calculate an
>>>> inserted value to force the resulting CRC to zero.  How do you do
>>>> that?
>>>
>>> You "insert" the value at the end.  Anything else is insane.
>>
>> In all projects I have been involved with, the application binary starts
>> with a header looking like this.
>>
>>
>> MAGIC WORD 1
>> CRC
>> Entry Point
>> Size
>> other info...
>> MAGIC WORD 2
>> APPLICATION_START
>> ...
>> APPLICATION_END (aligned with flash sector)
>>
>>
>> The bootloader first checks the two magic words.
>> It then computes CRC on the header (from Entry Point) to APPLICATION_END
>>
>> I ported the IAR ielftool (open source) to Linux at
>> https://github.com/emagii/ielftool
>>
>> This can insert the CRC in the ELF file, but needs tweaks to work
>> with an ELF file generated by the GNU tools.
>>
>> /Ulf
>
> That can work for some microcontrollers, but is unsuitable for others -
> it depends on how the flash is organised.  For an msp430, for example,
> it would be fine, as the interrupt vectors (including the reset vector)
> are at the end of flash.  But for most ARM Cortex M devices, it would
> not be suitable - they expect the reset vector and initial stack pointer
> at the start of the flash image.  Some devices have a boot ROM, and then
> you have to match their specifics for the header - or you can have your
> own boot program, and make the header how ever you like.

All projects I am involved with have a custom bootloader.
If there is a problem with the reset vector, then the program will fail
immediately.
The CRC is right after the initial vector table.
The bootloader application contains a copy of the vector table.

THe first thing the bootloader does is to check the CRC from right after
the CRC. Then it compares the vector table with the copy.

The header is only for the application.

>
> I am absolutely a fan of having some kind of header like this (and
> sometimes even a human-readable copyright notice, identifier and version
> information).  And having it as near the beginning as possible is good.
> But for many microcontrollers, having it at the start is not feasible.
> And if you can't put the CRC at the start like you do, you have to put
> it at the end of the image.
>
>
> I've never really thought about trying to inject a CRC into an elf file.
>  I use elfs (or should that be "elves" ?) for debugging, not flash
> programming.  And usually the main concern for having a CRC at the end
> of the image is when you have an online update of some kind, to check
> that nothing has gone wrong during the transfer or in-field update.
>
>

The last bootloader I wrote download using Y-Modem which has CRC
checking. Since it had more RAM than internal flash, the whole
application was downloaded to RAM first, and then when everything is OK,
the flash can be programmed. Finally, the header is analyzed and the
flash contents checked. There is absolutely no need to have the CRC at
the end since the CRC result is stored in a known location.

/Ulf

Den 2023-04-28 kl. 09:20, skrev David Brown:
> On 27/04/2023 18:42, Ulf Samuelsson wrote:
>> Den 2023-04-22 kl. 05:14, skrev Rick C:
>>> On Friday, April 21, 2023 at 11:02:28 AM UTC-4, David Brown wrote:
>>>> On 21/04/2023 14:12, Rick C wrote:
>>>>>
>>>>> This is simply to be able to say this version is unique, regardless
>>>>> of what the version number says. Version numbers are set manually
>>>>> and not always done correctly. I'm looking for something as a backup
>>>>> so that if the checksums are different, I can be sure the versions
>>>>> are not the same.
>>>>>
>>>>> The less work involved, the better.
>>>>>
>>>> Run a simple 32-bit crc over the image. The result is a hash of the
>>>> image. Any change in the image will show up as a change in the crc.
>>>
>>> No one is trying to detect changes in the image.  I'm trying to label
>>> the image in a way that can be read in operation.  I'm using the
>>> checksum simply because that is easy to generate.  I've had problems
>>> with version numbering in the past.  It will be used, but I want it
>>> supplemented with a number that will change every time the design
>>> changes, at least with a high probability, such as 1 in 64k.
>>>
>>
>> Another thing I added (and was later removed) was a timestamp directive.
>> A 64 bit integer with the number of seconds since 1970-01-01 00:00.
>>
>
> Timestamping a build in some way (as part of the "make", using __DATE__
> or __TIME__ in source code, or some feature of a revision control
> system) is very tempting, and can be helpful for tracking exactly what
> code you have on the system.
>
> However, IMHO having reproducible builds is much more valuable.  I am
> not happy with a project build until I am getting identical binaries
> built on multiple hosts (Windows and Linux).  That's how you can be
> absolutely sure of what code went into a particular binary, even years
> or decades later.

With the timestamp located in the header, you can simply compare the
non-header area.
Make with __DATE__ or __TIME__ will tell you when that module is
compiled, not when the program is generated.
That is why TIMESTAMP is best generated in the linker.

/Ulf

>
> A compromise that can work is to distinguish development builds and
> production builds, and have timestamping in development builds.  That
> also reduces the rate at which your minor version number or build number
> goes up, and avoids endless changes to your "version.h" include file.
>
>
>
>

Den 2023-04-28 kl. 09:38, skrev David Brown:
> On 27/04/2023 22:44, Ulf Samuelsson wrote:
>> Den 2023-04-27 kl. 20:29, skrev Niklas Holsti:
>>> On 2023-04-27 20:09, Rick C wrote:
>
>>
>>>
>>>> You are making a lot of assumptions about the tools.  I'm pretty sure
>>>> they don't apply to my case.  I'm not at all clear how this is
>>>> workable, anyway.  Adding the checksum to the file, changes the
>>>> checksum, which is where this conversation started... unless I'm
>>>> missing something significant.
>> No, you reserve room for the checksum, but that needs to be outside
>> the checked area.
>> The address of the checksum needs to be known to the application.
>
> The address here could have a symbol, and then declared "extern" in the
> C code - it would not have to be a known numerical address.  But if the
> image is checked or started from another program (such as a boot
> program), you need an absolute address somewhere to chain this all
> together.

The header is declared as a struct.

>
>> Also the limits of the checked area.
>> That is why the application has a header in front in my projects.
>> The application is started by the bootloader, which checks
>> a number of things before the application is started.
>> The application can read the header as well to allow checking
>> the code area at runtime.
>>
>
> Or for my preferences, the CRC "DIGEST" would be put at the end of the
> image, rather than near the start.  Then the "from, to" range would
> cover the entire image except for the final CRC.  But I'd have a similar
> directive for the length of the image at a specific area near the start.
>

I really do not see a benefit of splitting the meta information about
the image to two separate locations.

The bootloader uses the struct for all checks.
It is a much simpler implementation once the tools support it.

You might find it easier to write a tool which adds the CRC at the end,
but that is a different issue.

Occam's Razor!

/Ulf

Den 2023-04-28 kl. 00:10, skrev Niklas Holsti:
> On 2023-04-27 23:36, Ulf Samuelsson wrote:
>> Den 2023-04-27 kl. 19:09, skrev Rick C:
>>> On Thursday, April 27, 2023 at 12:26:47 PM UTC-4, Ulf Samuelsson wrote:
>>>> Den 2023-04-20 kl. 04:06, skrev Rick C:
>>>>> This is a bit of the chicken and egg thing. If you want a embed a
>>>>> checksum in a code module to report the checksum, is there a way of
>>>>> doing this? It's a bit like being your own grandfather, I think.
>>>>>
>>>> The proper way to do this is to have a directive in the linker.
>>>> This reserves space for the CRC and defines the area where the CRC is
>>>> calculated.
>>>
>>> That assumes there is a linker.  How does the application access this
>>> information?
>>>
>>>
>> Linker command file
>>          public CRC64; start, stop
>>          HEADER = .;
>>          QUAD(MAGIC);
>>      CRC64 = .;
>>          DIGEST "CRC64-ECMA", (start, stop)
>>          start = .;
>>          # Your data to be protected
>>          ...
>>          stop = .;
>>
>> C source code.
>>
>> extern uint64_t CRC64;
>> extern char* start;
>> extern char* stop;
>>
>> uint64_t crc;
>>
>> crc64 = calc_crc64_ecma(start, stop);
>> if (crc64 == CRC64) {
>>     /* everything is OK */
>> }
>
>
> I'm nit-picking, but that C code does not look right to me. The extern
> declarations for "start" and "stop" claim them to be names of memory
> locations that contain addresses, but the linker file just places them
> at the starting and one-past-end locations of the block to be protected.
> So the "start" variable contains the first bytes of the "data to be
> protected", and the contents of the "stop" variable are not defined
> because it is placed after the "data to be protected", where no code or
> data is loaded (it seems).
>
> It seems to me that the call to calc_crc64_ecma should get the addresses
> of "start" and "stop" as arguments (&start, &stop), instead of their
> values. But perhaps calc_crc64_ecma is not a function, but a macro that
> can itself take the addresses of its parameters.
>
Whatever,
I did not put a lot of thought into that, and certainly did not check
it. The important thing is that you can declare labels in the linker
and use them in the code through extern declarations.
/Ulf

Den 2023-04-28 kl. 09:24, skrev David Brown:
> On 27/04/2023 18:26, Ulf Samuelsson wrote:
>> Den 2023-04-20 kl. 04:06, skrev Rick C:
>>> This is a bit of the chicken and egg thing.  If you want a embed a
>>> checksum in a code module to report the checksum, is there a way of
>>> doing this?  It's a bit like being your own grandfather, I think.
>>>
>> The proper way to do this is to have a directive in the linker.
>> This reserves space for the CRC and defines the area where the CRC is
>> calculated.
>> I am not aware of any linker which support this.
>>
>> Two months ago, I added the DIGEST directive to binutils aka the GNU
>> linker. It was committed, but then people realized that I had not signed
>> an agreement with Free Software Foundation.
>> Since part of the code I pushed was from a third party which released
>> their code under MIT, the licensing has not been resolved yet
>> but the patch is in binutils git, but reverted.
>>
>> You would write (IIRC):
>>     DIGEST "CRC64-ECMA", (from, to)
>> and the linker would reserve 8 bytes which is filled with the CRC in
>> the final link stage.
>>
>> /Ulf
>>
>
> I like that.  Thanks for doing that work.
>
> Is there also a way to get the length of the final link, and insert it
> near the beginning of the image?  I suppose that would be another kind
> of DIGEST where the algorithm is simply (to - from).  (I assume that
> "to" and "from" may be linker symbols.)
>
>

app_size = .;
LONG(to-from);

should work using the GNU linker.
/Ulf

On 28/04/2023 10:50, Ulf Samuelsson wrote:
> Den 2023-04-28 kl. 09:38, skrev David Brown:
>> On 27/04/2023 22:44, Ulf Samuelsson wrote:
>>> Den 2023-04-27 kl. 20:29, skrev Niklas Holsti:
>>>> On 2023-04-27 20:09, Rick C wrote:
>>
>>>
>>>>
>>>>> You are making a lot of assumptions about the tools.  I'm pretty sure
>>>>> they don't apply to my case.  I'm not at all clear how this is
>>>>> workable, anyway.  Adding the checksum to the file, changes the
>>>>> checksum, which is where this conversation started... unless I'm
>>>>> missing something significant.
>>> No, you reserve room for the checksum, but that needs to be outside
>>> the checked area.
>>> The address of the checksum needs to be known to the application.
>>
>> The address here could have a symbol, and then declared "extern" in
>> the C code - it would not have to be a known numerical address.  But
>> if the image is checked or started from another program (such as a
>> boot program), you need an absolute address somewhere to chain this
>> all together.
>
> The header is declared as a struct.
>
>>
>>> Also the limits of the checked area.
>>> That is why the application has a header in front in my projects.
>>> The application is started by the bootloader, which checks
>>> a number of things before the application is started.
>>> The application can read the header as well to allow checking
>>> the code area at runtime.
>>>
>>
>> Or for my preferences, the CRC "DIGEST" would be put at the end of the
>> image, rather than near the start.  Then the "from, to" range would
>> cover the entire image except for the final CRC.  But I'd have a
>> similar directive for the length of the image at a specific area near
>> the start.
>>
>
> I really do not see a benefit of splitting the meta information about
> the image to two separate locations.
>
> The bootloader uses the struct for all checks.
> It is a much simpler implementation once the tools support it.
>
> You might find it easier to write a tool which adds the CRC at the end,
> but that is a different issue.
>
> Occam's Razor!
>

There are different needs for different projects - and more than one way
to handle them. I find adding a CRC at the end of the image works best
for me, but I have no problem appreciating that other people have
different solutions.

On 28/04/2023 10:56, Ulf Samuelsson wrote:
> Den 2023-04-28 kl. 09:24, skrev David Brown:
>> On 27/04/2023 18:26, Ulf Samuelsson wrote:
>>> Den 2023-04-20 kl. 04:06, skrev Rick C:
>>>> This is a bit of the chicken and egg thing.  If you want a embed a
>>>> checksum in a code module to report the checksum, is there a way of
>>>> doing this?  It's a bit like being your own grandfather, I think.
>>>>
>>> The proper way to do this is to have a directive in the linker.
>>> This reserves space for the CRC and defines the area where the CRC is
>>> calculated.
>>> I am not aware of any linker which support this.
>>>
>>> Two months ago, I added the DIGEST directive to binutils aka the GNU
>>> linker. It was committed, but then people realized that I had not signed
>>> an agreement with Free Software Foundation.
>>> Since part of the code I pushed was from a third party which released
>>> their code under MIT, the licensing has not been resolved yet
>>> but the patch is in binutils git, but reverted.
>>>
>>> You would write (IIRC):
>>>     DIGEST "CRC64-ECMA", (from, to)
>>> and the linker would reserve 8 bytes which is filled with the CRC in
>>> the final link stage.
>>>
>>> /Ulf
>>>
>>
>> I like that.  Thanks for doing that work.
>>
>> Is there also a way to get the length of the final link, and insert it
>> near the beginning of the image?  I suppose that would be another kind
>> of DIGEST where the algorithm is simply (to - from).  (I assume that
>> "to" and "from" may be linker symbols.)
>>
>>
>
>    app_size = .;
>    LONG(to-from);
>
> should work using the GNU linker.

Will that work when placed earlier in the link than the definition of
"to" ? I had assumed - perhaps completely incorrectly - that the linker
would have to have established the value of "to" before its use in such
an expression.

Den 2023-04-28 kl. 15:09, skrev David Brown:
> On 28/04/2023 10:56, Ulf Samuelsson wrote:
>> Den 2023-04-28 kl. 09:24, skrev David Brown:
>>> On 27/04/2023 18:26, Ulf Samuelsson wrote:
>>>> Den 2023-04-20 kl. 04:06, skrev Rick C:
>>>>> This is a bit of the chicken and egg thing.  If you want a embed a
>>>>> checksum in a code module to report the checksum, is there a way of
>>>>> doing this?  It's a bit like being your own grandfather, I think.
>>>>>
>>>> The proper way to do this is to have a directive in the linker.
>>>> This reserves space for the CRC and defines the area where the CRC
>>>> is calculated.
>>>> I am not aware of any linker which support this.
>>>>
>>>> Two months ago, I added the DIGEST directive to binutils aka the GNU
>>>> linker. It was committed, but then people realized that I had not
>>>> signed
>>>> an agreement with Free Software Foundation.
>>>> Since part of the code I pushed was from a third party which
>>>> released their code under MIT, the licensing has not been resolved yet
>>>> but the patch is in binutils git, but reverted.
>>>>
>>>> You would write (IIRC):
>>>>     DIGEST "CRC64-ECMA", (from, to)
>>>> and the linker would reserve 8 bytes which is filled with the CRC in
>>>> the final link stage.
>>>>
>>>> /Ulf
>>>>
>>>
>>> I like that.  Thanks for doing that work.
>>>
>>> Is there also a way to get the length of the final link, and insert
>>> it near the beginning of the image?  I suppose that would be another
>>> kind of DIGEST where the algorithm is simply (to - from).  (I assume
>>> that "to" and "from" may be linker symbols.)
>>>
>>>
>>
>>     app_size = .;
>>     LONG(to-from);
>>
>> should work using the GNU linker.
>
> Will that work when placed earlier in the link than the definition of
> "to" ?  I had assumed - perhaps completely incorrectly - that the linker
> would have to have established the value of "to" before its use in such
> an expression.

Yes, the GNU linker creates a list of statements with a known size
updating the location counter for each statement.

The expressions are evaluated in a later stage
so you can add a DIGEST statement and compute the SIZE by
"LONG(to-from);" before "to" and "from" are declared.

/Ulf

>
>

Den 2023-04-28 kl. 15:04, skrev David Brown:
> On 28/04/2023 10:50, Ulf Samuelsson wrote:
>> Den 2023-04-28 kl. 09:38, skrev David Brown:
>>> On 27/04/2023 22:44, Ulf Samuelsson wrote:
>>>> Den 2023-04-27 kl. 20:29, skrev Niklas Holsti:
>>>>> On 2023-04-27 20:09, Rick C wrote:
>>>
>>>>
>>>>>
>>>>>> You are making a lot of assumptions about the tools.  I'm pretty sure
>>>>>> they don't apply to my case.  I'm not at all clear how this is
>>>>>> workable, anyway.  Adding the checksum to the file, changes the
>>>>>> checksum, which is where this conversation started... unless I'm
>>>>>> missing something significant.
>>>> No, you reserve room for the checksum, but that needs to be outside
>>>> the checked area.
>>>> The address of the checksum needs to be known to the application.
>>>
>>> The address here could have a symbol, and then declared "extern" in
>>> the C code - it would not have to be a known numerical address.  But
>>> if the image is checked or started from another program (such as a
>>> boot program), you need an absolute address somewhere to chain this
>>> all together.
>>
>> The header is declared as a struct.
>>
>>>
>>>> Also the limits of the checked area.
>>>> That is why the application has a header in front in my projects.
>>>> The application is started by the bootloader, which checks
>>>> a number of things before the application is started.
>>>> The application can read the header as well to allow checking
>>>> the code area at runtime.
>>>>
>>>
>>> Or for my preferences, the CRC "DIGEST" would be put at the end of
>>> the image, rather than near the start.  Then the "from, to" range
>>> would cover the entire image except for the final CRC.  But I'd have
>>> a similar directive for the length of the image at a specific area
>>> near the start.
>>>
>>
>> I really do not see a benefit of splitting the meta information about
>> the image to two separate locations.
>>
>> The bootloader uses the struct for all checks.
>> It is a much simpler implementation once the tools support it.
>>
>> You might find it easier to write a tool which adds the CRC at the
>> end, but that is a different issue.
>>
>> Occam's Razor!
>>
>
> There are different needs for different projects - and more than one way
> to handle them.  I find adding a CRC at the end of the image works best
> for me, but I have no problem appreciating that other people have
> different solutions.
>
>
>
>
I'd be curious to know WHY it works best for you.
/Ulf