Dear all,

I would like your comments on the following issue. Currently ParametricMapStorage can only be stored as Native (uncompressed) Format. However there is on-going discussion to include JPEG-XL as possible new Transfer Syntax. Since JPEG-XL can store arbitrary 32-bit IEEE 754 single-precision floating point values losslessly, it would make sense to support ParametricMapStorage for this use-case.

The major issue is dealing with legacy implementation. There is only one defined mechanism to store Sequence of Fragments: use of Pixel Data (7fe0,0010) with Value Representation = OB. This will conflict with (7fe0,0008) OF (FloatPixelData) data element.

What would be a good path forward ? Extend (7fe0,0008) to support Sequence of Fragments ? Allow conversion from element (7fe0,0008) to (7fe0,0010) ?

For reference, here is my current simple attempt:

* http://gdcm.sf.net/pm-jxl

As mentionned in the README:

* http://gdcm.sourceforge.net/pm-jxl/README.txt

The original ParametricMapStorage was "borrowed" from dicom4qi project.

% du -sh *.dcm
5.1M paramap.dcm
720K paramap-jxl.dcm

Hi Mathieu

Thanks for sharing the suggestion and the experiments.

Something like that would certainly be necessary if we are going to add compression support for floating point data. The same would theoretically be true for 64-bit floats in Double Float Pixel Data (7FE0,0009), even though there don't seem to be a whole bunch of 64 bit float codecs out there.

The presence, absence or improvement of the Basic Offset Table for would also need to be considered.

An interesting question is whether this effort is worthwhile though, in that the there may not be a pressing use case for lossy (irreversible) floating point compression, and lossless (reversible) compression of floating point data in an image-specific manner may not gain that much over a dumb approach like deflate applied to the whole dataset (or a better dumb codec than deflate like bzip2, if we were to add that as a new Transfer Syntax).

Anyway, something to raise with DICOM WG 4 if you want to join that group and pursue this.

David

On Wednesday, April 13, 2022 at 4:21:32 AM UTC-4, Mathieu Malaterre wrote:
> Dear all,
>
> I would like your comments on the following issue. Currently ParametricMapStorage can only be stored as Native (uncompressed) Format. However there is on-going discussion to include JPEG-XL as possible new Transfer Syntax. Since JPEG-XL can store arbitrary 32-bit IEEE 754 single-precision floating point values losslessly, it would make sense to support ParametricMapStorage for this use-case.
>
> The major issue is dealing with legacy implementation. There is only one defined mechanism to store Sequence of Fragments: use of Pixel Data (7fe0,0010) with Value Representation = OB. This will conflict with (7fe0,0008) OF (FloatPixelData) data element.
>
> What would be a good path forward ? Extend (7fe0,0008) to support Sequence of Fragments ? Allow conversion from element (7fe0,0008) to (7fe0,0010) ?
>
> For reference, here is my current simple attempt:
>
> * http://gdcm.sf.net/pm-jxl
>
> As mentionned in the README:
>
> * http://gdcm.sourceforge.net/pm-jxl/README.txt
>
> The original ParametricMapStorage was "borrowed" from dicom4qi project.
>
> % du -sh *.dcm
> 5.1M paramap.dcm
> 720K paramap-jxl.dcm

On Wednesday, April 13, 2022 at 3:31:33 PM UTC+2, dcl...@dclunie.com wrote:
> An interesting question is whether this effort is worthwhile though, in that the there may not be a pressing use case for lossy (irreversible) floating point compression, and lossless (reversible) compression of floating point data in an image-specific manner may not gain that much over a dumb approach like deflate applied to the whole dataset (or a better dumb codec than deflate like bzip2, if we were to add that as a new Transfer Syntax).

This comment also applies for integer based compressor, why bother with JPEG-LS when you can simply gzip the octet stream. Based on my limited experiments:
1. I found JPEG-XL very powerful with color transform (even superior to the hp1/hp2/hp3 from the original LOCO implementation),
2. I believe there is a small use case where float point is in the range [0..0 - 1.0] is very neat to handle (implementation comes for free in most JPEG-XL implementation since rendering is perfectly defined)
3. gzip Transfer Syntax is pretty good for archival, but it is really bad at http-byte-range support (I believe there are some hack around to get things working). But in any case I see a natural fit for DICOMWeb and JPEG-XL (multi-frames).

On Wednesday, April 13, 2022 at 4:31:09 PM UTC+2, Mathieu Malaterre wrote:
> On Wednesday, April 13, 2022 at 3:31:33 PM UTC+2, dcl...@dclunie.com wrote:
> > An interesting question is whether this effort is worthwhile though, in that the there may not be a pressing use case for lossy (irreversible) floating point compression, and lossless (reversible) compression of floating point data in an image-specific manner may not gain that much over a dumb approach like deflate applied to the whole dataset (or a better dumb codec than deflate like bzip2, if we were to add that as a new Transfer Syntax).
> This comment also applies for integer based compressor, why bother with JPEG-LS when you can simply gzip the octet stream. Based on my limited experiments:
> 1. I found JPEG-XL very powerful with color transform (even superior to the hp1/hp2/hp3 from the original LOCO implementation),
> 2. I believe there is a small use case where float point is in the range [0.0 - 1.0] is very neat to handle (implementation comes for free in most JPEG-XL implementation since rendering is perfectly defined)
> 3. gzip Transfer Syntax is pretty good for archival, but it is really bad at http-byte-range support (I believe there are some hack around to get things working). But in any case I see a natural fit for DICOMWeb and JPEG-XL (multi-frames).

For reference, using `% dcmconv --write-xfer-deflated paramap.dcm paramap-gz.dcm`. Here is what I get:

% du -sh *.dcm
5.1M paramap.dcm
1.3M paramap-gz.dcm
720K paramap-jxl.dcm

The deflate transfer syntax, because it's the only one compressing non-pixel data elements, is a bit unique, and not handy in many use case.
Also decompressing (compressing too, I believe) using deflate cannot benefit from multi-core architectures.

OTOH, the fact that JPEG-XL includes floats but not doubles is really a pity. I wonder why the specifications didn't go this bit further.

Concerning the data element that could contain compressed float pixel data: using the original PixelData seems the most natural. There is already no relation between the underlying data type (eg signed 12-bits) and the VR (OB or OW). FloatPixelData and DoubleFloatPixelData were introduced because we needed data elements with the new FL and FD VR. Allowing encapsulated data in them would require to change their VR to OB when using an encapsulated transfer syntax.