Pydicom: Pixel Data with undefined length must start with an item tag

Created on 13 Sep 2018 · 9Comments · Source: pydicom/pydicom

Opening an issue proper here - I commented on an old PR and that's probably not desired :)

PR in question: https://github.com/pydicom/pydicom/pull/490
Issue in deid https://github.com/pydicom/deid/issues/65

hey @pydicom/pydcom-fff ! We are hitting this issue in deid after cleaning data (replacing some values with 0) and trying to save the image. Complete traceback is here: https://github.com/pydicom/deid/issues/65#issuecomment-421059020 and I'll also copy paste the basic operations for quick viewing here:

we read in the dicom
we identify a coordinate to black out, e.g.,

 coordinates
[[0, 0, 800, 59]]

Then we fill in with black (0), this is a copy of the array called "cleaned" We replace the PixelData with cleaned (and this may be where the issue arises, in how I am saving here)?

dicom.PixelData = cleaned.tostring()

and then the standard save for the dicom triggers the error seen here:

dicom.save_as(dicom_name)

```python
Traceback (most recent call last):
File "/home/vanessa/anaconda3/lib/python3.6/site-packages/pydicom-1.1.0-py3.6.egg/pydicom/tag.py", line 30, in tag_in_exception
yield
File "/home/vanessa/anaconda3/lib/python3.6/site-packages/pydicom-1.1.0-py3.6.egg/pydicom/filewriter.py", line 475, in write_dataset
write_data_element(fp, dataset.get_item(tag), dataset_encoding)
File "/home/vanessa/anaconda3/lib/python3.6/site-packages/pydicom-1.1.0-py3.6.egg/pydicom/filewriter.py", line 435, in write_data_element
raise ValueError('Pixel Data with undefined length must '
ValueError: Pixel Data with undefined length must start with an item tag
````
Is the error one level up where we replace the data, and if not, how do we deal with this particular circumstances? I am attaching the image in question to debug, and @fimafurman and I can add further detail if information is needed. Thanks for the help!
IMG00001.dcm.tar.gz

question

Source

vsoch

Most helpful comment

For compressed pixel data that will be written decompressed:

from pydicom import dcmread

ds = dcmread('IMG00001.dcm')
# 'JPEG Lossless, Non-Hierarchical, First-Order Prediction (Process 14 [Selection Value 1])'
print(ds.file_meta.TransferSyntaxUID.name)
# True
print(ds.file_meta.TransferSyntaxUID.is_compressed)

# Decompress Pixel Data, assuming we have the required dependencies
# Also sets the transfer syntax UID and the element length
ds.decompress()
# False
print(ds.file_meta.TransferSyntaxUID.is_compressed)

ds.save_as('out.dcm')

For compressed pixel data that has been modified and will be written compressed:

from pydicom import dcmread
from pydicom.encaps import encapsulate

ds = dcmread('IMG00001.dcm')

# Decompress and modify Pixel Data, assuming we have the required dependencies
arr = ds.pixel_array  
arr = some_function_that_modifies_arr(arr)

# Compress the Pixel Data, outside scope of pydicom
#   assumes that the results of the compression is a list of bytes, 
#   each item being the compressed data from a single frame
pixel_data = some_function_that_compresses_arr(arr)  
# Write encapsulated and compressed data back to Pixel Data
ds.PixelData = encapsulate(pixel_data )  
# Update the transfer syntax with whatever syntax matches the compressed data
ds.file_meta.TransferSyntaxUID = WhateverTransferSyntax  

ds.save_as('out.dcm')

scaramallion on 15 Sep 2018

👍3 ❤2

All 9 comments

The dataset is a compressed transfer syntax (1.2.840.10008.1.2.4.70, JPEGLossless). Compressed transfer syntaxes have 'undefined length' (i.e. 0xFFFFFFFF) and are encapsulated as per Part 5, Annex A. Uncompressed transfer syntaxes (implicit/explicit little/big) don't require encapsulation.

Assuming you're going from compressed ds.PixelData -> uncompressed ds.pixel_array -> uncompressed ds.PixelData then you'll probably first have to change the transfer syntax to an appropriate (uncompressed) value before trying to save since the file writer assumes that with a compressed transfer syntax the pixel data has been encapsulated (and hence tries to find the item tag).

If you're maintaining the compression then you first have to encapsulate your data (there are encapsulation functions in the encaps module that will be present in the v1.2 release and are in the current master), then use the resulting output to set ds.PixelData.

scaramallion on 14 Sep 2018

👍1

Would this be a better message?

ValueError: Pixel Data encoded using a compressed transfer syntax must be encapsulated however no item tag was found. Please check the transfer syntax is appropriate and that your pixel data has been encapsulated

scaramallion on 14 Sep 2018

A change is a good idea! I would be more direct - if it's hit for this save, then maybe the user should hear "if saving originally compressed as uncompressed data be sure to change the ds.TransferSyntax (or whatever is the appropriate field).

For this instance, what specific commands / steps do you recommend? It sounds like the upcoming v1.2 release will have functions that could be useful, but we should have a workaround for now.

vsoch on 14 Sep 2018

hey @scaramallion could you give an example of how to use the functions here?

https://github.com/pydicom/pydicom/blob/master/pydicom/encaps.py

I tried (cleaned is a uint8 numpy array)

In [42]: encapsulate(cleaned)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-42-8a1999fc0d60> in <module>()
----> 1 encapsulate(cleaned)

<ipython-input-40-24e31b7fee0d> in encapsulate(frames, fragments_per_frame, has_bot)
    567         # `itemised_length` is the total length of each itemised frame
    568         itemised_length = 0
--> 569         for item in itemise_frame(frame, fragments_per_frame):
    570             itemised_length += len(item)
    571             output.extend(item)

<ipython-input-40-24e31b7fee0d> in itemise_frame(frame, nr_fragments)
    501     """
    502     for fragment in fragment_frame(frame, nr_fragments):
--> 503         yield itemise_fragment(fragment)
    504 
    505 

<ipython-input-40-24e31b7fee0d> in itemise_fragment(fragment)
    466     item += pack('<I', len(fragment))
    467     # fragment data
--> 468     item += fragment
    469 
    470     return item

TypeError: ufunc 'add' did not contain a loop with signature matching types dtype('S8') dtype('S8') dtype('S8')

And if a function in here works, am I allowed to include the file (unchanged with pydicom LICENSE) in deid until the official 1.2 release (that I can require for install?)

vsoch on 14 Sep 2018

For compressed pixel data that will be written decompressed:

from pydicom import dcmread

ds = dcmread('IMG00001.dcm')
# 'JPEG Lossless, Non-Hierarchical, First-Order Prediction (Process 14 [Selection Value 1])'
print(ds.file_meta.TransferSyntaxUID.name)
# True
print(ds.file_meta.TransferSyntaxUID.is_compressed)

# Decompress Pixel Data, assuming we have the required dependencies
# Also sets the transfer syntax UID and the element length
ds.decompress()
# False
print(ds.file_meta.TransferSyntaxUID.is_compressed)

ds.save_as('out.dcm')

For compressed pixel data that has been modified and will be written compressed:

from pydicom import dcmread
from pydicom.encaps import encapsulate

ds = dcmread('IMG00001.dcm')

# Decompress and modify Pixel Data, assuming we have the required dependencies
arr = ds.pixel_array  
arr = some_function_that_modifies_arr(arr)

# Compress the Pixel Data, outside scope of pydicom
#   assumes that the results of the compression is a list of bytes, 
#   each item being the compressed data from a single frame
pixel_data = some_function_that_compresses_arr(arr)  
# Write encapsulated and compressed data back to Pixel Data
ds.PixelData = encapsulate(pixel_data )  
# Update the transfer syntax with whatever syntax matches the compressed data
ds.file_meta.TransferSyntaxUID = WhateverTransferSyntax  

ds.save_as('out.dcm')

scaramallion on 15 Sep 2018

👍3 ❤2

This is exactly the fix that we need! I simply added this check to the save function (and I think) it's working!

dicom = read_file(dicom_file, force=False)
dicom.file_meta.TransferSyntaxUID.is_compressed is True:
    dicom.decompress()

# cleaned is uint8 numpy array that we've been messing with
dicom.PixelData = cleaned.tostring()
dicom.save_as('/home/vanessa/Desktop/cleaned.dcm')

@fimafurman I will have a PR for you to test in a few minutes for deid.

vsoch on 15 Sep 2018

🎉1 👍1

Thank you!

On Sat, Sep 15, 2018 at 10:47 AM Vanessa Sochat notifications@github.com
wrote:

This is exactly the fix that we need! I simply added this check to the
save function (and I think) it's working!

dicom = read_file(dicom_file, force=False)
dicom.file_meta.TransferSyntaxUID.is_compressed is True:
dicom.decompress()

cleaned is uint8 numpy array that we've been messing with

dicom.PixelData = cleaned.tostring()
dicom.save_as('/home/vanessa/Desktop/cleaned.dcm')

@@fimafurman https://github.com/fimafurman I will have a PR for you to
test in a few minutes for deid.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pydicom/pydicom/issues/738#issuecomment-421581350,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Ak2KXGyVf-7hP1lMqhRIS0iG6HE-gjRgks5ubRMQgaJpZM4WnnrH
.

>