H5py: Recommended method to walk HDF5 data tree?

[biocyberman]

In an HDF5 file, I can see there are root --> groups -->[subgroups]-->datasets. Metadata/Attributes can be attached to any of the items in the HDF5 file's hierarchy.
I am new to h5py, and the documentation does not provide much information regarding this. In PyTables, there is _f_getChild method to help conveniently walk an HDF5 data tree as demonstrated below (code is from another github project). I want to switch from PyTables to h5py in a wish to understand more how to use h5py and also to shorten runtime of the program.

    def fetch_region(self, sample, region_name, tags=None):
        region_name_int = int(region_name)
        region = self.hdf.root._f_getChild(sample)._f_getChild(region_name)
        if not tags:
            tags = self.tags

        locations = []
        for y, x in region._f_getChild('Fragments')._f_getChild('yxLocation'):
            locations.append((y, x))

        vals = {}
        for tag in tags:
            vals[tag] = []
            if self.tags[tag].is_colorspace:
                k = 'ColorCallQV'
                bases = '0123'
                wildcard = '.'
            else:
                k = 'BaseCallQV'
                bases = 'ACGT'
                wildcard = 'N'

            for (y, x), basequals in zip(locations, region._f_getChild(tag)._f_getChild(k)[:]):
                name = '%s_%s_%s' % (region_name_int, y, x)
                if len(tags) > 1: # Cases of not fragment sequencing
                    name = name + ' %s' % (tag)

                calls = []
                if self.tags[tag].prefix:
                    calls.append(self.tags[tag].prefix)

                quals = []
                for byte in basequals:
                    call = bases[byte & 0x03]
                    qual = byte >> 2

                    if qual == 63:
                        call = wildcard
                        qual = 0

                    qual = chr(qual + 33) # Is there any problem passing special characters around?

                    calls.append(call)
                    quals.append(qual)

                vals[tag].append((name, ''.join(calls), ''.join(quals)))
                #yield (name, ''.join(calls), quals)

        for i in xrange(len(locations)):
            for tag in tags:
                yield(vals[tag][i])

[andrewcollette]

I think the method you're after is Group.visit and/or Group.visititems:

http://docs.h5py.org/en/latest/high/group.html#Group.visit

Example:

def print_attrs(name, obj):
    print name
    for key, val in obj.attrs.iteritems():
        print "    %s: %s" % (key, val)

f = h5py.File('foo.hdf5','r')
f.visititems(print_attrs)

[biocyberman]

Thanks a lot. This is much better then the documentation for this part. I did not really pay attention to the callable parameter of the two methods. I will try using the two methods and give my feedback later.

[iteal]

For python3, we need to replace iteritems() with items()

I think the method you're after is Group.visit and/or Group.visititems:

http://docs.h5py.org/en/latest/high/group.html#Group.visit

Example:

def print_attrs(name, obj):
    print name
    for key, val in obj.attrs.iteritems():
        print "    %s: %s" % (key, val)

f = h5py.File('foo.hdf5','r')
f.visititems(print_attrs)

[pmdaly]

Alternative spacing option

def print_attrs(name, obj):
    shift = name.count('/') * '    '
    print(shift + name)
    for key, val in obj.attrs.items():
        print(shift + '    ' + f"{key}: {val}")

[jinnang1985]

I think the method you're after is Group.visit and/or Group.visititems:

http://docs.h5py.org/en/latest/high/group.html#Group.visit

Example:

def print_attrs(name, obj):
    print name
    for key, val in obj.attrs.iteritems():
        print "    %s: %s" % (key, val)

f = h5py.File('foo.hdf5','r')
f.visititems(print_attrs)

Python3 renamed iteritems() into items(). If using python3, replace obj.attrs.iteritems() with obj.attrs.items(), and put bracket for objects to be printed

[Khoa-NT]

Alternative spacing option

def print_attrs(name, obj):
    shift = name.count('/') * '    '
    print(shift + name)
    for key, val in obj.attrs.items():
        print(shift + '    ' + f"{key}: {val}")

I updated for printing parent one a time.

def print_attrs(name, obj):
    # Create indent
    shift = name.count('/') * '    '
    item_name = name.split("/")[-1]
    print(shift + item_name)
    try:
        for key, val in obj.attrs.items():
            print(shift + '    ' + f"{key}: {val}")
    except:
        pass

Example output

    Enum_Boolean
    String_VariableLength
volumes
    labels
        mask
        neuron_ids
            resolution: [40  4  4]
        neuron_ids_notransparency
    raw