[Hdf-forum] HDF lib incompatible with HDF file spec?

Krug, Markus markus.krug at hm.edu
Wed Sep 6 10:44:08 CDT 2017

Dear all,

is anybody using libmysofa? I think it is beyond my resources to compile the sources and write a test program for the file I was generating. Nevertheless I think the file I was attaching in my previous email is correct in the sense of the HDF 3.0 file spec. As you can see it can be read by HDFview without any problem at all.
The problem is more on writing HDF files. HDFview, and I guess most of the applications that use the hdflib, are writing to certain positions in the file without checking if the position is already in use. The reason for this is the assumption that the file has been generated with hdflib. So the hdflib writes to positions that it has been left free in previous file generation- or manipulation steps.
My point is that as far as I can see there is no specification about the physical address layout in a HDF file. The whole HDF file specification is built on a tree structure where different nodes have different meanings and point to the possible next nodes. The physical position of the node is not specified. However the hdflib seems to have some assumptions where to place certain nodes. This leads very likely to a corruption of a HDF file if it was initially generated without the hdflib and then gets updated/manipulated with an application that is based on the hdflib. The reason why this hasn't been observed before might be that most people can use the original sources from the HDFgroup and stay with the generated hdflib for generating, reading/writing/manipulating their HDF files. Which I cannot, because of my limited hardware resources. My plan is to write HDF files on my embedded device and do the data analysis on host computers. This analysis will include to add new data and or metadata. So I cannot have the risk that after the analysis of the data the hdf file is corrupt.

I kindly ask the HDF community to check if my observation is true. After that we can discuss if the misunderstanding was on my side or if the HDF file spec or implementation needs an update.
Best Regards

Von: Hdf-forum [mailto:hdf-forum-bounces at lists.hdfgroup.org] Im Auftrag von David Pearah
Gesendet: Mittwoch, 6. September 2017 04:32
An: HDF Users Discussion List <hdf-forum at lists.hdfgroup.org>
Betreff: Re: [Hdf-forum] HDF lib incompatible with HDF file spec?

Just FYI... if the question is whether the Markus' code produces correct HDF5 files, then it might be helpful to look at an independently developed reader. If so, it might be worth looking at libmysofa:


"... The NetCDF and HDF5 libraries, which were intended to handle big data, were not originally designed to be compiled on constrained devices. The German company Symonics GmbH, (together with help from The HDF Group), has reimplemented the HDF5 file format specifications aiming at a light-weight HDF5 reader library called libmysofa.  With libmysofa, the size of a SOFA reader can be reduced by a factor of eight.  The library is open source and available under the Apache license.  It provides reading capabilities to access SOFA files and directly addresses loading HRTFs into the system."

Hope this helps,

-- Dave

From: Hdf-forum <hdf-forum-bounces at lists.hdfgroup.org<mailto:hdf-forum-bounces at lists.hdfgroup.org>> on behalf of Quincey Koziol <koziol at lbl.gov<mailto:koziol at lbl.gov>>
Sent: Tuesday, September 5, 2017 2:27 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF lib incompatible with HDF file spec?

Indeed.  :-)   I don't have time to look into Markus's file today, but I will take a look tomorrow and see what the best course of action is.


> On Sep 5, 2017, at 2:22 PM, Latham, Robert J. <robl at mcs.anl.gov<mailto:robl at mcs.anl.gov>> wrote:
> On Tue, 2017-09-05 at 17:21 +0000, Miller, Mark C. wrote:
>> Hmm. If I understand you, you have written code that you believe
>> produces an HDF5 file according to the 3.0 file version
>> specification, https://support.hdfgroup.org/HDF5/doc/H5.format.html
>> but nevertheless does NOT use the HDF5 library to do it. Furthermore,
>> where 'extended padding' is concerned, your implementation does
>> business differently than the HDF5 implementation.
>> You can prove HDF5 tools will *read* the file ok. But, in a read-
>> modify-write scenario, the file is getting corrupted by HDF5 library
>> due to the difference in how the two implementations handle the
>> extended padding -- a feature that you explain is '...not defined at
>> all -- not even recommended'.
>> Is that about right?
>> If so, it does indeed sound like a potential issue in the file format
>> specification for HDF5.
>> Your scenario sounds like a super useful test case...does a wholly
>> independent implementation produce a file the HDF5 library can
>> "handle"?
> Over in Parallel-NetCDF land a few years back, we took, um, a "rather
> aggressive interpretation" of the NetCDF spec with respect to alignment
> and then opend a bug with Unidata when their tools did not follow the
> rules as written.
> As Mark observes, it was a productive exercise in keeping both
> implementations honest.
> ==rob
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at lists.hdfgroup.org<mailto:Hdf-forum at lists.hdfgroup.org>
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5

Hdf-forum is for HDF software users discussion.
Hdf-forum at lists.hdfgroup.org<mailto:Hdf-forum at lists.hdfgroup.org>
Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170906/1df1c22a/attachment-0001.html>

More information about the Hdf-forum mailing list