[Hdf-forum] HDF lib incompatible with HDF file spec?

Miller, Mark C. miller86 at llnl.gov
Wed Sep 6 11:27:42 CDT 2017


Hi Dave,

Interesting information. Thanks for sharing.

That said, and not at all to denigrate the work you describe here but I suspect doing a lightweight reader....umm, well, I was about to say would be substantially easier than doing a writer. But, given that any file the HDF5 library is capable of reading could have been produced by any one of a number of VFDs, having a reader capable of reading *any* HDF5 file is probably a fairly complex under-taking as well. I suspect the reader you reference here is probably assuming something like the stdio or sec2 VFD and not something more exotic like the family or split VFD or globus.

This discussion actually points in a direction I was headed when first raising some questions about the impact of the *current* bytes-on-disk file format in terms of HPC relevant performance scenarios as well as SQE necessary to support it.

Mark


"Hdf-forum on behalf of David Pearah" wrote:


Just FYI... if the question is whether the Markus' code produces correct HDF5 files, then it might be helpful to look at an independently developed reader. If so, it might be worth looking at libmysofa:

https://www.hdfgroup.org/2017/04/hdf5-under-the-sofa-hdf5-on-embedded-and-mobile-devices/
https://github.com/hoene/libmysofa

"... The NetCDF and HDF5 libraries, which were intended to handle big data, were not originally designed to be compiled on constrained devices. The German company Symonics GmbH, (together with help from The HDF Group), has reimplemented the HDF5 file format specifications aiming at a light-weight HDF5 reader library called libmysofa.  With libmysofa, the size of a SOFA reader can be reduced by a factor of eight.  The library is open source and available under the Apache license.  It provides reading capabilities to access SOFA files and directly addresses loading HRTFs into the system."


Hope this helps,

-- Dave


________________________________
From: Hdf-forum <hdf-forum-bounces at lists.hdfgroup.org> on behalf of Quincey Koziol <koziol at lbl.gov>
Sent: Tuesday, September 5, 2017 2:27 PM
To: HDF Users Discussion List
Subject: Re: [Hdf-forum] HDF lib incompatible with HDF file spec?

Indeed.  :-)   I don’t have time to look into Markus’s file today, but I will take a look tomorrow and see what the best course of action is.

        Quincey



> On Sep 5, 2017, at 2:22 PM, Latham, Robert J. <robl at mcs.anl.gov> wrote:
>
> On Tue, 2017-09-05 at 17:21 +0000, Miller, Mark C. wrote:
>> Hmm. If I understand you, you have written code that you believe
>> produces an HDF5 file according to the 3.0 file version
>> specification, https://support.hdfgroup.org/HDF5/doc/H5.format.html
>> but nevertheless does NOT use the HDF5 library to do it. Furthermore,
>> where 'extended padding' is concerned, your implementation does
>> business differently than the HDF5 implementation.
>>
>> You can prove HDF5 tools will *read* the file ok. But, in a read-
>> modify-write scenario, the file is getting corrupted by HDF5 library
>> due to the difference in how the two implementations handle the
>> extended padding -- a feature that you explain is '...not defined at
>> all -- not even recommended'.
>>
>> Is that about right?
>>
>> If so, it does indeed sound like a potential issue in the file format
>> specification for HDF5.
>>
>> Your scenario sounds like a super useful test case...does a wholly
>> independent implementation produce a file the HDF5 library can
>> "handle"?
>
> Over in Parallel-NetCDF land a few years back, we took, um, a "rather
> aggressive interpretation" of the NetCDF spec with respect to alignment
> and then opend a bug with Unidata when their tools did not follow the
> rules as written.
>
> As Mark observes, it was a productive exercise in keeping both
> implementations honest.
>
> ==rob
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
> Twitter: https://twitter.com/hdf5



_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum at lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.hdfgroup.org/pipermail/hdf-forum_lists.hdfgroup.org/attachments/20170906/609ab3a8/attachment.html>


More information about the Hdf-forum mailing list