[Hdf-forum] high level API for parallel version of HDF5 library

Nelson, Jarom nelson99 at llnl.gov
Thu Sep 28 09:10:11 CDT 2017


herr_t H5Sselect_none(hid_t space_id);
https://support.hdfgroup.org/HDF5/hdf5-quest.html#par-nodata
https://www.hdfgroup.org/2015/08/parallel-io-with-hdf5/
https://support.hdfgroup.org/HDF5/PHDF5/

Jarom

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces at lists.hdfgroup.org] On Behalf Of Rafal Lichwala
Sent: Thursday, September 28, 2017 6:52 AM
To: hdf-forum at lists.hdfgroup.org
Subject: Re: [Hdf-forum] high level API for parallel version of HDF5 library

Hi,

Thank you for an answer and example codes.
Creating metadata (groups, datasets) is clear now and works fine, but I've got the last doubt: what in case I'm running 4 MPI processes but only 3 of them have some data to be written to the given dataset.
Since the H5Dwrite() call is in collective mode, my program hangs...
How to solve this?

Regards,
Rafal



W dniu 2017-09-27 o 22:50, Nelson, Jarom pisze:
> Calls that affect the metadata need to be collective so that each 
> process has a consistent view of what the file metadata should be.
> 
> https://support.hdfgroup.org/HDF5/doc/RM/CollectiveCalls.html
> 
> Something like this (or the attached):
> 
> plist_id = H5Pcreate(H5P_FILE_ACCESS);
> 
> H5Pset_fapl_mpio(plist_id, comm, info);
> 
> H5Pset_all_coll_metadata_ops( plist_id, true );
> 
> file_id = H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, 
> plist_id);
> 
> H5Pclose(plist_id);
> 
> for(int procid = 0; i < mpi_size; ++i) {
> 
>    hid_t gr_id = H5Gcreate(file_id, std::to_string(procid).c_str(), 
> H5P_DEFAULT, H5P_DEFAULT, H5P_DEFAULT);
> 
>    H5Gclose(gr_id);
> 
> }
> 
> H5Fclose(file_id);
> 
> -----Original Message-----
> From: Hdf-forum [mailto:hdf-forum-bounces at lists.hdfgroup.org] On 
> Behalf Of Rafal Lichwala
> Sent: Wednesday, September 27, 2017 12:32 AM
> To: hdf-forum at lists.hdfgroup.org
> Subject: Re: [Hdf-forum] high level API for parallel version of HDF5 
> library
> 
> Hi Barbara, Hi All,
> 
> Thank you for your answer. That's clear now about H5TBmake_table() 
> call, but...
> 
> H5Gcreate() in not a high level API, isn't it?
> 
> So why I cannot use it in parallel processes?
> 
> Maybe I'm just doing something wrong, so could you please provide me a 
> short example how to create a set of groups (each one is the process
> 
> number) running 4 parallel MPI processes? You can limit the example 
> code to the sequence of HDF5 calls only...
> 
> My current code works fine for just one process, but when I try it for 
> 2 (or more) parallel processes the result file is corrupted:
> 
> plist_id = H5Pcreate(H5P_FILE_ACCESS);
> 
> H5Pset_fapl_mpio(plist_id, comm, info);
> 
> H5Pset_all_coll_metadata_ops( plist_id, true ); file_id = 
> H5Fcreate(H5FILE_NAME, H5F_ACC_TRUNC, H5P_DEFAULT, plist_id); 
> H5Pclose(plist_id); hid_t gr_id = H5Gcreate(file_id, 
> std::to_string(procid).c_str(), H5P_DEFAULT, H5P_DEFAULT, 
> H5P_DEFAULT); H5Gclose(gr_id); H5Fclose(file_id);
> 
> Best regards,
> 
> Rafal
> 
> W dniu 2017-09-25 o 22:20, Barbara Jones pisze:
> 
>  > Hi Rafal,
> 
>  >
> 
>  > No, the HDF5 High Level APIs are not supported in the parallel 
> version of HDF5.
> 
>  >
> 
>  > -Barbara
> 
>  > help at hdfgroup.org <mailto:help at hdfgroup.org>
> 
>  >
> 
>  > -----Original Message-----
> 
>  > From: Hdf-forum [mailto:hdf-forum-bounces at lists.hdfgroup.org] On 
> Behalf Of Rafal Lichwala
> 
>  > Sent: Monday, September 18, 2017 8:53 AM
> 
>  > To: hdf-forum at lists.hdfgroup.org 
> <mailto:hdf-forum at lists.hdfgroup.org>
> 
>  > Subject: [Hdf-forum] high level API for parallel version of HDF5 
> library
> 
>  >
> 
>  > Hi,
> 
>  >
> 
>  > Can I use high level API function calls (H5TBmake_table(...)) in 
> parallel version of the HDF5 library?
> 
>  > There are no property list parameters for that function calls...
> 
>  >
> 
>  > Regards,
> 
>  > Rafal
> 
>  >
> 
>  >
> 
>  > _______________________________________________
> 
>  > Hdf-forum is for HDF software users discussion.
> 
>  > Hdf-forum at lists.hdfgroup.org <mailto:Hdf-forum at lists.hdfgroup.org>
> 
>  > 
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> 
>  > Twitter: https://twitter.com/hdf5
> 
>  >
> 
>  > _______________________________________________
> 
>  > Hdf-forum is for HDF software users discussion.
> 
>  > Hdf-forum at lists.hdfgroup.org <mailto:Hdf-forum at lists.hdfgroup.org>
> 
>  > 
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> 
>  > Twitter: https://twitter.com/hdf5
> 
>  >
> 
> _______________________________________________
> 
> Hdf-forum is for HDF software users discussion.
> 
> Hdf-forum at lists.hdfgroup.org <mailto:Hdf-forum at lists.hdfgroup.org>
> 
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> 
> Twitter: https://twitter.com/hdf5
> 
> 
> 
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum at lists.hdfgroup.org
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.or
> g
> Twitter: https://twitter.com/hdf5
> 

-- 

***
Rafał Lichwała
Poznańskie Centrum Superkomputerowo-Sieciowe ul. Jana Pawła II nr 10
61-139 Poznań
e-mail: syriusz at man.poznan.pl
***

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum at lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5


More information about the Hdf-forum mailing list