Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for meshes in device memory in MFEMSidreDataCollection #1398

Open
aaroncblack opened this issue Aug 15, 2024 · 2 comments
Open
Labels
App Integration Issues related to integration with applications enhancement New feature or request Reviewed Sidre Issues related to Axom's 'sidre' component

Comments

@aaroncblack
Copy link
Contributor

aaroncblack commented Aug 15, 2024

The Marbl code has test problems where the MFEM mesh data resides in device memory and has calls to the MFEMSidreDataCollection SetMesh() function.

The data collection source currently expects the MFEM mesh grid functions to reside on the CPU, as it is using the grid functions 'GetData()' to get the pointers to the data. These will return invalid host pointers if the grid function data is on the GPU.

I believe the source can be updated to support the MFEM mesh GF data in either cpu or gpu memory by replacing GetData() with HostRead() or HostReadWrite() calls to migrate the data if needed.

@rhornung67
Copy link
Member

@aaroncblack Sidre allows a user to specify an Umpire allocator for a single Group or an entire subtree rooted at a Group. Wherever the allocator places allocations should be accesible through the data access methods. I'm not sure, but it would seem that MFEMSidreDataCollection could leverage that.

@aaroncblack
Copy link
Contributor Author

@kennyweiss Hi Kenny, I chatted with Tom Stitt a bit on this.

For my use case, I hit some segfaults specifically on marbl problems that were running on the GPU where the hydro had migrated the MFEM mesh data, and then the MFEMSidreDataCollection tries to copy the Node data into Sidre with a GetData() call and segfaults because the host pointer is invalid.

He noted that this line would at minimum need to be replaced with a
GetData ->HostRead{Write}

https://github.com/LLNL/axom/blob/develop/src/axom/sidre/core/MFEMSidreDataCollection.cpp#L812

but he wasn't sure if that would be sufficient by itself, so recommended tagging to look at it.

I was able to get my tests working by adding a GetNodes->HostRead() call in the hydro, before MFEMSidreDataCollection was called, so didn't have to change anything in axom. I didn't test anything fancy though, for example a restart. thanks

@rhornung67 rhornung67 added enhancement New feature or request App Integration Issues related to integration with applications Sidre Issues related to Axom's 'sidre' component labels Aug 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
App Integration Issues related to integration with applications enhancement New feature or request Reviewed Sidre Issues related to Axom's 'sidre' component
Projects
None yet
Development

No branches or pull requests

3 participants