Mfem: Sidre data collection and reloading data

[rcarson3]

So, I'm currently working on adding QuadratureFunction support to the SidreDataCollection in this branch: feature/sidre-qf-support , so I can later use the output to restart simulations in ExaConstit. I wrote a small example following what's done in the convert-dc miniapp to test the code, and I've found that when loading a previously saved off SidreDataCollection that neither the mesh nor the underlying saved off qf data is accessible. They just return null pointers. I'm not really familiar with this class, so I'm not too sure if I'm just doing things wrong or if there's additional steps that I need to perform when loading up an already saved off data collection. At least at a first glance though, it appears that something like the VisitDataCollection::LoadMesh function might be needed for the mesh stuff.

@kennyweiss @cyrush

[kennyweiss]

@rcarson3 -- Looking at your code, I don't see any obvious problems.

Did you confirm that you can save and load the datacollection without the QFunctions (in this branch and in master)?

[kennyweiss]

@rcarson3 -- Does your branch compile?
I tried to compile your branch w/ mfem configured using mpi and axom and ran into multiple compilation errors.

E.g.

<mfem>/fem/lininteg.cpp(917): error: class "mfem::QuadratureSpace" has no member "GetElementIntRule"
        &vqfc.GetQuadFunction().GetSpace()->GetElementIntRule(Tr.ElementNo);
                                            ^
<mfem>/fem/lininteg.cpp(948): error: class "mfem::QuadratureSpace" has no member "GetElementIntRule"
        &qfc.GetQuadFunction().GetSpace()->GetElementIntRule(Tr.ElementNo);
                                           ^
compilation aborted for <mfem>/fem/pfespace.cpp (code 2)
make[1]: *** [fem/pfespace.o] Error 2
<mfem>/fem/datacollection.cpp(440): error: class "mfem::QuadratureSpace" has no member "GetMesh"
     Mesh *mesh = qf->GetSpace()->GetMesh();
                                  ^
<mfem>/fem/datacollection.cpp(441): error: class "mfem::QuadratureSpace" has no member "GetNE"
     for (int e=0; e<qf->GetSpace()->GetNE(); e++)
                                     ^
<mfem>/fem/sidredatacollection.cpp(1210): error: class "mfem::QuadratureSpace" has no member "GetOrder"
     v->setScalar(qf->GetSpace()->GetOrder());
                                  ^
<mfem>/fem/fespace.cpp(2525): error: class "mfem::QuadratureSpace" has no member "Save"
  void QuadratureSpace::Save(std::ostream &out) const
                        ^
<mfem>/fem/fespace.cpp(2525): error: a type qualifier is not allowed on a nonmember function
  void QuadratureSpace::Save(std::ostream &out) const
                                                ^
<mfem>/fem/fespace.cpp(2529): error: identifier "order" is undefined
         << "Order: " << order << '\n';
                         ^

EDIT:
I was in the mfem branch: feature/sidre-qf-support.
The errors don't appear to be related to axom or conduit, but in case it's helpful, I was using axom@a1b769e3 and conduit@19daba8e.

[rcarson3]

@kennyweiss sorry about those compiler errors... I'd accidentally introduced some extra characters before the QuadratureSpace::GetOrder definition when I copied some of this from another branch I was originally testing all of this on. I just pushed up a new commit that removed those, and it should now compile.

Also, I just tested the sidre example with and without the quadrature additions (so pretty much this branch and master). The data collection appears to load. I can view quite a bit of info when looking at this in gdb. However, the mesh doesn't appear to be associated with it at least when doing:

    SidreDataCollection dc("base", mesh);
    dc.RegisterField("u", u);
    dc.RegisterField("v", v);
    dc.SetCycle(5);
    dc.SetTime(8.0);
    dc.Save();

    SidreDataCollection dc_new("base");
    dc_new.Load(dc.GetCycle());
    Mesh* mesh_new = dc_new.GetMesh();  \\ This guy is null

[kennyweiss]

Thanks for the update @rcarson3

Unfortunately, it looks like SidreDataCollection::Load() does not construct a mesh or directly associate field data with grid functions:
https://github.com/mfem/mfem/blob/27acd4f56df4dc24481dcd24e239effe23541f1e/fem/sidredatacollection.cpp#L698-L729
This is likely due to the way it was used when we initially developed this, where we Load() mesh data into an already constructed mesh instance. I.e. it is the user's responsibility to construct the mesh, construct the finite element spaces and grid function objects and to copy field data (or update pointers).
The way I use this is to call something like simulation.init() which creates the mesh, FESpace instances and GridFunction instances and then overload the data with something like simulation.restart(cyle) (which internally calls dc->Load() and then copies data).

In contrast, VisItDataCollection::Load() calls functions to load the mesh and fields:
https://github.com/mfem/mfem/blob/27acd4f56df4dc24481dcd24e239effe23541f1e/fem/datacollection.cpp#L564-L597

I suppose we could add that functionality, e.g. if the mesh pointer is null.

Here is a relevant constructor that you could use to construct the mesh from the Sidre data:
https://github.com/mfem/mfem/blob/27acd4f56df4dc24481dcd24e239effe23541f1e/mesh/mesh.hpp#L484-L499