Design and use of the APF sharing object

This discussion is trying to address the issues listed below.

R1

numberOwnedDimension does not support a user defined definition of sharing.

Adding a sharing argument to numberOwnedDimension that defaults to NULL is fine with me. Since this API didn't have a sharing argument, and with the proposed change in R4, we can pass safely (no memory leaks etc.) pass delete=true to numberNodes when the user passes a non-NULL sharing object.

R2

makeGlobal does provide an argument for a user defined sharing object.

makeGlobal takes a local numbering object that has only numbered owned nodes. In this way it can globalize the numbering without being concerned about the specific sharing rule. No code change is required here, but the documentation of 'makeGlobal' should give an example of the usage to make it a bit more clear.

R3

PUMI exposes creation of global numbering creation without first creating a local numbering. This is a problem because of R2.

To create a global numbering the PUMI API should force the users to first create a local numbering as described for R2. This requires no changes in APF and should not require too many changes in the PUMI API implementation.

R4

synchronizeFieldData and numberNodes delete the sharing object that is passed in or created internally. Deleting the internally created object is fine since the user never had control/awareness of it, but deleting the users object is a side effect that seems unnecessary.

synchronizeFieldData ( declaration , implementation ) is called by synchronize and accumulateFieldData.

Below is a list of all SCOREC/core references and calls to the synchronize functions:

./apf/apf.cc:void synchronize(Field* f, Sharing* shr)
./apf/apf.h:void synchronize(Field* f, Sharing* shr = 0);
./apf/apfConstruct.cc:  synchronize(global);
./apf/apfNumbering.cc:void synchronize(Numbering * n, Sharing* shr)
./apf/apfNumbering.cc:void synchronize(GlobalNumbering* n, Sharing* shr)
./apf/apfNumbering.h:void synchronize(Numbering * n, Sharing* shr = 0);
./apf/apfNumbering.h:void synchronize(GlobalNumbering* n, Sharing* shr = 0);
./crv/crv.h:    void synchronize();
./crv/crvCurveMesh.cc:void MeshCurver::synchronize()
./crv/crvCurveMesh.cc:  apf::synchronize(m_mesh->getCoordinateField());
./crv/crvCurveMesh.cc:  synchronize();
./crv/crvCurveMesh.cc:  synchronize();
./crv/crvCurveMesh.cc:    synchronize();
./crv/crvCurveMesh.cc:  synchronize();
./crv/crvCurveMesh.cc:  synchronize();
./ma/maTetrahedronize.cc:  apf::synchronize(global);
./omega_h/apfOmega_h.cc:  apf::synchronize(globals_apf);
./pumi/pumi.h:void pumi_field_synchronize(pField f, pSharing shr=NULL);
./pumi/pumi_field.cc:void pumi_field_synchronize(pField f, pSharing shr)
./pumi/pumi_field.cc:  apf::synchronize(f, shr);
./sam/sam.cc:  apf::synchronize(f);
./sam/sam.cc:  apf::synchronize(f);
./sam/sam.cc:  apf::synchronize(f);
./sam/samSz.cc:  apf::synchronize(fLen);
./sam/samSz.cc:  apf::synchronize(fCnt);
./stk/apfMeshSTK.cc:  synchronize(n[0]);
./stk/apfMeshSTK.cc:  synchronize(n[d - 1]);
./stk/apfMeshSTK.cc:  synchronize(n[d]);
./test/fusion2.cc:  apf::synchronize(m->findField("fusion"));
./test/mixedNumbering.cc:    apf::synchronize(global[n]);
./test/nektar_align.cc:  apf::synchronize(n);
./test/pumi.cc:    pumi_field_synchronize(f, new testSharing(m));
./test/refine2x.cc:      apf::synchronize(fLen);
./test/refine2x.cc:      apf::synchronize(fCnt);
./zoltan/apfZoltanCallbacks.cc:    synchronize(gn);

numberNodes ( implementation ) is an internal apfNumbering.cc function called by numberOwnedDimension, numberOverlapDimension, numberOverlapNodes, and numberOwnedNodes.

Here is a list of all SCOREC/core references and calls to the above functions:

apf/apfArrayData.cc://      num_var = numberOverlapNodes(f->getMesh(),name,s);
apf/apfArrayData.cc:      if (!n) n = numberOverlapNodes(f->getMesh(),name,s);   
apf/apfConstruct.cc:  GlobalNumbering* global = makeGlobal(numberOwnedNodes(m, "apf_destruct"));
apf/apfNumbering.cc:Numbering* numberOwnedDimension(Mesh* mesh, const char* name, int dim)
apf/apfNumbering.cc:Numbering* numberOverlapDimension(Mesh* mesh, const char* name, int dim)
apf/apfNumbering.cc:  return numberOwnedDimension(mesh, name, mesh->getDimension());
apf/apfNumbering.cc:Numbering* numberOverlapNodes(Mesh* mesh, const char* name, FieldShape* s)
apf/apfNumbering.cc:Numbering* numberOwnedNodes(
apf/apfNumbering.h:Numbering* numberOwnedDimension(Mesh* mesh, const char* name, int dim);
apf/apfNumbering.h:Numbering* numberOverlapDimension(Mesh* mesh, const char* name, int dim);
apf/apfNumbering.h:Numbering* numberOverlapNodes(
apf/apfNumbering.h:Numbering* numberOwnedNodes(
apf/apfVtk.cc:  Numbering* n = numberOverlapNodes(m,"apf_vtk_number");
apf/apfVtk.cc:  Numbering* n = numberOverlapNodes(m,"apf_vtk_number");
dsp/dspSmoothers.cc:    apf::Numbering* numbers = numberOwnedDimension(m, "my_numbering", 0);
ma/maTetrahedronize.cc:    apf::numberOwnedDimension(m, "layer_base_number", 0);
omega_h/apfOmega_h.cc:      apf::numberOwnedDimension(mesh_apf, "smb2osh_global", dim));
omega_h/apfOmega_h.cc:  auto vert_nums = apf::numberOverlapDimension(am, "apf2osh", 0);
phasta/phOutput.cc:    apf::Numbering* en = apf::numberOverlapDimension(o.mesh, "ph::getEdges", 1);
phasta/phOutput.cc:  apf::Numbering* n = apf::numberOverlapNodes(mesh, "ph_local");
pumi/pumi_field.cc:  return numberOverlapNodes(m, name, shape);
pumi/pumi_field.cc:  return numberOwnedDimension(m, name, dim);
pumi/pumi_field.cc:   return numberOwnedNodes(m, name, shape);
stk/apfMeshSTK.cc:      numberOwnedNodes(m, "stk_node"));
stk/apfMeshSTK.cc:      numberOwnedDimension(m, "stk_side", d - 1));
stk/apfMeshSTK.cc:      numberOwnedDimension(m, "stk_elem", d));
test/convert.cc:  apf::numberOverlapDimension(m, "sim_order", m->getDimension());
test/hierarchic.cc:  numbering = apf::makeGlobal(apf::numberOwnedNodes(mesh, "n", shape));
test/nektar_align.cc:  apf::GlobalNumbering* n = apf::makeGlobal(apf::numberOwnedNodes(m, "nektar_id"));
test/poisson.cc:      numbering = apf::makeGlobal(apf::numberOwnedNodes(mesh, "n", shape));
zoltan/apfZoltanCallbacks.cc:    ln = numberOverlapNodes(zb->mesh, "zoltan_vtx");
zoltan/apfZoltanCallbacks.cc:    gn = makeGlobal(numberOwnedNodes(zb->mesh, "zoltan_vtx"));