SCS Saw-Tooth Behavior on Pseudo-push

[MatthewChristoff]

We've noticed some strange behavior for the pseudo-push operation on our performance tests. The code for this test is located here.

On multiple tests we've seen some seemingly random patterns in the times for this operation despite averaging a hundred iterations. Further, these patterns are not consistent between runs.

Sparse

Dense

At first, an idea was that this is from some natural hardware variation because these operations took on the order of 10^-2 seconds. We attempted to increase the time of pseudo-push by adding an internal for loop, but the same issues were seen then.

[MatthewChristoff]

@diamog Do you have any ideas about why this could be / what we should do from here?

[cwsmith]

@diamog and I discussed this and have a few ideas. In SCS with full sorting enabled the array of mesh element data is not in the same order as the rows of the SCS.

pumi-pic/performance_tests/ps_combo160.cpp

Line 137 in acf7a6a

Kokkos::View<double*> parentElmData("parentElmData", ptcls->nElems());

For CabM and CSR the mesh element data is in the same order as the 'rows' in the structure. We are wondering if this ordering difference could be causing the jagged SCS performance. Towards this we can try the following:

• run SCS without sorting, or
• sort the element data array in descending order of particle density to match the SCS row ordering.

If these tests show no significant improvement, we may be seeing an SCS sensitivity to the particle distribution variation. To check to see if this is the case we could run SCS pseudo push over several different instances of a Gaussian distribution with the same average particle density per element and total number of elements.