Correctness Checking

[juso40]

What are the values I should aim for to validate my runs?

I cannot recreate any of the 'Final Origin Energy' values of 4.3 in:
https://asc.llnl.gov/sites/asc/files/2021-01/lulesh2.0_changes1.pdf

Tried running with ./lulesh2.0 -s 5 -i 72
My output:

Running problem size 5^3 per domain until completion
Num processors: 8
Total number of elements: 1000 

To run other sizes, use -s <integer>.
To run a fixed number of iterations, use -i <integer>.
To run a more or less balanced region set, use -b <integer>.
To change the relative costs of regions, use -c <integer>.
To print out progress, use -p
To write an output file for VisIt, use -v
See help (-h) for more options

Run completed:
   Problem size        =  5
   MPI tasks           =  8
   Iteration count     =  72
   Final Origin Energy =  6.232687e+04
   Testing Plane 0 of Energy Array on rank 0:
        MaxAbsDiff   = 4.547474e-12
        TotalAbsDiff = 1.360249e-11
        MaxRelDiff   = 4.855244e-15

Elapsed time         =       0.01 (s)
Grind time (us/z/c)  =  1.1185698 (per dom)  (0.010067128 overall)
FOM                  =  7151.9901 (z/s)

All tests were done using the version on the main branch.

[juso40]

After deciding to give OpenMP a try, I actually got some usefull results.
So instead of:

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=g++-10 -DWITH_MPI=On -DWITH_OPENMP=Off -DCMAKE_CXX_STANDARD=20 

I now build LULESH with:

cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=g++-10 -DWITH_MPI=Off -DWITH_OPENMP=On -DCMAKE_CXX_STANDARD=20 

And this way, I actually receive the same 'Final Origin Energy' values of the lulesh2.0_changes.pdf mentioned above.

So either something with my build/run process for MPI is wrong or LULESH has some wrong rank logic, I assume.

But I still have the general question of validating results as a mean of testing various real types. The paper https://www.osti.gov/servlets/purl/1117905 mentions the type Real_t:

Can be defined as float, double, long double, or any other numeric type suitable for real
number arithmetic. Furthermore, since LULESH has been written in C++, LULESH can be
extended to support user defined types for fixed-point or interval arithmetic if those features are
not natively supported in the hardware, compiler, or runtime system.

How would I validate/compare my different types, the paper lulesh2.0_changes.pdf generally just has values for double.
I guess the smaller MaxAbsDiff, TotalAbsDiff and MaxRelDiff are the more accurate my real type is.

The three measures of symmetry should all be 1e−8 or smaller in double precision for 1003 mesh discretizations of smaller.
https://asc.llnl.gov/sites/asc/files/2021-01/lulesh_ports1.pdf

But that would mean floats simply are not accurate enough to even test this. Is this assumption correct? Since floats have a maximum precision of up to 1e-7

[cadivus]

You have to use the table from the Lulesh 2.0 documentation when using LULESH 2.0 (https://asc.llnl.gov/sites/asc/files/2021-01/lulesh2.0_changes1.pdf).
It says:

Due to changes in how the initial timestep is set and the initial energy is deposited the correctness values in are no longer correct for LULESH 2.0.

You also have to use Total number of elements from the terminal output for the lookup in the table (size column). Since your using 8 MPI slots, the number of elements is not 5³, but 8 * 5³.

I get the result for 10³ using 8 MPI slots and s=5:

$ mpirun -n 8 lulesh2.0 -i 231 -s 5
Running problem size 5^3 per domain until completion
Num processors: 8
Total number of elements: 1000

To run other sizes, use -s <integer>.
To run a fixed number of iterations, use -i <integer>.
To run a more or less balanced region set, use -b <integer>.
To change the relative costs of regions, use -c <integer>.
To print out progress, use -p
To write an output file for VisIt, use -v
See help (-h) for more options

Run completed:  
   Problem size        =  5 
   MPI tasks           =  8 
   Iteration count     =  231 
   Final Origin Energy = 2.720531e+04 
   Testing Plane 0 of Energy Array on rank 0:
        MaxAbsDiff   = 8.185452e-12
        TotalAbsDiff = 1.512035e-11
        MaxRelDiff   = 2.520539e-15


Elapsed time         =       0.04 (s)
Grind time (us/z/c)  =  1.4438799 (per dom)  (0.18048499 overall)
FOM                  =  5540.6269 (z/s)