Performance notes of PySM3

[zonca]

Running PySM3 standalone using modifications of this script:

https://github.com/healpy/pysm/blob/master/mpi_examples/pysm3_mpi_so_pysm_models.py

I have all the 8 components requested for the TOD sims, which load a total of 30 maps. Plus bandpass integration of 5 points each for 10 channels and distributed smoothing with libsharp.

[zonca]

It runs fine on Popeye single node with 1 MPI process (48 cores and 768 GB of RAM) at 4096 , using a max of 25GB of RAM in 6 minutes.

[zonca]

Timing, initialize sky and run bandpass integration and beam smoothing for 10 channels:

• 1 node 1862.6s 23.882 GB
• 2 nodes 1399.8s 15.213 GB
• 3 nodes 1169.1s 12.470 GB
• 5 nodes 978.4s 10.193 GB
• 10 nodes (4 MPI 12 OMP per node) 702.5s 10.319 GB

Checked for increased memory between channel 0 and 9, seems fine.

[zonca]

Only thing that worries me is that at 10 MPI processes sometimes the MPI job hangs after creating Sky, I guess in one of the all-to-all communications....how do I debug that??
Testing on Cori KNL now.

[zonca]

Breakdown for 10 nodes

10 nodes (4 MPI 12 OMP per node)

• Initialization: 111.3s
• Create sky and bandpass integration (each channel): 50s
• Distributed smoothing (each channel): 6-10s