Error related to parallelism when running snap.tl.macs3 when running on the cluster

[PauBadiaM]

Recently I had a weird issue when running snap.tl.macs3 on a cluster, my script would get stuck for a while and then throw an error related to parallelism issues:

0%|          | 0/12 [00:19<?, ?it/s]
Traceback (most recent call last):
  File "/mnt/sds-hd/sd22b002/projects/GRETA/greta_benchmark/callpeaks.py", line 18, in <module>
    snap.tl.macs3(adata, groupby='cell_type', n_jobs=n_jobs, tempdir=tempdir)
  File "/opt/conda/envs/env/lib/python3.10/site-packages/snapatac2/tools/_call_peaks.py", line 155, in macs3
    peaks = _par_map(_call_peaks, [(x,) for x in fragments.values()], n_jobs)
  File "/opt/conda/envs/env/lib/python3.10/site-packages/snapatac2/tools/_call_peaks.py", line 221, in _par_map
    raise RuntimeError("Some worker process has died unexpectedly.")
RuntimeError: Some worker process has died unexpectedly.

Despite this, the same script would work in my local machine.

In case someone has the same issue, I've found that the solution is to wrap the call to macs3 with the __main__ conditional statement. Here is an example:

import snapatac2 as snap

if __name__ == '__main__':
  # Read data
  adata = snap.read(snap.datasets.pbmc5k(type='annotated_h5ad'), backed=None)
  
  # Subset to make things faster
  msk = adata.obs.groupby('cell_type', observed=False).head(50).index
  adata = adata[msk, :].copy()
  
  # Call ATAC-seq peaks using MACS
  snap.tl.macs3(adata, groupby='cell_type', n_jobs=8)
  
  print('Done!')

Maybe this could also be added in the docs.

[kaizhang]

As tl.macs3 uses multiprocess for parallel execution under the hood, it would be wrapped in a if __name__ == '__main__': statement. This constraint may be lifted once "nogil" is merged in Python.

[1010shane]

@PauBadiaM Thank you! Exact same problem, has been driving me nuts. Your solution solved my issue. Previously, I was able to run my pipeline on MacOS, but not on Ubuntu 22.04.