No retry for big report (>100Mb) or limited bandwith (>20sec)

[Wazabiii]

Describe the bug
When we download a large report (with codecovcli), the worker indicates that the download failed because the raw report has not finished downloading.

codecovcli finished with success and we can find the raw report in minio
the process finish (Finished processing upload) without retrying based on MAX_RETRIES for the raw report.

In our case the report is ~200 Mb and the upload finished in ~30sec (so just after the first retry and end of the analysis process)

2025-02-04T10:29:15.251402808Z {"message": "Task app.tasks.upload.Upload[bdcf91de-4db7-4bfd-8863-0f77074d8f33] succeeded in 0.21610997524112463s: {'was_setup': False, 'was_updated': False}", "asctime": "2025-02-04 10:29:15,251", "name": "celery.app.trace", "levelname": "INFO", "lineno": 131, "pathname": "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", "funcName": "info", [...]
2025-02-04T10:29:15.272782515Z {"message": "Parsing the raw report from storage", "asctime": "2025-02-04 10:29:15,272", "name": "services.report", "levelname": "INFO", "lineno": 524, "pathname": "/worker/services/report/__init__.py", "funcName": "parse_raw_report_from_storage", [...]

https://github.com/codecov/worker/blob/main/services/report/__init__.py#L590 with is_retryable=True,
2025-02-04T10:29:15.278796104Z {"message": "Raw report file was not found", "asctime": "2025-02-04 10:29:15,278", "name": "services.report", "levelname": "INFO", "lineno": 599, "pathname": "/worker/services/report/__init__.py", "funcName": "build_report_from_raw_content", [...]

https://github.com/codecov/worker/blob/main/tasks/upload_processor.py#L66
2025-02-04T10:29:15.278819702Z {"message": "Scheduling a retry due to retryable error", "asctime": "2025-02-04 10:29:15,278", "name": "tasks.upload_processor", "levelname": "INFO", "lineno": 69, "pathname": "/worker/tasks/upload_processor.py", "funcName": "on_processing_error", [...]
2025-02-04T10:29:15.284786736Z {"message": "Task app.tasks.upload.UploadProcessor[da836cb2-7033-484a-991b-eb96923bc4b1] received", "asctime": "2025-02-04 10:29:15,284", "name": "celery.worker.strategy", "levelname": "INFO", "lineno": 161, "pathname": "/usr/local/lib/python3.13/site-packages/celery/worker/strategy.py", "funcName": "task_message_handler", [...]
2025-02-04T10:29:15.292041479Z {"message": "Task app.tasks.upload.UploadProcessor[da836cb2-7033-484a-991b-eb96923bc4b1] retry: Retry in 20s", "asctime": "2025-02-04 10:29:15,291", "name": "celery.app.trace", "levelname": "INFO", "lineno": 131, "pathname": "/usr/local/lib/python3.13/site-packages/celery/app/trace.py", "funcName": "info", [...]

+ 20 sec https://github.com/codecov/worker/blob/main/tasks/upload_processor.py#L16

2025-02-04T10:29:35.286932080Z {"message": "Received upload processor task", "asctime": "2025-02-04 10:29:35,286", "name": "tasks.upload_processor", "levelname": "INFO", "lineno": 61, "pathname": "/worker/tasks/upload_processor.py", "funcName": "run_impl", [...]
2025-02-04T10:29:35.317035932Z {"message": "Parsing the raw report from storage", "asctime": "2025-02-04 10:29:35,316", "name": "services.report", "levelname": "INFO", "lineno": 524, "pathname": "/worker/services/report/__init__.py", "funcName": "parse_raw_report_from_storage", [...]
2025-02-04T10:29:35.323623504Z {"message": "Raw report file was not found", "asctime": "2025-02-04 10:29:35,323", "name": "services.report", "levelname": "INFO", "lineno": 599, "pathname": "/worker/services/report/__init__.py", "funcName": "build_report_from_raw_content", [...]

no retry based on MAX_RETRIES? https://github.com/codecov/worker/blob/main/tasks/upload_processor.py#L15

2025-02-04T10:29:35.323662122Z {"message": "Finished processing upload", "asctime": "2025-02-04 10:29:35,323", "name": "services.processing.processing", "levelname": "INFO", "lineno": 66, "pathname": "/worker/services/processing/processing.py", "funcName": "process_upload", [...]
2025-02-04T10:29:35.346776092Z {"message": "Task app.tasks.upload.UploadFinisher[d34c7b52-df1f-4992-be74-71f46ea871a6] received", "asctime": "2025-02-04 10:29:35,346", "name": "celery.worker.strategy", "levelname": "INFO", "lineno": 161, "pathname": "/usr/local/lib/python3.13/site-packages/celery/worker/strategy.py", "funcName": "task_message_handler", [...]
2025-02-04T10:29:35.366633358Z {"message": "Received upload_finisher task", "asctime": "2025-02-04 10:29:35,366", "name": "tasks.upload_finisher", "levelname": "INFO", "lineno": 88, "pathname": "/worker/tasks/upload_finisher.py", "funcName": "run_impl", [...]
2025-02-04T10:29:35.374741365Z {"message": "Getting owner's GitHub Apps info", "asctime": "2025-02-04 10:29:35,374", "name": "shared.bots.github_apps", "levelname": "INFO", "lineno": 278, "pathname": "/usr/local/lib/python3.13/site-packages/shared/bots/github_apps.py", "funcName": "get_github_app_info_for_owner", [...]

Environment (please complete the following information):

• codecovcli, version 9.1.1
• codecov, Self Hosted version 25.1.16

To Reproduce
Upload big report (or limited bandwidth) with upload time > 20 sec

Expected behavior
A retry based on the MAX_RETRIES constant?
A way to customise FIRST_RETRY_DELAY in the config like get_config("setup", "tasks", "upload", "retry_delay", default=20)