Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Downgrade to Julia 1.11.1 #2178

Merged
merged 1 commit into from
Dec 3, 2024
Merged

Conversation

mathbunnyru
Copy link
Member

Describe your changes

Julia 1.11.2 has just been released and it seems that it causes our self-hosted aarch64 runners to hang.

Previous build was using 1.11.1 and has been working fine: https://github.com/jupyter/docker-stacks/actions/runs/12123374006/job/33801780215

Let's try to downgrade to 1.11.1 and see if it helps.

Issue ticket if applicable

Checklist (especially for first-time contributors)

  • I have performed a self-review of my code
  • If it is a core feature, I have added thorough tests
  • I will try not to use force-push to make the review process easier for reviewers
  • I have updated the documentation for significant changes

@benz0li
Copy link
Contributor

benz0li commented Dec 3, 2024

Julia 1.11.2 has just been released and it seems that it causes our self-hosted aarch64 runners to hang.

@mathbunnyru Let us see, if Julia 1.11.2 causes the same problem with b-data's/my GitLab runners.

@mathbunnyru
Copy link
Member Author

mathbunnyru commented Dec 3, 2024

Although we merged mamba v2, I don't think it's related, because mamba v2 was building just fine in the branch.

Julia v1.11.2 has just been released, so I suppose it can be a problem somehow.
https://github.com/JuliaLang/julia/releases/tag/v1.11.2

@mathbunnyru
Copy link
Member Author

Unfortunately, GitHub Runner seems to lose the logs, crash the host machine, so it's not easy to debug.
I was able to take a look at the logs online:

Screenshot 2024-12-03 at 16 34 12

Note that for 7-8 minutes there were no logs.
In v1.11.1 it started outputting some logs after 2.5 minutes: https://github.com/jupyter/docker-stacks/actions/runs/12123374006/job/33801780215#step:8:65

@mathbunnyru
Copy link
Member Author

mathbunnyru commented Dec 3, 2024

@benz0li do you have something similar to our setup-julia-packages.bash? If not, it is likely to work for you.
It hangs (or crashes, or something) inside this script.

@benz0li
Copy link
Contributor

benz0li commented Dec 3, 2024

@benz0li do you have something similar to our setup-julia-packages.bash?

Yes. In the Dockerfile: https://github.com/b-data/jupyterlab-julia-docker-stack/blob/a8b728c4e70ae460072cc22197bf632e5954197d/base/latest.Dockerfile#L307-L344

@mathbunnyru
Copy link
Member Author

v1.11.1 builds just fine for julia-notebook: https://github.com/jupyter/docker-stacks/actions/runs/12144295877/job/33863916105

@mathbunnyru
Copy link
Member Author

@mathbunnyru
Copy link
Member Author

This seems to fix the builds, so I'm merging this. When I have time, I will report the problem to the upstream.

@mathbunnyru mathbunnyru merged commit 7bdb19c into jupyter:main Dec 3, 2024
73 checks passed
@benz0li
Copy link
Contributor

benz0li commented Dec 3, 2024

Julia 1.11.2 has just been released and it seems that it causes our self-hosted aarch64 runners to hang.

@mathbunnyru Let us see, if Julia 1.11.2 causes the same problem with b-data's/my GitLab runners.

No problem with b-data's/my GitLab runners: See https://gitlab.b-data.ch/jupyterlab/julia/base/-/jobs/135429

@mathbunnyru
Copy link
Member Author

mathbunnyru commented Dec 3, 2024

Julia 1.11.2 has just been released and it seems that it causes our self-hosted aarch64 runners to hang.

@mathbunnyru Let us see, if Julia 1.11.2 causes the same problem with b-data's/my GitLab runners.

No problem with b-data's/my GitLab runners: See https://gitlab.b-data.ch/jupyterlab/julia/base/-/jobs/135429

It seems that you're using Apple Silicon M2 Max, 3.5 GHz, 8 cores, 48 GB RAM - very powerful machine.
We're using t2a-standard-1 machine (https://cloud.google.com/compute/docs/general-purpose-machines#t2a_machine_types), which is an order of magnitude worse 😃
I think Julia started using too much resources, and now the VM is hanging.

It might still be a performance bug in a new release.

@giordano
Copy link

giordano commented Jan 8, 2025

Is there any way to reproduce the hang you observed on a local machine? Or at least, a clear indication of what's the workload you're running.

@mathbunnyru
Copy link
Member Author

@giordano I created an issue, let's continue the discussion there: #2198

I also added the answer to your question

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants