Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate crate_universe bzlmod repositories in parallel #3234

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

UebelAndre
Copy link
Collaborator

@UebelAndre UebelAndre commented Feb 3, 2025

This change aims to improve crate_universe for bzlmod by parallelizing generation into a single module_ctx.execute call to the cargo-bazel binary for each module. This avoids the limitation of running each module serially.

@UebelAndre
Copy link
Collaborator Author

It's interesting that this is roughly as slow as the serial implementation. I suspect the bottleneck is the cargo tree calls which I believe puts a lock on the local registry being used.

Before

signal-2025-02-04-082246_002

After

signal-2025-02-04-082311_002

@UebelAndre
Copy link
Collaborator Author

It's interesting that this is roughly as slow as the serial implementation. I suspect the bottleneck is the cargo tree calls which I believe puts a lock on the local registry being used.

I should point out that this was for a run without REPIN=1. I also misspoke about cargo tree calls, unless splicing is happening (which occurs with REPIN=1), cargo tree should not be called. When all crates are repinning we do see a larger time savings but not as much as I'd expected:

Before

Screenshot 2025-02-04 at 10 27 10 AM

After

Screenshot 2025-02-04 at 10 26 59 AM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant