Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eval errors make jobset and eval pages very expensive to serve and browse #1362

Open
mweinelt opened this issue Feb 14, 2024 · 5 comments · May be fixed by #1363
Open

Eval errors make jobset and eval pages very expensive to serve and browse #1362

mweinelt opened this issue Feb 14, 2024 · 5 comments · May be fixed by #1363
Assignees
Labels

Comments

@mweinelt
Copy link
Member

mweinelt commented Feb 14, 2024

Describe the bug
Eval errors make up a large part of what hydra has to ship to users when browsing jobsets or evals.

  • The jobset page for nixpkgs/staging-next is 26 MB in size
  • 99.9% of that 26 MB page consists of eval errors

To Reproduce
Steps to reproduce the behavior:

  1. Go to https://hydra.nixos.org/jobset/nixpkgs/staging-next or https://hydra.nixos.org/eval/1804223
  2. Observe the download size
  3. Strip the HTML around the eval errors
  4. Notice it takes up >99.9% of size

Expected behavior
These pages should be quick to load and not create load on the hydra webserver.

Proposal
Create a separate route for requesting eval errors and embed that endpoint via an iframe that gets lazily loaded, when it comes into the viewport.

The eval errors should also be shipped in plaintext and not be passed through the template engine.

Hydra Server:
hydra.nixos.org

Hydra 0.1.20240126.838648c (using nix-2.19.4).

@mweinelt mweinelt added the bug label Feb 14, 2024
@delroth
Copy link
Contributor

delroth commented Feb 14, 2024

FWIW when I checked the SQL query associated with loading that page returned 225MB of data. I'm not sure how it ends up getting condensed to 25MB though, but if I did actually get the 225MB measurement right then it's an even more critical performance issue.

Create a separate route for requesting eval errors and embed that endpoint via an iframe that gets lazily loaded, when it comes into the viewport.

Or just make it a link? I don't think we lose much by not having the errors be inline in the page, tbf. If that simplifies the implementation...

@mweinelt
Copy link
Member Author

Given that browsers do the heavy lifting for us, I think iframes can be the drop-in replacement.

https://web.dev/articles/iframe-lazy-loading

@mweinelt mweinelt changed the title Eval errors make jobset and eval pages very expensive to browse Eval errors make jobset and eval pages very expensive to serve and browse Feb 14, 2024
@samueldr
Copy link
Member

Why even go for an iframe?

Using a discrete page is always the better choice. That is, rather than trying to be "smart" with lazy loading, and/or js, and/or DOM-based tabs.

Though since the expectations vary wildly with tab-based navigation, it would be best if it was a discrete link/action in the blurb above the search input.

@mweinelt
Copy link
Member Author

Why even go for an iframe?

No strong opinion. My impression was that it would be the simplest drop-in replacement.

ajs124 added a commit to ajs124/hydra that referenced this issue Feb 22, 2024
@ajs124 ajs124 linked a pull request Feb 22, 2024 that will close this issue
delroth pushed a commit to delroth/hydra that referenced this issue Feb 22, 2024
Ma27 pushed a commit to Ma27/hydra that referenced this issue Mar 13, 2024
Closes NixOS#1362

(cherry picked from commit b45f473)
mweinelt pushed a commit that referenced this issue Mar 15, 2024
mweinelt pushed a commit that referenced this issue Mar 16, 2024
@mweinelt mweinelt moved this from Todo to In Progress in Infra operational issues Mar 18, 2024
@delroth delroth self-assigned this Apr 12, 2024
@delroth
Copy link
Contributor

delroth commented Apr 12, 2024

I've done another pass in 6189ba9 this time actually removing DB fetches when possible. h.n.o loads considerably faster. There are probably some other places where errormsg is being fetched right now, so I'll leave this open for a bit, but I think we might end up just closing the issue if the remaining places don't cause issues.

zowoq pushed a commit to qowoz/hydra that referenced this issue Aug 7, 2024
mweinelt pushed a commit that referenced this issue Oct 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

3 participants