Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Microsoft.NET.Build.Containers.IntegrationTests are pulling from Docker Hub #42026

Open
akoeplinger opened this issue Jul 8, 2024 · 9 comments
Assignees
Labels
Area-Containers Related to dotnet SDK containers functionality Area-Infrastructure Known Build Error untriaged Request triage from a team member
Milestone

Comments

@akoeplinger
Copy link
Member

akoeplinger commented Jul 8, 2024

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=733130
Build error leg or test failing: Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.WriteToPrivateBasicRegistry
Pull request: https://github.com/dotnet/sdk.git/pull/42019

We need to stop pulling from upstream Docker Hub since we're hitting rate limits.

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "You have reached your pull rate limit",
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=733130
Error message validated: [You have reached your pull rate limit]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 7/8/2024 7:20:41 PM UTC

Report

Build Definition Test Pull Request
936288 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46254
935366 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46429
936159 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46371
936153 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46281
936140 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #44953
936125 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46327
935929 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46353
935565 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46433
935902 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46441
935782 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46439
935623 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46431
935722 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46370
935828 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46261
935795 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46195
935787 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46356
935738 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46361
935687 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46353
935622 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46266
935616 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46435
935378 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46430
935549 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46428
934380 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46409
935525 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46408
934464 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46358
935373 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #44843
934459 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46417
934441 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46414
935118 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46313
935357 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46411
935124 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46387
934463 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46266
935035 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46370
934775 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46281
934766 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46327
934667 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46419
934531 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46290
934424 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46356
934417 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46410
934374 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46336
934486 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46261
934448 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.MultiArchStillAllowsSingleRID #46416
934469 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46319
934328 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46407
934435 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #46415
934431 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.EndToEndTests.ApiEndToEndWithLocalLoad #46413
934379 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46315
934416 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46411
934420 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46412
933380 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46361
932961 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46358
932962 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46319
933991 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46366
933804 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46346
933925 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46377
933902 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46394
933866 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46254
933593 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46392
933816 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46368
933795 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46401
933789 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46399
933506 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46390
933154 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46371
933389 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #44953
933383 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46290
933367 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46389
932957 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46266
932869 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46315
932948 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46082
932952 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46318
932932 dotnet/sdk Microsoft.NET.Build.Containers.Tasks.IntegrationTests.CreateImageIndexTests.CreateImageIndex_Baseline #46119
932786 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46281
932941 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46386
932914 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #46115
932954 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46387
932907 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.DockerRegistryTests.GetFromRegistry #46114
932901 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46356
932894 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46385
932530 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46377
932176 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46366
932065 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46261
932035 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46341
931939 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46333
931899 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46361
931902 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46290
931889 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46327
931890 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46327
931683 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46281
931679 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46336
931638 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46341
931545 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46318
931540 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46357
931518 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #46355
931514 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46354
931601 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46350
931583 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.3.WorkItemExecution #45384
931589 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46281
931544 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46358
931543 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46195
931533 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46306
931522 dotnet/sdk Microsoft.NET.Build.Containers.IntegrationTests.dll.1.WorkItemExecution #46356
Displaying 100 of 247 results

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 7 247
@akoeplinger
Copy link
Member Author

@marcpopMSFT who owns Microsoft.NET.Build.Containers?

@baronfel
Copy link
Member

baronfel commented Jul 8, 2024

@dotnet/sdk-container-builds-maintainers

@baronfel
Copy link
Member

baronfel commented Jul 8, 2024

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@nagilson
Copy link
Member

Bumping this thread since it's affecting more PRs

@nagilson
Copy link
Member

We don't have a path forward to fix this yet - Aspire moved to using their own ACR for storing the images they use, but we need an engineering-systems-managed solution for our entire stack. There's an issue tracking this request but I can't find it at the moment.

@baronfel
Copy link
Member

Could be fixed at a root cause level by dotnet/dnceng#3389

@MichaelSimons
Copy link
Member

@akoeplinger the image in question is registry:2, which we need/use to validate our logic for communicating with container registries via the container registry APIs. Do you know if this image is mirrored anywhere that we can pull from? It's not currently on mcr.microsoft.com.

@baronfel - It sounds like the test has this dependency and is what is pulling the image. One possible option to workaround scenarios where tests are pulling images directly from DockerHub would be to create a wrapper buildtools image. The Dockerfile would just be a FROM statement. The buildtools images are stored on MCR and therefore you would be able to get around this rate limiting issue.

@nagilson
Copy link
Member

@dotnet/sdk-container-builds-maintainers Do you agree with @MichaelSimons suggestion? This has a pretty large impact on PR failure rates. If this is still an issue going forward a month or two down the line, I will sadly suggest that we just disable these tests

@marcpopMSFT
Copy link
Member

@donJoseLuis @MichalPavlik I'm told by baronfel that there may be a solution to this of setting up our own ACR mirror of these so we don't get blocked by docker from pulling too often. He said to talk to @eerhardt about that. Is that something you can potentially drive getting set up?

As you can see above, this fails a couple of dozen PRs every week that have to be rerun. It's not a crazy blocker but that's high enough to be something we want eyes on to get fixed eventually.

@marcpopMSFT marcpopMSFT added this to the 10.0.1xx milestone Dec 18, 2024
@marcpopMSFT marcpopMSFT added the Area-Containers Related to dotnet SDK containers functionality label Dec 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Area-Containers Related to dotnet SDK containers functionality Area-Infrastructure Known Build Error untriaged Request triage from a team member
Projects
None yet
Development

No branches or pull requests

6 participants