GitHub API rate limits are too often reached causing end to end tests to fail

[mgoerens]

It is allowed to do 5000 calls to the GitHub API per hour, see this doc. The openshift-helm-charts-bot regularly hits the maximum, causing end to end tests to fail.

This typically occurs when opening multiple PRs as multiple pipeline are ran in parallel. Concurrency (currently set to 2) of tests within a pipeline also puts additional pressure on the API usage of the bot.

[mgoerens]

Influence of polling retry on a single test

retry = 2000ms

When running a single feature (HC-07) with smoke tagging, two tests are run sequentially, creating two PRs in the sandbox repo. Here is the evolution of the API usage of the bot:

A total of 180 API calls are consumed. We notice a constant usage throughout the tests, due to polling the GH API and checking on the sandbox PR status. In addition there is a higher usage at the beginning and at the end of each tests (2 tests here) for the creation/deletion of the branches and the creation/commenting/close of the PRs.

retry = 10000ms

When increasing the polling timeout from 2000ms to 10000ms we get the following result:

A total of 82 api calls are consumed. We can notice that the polling is less resource hungry and we can better see the api calls consumed by initialization and cleanup.

Comparison

First scenario in Blue, second in orange

[mgoerens]

Experiment on timeouts on complete suite of smoke tests

Retry=2000ms; concurrency=2

When running the full end to end tests with smoke tagging with timeout set to 2000ms

A Total of 3022 API calls are consumed.

Retry=10000ms; concurrency=2

When running the full end to end tests with smoke tagging with timeout set to 10000ms:

A total of 1188 API calls are consumed

Merged plots:

• 1st scenario is in blue
• 2nd scenario is in orange

[mgoerens]

Experiment with concurrency

Given a shorter retry timeout, how does API usage looks like when we increase concurrency.

• Scenario 1 in Blue: Retry = 10000ms; concurrency = 2
• Scenario 2 in Orange: Retry = 10000ms; concurrency = 5

Unsurprisingly total api usage is similar, but concurrency=5 completes much faster

All scenarios previous scenarios compared

• Scenario 1 in Blue: Retry = 2000; concurrency = 2
• Scenario 2 in Orange: Retry = 10000ms; concurrency = 2
• Scenario 2 in Green: Retry = 10000ms; concurrency = 5

[mgoerens]

Testing how rate limits reprovisioning works

When does the bot gets its api calls back ? Let's wait until we're back to 5000

• Scenario 1: retry=10000ms; concurrency=5; 1 smoke tests
• Scenario 2: retry=10000ms; concurrency=5; 2 smoke tests in sequence

Conclusion: after 1 hour after the first API calls, budget is given back.