UC: Manipulative Benchmark Optimization

[jawache]

Optimize the software so that it detects it's running in the context of a benchmark, adjust it's method of functioning so it performs optimally in those conditions but performs worse in real-life conditions.

How this might be reflected in an LLM environment would be LLMs optimized to return lower quality results (but therefore faster, more energy efficient) when it detects it's being run with a benchmark of prompts, however prompts outside those known to be used in a benchmark return higher quality (more energy intensive) results.

Counter

• Usage based measurements rather than benchmark based measurements. For example rather than measure based of a benchmark of prompts, measure the emissions of all the entire infrastructure of an LLM Software (SaaS/Application) every day and then divide by the total number of prompts user made during that day. By using actual user data we get a more realistic figure and it more directly useful to the end user and also it's not something that is easy to manipulate.