Color operations are undefined

[lassade]

Operations like add, mul and lerp for Color in bevy 0.5 can't be properly defined.

Here is the problem:

let a = Color::sRGB { ... };
let b = Color::HSL {  };

// In what 'color space' the result should be in?
let c = a + b;

// Now, in what 'color space' the lerp should happen? (Each one will yield different results)
let c = Color::lerp(a, b, 0.5);

fn do_something(d: Color) -> Color {
    // `v` will assume different meanings depending on the input,
    // `Vec4` or `[f32; 4]` shouldn't have defined operations with `Color`.
    // `Color` can only mix with another `Color`
    let v = Vec4::new(...)
    d + v
}

Gradient and Curve<Color>, can't mix and match keyframes in different color spaces because lerp operation doesn't know in witch space they should happen and will need

How I think this should be:

let a = sRGB(...);
let b = HSL(...);

// Lerp happens in HSL space but result is stored as sRGB
let c: sRGB = HSL::lerp(a.into(), b, 0.5).into();

Using something like Curve<OkLAB> is much more telling and predictable;

Further more

1. We care about the color space because of the operations inside each color space yield different results (that artistically impacts our game); At the end of the day every color will converted down to color space used by the renderer which is in our case is the linear RGB color space, so why not store it that way before sending to the GPU.

2. Have the ability of keep colors in HSL adds little value. Plus its possible to easely create a variant ColorHSVMaterial at the user level, it could use the same shaders, and should be a plug-in in an external crate.

3. Color operations are now way more expensive that they should be, match is a language construct and is not for free (can't be always complied away);

4. A gradient in any color space can be converted down to a sRGB texture or even other gradient and sampled from that;

[mockersf]

Currently, all operations happen in the space of the left hand side color

Some thoughts:

• We should use something like https://crates.io/crates/palette for colors Use palette crate for color handeling (color spaces and conversions) #1146 Add a Color lerp() method #1402 Feature request: Use a fancy color library like Palette. #1797
• On lerping colors, it doesn't really work to do it on cpu if the color will be used in a ColorMaterial or another render ressource. Well at least I tried in Bevy 0.3 and it slowed down Bevy a lot. Should we do it in a shader?
• With Enum variant types rust-lang/rfcs#2593, an enum based api for color could be very nice

[lassade]

1. Agreed;
2. I have (somewhere) a working example for the bevy 0.4 for a color animation on some StadandMaterial, cpu sided no perf hit;
3. Yes it will

[lassade]

all operations happen in the space of the left hand side color

I don't think this is a very good idea, because it can lead to unexpected results (like operating in a different color space than expected).

Plus will introduce hidden costs. To put in perspective things like converting a usize into a f32 have a measurable cost in the animation system in the us scale. I now appreciate having to put as on everything;

[kabergstrom]

My opinion is that color spaces should not be types, they should be data.

I wrote the following in Discord, which should be somewhat relevant but also slightly off-topic.

...

I've been diving deep into color management for game engines recently, and more specifically on how to handle wide-color gamuts and HDR.
I wrote a small library which defines a framework for doing conversions between any 3-component color spaces: https://docs.rs/kolor/0.1.1/kolor/
kolor only depends on glam and has a core API surface of 3 types.
Color in kolor is just (Vec3, ColorSpace), where ColorSpace is 3 bytes. But you can define your own color types too, in which case you can use the ColorConversion type to define a conversion from one space to another that can be applied to 3-component vectors.
Conversions between linear color spaces (like linear sRGB to ACEScg) is a single 3x3 matrix multiply. non-linear spaces obviously need their transform function (like gamma compensation) applied as well.

kolor examples:

let srgb = Color::new(r, g, b, spaces::SRGB);
let oklab = srgb.to(spaces::OKLAB);
let srgb_again = oklab.to(spaces::SRGB);

if you want to use your own storage type:

let conversion = ColorConversion::new(spaces::SRGB, spaces::OKLAB);
...
conversion.apply(&mut my_vec3);

It's still a bit of a draft crate but I think it already makes colors a lot easier than the alternatives in Rust. When you treat colors as 3-component coordinates instead of some typed structure, add/mul/lerp in a single color space are obvious, and kolor makes it trivial to convert one of the operands to the other's color space if necessary.

When it comes to lerp of colors (gradients), you usually want to do it in a perceptual color space like Oklab. The great Ralph Levien has a nice interactive lerp visualization on his blog: https://raphlinus.github.io/color/2021/01/18/oklab-critique.html

On another note:
I think an exciting opportunity is for bevy to support rendering in a color gamut that is wider than sRGB, as contemporary monitors are already able to display a wider gamut. Other than just having the scene-referred color space being able to work with more saturated colors, rendering in a different linear color space could help alleviate certain artifacts that happen with colored light in sRGB renderers: https://twitter.com/Mirko_Salm/status/1259230116737138688

My opinion for how to deal with color management in an engine is that the engine should define a "RenderRgb" or "LinearRgb" (and *Rgba) that stores colors in the engine's linear rendering color space (linear sRGB right now, I'd propose P3 for WCG rendering). All colors in assets should be converted to the renderer's color space on import (even if sRGB gamma compensation is applied to textures after import, it'll end up correct when sampling), and all other color inputs should use proper color management such that when they end up on the GPU, they're in the correct space. The last part is fairly trivial if you use something like the Color type in kolor.

[kabergstrom]

I see the sentiment, and I recognize that typed color spaces could make sense if a lot of code required a specific color space to work. From what I've seen, most code that deals with colors doesn't need to care about the color space though, so I'd avoid using typed color spaces in most of the codebase. Let's say you implement simple sRGB lerp now for animations. Lots of user code (including UI and UI frameworks) will be written to store colors specifically in the sRGB type, and things work and are fast. But if you want to change the lerp to use a perceptual color space, all user code will have to change types.

In my experience, colors are authored, possibly interpolated, then uploaded to GPU. Which color operations do you think will be performed on the CPU?

[kabergstrom]

When it comes to looking up a conversion, in kolor I pre-generate the conversion matrices between named linear color spaces, and I could implement support for perfect hashing of the lookup to improve the performance of creating a ColorConversion.

[lassade]

Bevy pbr renderer should always operate in the linear color space. So it seams right that all colors given to it should be already the linear color space.

Let's say for example I create a StandardMaterial which has 2 colors: base_color and emission. Setting both of theses colors to a space different from the renders linear color space will trigger an color conversion (even if it's on the same color space it will have to at least check that) it's fair to pay for a single conversion cost, but now if I change any other parameter let's say roughness these checks and conversions will have to run again every single time, that's a hidden cost that few people will be aware of.

You can make this conversion really fast but it's always cheaper if we simple have keep everything in the linear space.

[lassade]

I think using different color spaces for things like gradients are a great idea and that flexibility should be given, but we should always encourage preprocessing (give memory gain cycles), like every gradient can be made a into a texture, every animation can be re-sampled and these should can work in a linear color space just like the renderer expects.

[lassade]

I also think #1572 dont really addresses #1193 or anything, for me was a simple case of just having to convert the color consts into a linear color space;

We don't need to keep colors around if it's in a different color space what the renderer expects, but again someone might want to do that so they could use a dynamic color space using kolor I think that is a good idea especially for gradient, or using another crate like palette if they really need a specific color space like HSV.