Discussion about Widgets and Styling

[viridia]

I've been busy prototyping some ideas around widget styling, and I wanted to share some of my thinking. While this is mostly going to be a description of what I'm building, it's also meant to be suggestions for the people thinking about the next steps in the bevy_ui evolution.

Why stylesheets?

One question that might be asked is, why bother with styles as an asset? Why not just define styles in Rust code?

The main advantage of stylesheets is that it allows the development of widget behaviors and widget styles to be done independently. A user can pull in a library of widget behaviors, and a library of styles, whose authors have never met or are even aware of each other.

In the web development world, there is a robust ecosystem of widget libraries such as Material UI, ANT Design, Bootstrap, Semantic UI, and at least three dozen others whose primary value is their aesthetics. The actual behaviors of these libraries are remarkably similar, because they conform to commonly accepted standards of UI design and accessibility. But each has its own distinctive look, which can vary from professional, to friendly, to just plain funky.

In the game world, user interface novelty is a virtue - it's expected each game has widgets that look slightly different. However, this does not mean that the behavior of those widgets should be dramatically different, and especially it does not mean that game creators should be encouraged to violate accessibility guidelines. Particularly for complex widgets such as modal dialog boxes, drop-down menus, radio buttons and others, it can be tricky to get accessibility just right. Game authors should not have to be burdened with the task of re-inventing combo boxes or rich text editors.

Therefore, I believe it should be a goal to encourage an ecosystem where widget authors can publish "white-label" crates which offer robustly engineered widgets with no-built-in styling, and allow other creators to add their own styles on top of those widgets.

Why not CSS?

There are a lot of good things about CSS. However, in recent years the web world has become aware that some aspects of CSS are flawed, in the sense that they increase the project's maintenance burden. For example, the rule-based nature of CSS selectors means it is not possible to write an automated linter that can detect dead CSS code, and I have personally seen how unused CSS can accumulate in a large codebase.

If you take a look at some of the most popular CSS tooling on the web (CSS modules, Web components, CSS-in-JS frameworks and so on), these tools go out of their way to turn certain features of CSS off. Coincidentally, these features (such as the CSS cascade and complex selector matching) are among the most difficult to implement, and have the highest negative impact on performance. So there are multiple reasons to leave these features out from a Bevy implementation.

(Background: I'm somewhat familiar with this area because in 2007 when I worked at Electronic Arts, I wrote an HTML/CSS browser that ran on Windows, XBox and PS/2 and used hardware-accelerated drawing - it was used in Spore.)

The approach that I'm taking in my current work is inspired by the TypeScript library vanilla-extract, which is a compile-time CSS-in-JS framework that lets you define CSS rules in strongly-typed TypeScript. This "language" is isomorphic to CSS, meaning that you have access to all of the familiar CSS properties and values, but the syntax is slightly different.

Vanilla-extract allows CSS selector matching, but in a restricted form. The restriction is that style rules can only be applied to the element they are attached to - you cannot write a generic rule that applies to all children, or to all nodes of a given type. You can, however, write rules that are conditional based on the element classes, as well as the classes of the element's parent. So for example, this syntax is allowed:

.widget.disabled > &

But this syntax is not:

&.widget > .disabled

(& means "current element" and is borrowed from SASS.)

I've written a number of widget libraries using vanilla-extract (one for React and one for Solid) and I've not found this restriction to be terribly burdensome. Also, one nice thing about this restriction is that the matching algorithm can be implemented in Rust/Bevy very efficiently.

I've also decided to use JSON as the host language for specifying styles.

Why JSON?

My first attempt at doing this actually used XML, but I later switched to JSON. The main advantage of JSON over something like RON is interactive schema validation in VSCode. For stylesheets that are to be authored by hand, I want precise and fine-grained validation of every property in the editor.

Another advantage of JSON (compared to regular CSS stylesheets) is that it can be decoded by serde (with some help from willow), which means that it can also be serialized again. This may be important for writing an interactive editor.

The current prototype supports the set of style attributes that bevy_ui supports, such as "background-color", "flex-direction" and so on. It also supports some shortcuts which are directly inspired by vanilla-extract:

• When specifying a length such as 'width' or 'bottom', a bare number is assumed to be in pixel units. So "min-width": "7px" and "min-width": 7 are equivalent.
• "overflow" is a shortcut for "overflow-x" and "overflow-y". There are other shortcuts for margin, padding, flex and so on.
• UiRects properties can specify one to four lengths, e.g. "1px 7px auto 3vmin". The order is the same as CSS (top, right, bottom, left), and they default if fewer numbers are given. (I'm considering also allowing UiRects to be written using array syntax, e.g. "margin": [0, 0, 10, "12px"]).
• Colors can be specified in hex or functional syntax: "#ff0000" or "rgba(255 255 255 / 0.5)".
• CSS variables are supported, but (again, like vanilla-extract) variable definitions cannot be mixed in with regular CSS attributes, but rather are in their own dedicated "vars" section.

However, there are some differences from CSS:

Asset references: Properties such as "background-image" which would normally accept a URL in CSS (via the url() function), instead accept an asset key:

"background-image": "asset(../images/button.png)"

The deserializer recognizes the "asset" syntax and knows to load this as a resource handle. Note that relative paths are supported.

Selectors use the same scheme as vanilla-extract, where the selectors are map keys:

"selectors": {
  "&.hover": {
    "vars": {
      "--button-image": "asset(../images/button-blue-hover.png)"
    }
  },
  "&.pressed": {
    "vars": {
      "--button-image": "asset(../images/button-blue-pressed.png)"
    }
  },
  "&.disabled": {
    "vars": {
      "--button-image": "asset(../images/button-blue-disabled.png)"
    }
  }
}

Note that Bevy widgets don't have built-in dynamic states such as :hover or :focus-within, so any dynamic styles have to be done through regular class names.

Applying Styles

The internal representation of style objects uses a sparse key/value system, so you don't get a giant struct with a bunch of Options, most of which are set to None. During style evaluation, a style asset is "merged" into a ComputedStyle which is just a struct that contains a bunch of Bevy style structs (Style, BackgroundColor, etc.) This is then applied to the widget components using a custom Command.

Selector Matching

The algorithm for selector matching is very performant - it's a simple NFA (Non-deterministic Finite Automata) that takes in an element as its input, and returns true or false if there's a match. The only information it needs is the parent/grandparent chain and the list of classes associated with each element. Unlike regular CSS selectors, there's no need to walk the whole widget tree globally in response to an event.

Evaluating CSS variables.

Individual CSS attributes are parsed into a simple expression language, an AST. For attributes which are constants like "flex-direction": "ltr", the attributes are pre-optimized into their bevy_ui enum equivalents, e.g. bevy::u::Direction::LeftToRight. However, attributes which contain a variable reference are left in AST form and can be evaluated when the widget style is updated.

Another optimization idea I've been considering is to pre-compute a bitmask of which widget style properties are dynamic and which are constant (by looking at the selectors and attributes), and only re-evaluating the dynamic ones in response to an event. However, this may be premature optimization on my part.

Relative asset labels

Imagine a widget such as a slider which has multiple internal parts: a track, a thumb, and so on. Each of these elements needs to have a style, and because we don't allow parents to style their children, each one needs a reference to a style asset.

Unfortunately, we also want the style to be a parameter so that the same slider can be styled differently in different contexts. It would be cumbersome if we had to pass in a separate AssetPath for every slider part.

The solution comes in the form of relative labels. If you have ever worked with json-schema, you'll be familiar with JPointer syntax, which allows relative paths from within a document such as #./abc/def. The # at the beginning tells us this is a fragment identifier, meaning that it's not a filesystem path, but rather a reference to the internal structure of the document that is relative to the current asset key.

We can design our slider to take a "base asset path" as a parameter, and then have the individual parts use relative paths: #./slider-track, #./slider-thumb and so on. So long as our style asset structure conforms to this naming scheme, it will work with any style asset.

Show me the code!

Unfortunately, the current prototype requires Bevy Assets V2 and is still in development. I hope to have something to show in a few weeks.

[rlidwka]

My first attempt at doing this actually used XML, but I later switched to JSON.

Please don't use JSON for anything that a human is supposed to write. That's a machine serialization format, not a source code format. It doesn't have comments, is too verbose, serde round-trip doesn't preserve whitespaces, etc.

TOML or YAML are infinitely better at being configuration files, and both can be serialized/deserialized by serde just as well.

Or just use a subset of CSS for this. Surely, writing CSS parser with serde framework isn't that hard...

[viridia]

One of the nice things about using serde is that you don't really care what the syntax is, so long as you write your serializer in a way that doesn't rely on format-specific features. I'm actually more interested in compressed binary formats like MessagePack or CBOR, which ought to be able to work with the same serializers. No point in shipping all that text...

All that being said, once you have a JSON schema, editing with autocomplete isn't terrible. Not great, but not the worst thing in the world either. Not that different from TypeScript object literals.

XML is great if you have one complex namespace (children) and everything else is a string (attributes). As soon as you depart from those restrictions it gets messy fast.

[rlidwka]

All that being said, once you have a JSON schema, editing with autocomplete isn't terrible. Not great, but not the worst thing in the world either.

You still can't make comments in that JSON. And comments are absolutely required for any config files written by hand to simplify later maintenance.

Well, you can, if you use that weird {"// comment": "hello"} syntax that doesn't get highlighted as comment by github or vscode. Or you can use JSON5, but that's not JSON anymore.

[viridia]

It really depends on what the end-goal is. If the end-goal is Unity, or something like it, then human-editability is not so important (although human-debugability is). If the goal is Unity, then comments have to become part of the schema (a "description" field).

One question I've been asking myself is how important is round-tripping. Take for example color: if the user types rgb(255 255 255) and we serialize that out as #ffffff, is that acceptable? Or do we need to store the original format as part of the color?

At the moment the format is about 95% round-trippable. I could make it 100% with some work, and a bit of extra overhead, but I'm not sure how important of a goal that is.

[rlidwka]

One question I've been asking myself is how important is round-tripping. Take for example color: if the user types rgb(255 255 255) and we serialize that out as #ffffff, is that acceptable?

If user picks it in editor, then yes. If user writes it in VSCode, then no.

You have rgb(255, 255, 255) now? Later it might become lighten(10%, cielab($base-hue, 30, 40)), with helper functions, multiple color spaces and variable substitution. Certainly want to preserve that.

[nicopap]

So I recently worked on a styling lib for a silly experiment trying to recreate the cyberpunk main menu.

This styling module

It's a rust module: https://github.com/nicopap/cuicui_layout/blob/3eea65e7975ac73242e977011696e592ae3e11b9/examples/chirpunk/src/style.rs

It relies on two types style::Bevypunk and Element.

style::Bevypunk

Is a Resource implementing Reflect, which means it is run-time inspectable by third party tools such as bevy-inspector-egui. It's also conceivable to write a bevy AssetLoader that updates the style::Bevypunk on asset loads.

It has hard-coded defaults, but nothing prevents you from loading the style from a serialized file format such as toml, xml or json.

Element

It is a Component enum of all possible kind of stylable entity. I think this corresponds the most to a "html class".

We have a system that checks for update on style::Bevypunk or Element, and updates various components on entities with Element.

That's it really. Styling is entirely encapsulated by this module, you can see only the Element and Plugin types are public. The rest of the code absolutely doesn't care about styling. This way, I have the guarantee that styling is controlled exclusively through that one resource. It makes it very powerful and exhaustive. It's really neat.

Limitations

I've identified a few limitations:

• The pattern of defining a palette, then define a design language on top doesn't really work. I suspect I could use something like reflection paths
• The composition of components for each entity is hardcoded and can't be changed on the fly. But I suspect it is possible to fix this by relying on reflection as well.
• The number of elements and how they relate to style is also hard coded.
• Element only works at the entity level. In this demo, it is fine, because animation works on entities and their children, not just single entities.

Considerations for styling files

JSON schemas are a clever way to work around tooling limitations. But toml and xml-based formats are not hopeless. There are two excellent error reporting crates miette and ariadne. You can't base your design around serde with this, because you need the source code spans. toml-edit and kdl tracks the source file spans though.

An advantage of such an approach is that you don't depend on VSCode, your users can just read the error output of your loader in the console, and fix their style file from that.

Syntax nits

I really dislike the CSS "variable" syntax (ie: prefixing with --) In a lot of languages -- is syntax for comments (similarly to // in C++). CSS uses -- for backward compatibility reasons, (the same reasons it doesn't support // comments). Since your syntax is not restricted by the bounds of CSS, consider using something different, such as the dollar sign $ (as in PHP, bash, awk, perl, scala string interpolation, Javascript template literals) In fact, I can't think of any other language that CSS where -- denotes a variable.

[viridia]

I'm not wedded to CSS variable syntax, I only chose it because I could.

I was considering using the handlebars syntax {{var}} since that's old-school HTML templating style.

[viridia]

I don't really want this thread to get bogged down in issues of file formats for stylesheets, that's just syntax. I actually don't care all that much what format gets chosen.

To me, the more interesting question is: "What subset of CSS semantics should we support?".

For some existing UI libraries - for example bevy_egui - the answer is, "nothing". The appearance of the widgets are implemented in Rust code, and cannot be changed without effectively re-implementing the widget. I think this is a non-starter for game authors, because games require novelty in their UIs.

Other libraries offer a smidgen of customization options - you can perhaps change the background image nine-patch of a button, but all of the other attributes such as alignment are predetermined. Again, I don't think this is enough.

On the other hand, there are libraries which try to offer the full set of styling options one would get running in a browser. I think that's too much - both because of the difficulty of implementing it, and because (as I explained above) some of those options lead to code that's hard to maintain.

So the challenge is to cherry-pick the bits we like, and adapt them to the set of design patterns that live in the Bevy ECS and Asset worlds.

The model I am thinking of is one where styles are assets, that is, assigning a style to a widget is basically assigning an asset handle of a style resource to a property of the widget. But I don't want to throw out CSS rule-matching entirely, because rule-matching is useful for dynamic states (hover, pressed, focused, primary vs. secondary and so on). Also, I think it's useful to be able to compose styles, that is, to be able to apply a combination of multiple style assets to a single widget.

Here's a sketch of a selector matching scheme, represented as an enum:

#[derive(Debug, PartialEq, Clone)]
pub enum Selector {
    /// If we reach this state, it means the match was successful
    Accept,

    /// Match an element with a specific class name and proceed to the next state.
    Class(String, Box<Selector>),

    /// Reference to the current element.
    Current(Box<Selector>),

    /// Reference to the parent of this element.
    Parent(Box<Selector>),

    /// List of alternate choices.
    Either(Vec<Box<Selector>>),
}

The enum represents the nodes of an NFA. Each node has a pointer to the next node in the chain. You start at the start node ("Current") and try to get to the "Accept" state. If you do, the selector matches, otherwise it doesn't match.

So for example, the CSS selector expression: .foo > & translates to:

Current(Parent(Class("foo", Accept)))

Note that except for the Either matcher, none of the selectors require any backtracking, which means that all you need is a match statement inside of a loop. Either can be handled by a recursive call.

I deliberately left out the "ancestor" matcher which in CSS has the syntax "A B", that is, two selectors separated by a space. The reason for doing this is because it requires backtracking (because you don't know how far up the ancestor chain you have to search), and because it's not necessary, you can use multiple parent selectors if you really need to.

For the same reason, the "current" node always has to come first, meaning that the search always goes up the ancestor chain, never down. Again, this is to avoid backtracking and searching over children. In fact, the Current enum could be omitted entirely, since we already know that we're starting at the current node. The main reason for keeping it is round-tripping, so that we can convert the NFA back into a text format during serialization.

There's also no matches for pseudo-classes like :hover because Bevy widgets don't have psuedo-classes. (They don't have classes either, but that's something we would need to add. Once again, this is a question of how much of web semantics we want to imitate.)