BUILD file symbols

[kaos]

There are some good ideas presented in #14832 that needs a little bit more design and sign off before being implemented. The purpose of this discussion is to drive that home.

Current state

BUILD files are just regular Python files with the added restriction that they may not use import.

• Macros

To offer generic BUILD-file constructs there's support for BUILD file "macros", whose contents are in effect prepended to each BUILD file in order to support globally defined values and methods.

There is no introspection for these macro defined values so they can't be documented properly.

• Plugins

Using the Plugin API backends may register new target types and BUILD file aliases. The targets are well documented and will not be further included in this discussion as they are already in a good position. BUILD file aliases are the Plugin API equivalence of a BUILD file macro file, and equally under-documented.

This is at the core of this discussion, to find new ways to declare macros both from macro files and the Plugin API, by exploring the UX/DX to support well documented macros.

UX - BUILD file macros

Introducing a new macros manager, users use this manager to setup their macro methods and values along with any documentation as desired. Pants will then make this information available in the online help using pants macros --help where all available macros are discoverable and documented.

Macros manager example:

# pants-plugins/macros.py

# Activate the macros manager, using the `.namespace()` to scope the declared macros:
with macros.namespace("ex_inc", description="I may want to describe this namespace") as ns:
  @ns.macro(help="My help text")
  def macro_impl(...):
    pass

  ns.const.MY_CONST(42, help="Describe `MY_CONST` that has value 42")

# May place macros globally..
with macros as m:
  m.const.FOO("my FOO value", help="Use this for FOO stuff.")

# Use as many `with macros ..` blocks are needed.

The declared macros and constants are then available directly in regular BUILD files:

# src/some/BUILD

ex_inc.macro_impl(value=ex_inc.MY_CONST + 24, foo=FOO)

Help for namespaced macros may be queried directly:

$ pants ex_inc --help

# in case the namespace conflicts with another help topic, the full name may be used:
$ pants macros.ex_inc --help

DX - Plugin API

TBD.

Some notes:

• These APIs will be the same that the macro manager uses, so there is no difference in the end result between registering macros in plugins or in macro files.
• Macros will be accessible to @rules to inspect/interact with.
• Maybe: Macros may leverage the parser state. Advanced feature only available when using the Plugin API.
  The idea with this last one, is to be able to refactor __defaults__ and __dependencies_rules__ to be just another macro citizen. (may turn out there are too many dependencies that goes out of scope for macros to be feasible, but would like to explore this option)

[stuhood]

There is no introspection for these macro defined values so they can't be documented properly.

There probably could be though...? It seems like it would be straightforward to extract docstrings from macros.

[kaos]

True. We'd get at least some level of docs that way. (def an improvement over status quo)

[stuhood]

Macros manager example:

Another option that avoids the nesting would be to have (require?) explicit export statements at the end of a macro module that would optionally namespace them.

[kaos]

Unless you could do it without having to repeat all the names you export I'd prefer the context manager approach (nesting is only one level deep.. so not that bad?)

[cognifloyd]

Agreed. Like @kaos, I prefer nesting. That avoids forgetting to register it.

[cognifloyd]

I wonder if it will be possible to use the macro namespace immediately after definition in the same BUILD file.

conceptually, you probably wouldn't need the namespace for macro functions in the same BUILD file, but constants are iffy.

with macros.namespace("ex_inc", description="I may want to describe this namespace") as ns:
    @ns.macro(help="My help text")
    def macro_impl(...):
        pass

    ns.const.MY_CONST(42, help="Describe `MY_CONST` that has value 42")

# this is defined, so it is usable.
macro_impl(...)

# but could you do this?
ex_inc.macro_impl(...)

# or would you have to do this, using the `ns` object from the with block?
ns.macro_impl(...)

some_target(
    # and how would you access consts in the same BUILD file?
    some_field=ex_inc.MY_CONST,
    # or would it be like this
    other_field=ns.MY_CONST,
)

It's a little more straightforward if you're defining the macros globally instead of in a namespace, as then just using macro_impl(...) makes perfect sense. But, accessing the global const is still questionable.

This gets dicey in the same BUILD file because pants would have to inject stuff into locals() and not just prepend / make symbols available before parsing the BUILD file.

[kaos]

I wonder if it will be possible to use the macro namespace immediately after definition in the same BUILD file.

I would say it is broken if not, so.. Yes. Including constants.