Build dependency in Windows GitHub worfklow

[JPHutchins]

I have a bit of a weird one!

jobs:
  build:
    name: Build the Packages
    strategy:
      matrix:
        os: [windows-latest, ubuntu-latest, macos-latest, macos-13]
    runs-on: ${{ matrix.os }}
    timeout-minutes: 5

    steps:
      - name: Check out code from GitHub
        uses: actions/checkout@v4
        with:
          submodules: recursive
          token: ${{ secrets.MY_PAT }}

      - run: pipx install poetry

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Build the packages
        run: poetry install

One of my dependencies is another package that has protoc as a build dependency. Linux/macOS, and Windows locally, are correctly installing that build dependency at build time.

[build-system]
requires = ["hatchling", "hatch-vcs", "mypy-protobuf==3.6.0", "grpcio-tools==1.65.4"]
build-backend = "hatchling.build"

But the error produced on the Windows runner is that python can't find grpc_tools:

C:\hostedtoolcache\windows\Python\3.12.8\x64\python.exe: Error while finding module specification for 'grpc_tools.protoc' (ModuleNotFoundError: No module named 'grpc_tools')

The source of that error message is in hatch_build.py:

import os
import subprocess

from hatchling.builders.hooks.plugin.interface import BuildHookInterface


REPO_ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", ".."))

class BuildFrontend(BuildHookInterface):
    PLUGIN_NAME = "build_protos"

    def initialize(self, version, build_data):
        p = subprocess.run(
            (
                "python",
                "-m",
                "grpc_tools.protoc",
                ... lots of arguments etc
            ),
            stdout=subprocess.PIPE,
            stderr=subprocess.PIPE,
        )
        if p.returncode != 0:
            raise RuntimeError(f"Failed to build protos!\n{p.stdout.decode()}\n{p.stderr.decode()}")

        return super().initialize(version, build_data)

Mysterious that it works in every environment other than "Windows GitHub Runner". It makes me wonder if the runner is somehow installing the build dependency to the wrong interpreter? The error is generated by this interpreter: C:\hostedtoolcache\windows\Python\3.12.8\x64\python.exe.

I can fix it by installing the build dependency before the build, but this is not preferred!

      - name: Build the packages
        run: |
          pip install grpcio-tools==1.65.4 
          poetry install

Thanks for looking, it's totally a long shot to see if anyone has seen something like this!

JP

[JPHutchins]

Narrowing in on this being a problem introduced by the Poetry 2.x rollout.

...

Nope, downgrading to 1.8.5 did not resolve it. I do see that the Windows runner is installing poetry with python 3.9 whereas the Ubuntu and macOS runners are using 3.12 or 3.13. The package being built uses 3.12.

Run pipx install poetry==1.8.5
  pipx install poetry==1.8.5
  shell: C:\Program Files\PowerShell\7\pwsh.EXE -command ". '{0}'"
Found a space in the pipx home path. We heavily discourage this, due to
    multiple incompatibilities. Please check our docs for more information on
    this, as well as some pointers on how to migrate to a different home path.
creating virtual environment...
installing poetry from spec 'poetry==1.8.5'...
done! 
  installed package poetry 1.8.5, installed using Python [3]
  These apps are now globally available
    - poetry.exe

[JPHutchins]

Now leaning toward it being a poetry issue since the build always works with pip install.

[ofek]

That's very odd; perhaps show the pyproject.toml file?

[JPHutchins]

Thanks for taking a look!

The dependency that causes a problem in poetry:

[build-system]
requires = [
    "hatchling",
    "hatch-vcs",
    "mypy-protobuf==3.6.0",
    "grpcio-tools==1.65.4",
]
build-backend = "hatchling.build"

[project]
name = "some_project"
dynamic = ["version"]

[tool.hatch.version]
source = "vcs"

[tool.hatch.version.raw-options]
root = "../../"

[tool.hatch.build.hooks.custom]

Redacted excerpt from the poetry managed repo that's having a hard time on Windows:

[tool.poetry.dependencies]
# other stuff
some_project = { path = "some_project/python/some_project" }
nanopb_pb2 = { path = "some_project/python/nanopb_pb2" }

Using a local install via git submodule to avoid having to host the wheels someplace. Though at this point it would have saved me time to just host the wheels.