Azure Function Support

[calleo]

Hi,

I am very new to Pants, but I am trying to consolidate a number of repositories of Azure Functions into a single one and use Pants to have a consistent development workflow and package them up before they are deployed.

Azure functions can be deployed in a number of ways using the Azure CLI, but I believe using the "zip deploy" method would suit Pants the best.

In short, any local Python files and config-files (json) are added to the zip and all 3rd part dependencies are installed into a sub-directory. It's a lot like the pex-files, but folder structure is different.

Azure docs: https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python?tabs=asgi%2Capplication-level&pivots=python-mode-configuration#install-local-packages

Another alternative would be to replicate this folder structure (see link below) in the dist folder and let Azure install dependencies from a requirements-file:

https://learn.microsoft.com/en-us/azure/azure-functions/functions-reference-python?tabs=asgi%2Capplication-level&pivots=python-mode-decorators#folder-structure

Then deploy from that folder using the Azure CLI.

Perhaps it's possible to combine existing Pants-functionality to build such an archive?

[jsirois]

@calleo you should be able to use a pex_binary target for this.

From what I can tell you need to ensure just 3 things:

1. You just need to add a resource target to own the JSON config and add a dependency to the pex_binary so the PEX created by ./pants package ... includes the JSON config files.
2. You need to ensure your entry point script has an import __pex__ line before ~anything else. It really just needs to be before any 3rd party libraries are imported. That import will install a special import hook that knows Pex and will be able to find all the code packaged in your PEX file.
3. You probably need to specify an output_path for the pex_binary target. I don't know Azure, but AWS requires the file have the .zip extension, and Pants defaults to .pex.

[calleo]

Thank you @jsirois for that suggestion! I could get it to work using the PEX-binary.

Here's how I did it:

A Function App is a collection of Functions, which are deployed as a single artifact (zip-file). Each function is a sub-folder of that package, which contains the handler function and the config (function.json).

To allow me to deploy multiple function to a single Function App I created one three folders:

• Function A
• Function B
• For the app itself, which depends on Function A and Function B

You can see how it's done over here: https://github.com/calleo/pants-azure-function-example

I don't understand how PEX loads 3rd party dependencies, but it works, even though the zip-file doesn't follow the recommended layout. I was having problems deploying multiple functions at first, but I believe the issue was that my handler functions were in files named __main__.py. It was late and I fiddled with a lot of things to get it working, so not 100% sure that was it 🙂

This approach will not work when using the V2 programming model, which requires a function_app.py in the root of the archive where handlers are defined using decorators.

[jsirois]

I don't understand how PEX loads 3rd party dependencies, but it works

The import __pex__ line is how. That has side-effects. The __pex__ package is in the root of the zip (check and see!) and its __init__.py has the side-effect of installing a custom importer for everything that lives inside the PEX file, allowing the internal arrangement of those things to be as PEX sees fit but hidden from the rest of the Python ecosystem, which just gets at those strangely arranged things via import ... and the import machinery from that line on.

[lilatomic]

We deploy an azure function with a Docker container, so things are a bit different. It deploys several related functions (a mix of Durable and regular functions).
Here's how we got it to work:

• Create a target to reference all of our functions
• Create a shim python file for the entrypoint. This is empty for us.
• Create 2 pex_binary targets, one for dependencies and one for source code. This is to allow us to shuffle dependencies to where they need to go, and also happens to allow us to optimise builds a bit
• Create a docker_image target which depends on those pexes which shuffles things where they need to go
  • deps go to /usr/local/lib/python3.10/site-packages which seemed like the most normal place to put them that was on the pythonpath (the normal way is to just pip install a requirements file)
  • srcs need to be pulled up a level so the file-based inspection works (Azure knows which functions exist by looking for descriptor files)

This is kindof gross, but we're migrating away from Azure functions.

target(
    name="application",
    dependencies=[
        FunctionA:FunctionA,
        ...
    ]
)
pex_binary(
    name="deps",
    dependencies=["//:application"],
    entrypoint="shim.py",
    layout="packed",
    include_sources=False,
    include_requirements=True,
    include_tools=True,
)
pex_binary(
    name="srcs",
    dependencies=["//:application"],
    entrypoint="shim.py",
    layout="packed",
    include_sources=True,
    include_requirements=False,
    include_tools=True,
)
docker_image(
    name="docker0",
    dependencies=["://srcs", "://deps"],
)

The dockerfile needs to do some shuffling:

FROM mcr.microsoft.com/azure-functions/python:4-python3.10-buildenv as deps
WORKDIR /home/site/wwwroot/
COPY ./deps.pex deps.pex
RUN PEX_TOOLS=1 /usr/local/bin/python3 deps.pex venv --scope=deps --compile /home/site/wwwroot

FROM mcr.microsoft.com/azure-functions/python:4-python3.10-buildenv as srcs
WORKDIR /home/site/wwwroot/
COPY ./srcs.pex srcs.pex
RUN PEX_TOOLS=1 /usr/local/bin/python3 srcs.pex venv --scope=srcs --compile /home/site/wwwroot

FROM mcr.microsoft.com/azure-functions/python:4-python3.10-appservice as srcs
COPY --from=deps /home/site/wwwroot/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages
COPY --from=srcs /home/site/wwwroot/srcs.pex /home/site/wwwroot
# you might need to copy in the other descriptors like host.json and function.json depending on how you defined dependencies and targets

[calleo]

Nice @lilatomic! I'll try that out if we need to use Azure Functions. Looks more robust than working with zip-files.

Out of curiosity, what are you using instead of Azure Functions? I ended up using Azure Container Services instead.

[lilatomic]

We mostly used timer functions and durable functions for fanout, which maps fairly easily onto Enterprise Integration Patterns, so it's straightforward to migrate off of the Azure Functions runtime. We already have a Kubernetes cluster in our org, so we're just going to host the workers in that. We looked into Azure's Container Instances as well. They seemed fine, but we already had a Kubernetes cluster, the tooling for it, and we then get all the other things we've added to that platform (observability, autoscaling, authentication, rbac).