Guidance on namespace packages

[amercader]

Problem description

Apologies in advance if this is not the right place to ask this question, but we wanted to ask for some guidance about namespace packages.

The CKAN project has an extensions system, which are essentially python modules under the ckanext. namespace. There are hundreds of extensions and the CKAN Tech Team only has direct control over a handful of officially supported ones under the ckan organization in GitHub.

Namespace packages have historically been defined using the pkg_resources-style (actually the extension template uses the discouraged pkgutil fallback, but we will remove that).

According to the docs:

While this approach is no longer recommended, it is widely present in most existing namespace packages. If you are creating a new distribution within an existing namespace package that uses this method then it’s recommended to continue using this as the different methods are not cross-compatible and it’s not advisable to try to migrate an existing package.

But also:

Also, pkg_resources has been deprecated.

We obviously want to ensure compatibility with the existing ecosystem of extensions but also start preparing if the current way of defining namespace packages will eventually stop being supported.

Would you recommend staying put and keep using pkg_resource.declare_namespace() in current and new distributions?
Will pkg_resource.declare_namespace() or pkg_resource for that matter eventually be removed?

Many thanks in advance

[merwok]

This is a question about the import system, not the packaging ecosystem.
Here is a guide: https://realpython.com/python-namespace-package/

[henryiii]

You can use native namespaces packages now. That's the old pre-Python 3 system. Those docs sound very old; I'm sure that statement came from when Python 2 still mattered.

[henryiii]

Where are those docs?

[matthewfeickert]

Where are those docs?

@henryiii Looks like from the Python Packaging User Guide (surprisingly): https://packaging.python.org/en/latest/guides/packaging-namespace-packages/#pkg-resources-style-namespace-packages

Specifically here:
https://github.com/pypa/packaging.python.org/blob/48ad6bff2ce76fdd3de6d975d853cfe3278b7af7/source/guides/packaging-namespace-packages.rst?plain=1#L221-L226

[abravalheri]

Will pkg_resource.declare_namespace() or pkg_resource for that matter eventually be removed?

Eventually, yes. And I believe that there is also some appetite for removing pkgutil.extend_path (some features in the packaging ecosystem already do not work with those old methods).

Would you recommend staying put and keep using pkg_resource.declare_namespace() in current and new distributions?

No. Please use implicit namespaces (PEP 420) whenever you can (the way to determine if you can is via tests).
There is no need to panic (because the removal will not happen next week), but staying put means that all the changes will have to be implemented at once when the functionality is finally removed, and that can cause a lot of disruption.

We obviously want to ensure compatibility with the existing ecosystem of extensions but also start preparing if the current way of defining namespace packages will eventually stop being supported.

If you are concerned about compatibility, you can have a look at this table: https://github.com/pypa/sample-namespace-packages/blob/master/table.md . The rule of thumb is to stick with regular installs (and not use editable installs) for better compatibility. More info on the section: https://github.com/pypa/sample-namespace-packages?tab=readme-ov-file#remarks-on-staggered-migrations.

Because we are talking about deprecated functionality it is expected that packages start to move away from old methods and maintainers start to modify the codebase to adhere to PEP 420. But because tests indicate that there is some level of compatibility (as long as you stick with some rules as mentioned above), this migration can be gradual (which facilitate maintenance of large ecosystems)¹. Automation tools may also be used by large ecosystem: the main change to be implemented for implicit namespaces is to remove __init__.py files.

Footnotes

1. Note however that the deprecation notice is already a few years old, so maintainers are advised to start the migration. ↩

[amercader]

Thanks @abravalheri that's is really helpful and gives us pointers to plan the migration.