Use up-to-date namespace package setup for plugins (#5505)

Refactor `beetsplug` to use native namespace packages by removing
`__init__.py`. Update documentation and `setup.cfg` to support namespace
packages.

### Motivation

Adopt PEP 420 native namespace packages to simplify plugin management
and eliminate the need for `__init__.py`.

See https://realpython.com/python-namespace-package.

This setup is backwards-compatible, so plugins using the old
pkgutil-based setup will continue working fine.

The advantage with this setup is that external plugins will now be able
to import modules from 'beetsplug' package for typing purposes.
Previously, mypy could not resolve these modules due to presence of
`__init__.py`.

docs/changelog.rst

@@ -74,6 +74,8 @@ Bug fixes:
 For packagers:
 
 * The minimum supported Python version is now 3.9.
+* External plugin developers: ``beetsplug/__init__.py`` file can be removed
+  from your plugin as beets now uses native/implicit namespace package setup.
 
 Other changes:
 

docs/dev/plugins.rst

@@ -3,47 +3,57 @@
 Writing Plugins
 ---------------
 
-A beets plugin is just a Python module inside the ``beetsplug`` namespace
-package. (Check out this `Stack Overflow question about namespace packages`_ if
-you haven't heard of them.) So, to make one, create a directory called
-``beetsplug`` and put two files in it: one called ``__init__.py`` and one called
-``myawesomeplugin.py`` (but don't actually call it that). Your directory
-structure should look like this::
+A beets plugin is just a Python module or package inside the ``beetsplug``
+namespace package. (Check out `this article`_ and `this Stack Overflow
+question`_ if you haven't heard about namespace packages.) So, to make one,
+create a directory called ``beetsplug`` and add either your plugin module::
 
     beetsplug/
-        __init__.py
         myawesomeplugin.py
 
-.. _Stack Overflow question about namespace packages:
-    https://stackoverflow.com/questions/1675734/how-do-i-create-a-namespace-package-in-python/1676069#1676069
+or your plugin subpackage::
 
-Then, you'll need to put this stuff in ``__init__.py`` to make ``beetsplug`` a
-namespace package::
+    beetsplug/
+        myawesomeplugin/
+            __init__.py
+            myawesomeplugin.py
+
+.. attention::
 
-    from pkgutil import extend_path
-    __path__ = extend_path(__path__, __name__)
+    You do not anymore need to add a ``__init__.py`` file to the ``beetsplug``
+    directory. Python treats your plugin as a namespace package automatically,
+    thus we do not depend on ``pkgutil``-based setup in the ``__init__.py``
+    file anymore.
 
-That's all for ``__init__.py``; you can can leave it alone. The meat of your
-plugin goes in ``myawesomeplugin.py``. There, you'll have to import the
-``beets.plugins`` module and define a subclass of the ``BeetsPlugin`` class
-found therein. Here's a skeleton of a plugin file::
+The meat of your plugin goes in ``myawesomeplugin.py``. There, you'll have to
+import ``BeetsPlugin`` from ``beets.plugins`` and subclass it, for example
+
+.. code-block:: python
 
     from beets.plugins import BeetsPlugin
 
-    class MyPlugin(BeetsPlugin):
+    class MyAwesomePlugin(BeetsPlugin):
         pass
 
 Once you have your ``BeetsPlugin`` subclass, there's a variety of things your
 plugin can do. (Read on!)
 
-To use your new plugin, make sure the directory that contains your
-``beetsplug`` directory is in the Python
-path (using ``PYTHONPATH`` or by installing in a `virtualenv`_, for example).
-Then, as described above, edit your ``config.yaml`` to include
-``plugins: myawesomeplugin`` (substituting the name of the Python module
-containing your plugin).
+To use your new plugin, package your plugin (see how to do this with `poetry`_
+or `setuptools`_, for example) and install it into your ``beets`` virtual
+environment. Then, add your plugin to beets configuration
 
-.. _virtualenv: https://pypi.org/project/virtualenv
+.. code-block:: yaml
+
+    # config.yaml
+    plugins:
+      - myawesomeplugin
+
+and you're good to go!
+
+.. _this article: https://realpython.com/python-namespace-package/#setting-up-some-namespace-packages
+.. _this Stack Overflow question: https://stackoverflow.com/a/27586272/9582674
+.. _poetry: https://python-poetry.org/docs/pyproject/#packages
+.. _setuptools: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#finding-simple-packages
 
 .. _add_subcommands:
 
@@ -249,13 +259,13 @@ The events currently available are:
   during a ``beet import`` interactive session. Plugins can use this event for
   :ref:`appending choices to the prompt <append_prompt_choices>` by returning a
   list of ``PromptChoices``. Parameters: ``task`` and ``session``.
-  
+
 * `mb_track_extract`: called after the metadata is obtained from
   MusicBrainz. The parameter is a ``dict`` containing the tags retrieved from
   MusicBrainz for a track. Plugins must return a new (potentially empty)
   ``dict`` with additional ``field: value`` pairs, which the autotagger will
   apply to the item, as flexible attributes if ``field`` is not a hardcoded
-  field. Fields already present on the track are overwritten. 
+  field. Fields already present on the track are overwritten.
   Parameter: ``data``
 
 * `mb_album_extract`: Like `mb_track_extract`, but for album tags. Overwrites

setup.cfg

@@ -37,3 +37,5 @@ allow_any_generics = false
 # FIXME: Would be better to actually type the libraries (if under our control),
 # or write our own stubs. For now, silence errors
 ignore_missing_imports = true
+namespace_packages = true
+explicit_package_bases = true