datasource/sbt-plugin doesn't support registries that doesn't allow dir listing

[pnordh]

Tell us more.

Same issue reported for the sbt-package datasource in:

• Private gitlab maven repository doesn't work with sbt-package datasource #16761
• sbt datasource incompatible with non directory listable repositories #18589

... which was fixed by falling back to the maven datasource (parses versions available in a package's maven-metadata.xml) if it failed to list registry directories in:

• feat(sbt-package): Fallback to Maven datasource for GitLab packages #16817
• fix(datasource/sbt-package): Fallbacks to maven when no sbt directory listing available #18590

I'm thinking the same fix should be applicable for the sbt-plugin datasource (I'm working on it, including reproduction).

What do you think?

[MPV]

cc @sschoebinger @zharinov @fernandomora as you worked on the previous implementations (linked above)

[MPV]

There's a first draft over here:

• pnordh@f87568d

...however it doesn't fully handle the oddities of versioning of sbt-plugins such as:

differences in maven path for plugins published by sbt >= 1.9, which includes the extra attributes scalaVersion and sbtVersion

i.e. a scala package:
com/myorg/myproject/0.1.0-SNAPSHOT/myproject-0.1.0-SNAPSHOT.pom

versus an sbt plugin:
com/myorg/myplugin_2.12_1.0/1.0.0/myplugin_2.12_1.0-1.0.0.pom

Ideas welcome. 🙏

[MPV]

Here's example Renovate logs from when it fails to look up an sbt plugin from Google Artifact Registry:

DEBUG: GET https://europe-maven.pkg.dev/myproject/myregistry/com.myorg/ = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=119)
DEBUG: GET https://europe-maven.pkg.dev/myproject/myregistry/com/myorg/myplugin/ = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=380)
DEBUG: GET https://europe-maven.pkg.dev/myproject/myregistry/com/myorg/ = (code=ERR_NON_2XX_3XX_RESPONSE, statusCode=404 retryCount=0, duration=118)
DEBUG: No versions found for com.myorg:myplugin in 2 repositories
DEBUG: Failed to look up sbt-plugin package com.myorg:myplugin
{
  "dependency": "com.myorg:myplugin"
  "packageFile": "project/plugins.sbt"
}

[MPV]

Could this have been caused by:

• refactor(sbt): Decouple sbt-package and sbt-plugin datasources #31835

What do you think @zharinov (as you did that change)?

Asking since we'd like to fix this, and our current reasoning is that we will make SbtPluginDatasource inherent from either of SbtPackageDatasource or MavenDatasource to be able to solve this in a reusable manner.

[MPV]

I don't know. What's valid url for com.myorg:myplugin?

For example:
curl -s --oauth2-bearer "$(gcloud auth print-access-token)" https://europe-maven.pkg.dev/myproject/myregistry/com/myorg/myplugin_2.12_1.0/maven-metadata.xml

curl -s --oauth2-bearer "$(gcloud auth print-access-token)" https://europe-maven.pkg.dev/myproject/myregistry/com/myorg/myplugin_2.12_1.0/1.2.3/myplugin-1.2.3.jar

[zharinov]

Could this be solved via packageRules by changing datasource to maven or sbt-package? sbt-plugin datasource targeted the default place where sbt plugins are hosted, so private packages could be just out of scope.

[MPV]

Could this be solved via packageRules by changing datasource to maven or sbt-package? sbt-plugin datasource targeted the default place where sbt plugins are hosted, so private packages could be just out of scope.

@zharinov Ah well, that almost solved it (good idea though!).

In addition to doing that, I also needed to switch my addSbtPlugin() calls from % into %% to make sure renovate can find them (published using sbt 2.12) even though my repo is on 2.13. 🤔

After that I got Renovate to make PRs for myplugin. 🙏🏻

How might we preferably solve this to become a built-in feature?

[zharinov]

Maybe we could dispatch plugins between sbt-plugins and other datasources during extract phase, depending on registryUrl or some other clue?

[MPV]

In addition to doing that, I also needed to switch my addSbtPlugin() calls from % into %% to make sure renovate can find them (published using sbt 2.12) even though my repo is on 2.13. 🤔

Here's a follow-up issue on the separate problem that "SBT plugins are named with the standarized SBT version of SBT":

• `sbt-plugin` datasource should always `overridePackageName` with `_2.12_1.0` suffix #33939