Coursier: [Question] - Retrieving sbt plugin versions from hosted Sonatype Nexus

[ckipp01]

At my work we use Scala Steward to update all of our internal dependencies. We use a hosted Sonatype Nexus to both have a mirror of Maven Central and also to have our own internal releases Maven repo. From looking through the Scala Steward code it seems that it relies on Coursier to retrieve the list of versions available for the artifact.

https://github.com/scala-steward-org/scala-steward/blob/a2cea0f1f72f13b38f3b5cb01ae3cad04a58aff0/modules/core/src/main/scala/org/scalasteward/core/coursier/VersionsCache.scala#L54

If I understand it right, this is done for artifacts that have a maven-metadata.xml file, but as you know, sbt plugins that are published Maven style don't have one. Which again, to my understanding because of this, Coursier then relies on the directory listing? This is how it seems to work for our maven-central mirror with the sbt-scalafmt plugin. By looking at it with a path like this, I can see an html directory listing of all versions:

https://nexus-redacted.com/repository/maven-public/org/scalameta/sbt-scalafmt_2.12_1.0

Scala Steward (Coursier) is able to correctly gather the versions (again I assume from this and sounds like that from this thread in Steward). However, I'm finding a lot of contradictory information about whether or not this is available on different versions of Nexus3. Plus those that manage our Nexus is saying there is no option to turn this on.

• Community post saying that it exists but is at a different path now <-- This seems to be the case
• Sonatype ticket to create this feature
• Another ticket for the raw tree view
• Announcement saying it's now supported in 3.7
• SO post saying it's in a different location now

From what I can tell, for our own internal nexus where I'd expect the directory listing to be like the sbt-scalafmt example above, it just 404's.

https://nexus-redacted.com/repository/redacted-releases/redacted/redacted/sbt-redacted_2.12_1.0

There is an html index of sorts at this url:

https://nexus-redacted.com/service/rest/repository/browse/redacted-releases/redacted/redacted/sbt-redacted_2.12_1.0/

However, looking through the code here, I don't believe that will work:
https://github.com/coursier/coursier/blob/59bf7739079f6b906ecb77435c8afce2d3658f2c/modules/core/shared/src/main/scala/coursier/maven/MavenRepository.scala#L208-L252

And when Scala Steward attempts to get the available versions of the plugin, it gets none and therefore doesn't send in prs for our internal sbt plugins. Which sort of leads me to believe that because of the lack of directory listing there, Coursier isn't getting the versions? Do you know if it's possible for Coursier to get the versions of an sbt plugin without the directory listing, or does that have to be there? Or could there be another check possible to capture this alternative html view the same way the other is captured?

EDIT: For the time being we just add in a custom release step to create / update the maven-metadata.xml file, but it'd still be great to see if what I mentioned above is the case or not.

[froth]

@ckipp01 Could you share the code of your custom release step for maven-metadata.xml? struggling with the same problem currently

[a1kemist]

@ckipp01 We are also encountering this issue when trying to utilize scala-steward for our internal sbt plugins. We tried to use arktekk/sbt-aether-deploy to generate/update the maven-metadata.xml, but Nexus3 complains that the file is invalid when publishing.

Could you briefly outline how you were able to generate the correct maven-metadata.xml?

[ckipp01]

So this is a pretty hacky solution, but we essentially just included this as a custom release step (if you use sbt-release).

import java.nio.charset.StandardCharsets
import java.nio.file.Paths
import java.text.SimpleDateFormat
import java.util.Base64

import sbtrelease.ReleasePlugin.autoImport.ReleaseStep
import sbt.Project
import sbt.Keys
import sbt.internal.util.ManagedLogger
import Env._

import scala.xml.Elem
import scala.xml.Node
import scala.xml.NodeSeq
import scala.xml.XML

object MetaDataUpdate {

  /** An extra release step that is sort of hack specifically for an sbt plugin.
    * This step will look to see if there is an existing `maven-metadata.xml` file
    * (which sbt plugins don't produce when published maven style) and add the new
    * version to it or create a new one and push it up.
    */
  val step: ReleaseStep = ReleaseStep { st =>
    val logger       = st.log
    val extracted    = Project.extract(st)
    val name         = extracted.get(Keys.name)
    val org          = extracted.get(Keys.organization)
    val version      = extracted.get(Keys.version)
    val target       = extracted.get(Keys.target)
    val metadataName = "maven-metadata.xml"

    val creds = Base64.getEncoder
      .encodeToString(s"${nexusUsername}:${nexusPassword}".getBytes(StandardCharsets.UTF_8))

    val metadataLoc =
      s"${nexusAddress}/${nexusRepositoryPath}${org.replace(".", "/")}/${name}_2.12_1.0/${metadataName}"

    val outputPath = target.getPath() + metadataName

    val existingResponse = requests.get(
      metadataLoc,
      headers = Map(
        "Authorization" -> s"Basic: ${creds}"
      )
    )

    if (existingResponse.is2xx) {
      val rawText          = existingResponse.text()
      val existingMetadata = XML.loadString(rawText)

      val versions = existingMetadata \\ "version"
      val metadata = createMetadata(org, name, version, Some(versions))

      logger.info("Existing maven-metadata.xml found and being updated...")
      saveAndSend(metadataLoc, metadata, outputPath, creds, logger)
    } else if (existingResponse.statusCode == 404) {
      logger.warn("Existing maven-metadata.xml file not found. Attempting to create a new one...")

      val metadata = createMetadata(org, name, version, None)
      saveAndSend(metadataLoc, metadata, outputPath, creds, logger)
    } else {
      logger.error(s"${existingResponse.statusCode.toString()}: ${existingResponse.statusMessage}")
      // If something goes wrong here we just blow up
      sys.error("Something went wrong when requesting maven-metaldata.xml.")
    }

    st
  }

  private def createMetadata(
      org: String,
      name: String,
      newVersion: String,
      existingVersions: Option[NodeSeq]
  ) = {

    val newVersionNode = <version>{newVersion}</version>
    val versions = existingVersions match {
      case Some(existing) => existing ++ newVersionNode
      case None => newVersionNode
    }

    <metadata modelVersion="1.1.0">
      <groupId>{org}</groupId>
      <artifactId>{name}_2.12_1.0</artifactId>
      <versioning>
        <latest>{newVersion}</latest>
        <release>{newVersion}</release>
        <versions>
          {versions}
        </versions>
        <lastUpdated>
          {new SimpleDateFormat("yyyyMMddHHmmss").format(new java.util.Date)}
        </lastUpdated>
      </versioning>
    </metadata>
  }

  private def saveAndSend(
      metadataLoc: String,
      metadata: Elem,
      path: String,
      creds: String,
      logger: ManagedLogger
  ): Unit = {
    XML.save(path, metadata)

    val putReponse = requests.put(
      metadataLoc,
      headers = Map(
        "Authorization" -> s"Basic: ${creds}"
      ),
      data = Paths.get(path)
    )

    if (!putReponse.is2xx) {
      logger.error(s"${putReponse.statusCode}: ${putReponse.statusMessage}")
      sys.error("Unable to update metadata")
    }
  }
}

It's pretty self-explanatory, but it will just check to see if there is a metadata file, and if not create one. If it does fine one, it just updates it. It's using requests-scala for the calls.

[alexarchambault]

Which sort of leads me to believe that because of the lack of directory listing there, Coursier isn't getting the versions?

I'm a bit late to the party here, but yes, for sbt plugins, we rely on directory listings. As the plugin artifacts' paths aren't standard (the _2.12_1.0 appears in a directory name, but not in the filenames), it seems Sonatype doesn't generate or update maven-metadata.xml files itself.

Do you know if it's possible for Coursier to get the versions of an sbt plugin without the directory listing, or does that have to be there? Or could there be another check possible to capture this alternative html view the same way the other is captured?

If you push a maven-metadata.xml file yourself, as you do, there should be a way to have it picked… It should actually be checked if getting the directory listing fails (via the orElse). Maybe this code doesn't look for maven-metadata.xml at the right URL? It could be fixed then.

[ckipp01]

If you push a maven-metadata.xml file yourself, as you do, there should be a way to have it picked… It should actually be checked if getting the directory listing fails (via the orElse). Maybe this code doesn't look for maven-metadata.xml at the right URL? It could be fixed then.

Just to clarify, yes, it does work now with us creating the maven-metadata.xml file, but was curious if there was a solution where having that step to manually create one for sbt plugins on self-hosted nexus would no longer be necessary.

[jchapuis]

thanks for the great workaround, on my end I had to add check = false for this to work otherwise scala-requests 0.6.5 would throw

val existingResponse = requests.get(
      metadataLoc,
      headers = Map(
        "Authorization" -> s"Basic: ${creds}"
      ),
      check = false
    )

[laughedelic]

Hi! I came here from https://github.com/scala-steward-org/scala-steward-action/issues/209#issuecomment-799687563 and then https://github.com/scala-steward-org/scala-steward/issues/1628.

I have the same problem but for libraries (not sbt-plugins) published maven-style. There is no maven-metadata.xml and Scala Steward (using Coursier) can't find any updates.

Is there a way without the workaround with manually creating maven-metadata.xml?