Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some dependencies not updated #2664

Closed
james-s-w-clark opened this issue Jul 10, 2022 · 9 comments · Fixed by #2816
Closed

Some dependencies not updated #2664

james-s-w-clark opened this issue Jul 10, 2022 · 9 comments · Fixed by #2816

Comments

@james-s-w-clark
Copy link

I didn't see another issue like this one, so I'm opening this.

I saw a few dependencies were outdated in a project despite daily scala-steward runs (on latest image). I confirmed there were updates with mill mill.scalalib.Dependency/showUpdates

Here's some examples:

  • log4j api/core/slf4j-impl (2.16.0 -> 2.17.2)
  • refined (0.9.14 -> 0.9.28)
  • os-lib (0.7.8 -> 0.8.1)
  • wiremock-jre8 (2.31.0 -> 2.33.2)

In the case of slf4j, 2.16.0 was Dec 2021 and 2.17.2 is Feb 2022. I manually updated it in late June, so Scala Steward wasn't picking up those updates for about half a year.

Some of these in an dependencies.sc file have the version as part of the value. Some have it interpolated into the ivy dependency. I don't see an obvious pattern for why some things get updates, but others don't.

Posssible causes

  • Does scala steward cache failures from fail-fast, so if you have a failed CI check then you won't ever get e.g. 0.0.1 -> 0.0.2 of a dependency? This is a total guess. I don't know how scala-steward works.
@james-s-w-clark
Copy link
Author

james-s-w-clark commented Jul 11, 2022

com-lihaoyi/mill#1954 as example in a live public repo that uses scala-steward

@james-s-w-clark
Copy link
Author

james-s-w-clark commented Jul 19, 2022

Is there anything I could help with here? I had a dig and I can't figure out why this update didn't update, even though there's 9 new verisons of scalafmt-dynamic for example, spread over a few months.

The group/artifact IDs are unchanged, so there is no need for an artifact migration like at release page

There is no pinning to minor version in mill's scala-steward.conf either.

  val scalafmtDynamic = ivy"org.scalameta::scalafmt-dynamic:3.4.3"
  val scalafmtDynamic = ivy"org.scalameta::scalafmt-dynamic:3.5.8"

@james-s-w-clark
Copy link
Author

Maybe it's a problem with the public runer - more info here?

@exoego
Copy link
Contributor

exoego commented Jul 29, 2022

Can the reproducible repo be provided?

@james-s-w-clark
Copy link
Author

james-s-w-clark commented Jul 29, 2022

Can the reproducible repo be provided?

See #2664 (comment) (mill#1954) for an example.


I haven't scanned through these updates yet, but to illustrate what I did: In the mill repo, one can run mill mill.scalalib.Dependency/showUpdates. I didn't check where these come from, or if they're easy updates in dependencies.sc/build.sc yet. Versions listed are in build.sc
Example output of mill mill.scalalib.Dependency/showUpdates | grep ':' | sort --unique gives:

  • ch.epfl.scala:bloop-config_2.13 : 1.5.2 -> 1.5.3
  • com.lihaoyi:utest_2.13 : 0.7.11 -> 0.8.0
  • com.typesafe.play:routes-compiler_2.12 : 2.6.25 -> 2.7.0 -> 2.7.1 -> 2.7.2 -> 2.7.3 -> 2.7.4 -> 2.7.5 -> 2.7.6 -> 2.7.7 -> 2.7.8 -> 2.7.9 -> 2.8.0 -> 2.8.1 -> 2.8.2 -> 2.8.3 -> 2.8.4 -> 2.8.5 -> 2.8.6 -> 2.8.7 -> 2.8.8 -> 2.8.9 -> 2.8.10 -> 2.8.11 -> 2.8.12 -> 2.8.13 -> 2.8.14 -> 2.8.15 -> 2.8.16
    • mvn dec 2019. This one is an interpolation, with version hard coded in another line. Probably don't expect auto update for this.
  • com.typesafe.play:routes-compiler_2.13 : 2.7.9 -> 2.8.0 -> 2.8.1 -> 2.8.2 -> 2.8.3 -> 2.8.4 -> 2.8.5 -> 2.8.6 -> 2.8.7 -> 2.8.8 -> 2.8.9 -> 2.8.10 -> 2.8.11 -> 2.8.12 -> 2.8.13 -> 2.8.14 -> 2.8.15 -> 2.8.16
    • Same as above, not a problem?
  • io.get-coursier:coursier_2.13 : 2.1.0-M6 -> 2.1.0-M6-26-gcec901e9a -> 2.1.0-M6-28-gbad85693f -> 2.1.0-M6-44-gc47ecec07
    • Mayb not updated as it's on a milestone version, and newer versions just have different hash on the end. link to mill. Seems OK. mvn
  • org.eclipse.jetty:jetty-server : 8.2.0.v20160908 -> 9.0.0.M0 -> 9.0.0.M1 -> 9.0.0.M2 -> 9.0.0.M3 -> 9.0.0.M4 -> 9.0.0.M5 -> 9.0.0.RC0 -> 9.0.0.RC1 -> 9.0.0.RC2 -> 9.0.0.v20130308 -> 9.0.1.v20130408 -> 9.0.2.v20130417 -> 9.0.3.v20130506 -> 9.0.4.v20130625 -> 9.0.5.v20130815 -> 9.0.6.v20130930 -> 9.0.7.v20131107 -> 9.1.0.M0 -> 9.1.0.RC0 -> 9.1.0.RC1 -> 9.1.0.RC2 -> 9.1.0.v20131115 -> 9.1.1.v20140108 -> 9.1.2.v20140210 -> 9.1.3.v20140225 -> 9.1.4.v20140401 -> 9.1.5.v20140505 -> 9.1.6.v20160112 -> 9.2.0.M0 -> 9.2.0.M1 -> 9.2.0.RC0 -> 9.2.0.v20140526 -> 9.2.1.v20140609 -> 9.2.2.v20140723 -> 9.2.3.v20140905 -> 9.2.4.v20141103 -> 9.2.5.v20141112 -> 9.2.6.v20141205 -> 9.2.7.v20150116 -> 9.2.8.v20150217 -> 9.2.9.v20150224 -> 9.2.10.v20150310 -> 9.2.11.M0 -> 9.2.11.v20150529 -> 9.2.12.M0 -> 9.2.12.v20150709 -> 9.2.13.v20150730 -> 9.2.14.v20151106 -> 9.2.15.v20160210 -> 9.2.16.v20160414 -> 9.2.17.v20160517 -> 9.2.18.v20160721 -> 9.2.19.v20160908 -> 9.2.20.v20161216 -> 9.2.21.v20170120 -> 9.2.22.v20170606 -> 9.2.23.v20171218 -> 9.2.24.v20180105 -> 9.2.25.v20180606 -> 9.2.26.v20180806 -> 9.2.27.v20190403 -> 9.2.28.v20190418 -> 9.2.29.v20191105 -> 9.2.30.v20200428 -> 9.3.0.M0 -> 9.3.0.M1 -> 9.3.0.M2 -> 9.3.0.RC0 -> 9.3.0.RC1 -> 9.3.0.v20150612 -> 9.3.1.v20150714 -> 9.3.2.v20150730 -> 9.3.3.v20150827 -> 9.3.4.RC0 -> 9.3.4.RC1 -> 9.3.4.v20151007 -> 9.3.5.v20151012 -> 9.3.6.v20151106 -> 9.3.7.RC0 -> 9.3.7.RC1 -> 9.3.7.v20160115 -> 9.3.8.RC0 -> 9.3.8.v20160314 -> 9.3.9.M0 -> 9.3.9.M1 -> 9.3.9.v20160517 -> 9.3.10.M0 -> 9.3.10.v20160621 -> 9.3.11.M0 -> 9.3.11.v20160721 -> 9.3.12.v20160915 -> 9.3.13.M0 -> 9.3.13.v20161014 -> 9.3.14.v20161028 -> 9.3.15.v20161220 -> 9.3.16.v20170120 -> 9.3.17.RC0 -> 9.3.17.v20170317 -> 9.3.18.v20170406 -> 9.3.19.v20170502 -> 9.3.20.v20170531 -> 9.3.21.M0 -> 9.3.21.RC0 -> 9.3.21.v20170918 -> 9.3.22.v20171030 -> 9.3.23.v20180228 -> 9.3.24.v20180605 -> 9.3.25.v20180904 -> 9.3.26.v20190403 -> 9.3.27.v20190418 -> 9.3.28.v20191105 -> 9.3.29.v20201019 -> 9.3.30.v20211001 -> 9.4.0.M0 -> 9.4.0.M1 -> 9.4.0.RC0 -> 9.4.0.RC1 -> 9.4.0.RC2 -> 9.4.0.RC3 -> 9.4.0.v20161208 -> 9.4.0.v20180619 -> 9.4.1.v20170120 -> 9.4.1.v20180619 -> 9.4.2.v20170220 -> 9.4.2.v20180619 -> 9.4.3.v20170317 -> 9.4.3.v20180619 -> 9.4.4.v20170414 -> 9.4.4.v20180619 -> 9.4.5.v20170502 -> 9.4.5.v20180619 -> 9.4.6.v20170531 -> 9.4.6.v20180619 -> 9.4.7.RC0 -> 9.4.7.v20170914 -> 9.4.7.v20180619 -> 9.4.8.v20171121 -> 9.4.8.v20180619 -> 9.4.9.v20180320 -> 9.4.10.RC0 -> 9.4.10.RC1 -> 9.4.10.v20180503 -> 9.4.11.v20180605 -> 9.4.12.RC0 -> 9.4.12.RC1 -> 9.4.12.RC2 -> 9.4.12.v20180830 -> 9.4.13.v20181111 -> 9.4.14.v20181114 -> 9.4.15.v20190215 -> 9.4.16.v20190411 -> 9.4.17.v20190418 -> 9.4.18.v20190429 -> 9.4.19.v20190610 -> 9.4.20.v20190813 -> 9.4.21.v20190926 -> 9.4.22.v20191022 -> 9.4.23.v20191118 -> 9.4.24.v20191120 -> 9.4.25.v20191220 -> 9.4.26.v20200117 -> 9.4.27.v20200227 -> 9.4.28.v20200408 -> 9.4.29.v20200521 -> 9.4.30.v20200611 -> 9.4.31.v20200723 -> 9.4.32.v20200930 -> 9.4.33.v20201020 -> 9.4.34.v20201102 -> 9.4.35.v20201120 -> 9.4.36.v20210114 -> 9.4.37.v20210219 -> 9.4.38.v20210224 -> 9.4.39.v20210325 -> 9.4.40.v20210413 -> 9.4.41.v20210516 -> 9.4.42.v20210604 -> 9.4.43.v20210629 -> 9.4.44.v20210927 -> 9.4.45.v20220203 -> 9.4.46.v20220331 -> 9.4.47.v20220610 -> 9.4.48.v20220622 -> 10.0.0-alpha0 -> 10.0.0.alpha1 -> 10.0.0.alpha2 -> 10.0.0.beta0 -> 10.0.0.beta1 -> 10.0.0.beta2 -> 10.0.0.beta3 -> 10.0.0 -> 10.0.1 -> 10.0.2 -> 10.0.3 -> 10.0.4 -> 10.0.5 -> 10.0.6 -> 10.0.7 -> 10.0.8 -> 10.0.9 -> 10.0.10 -> 10.0.11 -> 11.0.0-alpha0 -> 11.0.0.beta1 -> 11.0.0.beta2 -> 11.0.0.beta3 -> 11.0.0 -> 11.0.1 -> 11.0.2 -> 11.0.3 -> 11.0.4 -> 11.0.5 -> 11.0.6 -> 11.0.7 -> 11.0.8 -> 11.0.9 -> 11.0.10 -> 11.0.11
    • Version is hard-coded in a separate line, then interpolated. mill
  • org.flywaydb:flyway-core : 8.5.13 -> 9.0.0 -> 9.0.1 -> 9.0.2 -> 9.0.3 -> 9.0.4
    • This one is fine with Steward. It was stuck for a long time because they required you to add a NOTICE-level logging. mill.
  • org.jgrapht:jgrapht-core : 1.4.0 -> 1.5.0 -> 1.5.1
  • org.scala-js:scalajs-js-envs_2.13 : 0.6.33 -> 1.0.0 -> 1.0.1 -> 1.1.0 -> 1.1.1 -> 1.2.0 -> 1.2.1 -> 1.3.0
    • That's fine, it's stuck on 0.6 by choice. mill
  • org.scala-js:scalajs-sbt-test-adapter_2.13 : 0.6.33 -> 1.0.0 -> 1.0.1 -> 1.1.0 -> 1.1.1 -> 1.2.0 -> 1.3.0 -> 1.3.1 -> 1.4.0 -> 1.5.0 -> 1.5.1 -> 1.6.0 -> 1.7.0 -> 1.7.1 -> 1.8.0 -> 1.9.0 -> 1.10.0 -> 1.10.1
    • Same as above, no problem
  • org.scoverage:scalac-scoverage-plugin_2.13.8 : 1.4.11 -> 2.0.0 -> 2.0.1
    • more complex migration, work is under way. mill
  • org.testng:testng : 7.5 -> 7.6.0 -> 7.6.1
    • not upgrading so mill keeps it supported on jdk8 mill pr

I will edit the comment and note if any of the current versions are especially old (i.e. if scala steward should have updated them by now).

@james-s-w-clark
Copy link
Author

james-s-w-clark commented Jul 29, 2022

@exoego I think from all of the above, the only valid cases of Scala Steward missing updates are:

  • scalafmt-dynamic
  • bloop-config

@daddykotex
Copy link
Contributor

daddykotex commented Aug 15, 2022

it previously opened: https://github.com/com-lihaoyi/mill/pull/1741/files

so it has worked at some point (edit: is:pr is:closed author:scala-steward scalafmt )

@james-s-w-clark
Copy link
Author

james-s-w-clark commented Aug 15, 2022

The problem was with dependency configuration, not Scala Steward itself. Steward could not find a heuristic to do an upgrade with the dependencies we gave it.

If anyone finds themselves with this issue, you can have a play with Scala Steward and try to figure out how you can tweak your dependencies so changes get picked up properly.

  • clone Scala Steward
  • run sbt command testOnly org.scalasteward.core.edit.UpdateHeuristicTest
  • add code like the following to the test:
test("check".only) {
    val original =
      """|object Circe {
         |    private val version = "0.14.1"
         |
         |    val core    = ivy"io.circe::circe-core:$version"
         |  }
         |""".stripMargin
    val expected =
      """|object Circe {
         |    private val version = "0.14.2"
         |
         |    val core    = ivy"io.circe::circe-core:$version"
         |  }
         |""".stripMargin
    val update = ("io.circe".g % Nel.of("circe-core".a) % "0.14.1" %> "0.14.2").group
    assertEquals(update.replaceVersionIn(original), Some(expected) -> UpdateHeuristic.original.name)
  }

This does not find a heuristic to do upgrades with - the test fails. Steward PRs would not be raised for these dependencies.

test("check".only) {
    val original =
      """|object Circe {
         |    private val circeVersion = "0.14.1"
         |
         |    val core    = ivy"io.circe::circe-core:$circeVersion"
         |  }
         |""".stripMargin
    val expected =
      """|object Circe {
         |    private val circeVersion = "0.14.2"
         |
         |    val core    = ivy"io.circe::circe-core:$circeVersion"
         |  }
         |""".stripMargin
    val update = ("io.circe".g % Nel.of("circe-core".a) % "0.14.1" %> "0.14.2").group
    assertEquals(update.replaceVersionIn(original), Some(expected) -> UpdateHeuristic.original.name)
  }

This finds the original heuristic to do upgrades with, and the test passes. Steward would raise PRs for dependencies in this form.
Basically, our problem was version needs to be circeVersion for example.


I don't see information/examples about heuristics in the docs. I would have found this helpful - it is a little bit odd, right? Do we want a PR for some documentation on this?

@lefou
Copy link
Member

lefou commented Sep 1, 2022

One reason for missing updates could be the fact, that the Mill plugin of Scala Steward currently doesn't not support all Mill versions. You can find a more in-depth explanation in issue

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants