Skip to content

Commit

Permalink
WIP: check
Browse files Browse the repository at this point in the history
  • Loading branch information
eikek committed Sep 26, 2024
1 parent c95e685 commit 13af04d
Show file tree
Hide file tree
Showing 6 changed files with 165 additions and 29 deletions.
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,8 @@ jobs:
# - name: Coursier cache
# uses: coursier/cache-action@v6
- name: sbt ci ${{ github.ref }}
env:
README_BASE_REF: ${{ github.base_ref }}
run: sbt -mem 2048 ci
- name: Log in to Docker Hub
uses: docker/login-action@v2
Expand Down
19 changes: 13 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
# Renku Search


This provides the renku search services used to provide a efficient
search across entities in the Renku platform.
This provides the renku search services for efficientlyf searching
across entities in the Renku platform.

The engine backing the search functionality is SOLR (Lucene). The
index is created from data pulled from a Redis stream.
The engine backing the search functionality is
[SOLR](https://solr.apache.org) (Lucene). The index is created from
data pulled from a Redis stream.

There are two services provided: [search-api](#search-api) and
[search-provision](#search-provision).
Expand All @@ -17,6 +18,12 @@ elements out of the Redis stream, transforming it into solr documents
and the updates the index. It also creates the SOLR schema and
provides endpoints to trigger re-indexing.

This service is internal only and can be used by other service to
publish data that should be search- and discoverable.

The data in the index is received as a Redis message pulled from a
Redis stream.

### Messages

Messages in the stream must conform to the definitions in
Expand Down Expand Up @@ -69,7 +76,7 @@ is sent. Search supports
- `V1`
- `V2`

The `V` can be omitted.
The `V` can be omitted in the payload.

### Endpoints

Expand Down Expand Up @@ -197,7 +204,7 @@ openapi doc should be consulted.
"lastName": "Einstein",
"score": 2.1
},
"creationDate": "2024-09-25T14: 15: 23.779683809Z",
"creationDate": "2024-09-26T09: 22: 00.611173804Z",
"keywords": [
"data",
"science"
Expand Down
23 changes: 6 additions & 17 deletions build.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ releaseTagName := (ThisBuild / version).value

addCommandAlias(
"ci",
"; lint; compile; Test/compile; dbTests; readme/updateReadme; publishLocal"
"readme/readmeCheckModification; lint; compile; Test/compile; dbTests; readme/updateReadme; publishLocal"
)
addCommandAlias(
"lint",
Expand Down Expand Up @@ -444,16 +444,12 @@ lazy val searchCli = project
configValues % "compile->compile;test->test"
)

val updateReadme = inputKey[Unit]("Update readme")
lazy val readme = project
.in(file("modules/readme"))
.enablePlugins(MdocPlugin)
.enablePlugins(ReadmePlugin)
.settings(commonSettings)
.settings(
name := "search-readme",
libraryDependencies ++= Seq.empty,
mdocIn := (LocalRootProject / baseDirectory).value / "docs" / "readme.md",
mdocOut := (LocalRootProject / baseDirectory).value / "README.md",
scalacOptions :=
Seq(
"-feature",
Expand All @@ -464,17 +460,10 @@ lazy val readme = project
"-language:higherKinds",
"-Xkind-projector:underscores"
),
fork := true,
updateReadme := {
mdoc.evaluated
//copy query manual into source tree for easier discovery on github
IO.copyFile(
(searchQueryDocs / Docs / outputDirectory).value / "manual.md",
(LocalRootProject / baseDirectory).value / "docs" / "query-manual.md"
)
()
}
// TODO task to check that the copied files are not source-modified (unless its origin is modified, too)
readmeAdditionalFiles := Map(
// copy query manual into source tree for easier discovery on github
(searchQueryDocs / Docs / outputDirectory).value / "manual.md" -> "query-manual.md"
)
)
.dependsOn(
renkuRedisClient % "compile->compile;compile->test",
Expand Down
18 changes: 12 additions & 6 deletions docs/readme.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ import io.renku.search.provision.Routes
import io.renku.search.http.routes.OperationRoutes
import io.renku.redis.client.MessageBodyKeys
import io.renku.search.events.*
import io.bullet.borer.Json
import io.renku.search.config.ConfigValues
import io.renku.search.provision.SearchProvisionConfig
import io.renku.search.model.EntityType
Expand All @@ -15,11 +14,12 @@ import cats.effect.*
import cats.effect.unsafe.implicits.*
```

This provides the renku search services used to provide a efficient
search across entities in the Renku platform.
This provides the renku search services for efficientlyf searching
across entities in the Renku platform.

The engine backing the search functionality is SOLR (Lucene). The
index is created from data pulled from a Redis stream.
The engine backing the search functionality is
[SOLR](https://solr.apache.org) (Lucene). The index is created from
data pulled from a Redis stream.

There are two services provided: [search-api](#search-api) and
[search-provision](#search-provision).
Expand All @@ -31,6 +31,12 @@ elements out of the Redis stream, transforming it into solr documents
and the updates the index. It also creates the SOLR schema and
provides endpoints to trigger re-indexing.

This service is internal only and can be used by other service to
publish data that should be search- and discoverable.

The data in the index is received as a Redis message pulled from a
Redis stream.

### Messages

Messages in the stream must conform to the definitions in
Expand Down Expand Up @@ -72,7 +78,7 @@ is sent. Search supports
BulletPoints(SchemaVersion.values.toSeq.map(_.name.backticks))
```

The `V` can be omitted.
The `V` can be omitted in the payload.

### Endpoints

Expand Down
127 changes: 127 additions & 0 deletions project/ReadmePlugin.scala
Original file line number Diff line number Diff line change
@@ -0,0 +1,127 @@
import sbt._
import sbt.Keys._
import mdoc.MdocPlugin
import com.github.sbt.git.{GitPlugin, JGit}
import com.github.sbt.git.SbtGit.GitKeys
import org.eclipse.jgit.api._
import org.eclipse.jgit.diff.DiffEntry
import org.eclipse.jgit.treewalk.CanonicalTreeParser
import scala.jdk.CollectionConverters._

object ReadmePlugin extends AutoPlugin {
override def requires = MdocPlugin && GitPlugin
object autoImport {
val readmeUpdate = inputKey[Unit]("Update the root README.md")
val readmeBaseRef = settingKey[String]("The base ref to check modification against")
val readmeCheckModification = taskKey[Unit](
"Check the diff against the target branch for modifications to generated files"
)
val readmeAdditionalFiles = settingKey[Map[File, String]](
"Additional (generated) files to copy into the root docs/ folder"
)
}

import autoImport._
import MdocPlugin.autoImport._

override def projectSettings = Seq(
libraryDependencies ++= Seq.empty,
mdocIn := (LocalRootProject / baseDirectory).value / "docs" / "readme.md",
mdocOut := (LocalRootProject / baseDirectory).value / "README.md",
fork := true,
readmeAdditionalFiles := Map.empty,
readmeCheckModification := {
val bref = readmeBaseRef.value
val dir = (LocalRootProject / baseDirectory).value
val additional = readmeAdditionalFiles.value
val readmeIn = mdocIn.value
val readmeOut = mdocOut.value
val logger = streams.value.log
checkModification(logger, bref, dir, readmeIn, readmeOut, additional, dir / "docs")
},
readmeBaseRef := sys.env
.get("README_BASE_REF")
.map(_.trim)
.filter(_.nonEmpty)
.map(ref => s"${ref}^{tree}")
.getOrElse("HEAD^{tree}"),
readmeUpdate := {
mdoc.evaluated
val logger = streams.value.log
val additional = readmeAdditionalFiles.value
val addout = (LocalRootProject / baseDirectory).value / "docs"
additional.toList.foreach { case (src, name) =>
val dest = addout / name
logger.info(s"Copying $src -> $dest")
IO.copyFile(src, dest)
}
()
}
)

def checkModification(
logger: Logger,
baseRef: String,
sourceRoot: File,
readmeIn: File,
readmeOut: File,
additional: Map[File, String],
additionalOut: File
) = {
val git = JGit(sourceRoot).porcelain
val repo = git.getRepository
val base = repo.resolve(baseRef)
if (base eq null) sys.error(s"Cannot resolve base ref: $baseRef")
val p = new CanonicalTreeParser()
val reader = repo.newObjectReader()
p.reset(reader, base)
val diff = git
.diff()
.setOldTree(p)
.setShowNameOnly(true)
.call()
.asScala
.filter(e => e.getChangeType == DiffEntry.ChangeType.MODIFY)
.map(_.getOldPath)
.toSet

val checker = Checker(sourceRoot, diff, logger)

// check readme
checker.check(readmeIn, readmeOut)
// check others
additional.toList.foreach { case (src, targetName) =>
val dest = additionalOut / targetName
checker.check(src, dest)
}
()
}

def relativize(base: File, file: File): String =
base
.relativize(file)
.map(_.toString)
.getOrElse(
sys.error(
s"Cannot obtain path into repository for $file"
)
)

final case class Checker(sourceRoot: File, diff: Set[String], logger: Logger) {
def check(src: File, dest: File) = {
val srcPath = relativize(sourceRoot, src)
val destPath = relativize(sourceRoot, dest)
logger.info(s"Check modification $srcPath -> $destPath")
if (diff.contains(srcPath) && !diff.contains(destPath)) {
logger.error(s"You changed $srcPath but did not generate the output file")
sys.error("Checking failed")
}
if (diff.contains(destPath) && !diff.contains(srcPath)) {
logger.error(
s"You changed $destPath but this is a generated file, please change $srcPath instead"
)
sys.error("Checking failed")
}
}
}
}
5 changes: 5 additions & 0 deletions project/plugins.sbt
Original file line number Diff line number Diff line change
Expand Up @@ -27,3 +27,8 @@ addSbtPlugin("de.heikoseeberger" % "sbt-header" % "5.10.0")
addSbtPlugin("io.spray" % "sbt-revolver" % "0.10.0")
addSbtPlugin("org.scalameta" % "sbt-mdoc" % "2.5.3")
addSbtPlugin("org.scalameta" % "sbt-scalafmt" % "2.5.2")

// sbt-git comes with quite old jgit version
libraryDependencies ++= Seq(
"org.eclipse.jgit" % "org.eclipse.jgit" % "7.0.0.202409031743-r"
)

0 comments on commit 13af04d

Please sign in to comment.