From 416605cc6b65d2032c207189e8919758d6d9ea9a Mon Sep 17 00:00:00 2001 From: Loris Sauter Date: Mon, 1 Jul 2024 18:14:39 +0200 Subject: [PATCH 01/71] Format README --- README.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index b1747458..d87b42e8 100644 --- a/README.md +++ b/README.md @@ -8,16 +8,16 @@ vitrivr engine's data model, ingestion pipeline and retrieval logic have been re The project is set up as a multimodule Kotlin project: -| Module | Description | Maven Dependency | -|-----------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------| -| [`vitrivr-engine-core`](/vitrivr-engine-core) | The core library of the project, which provides basic interfaces & classes. | Yes | -| [`vtirivr-engine-query`](/vitrivr-engine-query) | Query / retrieval related extension to the core library with various retrievers and data manipulation operators. | Yes | -| [`vitrivr-engine-index`](/vitrivr-engine-index) | Indexing / ingestion related extension to the core library with various decoders and segmenters. | Yes | -| [`vitrivr-engine-module-cottontaildb`](/vitrivr-engine-plugin-cottontaildb) | The database driver for the [CottontailDB](https://github.com/vitrivr/cottontaildb) database, used for NNNS and other queries. | Yes | -| [`vitrivr-engine-module-features`](/vitrivr-engine-plugin-features) | Extension that contains specific indexing and retrieval implementations such as fulltext, colour, etc. | Yes | -| [`vitrivr-engine-plugin-m3d`](/vitrivr-engine-plugin-m3d) | Extension related to 3d model indexing and retrieval. Contains various feature modules and capability to process meshes. | Yes | -| [`vitrivr-engine-module-fes`](/vitrivr-engine-module-fes) | Extension that can be used to harnes feature extraction provided by an external ML model server. **Requires local generation of bindings:** `./gradlew :vitrivr-engine-module-fes:generateFESClient` | Yes | -| [`vitrivr-engine-server`](/vitrivr-engine-server) | A [Javalin](https://javalin.io) powered server providing an [OpenApi](https://openapis.org) [documented REST API](vitrivr-engine-server/doc/oas.json) for both, ingestion and querying and a CLI, essentially the runtime of the project | No | +| Module | Description | Maven Dependency | +|-----------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------| +| [`vitrivr-engine-core`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-core) | The core library of the project, which provides basic interfaces & classes. | Yes | +| [`vtirivr-engine-query`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-query) | Query / retrieval related extension to the core library with various retrievers and data manipulation operators. | Yes | +| [`vitrivr-engine-index`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-index) | Indexing / ingestion related extension to the core library with various decoders and segmenters. | Yes | +| [`vitrivr-engine-module-cottontaildb`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-plugin-cottontaildb) | The database driver for the [CottontailDB](https://github.com/vitrivr/cottontaildb) database, used for NNNS and other queries. | Yes | +| [`vitrivr-engine-module-features`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-plugin-features) | Extension that contains specific indexing and retrieval implementations such as fulltext, colour, etc. | Yes | +| [`vitrivr-engine-module-m3d`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-module-m3d) | Extension related to 3d model indexing and retrieval. Contains various feature modules and capability to process meshes. | Yes | +| [`vitrivr-engine-module-fes`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-module-fes) | Extension that can be used to harnes feature extraction provided by an external ML model server. **Requires local generation of bindings:** `./gradlew :vitrivr-engine-module-fes:generateFESClient` | Yes | +| [`vitrivr-engine-server`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-server) | A [Javalin](https://javalin.io) powered server providing an [OpenApi](https://openapis.org) [documented REST API](vitrivr-engine-server/doc/oas.json) for both, ingestion and querying and a CLI, essentially the runtime of the project | No | ## Getting Started: Usage @@ -40,7 +40,7 @@ configuration. #### Schema -vitrivr engine operates on the notion of _schema_, similarly to a database or a collection, +vitrivr-engine operates on the notion of _schema_, similarly to a database or a collection, essentially providing, among other things, a namespace. For this guide, we will have a single schema `sandbox`. From d84dcf217c8a5f55281dc3febc6bc04df9b8a482 Mon Sep 17 00:00:00 2001 From: Loris Sauter Date: Mon, 1 Jul 2024 18:17:33 +0200 Subject: [PATCH 02/71] Harmonised InputData's payload property name to 'data' --- .../engine/query/model/api/input/InputData.kt | 8 ++++---- .../vitrivr/engine/query/parsing/QueryParser.kt | 14 +++++++------- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/model/api/input/InputData.kt b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/model/api/input/InputData.kt index d149b197..342f9625 100644 --- a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/model/api/input/InputData.kt +++ b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/model/api/input/InputData.kt @@ -104,7 +104,7 @@ data class RetrievableIdInputData(val id: String, override val comparison: Strin * Cannot be converted to a [ContentElement] */ @Serializable -data class BooleanInputData(val value: Boolean, override val comparison: String? = "=="): InputData(){ +data class BooleanInputData(val data: Boolean, override val comparison: String? = "=="): InputData(){ override val type = InputType.BOOLEAN override fun toContent(): ContentElement<*> { throw UnsupportedOperationException("Cannot derive content from BooleanInputData") @@ -116,7 +116,7 @@ data class BooleanInputData(val value: Boolean, override val comparison: String? * Cannot be converted to a [ContentElement] */ @Serializable -data class NumericInputData(val value: Double, override val comparison: String? = "==") : InputData(){ +data class NumericInputData(val data: Double, override val comparison: String? = "==") : InputData(){ override val type = InputType.NUMERIC override fun toContent(): ContentElement<*> { throw UnsupportedOperationException("Cannot derive content from NumericInputData") @@ -128,7 +128,7 @@ data class NumericInputData(val value: Double, override val comparison: String? * Cannot be converted to a [ContentElement] */ @Serializable -data class DateInputData(val value: String, override val comparison: String? = "==") : InputData() { +data class DateInputData(val data: String, override val comparison: String? = "==") : InputData() { override val type = InputType.DATE override fun toContent(): ContentElement<*> {throw UnsupportedOperationException("Cannot derive content from DateInputData")} @@ -137,6 +137,6 @@ data class DateInputData(val value: String, override val comparison: String? = " */ fun parseDate():Date{ val formatter = SimpleDateFormat("YYYY-mm-dd", Locale.ENGLISH) - return formatter.parse(value) + return formatter.parse(data) } } diff --git a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt index 4423327c..828dc844 100644 --- a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt +++ b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt @@ -106,16 +106,16 @@ class QueryParser(val schema: Schema) { } is BooleanInputData -> { require(subfield.type == Type.BOOLEAN){"The given sub-field ${fieldAndAttributeName.first}.${fieldAndAttributeName.second}'s type is ${subfield.type}, which is not the expexted ${Type.BOOLEAN}"} - Value.Boolean(input.value) + Value.Boolean(input.data) } is NumericInputData -> { when(subfield.type){ - Type.DOUBLE -> Value.Double(input.value) - Type.INT -> Value.Int(input.value.toInt()) - Type.LONG -> Value.Long(input.value.toLong()) - Type.SHORT -> Value.Short(input.value.toInt().toShort()) - Type.BYTE -> Value.Byte(input.value.toInt().toByte()) - Type.FLOAT -> Value.Float(input.value.toFloat()) + Type.DOUBLE -> Value.Double(input.data) + Type.INT -> Value.Int(input.data.toInt()) + Type.LONG -> Value.Long(input.data.toLong()) + Type.SHORT -> Value.Short(input.data.toInt().toShort()) + Type.BYTE -> Value.Byte(input.data.toInt().toByte()) + Type.FLOAT -> Value.Float(input.data.toFloat()) else -> throw IllegalArgumentException("Cannot work with NumericInputData $input but non-numerical sub-field $subfield") } } From 7de5285f42feedceac52c236bf9ea4e0fba05f4b Mon Sep 17 00:00:00 2001 From: Loris Sauter Date: Wed, 3 Jul 2024 08:05:48 +0200 Subject: [PATCH 03/71] Repo cleanup: Removed old configs --- config-pipelines/image-pipeline.json | 52 ---------------------------- config-pipelines/video-pipeline.json | 51 --------------------------- 2 files changed, 103 deletions(-) delete mode 100644 config-pipelines/image-pipeline.json delete mode 100644 config-pipelines/video-pipeline.json diff --git a/config-pipelines/image-pipeline.json b/config-pipelines/image-pipeline.json deleted file mode 100644 index ce6ae026..00000000 --- a/config-pipelines/image-pipeline.json +++ /dev/null @@ -1,52 +0,0 @@ -{ - "schema": "vitrivr", - "context": { - "contentFactory": "InMemoryContentFactory", - "resolverFactory": "DiskResolver" - }, - "enumerator": { - "name": "ApiEnumerator", - "parameters": { - "depth": "5", - "mediaTypes": "IMAGE" - }, - "next": { - "name": "VideoDecoder", - "nextTransformer": { - "name": "PassthroughTransformer", - "nextSegmenter": { - "name": "FixedDurationSegmenter", - "parameters": { - "duration": "1", - "lookAheadTime": "1" - }, - "aggregators": [ - { - "name": "AverageImageContentAggregator", - "nextExtractor": { - "fieldName": "averagecolor", - "nextExporter": { - "name": "ThumbnailExporter", - "exporterName": "thumbnail", - - "nextExtractor": { - "fieldName": "file", - "nextExtractor": { - "fieldName": "time", - "nextExtractor": { - "fieldName": "clip", - "nextExtractor": { - "fieldName": "dino" - } - } - } - } - } - } - } - ] - } - } - } - } -} \ No newline at end of file diff --git a/config-pipelines/video-pipeline.json b/config-pipelines/video-pipeline.json deleted file mode 100644 index 2a2ab108..00000000 --- a/config-pipelines/video-pipeline.json +++ /dev/null @@ -1,51 +0,0 @@ -{ - "schema": "vitrivr", - "context": { - "contentFactory": "InMemoryContentFactory", - "resolverFactory": "DiskResolver" - }, - "enumerator": { - "name": "ApiEnumerator", - "parameters": { - "depth": "5", - "mediaTypes": "VIDEO" - }, - "next": { - "name": "VideoDecoder", - "nextTransformer": { - "name": "PassthroughTransformer", - "nextSegmenter": { - "name": "FixedDurationSegmenter", - "parameters": { - "duration": "1", - "lookAheadTime": "1" - }, - "aggregators": [ - { - "name": "AverageImageContentAggregator", - "nextExtractor": { - "fieldName": "averagecolor", - "nextExporter": { - "name": "ThumbnailExporter", - "exporterName": "thumbnail", - "nextExtractor": { - "fieldName": "file", - "nextExtractor": { - "fieldName": "time", - "nextExtractor": { - "fieldName": "clip", - "nextExtractor": { - "fieldName": "dino" - } - } - } - } - } - } - } - ] - } - } - } - } -} \ No newline at end of file From dcf3c7d881d966eb0294c7d266ea58911a4f7de6 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 5 Jul 2024 10:18:41 +0200 Subject: [PATCH 04/71] Refactors some basic functionality of the CottontailConnectionProvider into the AbstractConnectionProvider. --- .../database/AbstractConnectionProvider.kt | 47 +++++++++++ .../CottontailConnectionProvider.kt | 77 +++++++------------ ...vr.engine.core.database.ConnectionProvider | 1 + 3 files changed, 77 insertions(+), 48 deletions(-) create mode 100644 vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/AbstractConnectionProvider.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/resources/META-INF/services/org.vitrivr.engine.core.database.ConnectionProvider diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/AbstractConnectionProvider.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/AbstractConnectionProvider.kt new file mode 100644 index 00000000..231dbb5b --- /dev/null +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/AbstractConnectionProvider.kt @@ -0,0 +1,47 @@ +package org.vitrivr.engine.core.database + +import org.vitrivr.engine.core.database.descriptor.DescriptorProvider +import org.vitrivr.engine.core.model.descriptor.Descriptor +import kotlin.reflect.KClass + +/** + * Abstract implementation of the [ConnectionProvider] interface, which provides basic facilities to register [DescriptorProvider]s. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractConnectionProvider: ConnectionProvider { + /** A map of registered [DescriptorProvider]. */ + protected val registered: MutableMap, DescriptorProvider<*>> = HashMap() + + + init { + this.initialize() + } + + /** + * This method is called during initialization of the [AbstractConnectionProvider] and can be used to register [DescriptorProvider]s. + */ + abstract fun initialize() + + /** + * Registers an [DescriptorProvider] for a particular [KClass] of [Descriptor] with this [Connection]. + * + * This method is an extension point to add support for new [Descriptor]s to a pre-existing database driver. + * + * @param descriptorClass The [KClass] of the [Descriptor] to register [DescriptorProvider] for. + * @param provider The [DescriptorProvider] to register. + */ + override fun register(descriptorClass: KClass, provider: DescriptorProvider<*>) { + this.registered[descriptorClass] = provider + } + + /** + * Obtains an [DescriptorProvider] for a particular [KClass] of [Descriptor], that has been registered with this [ConnectionProvider]. + * + * @param descriptorClass The [KClass] of the [Descriptor] to lookup [DescriptorProvider] for. + * @return The registered [DescriptorProvider] . + */ + @Suppress("UNCHECKED_CAST") + override fun obtain(descriptorClass: KClass): DescriptorProvider = this.registered[descriptorClass] as DescriptorProvider +} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt index 68719230..415a02aa 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt @@ -1,9 +1,9 @@ package org.vitrivr.engine.plugin.cottontaildb +import org.vitrivr.engine.core.database.AbstractConnectionProvider import org.vitrivr.engine.core.database.Connection import org.vitrivr.engine.core.database.ConnectionProvider import org.vitrivr.engine.core.database.descriptor.DescriptorProvider -import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.descriptor.scalar.* import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor import org.vitrivr.engine.core.model.descriptor.struct.RasterDescriptor @@ -18,8 +18,8 @@ import org.vitrivr.engine.core.model.descriptor.vector.* import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.descriptors.scalar.ScalarDescriptorProvider import org.vitrivr.engine.plugin.cottontaildb.descriptors.string.StringDescriptorProvider -import java.util.* -import kotlin.reflect.KClass +import org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider +import org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescriptorProvider /** * Implementation of the [ConnectionProvider] interface for Cottontail DB. @@ -27,7 +27,7 @@ import kotlin.reflect.KClass * @author Ralph Gasser * @version 1.0.0 */ -class CottontailConnectionProvider: ConnectionProvider { +class CottontailConnectionProvider: AbstractConnectionProvider() { companion object { /** Name of the host parameter. */ @@ -49,36 +49,38 @@ class CottontailConnectionProvider: ConnectionProvider { /** The version of the [CottontailConnectionProvider]. */ override val version: String = "1.0.0" - /** List of registered [LinkedList]*/ - private val registered = mutableMapOf, DescriptorProvider<*>>( + /** + * This method is called during initialization of the [CottontailConnectionProvider] and can be used to register [DescriptorProvider]s. + */ + override fun initialize() { /* Scalar descriptors. */ - BooleanDescriptor::class to ScalarDescriptorProvider, - IntDescriptor::class to ScalarDescriptorProvider, - LongDescriptor::class to ScalarDescriptorProvider, - FloatDescriptor::class to ScalarDescriptorProvider, - DoubleDescriptor::class to ScalarDescriptorProvider, + this.register(BooleanDescriptor::class, ScalarDescriptorProvider) + this.register(IntDescriptor::class, ScalarDescriptorProvider) + this.register(LongDescriptor::class, ScalarDescriptorProvider) + this.register(FloatDescriptor::class, ScalarDescriptorProvider) + this.register(DoubleDescriptor::class, ScalarDescriptorProvider) /* String descriptor. */ - StringDescriptor::class to StringDescriptorProvider, + this.register(StringDescriptor::class, StringDescriptorProvider) /* Vector descriptors. */ - BooleanVectorDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescriptorProvider, - IntVectorDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescriptorProvider, - LongVectorDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescriptorProvider, - FloatVectorDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescriptorProvider, - DoubleVectorDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescriptorProvider, + this.register(BooleanVectorDescriptor::class, VectorDescriptorProvider) + this.register(IntVectorDescriptor::class, VectorDescriptorProvider) + this.register(LongVectorDescriptor::class, VectorDescriptorProvider) + this.register(FloatVectorDescriptor::class, VectorDescriptorProvider) + this.register(DoubleVectorDescriptor::class, VectorDescriptorProvider) /* Struct descriptor. */ - LabelDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider, - FileSourceMetadataDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider, - VideoSourceMetadataDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider, - TemporalMetadataDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider, - Rectangle2DMetadataDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider, - MediaDimensionsDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider, - SkeletonDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider, - RasterDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider, - MapStructDescriptor::class to org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider - ) + this.register(LabelDescriptor::class, StructDescriptorProvider) + this.register(FileSourceMetadataDescriptor::class, StructDescriptorProvider) + this.register(VideoSourceMetadataDescriptor::class, StructDescriptorProvider) + this.register(TemporalMetadataDescriptor::class, StructDescriptorProvider) + this.register(Rectangle2DMetadataDescriptor::class, StructDescriptorProvider) + this.register(MediaDimensionsDescriptor::class, StructDescriptorProvider) + this.register(SkeletonDescriptor::class, StructDescriptorProvider) + this.register(RasterDescriptor::class, StructDescriptorProvider) + this.register(MapStructDescriptor::class, StructDescriptorProvider) + } /** * Opens a new [CottontailConnection] for the given [Schema]. @@ -92,25 +94,4 @@ class CottontailConnectionProvider: ConnectionProvider { parameters[PARAMETER_NAME_HOST] ?: PARAMETER_DEFAULT_HOST, parameters[PARAMETER_NAME_PORT]?.toIntOrNull() ?: PARAMETER_DEFAULT_PORT ) - - /** - * Registers an [DescriptorProvider] for a particular [KClass] of [Descriptor] with this [Connection]. - * - * This method is an extension point to add support for new [Descriptor]s to a pre-existing database driver. - * - * @param descriptorClass The [KClass] of the [Descriptor] to register [DescriptorProvider] for. - * @param provider The [DescriptorProvider] to register. - */ - override fun register(descriptorClass: KClass, provider: DescriptorProvider<*>) { - this.registered[descriptorClass] = provider - } - - /** - * Obtains an [DescriptorProvider] for a particular [KClass] of [Descriptor], that has been registered with this [ConnectionProvider]. - * - * @param descriptorClass The [KClass] of the [Descriptor] to lookup [DescriptorProvider] for. - * @return The registered [DescriptorProvider] . - */ - @Suppress("UNCHECKED_CAST") - override fun obtain(descriptorClass: KClass): DescriptorProvider = this.registered[descriptorClass] as DescriptorProvider } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/resources/META-INF/services/org.vitrivr.engine.core.database.ConnectionProvider b/vitrivr-engine-module-pgvector/src/main/resources/META-INF/services/org.vitrivr.engine.core.database.ConnectionProvider new file mode 100644 index 00000000..497e745a --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/resources/META-INF/services/org.vitrivr.engine.core.database.ConnectionProvider @@ -0,0 +1 @@ +org.vitrivr.engine.database.pgvector.PgVectorConnectionProvider \ No newline at end of file From 82c4ef035af556ce9589bec3a7f86773774aa0f4 Mon Sep 17 00:00:00 2001 From: Loris Sauter Date: Fri, 5 Jul 2024 14:54:43 +0200 Subject: [PATCH 05/71] README-ified the README --- README.md | 1037 +++------------------------------------- images/vengine-256.png | Bin 0 -> 12934 bytes images/vengine-512.png | Bin 0 -> 27718 bytes images/vitrivr_256.png | Bin 0 -> 9551 bytes images/vitrivr_512.png | Bin 0 -> 76376 bytes 5 files changed, 55 insertions(+), 982 deletions(-) create mode 100755 images/vengine-256.png create mode 100755 images/vengine-512.png create mode 100755 images/vitrivr_256.png create mode 100755 images/vitrivr_512.png diff --git a/README.md b/README.md index d87b42e8..4f4a8303 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,52 @@ -# vitrivr engine + + + + +
+
+ + vitrivr log


+ vitrivr-engine logo + +
+
+ +[![GitHub Release](https://img.shields.io/github/release/vitrivr/vitrivr-engine?include_prereleases=&sort=semver&color=blue&style=for-the-badge&label=Release)](https://github.com/vitrivr/vitrivr-engine/releases) +[![License](https://img.shields.io/badge/License-MIT-blue?style=for-the-badge)](#license) +[![issues - vitrivr-engine](https://img.shields.io/github/issues/vitrivr/vitrivr-engine?style=for-the-badge)](https://github.com/vitrivr/vitrivr-engine/issues) [vitrivr](https://vitrivr.org)'s next-generation retrieval engine. + +[Read The Docs](https://github.com/vitrivr/vitrivr-engine/wiki) + +
+ +## vitrivr-engine + +vitrivr-engine is [vitrivr](https://vitrivr.org)'s next generation retrieval engine with a flexible, modular architecture. Based on the experiences with its predecessor, [Cineast](https://github.com/vitrivr/cineast), -vitrivr engine's data model, ingestion pipeline and retrieval logic have been reworked. +vitrivr engine's data model, ingestion pipeline and retrieval logic have been reworked from the ground. +Essentially, vitrivr-engine enables the analysis (i.e. ingestion) and querying (i.e. retrieval ) of +multimedia data. + +## Built With + +* [Kotlin](https://kotlinlang.org) for the JVM, e.g. [OpenJDK](https://openjdk.org/) +* [OpenApi](https://www.openapis.org/) +* [CottontailDB](https://github.com/vitrivr/CottontailDB) +* ... and more ... + +## Getting Started + +See [Getting Started](https://github.com/vitrivr/vitrivr-engine/wiki/Getting-Started) + +## Usage + +See [Example](https://github.com/vitrivr/vitrivr-engine/wiki/Example) ## Project Structure -The project is set up as a multimodule Kotlin project: +The project is set up as a multi-module Kotlin project: | Module | Description | Maven Dependency | |-----------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------| @@ -19,987 +59,20 @@ The project is set up as a multimodule Kotlin project: | [`vitrivr-engine-module-fes`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-module-fes) | Extension that can be used to harnes feature extraction provided by an external ML model server. **Requires local generation of bindings:** `./gradlew :vitrivr-engine-module-fes:generateFESClient` | Yes | | [`vitrivr-engine-server`](https://github.com/vitrivr/vitrivr-engine//vitrivr-engine-server) | A [Javalin](https://javalin.io) powered server providing an [OpenApi](https://openapis.org) [documented REST API](vitrivr-engine-server/doc/oas.json) for both, ingestion and querying and a CLI, essentially the runtime of the project | No | -## Getting Started: Usage - -vitrivr engine is a Kotlin project and hence requires a JDK (e.g. [OpenJDK](https://openjdk.org/)) to properly run. -Furthermore, we use [Gradle](https://gradle.org) in order to facilitate the building and deployment -through the Gradle wrapper. - -In the context of retrieval, often times a distinction of _indexing_ / _ingestion_ (also known as _offline phase_) -and _querying_ / _retrieval_ (also known as _online phase_) is made. -While the former addresses (multimedia) content to be analysed and indexed, i.e. made ready for search, is the latter's purpose to -search within the previously built index. - -### Indexing / Ingestion - -The most essential prerequisite for the ingestion is the existence of multimedia content. -For the sake of this example, let's assume the existence of such multimedia content in the form of image and video files. - -Also, since vitrivr engine is highly configurable, the first few steps involve the creation of a suitable -configuration. - -#### Schema - -vitrivr-engine operates on the notion of _schema_, similarly to a database or a collection, -essentially providing, among other things, a namespace. -For this guide, we will have a single schema `sandbox`. - -Create a config file `sandbox-config.json` with one named schema in it: - -```json -{ - "schemas": [{ - "name": "sandbox" - }] -} -``` - -#### Database Connection - -The database is an important component of the system, since it allows for the persistence and lookup of descriptors. -The default database used by `vitrivr-engine` is [CottontailDB](https://github.com/vitrivr/cottontaildb) This guide assumes a running [CottontailDB](https://github.com/vitrivr/cottontaildb) -instance on the same machine on the default port `1865`. - ---- -**NOTE this requires [Cottontail 0.16.5](https://github.com/vitrivr/cottontaildb/releases/tag/0.16.5) or newer** ---- - -We address the database with the [`ConnectionConfig`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ConnectionConfig.kt): - -```json -{ - "database": "CottontailConnectionProvider", - "parameters": { - "Host": "127.0.0.1", - "port": "1865" - } -} -``` - -We add the cottontail connection to the schema's connection property: - -```json -{ - "schemas": [{ - "name": "sandbox", - "connection": { - "database": "CottontailConnectionProvider", - "parameters": { - "Host": "127.0.0.1", - "port": "1865" - } - } - }] -} -``` - -#### Schema Analyser - -The [`Analyser`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Analyser.kt) -performs analysis to derive a [`Descriptor`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt). -In other words, the _analyser_ produces a _descriptor_ which represents the media content analysed. -However, this is only for _indexing_ / _ingestion_. During _querying_ / _retrieval_ time, -the _analyser_ queries the underlying storage layer to perform a query on said _descriptors_. - -#### Schema Field Configuration - -A schema consists of unique-named _fields_, that have to be backed by an [`Analyser`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Analyser.kt), -essentially representing a specific [`Descriptor`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt). -This is configured using the [`FieldConfig`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/FieldConfig.kt): - -```json -{ - "name": "uniqueName", - "factory": "FactoryClass", - "parameters":{ - "key": "value" - } -} -``` - -For images (and video frames), it might be worthwhile to use the average colour for representation purposes. -The built-in [`AverageColor`](/vitrivr-engine-plugin-features/src/main/kotlin/org/vitrivr/engine/base/features/averagecolor/AverageColor.kt) -analyser can be facilitated for this endeavour. -To use it, we specifically craft a corresponding _field config_: - -```json -{ - "name": "averagecolor", - "factory": "AverageColor" -} -``` - -There are no additional parameters, unlike, for instance, an [`ExternalAnalyser`](/vitrivr-engine-plugin-features/src/main/kotlin/org/vitrivr/engine/base/features/external/ExternalAnalyser.kt), -which requires the parameter `host` with an endpoint as value. - -For analysers that require a running [Feature Extraction Server](https://github.com/faberf/feature-extraction-server) (FES), the `host` parameter is required. Additionally, the `model` parameter may be used to specify a non-default model which should execute the task. Of course, this requires that the FES has the necessary plugins installed. See the [FES documentation](https://github.com/faberf/feature-extraction-server) for more information. - -- For [`ASR`](/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/ASR.kt) the analyser will perform automatic speech recognition on the audio content. -- For [`OCR`](/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/OCR.kt) the analyser will perform optical character recognition on the image content. -- For [`DenseEmbedding`](/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/DenseEmbedding.kt) the analyser will embed text / images as float vectors. Additionally, the `length` parameter is required to specify the length of the embedding. -- For [`ImageCaption`](/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/ImageCaption.kt) the analyser will generate a caption for the image content. Optionally, a `prompt` parameter can be used to specify a prompt for the caption generation. For example, the prompt could have the form `"Question: What is in the image? Answer:"`. -- For [`ImageClassification`](/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/ImageClassification.kt) the analyser will classify the image content. Additionally, the `classes` parameter is required, which should contain the classes to classify the image into, separated by commas. Optionally, the `top_k` and `threshold` parameters can be used to specify the number of top classes to return and the threshold for the classification. - - -Other fields are for (technical) metadata such as the [`FileSourceMetadata`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/file/FileSourceMetadata.kt), -which additionally stores the file's path and size. - -For exif metadata, the [`ExifMetadata`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadata.kt) Analyser can be used. For each exif tag that should be included, a parameter with the name "{EXIF_DIRECTORY_NAME}_{TAG_NAME}" must be set to a type. Keys that do not match an exif tag via the aforementioned pattern are interpreted to be custom metadata tags that are stored in the exif UserComment tag in JSON format. Each parameter corresponds to a sub-field. Here is an example with custom "time_zone" metadata: -```json - { - "name": "exif", - "factory": "ExifMetadata", - "parameters": { - "ExifSubIFD_FocalLength": "INT", - "ExifSubIFD_ApertureValue": "FLOAT", - "ExifSubIFD_DateTimeOriginal": "DATETIME", - "ExifSubIFD_MeteringMode": "STRING", - "time_zone": "STRING" - } - } -``` -For extraction, the exif UserComment of images might look like this: -```json -{"time_zone": "Europe/Berlin", "hours_awake": 12} -``` - -Currently, there is no list of available fields and analysers, therefore a quick look into the code -reveals those existent. For basic (metadata), see in [the core module](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/), -for content-based features, see in [the features' module](/vitrivr-engine-plugin-features/src/main/kotlin/org/vitrivr/engine/base/features/). - -In this guide, we use the _average colour_ and _file source_ fields, which results in the (currently) following -configuration file: - -```json -{ - "schemas": [{ - "name": "sandbox", - "connection": { - "database": "CottontailConnectionProvider", - "parameters": { - "Host": "127.0.0.1", - "port": "1865" - } - }, - "fields": [ - { - "name": "averagecolor", - "factory": "AverageColor" - }, - { - "name": "file", - "factory": "FileSourceMetadata" - } - ] - }] -} -``` - -#### Schema Resolver Configuration - -Some data is stored e.g. on disk during extraction, which later will also be required during query time, -therefore the [`Resolver`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/resolver/Resolver.kt) -is configured as the [`ResolverConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/ResolverConfig.kt) -on the schema with a unique name. - -The [`ResolverConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/ResolverConfig.kt) describes such a configuration: - -```json -{ - "factory": "FactoryClass", - "parameters": { - "key": "value" - } -} -``` - -Specifically, the [`DiskResolver`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/resolver/impl/DiskResolver.kt) is implemented and configured as such: - -```json -{ - "factory": "DiskResolver", - "parameters": { - "location": "./thumbnails/vitrivr" - } -} -``` - -Therefore, the _schema_ config is expanded with the _disk resolver_, named `disk`: - -```json -{ - "schemas": [{ - "name": "sandbox", - "connection": { - "database": "CottontailConnectionProvider", - "parameters": { - "Host": "127.0.0.1", - "port": "1865" - } - }, - "fields": [ - { - "name": "averagecolor", - "factory": "AverageColor" - }, - { - "name": "file", - "factory": "FileSourceMetadata" - } - ] - }], - "resolvers": { - "disk": { - "factory": "DiskResolver", - "parameters": { - "location": "./thumbnails/vitrivr" - } - } - } -} -``` - -#### Schema Exporter Configuration - -In the context of images and videos, having thumbnails is desirable, which can be generated -during ingestion with the configuration of an _exporter_. -Generally speaking, an _exporter_ exports an artifact based on the media content. - -The [`ExporterConfig`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ExporterConfig.kt) -describes such a configuration: - -```json -{ - "name": "uniqueName", - "factory": "FactoryClass", - "resolverName": "disk", - "parameters": { - "key": "value" - } -} -``` - -Specifically, the [`ThumbnailExporter`](/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/exporters/ThumbnailExporter.kt), -can be configured as follows, which references a [`DiskResolver`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/resolver/impl/DiskResolver.kt) named `disk`, see the previous section. - -```json -{ - "name": "thumbnail", - "factory": "ThumbnailExporter", - "resolverName": "disk", - "parameters": { - "maxSideResolution": "400", - "mimeType": "JPG" - } -} -``` - -Resulting in the following schema config: - -```json -{ - "schemas": [{ - "name": "sandbox", - "connection": { - "database": "CottontailConnectionProvider", - "parameters": { - "Host": "127.0.0.1", - "port": "1865" - } - }, - "fields": [ - { - "name": "averagecolor", - "factory": "AverageColor" - }, - { - "name": "file", - "factory": "FileSourceMetadata" - } - ], - "resolvers": { - "disk": { - "factory": "DiskResolver", - "parameters": { - "location": "./thumbnails/vitrivr" - } - } - }, - "exporters": [ - { - "name": "thumbnail", - "factory": "ThumbnailExporter", - "resolverName": "disk", - "parameters": { - "maxSideResolution": "400", - "mimeType": "JPG" - } - } - ] - }] -} -``` - -#### Extraction Pipeline Configuration - -In order to effectively support a specific _ingestion_ / _indexing_, we have to provide -a reference to the [`IndexConfig`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/IndexConfig.kt), -which is configured as a [`PipelineConfig`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/PipelineConfig.kt) within the schema config: - -```json -{ - "name": "pipelineName", - "path": "path/to/vbs-config-pipeline.json" -} -``` - -We will create said _index config_ later as `sandbox-pipeline.json`, hence, our schema config -is as follows: - -```json -{ - "schemas": [{ - "name": "sandbox", - "connection": { - "database": "CottontailConnectionProvider", - "parameters": { - "Host": "127.0.0.1", - "port": "1865" - } - }, - "fields": [ - { - "name": "averagecolor", - "factory": "AverageColor" - }, - { - "name": "file", - "factory": "FileSourceMetadata" - } - ], - "exporters": [ - { - "name": "thumbnail", - "factory": "ThumbnailExporter", - "resolver": { - "factory": "DiskResolver", - "parameters": { - "location": "./thumbnails/sandbox" - } - }, - "parameters": { - "maxSideResolution": "400", - "mimeType": "JPG" - } - } - ], - "extractionPipelines": [ - { - "name": "sandboxpipeline", - "path": "./sandbox-pipeline.json" - } - ] - }] -} -``` - -#### Index Pipeline Configuration - -Let's create a new file `sandbox-pipeline.json` right next to the `sandbox-config.json` in the root of the project. -This file will contain the [`IngestionConfig`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/IngestionConfig.kt). - -In order to address (reference) our schema, we reference it in our index config and provide a _context_ as well as an _enumerator_: - -```json -{ - "schema": "sandbox", - "context": { - - }, - "enumerator": { - - } -} -``` - -#### Index Context Configuration - -An [`IngestionContextConfig`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/context/IngestionContextConfig.kt) -is used to specify the _context_, additional information to the media data. -Specifically, a [`Resolver`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/resolver/Resolver.kt), `disk`, is referenced by its name from the _schema_ configuration. - -```json -{ - "contentFactory": "InMemoryContentFactory", - "resolverName": "disk", - "global": { - "key": "global-value" - }, - "local": { - "operator-name": { - "key": "local-value" - } - } -} -``` - -The [`Context`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/context/Context.kt) has two properties: `global` and `local`, which -enable local and global configuration in the form of key-value pairs. -However, `local`ly defined key-value pairs override their `global`ly defined counterpart. -In the above example, for the _operator_ (see further below), the key-value pair with the key `key` is also locally defined, -resulting in the value `local-value` to be used. However, a separate operator named other than `operator-name`, would get the value `global-value` -for the key `key`, as no local value is defined. -We refer to this part of the context, to the _context-parameters_. - -The example index pipeline configuration then, the current context looks as follow: - -```json -{ - "schema": "sandbox", - "context": { - "contentFactory": "InMemoryContentFactory", - "resolverName": "disk" - }, - "enumerator": { - - } -} -``` - -There are these special sections for the context-parameters: - -* `content` - For the local context-parameters for the _content factory_. - -#### Index Operators Configuration - -Next up, we declare a list of [operators](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/Operator.kt) in the form -of [`OperatorConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/operator/OperatorConfig.kt)s. -These _operators_ must have a unique name in the `operators` property of the [`IngestionConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/IngestionConfig.kt): -These names are used in the local context-parameters to configure them. - -```json -{ - "schema": "sandbox", - "context": { - "contentFactory": "InMemoryContentFactory", - "resolverName": "disk", - "local": { - "enumerator": { - "path": "./sandbox/imgs", - "mediaTypes": "IMAGE;VIDEO", - "depth": "1" - }, - "myoperator1": { - "key1": "a-value", - "key2": "1234" - } - } - }, - "operators": { - "myoperator1": {}, - "myoperator2": {} - } -} -``` - -There are different _types_ of operators: - -* [`Enumerator`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Enumerator.kt) enumerate and emit `Source`s to ingest from some data source (e.g., the file system). -* [`Decoder`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Decoder.kt) decode a `Source`s such that the `Content` becomes is available for ingestion (e.g., a video or image decoder). Create and emit `Retrievable`s. -* [`Transformer`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Segmenter.kt) transform incoming `Retrievable`s and emit 1 to n output `Retrievable`s (e.g., a video segmenter). Result in a 1:1 or 1:n mapping, depending - on the implementation. -* [`Extractor`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Extractor.kt) extract information from the `Retrievable` and their `Content` and typically append a `Descriptor` before emitting it. -* [`Exporter`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Exporter.kt) process `Retrievable`s and store external representation thereof (e.g., a preview image). -* [`Aggregator`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Aggregator.kt) aggregate incoming `Retrievable`s and emit - one [`Retrievable`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/retrievable/Retrievable.kt)s, resulting in an n:1 mapping. - -A minimal extraction pipeline requires at least an `Enumerator` and a `Decoder`. - -Notably, `Extractor`s are mostly backed by a schema's field, indicating that the `Descriptor` is being persisted. One can also instantiate `Extractor`s through their -factory, in which case the associated `Descriptor` is transient. `Exporter`s are also referenced by name from the _schema_. - -In the following, we briefly introduce these configurations: - -##### Index Operator Configuration: Enumerator - -The _enumerator_ enumerates the content to index and provides it to the indexing pipeline. -It is described with a [`EnumeratorConfig](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/operator/OperatorConfig.kt). - -Requires the property `mediaTypes`, a list of [`MediaType`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/source/MediaType.kt)s. - -```json -{ - "type": "ENUMERATOR", - "factory": "FactoryClass", - "mediaTypes": ["IMAGE","VIDEO"] -} -``` - -Currently implemented enumerators are found [in the index module](/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/enumerate), -of which we will use the [`FileSystemEnumerator`](/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/enumerate/FileSystemEnumerator.kt). - -The configuration **requires** the context-parameter `path`, the path to the folder containing multimedia content. -Essentially, for still images, use `IMAGE` and for videos `VIDEO`. -Additional context-parameters are `skip` (how many files should be skipped), `limit` (how many files should at max be enumerated over) -and `depth` (the depth to traverse the file system, `1` stands for current folder only, `2` for sub-folders, `3` for sub-sub-folders, ...). -Let's assume we do have in the root project a folder `sandbox`, with two sub-folders `imgs` and `vids`: - -``` -/sandbox - | - - /imgs - | - - img1.png - | - - img2.png - | - - /vids - | - - vid1.mp4 -``` - -For an image only ingestion, we could set up the configuration as follows (`skip` and `limit` have sensible default values of `0` and `Integer.MAX_VALUE`, respectively): - -**Context**: -```json -{ - "contentFactory": "InMemoryContentFactory", - "resolverName": "disk", - "local": { - "enumerator": { - "path": "./your/media/path", - "depth": "1" - } - } -} -``` - -**Enumerator**: -```json -{ - "type": "ENUMERATOR", - "factory": "FileSystemEnumerator", - "mediaTypes": [ - "IMAGE", "VIDEO" - ] -} -``` - -##### Index Operator Configuration: Decoder - -The [`DecoderConfig`](/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/operator/OperatorConfig.kt) -describes how the media content is decoded. - -```json -{ - "type": "", - "factory": "DecoderClass" -} -``` - -Available decodes can be found [in the index module](/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/decode). -Since we work with images in this tutorial, we require the [`ImageDecoder`](/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/decode/ImageDecoder.kt): - -```json -{ - "name": "ImageDecoder" -} -``` - -##### Index Operators Configuration: Segmenter - -A [`Segmenter`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Segmenter.kt) is a 1:n operator, -its [`OperatorConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/operator/OperatorConfig.kt) looks as follows: - -```json -{ - "type": "SEGMENTER", - "factory": "FactoryClass" -} -``` - -The `type` property is mandatory, equally so the `factory`, which has to point to a [`SegmenterFactory`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/SegmenterFactory.kt) implementation. -The context-parameters are optional and implementation dependent. - -See [implementations](vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/segment/) - -##### Index Operators Configuration: Transformer - -A [`Transformer`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Transformer.kt) is a 1:1 operator, -its [`OperatorConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/operator/OperatorConfig.kt) looks as follows: - -```json -{ - "type": "TRANSFORMER", - "factory": "FactoryClass" -} -``` - -The `type` property is mandatory, equally so the `factory`, which has to point to a [`TransformerFactory`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/TransformerFactory.kt) implementation. -The context-parameters are optional and implementation dependent. - -See [implementations](vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/transform) - -##### Index Operators Configuration: Exporter - -A [`Exporter`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Exporter.kt) is a 1:1 operator, -its [`OperatorConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/operator/OperatorConfig.kt) looks as follows: - -```json -{ - "type": "EXPORTER", - "exporterName": "name-from-schema" -} -``` - -The `type` property is mandatory, equally so the `exporterName`, which has to point to an `Exporter` defined on the _schema_. -The context-parameters are optional and implementation dependent and override those present in the _schema_ configuration. - -See [implementations](vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/exporters) - -##### Index Operators Configuration: Extractor - -A [`Extractor`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Extractor.kt) is a 1:1 operator, -its [`OperatorConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/operator/OperatorConfig.kt) looks as follows: - -```json -{ - "type": "EXTRACTOR", - "fieldName": "name-from-schema" -} -``` - -The `type` property is mandatory, equally so the `fieldName`, which has to point to a _field_ as defined on the _schema_. - -See [implementations](vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/base/features/) - -##### Index Operators Configuration: Aggregator - -A [`Aggregator`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/Aggregator.kt) is a 1:n operator, -its [`OperatorConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/operator/OperatorConfig.kt) looks as follows: - -```json -{ - "type": "AGGREGATOR", - "factory": "FactoryClass" -} -``` - -The `type` property is mandatory, equally so the `factory`, which has to point to a [`AggregatorFactory`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/ingest/AggregatorFactory.kt) implementation. -The context-parameters are optional and implementation dependent. - -See [implementations](vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/aggregators) - -#### Index Operations Configuration: The Pipeline - -So far, we only have _declared_ the operators, with the `operations` property, we define the ingestion pipeline as a tree in the form of -[`OperationsConfig`](vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/ingest/operation/OperationsConfig.kt): - -```json -{ - "schema": "sandbox", - "context": { - "contentFactory": "InMemoryContentFactory", - "resolverName": "disk", - "local": { - "enumerator": { - "path": "./sandbox/imgs", - "depth": "1" - }, - "myoperator": { - "key1": "a-avlue", - "key2": "1234" - } - } - }, - "operators": { - "myoperator": {}, - "myoperator1": {}, - "myoperator2": {} - }, - "operations": { - "myOperation": { - "operator": "myoperator" - }, - "myOperation1": { - "operator": "myoperator1", "inputs": ["myOperation"] - }, - "myOperation2": { - "operator": "myoperator2", "inputs": ["myOperation"] - } - }, - "output": ["myOperation2"] -} -``` - -Specifically, the `operator` property must point to a previously declared _operator_ and -the entries in the `inputs` property must point to an _operation_ with that name. -The `output` property of the ingest config then specifies the _operation_ whose results are persisted. - -Currently, there are the following rules to build such a pipeline: - -**Pipeline Rules:** - -1. The first _operation_ **must** be a `ENUMERATOR` -2. Following an `ENUMERATOR`, there **must** come a `DECODER` -3. Following a `DECODER`, there **must** either be a `TRANSFORMER` or `SEGMENTER` -4. `TRANSFORMER`s and `SEGMENTER`s can be daisy-chained -5. A `SEGMENTER` must be followed by one or more `AGGREGATOR`s, multiple `AGGREGATORS` results in branching. -6. An `AGGREGATOR` must be followed by either a `EXTRACTOR` or `EXPORTER` -7. `EXPORTER`s and `EXTRACTOR`s can be daisy-chained -8. The end or the ends, in case of branching, must be of type `EXPORTER` or `EXTRACTOR`. - -Notably, currently multiple `ENUMERATORS` are treated as separate trees, since merging is not yet supported. - -One example, based on the _schema_ further above (without branching), might look as follows: - -```json -{ - "schema": "sandbox", - "context": { - "contentFactory": "InMemoryContentFactory", - "resolverName": "disk", - "local": { - "enumerator": { - "path": "./sandbox/imgs", - "depth": "1" - }, - "thumbs": { - "maxSideResolution": "350", - "mimeType": "JPG" - } - } - }, - "operators": { - "fsenumerator": { - "type": "ENUMERATOR", - "factory": "FileSystemEnumerator", - "mediaTypes": ["IMAGE"] - }, - "decoder": { - "type": "DECODER", - "factory": "ImageDecoder" - }, - "pass": { - "type": "SEGMENTER", - "factory": "PassThroughSegmenter" - }, - "allContent": { - "type": "AGGREGATOR", - "factory": "AllContentAggregator" - }, - "avgColor": { - "type": "EXTRACTOR", - "fieldName": "averagecolor" - }, - "thumbs": { - "type": "EXPORTER", - "exporterName": "thumbnail" - }, - "fileMeta": { - "type": "EXTRACTOR", - "fieldName": "file" - } - }, - "operations": { - "stage2": {"operator": "pass", "inputs": ["stage1"]}, - "stage0": {"operator": "fsenumerator"}, - "stage1": {"operator": "decoder", "inputs": ["stage0"]}, - "stage3": {"operator": "allContent", "inputs": ["stage2"]}, - "stage4": {"operator": "avgColor", "inputs": ["stage3"]}, - "stage5": {"operator": "thumbs", "inputs": ["stage4"]}, - "stage6": {"operator": "fileMeta", "inputs": ["stage5"]} - }, - "output": ["stage6"] -} -``` - -Here, the linear order of the pipeline's operator is: `fsenumerator` -> `decoder` -> `pass` -> `allContent` -> `avgColor` -> `thumbs` -> `fileMeta`. -Note that there are context-parameters defined for the `thumbs` exporter. -Also, note that the `output` property is set up, such that the `stage6` operation is persisted. - -#### Branching and Merging - -By defining the operations accordingly, there are two thing that can happen implicitly. - -**Branching**: If an `operations` is used as input for multiple other `operations`, this results in a branching. This is handled automatically by wrapping the associated `Operator` in a `BroadcastOperator`. - -**Merging**: If an `operation` has multiple inputs, this results in a merging, which combines multiple flows of `Retrievable`s into a single flow. The merging strategy (`MergeType`) must be specified explicitly in the `operation`. - -Currently, `vitrivr-engine` supports three type of merging strategies: - -| MergeType | Description | -|-----------|----------------------------------------------------------------------------------------------------------------------| -| `MERGE` | Merges the `Retrievable`s from the input operations in order the arrive. No deduplication and ordering is performed. | -| `COMBINE` | Merges `Retrievable`s from the input operations and emits a `Retrievable`, once it was received on every input. | -| `CONCAT` | Collects `Retrievable`s from the incoming flows in order of occurence, i.e., operation 1, then operation 2 etc. | - -#### Complete Sandbox Configuration - -After following above's guide on how to build your _schema_ config and your _index pipeline_ config, -the files should be similar as follows. - -The **schema** config: - -```json -{ - "schemas": [{ - "name": "sandbox", - "connection": { - "database": "CottontailConnectionProvider", - "parameters": { - "Host": "127.0.0.1", - "port": "1865" - } - }, - "fields": [ - { - "name": "averagecolor", - "factory": "AverageColor" - }, - { - "name": "file", - "factory": "FileSourceMetadata" - } - ], - "exporters": [ - { - "name": "thumbnail", - "factory": "ThumbnailExporter", - "resolver": { - "factory": "DiskResolver", - "parameters": { - "location": "./thumbnails/sandbox" - } - }, - "parameters": { - "maxSideResolution": "400", - "mimeType": "JPG" - } - } - ], - "extractionPipelines": [ - { - "name": "sandboxpipeline", - "path": "./sandbox-pipeline.json" - } - ] - }] -} -``` - -The **pipeline** config: - -```json -{ - "schema": "sandbox", - "context": { - "contentFactory": "InMemoryContentFactory", - "resolverName": "disk", - "local": { - "fsenumerator": { - "path": "./sandbox/imgs", - "depth": "1" - }, - "thumbs": { - "path": "./sandbox/thumbnails", - "maxSideResolution": "350", - "mimeType": "JPG" - } - } - }, - "operators": { - "fsenumerator": { - "type": "ENUMERATOR", - "factory": "FileSystemEnumerator", - "mediaTypes": ["IMAGE","VIDEO"] - }, - "decoder": { - "type": "DECODER", - "factory": "ImageDecoder" - }, - "pass": { - "type": "SEGMENTER", - "factory": "PassThroughSegmenter" - }, - "allContent": { - "type": "AGGREGATOR", - "factory": "AllContentAggregator" - }, - "avgColor": { - "type": "EXTRACTOR", - "fieldName": "averagecolor" - }, - "thumbs": { - "type": "EXPORTER", - "exporterName": "thumbnail" - }, - "fileMeta": { - "type": "EXTRACTOR", - "fieldName": "file" - } - }, - "operations": { - "stage2": {"operator": "pass", "inputs": ["stage1"]}, - "stage0": {"operator": "fsenumerator"}, - "stage1": {"operator": "decoder", "inputs": ["stage0"]}, - "stage3": {"operator": "allContent", "inputs": ["stage2"]}, - "stage4": {"operator": "avgColor", "inputs": ["stage3"]}, - "stage5": {"operator": "thumbs", "inputs": ["stage4"]}, - "stage6": {"operator": "fileMeta", "inputs": ["stage5"]} - }, - "output" : ["stage6"] -} -``` - ---- - -#### Starting the indexing pipeline - -To start the actual pipeline, we start [the server module](/vitrivr-engine-server)'s [`Main`](vitrivr-engine-server/src/main/kotlin/org/vitrivr/engine/server/Main.kt) -with the path to the schema configuration as argument. - -For this to work you either build the stack or you use an IDE. - -1. Then, when the server is running, we have to first initialise the database (since our schema is named `sandbox`): - -``` -sandbox init -``` - -2. The extraction is started via the CLI by calling: -``` -sandbox extract -c sandbox-pipeline.json -``` - -Which should result in logging messages that confirm the usage of our ThumbnailExporter (including its parameters) and the message: -``` -Started extraction job with UUID -``` - -3. The server (by default) provides an [OpenAPI swagger ui](http://localhost:7070/swagger-ui) with which the job status can be queried. -The same can be achieved by this cURL command, where `` is the UUID printed above (and again, we have named our schema `sandbox`, hence the sandbox path: - -```bash -curl -X 'GET' \ - 'http://localhost:7070/api/sandbox/index/' \ - -H 'accept: application/json' -``` - -### Querying / Retrieval - -## Getting Started: Development - -This is a Gradle-powered Kotlin project, we assume prerequisites are handled accordingly. +## Contributing -1. Generate the OpenApi client code by executing the (top-level) `generateOpenApi` gradle task -2. Start developing +We welcome contributors. Please fork the repo and open a pull-request with your work. +A good starting point are the 'good first issue' issues. -If you develop another module (plugin), please keep in mind that the providers and factories are -exposed in `/resources/META-INF/services/` +## Contributors -``` -./gradlew openApiGenerate -``` +* @ppanopticon +* @lucaro +* @sauterl +* @net-csscience-raphel +* @rahelarnold98 +* @faberf +## Citation +See [Citation](https://github.com/vitrivr/vitrivr-engine/wiki/Home#Citation) diff --git a/images/vengine-256.png b/images/vengine-256.png new file mode 100755 index 0000000000000000000000000000000000000000..a59b8cdd5b04c6227736ba126611da1c404d090f GIT binary patch literal 12934 zcmcgz^z)s*xg2nPNOgK)9IA2ZKVd+-O(UCr1F zTs%bogDt66?ShN6-pWSa`fm2#zSf?0kgu;Vztb}pFI#JOJAOA$hwNQxItXHc)Rp87 z{Bm}3eT@tz#(NiNC2hDKq!dvq$L5wsTkW+L^=6cgrDxk{@Qr0<%{t!y7hLv+UZmD$ z?@crt4+bs|?sN9g*a$A=fLw_MW-kX}2gH+22?u*r-n#TKt;4W|@KEr?^yPMIMzr-? zLN3Z!b+)x>2tT5W(E#G2q=>|aB5b;JUT=;Mu{lbxBQfwU49DJgon>my<@x zHI=7O$I6=Q?{Di3O-&9B7)TFq?iHTJ=?Y!5_z9($2;lPpVWzyucJ*jKB zZrOYy6>0re!^DJ#3dRT?^Glj;^D6dFl3cLE(9jSKgr3{@hfC$+#wzpXuAqy*IU zP4ilux8hzCGfE2XwVn#t@6F3-SX;lGmkUu>C-d5!rL~{>_)a5}-(2l$agg`!?CYtI zcEbAR=4B_xsH;P9TRS`Dss{V-&f_S3ve6xREJQ-FP_CuN)0xIsx8ZU(XNtu$;@%r9 zKdUVl0;F1_qHqp7arT5&6B+Ce4!tB$S|987gk8of?Ipc8bovZKLPG4jSo z;u{)7{859iEGI4{dEj59N}>A)Iy+a1iHSq+9#YchbZXi2F==X^wy7SYfETNO-&EJ2(+EG=oOnA^n4n4LX5SP#ck4~^4`32p_w zO>@yF(?O96#izp}SkeXr15X&9&bMGeqhJxge$7$pOJwvJSJBfW*UE|R-Kri5Z)$29 zuHxnR@$+Y`=PFNlc(@gKsCU2|37zr1-dgL+6IVfD;Xjl2ZshI$g^UinNqcWkk_icMj)6x*9pGRHxlViQ z$;apsP(k41cpWlFhC2dKtWbN=p_t1!p~G;_``iUwooGaz{dcHe*=jgP?60BCZH-`J ztJ7VXENOpyXz|r!3wg6y<6NyX-A|k{GK{T(r^?pWtdP}2Rc_XO?o``~vzG&^kcCCw z6VK95pJ>R{rTcE_dVgR2=ETL%j}2wLe%<-)Tbga$S=ayp!pMjQTdBTv|Zgys@p| zdgHm+A+PcJfX><3nT0%N|C@lE?R?8;@C7N2U~-*TfBH~g&;C*97#rh3X9wS?&bI69 z7)+IQqrz}Vtbe>6D6OiB1kcfyyN>omF7=}D*chjf z{gBat*yURGBcpmp);`03^mMQmB&#C!pV7jwn%Y_zRB6$Q9e3;Q+9c0loVX890=s|!Nt4?`Z>|(=|F>_7PEOqaGBn@Zf0>)hn3Iz;`d-te-gPFU zGM5@Yk}nq(6-7NbIB4bK5;rtN`02|RPuC19!S!jVaPiL{qR14NcX4q9($do64JE8q zmhFk4>PIIeCr>GZLns9tf;1jE9tw7335LsCT7R)-&oI|KX?4` z77r?@ucy3y`*#1pIS%-K*Fp#?p}yYx?c2A%rlbUJ#R&ypiKDC7yiW~!EB6;!A*61x z$}%Xw45l2UpR^wIpB9&whvE?ua`Z^1rj2+#qv-C}cO{`0t=Dpu4GavlAI_N?0-3G8 zoa*e#U)F1--`f6U+t`N(BTk9U{%qfR2<)B&iYYxQPbQH3a+4RKkn{yqhD?TRQCAMR zE91KP#o4^nR5tsuqKIR|xv2&hPJIiD@2WO=fm74d2St873namWPy7e}{Rvs_O`wO6e}9&Ae*c~z zY#My&iqX{_$SN#M0WGerMcy?P{j)t~bhI@wVfCDlW|Y~(!@&bg*e_s^!sW@6CoFt? z402cd#hu%wBq?g)O$)&UkdC2YqR{nQ&nBqH)?DBEq= z+JBZ-_`Qor+r;mGUokh z3zFR+UBtRNnZZ6d`;$aX({jDJ=^W1}MGT4(28(47M@mjN4s(l}yQxKQJnc`9wIL^Q zDJcwym{~S7hJt+{!%UHamRtLbJ8i&yS?NBfC@vPLSUTqB35rp~7!Vu|CuL%KHhMxE z**W|9aDT#*6N2XF=cD7KzHJ$-f`w29mCvN<84*G@Q47B(Yfg$RKV&z|vDnSHj|KZ$@-K~&`x6$2!q z;^LD9^wzrz?MZ)sIv}lSK%ucl-Mk3p(<5hKhzB!N;JX!OO*51ygEKWXh23D?frPDi zXb>`Dw*Z#GqsH;Ce`Ev)(#R3-EO2eVK9S<$;({QMAStgePlX*XzdQl!VbtPB2Aw}& zk?@+o?|9i7pxiMyh!4514bg@*-sLETN?B6W`u`LvM$uHx%{8AUtGDm(yBGH>wb=Af zZ;Tb=KcDhZSy^F_m94F94Wk>#4MsJKwh$7) zv*2r=T-{$E#l*z=s@to=K7W?p`5MJM%!&7iQ2umx4ua0l&-?rP1&x{&Lh+I0dS#T< z{AQRe?H}x%oSc%MPgJ9*l9*)RP|(>D4lXgVs9JMm0f-8}z19#I^s%nac#HQ)6AU!) zXJv%|+vTwujkzrH>MBS&;pUI}fU?TU@M<>~>EJ7x zlarHAz5*1)#KZw^6UmgfZWTS^Wk&x|j=R-AG{oXVeWY$^cq{0|>J~ZR3#RLPAVk%A z79-u=-H~Z&X%Cbae*L&62E)&0hKiXOd1t2=@O+d`yo(^-AE)gs>O(3~te;Qp^_V@E7Yc^J&&VIr} z1`RKDlgJ6GWt$0x+(doda+^MC?wsEF&8x8G0w@*D4ry0uYUsTO!mVztA|&8Y(3zJw zj0lE>FNwL$e+7^Mf^7Q|<-x%}p1)2B~Xze-ak>l_65%|0Pbo1W2fR2%>ZKmgPL zy$M*dK#jA;gC>WOyd{8TAgH)C&dLB@FyvCLDX$cMvRSPS3X>L-cD`(`;zPpjcv|5a zhoqoatBH)dXPdm&6J{Vy+-^hNpza4$E3vV{ttx4LlOi0IRP!-ew7X^87m=AQCTE09 z($O04xn-pyOiLkmrQl%MA3uIT5U6KDv6nXj-2hO%xH!Uw-oAgI`X~tyVt5DH`fyK* zxih%DG){x@qdGsxzs`V1*(_(}cJ=!WdLD?OaQ_A(eBSqxge$~wbr0D^WB`dTkAEz%r5MIwvAIe+rW3qZ}p4~06g#pszP6gJC(oLlrGkD$)!X*vrF3&^=01K$_@ zVH627h`JZ&k6pAE&d$%_$WAXzoz1Sll2DmJY*7)LfuUj1`}f$QqM|ho4G|z$X4=zY zD$9U85SVM(TwWfV4uPPL0(|q$eJ6#JKCgsjVcs?{IE?}Hfk$9$`U_)AeQJ6W_jl>z-?V4WhHF zJBnnW;gQpHJ-3;e+2YO|esFMG&If6Optaaoe4r-KKApJzjvt#0SzgZb_PvG5g9pyV z&n<-r!$L#hK&4m&bqS=%{li1WiK^i#E=*I$X+V5*(V4yT_YG4pf z!C91&k&$TK9l6npZ3EyU7Di4Lz#_iy2&jG0F5;Kd*w#P z#+@MBf=si&zmEX4o#oX7kFD{$9Gsl&oSZnwAbwUMAu^EZx;}Z>YIMb=XJ!Jl%-%mZ zXd|;wq)ZSV6(#m5qXPu#`1rWHkB@Tm^Bo8Rc~8;VndjL;o96Q`t1By3hie};ewK(S zC^on62TT{KeJM}E(}@k(FoX2;^gMukGk!9Vao5ylf3Z_w`;5ri-ac}1(Fzj>=g-CQ zR&-1ZE7J;C-pN{9&i5LxztTOXqX+vr`PoYh;4GEAy4u>#(LzO$dA+Tj9U?b3H|ZA_ zlz@13&NchO`g=q~PY9W0gTjSuRZ@b2zi|>k_S8L7z>7 z+*w(JJ6v42VAJo|wL<4X-|}QGy-ijQH-A^G1VpOgDmc;P;&y;TSb04MctU5r)6b(@ z%P~r9gv_!zD=%e_@ZW3YKEMxxA5g`C zKpmeteFMrU6R*S0JNK}0xDJRgcGbjGV<*7FT27{2B=OGa6*$PP`;%EKXRm$t=6j~< zJOW%NNDUr8wjIfP@fN|&B5gyRm$+D%W zRZ}FS_ZBul3=*F4vlCQtJv4H`E%Q?C_W&upll>4{ygEM&wY0sNupINdIunxE`|*w- z;CR!n=A_3)2;ephZEfuXGeBgRr2Pth`&3j^cmS{l6@dDpl8d3s<#-!$?BiTv%8bP#0qKg#CA`+lyFVJs@(9s!~%|?`d$E==Vn;AZ!w*r|wci-@gkzn-%x; zjj*Qq&_3a{zT6+w3jl|=4X3iy{kA@RW8;B0M~4Ro3h14T*UJR*iq*zw;bd!|804?M zak!p_nd$fk$Ox4W8u|J8X(T+~${Vi#jsgTuC7wpUZK9<`+U%1%nM{D=iSNyyq-o4- z+ur!`n1lp7)D4DnCQh^OUS4qg5*Zyq$GeBzPc)m2_m4e{ajf{-koio9mp`oGfe|m|agT01_nZwT2i-9NG zv?qIV5QK||CuB6I42c6x<&H*H>2HxwSFT$W36YS9h{)SWCYEZHeZL1Pjz-*_<*(qQ zM~}cdtxgv8QZVO7&(4~8$_Aae&!-n}&NK;3@flcHRE?c}c{Uf|nSJu1Y3=W?w#9X<@#(gUnXGSbpVA5Aq(L5V`=SgafhL6}#u1UBaEQ2QLy3e<$fr6udplL+KAyr4z; z6^SrPfK6ID=TsN@@$~Vq{ny9osCmNf!$A+Dvt(em9% zEiJ8Nwc*abKAf$IYT+~dF+l6kEYBv6^q#WE%jtIU)%dy~0mniHz0CCVeI9($H>Zya zTA&TPnQAHT4J1(Lt$w^!dy)Mp3h3o2cAG%GV9mm^j#P1l>6{b=RXffAbq^C8JJqXA z36{hxr($ov!x!M^C%)I(4quhX?+wZ{<8b%@l#KF63sACDS(5ABBniZ)p51gvB$E#7V-il`nJBV(b+>S^9i^lMPyTo^mcX)WX@{y03 zDoi_1npI84v>_-vd-dQ$4>>2;}77HnTLGB9ASURPVR4nEML zBmePQ(PgtskWlu_N_I-?Iv-W}PFa=iB|v|?udf|wB1Q66g+Xqt$Jhyo-pmS_NYU)hvd27F5-~i=F`4U4wBSOT}iKN=bg@fHj5D*ZIr;g4v zxo!PmY9%KllP^L7ARMLw8=PO5pSRGNM;)wwt1PN&4T$h^)HDa0NvO80F)kw5VYwtf zcx94psUU>_AGh@QEiEl=%e-qzJUF7fGp;tv{$WZmU|VQ2?v0#^P~rnpoRXELHjDv> zFK%uQ7ApJr0~2K6>jF?EYj1kfBgfRBtLvHM{RV)u-^P){mzEmlD@sdoA6GdVzSU<0 zhs#r!qAe{oRe?teNJjJiC{W~53~X~HeY!aFgNACAyA;@AK#R04+m0Zh_O?%M^YHPp zo61X0E}h_71VPXVR4vWn?yfFO;gjm}as?oIOSnx`F;+jjewCiSXo(6SK?X%Ezt^Pq z`z-Zp#PRCk_H=`2-5@{>n|CpWAG5Ks_5T5@W7Zn*;pJl&)u15Pv#UYgYTUs~PEO7` z^7hE6D37OP+1uMs_2fE#{77w3i}@jb{}v=5AYjPozv{uZP$qz{!yz$t8_+E~&D6+< z2t503OI_X0*lo$@+x^XR`7d9-e3BaivTGzUqt-5_4qcRCtz*+O^MOB=m3)Tff^Egp z)H+5+rJ0|hM|&epM@~d3IXN090a@AEhr^e=M!=Xt>vYgT3rTlmg4J)sg8T7?a8d?_ zaNy>pc*Q6cHONPHeg94bkT&c3mplT$lkL}5ts(yY66!z(1mQF8FJfbUa6EzP{b=_P z*3bSD=zM>EmC+=AAjdYr11_e*NM1u8D=+WEtqYGxC4c^cni|rHD)Wh{hTEx_db*D{ z=H>3EFBrdqEml;weFfVKOD34?Pbqdk+7L?ouq!|x+P zU9fo_c(lQ4w6_3EB#XchJHyH|I8b>RkUe70^$kPr}147ED>v%Z@CXbIGh%AExTYallowVLFSm6n&| zr=+9-9B2*RgHw_W`pvaC-x?Si3kT>(lZhJ8uM3~=k44J^LKJUx{<>wi=LU=jaqvD} z5&4TVZWB-`6;NV{J)cscMyFEi3kne9#6qldNj_d)VSu1WBqeE&a^d%X|895tJ89~v zz}>rc{;0>6FJ5I*#r|3XD%@i8UK4B54Id@@A?(3 zv>)l`G+llDu%V&MOc7x!Zyu=J3r4~$d;c{T;=HY$w6&d0`OT|Wug3gBpijZqk{8GH z2>fYak5K{d1k6EhBgpg(*i&X^X3zOabX+eKtC9fxwb7$u&aVB+!c_1bC_5_pN`C

qR^v}-;H}#aiAeU;ndE}bNC6J|sVW>G?{AI>4h}9ZLY|#X zn??atFTXx~HJCX;`tRRAmA}_+b6+BYud$QAHl*EAwgC!4C~XEUek(2lfiN|sB&DV< zNwPmuc%h4AtFIZBfxlNj-)r_HGR`=yhrj|T zAXZgaUS3}1*AhIum}3>#v466 zZ9aW2qPjQ5{b~YBI^Zb!(XH$~u0673w}Y~|9Ne}Vo+n-D(-sf6M9q~gqH?Bsy9aD7< z$8&Fo_m7U0zGd7U_^d=u5@~(Yf7;d^Fo@j@1@qXvGoozuOca zPiQE~FH6D1lvrBI1+m;Ng_itL0QZguWBIg%04-%WH%`&h^8l4dQ0d$`*e{G1VRLon z5gng{>+5?OU|BL4ipAUs6vD&xknlHe=zxx}6M||}u_*pF>}F>d0lZSc47k#$U}1QO zt?^)YUti7$hUMjDOm?(oP|1Y=)aQa$ZYvCss_{1wH8s6^k3m}wF5cg1E7r+^0v5Cd zgMm$m$d_#u8D>8NT@h$O62K4HsWwxPxj@T^OV+`>0tw1=I zmX>yu>K50!Em#1X(Tx2vB2qq6Raq$zj^Ni@(h$Pr~pX5*q?o?2tZaW zPc5m5GI?lAb^A9AZ8x7?pK1nJnn=0Ln*o4eyF2R-*2SoH`Qbsxm+dQjjYpV-7Vq@@ zeG1s{N%3Qy6+oH_J-7i$!#_{q_#-7RHSn0~%R0w@kL1Y^w4>T$evciGom26do#Cn9 zDJO;bOx(sL3L^l7S^)T?0C*bWT%4VCeEUW)_k}iiuH}8{I2SI-UfFXT!&}gcUph~t z!BSg+TnqLPeX8Fz4Oe&kQbMLl{vOmtI-fsn1Ktj^_s;b3s=x}~`fx5g8ig`3N|jsY z8u{anj*dINK&ySy0qF@oTft7==uRBDw}+<2#!DS9uLX9l@Ne;7Ve%RV!4Uj#3QiTe z9Z8rLX2-W@e^U?~R0n1fR#wlz!bsxS0y{rbybM#&dN7nB`d}dzNii z!ffzZ)9_v{cn&VqVZnH|JtZu!3{#Nz_p@_zO9EX|PTq`F?2u*_=xFk90bLS?({;Tz z_-q}oArSXwiQVb62(auc%qr;k7 zKFb^0*AzOH6oT(1V8fVDDLvNx`qA(rrojOtMn*=FlN+QU7J`F(Dbq6e4jYQM1tz(O7E6d8FD=O!=Ng%Zi-VzX?_+CWp-1ErFe-G6JjifS zRaFJbpp%abs|th^)HUS84%2fy1Yry%~WPFD@xL z9_Z)qzmy<0Kl*AQMX_4 zF|xOC``R3+`g;QL9K1mtzt~~(c*v5Hk_SHXe)1#ITiHWH8maGalSi(8`coVW+L0~h zLePDD9h}w68vs^+UAjb<#X;7}f7f!?t*}r6e&*3QXVJvID&AV#jQ^**XGY?p|KD$}ncw`PEJM)%gD$q?aYsEtLkU?PI!DjHMWgMwPrRbb`wmc}&P z7kxC!2-5%J64-jnK*41&vGoJ=u&GImHwM2)kR}@!R|P^7i&x>Ack$7X6D@TQcXxM- z*Z0;7-6125|L@WNkxR3sQM&HJ&FdFIK}InK+>w1oN(db^Ma2(6O~2R$V%8lpEVJh4 zME?H%D>(rn!B&K9EL<=M867<>2?tVxnzr`(DBX3pAR##g#lU9(c$Z#6$*@x6Hs`4qYF~PxD$`B8dpW)kWWfl%_OPf&o2Wg&GUAe~wQs zEG#gCeOer4dtgiY1{c>50_`bdjjpbnTjAlD#j$YE)@3~3?l8i{V4|pt!y1FM0+w&o z*Fjx`i3wfbt-si127?DTz*T!UZ)t_AjEQ*{I)KwOtX6}`C9vTGM@SKn8i^9y5NkF} zGSYeo@+k)Dynxr!l;@=ZsE!TX$cM@v z9zxK0QdHt~1WCR!VjpzHX7=(FA%FbP>r{b(*{VHwz_$MDlAa`p3RX;trdeniJ05P( zDrbJ^X11@38Lt3ACtn~|bWF&Ji_?717F!=>i#-&LB!jA+i7gux+pgl!EOi+yb z_2lj_XKf?Y{pdp%oNcP!|axPqHj=q0|PSD?2a{ zbRY$f)myf+vkQ%ZqXi5=wTEXKT%rJ%_5gEShz9*&Co>^2xWMlN{=Gb1Ob4)si-4I8 zA%S7HJ=a|Q?|JDQU6O7Bd^lfDAew6t{U&$sVu3@zT}VKd{NP=d#Gy|rL*MV;RC_Il z&;PjUj6At1ro#eGspnJOZs7SXbwv=Yw16?)Jw2DfELq9odM0a@#O`q>XJ$sVy|@%} z)ST22tNw+1)~LV|83&=>Jhbg;Gl#rL5#0no(>B8i(jWF0-H>%-!2t z2^c&Q>sw~%v2-mgh!MNj-CYV2jnQ4V6UboZ&x1SEKulE=8z{k6S66`+wlv8mT0Y+n zdX;`%>eEre%zcA9qOm|DMapr{zAOJCm>be_ z-QwM+0d&9KZ6OaG2_H|Uyn4`YjBcP46cYmx{Ghco7Alw-CZYNK-&OW<9$waU(AlstC)H)OqtA70O%6_t z7=8kcP)<%}MurXO;;j5Nb5R6Rny#*HC6OGvDluoMQlWdq04{_ka!DXgk`ICF3tG1y zRd>JS)XuX4Y>5kq@gv833rQKL@Au(pS(paOY%x(vdZu0@03mlr68#L&jo|tF_iuD! zqWwD_5)zUUELrZBP7W-fC6JSQ?3yubNyy2`5z^jM+ESJPvQ`g(A^#RTF~$$;5B1L? z2f%L{W?TFTzzaIuCIyjfKYmcZ{KD?903LIrnw?74*^3%LOacQWj>4+xl zCZnMl?V#eK?6&;@Cfa%oOky@p&XQr2i0d>Z6B84_k5OS^7?mGBSi8Csz1D~Xrw#b4 z=w2Y8IrskQOB$FCwv(2T5%S)!0*y)Z%YSsZoUISw5Cj@Kf4rmo+N0i2Zte=u0J>{$ ztjc`i+o?1GL&Pm}(W?XX1gJ t+0q93Z`tB2Z~X6f5u1bmzdw{XM0E)5dfOh*gTRL%b!9E3Dg~?1{{x-dPRRfO literal 0 HcmV?d00001 diff --git a/images/vengine-512.png b/images/vengine-512.png new file mode 100755 index 0000000000000000000000000000000000000000..5ef04d62b9489b9d0aa1b32bf29a41c630114bfa GIT binary patch literal 27718 zcmeFZRajMT^fkIcR6r>yDe07!lJ1g_R6>vzkWOhq1nCeE5KvG7L6D7fgOVx@(jd~U zbnS1h-~ap0bI!%NI=AP-M>g!e_PgGgbIdWuTrciwsSw~(ZitF7YH4Sq@GsdV2{&&Af$$HLtP<>TYSYwzsnVP)ZJ z!|UR1m$@lPjY2V?)D-3PeP6Fn`IwU3I}zD-EDVjhSmvX2xA@m1uF$ChffpH;mQf!L zato;kg}yzCS{Z(n&F+vPRTes+lYxuRgsZM1>z8>z6eR8P{W~{9-K{!_(Sv{K!9R_> zw=R1326n&BBYozFrKqU*j?>WLvMl`2ZG{d0y_3^GMMg#nMdG1IND?mzU@nKTHH#M*_v<;CX)fr2hQan1wwjoB`J% zVvtKbi*R$(+4pQu_`1-egz-7zM4T*!q09qOg2amj1qC5>C;Zm7wjb%D z%`Pt&cylQ3^1a4p5X9<;yJ%!OPlQTOPZ#s%R^8x_RKaGe*MMG$LbWF%LRhK8noZzPjT(PMt3_zB(c@UTOKoZ^pWoRCf& z)mYSh6BDc9k15EXr-M%DU~kxl>Y)_bn!D9cbsj%@beHiO(*QpFnoZULr+M-Kn}KPt z9r>&)?4Ota{lpL3>OJf&K`i$wEO+XV$e3_ER3csiH5N(?Uskcb4SD`X-uA`%4mpdD3aUNd3 zk_pB_{+x32-rJ8K)i)9%cagQtb>zBvQy};pn^xTOy6ilqgX^{%t=MNGGcz+T6+2T} zEPj`=vNGJOuj6uanNYEqIyiH`s$7W1E1inmJrxJUuw~iiW@0hItgNiBZhMHxD<~L5 zX;7KsJZiMGv@CHvEyW`wZ2uw3r)XpIUOkayr2{8KpMCVuT+i4zit3i_+;YEF_oWXE zH6$orY$2zeot;?6QE<_?H%!rn9KAayu%CNyDO|A_YZ796~zr7=;Kz zW+o5A`dptA(ZBerxYB<$@Dd?yq{L=5ozIjP`*T7%hi9Ev=hw%| zw-1P9wY1_2es*``n+Knfr}8#jiKgUh$qC*}3Ls>|vgUe)O32NaIGh+8!$Ns)O@}E( zQ{354*x1-8^H?IB^c*zJYY$+C1$CQkHa|Q&m?`twk|v|1tUA2CzP_H2lXHcgot;_n z5pC1?F^zN68R5|I@YJ?P44nkUowVuj2^aIw&8a#X{@M{WHMOTr!6;tyM(&S;E1rjc zd-y(Eesf33YHAX|Dg}n#<=g)K{;PqG4x2_M1qUjjo`U-opZ>jjQ9;gI(^B%8B2Fy9 z?LU8N9nQzk7zDe(`(M5^C27Q*OISdt3k?ekTR(Zcxz|qPXb`L|>dwl>hU+*|Okw-; zHKXTp-`w+=rizLR*B+iL;dv_Y1?SnixOe2HxKh z-KCwX^-_>+)`#&lcXZG&tHeEvja_j0nVygk5_qz2HPd*8OF-b%?+{?NMqXa!M~@2h z&bIRQ4fORH0uH9C1p4di=fFkGkr}s zYZFCDiZWBKyE=7fUKbh~hrcmik(?zujb2a*YmEQ+5r3|zw2=%B$@FjKCYAUbtqMd^ z91tln&!>x0OTK@-gqE}j*pI)#rKY;!2xZ@#+ zH*4eY-#69748u2Ry~d?@{rWXa&O~>&`r&4si^@!Q5m!@Z{P+h!IPjICKd(~Q5y#)Q{=rb`}3%*Be1;6sG{6tq!X9NjwG5@uuW zI92Q8d{~o?vn}`@k`7;(ImAzo#qShwed9Cn{{BRHSQ=Jy1=zBgqQQO&y(SMHa0VPL zP|9yjH)KvNONfc#MpI0&t&i{J-@SJ)?|7zJx8(Kfp4Q{NwT!bWS2s5jSUQfO5yP47 zH3B+u{r#hvfPb-uP5BuaTyu+=l%0+3f4Y(hk~+744-b9A2}zUVvpP7%1S%;%zmg^H zm86ON&bc?KZgtHue5(mlqD0VCRaN!brYHWiRphT14<`-(L)r3k-WxY=w3-LgJ?eL~ zwllvrIXP*a#{e4D;bY9#JyHo zip|b460!o-_GYR^N^5JWEqYSvGliJ|`fwm{1>I$5W)?Tyj#)@E7aTvCRYma$^Rc1o zjROmjP4|1=D!k-DU3dFVZ^}eP#c*rgR*B0<9Nd)GwULs*2OPY-x@SH=){culew0VH z`&b17rv4Jq4eryk1M|_dfS{m~={iEgPsaAU2hA)YX~xJt`&6s(ey7PhOPY?)b8SRZ zB-?9kn#9sl3qLzM`|%;qG8Y#YadzS!T^GMobsgvD1mV+&20e-)wsGr_MF37M zbC1o*+c-EHjyoa^<+eY*M$kL+uQ1ye+Mn!ibm(bI#SRS(QDD|dwI$=?rN?oywCtXB z?pj@wRR@gEW4j6{q_D6M7xxw0lgOT?n%dgmdGR6^`|C#dipuH*A3wHUrhnXA>$R2{ zs5+PzFCD5abEcTmj-h75dPl~F8e&=;I5baAO?@5`VxJUIgNkiC>y`W`1XG`7UO>Uc z!$UwMbBYTw_T7OSk<>1OH0Jb{0S(6S6;8snSE#$?!o!EX>9=T_&h~GvZ)}jqX*5CX zWMX(o{WC`<_`y*m3G?| z-I$<+u=G4Wo(T2Esw2+#WTVRUuQeG>FoDsRWs%p?bny~f_4BoV$#14+BG9|Cvhr{t z4PyzRG-@9 z@{b-}mz0#`cQSr6KDRtP1}bb#$0FTYrtx1k56A{T!G{*#D-#NZO`cx62N0n zfQ22knAf%*he>|RkM#zFytn}`7y)~u;3MB5-#hTIp{z$TPl@fRM6cwI0FL(E8y@Ah zOr~ezL6?`e?Qb-7vSCFK(YHVd|KO@`O;}ArY_WWoh=rX!y1Svio|w$@WvomhRY75) z9AL=TVs7&KOPiZ<@7^hr=Q8QFlc3~tq#Ia!^78U_)=HZ=X$eTCA|dfJa&f7nwKi{Q zjl)}73-$76x9CHP?v+NZ7JvG*1JPx+q;6B6QIN?Y4#}(Mrx%kv&LFQS-0jox_4E5k zOqDJeha#4gk^)2|Eq43%i&Z#DWsk=i4(B514I*PAGu(fF#p5>hjkccOT3(KPPUVD3c)tK|k+eb_01;kV&U3hqSE$d?*tS);VuCC1hn)tmw zHvN5?XasYEx#hinpH}iP&H5SxE;h0S1NO(3xD=HYF%%ROZ2!SGi>04*aE?C8TU+zK zLiKqeBV6vAarO457{&_?0c3%l6I5#y-Wj=^p_i{^u{rr9Y0Voz-1b_jgn-@8$K*`R z9<4YbitRB}VG0rF%4xh06um+jX1myPS|QcLg%!z2NyAnK?S6Ns+{_i1kdRpaRYoT| zC3G*mvvU#_R}JQOBojw+Z)ork&;v|{YMrH(l|bOJ1{pc|i`9}1e6iqjN#u(F1Y5^a z3*)!6bo4d&`{-mb;QAh~*nM`M@`FDQWFH>-YGjIB2oDd>6^D}O@z~$I^%|6-F2GeY zSeI}8c@h12Dnwn!Y34x}x9ZEa<_aReu-?8+b<6JO$i(4H(8)FAYoIR7_yfBF-kN9E zo1L^*TK|`8+HcJ%JdbJMdAVILa@O75-4(r{Mui33a$Cysf#pb-fZ)OnC|k^Lm)~!1 zmDLC{?pUymwpqy9&Mq1OLr7HNe&CmepP0xxYuG%*G}@SeMx$BQ z?tLFcOht(#G3C&MMMTL}BkLNJ}#6Iuj8v^Fz8EFa*rj_u_5rz2#VvNzivd!hVoPZ3=jcf=HYDfQ~08W18Fl~i}$TM zBoU6c1-+ z7IgCR^13eSjQ#ufZO~f6 zMIz|*+2i>p+aC@h=hi}iD7LUNl4_~ zN#k7}%ojaA*op^aYt@;+cwN9!3|1+>Poh8j`SVLL*9GwHe!eCdcA7dS-K<^Hg3!hT z@!--uQ`4BY3Q{ZwyfR)~)KClSFmm;{Ep(GVd;XkJNQjJ)@oi*mEK(Tjn3uznUzhSD z3kVFHo1ecxPwxWtbALf;A*-C6+$(Y6yI<~Cnp8Qfs;evJYh~$2#c)%Rq6DkqX2{F~ zcHS49{*>5i20(8R*;x;Pq@aLfeI^)nazHQZ=_v%U&mgu7W{Z?fqZtZLe#a4w7#h*I zE0QKFjVGHOkbXJ1g8w8&?2OrE8bn5N)4{b1Iy+T;{ZDe=zMZT63&kx1C#TN-eT!?+ zQBkdYO^-X`=p@{Bek)b9^%t2p1wmRRN#!<*DYNSMNEc_s#m)V-@9mAw-rlB?NytyY zQ(22nG1cm5pgJE?Q&aO>gi<1P_s5gfBEB-HFv)WT`GDzw#JO<@Am8TR^UPW()hx%* z{@cwL4v=Ijgk!DP9H{^r_R^n6E{ zm}0StySt!>^W-xglOPB8mTYM$|NXCTr=C%|0;)R^5Dz3aj zTS8{$MD7ycnb!Izq859rqq8e3_I-!k{f&Vq0zR9Xxqsi^*=p66anfOALhHkHGfY3R zqceN5V4!mZR7@0t&B7x!OyuRU&HSB&y*I|E)}4|jzSxIx>akhvua5yY^ueKF_0Jz` zTCr`){jHh!H*X$7g)h3c1heGa_ylX(*NO+~HRnGZKQgPT_*&cAN^h1|Rw~QO(*mpU zQCHvE9$;J1!oot!@G!yq_wPqjs$6DnaB|x9e=-qGVY(#+EA6{cF^sHqW<}D>&!2>l z5t@I0iw43>64Es%vCY4fmoFcVmQ4L>43s!OUfaTG-dwh_vI^_C;ZNpLUpDe*I==Am zWGf7~vDAXow-GnKQkn;p$SYdEK)ze*WZmOewOew*DHMC_3>?NxfDf2 z%;bOnnF>`pK`JGxaC=bu+ zgHPqQ7G>d%T9*5>aq;kU_{9gkFlPs9aqj~E{Sf6+R7ic!3`rDEY?A=N_ zb9pEn%bX?*PY>sbApgD$_zuCEt&_TdBf-= z-@fJQU3ifJ{EYBK`Yk)x)gexuBiNk<#Ib)J932t5v+L(ksd2?Mpr1Y-os1A5P$?&{ zl4@0~AU)8T9w0lvz+?AsoT zk==kFh5VygDhOdMQmd=Q-sR^URGSqo;_ih4ye?kqw z=O1YQs>7gp$eVe9-nbO=VgL<7iBa_@a0w}YfUd^!_xC4f8f5$5Wf;^dxwQb}Cw*ql zGVT-c+jLv&7~Q@589(vA3u@NFsX$;6SB|JlHtdn5hJ)!eQ;Vq&nlz_NG84WzP58iR ze)d@M?s5wA0U;okMzoMP1^q!Ior&v?s%qa?TwE#&n6-iFh7%Tl!+(oY@8pHLRW&re zt_}|r^@_TjljSML>WVSd+8%Dt{i@zk=&DUK^P_t4;swk;cOeN%j@b3|5RW{U@xQXn z^KjcjXABq?{)f#d@>~IFIpXcFq}u8?6zZ(IEGL7T4t))EN5Y6ong`zdyy%9g2Wai) zU32qNdHrq>Fp2#RHhH*~p34sDg3iPT7hiwn`?SB<=$1jGG8F*YHYD6EY;JBIE@qad z5Jj>C?vL4Cr4m%szd=Is^hc(Q9wXDj;bElDbWCI<$cut@aGg>>3k;&7_w=Q`R`p?2 zjROO^#m0NN1(Tql%mf|~Ady*KpXEJ7yI<&q%UyN+2-bltuOF#SMar&4mI9&vpDD_L z1R+)QwSNJ_*4QZBeK{`#nT@frvE09xZ&aJ#TB3WE{DPGIcfSffNlIF3s^!MGsD7j> zat-2?l(e+@@9DQ7=`2R!p;o+hmvwZ;4tM6~{``SiZzrdriRNoMRoCBKUylKD5CEA& zT-8j_9e1r$AUj%C2J-=}9|8qn0q2K{h-gU&B}`9>8o0@#DDURa`G33s0o$=nO@|^Z zcebIrWu=U7IN9`r!la0pL|6sFFl(;`=o)ELE&Y?X#l#+djU>r+FS+n06(SB=P*9L% z3B#^UG4tGdGdC{}(S>x5V2ExoOI-BC6`jC#W68+K=o|sv^%SZGawb9PLVHB)Cud^x z0BYGQFC>5eWd&c5ztgoGkDyI{NzW zyg5)I5ZPus6BUpGdi?F(TW*gBpEc}{J0v@tK5f74c1QQ_-4LgVAc~aBmo9zY5mf-V zl;ppF*&Q@Yw&x&F${)07#Th)3w6(SUT`_7{M0cqIMT#O#eH~!9L9EE_bRG;)UayFg<*p}=l9v9t*)(&>P93$I5H)-J#35UTD>$jB%*gAmT97^sv=5D`Je!} z2xUw~o4~8>ZC5D3Uv%FqzE7B?l@HJw7ZlU`ji*>TN3eHE=;%JlE4JT)+bDi;BL3LP zX_zqlJ253z#%G>9_xGH!Hd=r{ck&&{7hN%SsJHNldbhUqe)X$`2q^$h>o8|@LDGrDd zYEVlPrywV6x;_(tLJT66Az;5C0P-Gu#@Y{PFE%w{?59t$5KbAAgaDK>7Tsh!HRP8G zPv6T6DSn8+L)mKqn6R|`dL;!A1^m)lquegqB^fEJ!9oC6?)(BQ-49?IQ|is1j1;FM z<6+qo{RD?xaxp{I5x!;-z+hxIP+sA3VV4St38<1YGid-DGIFII(=!ZE!+3AMeQS{` zC?m6jC1_0*5csX`o;!Fj;OjLM6fQu5Ra{7|{3dV~tQ$N58Mc3?SJAi)Cq#%9*mxTo zn_PG8UIF-G^@QVz)YMcaW@d%(N0AzrAv(Q$_G~a$FiR_fgjutlNQyn9Wu?^q#x>kSMJD%q5jgryKYBNO)h zeCWdcfBorQi-_o>1lwD(ZS}FLFBWSLr`Su7|F>!#ZM$oCxoe>3T1dxP@t+m{r(Om1SM9rv)C(F zuE1M6J%1I@g=CbNHAI0416ZHD+S;9uMMW1-za}O?y=#GT4_Ia7PJDG=r`P%IaRIzi zTKM@h%OLW@5zs0+y1Jpie_sRdL|#b=A=qVzaS$U7K)A4m1}RW0`YS&%zs}0Cu*zGB zuZ+~kzN@Df23RhrM6F!tACJzxdl6SC2Y?HF3Z>+qg@w1kNGkPOI61{?Wr@E9*0W%A?d`w3Sjq~>h4Vr4q01U3N)IR@#f7da9nuCE415;3?QZp_2iO<0F(yT(YG28PB>_3>t_{;fq_y$qTPkK z_3T-@rXnpZE&roFhU!|v_BV1DgwI$VR~;N2Sb8+)Tv9VLqu;$Nhsw}lPp1N=x4QZ| z3<2Sd!5Q`th7V+)RddMsb-}L&9)mZ4rAu0X*}+$lw-)CX#Hw~*z|YOoW|WqmUZ&p$ zwE0>(kUC#G=Q#-W&k1NQY4`Rgu_q+fYVcqontl2$waBO_W$;M!o8JU@4~#3=RB!{2 zccBc-o88>J1k_4iVAJsRIt2 zFzcEW@=>{DZzDx)SFlpO79)xJ_3M7}?YWy05;PLt>x#YEQgdr-ZHmn}ujKjoIXS6d z4Hewn$}CF@wY|KAAqSHHJr8lKwYRr}i4?0wv1e!`n3%Jxm^9@0*biX_B_(r*|DDKM zximwdgn))_`^MvS^w$P>d{O|AKzR5?Q_yPWNE0YLuQt242zIfA>MxyLU3gVf-bDqb zFO+A2`iB%sFL;Zf{3r(I6gUSRJw4`$I;R)#C11aOWng5)0?ZnolM}yQUvr!5X0jHl z8kO+n`(x4&*ojbMU4HW93D5CuIOkC8AhbNh>zlRS8&N<7HcwBR2~iru=8cNNfBEv| zn|JSU08n)TYwPVT0=2jCZY_L5Nm=<4HueWgz(qff(}|>go0LlBj=doCN_aoMw<~;6 zPS$f*jaGtU=r>CRl$XoHMU;RSV|gFcz5($Ab7Fp$55q+`U#OMtm6!r~frXHA$;n~$ z_4O37QLv>LnV7JFeQk-M8k}1cd8Z_h$yL}}l#)U|{>45I3Arun*Fpf7KMN0^|N4S> zXmnI`%??ltcyVx{EQg#ZQrBT;Wpx28G7zrcYUfCg4wF+*FtM^;1UaDd`}fBqX^#yI z$Po+B(9l0n2TPRpKC2dkb0PS=hGwZZ$7R3w3-DWT^1|!tBp@me7w_Rg6aerM5*Bs= zj(@4d;Rm>dm6a8s;T7N%m6MKwob`LuJUFbV=q9*d2-(^5ZJRFZ=^GldaRI;Zho}CX zy=@8#UjL4l@dgq7$`3!tf>Llip5u|SH3LR{QwA`w4MOo)d!a&MO4@VwPq#AvJ7f|*=Oe}6 z^dQJ=|NYwv)ihbnw<4#B%I*FA=V@t_KVRPto9~Ff<9z!>8&vGfEIf;pkT=So5-+9?4qF}IuzzW&BTV!kHw?R-4c=FfU!0-TsA75YJhs%9$Bje*c zfSleXtbvt5CcX_J!X1WGFEX=QscA@tP1PL;jn#R(ySK(r-Qu^76!%!C6ZdQefIqjf z5u1=e{I#P)G#wX4G!H(V>V?$IAI^IL#>N>1k7s3f@SZ9fTmHq9m2b6&>sEE$-iKWb zP~iz%;;;$Sl)KCQ*9EN+kvsD1SH-ao5vZ+!<%4Ad*0(BDC+RI%EW!Ys)pcQpK{R9L z=kH?~2|)k^!pVxcF%i;=h3xKnHpKUS|Nb?V$HeVqBf7s(kKHCzCdj8I&=7IoqB z`M^ojI3O%;v-;}*p9EgCV{*iOHe*2DCjGb68UX-My1IIH8s6a?-pENU9UyiiV7rBFo|})aW8{;}Jn+K!O5XG8>gr%+ z{~uA9iG-+9HTXPpT16U0C>|$>;SD3Ctx> z`wj7K9v*Ela?RggTT=h-?6g3oRp_G*q|2qIQ*_=~mBdGg^rrfYP#nsEu{xnLT^~Ck zEiD36(Fa@j4#=E&PdM2gh2snH62F!SmTr%uyLoaQ3UE+~Glpr4M!vSU3*TOsaTwy5 zbnoH$Tv=(z!PN#BiEk6A5`+fa7XW7Za4WcKes#DAUwI@z7q1y8)oMIq`|K_j6!OP~ zG&(;bTPm1y=f)%3@^kqH+r{tc+q=6jKb001w#3mj<$`shrJ&&U&#XHw=VzB;7Jh$y z@k+v{L`g^xBnl8i{C7(%AWXFIHLc4&c#svED6C10)8!>RJzYQlywmzx2na7L8*<1& zgact)gZJV!U~-E;sz)c7J9qBvEalW`EiqIb(UF>jf$8#Stozaog zLrp)@Z>emxHiJtKKr`2Mk}ggTj%Ef$@Rip`lp_F)9WIDeG2X4t^REg1VTeYX+V=IJQ-5Q+V5@P3O)#oLTLE;f|xa z*)(3g@zK1HXjqVe650%{GWWis3;H9s5&G-z`1LPURUNrX(X+F&3`R43eaL(YpX@3K z);IVctUXwf_T7H?Jxk(VOXScX1O)hyqW4zp__#4dBJI7xB4Duos}~?uK?Se)L1h?+ zK~PZLMH}n|HMYT5?LEAf`>_5kb_EM8EG^9+y6vPPl?Dm~r|aY$m@hG5Y+l5x<`}{l;)(8C3Xgzn=w!f$%0aOsKBY z1D3hDg#}gC^gj!Q1iE;j_Bx1PCaXvs*ivWTf!p9<5cT_5Z?hs zqahW(0lEp&WV+Y?=z{>RhG|H2t-o-1+P{dZR(=7m$^2 zK$yf=w)TVaSHN+E8;qTr=yRC&YR{ExfCSz@G6hF3!GBc)5cJWxZX6q7L?W}%e2t1a zI*I+}cV=8`UQ>>x4u{ly^Lp&c3V?o~T=?obPGP?R$&jElO<~VvX*JKH zols{xS?xxvelBx%yw}#5$V@^{&j;3@0BoPhGQ~3A)z#H)sDdNAJHRCZV1a`oV&FgZ z3xp5w^V=%!aE5@IIe+?OC2*4tU_Q`j;mm@X$!wbPb>15+Yk&X!OMJ;96Dt1+X&9iD z4p6LYyjNaSGgu zVn-btqJaPveL_)V2;q4}^Es$fI%gs`?|98N&!A0oB_JaKv_Y;rKRb(}r>7tDW8Hdq z1nyxa8z9%K|NWF8i*}f$gGw~$k-T+s zcL5+moYE0v>hW3BxTdT7C%DwOR=DClkZd+8pP=qc%*k=ty&gw*rS(7J5p2Bqj~}N3 zTX6$4$&c-b%>~$u+K&QvdpK$@F^CxY#SuT6%ruFv(p||LT$x>3LISiVTAGWC+Sz%$ z)?nZj;N9(mnZN#H{=UA=!zogfmBm@MpQMwwz{WHbj{ zqQL6fN6pn#2bdhhG{e;P7b>LfgPhhszrJ&%qJY3+V3hNVcZCqJe{NGM~LbQbuWFy?gggJ%|$(t1z{G)-&kiZshCx zE;R9e3G5YS0Rf%LAaEU-<_uhg`$Myv0vYrhyim&i8^cATk=@0`#h<~#k=O8nlNd+N z9jCdsSCHsEQ=1Mqma2v%vxz9k+V5NW?*Ph7mI_z^Ts2A8huY=Qo!@lhxdlmVOgwPr z!lyE?HIZk}qqKjRqOAqM3m5Y@;{z0D@_#>nR!e3Bt|OgP=^C)q3E-bG0uaqNmbF!o z2xk?Rm~{;pI@QSIqTnCtjV5~Fk4MfVOpFU?E6OvCfvju5h+YjU7zUeIiNkWble(# zHLO8PU!Sb;c-2wo3AoeBg3iRl8sZA0;^LGM7cj03E%MtSuc5 z_Enu4pb^1$*=Q)!R0zEhZQ3#~sxVm~L<5|@%g7`ss;rFyQnQ@ter{+Gyu zbG?7prLZ_S5}K4WH)-9HmcYsV*C8!3kB3W}mmf06cYTgIw};9!k=rhHTpUV2^aOhj&?k@-@l~v)clXZsrtIZ- z*bi=}otQl;0fPr(I=tT$M2C1MH>zgdKxzKTPs;;BVLY(@a{4!}Uq?YHyl{c3{Jk;u zLl+m9^|Z=IP{XCrU3~QD(JUR7AS8snpJ(wjqFQs2+?kV0^|iIN!@ZvdSHRY$4xSgJ zbE5M2==(P?lD(9?w*UgM$}XPMOW@+P+y!i4p{?N$sFX@ zB(L=|nuy7zNn&P{@fZ6+m1@7ej|Hc&8bad&J=Qf}$}HPMpZNQ?wYMvyd8MVZ^H7*K zE_HuEJ@hc{#)S;Y(z9@Gr)K8nCLj$y4V@#{;2;P-0N>c+&t<%j$;%I%twW4I6zYV0 z{)`JdN%QFF2;C42Ase>AI{?M#*FLC1{V@oK^`CM}VuTGFCN3^6X+ihj zeNQWpun_a)osr?uQO6*%H(N5@{BqozUR#W52=)RowP$2+eN3T7_PZ(cEd?Ff z)D=v&wOt^RbZx>X0VA{@c3)$ICpL5@{pu-qSpC+mS58e*3dXeRhH5MoHpt0d!Z%XqR7qO8~c#NmCQncvvp@xPg1sDcH% z6B1AFbW`HXloW*2Nb&**W7RgSVN!Yetncp5)J>!j>*wn+SclZFuny7D(T^Vl1vQqL zo9V$8=o>c<`QV2EBimgurj5v$7)6AY%ZljW56=_O2CtFt%KJ1WWMsYvQ+voo+TkKu zS#;vw>l{XDDJeGVX`{4(c_?VU%P{tbYOQ^`2_IVZ+(1O{5(nv{x`qJo!@u2uJABN5 z67|o)e%3ZPsE1f^IM5^slZrj6TK6i94sggq;|MSv%tjqG!ShiuF)tf_ zjJCCf6^&%h4HfD^7ged^0;HgJD5XI&v3>!eAEX){BQqnTE-#ew?EHM*kq#=b0Rs%h zUujR~!h!-Qre9iv>=G6qpVmr}o&!XfP;`rd*B+a0g8N@Oj|Kimqw+uo@U-VDOB2T{>Oe5VM8=Q> zE3-p-m^klz1LJ~L0_)z3+M1dMOgKOKMe&J&uKdq@p!@R$>09&_klJb3>u!@*`~-su zBC&Oe@1JL~OG(iIp+%AcT*useyl6{zR7X|99LL|8cY`Yki`-QUKv^x@N%Ip>)R!Cb zP^q;P2r0jR?g7W?CJT$={x&@k5mAE=j?TOk=BA+}NBV1-^FyoC8P+g8?krl#`{G!D zEA|a}9O%Ff`Xd%wNcaFYk{j@k47|KrXrz0tv2hw=U=XyMU52*H`})(M*w>%!n;`o@ zs|*J`wOHuPU?7`~3;Bh<0b>enMG3{_Un@|+P4!Qlf=Ivq01Wy@D*R^HlQaj~;O0Q3a>j{+sOpiS#2QzsUBJ zvmSe+qGMwX_^)j&|0>IazzNNe-(s+Vd{Odd=^7bHh3nxH5~gN(L#yXi*oSbn`qQ0` zoL^;D`33cW77w@E2;jDj-#)_T_uXa~n;QK+KW`I3T<8UW#YP1Sm_i$pJe20%`NF3B z+mWuGo@6P9F}U(gOy*rMu7c0b6uk*n``<%&R4KW5Apo%ukZp8NN2ldmi1I+0mCD#m z9cv6WG#ajLocTW~BN0q}^M+w;8dL2y|H@Ywx~mMT1CG!I0kA!hejW%esT@Iih57lr z7Q!eN(I$qr&wq4K`{%=nSy{1QnClXU=<54qr&2iZ+O=!jKSXDux+Of9L%_CMEU)-Z z7%B?b_HzFLIS}&!0+u|MrAA0F;1*OzhDSyKDEgfk>fO`TWtuptDl40tY$Acgu2Ja> zcK=D|8a#HbXR2}8z{Z7LHb`XBA2W1Vzkifm3?4@E7)6uQ(^G4hnrunv#lR0|c4$aQ z;2*1ChK`>4z;0E*j;?d97ypB08n2q$QZocSm+yA{1*;^|vjd7#D@o!=K@ti8d)Ez; z*U<6u`ZaAh^T6{D%2?l>cx2Vo@L&;G*T7jeyYb_PT@0(3*!{|%J&sJ^@F$CDmpnSr zZiyoAVO>K=2x9)&i1&#)g-4Me`u>j>APEIl>jtT3K&e8JSb2DSET2E9HUgbY-NnjS zU=O{0P#Ovbaikj&4Hj2#uwxfP_$UP<0(QLgaisMFNWowiiCSNO4EFj@cQ>qG=;SGP zJAa2_zctv8d1bPu9QGSDx{eiJeDv|4HwDw42WW3?jk#`~{RJ4Cz#hl$3wwZYER~O1_w;E^!~v_KOX-=!q>2yYaOVrj?bo< z!Q_8%QRhx!1`5?&?et zUO+%HuG{x2HgD2G^WM5es+WI>Nm$rmuAo!*l_CtunTT5yKOr?0?vi9ej; zt6*u&#~}&b^vFdLvQWaOOd+-NR9(RsgCfIKO>CI-P%!&&K=T21^3fj=Vf;5+cOs4L z2HvTm^H7P29|Z8hLdZLDLNDw z^M?T%Nv@HSlE+1T&jJE8&BPwWvVO@hdR_wrqC6r+~E68fg+z9#= zLk?zwwb4i?0T$MvwRSs8lBI5r8EB5aPrB%5x{-COqLAJA4=L`S8z8NCW7EMP!TvhA z*YdAd2C>^F6!-M>C?HJ@EG%$4={J+rQ6VrHHBP2{rqv-(g>wNm1R}io#}7?(2{bwO z_peTp+=a%*SFf%kNj5hBc0+K@v&5LSF}s|WrJP{4qm!mooX(AxYP;HOX{cYne!235 zs3Qk2GE(1$>`mcx(`4DekT_s9^iqEvTzcw(xDksDV(mn$$%4ZjabVIpQD5=x@=Q0y zMlI7{1TA=pBP0>Vf!NrSyg*?Sd-C^X?#}0fQo{Z7A8pW`8wtTg-kq);ifDf58d6kB z?^wxxN0_XEq9*c8G{4e3g%xY}_fHqwEqr1Mil;zCcGZI7iH2E)LJCmG%VNjZTuEBj z)Slcl;guD1Xf1!*S_zzDho9BJ1D6~%9}c=4E@qZ*GE9miN64s)%L zUQ(OT5Wzx)2w458^7+x-{Tn7pD5(Dq(LaEi)Ms%akT3vfY30vPHYGxJxSVy8BWWZp zz_K?Lc%ZBCnq$gd{mwi|D~s&_UI+ky`i6xn-vy44on=7h@X(c24%z$npRNCk2gwy->-cC{wwP#|f*0r?At6bonxpSzuVWL}^>^tCW4KPnWU%G3}%@G&d(sw)*ofLyu8`tKtwX z(j0d%eFFYw0A%YERVv=mUIQY@+9=e+-A6|oX60zK>m2q{Rm@wUiLXoE_xn|aMqkU< zQac!O(VzEFkLQL;LK3dx=Z`r**CUI0o>A1;c%vtEDjJMasl^9-d*99hSZIiJ2}a{I z!#>(FM#W)Y@BP#DSE1?ojf;JwC-k6P_+ymeP=|qB*j4+#eFy5#6@E0g=Ui>!zJ`*; za3CFBV)zePpvj^GIzU*~fH=DvASHU;JWi(pn?cqvq`$$Yh;I8y6pLnbW}&Fa~*h=-8&59C||9 zUmfi^#@|69Zx4I;=iuTjY)GLBzUJ5%Y&eYIhA~DvgKZkpL>Ka`4Kp)dBT+Ddf~({n z+^@Qa$kA9+aOwKaisJFB2QU_9m2Ub=fC>=y3c)>Q4Sp8D93^?h<{wl%=8>IW2{k<` z{Q@fq4;AVhIXSYQKj6j1Zf$J^)2SP1^F?%VxUc{sPg!3Q@pq)Y01xeA~Enw&iI2QqT!t((7^R?;2nRU&4eZ@d8Bci95(a2tp zcbjKpfLd@$El}Gmp$`PX5(2f#Kc=SBVr42)$JkI10Ji@nW6%&1c9u=9^iAr7barwy zG++RKO}hrL@ImqfsV&JX>@tyZsaXSAoa*g~%CoCJ2Ns9W2FDU{s;8@qG`K0G2A}tW z+eU;w^iODW1bIncjAS>iRVM#0l1ltd}4=CvL8*KkmeXfGzz z8E`{l1f-=W#^kg-RadpHEC3ENpd8YQ9RWX{oX6(v9Nb_vap7zT1_SW+EF(0H(&>HB za6*$Z_>9|tO9b)|8I=-r;5rDL;Jmq=hYDrjxSr^}JVD3L!lLg^#=xZ>w9o|R3goDh zkT9P3`*?UL#NUvDvgb1ZuASj~<{%#F{O1>?m(D%d0YuyiTDIH%<7?0>$$s<-EIr6h z0P)@G+0~FM=c*5Eb0E!Z&;6bFwz#n1M%JuqERNp?=J(#--Zwisu*JB>BzRH(9vt!E zV#_NiK$lt=AVta>e#>)gG!QNwCxbD8Vm`c7NM{eieghR&f3!%G6c6fH%NP4w07R^} z4~NaReGa!tBzL|h`~Le;1uce;UXC&h+LGYO&Zm|g|99xzmL0#ewYHX;IMLAs>UtB7 z|M(FLi;71?R8^pp2KgV{%}n5W!!e&x1EaKy&DP`UcBprKCfy_;^g;Xk3?f904mUx= zC7Kxvz%|mx3NB58{To9!_GdoNDQM(KUx40XDk#%ip5d8wc6Rc2J0`B6BXLkDNy$G| zu|L3(1dp@<9gPOM8`wx}p<6X1!E;4Eih}mZ7O_1$j`W{|;!zki#`Jw8K8*Ok@`)QmYV{Bp%BoSg#g9iZM;^Q~Y zc}av_B4~vsvu7Y|YJ0~gd_YQ+&lMHlpvGog6@uq0A^jsDdtQK=%``gu-bOnHkqW~p z4Oaude~YGs_8K~HlP$XSQNICOb@VC)J6raD4+$#vaC2(`rGmBG-ysV;BB!V2yOwNO zTI>?Xg78=*#7EG6RkIv)Aa4KV_=t$h@X(%oSH)P!2_#@r2eUUEe2`O?xQzV}G`vG;o9 zdgY8Y6}JOyDex&p{jc(_{2R*tf4^s}p(1;QknJgJky26gPzj|_$X3XbB^jd3Q({Q6 zRFW3$3z8xvOG-$k1(obn)*nUxJ2Z58&GSHtE325y zOdP%LpuKF_3-s!0z7Z~Q0YvAS)L9m&M2Dkho!~rjKM)2=Rn4!@b;WvxZv+o(J$r1$ zm!Z&|_E#%m4z%{yWV4?^wsQ$Tba9m+qdl7rkeU3rl@kqa0YqnK)Sek<78Xhmd`{~? z${~5s9nlk~u$+B)q^mrzb0fs)5~$Ty`R10czs3wSwAndks7B%UEFy#;3OC=s|6G?u z=8YRX(%O5?P~n=tX2law8#^UT=G3zzVn6klzJc%nuxu zKFzMyo`ombmx6&^#72bDy@*Ganoz5AhsgIZ%}m0>S6lOj)$Y^o$9J6A5u4$_6F6S( z)A(0z&Lzz2wy^HEvTQ%VYpy@vrHz6*)b;B0fVa0{Y&+}6XNQu|)7DzE-DMoMH1{*u z2XtM$S6;4wje2F`;?>jRpT~kn6$*VnXZ@+?u;y3Q{|=!C$p#~UsY_FHon=jV2LxQT z8R+fhmbq1ru4Jfc8yHAI*$7}OrvGOcv)yQVdYX%mJCtyQ{D)g-%EN#EILjOo;Nw%& z*O&bIbvNjk8(`xD;&f?KyCuwD=JE3KVlD5zB>Ipaf`IVfQiH7}0s!hhG*(3|R-^pz zD6%E9=%yE8!PoXPgtSC^Do*Zm(JzQUcY`l`f%IKWFn}a zo&D|Y?Fv& zdfI<$)tW`pWhBI{ML`BS!+ESgE?0{ohsCHA5Hmt=0GT zj6eMqC1XktN>>7gKjTD{b9Y1@h!oP9Oy^Q5g#wc`nK;TR*@7)BDK%9kNBZ#)CCmZt2Z{3Qy2gi*wgme?ks^_lh7K zCw1Ox|Mm6B!PeJ}1P?TSO4Xr}mWAwr)Kr(BP?XSU}?=y8>G>soy?>dj=xv)ns(%tr^XN-Zuw(&d#o z(ML6jeD-I<%6043U7{vw-Cg%WNk_+Kx?e$5fV5FZE-A1&X5w9pgL~9L6q%P+hX;-! z>=n+0G)z+OUcRIHF&+#_LG5f%$}11`we6eOM%QC8Y2{V$SGwh36xgu@-MQwLwBFkW z0A-46uvV_M+L5$*9`UL2S6ENH^yluRBX2HtoVNp5z7z=|Ai^gonOdgwmsNdoI~}p3&}bXTs^Ql8aA3 zK=t>!G)Z5(ENm(1DMr)$XDF{-ioQ%+zZ4xyaWGcS7-&jp;kzLj< z_f_s|`R~rfrFM&uC~0x=oHcyN2vb|c!~(lt&0VRluhFz|<3>~kREw2$A!@4<#ZyM& z`?4ZrzDD2U(` z7q7nP$mY33yv0mO9Pgi}l$_o9-eRErOSXHowzr?*Di@b+oh@mqb$Z}Zt`(Vf1ipIt zjZyyonq_~wYHk|mhU9sXkmq#L$;JQ^*RG8E{C#T=dnf?7M_!gU!b$?CVXBSSI zEL-jwUnY8SOs9=%^AM2%kO}psS9ge6TUVfA;83%E)y21-AgRaqBzne(lcA0wazvDe zPH)~zM^41#Tav4xtu5UIHx@C`%-rYB78ZYAZIIeK#lPT#H!x z23uQkJm~|6$6x&{Ht*IfzL6e>s_pje*K)sQ#_I5jR;(lUg@k{f zQ9QJ>Nzy(#t{^291L#kKQ;aAFYP2r<@@X3CpvrrD6FlQ1s5Jq8e$hrDI}uIAR*Hgu zw6LeA$AZa7=m+B!C9wIDz6i(u^jULcXJ=aWc;7S&z3aw!H?kCqj+msPtc%&6j-3~7N3WkI3|6n#a~^tD`vrK0fP zx7{_@u3emEmva`6JI9p1T9;UE49W`=w}oop4q55pW5g+*R+E*XzKNx{E`TwVZl{E6 zV4$eJzCLD{@&P|TGhg2bL% zD$jbWw;5{%<+M0=e_R7Buk$Tpvd?ks2hL3RR{>bz1rVk}GkYoODkVZ}GoMIT94vhC zJ$UnAOoGW!Ta=p{#h%l%0;gYfyOyn}tZW=?oT?V0p}XKKXX5CX;qH%GwuPmWVzKh| z0GzM}{B(^S5nA-tgK5(zOs4e>3}PVJal4!q^-(B>baw?@&_K=mB|=$AUx<>wOVCv} z`Hve1w5@$GyV+KTnh-~EUD$A&087RueLpWxw{UV6@Z-`@vmSX<`lt@Ed;w)({QkQ0 z8cl|LOK4no0yrEvWPtU2UfVjUMeIY54zG+bPsK_}sUO0=Ey5;B9v&VzoJ5;W#S5Wx zx1pKy_~{3BWKfHov@WeVyd~KDrT{<*@dvZg%N^+Z>&{N5i5|e{=<1L+CK#OpJ7?$9 z&!2Col}kQf{B5ef(YOzv&gzmk6*w_h^JQ+%f-=b7KFwWIPcH?JDi=eJdXNj(t}Bfu z4ndw0BxFM9A3kiyv&4-t6iv!yh^IGVM+*y^wt}JHVjDQq7At?e4quKg7(T=o%vfL8 zd90o{Szd>?SDc3~AR+Z5%Zh?va4F4fkKUR!_m2#h7dfzQZ&oUG6@0J2Pv#dzg+>&+ z`+IvcY)PsSvN1#w626E-9Ud*U3N|-udoG%(Ekfk48}Jz(onCo|sOszKb%p*kPEN)> z1bFDNZ!%WRUuLInTz=V>{9c1AAa0fLMV4*m=a=v%lyMVx62P7QQm?UX6@K5|u3fDB z>sDu{8^myP2AtTb$-2DW3TSL=)G5d7?n8iAFHtX8q8i=bZ>*}i2aO2EHx}Vm+>vST zJ_hB86~)p$EKCN7r9{)IQy85?xQiCUI9mV8@_C-{o?p)1mWIKskcbKWZrdeA+&$%Ip@Afn4#Q%#nDIB$4|@Ou`1 zJ6V2~TNnT0ab`1eXR&dfm{WS z-MixTX}PCWh8}$z3M_VtFOXbj=L#hVx7&Sj)h(s4>1EIa!R!_kK=6W7Liv3&6iqLtl2g z3y#d=CkpV;K-yV7Ju|b*p7U(dx+h$Oa;2Jd@c7tr7aV?9goCK1gl3)hB>4LlF7+Mr z@JEGc2LgljJ8Y6-aadF}1Ir2m^9`-pXL#AFu`vpzG{iw~PAXO$k|B{VJs;n*q7+G+ zWFYtf(=^**tCiIc3<))c)Vq?AYb=(N6(!^)fFcwJ@(jQvZJSX`VJN!(z^`jOA1kV; zY|498N}6pU9DF*><&ASMe_GMi6JLOM#oUk&be*3PCd$Z2f?$_f)ttTiaoF(H8gm)y z?K6~$YCnTTFxi&$h5sQoGwgF3VE$Cp)G*vNRa8!9MJXw`3to`tCv(vy8(dv8-TTJ= zo+d;%m-~)@9Mnpjot;yRt_zHhjb)aV?ali!Jbb35#jjRQ?86J(B9UUk1LX3gEggF< zlW!ysXEypFEUX9(wLnKAVr1ccjfY`IQc@r92P) z@1hK9pf)`K#p5~l9FPygdXmktbj+Ced>#)y=I%BMx3Ul8=UTaRM;fgV+k=|gRtyV* zH40Ogcw2y%3}u-F7D0`@om~tDvTDzsJqztQW(%uDGtpF!;6sny@*E_5QLVZ}wfNZB zh*&~Py{Twg?c@9yqRUTCqkP!8m&~%YbqFl%k0&51l_qsKZfCPOh>s z3c$^y$189Inysi3)e*yIC!?e9GWj63N^40j#zVOI`B~O@!1wdR(4kcCg9j&GPkNH~>IE8Hc;J0o8v@ruKn$vJ z221!#o#Kw$<;Qz+;Ouoqhj+1IpeSS?X$z@sFvN2c3^ZK)ibpFo3KphYP zC4D2;WKJSh;|B5z7ARpc{92h4Lw8|yxk&n=LRP6aUXr-ZRGPxLwap|V8(e}LV-K^P3&d|Bxw;SbyzAm4@CO-*)ArAKz=gjU|?}o_Ihk$#LRdrZsRC zpZc<8sb5Y9EjB*pk9H%w5S$|7Z9Xm-9OxvU^RU-0T< zxdjW3kA-}%))h}44PV3SJ=8T{hxEob`s|Y1d`P>Z;qDOYXa7~kT1jet*r8XCV*Aq`JezTI-d;H&Pykp0HWVl$o_(`=)MB38Q zvKW2e>-%XgWe=bf5%TnGdb(oda(@pG-wxP{ z{%;Y^v>JM8h7ukCA(c>5vm?4+MY&Pd)W2VGgd;$f)!P2j7H+4zx1*8<<5-iZv)J!q@Xn?!IpJ-VV%7IBK zkGQeLO?##)W&Wp+ucH?8&=YqrmjdHNcmT3Nsj~LAgu&2~hv(DMveI{2nN7PE?18(9 ze4!#hwky>}!-GvN!!rEjGT1iaOunMS15#YhFYV?w>=E8RK0C5c;$$4}-^X44Vnro# znFV>A6BwFz?s?FT+dCMC4$2@{S{izDn@@xiOq%qkUTij-);VSblj@X!qwFeam-j7S zI2%L=e^PRCB5V)-YQn*N3NlO!riJ4kM@OzbV$~k3@RU@0?oO5P%BP?!)q0|He$*d@ zXTM(8kG?Z18AXZ0`$$mdDg%ScjFvqh7cWy%YVYQ)Skl$yo_yhgdbYyq;Fo6ugtPD66Whx=ar}7FIhFNWHk<9+ zht@qh*h)LOhzi@mJg<6rk$LmBFkjQw7#Rhjz$AtIor@L~kSn@UPb z@&ek|CnP3r8GOAe2EU{Y?-iLlSEC8^EJOk!wa$1C45F&%*$#P&v+PePcy}Xr9O>8) zCpYy9)RaVQQEsI-_nq*h{a7GYTR~h+2AYma|Nik-ylJ4nf2Vo(5qrF+0wrtV0nnO} z#wIt;6H8XrB{DW#+&$cL5kT_jtM{_AVFh;b<%6EP4<6XDX`9T%OGkN8ECN& z6l`Cf9u@|lEBY$xFjX+?SsMD?nM+#`6@b6+}&pw7JxaIMN=ya~rYQZ#AK+4JX%OQ#+7pKMX<(k}Z$)k@nseh3}( zCI6n;VQpTqWTg5R-axwKTKKK?*un3w&p8wxyxhN;>QaO^K|mr56s>dOF;JCP;OT~q zswwS2-){_!(4Uafr4P5)z8ok;o2(gv0Bv zGAyDg6*sZ*M+1VW8X0-`t^RKW(6E|7Yd)@;6H24qKmd>FaM*|`Ku5sk*Wz&e+_KcA zAy~F9B1AP5>1ZTtRDNu1%(DMCP8+yIY(%gJB>7oWTHW`J*t3x60Z&ssYNiVnKi(uz zg*33gpbbmqIHjToadKW1$BX+CiaTRJu`6)7$S8(lMsP*ZpUOTngNxrcHO2h+;fby9 z#;sfXcX-SSbbvv_|=d5Z%+;! zFQt}C3IAtktsKLMJdJVjyUu_saH(Y^S=6Vrw2~CHACem7&Qqpq}LaiBOG^S zp!4n-uxt4>ImtDVMeWe+?J-=PS)tS(7%L>mn89vSpiAi~)L?KJqn zbNxWwyM&f5UEb};VldV+DD>Ms2%`fH%QhbRt)HKtZ`r?^Y6}285n^N%y_GTR6%`fb z*)wsB-t}nQ6Fg@~NJwT*j-0>$P%o1i^Qq4ap#!3BUF4#Fej;KhU$5c%_^S?FJ(fE> zB*Ic}V&8hEvT~Ow>8bIPU-yB5=boT9`g}h$6~|b4oC4RVn&`-x7-{IWe$CnJn$Bhff0hgHf^gJ;Dy-k zndS!wBBH(e#jDszx(zZpf zcXEdAty{O+&W?8ljG99Td>9goiz#SmXzZky@l$cn*Vkgv!FXhss&#QCC!gCYf#`3!Uqo{j{<|o++ z=WFjqP+=_l989R&Ym;?)LasY=uG#%G%Y4_KKGo71Q3*`0)va+Weayv`qj5(wf{~N+ zIU5g8-JFBS;wi@aknyt?p7X<)b|$mvi5k=PXmJTN1Ywbk+}t5i4~^vf6A~sPV$zdB zS5wo|PgN27vR?EO5F`aJuo`}3+7T=Ko3z2IE`VdAaIB0FVk|+g^kugF4WQ7)dQE*K zqV{1h6to_32#tKb+kIU#;_?9(*NfiC=E=CBp*NqCGd^WzW(HN~jL>yjTS8{hXs3y4 zR||q2UZW<`8<4uY@&o#sdfv9(-G%nv3JygqvMpXrTef#EY@!d>Y;6VMOHm8D?U~7% z`1tzXiT?S{1;tG7$q;Cm(esWHfsi?l4?p~6O;*oPL9ywYVeecxe3(<+W_^0Smw&ux zWF#x2Eu2#$)`vmO?dm#7>>NQ_XI5$H>)ZI#sgY_s-Hl*UGRkK;Sy`>iS=C%PyPY`{ z9#p%RtR?Oyg=k!7Rg9n04R5A?YGT67irZPdn$@~Bbg?>AXf#u5IjDo>dGH{d86}cA zxca;%Cpz@rGwj}CYpX(L&3Gvs1Gd@dOG9_+;OcJ8|CRuY#de^ud#8#L=Y6fa=g!O+ZWI6qD$m~hDnAgP`>VOA6U-<&~aly*uWJC35&tMdAxLt#zW<*kIs?0={bG|M%m380ke40^ZKV+tioS^m^VCd8|>$cs2Q*|7S`6KgZ=%C5ZjE7jb7$MU!Fw>!7C?AEVNos^W6 zBo?Vt<3?#=VUapBV~EGN_p#sNoDNr%EV#MY+_umj^Exgr&K`;CkcS-yBo9f#|2h-; zCOKs|R`nirQ+xj|vY&0b!x5gfwB1ZUC@UkgcT#0XOH}2wUZhv)uw2k@5p1NXUOVh| zvNese8I_e(^m!;p6TGlHkBhWCJvqXI_-V+VobUCHD5uDcXZKrN?wYPFHGC1Q$fd#) zBY4>xN=h$&_UQ{f`i;O>E@>ExVtm{8hzM>>K)|uKx_ZRA>C)AIZv{h;WUjgd)tx(; zA6}z}3Jl=)?uFp#2UGR$OgF3^k=7q_-5@4bF1dI6-o3njGk(tMw68bm1+OOnd`4kE zuR3tw1SniTIY|#O{N=oHyk#G@b;o;?5w5ZTXM|njZK@SO(3cB{U!Q7dro(hXM_cwp zvwV=a$Jp;9QtzAe&b(^2moHGLC^`6FF&JYw67>#=;xN=4m?k4oQBm1off=GBxfjpQ z&q?08uM0$m$?FgqVcJZ0Yl^r!Wntaav%5&ty{^@L4YB)zeeM);5uX4}M1skC=FN981MykIXbI=WIRSG`UV{`dR#Y%{3v zTe-QpD)RC~pWr9H<2H)heYk2+6q;JOP9)MxdBhYf&yXl-OH0mLw|x#-_}?n(*-2I; z@{LX=&iK*!cY_~?C0f??01nv$lm872s5gHip;AMl&`rB%cjZ#HRrKb2dwZ?xF81?F z6?8t*(b3th4Hb&#f1rIQ;k*&+vOXe!LNpFyH8y15M35z#y9LXoxGKQ?8ZAmk zgaP=uf`TYf&wd@0yfi2D%EH!`YGh=D8dFFK9qc^0j7&{UP5xbG3)sS#dLf@Cn8mIq z6@_Tv?E}zPvAfur{R}(CgzB-SxP68y9Ys6_{k{z4G5=dkW)cz-lv412({}#g{GEY`d>yw*H;?9?6WZ|1Rj(Qi zcfX9pY#5wxlBJgIzomDPTh}rY<+_ng$*}3 zUZh}s_1(1X$Fv}z>@JJa^N1>s<41i10}3W4Cb7NCR`x=)0m*FWjSjhtm*6se>IDgU z^pSzJjLcuML6`V14PS1GK7an)+J_;B_TNxyTAG-ga$ob|q*tBk#@H{JL+5{nh4rC>lM*G<&RM%TN#x*#yeh=_>)u0a<_+z$Q8U4oRn zyB=@SRII%&ju%UrOtF z`R=_CAa1NJipYB;Crk8>#f(xWp$U#K!6j#=NkwSg9dx zSo2zldRGF}*LratC>s&L1Fh%h=P7#VW0S~T`2#b{j|yq6=(WP;vp)?Dr{A-)vi|6K zojbC2?tTEO@9wxc3B>T=fi>Wp&0soFbvg=)S*qJJZ*w%=-91yVaM3ci}@q^p6s&$0~?!vUT$uU?e;kq?!FF;=FfF|Sl@*1GGNYXdNm z(a{-O6p7n#&eq7kcaM(l+&2fM&}GKY>$svOa{s=Mxs6S2{;yxhEhe}KPfs9jIyT<> zapCg)K7Hyw-)|aZMBP@;c{81^lKHrpEBol>eVAeMpHCzLdG{76>F8b>VItGh($1#_ zSy4OA#is4aa=|q7D!R(b%0TIVd+NwP^qpN*GQF4JEAoyf-^eJKwe6kH!beLfp@g|_ zYerI1Vl<0QlQiDC%2FT@d$#X7;~(%xz3_ByfPiYqPk$AsKuAPXrFQ8YnGS9y zyI|s3j`!yW%(DZq51OlI@ddr8KgTsPFdMS0-Dr6NM2ax60uMdj{ z61Ucx=32rS+1UOmrm57V%G?TTDUnW7Y1gcDU|+Ym^VvXuq{JLwdMO}&kS$P4)cwFJ zG8m4+@_L>)+<@*mmSWNa{QZ?w5tOs*PLIG(saCG}c%@_XM>lTg?q!Fqwc^f|`%FxW zJ^Z@i(a~-#&1kr6sc!AnoWo zc-zX^+1YQ^v;-szNi7%Ch`I6iHy20KAP}h;c|U$oKY()HX%5WO%cQB8Z~^p{?d3ky zfFL+6?F$uzz$oP~aBu{`!JYo4P(13474BUf{w4kN>9j`ONjM;u^8u^Jb>v&;EYzA2 z;SZKR`}s-Q*%gfRD|@#XYyIYZ@fUN44>6gFL_v@s+rJ_%Kr%DJq7A&>@EdAbzUg00 z?(jDG?j-Jg8MZ>prgT>yjD6@_Ce#`~L{3jn!|CX9d(HU0HrFe;a29Eu3TY~M(B8ps z1!_)hX<|}RKOl%xyUhlw?gJ69fLC7wHb<3~ZEzn$be5>1s><5K@2TIP^;%-E39;Xt9`Dv_If1$G$6$~!pC8uXzAnXdeqH9y6N6XiMgpw@Q^;?{py}-H z1$hG>H0lyR*`M(Lk{nP>Oxp9vl?{M1G4YWfj<%+Goi}s>lQRa1l@W)rB9037bK`!* z6bwQ_Z}_^{fbMU8NXqAWvbP*anVX;gB_t%IgWeaxnM~KhGoh(Z29&%5-zGr{~&Y zM_iao{AGrOaL6oL!e*=j1z@8_Zf1h?=gSzmUjB1xM%W)cfba!u95=Uuf`j|X_-`bd zi|~QUOn2D02^Dxnh}$0JX>slnna}F`C>c& zm;`M@21r1$L&^wW|GBw2IUfe{$nbFXkhN%DgKMC?~OQq?xHadStswzi76#(Vwlbu8w(ix1^DH#U+jlMp}*f%l(b`7<)A zK3>Uqz)Lv=1xnjmt&%BF@$eJ+v9rwO)SXp8b=RX<}!T}vae8vkWC?eh|g%%`vG z*!55`H_x;xW}n=o07L$!uA7LF_x`zz0X0?GfBMeEWEhEhbWr}S-S|;EBg}9cAoi1) z)erCb&~ZQ%F`$a#uR-7;nbH@NzaR&tks-)@wSA4aMo)>4wkLnm(&O4up z6U03$XY%W^vTj&C9wn%GTi2}5%k}`RD&&5^<#A>|dFl@Hfm46Z1P?V{UteE$@Lynb ze7rch(V1oRPx>&VxephPR8~2cde)YfyqTU*ga<+jsgV1z^NiCG8( zqt6t0p8#TGV5ol#aMw{y4Y9t`tY4~?R9N^ekM&j>+qnK!OO|g9eM$9ni^{^0c%Y4q&+YDxNt$*@K;_=uJSDAAho83E`(Ni zCL=qWB3C_|Uiw+8xmN5G*CGT!8PypJF}J;p_GvqhqkLId{NiOwVq&5TSP_~u?idj0 zj7wZ>hkkF55%jVfzm}!OQ7nq~!aOv7_fDHY%H>>r^Ss}}asIAt^|mkdr;qxY5kNTy zKwtU6!x#c`$^tGKPgdCup#C->RJY93)VU}O22;N7R8^QCEQpX5dt};1?fTdMR-j|~ zuvLNOAonOR57Dmiib-rM-k>-DcA6BvS@@5N%2o({cJ!2|_9)pYvF19p(NL8$&zFa2 z9!AC_*}_LQ^}8NBg+|Q-GP1HCRr2Luyud8%E%)XZbDa`Jn217<8prFORW+9%uM_1Y zv#}hMqe4tHOOdE-fCZ}qfArAk+G213>ZxrPTrcR0xHO8nIDh{9`6?qfS9f%1=qTZ) z*xuKUaPo3c7J3zSNkE2Q?CZ6H@TL`P&Kjtyyo$pKcOros)Gs!@K&oUXLjj40|@{V@V91afyk=odYFvF?*F`1k+2@4ps_<_~zGL zL|@s6G`SR%T2@dMl9@RU*VcBHnwU5hMMUqR)D6p6WYg!Z@iH=EDk&}1#FrWxNr+WQ z3k0l!I6FJj3E`^l09g`FLtTBq0HeWzM86!5I6L;ujb?Sf7rv*N%EKlQK` z{hJr|^=rM8lhcL8XqmN!#M#!<=gP{r^0Kpi0kjI)&)|CiT{R}USv_lJWi@6|s%12W zntMczFZE2^X^pzWBM#1v zs*Lz)6ah2>OIzmvnDyW~5xEdB5{ZR{x3aRc8ChB7Pj(Y7t=d~#v#`iz)_{P3W;{1z zd^a#(#oGM=?bD#3AbPDLvP3R-nhj#_dtB)ztLk7Zq7qS?QIu4xFw65@S`> zFjqj-A{sw6ElO$B^K>##cW^wHBv9=Ny4~t==vSoq50yGEZSA}K0s_M9fhz)G%8DR6 zVCBN02YPyT?0`R~MPXI(BnhB)+GZ0YjKp&Vh04Td#5WE2X=3;uJrd_&V{@@@45SQ{ z#{+U+=H<(Fc~}ygv--vA>S|I=O^w+7`!n>{Te8%$+{p;Nm?gjE`QPI&3`?1x)8xwM|;+}553k(``CMby@_<#QqLcBmqDR!^! zlMP<-;CB4Hy&-bO)Qr)r{Cpz6E-Hmb^A7FS&d%rKDj@Xi3<@>{E z(fZ%8VPV;~q$d$nRGbIIfU)bVU3cr_GU~4KTV(|2Rf7OQRr|7@T9K5LRIQ{S0Pa&? zFZniNzF^}r(nMG(rJL<7a(!tD?h=1knZ03pwb-|$>E{*~IXg6k-_Jb*I!R`u?cCDR z^0#4ONnrAlN^Bq!g=bM(wCwfn9jg+ef301z;;pR?9N$w}`P-ZwTjzBeDk>s{#S1L* zchb`@fGQ`H#uAAnVF9`^Av&K8sOzaY$trIi93E;s={BY&G{D%x_LA5X)?ei_!+c0A3wQY<@t6WHf$;Ir^z%_Qw%WXX`R`e+|AjYS%WczhOOD& zKRLrv0XL2sT0iIuV^y393Bem|C2ZUmq6BgCg?gx}{U>oU$@lNyi$NZ5)zs!n&sU+V zEaXwsk^gUfrP$(jReg-2ZcWzNk+*Enf5+zhG#@n32WZe63ANylzO{jLEz!!QiC!7jsi{dbb$P>7NWcK0XE^wa$B#D@puf z5QvmATz(q?vuy@Ys<@}OSM@ZEFQz?=E%0P3>B~KpHKL8BCCKkOBQ&Lg6Z-Cj|bMp)kTydtVgZ5t{bR~^2P)4AnTSD$TPse*l|%MadTVrtrPgOoIN zc4p=bq}^)~HvRqm>YxW=`}q$2B~i0|jD@A;Wle4_*^?+*0wYo|MA602j2hwIATtX~ zJTq!XRWbBhU{DaV4?|Jj7bH0|7cLN%qzJ_K_v33-N||Z}9>?$dK}Z?J8t-B_vNMm< zxY$dV2u3fbMaos5tYq7Goq=QC5ij-}9CK}MXi@!>$uK!)J7H&M=ZVOTD`r@5t>qRL z7RH6`lM8SQuQ8(@MTCd9C}^~v`YgJ#adA0}Se0p;=I0|(m!2R1GvIa4VGs1_>QVxM z521{V3~4112Rpkx7Wpg2M01{eLzEh0Z>p65u_GN5LClqrHn6W~(Kvq)121dm`XU^&O|KfX>J&LQV$8zH|#z7jXq@%M2 z;?;=d^ZZ#mp|_nK9ZnS#c+2KRMMW+?4C2f0n@^Pd8>as~@jNMUIDVhstWb$TqQ1t& zSkhZ4hL_t7Lk^8PUi?GKk^q4G;Uj!XWid$%fa=*G5BsgD-;AXYh$@wIb=x&_2N;Qi zFCNwxMM#aR4X24oO2nQObk#|Tx)2Fe@EwlJf(o3LqoSYH$_Ou{Z^naDEx~Wbe zh^LIu%gxGK$;rG!b81|wWpKDL4&v0aU!2;Ko_l0ca4#z>D+hN3up=OU6BY`&Tg3L- zHsv2=HB^xFiNXJHeAtx$QhM`dNe1RQ;HjM*^AEpl!)J5CirqUnl`EjKNwbUZw&Uq;OD z#m3T>%i)2eP{f6kRS1#VB=P&2!#~Z@L=Bg2r7yaMu=On&wyFbR?BBsWc9NInqw7jq zS{!bM%0SddC9?&o|ez#$%*0?MBKICHRIwt!JyJokjX@Kc;O@YZ_a2kVd1$J9n$)e;OcB zW$bjq&azW(Pr@9W5p|MWxH3b`q$7BMad%rr6I~K)veFP)b4URLt$JWQ(ni|jScGC~b)r~-(_uJ8a`A##luwZ(F zQEek4*~SOnYH4Mq)^{E!?m5!BdzQ%e)G_$li{atnKrSUvuYPb5V}2U39qpE)*zvBe zM6)@^=|2P@VI_z+tK<1jKX@IYZH~&y%CfVx%zplam4^y~@VloPqnXR%qpuvnE{b% z_^C$BcRopbYFl9fRm$Prb=sV;{xF}kFLPYc&a%=qs)h{Va0-+LmINYnyKW6(W3XK*##wtI~r z!el*}EpN2Oy*zl4$jNYiZf=WuHcl?3+bAmRia?Mkz4hVZ=Cp4gUcCzOe%QfB6U_R* zr-7i`w{KrG8y3=@&4npiookkQo*rz2rOYKDBv(*BGrq%hnO?zjjvpNu= z!$34w>7%e7nIP421K#f7Vj>Q7ikCtEK2AM*Us!RqD@=-x<(E-2`A%CD-zErqj2#?G zoIpMcY&evo&I!V9;u|C+xuDC3ygR9t2#r}zT_3DR##WrRNF z_q3fP=s*GoCiV2G?>v|XKML#kq_b{zak0En)x4y)r{`x6|9St`bVD<@PDPjPA3tKy z%F0U^@V!Yd-iE@K@bU9I#BmaH)ezu{Qh;+iSaPV0dLF^GO+rRSfJIJ^lv>)Y^syI~ zl=KlpUjCt>UcGlmt*bhW|E036^fNz?Fq4*+?gB#Gdse6dbh>~P*3Q@Y^&ZcNw%DM7 zFrU`$Pf^ZDXXmCDp8iW?W8{ZHUFUCjcz6t`+x&up4d0mLts+6w?vMT0dAt_S>dnwt zmKZGdeH{pk9zbpX$vy>$roHYtmY)JL!yqh^Ulwh);`#{)Bf)Qq^!+k|MM2BAti+e$Lv!`N4gcv$4k zgUC_Oua$5RAS890q1kaeI+_uVNR-nx$vg&_?akMfrSbZhZy;RN$yL9ULo^C+oflW$ zNA*O4MgG8dpI#N@nwkA8`>>?rWH!Pu6k6p3;35&EWq< tMwvDMe!EiS%OQvpkELwoTU>oFZ%Q`xPK^xe&44yMgpz$FlP9h3^MBvFx}X36 literal 0 HcmV?d00001 diff --git a/images/vitrivr_512.png b/images/vitrivr_512.png new file mode 100755 index 0000000000000000000000000000000000000000..f3fb5c9c2046cd71cc88e2327f95a94e9064c1e1 GIT binary patch literal 76376 zcmeEusmNCL_Rn;csRs_ z@JaYn!a?}!l&gZW4l(@r5?efjf0H=fGH}JgL5(B-;To<4--Hj(yWP-t({{9S^LXH5 ziR0nnA!y@Z>uUbM$x_hK#X4<8h5-lXB98L)YdW6k%i~_@8J2#>$Ag7!cEj8pSI;Vl zQ4zD!qpwDUaO#|i5qMP4v~K@L?$M~}(!h3~!@ZJi+dq#!jNB_Iy0~A2$5tI%N`Qas z%;yGz@|aWQ>H}@UOV}upAdbuP=_!&se%&cvvMf;>$;vA|epwz|ZJmC`SqBX(VJj;R z9LkD%5oKqS$(bnCPDT2~MYZ7E4gEbpVTWJIv_0b8Ki~6|N;hXCFPcL+R!7h*;)O-? zMMtr1iB0FuJGH;r4|dH(R@0~yR7nF)DaUf;Tj&feCuG0kZq}zXI%Py?L>i&FP&9zo z5txHEW(~POQXd8VWS>1}E_2RPJO$NCfG#~XkxS+o(z$ESvwENSm2A*X3_POpw97=9 zfz2#}x>eL&2w(8jFRjRL1Ttz?eU|4_o+m}di`4F?X zsP2VdxCtxJaz{0P5A2EaVs@0=CtsDsaI>D`ZiBh&xSzD8GPlDyqS@uRT;A$T`hanj z6SKq`GX8hm#apOm8?<%M4sODYNNwUM`~b=^v+sJ)>+Dyn5{`s2dUJu*Gnnsa)DxNQ z(g$B2+Sz62=Ra?5zRAGIn4h2j;P~jz`^w64$4R;E-=j*lwtNRWSngL@+|q$1c%~$* z%eQ`#!%R44g_=G2SM|`3X?9~!NLgNN$$$6GtUaEmGhH&ydAhlNV5}&=Enp(Iw)Wry zmlF(LQnyZ8pjTq`$$-0oH4sWOEN zw-KTF2Skfk1l3H6acnbEH|@?WVoGJu7=dZhd{mh%r{vBl><;2J#X)P*Zsd$&=%;UCZescxRz~~#zbbQQeCCMW?q{Pc2X7v>adI*hU z1L>zj0|SVj?pQXGP;)sQ?ahlk{?%{s&(`-2p~lWIT0Z>>EA;?(Q~WxikX3ntNKP1% z{28+E_^4*SAS}*{y%iDPU8iozd8WCpw86Z2Y_5GzrtIPVV58M+dxD#r+vejPmYAB) z{?>TCFkm(-PqRL`FUjt7!?L(lC;d4=qt-U=taPY<)z}^Xj2?h z3Ra0%T+k@?J}gp2AN1R!X*4E5g^SlaEH~yhM;e_vW*( zOU3d8#pJAZnimscrv2PvKAid2o9J8cCW!>Q^BT&_V#ma%!b;#f`T~m zq?bj8RqF+&f5PT7eTs!@?Qs%AsF-hELdK1!17)Z4k6FYPx*lEU`-_o*!4vB<*)h`= zN8Oc>Yd>1)yi`-n?x&7))!;qu1j=H^-i(_T|j zBJvn>CW?|j@-%fn&?26iR<=Y$hA@qO+5$6({_6W{a(|%81;jG(M{L*LBO9Bk0YgX2 zjk&J96&IXp;gHphQ|5?>)|U_SDRXi)?_R;Y+3W945eaPyr_;WBH^J%M&tEOH+sCAb zcZff+xWJ=Y=TOZt=s#$lnnwi%1zz*glWqBVnW?^|Y`>P5Er!c2W%d`|?5*?>8c{1$ z3w+XtCVgoq-E%bWl3=o2e%!r#x6E_RY_2mUPT^hOwU4`P0AU|@upC~EvZwjH4QRda#Bep5h9u5t)Mnt}1pAciFRdXQbCF-50klbZV}U?oZzis0<>OMa1ghjmk!>KhiGH4&NfjQJ`o{ybE6FodCeawl=?>k&v9V4f zxm8>S28KQo-Pf6Z+XnvYaarGbdt)^B96^g_xxt0t$WQ-Df6Tn=R{T+6X;P4-&7ZW2R_lj%g7#QnAQ=}gAswVQkf{oK# zbi%Ztny0n?cK)7zdJo+=`z{SsUXrPQ&5MaI=^hha{6^K6v--Zb@-HsaX+p=(D?ogj zXoILLXmxaSRws5El0a8w)8zZAr3gRaR872XXqZ;8Nxxq0dJP_D=roJ?66A>cWBAw2 zNT#+cgGO;ZPZAWnYvwaMq$k!QeN`sO5AT%QAeRQYX`-m+8nlwj4*`?9WSj4~Ini%r z)H7vL3Q8JM+uPIS`~3NuJ&hSs;PLFUTRbjCc2c1468Q}j6crU!V%mES_ckOCmMb=R z{WEzy=lz{Ejdg&v7T5} z`mpzP{LlDCL%FBEG&Yw?3Hsu#jKUWwr%ewy z+S?n;cxTd?DVqj%52ZKE@|E70-4Grju!f7E)whv8R1!DqGbP?+hGFyR%_8CgHLVLIw};Fn|7?EORuT2A-^jpP43!ps3BQLa(Fz0U z&t^p&wVZiB4Kh4i0SI+lzOiHq?tZ# z!hSR2{-=KkYTAompfpt35x4p&1;}N5rhIp$GwefWmb`osfAB=L^R&cPzqV__ALX74 zgRqdkm*9|68pRb^fvs0dmq6uQ51vf1&Q#Xc)~<|wgz5QrYM+s*U2K+M^fatYB@?QP z{`Kou{8@5xvUB_|%Y~PsKH8v1wUJeDxyLUgqo#?o` zjmyG~tW*IJ!o}UmVKI#isRV-xtDFSOR(epUu5+DjcioK{43L}h{wqCS+gfYv6EZq2 zJ-sBY*RQ5GziakvJWzj=uGP@^(cruax#KEqXGpoIh9h}N7b zJe*j(+|9}|uw+@`Y(FRy5)wMo#D(;RL%+AO!7#5rA$>TKd|L`mYmQ>+$yScjiV%4p z!EhyvN83AN9`7ywM3NRFKRo0C(e~5nlCCbYE7$Rz+IjM3_V-Pg%w@!iWMD5D#$+mz zvS*A_?Wf0TT)(%YEV?pfrS;lszqhhNYfgb?38w0oRlngDMP56rYiim93b8yvC${F` z9@6|Wa$+y(&B+b8?me=#m4t)RRudJ?aO+d<9~iLW^YV!(lR6nm>z{M62K965UEn_F z=9WSQH3*%V(>Hd6eSj?~eUun&mSKqIji%O+VAX4TO=E^= zs^`y9%tV{sODiEH)Gp)p@9^5qPyf{F{_pyu!LfI`=Q!H(f^oj0T&sU*sCB5!ydohz zL-*t`S*5~5%<27-KCU0v@<_OrKS)Z>W!dsrMx^cG{)&Nj2R zqnOO8T3G%&!Sq7CAJV~a{jf_q-QKcUj}PxK6$`?@va^x>o=G1Xefsg^N24AGEj+mY zH{ho-Hs8h)(Ve8=^%8CF7t*KIH^oeMn>d|D#1|L2OKs zx(e&`2a<&U>v5@v9exU2+cX@O{FJ#TIJ~8#9{xAh0#t5nMO76!gqUg~p%mW-1`_QzR^GcHXH1Z|WhB<( zReCnIWvFD(`3*FPr#u-mt)BBz%isjuapl3MPqY+gMxcdnc+4ZSr&@9+JC)f{={#C# zkBF!YS9SSO2K1=RSvqq45g>~VK@FL%w;6i_ydM72Jp`rL4GcWj(aq^r$4Xn(zb};Xbk+R@M-8bml~5iE*u1{pHsM1>s4iD_du*Z~$g0 zvYACh`E2yo&nAdIauO8LPste<78Y7{e>Ua9{QqjfLL%?cQrp2|-%Qp5yy0qRc8J1< zRaP|l5TS^VKF0O_Em&+)FV%FZeR^qp$@31hY(Kfq1Lby%8FtE=%pO|Se`?|5SQUtK zDOZ;!Dji2FU+8~ffONHnh-Bp*xAqJb?VH$x_EV8HmycA!w1uO&LEWpL!Lbl`)G z;~s4Pp8la{+NHa$L;OJXe=A2>61x@q>5q?>Tm%FP@euSQP0YSscU^(_f#1n<*{9$_ z72o^7Tg0R|qAFA$U+}fo8`(K3WXNl*A%&{0;K!^jSM=m2KYZx@D?e}H*ROV6mw&}3 zME3MDj1Vr=aLd-#Hq7~HZVO9*jNWs&Z?PP-A;`6x=(A&F@mVedHE!Ug_TJ);G_vOQ z8^o{htKU@pHa0evfw+N(i^1D{dFa#rmH{2b2+>&Fi9I8CL8jqVg;Q@L)#yBZ69AG_ zv-DT933?`NTU+CVhcY=5(3=)RL!A8^x^ynSsBm>h3RX{_4=@BN;u3K3hm_J)^BAf# z$Kn3UQ)2NG)oXjf8nP{VqIh{~@I7;?UI+*X)U3$dm4EE+zW$Y3bB6bz=j3~vhox|n z7F(r~O>10odb(6V^hY-gUUxq~aX)!kLo_hWc2^)$xW&`;ZGH6%WrYWJ!j4l2cPHn> zhA9RjJ8+G7N)P>RqOz*0%5~B&a)<2qN-Z|Gpn%V?%~63^u2c<{5ZvogZf{iUE+ACD zsikt~>C-<4I~!h|FpW&PdRGsw=$9+TQF+M9cL zi0_QKYKA!zR@Gt~z#({qY1DcKdX$5&P5w*wT24ZpV{p4_V@kb;~` zA>Sx`r3||OFY{cfnTLUQW!78SVL>8#!`yHP=_iL^LOg}S_zXW{_Qc?BV<)q~N#~9t za~H=mRbdT-sXe`jXzYfx)-i*T0-u$d(Bk6a_@AelVQ(=<2iTsw?mom*4S(0l`BMb_ zX+n%y#xt6hPdA(1dbM@qj!XcpLTQ>H%HoRXepD+=yCKtSF2y&|af0kn zofJ8;>{x1rA>6NaSA2F>W)lq^UR6|xNh|cWdIQB%2w~tbN}skr16D zWc8Gjd1Y#4rI+C9Y6O~FD@0;!hIfdAGL>oP^hS=Xw2=qIv{j2 zyneX5jXc*u3}aPviKlQ@pBp{q(<`<+FOeXPy0&B}@;f}a;2vmI=8A;X3?yC$q_ILQW$OSHvOF}Vj@oCogc~l6} zf>^_xiy^B$Xb;yP)u02r+q7UF#IBj%^SGdl18|C=47V^HAeez#qMWu}eC(45Fp{?z7?c^Upz`r$k zv}_g^7BH2g4%5gD$U8J-4JqL|vGlx3vgBh9}90?z&K2M_`*Ziuq8~*IkuyFzYjs;Mx%>DPh28dxx{d`Qse}TiG z8QYCKf1&7gZEfw)c@E{!=P1EbE2rMM{>X&8i~urww{KK>ZCD{Y>yDks(A>Ls--P&q zVR=3@%>KmMyi0(ItN8`~Kr^kL(I*Hyv~S%tijv#Eq?#m{0}4HB9XtpYH7Nu!-?XX& zL03^DejvG8=KBcS2q)I~$6ZQVnp)$oOppArjQE(7V#AAxdbFh&Vso?Q_Ehg16u%B8 zqFfJVISk~^^kjQ-tEqQZ&oNE#qrG!D`toK1fbq(m;Q^~HsT*T$bHbxRhw$( zE3w#JooaikGK^0tgRbA1NOSb!cDg}4wLNndk-A{*2f?T>s@lK=Q?7qgr0kuTm`E~~CpuIo{zP&G z5zSM`cw7i!HyYRr9D-##jbOvBS!!BZ9GTcv#9VbtV=X|d`;Bjh_&Gp8)P?`}~CLycmDUl8*Ku2n3)#pVntPhCz9ox{p&Hg5heU4nr{0 zt#Gi%yzHpvxIj}J?wMN6e8Jy!K06Lz^_VmL_ZJ&hMo4^RLRJ$_ndJkOm~fRkMY?b2 zV`m1qQG3r(4J^A9(L48Fzj_sPBmA86h)o`6VuAcXv2jO+$3$DAfZ_hfB|fx|0x>1` ziN0Z=LY8O+%x7R0p7pk6~ z`ezlY*@GVb3I;6U@&$>#(!taBX)gbx7FRX(c3di`9_?_SvXsD+!n0w1+7-P|rJ(nh zu$|>;uBNI^kb`}3S}tpG*&enQu^r&&<>P~a6v3hu=obC#RboLT`H$pRb_<4jMlvRZ zr{E<<=K0htf*idQ)Up2&LBt%rIt-J=9Zeqfe3x><;AqPK5si}FriTDx1^XbANRX0M zMi2e4T|1(TL4vWGxSBqhNQZk>mni5McV*e@k0$Y}9z6nZYw}CYBDnG8^C$~WU+4ok z0I(Iw=?IaB4<+`!(H3@-4H+Oa>D*bqu>%7GN6d3X7Ksk-PZ$sZ;YViGY!;!1Yf2c2 z?KzrmiLxGfZ=;qamoXhBpBWZb$J~^l4GRNVG5bC8q0qQC@dcxJ-|+D8?04_N z+TyrH{Px|z9cl)6pEU6|Itahev@sLNzj7cUPl(7bvP8Z>Z+!nt0G>4mZPsqoW8Cn3 z$!YsPxc~tKzNgs<*O~= zIKUz8h_|z5pv8k4Nmx%m%)RMHZJv(nd#d^zz2TUsk_65neKoWGIfaEC+EXy~UyYd? zO#!&)C}$+zC`HXxPe;C~sY#ievjTrWNl_91$`wbs!o`)9In1YExVan8))12mBnlZp z^7IJBp};D+UJ^JSS~Z9>9}FBKhcLAu-MRfCxOe8jH3EUyJ1}t5$4C12@83_7)fT&i zf;2((xbk>%v+bxA9CDwqU_y72DM_XIb-d+Igw2rQBUvKZ)$XN_59$5OmzCNvI*3vA$QJ6T^ z5NR^M`ylPJ0~-&clO$hF?)5!kLCDC<-Uxk8C%jk^%mTh32PJ5M9$)vH&S z^OVCC)|7w-AbO|p-iI(S;oCd6jnP|Z)2CEobb_y7=ykw*DU7aL9Xj(|Mm?#rCt_w3 zIjUw4S-nk)<$>LyeS!@C8Ibr^wQ7Dm4We9BV6CjrGF~QrCV@WzKN)MC4BGq=j(29) z0Xb&p;3F1pZnCt8V1;#$6mhh% zjchM|Cm|ON<{Kkqk!0em5GICFZPQ??LE5z15E{x`J)$hLQ@AG&SG!ea-lP9P$)o$4 zJ+?a!vRG1$yt*xWRa7I&2&>Oj=L#LXV3E&6EXB{~++5U!ED)L))VOd`%5L5cSlRxM zWqO9t+x}rFvACMmP0^Pr>@6(pff^GMM?>U=Tq1;C6V1i2Lr;Z48*-3q$#xwW2CA-) z=T%lF5|Ev{>^LFA%SYrL=OGN?cT0)?7N2gh5yUTgrXEpfYX#y=9N0eOm)8fv6=jAH zvk3N}EZq6I+MYs#it62;;Wm#RserAv2l)X1J?$>U%Z~>~11BMCME!&YN1Ep}gZwK8 zt3^1=I1ydu&f358<|ab2f|5+MD-A)^W9`bva3EB3glc68jLs?O$KB@tcW9M#IK5ax zNRY6=XxzV9Jo4@xFG7)Y%kCHXT)sJC@z z@~2o%)u$G3=w5=n6cm3914KbQYkdzb{x?8CTAk#NWUp@-+&8KG$2?NH0Clats~Ke`T0}p9YT3$AnS_lO5hm(xNPOgLmXiaC%&s3dHloRMHD{j_$XLoJdp!EJVfR#cwx}~FlMVsY4h@*dF z?Dp?`I1y|{*%AboWcPk$D&)Cn8!Sd`A0byZzqq(tSD0zn=mZl&y5|wh^7-JkBe-p12>-Y7LSRbr{I!X@F428ifi^aWWh0z${?{4|T`LLay50zo|w&U6f5 zIyOg#d!nME7JT`q8lq3&Fl0jQZOCkLu$n>p6Jlalt{~2D4!yP0yP~45`IHvA7J4DI z6Xp_`%&ZwA9FQ5vgzGD6*afJQI^<*k>{SJ4cNy>;BTLg~C~pb`ae?3bh4i=a9Qp6- zS?GsHn}K=kmY9wJ$jSk@KK^^}FX;`RM_clI{m-r!p3Hp2ACNu>iNMUp=_ay1K(e(` zggNzwKk!`tzn^?<7IEcjNmUB?mo9TD)&SKmRAc@jb4jQZX&y>HAsLZ*{Ca_$jtXeL z+zGKvV9UUU!wMIYFF(K&vFdEq)xtk?|L=Y;s-H-Un?QHv8la+T06dEVJY>WL$^Arf*6RV6gj!~_t~ z`s;IQ(C&a~Z_}GUKEbq+2N#`Ax`MnP(sF)EBQV(rX{sjrKEvrV$*`C>u60h_2@(Vk zM z6d4VfS{jNtb~f@fR!WURRE)B4h-I~ zm}WR?AoqmvL;9jX$Vsim^I7)h@Pj-AdVE++4_8Dh>_0cT&PTJP%LMyNAW|l^KJI^(68d*3iywxUHoU{l``_NFQAzDP;ee4@{^#qh zlzK6r@HYU_>9su*Jz=rn+rFt2;Z!h-P`_$?!uucyFfwttJgp~JC})r}{gd_fIn^6C z2!8zdF|2~LIC(%N<7AsuOH?KB5mS+%sMO}{QPKly>7c|<$??HT^GJQ@8z|4d*-y86$zI#2&Q2l(*+H~jyb^Z$^X zSHH}NqB#;W5vKq2<+JpUhYA=T5^OVXkKo%8v%aU}F48$08ICeBT&D8tCuhR69(VWz z{bmjM-E^8uIY+R0=<1x&;x}84+s^vR;!adUenYlBuX_rg_A4>skx(%W!al zeS$K8io6j97!#7AB?YPBB(f|@&NKR_BALRMC6_nGfjimk;}xJCA0Kb({$`D8UPIfU zdC=E|@t4ko)^FpEQux@*{~9z@BsT2R6wKjnmWb^O}rm<9cmFk zSD-~$LtZR$>y3DG-JKvzr|4Yqswbcy2>s%QV;e{pIPW7TM=mpCVKw=ceiC==`{*{p z2SQ+zM|Z1auU)S}0T{kA^K47*l;f5HZ<8LJohK06TuI>Iywp5&YffhGTXx%2$(w)@ z=Rik~z&BjRo_=grA}Jedit{r9_`X8`cDGdQNd?aWQ++-sjQk5*D7 z?l~t`q;jX{hIn3d3QPLu7}sESpLXvwI+;x$h+H!kxsv**&6x<9I$}10lmhJLTfIKj zqbY_O+w!yKGNM%*4#FEU_!mmHt=Lo6T3(?F z=?cj4AaSrQ;(-vG$fPqH{;~LbGW}0W8^L_`NAJd(=rwHAEt;WsU8+R7_q4=+|a_e+Fy667<6rK&jIj_sPr%>7}85ylu>6kk;t zk&3iCDpsG8&B0wv$(C}%^T18v(~s*XI3%nN?!@dJ0o?W(n~(P#C5GEb=9}C(qk^g< zJrrLRU~F{khP9Rt9}r$Z_XHlF@?ZHe5evumhDYP)i#Q(hu#|$U?#?q`JJ>C{dyEs+ zIoX>f$afX*lQppb7FyYE|mhu4Ge@v8vM;NSW@+t=@k-+BdoOw`HelKvY#7!*Ll zYRBhtZjo4aUM7IR2)CaG)oy|zLRTpgQ=UVG>bN0s&|aO~xfF6ykdu$w_>F9?Z-)A) zK5O6T7Ripk4lDFzNQtYS`l8dfv9^Nt2Ohw_HXALIt z{AUtMav;&4Y}Z;?&polc-a?RJ*X{(eqDkwT08dR^$QZp7*tmA)d~6?;c_F*2c!zXb zA?e+Bswlb+Z*PXilhT(P-EXczY-fUD0q{$!eE>f|)2g^7*OBB>8RON+%BardE!E8F zcSB|o6>Xj5Cd!Kfd;9z)lQhCJ^B$tRNCt9M-1*SDxIXz~`bQ+umZ@VVw zr|B`rID>b*<556hXxKjid|=NLDiZ-XuVB>(XWkNV4+zNw=>_j*)E(VQ z%t0#_o8>4#*}mAM$B*%@Ww=|M}R?=61ky~ZHo2!5AzBdH)3JX2zTnwBp!-1APn za%?MH@F!tkW}gC6fgTK9Ka3>I%d~DiK9^F~N^sTOnL6QxgRrVB%SWm$+(vmC-QLuC zR5K;6Y4+g&_oU_=X-)tsYayRIJ-Gv_*|s&kHA8Ee|Bn+<%C(9fOZ7Vu&Lpb=Jlz|m zq?WX+ZB(D_n<%tSi}|P~bxrsWD_KP!-_Y}m?l+ZH_0DWr((BAJ8oM6cEQoQyD4_*d zL;k?YKul_fBZm*rtAI;-(5~d_bE?l3u&DU2^z(R=FLyhx?6hcMHdsRr!Y*77HiQT< z1@)sgJR^ri7dMNP^{{&ATJ}*GFdB~2TjA;%@q7{W7R>Ule5;BqYvNA%;yyl^7xo;{ z(xhHv?d>YS#z6VatD>S80BfCFUv;#iTMcCm!E*n($>41k@g~p_PmkOA5CUTes25gN zkV=&8ZTGYLCf}5;Y~~v&SnZU5lExl{kg2-WB#1dc17RL@jya!r5s?&6bW~pIeq|o(S1W-0g z#z?lh2#bfv#P&1ZdID+ZN>=>q8o~vM5rS5k@Km~a4E)Ky4GiqU<7MZCL8aq~Abg^* zE7->i!;_P6)F0__34{9)`J6)M1!b2$79!6UShdBrdEdT~qI>dPk zfbj0Bc(L1HZ||p-@!H`AKk3mbM{7v^LJ(O2!^&?Hau{0LZJc8qU&#=s`!9*UFL@KT z1&|$*+|H=q?}8Ws1+`HK8v#(zL%ZeUU+j<`orv8EL#wcI_HB?o}AOl7iQneny zZ16L%!#8rz?ERopxW*N^`7ty6)X{l~%ElJhwE5lXDC@)BHON-~4h0ddG3=9@K3i~p zkf=4uI7cK7B~c?EY{?#%p{LPAkkmu-^S&0KlPNXPs)R1-AcMJ3eE|V*R5JKUWv_p> z{&)#BWy6gD$7eSF37sP(b7FN8Uo{rqkY*(GzMTKfTd-9B`rO`HQryosYfx2p9I(A2 zBh!8$wYlqI_5P)?0vVn%vGPT#dslB=gm@!6JQ>Xx!;&gZgRk$PfjoRy!No1dd9pQGr_ukJ*hXR;H)VlUXt{g4)! z-;nWwas!c{r?Ptw8s0LjEcTDu~*;c(=R#a*#148_mo3nFru1Q{ZK05S*g0mbC7Lx`*jnZYM zF^_Nb;OprpZrwME)wbbTcj{m>st(_?uyuDAGjEOnygmr{Vp6i{?}tt+W0EGGQv^^C z)A}ixNYdNwHu=ZUAWi&ZOM`)L0;e_4u$8B-S}(QgPo|}$D8tEua)Pw$U-)#pYnvms zC2Ify0-h@vd4X{iyAT&bdD@rrhtj7+SC2RD+VSIc& zO6wQinS8wB=ZDsAY{&drJxT@!!?l#!f`{Dj6BA%(#3b*3I38L8BHdnlEE}XANMQ)TeqrI^)0vY5beTKLqg7D|%~& zN!zm7m-wt7f6^kbsQ}Tm2X>_ZKRN7_meg-EAXx}KvO7$nlq}3@*Oo-59$yb&N&2cY zayI5h2w6S5nP15y79h_s*fFTp0i_gSXrdIwJPRt5Ny_cnmrzQ|+ygAl)RuXGjX~my zvX-L8^_w_`K}+;k|L~rJ;{qaxax8piZjNg`>P0wc{D7ClzJO`5Bp@@qyu8*lSNys&q+4Rxm5{B}l-=sT%50L!ZuD$7 zE76j?`cYw@_6+Xe4n5j)18xLxK}d0Cu+*h|yX{dC8WHd->VTap;Y8#etnIV?;Sh1LF@WZdC-{qOYW?cI~=96;H z=We-#*E$E(Nm|W!gM;DS$uch;IGbo$A0OBsucdrZOZ1*Xb+`wQGt;!!HC^3vp!Bg` z(OM=ZM?q(-+g9MJGRgVwjuVwSGuil*SqB9W9^H_y{jO~xjt#)=F!##pnSTTF85v0d zR*KrYbyEvjI-(H}DT4fjt2g&EiRA1J2K1<-O-86bvkXMq|6mlC@_`SoK3s)JiQL?H_87s5Jvc7im_We^w;DRH=@;U#Q?0c- zY{BWP^j!}ywP+844sW?L0atl%>BFrhCCS3eWa&6iQB_6QGRVqeq3>Ew6rb=6Mkoo$ z0j9{NUn8D!NKhoXrJi%~jKQ_tFYiPXcdm$u3By&b&g=`n5fC6BN^!xgCGrB3#O0T? z5skRIaJSPW@)G;lE?@$Pd@m*L%+Ak?c&r$Kzi>8AKLYARz95<$tWch2i_Gu=&*w>L zo;}HeCX)oa{ey$ydLQvuw=WHZ@6!cYCR+a1xcx|lhw^L;!{-Ziqk*ZO{=ctYL@)Hd znKaf!foKepshT)#F5IEGAoUfLgcnc+gd(4QX?33wv^ZpUs&zle4ZYRUp&8h%KLMEg z7pQ203ZPqRW+*BDLkT4F@9AY4f#7ha^mLmY~iwLDOaP`?!A=DG#$n6Xan>KH8o&u^6KhjGEL=dn7~gO z_lbe4+v0P$;e|2zY@X?;zO@)trg94o?dF-4hSZr&*_4zNfY^hJiv{&QQfEl`?TbQ< zqy*$g=ogwBnc&SCBz};;mM$abXVZz9hWJ$Pw#qK~kuFgN7UfLd9Fk6E}q&3K2ECn}3 zpPQEne<1tkuCKuMJB!EL%LdE=htB?61*T2Hf@`NrN6HJcxG!JH>8A>JTNC8;b@3;0 zUjEtD(SabQz`7)YrqeXRp3VV^w85IQ1MbAx9w7(|*_^`9`DrITi{LiGl(_C^!qz}t z?|e#dE_SC+Rmd4hANm=NDv9uoUX^{e66JdJx^SQ)wH#)828shr$Cqikm@+$o=!gsWrsY@ysEADk-iWB{Q@+3Bm{^J`5!*@~tM zYeXF}U)dvv%6j<^3ykM>%u1fAarm-hJO}&02R3;B@C+CR=PphYM4XT;A)$c14tKiezg9YX$V;rcDoSp;%c1_R>M@wzHh zF~S)4`yapWz#NdQtoHWyvq}2tAU_e1ikk$%K=XQ~NQma(v8q`)1H--LyKO5_W;5nB zkECry5{cI-@1`yelBlWjTsg%?^y7$lm;N?wNr^Nl4G*$9hZkFFw2qKNS<60vBqn+O z9`cS|4Ngq^u#k4|dN$E6>hhR!9he_T{k?x#f`iZ8>;u&P`yXv&wRj(UKUmERBlde^0Us1GLoNq z71x%R?I2}rX84OJ)yif)w&6g+?b6%>yJHf&3Vy!1HAfX=6${b*$7t>#)Y|?>AT2z1 zOe_E>w~+ANnA-1&xI~sx# zNd$|$oP#7sUzvGJ_3z423GdCf->d2<_$&FBVvrd{KRcvd<;hzEcNePod|Gd&wfuxU zHY7Ru@Rwd=*w$2URiOMp#vls^8+nMPSLEZfruzD=P>w14`1ALCI0P3ER+nXcwb0pP z(%lG41C)^A4jbGToifObUTHXR$fLuS;8*Z>jKpEy$7=~v#XLZhLo9F&%hcB1{u*9Q z0Pi>G{r2suiOIxESDmR343NZYiP*PIEu(=|xh9sd1|DbQ#O~|al)K_gEBY6}zJ{L@ zSl^tQQI+&q8LQnXLCHAz;tyrBdkd*wZx50V%-8sdAA|$mab8{?H_tYG!=LTZvB!hP zWfAyTZeR2tN9yE66$Q5PkX6#JOx$b0oE4hXk5qJNVRt%3dYmTxcGe(=1H?;-Sks?j zmF<4cb&j~zw{vOFzKkSycB%mrw&4&o$QQ>QI}QjXc)appuxD1MzO3!W_D~7#`wJO| zH#qZchqX$57RPIzP6Jm2ipad&3*TlyX6stG>ES<0%+5xQvbg+J{7XT|QCznO8JEX6 zMcgqlVTP3cmkn~oe5Z_@*JqUd_c~0=VOR8e z@5!2f&YgLYhP27WZ;h?Ay}Xe7Y;vl~wV3%8!@!<1U-q9r1en0a(GdcA6b}VEXT_k2 z&kL~d<<>o!`h)o$7XeHHt_aj^vDL zH127fnOJi1_~*M8^Et2GEtab#al4?(A?qV$GlLBAKfzInvgA7c9@1iV;nw|o%B(K! z8=+T=dBj9S;(iq9i8zc^f90Nw1%J~Ol9^k|4ISvF?vy$k-<%gi4gsn-&pXs_>E@Y_ zWD$Mj?+9CBqs!4TYuecKg`I>q3%stcm$i}7(`tii*&&dwTKC^Sp`3zoIcZd>df~uN zV4sWf%%0S$Gk2M#rKR0aDQk|uE+vx$Ix67M9NwL9A7Bzn+1o%PkGER4p{x53bJ;0zTH7wr%aljF-lHu*GXhTCe-@47NEi-mD4qYh~mKl4%*=6u?#&j7--~@5J~0=D5u5h2y)RC6qa|W{mLQUf=bZGZp9v>v1rQI`0J5y7`PrszV*0qm9jjz4ck1Ju=eI;*Iiq zZJI+B)=_b*^MUz^mKRj*bldr^&ty53z7c0<4+aJEeQ0PwL054eUfc0AQhEimY@kGB zv%(?MCuyJfPKi15h(`ZmSP`J6B_&bm>FJ=0a1j*(+1@jUUP|PwS(W@9y*36Vy=EmU zk0IHqHqJJ;Dv+Y1oGBw$Tcw6wn9Ux4>Ws}a(f(rq_>zS!!`#=nULgWa#Nr!`Pek~^ zd3HX8a%;i5PU%@IJqCr|&911eO#@@dEybxR;j_(;yw(IFkNCJ%?W>o^-PLY3?@kh( zQZLJw*gp7^>AS9sR3mUxzV5*b=kM6BA+zD3q>A}~?vi+zoGNC|urV*qa60)t1X9hE z{;T+TF{u#aH}(s6fjYLP+?$AWq>EHQEifSg1=|g){P*8u!=> z;g_Zg=g*&q@*sHk$SdPI9t_;w5QSHtX#6H|8*qJl%L}h{J<|p^!miV?BXxSL^|RKhKt^@vD_5=*8deE%w)i(rLQO8B z{L-1U{1cpTmtK{Ov3r|wuv7{^!d|-SXUm1<~X0-x^szUUusq0rSQNG5gsno%q;%?Z3$4R z&Du-(3p2B`DCeKok>;>v67KwSpYGzmB&z16A+4uBO?csh#=BjzCkn@eofYA2ot@cm z2O*8}V|+qY1hXuHuS5CTd2QShs^s&ePa6IYPj4O%_4>b$k1a)($)1prWhh(8zQouj z>rlv&L?YR;lwHDDOGXGOSwh*8ZHy?gR48QMDSNV~vg~G z`?*}t>$+~P!w1a%dA=o%Xxui=c?EbH$Vu8@>=8uZH@SH61GKG*o)cm7e}uWxxELb; zo#H?!I7@G+5uj8@`E-WT+@&Uz`W{`A3wtSL%>kLGtK5I zO~iaoa39X^BWC*uViv=Gvi_SO7D2s*@E+IJy#0@?KY&vPY~4FRFuc9|rr%&PPU~5B zM3eohA{T-8ND`Z?`T^PHDG-=h*558y*1yvW{s9P43|!abmv6UIKZ?|Rwtw(IMB-2z zMFUa@-_BkL)PRF;Zlul&TKdBWbdu)oO|r!^QK9E)R#4v^YD$9N0<;1Uza^eKR78iB+EQ8tYz4!<(z(72TwHcRm9 zIE5h?-*!#C$$RBEt9@V51()B)Mt^QYJlDH+O$3l8*+_Zn->?eqTm&=z%W64A*lhc9M)u`(IvJ6PRjq#`&+vk^Wp5l>tPofb4ITQ z&Ctn=>VJx$4o5)=`vVH&`rrA>8&#-D$vzB;Yx!MJ;|OfYQG4m@fpuC~4t0#I z5<5dMOU)rM;w*(rCwCJzwT%_uMuQtn1u(A2Dm11 z)mR#BMH|HRem%=xx(&YTh`w(xPj$aH=) z=^_AG2&4{D^Xm-D%#oT>@YAH>H--N+aN@9M{83a6L;F+fzY$>o>=;<@WcVxXL`uQT z%uK!Y2A_R_hn%Oa82eFlEtg_e*crA?FDL*mg=O0hNCBeOe3y>#K(^IG#99t~(tae! zkCibpvaTroahK*kBl*;POcZnV5ZAI>>uX0Rr*hzBRxZs2t3yo>_!_8arJOqlj3he* zgF3zkw(o`qVlfZT)jaR_`+WfldBphX3687>-34I6M($6F!*}haZx=JVFg*rXS*h^l z%NfW#^}7qjepgyF3@iQ~mPq-8JHg5jsmA=5^?J-KGW&q)9+X(;Z`>&6ywlU$qCE~B zcI%}LiuBN(^Ny9Te_@Q73}zZOo$s}_>GT2p5O~oe`r`Tsjd-g0P{=lUJYCZnK3;I) z*&G|3cNjUm+|R3}rNwQiGUk)P@aa`-DZs&SM4=dTX2{}6cVCT%WJMdN^5gd*`=xuA z>Tk>ZS8|FULUb4KQiJUqZ}Xm+%!OqABL(1^Awas*y3;5*1n(m_+3RaeAWQt>_1R@X z^GkJJp3`yd@mzuBr5CCOxV3}(4`zp3E`Uc+(C@X|GbtOe*(nqXJ3BYaI}7U2Wlg+z z6GmQD{lthq_l4uEPa%qTbfj54$$RhSSTJ$O$%q9Hy$I0F=mVY{#_-| z^n!JBI3Bn(wOBa zUUz#a|Jb=&eRk~fUO<3EJQUyb+kJ%7u4(eA7|Ok&4#jRD{|pGFrCa=A^-E6`CW_;38ngUhVFBkjFzaV zO*7;S*FJG^@pyrZ^u|{h*^bCc(s@im6btqEyH&TnJ;38@!E?=u?Dy*IMJx)B$6J;= z&_N?Jt#L7l#Aes^TLP_L#Ntx$T0@=g%m5v1@$UZq)UARo$w#TFKPtm~-w;3f2@0oJ zj?;<^(m}F?=2`IpJwLV-WR{>eRybsP0<#EZJ}=HuVb#iPWU3_&EHAwUpU(i?*OIGW zr5?Z)bOE=KN+R&XIhx*B-|sq-6-SuY!>wu0@HUoub%%SGLtgdX+w6bsJ=;-fXPA>C zAtmJu)dR>bKD3{k+sH+LNx6tT`?=UgBK)<^Gv6Y5i~v7} ziSMniDOLe>?Y^$#J#;l15W8by){ z-)(c;KONjdhu-}10SSzd%-@G=5NWJl>lk}o)^u_b{`lzBts}cEn4H(04@pED8k?K3 z)-|42hJH?#_4M}Md;jZTXgWq+D*L@>PQ&ZhJuZT$Um>3=q!WGs#PX zD(`PdfpaHS8>U`)*8=l3Ep6$m?##sW<)Idv+YQrCAEbZt_V$LxnvyK1+#jp=npd%Q z;s&>$3JA;>Pv1Y^_UV%&44!9Zd3gQR#W7V9Pe}eTZ3^#_+1wDV4pgt?2d3Vz_j2Gw zX*Aw;2}mI}`r}?CB2)N{VKRvVKdJh-4ekp!r;xb%&6N ztM}p>ZpO4rhGdR71s{6PT?*mQr64B(WO={?cW1akdy?y`sTXH0>wciw2n36{IpUpB zjiEyaB8=RbTTueWL1$xxFiefnOAb&pq^pNEoaYg)hR_28@2q1=yY<95GnYJ->ySB( zjHdq;^45x_EMP?E_S093p12nLT>rBNgOI~MXG-z6tgNiyxTgY^<$ioz-7H)bH?zCr z{OF=Pt()`UXZSbpB~riYBNKi2$_KK$s{ZjDr@IKNg14`-H2@9a>udAh60S2aG9m{q z`Dg0B1V$z+)>O+F=N)c+$bUmuE@0z2goK;BYI=ddRyb)}*USE$zF|dENxTzx;ymu( zqgCeH!!8v_l>p5IsHElRnlK!%pJYx-KsB8oJ6q&`*fI8AOt>7=)>;U6laC-5^ zKJPWLuxO3vsWL04;*4v<$gk<&YvbO0TM>uFV!u#70{x{N46z=+%cl;Vf#qoU7w#!@ zyeKCJ187E;&r9*AZ{GCjulKs4t(JAv#MN-QoCm1{V3Y-U5L)_3fb5g9vKZd;_nFj6 ztJ(waq}-^YT|{Meh9CWFra{|`W3t#!Mp04mO6o&=R>o4GRbPfP|D?KX4$@)a)fe&s*_GyJpMi&jf;$oG{{owdu2oD zgo#7@r%Kz+q;m}M1T*mqwtj<7`44VrH{Wh8CGAGEy{LRk(_p;Qm?^{+>&0BsPCEMF zT)bZi?FxI+TMqQ}*Ipm4`JCTZ$!UNQErusjb-)|I$;~h0oH)w$HKtfkcPD$e1rp1b z?upwB&O;wNI=sIsm=T|4T)Ly>Xk6+=T7~Kh>~YP0PH+DKr_mCu+sPrOpDq|!mA5E& z`eWlCFN=L(ny`OLNQB)T5D;)ccBqziC!F|K9UbFhZi=y8D>zZXJ5}uG0a$SHT+=l4jR`r=9(*&^IrH$vNFkl4VVry{DU}c$5dqZwMMwyNjj$lBeWO z06tTL_`ymG<|pv7g+u5|4!FJ>0mavV8yBKKR~gZ8X5c>Lh=X@`X-hJQujF)Oyv|u-w zO#px+*fYcI-(n}5QVFit!AiUR%%Qx@)kAmyF>pG0!E`$^lV~#?NIW*dn|)yF*j34V zuJB$*d;1j-^PdE3o=3h4R`&{$MVyZ?_1V6=710ClnEL8tww{q__@3&7DU1XRrwQmJCHrT# zp8!XE!;030d5(-KBliKDd4dv+2xhbK067f?Cap?-wiIJfYPUyR2`;T=+tZOm1qJm}5DU8hLL+1F}=g*(N-&no#DAB=Ufr5$F z_Mh@JzXa&SmmF>Af0Z6FVIXbAu*#Ncu6 zV&ZgRP0c{NQtVCl8CgAx=X{fgeysn_{HqGNPUaM=d5kc|;;MokBoo#}ZWE^m05v$J z+CqIQh1oUnBNE$u-bpxeD#vX94uWMxJn37%o+~8XU|?Vv0RM0(>*MmurC4@CF>Ain z*;bSv?)tc)@E5$(G~r>%T}wNb0zg_cr*Huz_9~CR=9r>w*K|H{E|1{#HF8hPIRHqIT{8J(4!Q42;=b5 z@-PSb*VUad-RBYP@UeQcyX(26pGIaw%s|5T3fpV<$&3RjZe#LXW@!OyKAMBjDR9wEE5G9{Z z(Si!d=mXnz|5AFQ9(L8=PX-Fo0GU(`u4LE@5THDw@E$+zhIU~V?p zN13%fIx+bBrTLJTw4|;sMvRWwCK?thsM64R9e5u$w?50oA1l0py0C_r(>}gY60_;K zd|=`n+G}G%rsX7cj+!Tob*@3HE193HR`L3E*3zT;4+%C0t_w!}?unAzkNIqu@O9*MF4$(%y4`n2 ze25_}LalAgtQJwFZ6p}@`O@))FYkf8M=+gyaY&a*1NpRW_<+1LLc+Tcs#5lqIw@2v-9>oCck7v#Sg{CXaCsp-~rV;YmU`x zdgqy$grcE?vvfIVjm<0@0Fj8H`Z>dvaC1G_y6kY+m2+M|O-tfzDZ8^00 z2kNU|`}0hC-A*{0lzG4V?()nWBGg!C6$b^K$!zrQb3UwlSXhb#l`0Q zsqSflQn{tWb`F28z6!m(b=5x9=+6N8a#PZi)57_Dd4WHV)Hy#=y22;wSD6Gc?A*7 z@5Z9*rGJ9d_Qv$XAgqht;7~h-sp61K@eFm}cDQye-{|kHa{zRH{`}$6%B=g$!xdzd zQaUx!?P6ra%q#Ei2-sYJy`C@y7@x?W-NXJhj)RSV{%;Iz{^k>h(wi2^oI*!ho7{Fk zV;snpM#8aqpJl_D1;HYtr z;fk4CMG*)%F_8x@+f@yzaM9ZLkGf#PY{=`tMrB)m9`qY+#R8q4&Ob;&mB ze{kkSD!R_57`9xi=!Ot&sH}P6n1X`m(M>h4gP*B#B&Yj+W;KIux5$OXTW)An*%9Y> zG)ijwH8oJeW1x2{WT9sh3m0i!=7ThOZTsmrreIP(uvQk59~f z!=Qo;aSa(S-dl5xqff|`)ItdkwF6?Mq&Kwf`yqI&|*m+!R7&)xZXey%V zI4t=Id3pKr7_1x)xT`RSzk2d&mimDaEJpZ{!HpN%%L{Zq7NzH*9WxNLJGnJ9W_-iK zV+x8|r8{H4c(PniJGzkrbQ6yivhCZz#&mg18kEG$y3ZyX;ZD%ka11P*2myw=4Z!^Q z`U!a9R{)@t)J(-%eo*odR`C*Bxk}nYVPun550#%ksv1xUAGhttI#>rEHXE45Y&>7m@6BZM^Dg)_E+s+gnj+2#V+sP zac`}i^Y)Lh-vH+VFbBbk@n%>LID~Gabtx6Ht<_YFVbR*T*l-snVG8}Pu#piPFmYPe zyo77s`tElSw;Tlx{w^iiKGoJQy--(^}cTnzM-P zH2yRa2IT;cLb-_CKT*JDNy1+OP7yRvpM6IC%Rh#-;vDfR>ou#%*8;tbt(VwJYikGf z{eL>!;&rK#lEpu&#w_R!4R zToKIkFnZM0R-_hyNrjPLl0=Ful8q~z&Yxs^SR78T?HF9|U1PHJMl~e2h1d+>1AroF z?@CzF)5nf;$P2Vy>oPSmN=&@YgJD)nuJ_56T_CJXO*!E4ggXJ5)c_2q^BZw;Fl&UJq}&wShQUoeYxtHS6v52C!dn54-9-KD7hHpl)`5HWSZ=|^W1t{K&WmQ zSVHMN_^=n}cnsK%7tP!t8ZGvhI>XE>#@GORzrC#ud0C{ep7(imY~l~hgwZ@33MGs! zV=vT$>FDT4sf)xwd6>ndF{txE)|J3q#j3e-A-C(O_9@|DByAwC{-h>{H3{#=d>DdG zU0MR52EDxl-y&QwWL*F>rzb9%OI^IsaU?0daWBkE#2j#W#MfnhJBxt*4ErCj7=U{W zb#!_k|7iK!csaH$nl1CO^@m}~ixxZo3litrf z!GTOx0U69n;N1)C%>LVUJUo;6M?O{+`-4GMU9tZ`r)$?9t7WK|F3}xbw(E^J1NYI4 z@09Ua=aY^qLHU%=x~FjgtB<=zqfQfl+TuJOj~J@Fc(FRPtD6_~0f^`S8Di1V0q=te z9aC3&CT3tPV(v7*|L?t!o({?3P^nZy;?~`+A3ftzk_#&(*<(ZDtu!TME3;9O$KnC^jM(yW6WCf%xIo{d2-$ljj(3FFH~P z;+n`jF-A{$Rj9EhIRK<02n;jjaMR9Y!2xiL|B#U)anVVt33|*_O2l&-j?I8?Kercu zG_y*NZ1?FrmaaeUYMPsy+iV_W?qzOi37#?;*iT2$!#q{4-ERJy7cX8!K+S4z)6|Kl z?$27rsE#E4YRuATz*fPC5lm$8*|)?u)evsiXr0`ncBOH2(mc)H`d^J?tl%vF+wq@{ z{y#a2i0w4IzPI(xj6z< z9B{LUbPi$_Oj6)F2Z8J^XNFRmf66@DS7P zg}yryIu1_wp=JE~-ewED*P^97y_NlbWvm$I7|%GLn{l7oRKl~axL^4g`*MAmiJ0rJ zXJuvG?GYjz&TwE3^IlVCGLu-v-!XvzIe*#=b{y~y4oyZ&Ts0_17=SB#LGzv+Ch^-zm+ZtF8Uv&7xgRVqsOFX*Lsc!jmWbW`Z{k zd!{H4=E79>-m_wVV5>?xJzoOcs>k^yY>$A_Hd1sp>T~`8z~$dcFB@OWeVS#P(K>%wTlZErP!8N5+u;r=I%aapv*64t{$YH zJ{NzL#g=uGHZi1%neTm!mQ>KVSl(FckM;BrKg#L@^TKnrbCdWuY!&n|U;{PZB$Ial zp%}o`Wy}j!v8m)b*ex zt%-Xh7$23zT)b;6L)Hsz)5RZfrTErvH=~@O9I!P|3Q#8@7)M>MEzFY61 zL+SE%;@DeeK6-NCV)x2e4La$p{Tti(m4!{t_dk-=roA+9L5=?5!ubOLs&Go#W$xsX z@K*o@1IS+C&>o5C=DX{AGmbJ8rCtd`euL*?vf*7dBqKo%)7{wn&OTeh!x&6r1G(tx>th1@@&CHcKng&r!qWF9Mr*z6J-X@OQ zn&tgx>4Vi2d_cA3a`UEKq>I1D-yr^sk}=?=EH7+wzjuDgrFJ>FX&jZBfjCqF!70y( ziy+Pnpt7E=ZB}x3C@cDEtLk;<(gMY$70M-CJY!EvQhBjIlzEc*&oiZMx?RdGYzaq@ zZ6Mhs{h_w^l?n+T<}oobiHGhBW(MD)CGahv^MjdcSG!uD%|y1tXu2E@7V^z&|LgF} zM^7S47%pLp7j=r!lPLQpsY4E+w62m*NqLtMv;{FB6L5{?1wAZGY++w>RS?8Ce+Q`{ z13fFNbnF?%oz}K9($*oz7m`nBk<~Ut3?Dnz{r;p$4y13#cg63+c?cGqXJK_dV-7VjRX8Z?;LT=qjiUt*YI#)ND+}JH2n+vO`~U59>duniMX;9S-|wH2TRD%r@$t)8?nV*q^d=%k>)8UXv6}z+?Td16||@I{naJ3RY5GJsS+%lVG5%c@dvi3N0aADre6iLH)0q zab8T0Jx7g~7Q|&F&FSCOV+h{bHVKf@gwiSPq7+rz4r(duqIA!M#e$R0|&JrOLBmJxH7Bvu!7Y z>2se+c2JA|{NydPwWEEDrp&Pv`jIv*Y@Yel=Js|5zbuF@g_+Rc=*2RScLMsU}$iW2|TZ)edpw#j#6aUl43NnZR25A_5g5b zxE*t8DOdoFr3xWliPV4+mG`Wh=l^~TQYIK&K7PkZY+JZ4HN+WsPHL{+0esznD>o47 z_#A!=ouqQumI8n3%*-5vvDDX2yF3zJK-Boj89t3(V0dfy;7c zL^a-eB5@}ylx|^Z3K$eH@&`!hjh&g+B4vYrABAVC2 zPTmnZzVPb=Eagp5n#PWP+?$(Mg~#NHLU%p|rJM?FnhO`$Ou|I;R9~M`r7pG_IzQQ4 z`OEjk8g0;DIHgitE72$Pk%pr#u$4L$cGXw5BoX^8TLWX``{xH=Sn?@fQx50f7@YVM ze8AG3YXYchKw|Lqt=hEDGpB|YRu-Hc%?2Fp`||9{Mh6*Jfi=f`@-&BkvZ%4&*U>-D zl6lF!7+XqqWgH0HyG!iuEBP+4gP8WEZ5#SyMSKcX9*NL?4l*|funQ*4@@v3e5#Y@U zO#oQV0}vn%x*chVu%&ppy4UjhaBrrK|MrLeA7%3krz9kLO6*ty5vG*jrKJScq&tmV zhF*+Zl8%rXB^`4~btItM=-lVO$v}4WFBN$IzI=Rd#XmyNu*K{+>jZe^osP5>#VB2E zTGGf`3Ljw2{rJs1O7#!>kKNmUD;8%m$Vw`Byx?kQN|K za2)SuuhPK9%m2U&C-sM|f3Z7b@QomOt9w+Ld0>g4q<!3erYKe@Z+pp62VllhAaK>%b@-JIeNcg%3}4~z2(N&1h0M& z(}sjJ*tSVmlKUSKXQrMO1LV-37jJ$Ng=Kbt=xuxeF68vjO@Q?Y}^=ZoVcT zZP$G#?)@mGKdwy=rp$zWnis?^z^p`&0jmrjM;sz{0?`Ed!PUlX=-jb+Bt}qfU*Gmt zU#&l1C?>}`eqd;c8X9EFH?ApuiDI*Yr>}AMua&SSO}fwIkk0cbY`wGgptLNOzFoN{ z+0l5;-$SVEOPh^P6%9t+*L$;O%uQL1zl=s%B4rLlJ5^4vYA53d@_Ds9f z#PV#uPuQIh!)$Wxvv7sn6^PkNM3gS@h%B;l9LE2*a|>YSnq#s!s>omS4bpR9Nxo0a zjHz2a$`iPR$EIM6G8LNO5~gKMKg&mX#sVp#F>RZToMdhwlOf}qT{GQ%1zpRzCV-6UKKtyL*r`*#6&CBaFgbl0=gEh(EJ{~SZI&Ru=kWOC2YZj4 zMt@vXvlGTFhO1Y_{Q`0ec+vvYkXk{&E5Z#-`dV5MwnUbiwW<9k%O$$e;YB%^Wf}Qx zj~&OPK%jMWr`5p)nnKE7hOzSxFgJX|sHj^Lr~?~1R@!HidSq6IAK!w9-u|sz%jazM zY%u@ek|p3HP|W2s)&m>Y&hCk=9`n%XAO55gyWzLV zFy2f1XV=F%1oGa!y*)oc_41SFb?<;FCxgtfHL6ov;w-W8etUwGoo`>jFcpr6k0?C$ zrw@sI=b_TK)hi%!b2*ErobaPDwVp-ps^u~>H3B}h;vlwolp^56#_F5>3;35=OSp?# zj^HW`g2zviGqkJwHu%>a#`Eu(&&5y-*N3Ub|LDdPs163PW!L!CRw{rBiWyvq9jt076d} z`CF#Uu--CP-qD(+`ECDea{f{>miq?OibzHyKM466bnh0B0+!R(G>&U{ct`;i;}%vT zuU7+TcIld96t3DGuD!i4Ie-5A`Tf}=!auc$^MZs8cLVLU~|7i-i26FJT||>{QMt zrJTd(-6b#E`FLI~iE0G?l7XOMV*nT7*BLYqqsA&)N`QM-CPqb_81y+Ab=H`$H9oo#yybBF>vvT9Ku}jq8^|9di_6eaw3%Y_O zyH0AK$t*mFI?Y)8))eT{eMztBQ&LkGSL~bt5C#A!)8LkgmLtfe{{Gv3Tg|A*64aUJ zQ@?N{>}HPz++Pj)LLj`5Vxs};9iS}HTzelFN7+WTKBr70>{sCX#%j5WKYFZg6*H8k|V*kB`$+ z9qP-dL`2Ns`D}P)|BBT&JNdO=7_=42?M+7l&RIE;ON!;yD(Av%V}5c5!jsVa zRk*YPQZD4zptJJ9JWP02Tn&aqFjG&JC6s%2_mSCZS|ZI!1bppW6TQHh$m)@dNPnm1bH&g9aV_VI*sUDsSgYXjrbPI=lYL8Zw6B;%D4c~j zJnR4VPZoAb9ijm?`rpwq+aNiBDq~~aL{%j6RK(TQ$19!5UtU4^*kl}T_QkH(v~z6M z-rwW2%K%Ued?ClA0YIVp;D;%6S(DGA`DG*bUOQQJ0mno)_)yuX;+LKtEh{UfM+Zpw z5XcGU7qm}#QJslxqYsH2Wy0}=gwAyf8fC=>Pzs_rRyL$ktBOlk zxl`e{$%OJa_W!s50cjB>W0ygK%_i?{ipS3K)7w(i-<({Ofq(44;l5K1FFC+;A1sVK zu|j#~nD}FgUsq=FXso9MSKg8KikD`MSzqkbLDB?leoq;pS#z)zg78r`&Xam!R8%l^0d6lExoJSVki1M6J*Og z*-Pd&iX)?cbX~rU1~QrOEw=#Ff<$OUf(-lHY`_pH8wee$APewb`<-1SzusXDQx$rl z9>AaSl_yAeZ((8K{0Cm{GtWrp-Eo4JUsN5jg`oW2svPa7II%AWdo5!)=*0!f0FJoBzyi)t0$`9-fi)lpq!7eg>f!1m zpll7WZ)yK;LnwvaA8wgzJCjFjHM2Rd}Io7;wYL2;+D8d!*1~-4`@=&{11!$PqEz?)Po*oBCl@W;XP(1$-~&nX zEZhTiHga#I;^fuEQbG*O6cxDBt9<4O)Zs4#@r5S{Fx_Rq`U|n-*=)RBwi|(W2CIX7 z>)IFGi0hHpbo>wFug}Q+n&g1wZxG?YGr@mJGlV6m4^p}C+Uq~}HL7e!;M$+|Pw~4# zPf2(Te3N`<#+EOMlEJ(RY|FMvB`xHNiVEnCLvXlwU8;uzeT{S1rCcpatCt5~BbTo9 zNdnh?;R3#p1a$swh4cm3+#M2XbOu$7bI>#t6?NqKkO(o?6wH~P7Q|e@m!DiTop^S= zG+RB?4u%sb~8*M0m>Kpz)Qgq0vQKXo&N9zfcn@$?_FxV*`EGmX>n_ z9dSF@pxgRC4OW#*BTp9CymMdjC7EFU@9%$1nu#OU#@-XW&}y8tLV%SM`b ztObzTVeGu5xi$)&7FY25l@`#Odqa@{K#UAn6&Y-=*bSGAy?dt#xEcV!j=OGm=-&S@ zD~f&T{FjlOhM0bIW&ebvgstXQ+FV~wlp0LDfp`gs8f9{u5X8%2eF6G?j9n7K5T_A7eaP>vXu_?mDtU4T&9g0qyw%h zPB$N$au{YwFpvQ(H`$=cW-B(Lujbts2UWS5<|$9nvw zrsX^FKRHV{&ph`&zomE(mlQ1f)EN%X8z}dL3}0^frZz;m#PKTjmBgjyqNmrrsE+7S zQ&k*iee@ubwL;YK^2*+xmj|oN55*rM+iKe*GwUIgH9}y+_IK+@&#pU9y4LAD$>A5{ z33$VM5$005H*3I>n*(?x%N_izNeuysfA6CX_i%uSv9mqq1djbmUCI(23f)d?ugKOj z{p+}p&~y4%reddT>)0a>xMZJWqFYtfTIQZK-i$4YDzw{n_3+Te<6~_H<1pR>ICUJ3 zJpjaRFuIHzJt=}3IEnfBcrFHb@^S)=Z|9KeAR4rMM;C}hg9@u9r;5}7KC}z{DzQXF zRdv)vp8dr~N%CRnvZ^J>tvIQuy@O%1EK49BbO6ZY!03ZuoJ~` z2Ag%(K#~~kO7=r7e)iGA;Y2_Y&IkC6pOYw6P>Y5#&S-?bagE<~b)uc5y01-oWK)_*yJb1%KViQMCwb<7<%2)OzE8!k6)Xh99_zl)&(X2MtJuM`0I z%JerSEz3~vqg;$@pGte}yq@X8c)$DK3Ii^Q#4?E+RIB$f1m~y1m{WPZcvSVgDGT0-`AuK-;Y`9=ymieITaT~A;Z}YyIwF>L`S}tqL(k5 zC*M~k53itMw5LJo4Mv>|{l&~9OMCv&Ntpp=bow=aDTmZ0x}evJ;x8rP2F|O{(1mYl zM=7L@Dj#n2XL`eSFXJ#A^b4xdx z=;GL!Rq_OV6_rb?V6{6BOPTy>9u;OGFv;3y?bj^}mv zDn6FBXiV?$D5GIrH2^d7c3ig}tWY$un=WjWdv^)0DsC$R@Vrvd1vDcPm`Can0kwaq zESVP#^zH2MM zB|N)G6tjWJ{u2JAF-~ZK2{$7aW>|NWe*G-g;0POA7ObmlPAcn_BKfwS7IA~QcQ`th7-n;MJ# z<38>DfKj+ACiuFm>(`Vs)VCW=lp%akBjF2pkOaoi;@|UE~@b9?r^_6FUs0=e5)o z2yc8^%e0X_<{p}wcb_GSu>vlwXAJQ%&>Yv9J{L_S-YqRGJcL{a1ZdObhAT=>mc$Mn>ifawv=pwvO1Y-n#yk0*j^qO8Nf_wdYd^@)>i& z+$2IO)z0JpGLVx0$}zAnxMophhua-tJ`&nC4?7>MKkavYKI%^6dgByoxV#z$7OSbk zoH|P2ersgkr>*Q&J@X5=S2))8ZDa%{z=z2d(b~=47ZuRxS~CD4L3E}jW7@CC+PlL zWVtKv?u|WKd1JU~=hec2#76Ks{a7-?8V09yO6M|4+S;PgT*`FZ)UL!<#D&UJ2_Sj2 z%U#2|Z2*k^Z_k!FwjUu^0O_hFW3C|jA5Wvf{}LDoU?6Gu2ydc7Zoc5dU>NQKu^o7# zG{7~(0UN0&wU4SsANTL*yKh8l;Ezyi_{agd_dV!OnS61$Nx0Gedi6kta}gINwL3o$ zc68c`Z#e`e@o&+Bi^@-yyWxjtpq#sJ&Tz!H-IX6$Ju zw5g@9kIz%+kS)Aza4O*4>26oA`A*vtj9Em>gy;A?dO|_v^}${lHVd$Y{DiA$=wkxd(2C3xivL@D@D4a3EIgQ^ty#BCk8*%` z)wk%*OrWG=)*i>$*~|%`48Hw$K>XRcxSj)H+2SEJib`s0w}-+q=L&@!D5ALLtdptA z%gc>mTTdIGuIaNyNmkc!b8yT$reY%X0|M%7i>jF$J(%7=bW?qhoqZpAdC3Pt8OpO% zmcke-LDz&A1r3Gd+btXx_Q3JU4-m|eRu0~+v~6dmxyf4lxr?gO+7|FAq+0lIk6+uC zw7`#E&q-Y|!6(qI`r3Yu<&co}oY2tXGd&K|H&2lGk}LEEJXytu0+RqynMP(5eT?Wx zD+K&+57dz74BX?7mP+^{+_~ag4YQKryg=P|dGmImMce!Ht`Oi$0g?xRYP@9W9J)M9 z!3cK47dQkP8$K#q16ObVYT+`qwB(+%c0^g8!W2Y|^O<&G^Dfd@+WQ8B!wLesHzg9q zM%*|9GpJUn4sI%9ypJ@#u6qm=ISr#=!3dzt6M>@JbgpqU?xs3QdSC%#Mp^lOye~r# zGi<=|{J>F(vu8OGvX*VSbq~d# zI?c&9>2<&;gg++GCf7Z@gnE0rml=&#^Z_>nFeyEVcmwkX%n8>2Ss}kDVz_B4N#So*Lh1}{RX8(FC0e=a!uSQ>*{vI^$YX+p7y{GCE?*3#+y_~*?@6s>w zha~-%rYjHhz)9)pE*4vX=t7wR)!%#}T+qM*rxXg{zxCf8;MfFeJCbAU zMs%M@7E1;{h3hCk9iwFxXH~5!%#n&5yqgtr>pq-qG1vC2^;8-J^~I;@i*o z!ar1hzZ8i`NH8LIcdAHmE&QE;Th#4r2e(rQ_lbHMT05F^0#u)>>e|dp$;+eUe)k8r z<-*jtU89L&_Q(CwxCWW*K^JPEnd0P2Zf8&NQBQ=^kFD7Cjz~roe>I{ZP)?4bec8o#U z0wm&@NHR~)--09?stF~O1x*_D_>Ur5l<8=Ybt$3fNJHd!5Gs^@wT{TXS#yQj^NP&B z8*q4WyUh1nBOuEkS63GRg0~nJ8{WUTy!A)!%o%9PsGxnyuK5NL9OCm87z=?S;Xu@3 zujw;c4f8Pjy6*n%Np~g|Q#HIC6L1>cZh_yUYhW_7#TyzTgGD%_&!DYV*559P{`tCq zH%K6CR)q;E=C)Tx{e-dZgO;o!LkWlfz8UquO|gTZ1p5-M>pHB$Z(KcIdMmu9I4}w6 zSJr0Qkv|#a&N0wpZsqziMVgUFm&;8%9QB?*!?K}E$udu4VaC2%{3GU^@Af-HN&-y8LTrNXu8X5{_J6NpnzO0BP^i8`cd^FaJmy7emjK{ zv9d*6-L7?2sW?Ujgv00pVh{tes$yaXxAf^B0kY)@=36k;IRfW|6C{HElUEQ&gZK@N z?3v8~#2}JfQDCs4$xanQH4z7q?)B@}NaOd~Hc6d3e z)iC;xCAKYN()|N%UN3Hk)30=M-!~}jQti8a-%D-c@OgoFgA zwJ$BlJ}r!5F+T@6HXR^igV5pEi%~n!@iGH|s|Ac?NJH=85uUz$c^D3cp1%IMpdb}k zws0752+5Rxd>@R(!L-}N#N?^OJZd#HZaJFrKO zFKiOoE%&5HEq@b+6LAA<{QL^X7*H&{5J!o3i;pxgT{*2iV-|ennFaQZ4R(_uXq}iH z>m4oPK3v2QVXd(_U6NGNZ|Fh=L zyv}U(`vB@pfYU7_@YU~u*>4m%Pwjovf%Iza+?lC;LGE_DMt`3#ELXgi(jgQ=ZUghj zZ^wg`8S=pj1M!CQn;$Es*~zLc3Hn<&mc8H#4lxYaL&3*mMky?4+w@B zhFJVgqU#vL#09j>ttyi=Y@R#yYAEVk6qzgf2r7uC@hvuwpK3LXXdVT^l0OH|tdk0i zvlC(a@VG^5&cfJG4+0jn-~neM;cJ?iJhH0WPCOX zt2P4~yIK(V_Syw_IsTKDWnrRDHQiviBMn1TC{hD2Zt|EkpQp4(sxtbL8vrPR~p}*7lxf%20w*prKeW3jk1&?f!v6MT*R~`)G*2f zYQ|9)UJb4rzUQo#IFc^zFO9XnbOrKVdBCRY*n>I~=Q?>+Ra=X#xA;OP%vZ-Eskfmh z>C4czlm-F-O%LAIbq{5)UkhPFCrO+^wMG3LsFJ60BoamIPo+O29Ysw@qCg$Tf7&d61oh|9k6Ghw2kqs{wf3*3v`Om^+{iRbqTo^ z;<$K+cQ&=J7NkeNr83pR4JPy;e4?C#>x?`943=f>tY8y@jNznR{eYXrOxpzu+eog< zy-$onQwR%}dUTh?^K?)<#;ms|33ZCh8yHk*J@%=j+iCvF!{p}>P7s0%TJH~nKuqqB91@2>jZb# zBUK$=wK8XePigt+MFZAf%MU^hjf zLXZta-zuIXGHf+er0R7Hg|8;vS0Hw_hO{8?yfWtT;{Xgsz5U=`c0mC%jBRQB@`b6s zxAR?@XJOw~XV_L1OVyzJh{SL1U1kKG>cY-xHAoB~h&X^H+zHQvg!$?%#g^2ve%9gv zWl;iNoe&zMr&H^0elG5`_3Gkq!8!~bpbH5Jk#+v@FmL7{@?t#aFH?X=Zdl;Xedalx z^}GO{iKC^(tX_q^h*klG6H+DsU|{3l6M4_0&GU8HSEFJ}^dMiqzlo2MloaQn!r{ZT za67EcSSy9$4}1A|tNMY}t{TutNmt@{Ebf&;ns|qpeGOTQHoO2L3ShEh7k!J1Js1Gx z&A^+C;L=+4+gU?H!a*?+n~-1|H+77zpccDXpReSR`#9$Wv#d9Gn*Xv^F}0knz|jO? zksQP}oC+Qeao6Cuz<)^Y+{b+%z)_e=I0Cr@2*auuI(~nF6#J&_)Hn4)UBr)m$eWm* ziu1Ti5OZH&#t*iWMtxp2wnOR(Q08HvR9&|BWUFFj{L47R9I!YyzyT(vwX0t(j|cSP z`aorKYV5I-^+L$hFs94C=p<>smam14`s zH5;S|dx6N<8v9C@NIHzpCr&h^PHmRvS%IE|95T5W#}m{IInSXGk|F}-ToUk@^< z=hv$JUU)q}1TLiD(kTx`;#Wnflp6I_bQcoiDx;hk6B;~kYJpRvaB%e|uVP3E)^PEe z(oe0sN7rM?>-4Osh($!C)ccy-)lDwm&lVEM23G*0F*TLeIc-VltOq-x5D2;xt<<<$ z_gC)&pU>4px(EYuF%T6TT|h5nX|$ru8BQmJqEkQyp$T}2GZ}1K1>Uo&o*oN!FM@;< zD0zK9Z-D?BOn`TPN-l&G4vjAC{)OXqkwoh6mlmoiJO4{Nvz>(Mee0JAwIF7PAY$o< zXm>IUUAVOa`{`KU1*A;N6L*FZHP_q`c@U9poPD{G{l{mklYQ%_uTkx4!I&GOPvcn) zjC;_M&;fc;1NKflucX(L0NQ=~`$uA_rCCm#259qQ^>k>~o)}($g8`iDMwfj03jp5Q zP&ymUB?9Ja1v@IC&!_tO;xpLmWnqn`tkm*IJ=@&@)35SC4e+yTeK+XvLa+whd6*DR zQ>D*#dR|R?SGr7%Ow@kmyIa%yG7XfFPs5t=gN$Zwhvqv$7725bF(z^O&ksCu<}VTz z4l=-ou#e|1Eb%kSRUvZF1~*p9uK z85uPr-@={6a0}(nLDfwL&u@2}2U|kAk1MvtVYA$T{8ocVtb^*}Uhj|^1Qf}CHQt_1 zlKTO;8N|glPIsLejGaa-j%LnHr2i7K?;s_eux2#BaRZR7vjyP8{&sc1TiY(G(uud~ zOP2qx$2X>Sd*dMsf#-u$@_XUT%?h4?k#f7MR}V~{3Hd)R0QXx{iM+~|#dv<3xA3f4 zSq1&8i{5}W7^)J=p%e5D?f3pywv(NONgo08f2{{b4Koaue88DR2ESp}J2B8lIqmp; z2b3$fjQ;D?r%sz4AK^u?%m~tT2!38ig@krNpIP15*x23F6am<29+ADdVEUfcEPrcS zotQnj&>z-E*Zg5Z^MHZ}Gu&0>V^56BhkEzHzjBuv+>q-}rPgY%(C}q?SN1*Xi=U%{ zfv3vC>+F}<#s?zeR4a94NOM(@?faz?|E@1da;!HR+@@V{dkg>7dtI^pU*zrrY-~r& z9!4M=km&dHH8)4Ge-4uo?$26TVFBorEgSYHFt`M6VoogXsj4a;&Je+3&T3Q&|%>JD{s}kGmA!N4{Grnj29v{6U8^687O9>5Ca(HSJC~t_oTBs9E z)i<4LLt;P8(D=fm)WX>082>2RWS|_{kt}vVd4}mXxPHf4FW^ETjPZ(8`5nD3VA}FAXygusLzI0VQKE|+| z9^kUC(qnAzG*p9|>7Sn1(X=(h$(nSN&J8WMbk^e`l3_?0O~FV(w@)N~7tvPt9MFy2|Sh;xTK#$Wne0Nk$teINdc zmwGTnuW5JO)xcS2&ATrx#*42;A5pxungHn`L0ui29ESZC;JBN^+%j!qSx94;o7!2O z6&en^bN+d`zu_1`-jU6(8Z688_wT!l8HzD_Wp!o7?^&>{BPz;#_1SCYnschN%y_)- zL%k^{1duHmUhySHPDupXk+m{aKlk;y_1?+5HbL!zo|6O2Xdl5gEA)0*8jpu)4keD< zs(i*aT+;C3Lh9T^yUE^YP;0;4Sp0;I3pqqjRYy!j(cHt9&sWV+y}g6C%lK#t`fCfF z$F)bt#~-}S*D$<#H59ven(|ZIkSe}E;bgDOUC++1{@2-gqJP?)F{PT68?*P1>gmjO z$u%I%LHGXTSN+?KkD&k=hPs5pBH_~zu+t066k1vwMZ%K=T`>HX5sO zd4l?sY!KF{gsicj^e=VqsrIjqJ}%;W31~#%X*V<4sUs4QFYo!~vDnI=GrI;oM@Umz zt5Nc=O)6UF?wX-KO%Y~o2ZxV?{f`OQO!2Rkw(I||m$Tg!x5Y}Vv@ns(Cx`}4^2_8c z&<@?t_np0UxyW$+03Qvc29nJtuIdl zo1ju5gKPr!G5fG_Y%sEetMQXG&1Dpwe~@+TFyXx@TD`;JaKJgz>(1$VaF_`OBd`StjimGI!#$CRq+659*afk>vi9j zW}CJ^dvsU!fRWmJH(5)*n)-p=9pAp)h1qwqo{WBl)ws*TXg+-L4Z^4{z-3u)a(!Vi z0oU4$3fRwKEaa0iy?OC6NVy>)H2JN4BfEF<=ebSTbGo|E&sZhY9Z^&qXqp+|JDQmm zxO`@lwmNPC1MIP{1fWg2Z66*FqxgP#4hGue+AaaO=Lnv@-+`qRKA|?C2`VQB_d#Eo zCt~0H`Y4zE5r0GCF^22L*N3;ZPS#HS0ex^hyJI$_zu2Q=fKFRnJ8q2w|2vd?lSBP* z$d<-%;;=VjmR(qgSqJS|x>;fR8jw~A>Jke#cgfZi@{F#=nlPGmx0LKve2u0c$(?lV z8djEb-hMz6nUr+f$SGOxV`-y^Q1*a;=92~I1kD3-PCEeCps>8~!U<;zA_YDRieR(H zaNu;infj=Kx3@fOr-#7Oc`0GDbKw~N_bakNP{FBm!V!QTkP;pgZude6_{txM%U;Pm zakp;W>HH{<^l>#tu0GdVp#coZ5jtY4lwJ@g&a_fl5g!5aV5@Y;w3#}8&{^=9ioU@$ z=RESv6Ef8P7`;^70(*dm&~mzZ#D}=5B>@SHp5FJRyDIr`$J`xH^&e<@ z;Vcm#5z;75dk4R4`}gof{M@69pg?kPsxy^rqYmOE`=-oYTuwqiVGN!p!%HSiBRs^< zBzzUcXHlsKV6R`Jl_O24BA+(ScbTdaR&eD9tD+smfb~04&vA`lXqbOxdae}m4~$X= zpw+ABMf(tPIshmt6eyw&6vX#?1FsR(#t(}2O}$uK^AIz+$uqj9D~Shx zUqyFKo&E>DZgIDD0Mq5Ztn_qR?Bs=p9Y{7CqgyaWa@CGJD^5B5$p82vBXW>L0;9DQ zkMDc1hLhF}aq!r<`-n%Trx`!9ls-T?A0Vvdx2O5rFBsa{_}PWmg#R6{LPVvh$Q#Pb zL7)w>)vUUlg0OlP#K$_-`-1`kSry>AIn^HC>sTmL2ly^#pAF@&`=2NwL~M38k!K^j z%^0~!YYNIh%pp1Q$rG!x;IFq1K#xdA@%It9_(n1|aClFJuk-jRxK`gHHBy}|F8P&m z8mrm_hgND2DFB7`4x^t%TwBxUW4G_mJ$0ymw9*O$Ys*Q@bC|dV^8>og&#r+uBWd*`_U9184#Sb0-Ws2 zNas}&-^rWIf?5-I9voGxf>_aMlUBB~Mk%Dxf9VBY>NHx!QNuJd15~FJYT6lm-^F!(n*B{;a+qB)b-rWgXK7& zSaIo7q3I93z6!*Y&TRd+1;9>g*svq0O&(Mo=LmiDs6N?!U~;Lp`*auYJ5GMP;2(-) z`N@8vGiMHTcXw+O>5*5=kXQ6Y@oe(e$ZPMFcmHo5FsFkN(#*U+!Rc-RhLb*f_N-=7 zvlO>>X}2*t^~sMe(M7^~l@b>??j$3edC7Hkb5p4Cral#OFn}{xZ(y&S2V5F@Pmr)s zosupdk%!R{&}L4Gc|xPX9$6h+E&$iSu3>q@=1VpZ zLl8-|uNBc(=}Ps7DkW+_6zZ6`L7~}Y{YU?)EWi}d>r<&*w+9P159Jm;cXCZsAY=8t2R z_uk_Nhk4HezvC`cV_$Sd0&)qX24rI=0sK#-oF^dNIS-K)DD$-M@YZIs6K1kDD^;%=*M< z573vW%jzUbHyiMcDueq&sr}$wMhRz4kK*6y4N)XqUS!Eb0RuPnbf_ipH$!64bO*+j+KuBohMWWsJyveXWm$nPokc{Zob%^8i zu1e3JLBnrI9{_5A7NEEdJP@f31AzaolvT^cPcQKUY2f=r()a+8a1`m=^Q?RF7W9B0 zK=?6-11{@&bg{GPmaQ$!pku^_qR6^wRSs_Gy~On$MfW|Qd`RB#+gG!=$_3lW*vYAg z)CBp|N9DmzKxX(z&EVf20)mWPfBv(mfr*>5A5l_4p`0>-HU60B#FN%taI!ooQ zTK9435C}Ur8Ni8D4el#>7)$@g||jBHw>IK zC-t2Z`|Yr-_y_lXn(|A87Veu(%LEIifFOj1QlSWlUr+s-;RPKW=ol3|9>^0W3M(tCr$I#+hR|s1#akkI_{g^B^NpK?BRn;{Yvnc*Zqq?u^?EXB$=g)a@=w zp-Iy{NBVPPD7XN(lwFX_zkxwLl~L)4spQwacsxEv?-sIAlV>;fF&1O!-wj9g;hKiw za@x=c?u`w<+idL}*P3EDE7`5J6T8fsk~)4ZWFG&*byLKAO`q=8aN?P159>1s7;1i> zEgSs6x3#8@@o_*=$v=6V6TluuKE;PGQ~kI>+bRZl9s5o0XDl`2gwbni8PbG1EZ>AS zxs|CM_5WjMKofEIt}l~{nK4dprE-D5^TO-!B|t`jTk0Zkfc)7z_~N&LFpP0%z-Ixt zBu7I;Z;D1A1DItLQf($R2^p6Oc0u;+{20;GxGK?_oc5&hk5l`nfzO<^j- zFbuqU2TjAs#Kh?%bbf`ukSY`{qnq~o&!UI}F%6z4AJb%He=Mwp86yKDjb@=98G!Uk z9>`6|K)4n-TdSF-%r>#D{*m#7q9`av&sI3V+U!@i8`kRpr@7B3_t!!P-4WV>=;_lv zc}t0a90_{uMQ3T*cNI@G27qqpEkGD9?nODum|ms8Bbmo=hcQCeUq-1hR8*gN_|Y z>JcD%C&deA{uM)KDQIT9D{^W2U;3|}=f8XK2P2B2tOtYbGwuWYccjm2`gL$mh!2cV ztv%r0cQ?+|>wKB~8zrBuxz2>?R}Er(1q3xUxifVMD!0KGT|0bXql&!4Md9+bBWMs} zr$_X~)$l!&6D$@5I2LAZRX@H>j)-Ugj)Az(BVHTuc@PG73`}dv#?GDweFhN0+_di; zu3X)H44#1g zf+JFupIh-gR1|B$z@!QTqgxG{rL=1m&)rZunUr$@0I#QmV^`jcHM*D<;)llj`@B;U z%{1LnfU8+k9vlJu3Tz12kVE3)b2nB|NZ=Xm-o?RSq6WCeE*coz2Mqvp$yS$4Gw5n` zkPUG*e!288s-HjIf<83o_rhfHM_+vBoPV4~8(=d(g5S`uS}lV7-5=1Ox8qKZjEG1G zhB;y>^$oHAPIPDjF&uITnzeg*R?zk@9|@P24LhX_u@ZAp#V%X`(%^N;-LrE!##eCU z$Pu$4hr8ji|@!zo!(eIlIEaVXoVqz_>fQWUzk)qzLS`(i->(W z2MPQE2thVRdbC5gc3z!y-6($c_q@pRl{G6X?deHx(?2z1ddH6+=QG|oAZb?9a4-YR z`RSaGcM?zeFg@`TLC<<$RVAGH4+=QoaKYH+mHEL4V1UHWhVq`**8Tu|@f^{LD6S;XopX`D)y2P&$~efyT2PGxy& zD9D=Da+Jo_B?G$8_YM#J{SmtNYKGS^y#kP8;ViI@da$b`@Pl4&JvP&7A#d zc;>p7mHo@+g=4#~I5}vE1%(85Q}v(gbU~I2!gS09tg?+es0$8-!(ztli?_=gv=Jb< zxJc}0Fng3|1=Zb#$v0YP+BbkY~N3vBrMAXC&G zA9)mEnzWUz?4)8!p3ya}{$JJhGzC6^INF_1e(C`j4;kbe(e+~TXySy>z0X1R4BBxX z5-fheHEH}(wrRP}n!?g{0$ofYf5sIs)<`4##_Bik+))E_8)ylHK_hz@yb^!~1I)OJ z5bsEHWNX(TwNXcf`j!5^O<`v>vrB&7kF}p8kSCl1u9G<&;ioe8DWvX*SiMw@aM8@X z;AYh;K$G}%HfNO?C==R%NSR$Es^KnD5`Y$PW41?wTWDGvNI?xtlWp5veAkK)KQHCV zu220-!&v=(d(GE~kRts}$6g3Mi0?j%%G9hb{B&H4+Dok!M^bSP6=Vez-8XCprn8&u z%E=w|@SbOSJ1r$+Ow2$q;U-WfocYA$kJ8c1oq@oH0EjGza{AL?De`si5C`m@BYhrI zPK+tu^MMl2&!0Vn`g24Y<4hO_f8_tVZYOeC3fH&KER?)KB-m>CjyMQ-X>p0v@^xth)_A};P$R0+WjMWx0VyZR*ajmyHFy#hzU zcBX9IoonAvTyW2%xT2Y3At*2*p{>iCc_8RDd5bWr3iB`n_AJaQN-#KU_J3m>qC+$Z z7O%-bSmIZTenj>n6!cljnAISllqnK= ztxE=dfE`;T4eKU>(@w#)5TYg+Im@Z?DG`QOn zW`e}G8=Z?X81Epiib%*&8#P*ZH1?Y}zf`o4@xjm#YaP4N_XX$dKS!0GF{F*k~pNHi8r>K8IZn*7R=kaN% z5-&Q64UfmtRp$edH?TJksp4DHPlv``)>^4w*0!!}zkU|Jj}!wa>9kW7?--RTE+s?8 zh9hR%AbD*a80R6FGAVyrY2D`F!V>L*Yq_Lvb`1A!v(gUNxeTU!=wBr%WcSz1;uiE( zXO}$xc9-TO9`#$^w*BG=>0o`35ldK*HA^yUA8F`?q@QH1cW003#>^AdYE7@@CHCMc{DlxN7E# zb@C@VC6aa(tlR+%0voYR>ec~PTb1Jpq?D#>iit)11GK)IkaNL>RMxdMDmm9G`hhEU ztVc0GR9`NSOg zTF{FhMQIlJ5V7&|(0qEJz8I_wH9=m*4{l5vKq<->&ICzPS#p^yPgVMh%J=mH3t;+w zw;}xu{!T7=^y$+u-`c{$Vkjp!PpWr+HvsBYD+w#l%6en;j1XBGkdnZ7KS_(m$7ZD(XR+DW zKx3FDCg6lsX9;YX?MIM3mpVNqSWIDdt{PY&=2^s%yHc)c*NzX~rwnc_%V!#}1yn5b z1EnTc7SM=FPDD-3RG+E@_| z2@Az#o&nin7#eh$Kk<)VPgf?l7xNqeQ>5^ouP;hNfMup8vVULVm;GTJ#(u9d4kpOE zi^DY>(sT7w8@oDgO$hmtF#QqVGVNLjJ(|FT0+Uq&1j%5<}@8jpyOg`6A8{4+O?OQ>`-~4^ZvzP>$!F%(W zs|QH^5ezgJA4SZ`xvU=kjp|*W*t&Z0zHect{TUzHhm|fc>M%*pc_%PpAbC^=rHen) zpuM1GU-G0Wqt)_@kVZG4Z^22rOkd+P?X03g)1Y!6vNbjk)I4NTTvIk$$tM@oo6fyp zN`+pifx*rB=bwfWI|qByu&|hSz+RV|wM^lp?;HUztW^SNoz6Kc$HZyx} zDXw>To!8z54DBH;G61_AunKk&xPTm#Y}Y>W-8^3t0~ z2gmyB3+UqM04>s^Ksuql*x2O5M?WqeQLC|f?hd1Cu}=ep7Q6)-%pCo=@%VU`z`5^r zK|Sj?_ud10>@qO-~wyOVLf>OQ;42Ovz;7hcNS{ zX5Qr#&>A0BY_0qs7eEyZK{1gB%;T%(`lIelWEdIGxb70qDrhl82DdPEf+z zQsz*Zl;DWKHZhqn^2&x2TINu1oiur&wsvZ6RexZ@aM>Sk*!LG{-lID;5Dv$C*Angtj$DolCvDwuhG04_XX zbT3n#`&;ukkkUL(NwF(9;FRO+X77INYRPU$Y|qY+@6|(J8k9~TA~VwK+&(7 zOeaun=@jmpfx^Nj#eljS5A&4Vwt;XKOnI&hrv)?@l`;P`R`;JBp2}_+X}jZaNdk}O zN{`{*Q{!}K9ugDJr9^&4=|BAzX%Wa}MKdl1kAfz}3Dh*dbFQmI4_|m7O^raH29n!r z1^X-gxBMVKGVt?Ykxn}{p2MK8N%XQhL?M2qxb)o4Z7uXQg(+P2SSIvm{cz3CF>E@)}Jhl)-W`tzOOw|!Ah z&)n#P0jyAb=_U8$A|g!}dDspdcm#hnMHCdxw0b(F4*dY6=3?%&PYsJAqjn_Rj@TPK zh&mmtv3JGx-3>}-CZ;il9H3ztb~ZSm3k%+1r9O#dkCN?hra zQkKId5M*JMBAT_mkfCvGg(f4Aqp!tJo7_*6+msxp-Yk16_)Q`kZN>-;G3**qxY2cZU9z%L`ZbKg{u8qkB^O%GL0V^Y6;8CU2 z3$GBeFt2zSMbc?hqw+?ONF$6zwKtN26=1wPm$9q7i$~kg@X1RT8fE$bA9(rr$gs6G zYLD6x&;VgmyRNS}e0uYm{?GDs1K@8kp+mO&;?HN~uDmDsbIQd#-+4za6n1+|(K$3; z6eXM;reKSsqIdS}-RP%hf@(oCb|BOH-+>ePD~|GN1HK40QNc*T!?|$I<8@{-JJz&a z3y(Lz?{O&X%mitWPAIv-01~ay(@eAl3I|Kyyx9+DeP=KKOZXAv!VE>&0`W+HR#s=r z0{IhlEk3k zfOUC(NthXHA%X}^EdK-solEQ9=D^RMV}K%pdDuLkzB|KPV)aYbUKvQEpF_J2dR3%7 zdCA+6pKym9P(Bs@e zCTD%|*by++vILfVGyR`v8~Ykl!VLj=XEru-$G$JdMUR64lI$*X8PPlXrcc5cYm zJY`r&)3}43KHZja7pRIm#MRV+GvijF+}S_VT8BKObcK;N|_8v-nx<`Pjy)uK42 z%O;Jrv`@TPec`}VhG-lPxHf6a$SXTE`hNG`Kur!e*z*5hEC(tX>$FtU`wg;{kDGPJ zka;y>8u$&J_S89ZK*Ax{ceSC!fk>@sdiiws$)5>iiv&!e+j!6I+t=E(!$HbgS`SGf zxzSW$H3Bn;R%=DY&tygbO+#)Y4lPqCG#ZSm{Wc41J96l^-Z~7cfbQYwv185R9mcpJ ze#p256+Uzac}S>GBd0+sH%dZ!VzWql06KVblcpe1+fJ%IYO!z{l8nQ%ZZ}{{nmsno5!QZ-IY8N|BF)2t z41`_W;k0R)aV5C;r;GqdiuM(j3z9EvmKX4gx^rQz+kwW zD&A4i_zBqMfDU!vc`piz7|`kRe45A$)Q=J-fV)cB zK5Q3Ad7PLSmUDFWttW)c&uw<eT`>T;L(`F=*px;)m_Ur%u6_5#Cd{zSX0X9IvJOz5l2GGmu~lfU+>S zl5Xl=!-!+lRKFN12R;I6&^D`=m0gK~?|afBtna&Y9ecY#JLo+~>+s){IdB~?*Fh}F zfUJsB>ixd9GZT8=D>G-ky({xZd3~C^P}HfAj4|<(k%3OI7I=MNR+GR1_3K#}C6M=? z7M`kZ`0X4TO&xw$KTuF?GV>@b>_qo2fXDDbcgZJdJ(P`{w+jR3b~KU|Jhf##7M_v1 zY5!JqLo7DO7lRPm)ID<26t#wq#J>(Pdn6Mk&*yt*k+0b;w!&yY+$Ynac@1pCXe3*w zPDew(l0fkNkcz}yCRtG88s&hFQoJBA6+&(tL&1=h;D+7fv2X@?94z^6bs0kTH!juW7`HO$aHu)j6w8A7C#wa<#-?V_prJB zWt**jr$QMY-rLIAr@mopvMl0HA5=V+{uV2Vd7HP=0#y^@fAo&D+)|z_8kykp%}mzN79}LKj5=TT`@-b#>74m4c0J8DSa@z}U(BOo@Xq2Z@LBi&k8G;bnfUWiu zF%ET)ht6w~-8`3i{L7NMc@!D_1b>Q%VsEl88VSV=7Bnr62?*4~`QJS_e;j@=Grj_T zFbFiU8z62Xn)mZ-uupLYHd^E+JQgQz(a&h3LRY&)=>)4>TBQ5sz&z6pVWW!=cn38FTwM~{{a{GAxS68n|7x36=% zfwK49j@&m7*mhQaXL$21T8C61oM>R5j4=Q}mE$VV89$ZDbTz$l!YoB9h}>*$A4Xny zz>Y1%eI?JV3#(R^mdE=wM~z4%479H-T3_QzAqy`OX8y$`=+gs_Tx(fn^eUr1)t^1m zhjo)V@)#_LFjOnB_U*ApL^7Yw?UTxEH$y(X%8THYi)tE>EczXv(QkPUdz>9S!-*0- znLrYU_6=~08>#;sQC=b&_1oG#0v;2xPD1OeL$Y6U2bdp%cJi!I>s~mKefOJ8;}W1T zO=?;e{4;Um60YE(0Zb!{pT4hHYgHJhF3SqrMdNoHwfbwvHacATAH0q^V6z9oNR-N= z)G&=oPzI2N%h7-L$#(!I9A>(XLnc?D1D4L9G@SXE!h-*lTtKi$8lyQ6q2#S;K30Y> zva`dScpsx_aR6nzFY>hS7hhq4`;J~=-?JJ_6AByU&)b+7wo2Jt z3`nobDXmq#Yaq?;rAg$HfzMuFuoG84 zC-EVU_2t&~36^jX`MMHWrNsJkffRlu;ugSeQ z6T`MQ=y7uLo#TwU8=MS^hBRFk{E+9>UnZUc(e92KM#D$7|LrH7mjDg2zP^4=)OjT0 za;Pfd*U0JqPgeFDTuHs_65%5zX8gpX>mgGBHH25+y^AA-OiPEtIuZ)ar236@b!)AU z5U{*CpZ3@oT$e5OP*)y1HJjo^=n6=N?fi%!(o3 zTqPYzD|d$Z@k5tdLWZDk(6Bi6$xvXNX3bsGQ7?Ez)*m-CZ*U42W5t;#Gt#<0R9qvh z?Tm%*{cd>x>9pg3g^T6VT4oO;J6#SXlcO|=((vAR6YdDh_Jhr{goKVvhdkrZaZN|^ z&FdyY-njHxeUT3>vLg(Az-gis@yUiehd)vf8Ev@Om!5a^&0a7TDB^t3cT@3!EY`Ut zIIXSqZXS4#^&?hsRS7I@(ENZa+7EeQ5xF zqvdhNisMi3IrU$;;OLw*mG)+BcqB7GKG7npYUp%4z=*fuin)lMo)hmlJECOJ;gBQa$oZCS z%mRzV0{50L_Qg3+;`Bml4eV3MZ*Bg4ek=A^I||MlzAN??*$;m2=5B7645es>9I;pT zvs~I+{oJ{=U!rGW5GpUQk*IkyVdNw=&6h*6Es)9Vt*6=mxdUUDNU(_%c4M!uNtvMX zHL2ei?t3_rnjvI!Ab8P2KZ#P5+S>@A%u1$9vCNbr!e17qs;UarP6_9On~q%C%P{f% z+?`SWG_{UVH-5Ig*DWmGTe0hvPrgVj067;aenhsVS$grvb8DQBvLWWimaxGUEs$U} ze-fEZ6ogzI)f_-Bj<`j(QDgQpyznYk8x43Q1}l8+qYP|M?wWIk`(WqB!*J(A%)7IQ z!N|sca00KI{c~G4PDKGj@nJ_3zTlLZ>kzse2dh?M;4dJ=i&W5uK^2 zx$+T4n6UU+1(iB}=FFpzkUfTb${gTn(+I_TLdQj6cSxOhI@Qb;`A)is8L#V;p;l}} zux0Kpsy76Iq6$Cbp^Hvf4I;^D5X*pge8isECkT1bC+_Cra`~ak>6giF+*t1^$@RZ0 zlva7(hTqR0X@z`tOlrfuqJ8{4XZ^j9;$2M%YCBO->cq*D8UG7GETOP(VbfZ)!wBL< zy8w(CDI$I>fvSt`WGQLsU_ie}-T4H9oR=ygM3Z2Jj8aH^+}tq1s`x3{WdN-DN5J|_ zM#T5MV^T!JX`OqOOaY5u%s`KeNxj7C`Sl@oQ=(4)i?Ba#vLc;v{=*KOP^FslrUk)M zpE-)!Um?Sua*UD2b~0Uo>2{D6ghZK=J)nr_7{jemnC{~=0+DVUYt6`2uQDK&2O56c zv!en@6LM4~{f;EZ4_p{&`XL11t3P3SYo2_uCd0JD^+0KEmAmWnw}Y`tUB65uv-)|B|Cpi@HC(PGW7l zw%KuQ*mvGLHE%`*QHLLGS~F zwM&tURU0EN5(4xPl2p+;|E~ms&7s!`R1}a~SHlZ1WgfQmFxQ4WH21vW;0l+9(>`^r zY-}2MUjiqm53EZuT3p_eIf1*%=rVb~J6vRpL#J-eoy`Xomd{c;-y7D@9~t@iOkhfo z6Lt98iJ$PzWEO1~iu(uZ zFgS4-{azONfNTzf5pZl2()|2NC_lgsFM78uBF@X`EIPi!4WVAjT9alHy1-?g(|n(df| zwY89yH7SC5*S`z+eBJ2G0LX2YR#tlU5G)!O2FJMRbzwK&&9H>1b{4^vh26Mu1CIKA zsgoz~gR==K{xI4my~rGS29?@bpYsb_F52Mlw-4?ce z!m%N1SxucjKTf0`@1INfp7rL9p;6s0SYNWR4Y`pKsn6kt;FflLO{PMhu=kDy_$9uN zENvc`V0X^XvIPn`p3`Nt_b`oe4~FukZ*>?G5#n`=CuAMtbF*dfGykqwUyX_T(wJMlXKdpXzE>Np%e9V>e;&3h~~ zJ6puX-948`Bv^dCbP%qxwE9D6CBl|%%g~}O5L)3N`w$}-cDvz43gL8-Sw2nvWMtMs zTVWf2LaJsjVFE-fExk-}$*#8`_=}5*YRYu!yf~H`<^X;V+ z_=#&jWU8Ca*++f5J78FuIzeKl|l)w4Uy1i4X4Nebqh zYvam0E`V&!T!I)F7~!{+4x77NvONCQ&V`~NVFH`~1DsFetT|Zleo?$2C2ExS9;Ub4 z!C24WM36cimr6Ti{Oy`zgJThnmu`U_HJ;LzIGH*k`;xYNZ_P`b&(6pXb3wm0H}j|~ zSAG8eN^JnMc?Of>ak&O{qkj6Gw{|18Zm5#5$v{rmD2@1C?~H$%2O7a z?f7%GN5(t-pvtJ)q{vJ24-E~vFolrEuDsS(I1MJ`!U&itFbKwX7QYM;&b*B*oGA;t zy(0M1xkYZ5d%r6Ol*$eukzMnt^4ye06gf8O5C>lC9m)~`DHHhv(SPNUArnM2ty)}h zvRV{Licss?^CQW*r54ljn)ohv&~LsLW`tRB#~~Lu+QbqmV(f`s7VL@S8CPB*zM6)9 zTOgAZpeZZ!9Ogvqf?J zw$G&IwK|#V-xsiU$wY&cmLU>)aIXwIKr+sI7jge2AP>vuD&YF{q%T=m zfSO0SOd{P8-Btf}eQ?XK4~)XkcQ27DoCM~mtEN69JcBg?aV9MDvL)E;ji)J>4sMwo zLQO2ONS1B>XzF`!8-S+<1_ovx9+wSSb~sc+wL*xg>6+Pe?wK3_&U2WfFv->3JspF@ z-1uOFwdySKrZ|;Nr>FPE%8~w$3y==({{>IU=99@62ATZV-pT&@d}8FlWQ?}_>`4>X z4#RSAPh5Cy?@JH!fIlXG5ZMzOSLxUH_Cm$wDA*ms@#IE7qybxmUm@&bY zwFy&@@k3z$U^OpK*4Py=Zg?tbPJoBuAa4%t?RBW11q8=j@@t>u##ju1l{c0!Hmp`} z`C?KCjDQLbn^7JObs(;L2IE33|sVT#W$#jY`>KWf?y580z%ZG3hpv z>VmcR-cwfxZXMsB84dH`rLcz0$sYKALP$jOKr~t7C6wzs7D#0YfJU15Z!XJ@JUk&L znC^02vcq_fSu{khm}i8HV}BNhC3IZ-0R;d}C`>jD8ndv_huRIBAZ`L-0SPE9JYE9s z%fJT(K?f9^PoF+rYuh@ocd~Q$u`5M8z)`G0ZagUv{8p7VSNrGvWH3ETm^elFzb$*+ zd%GXJ>#K{0H{IGTf$%z2cD1(NYiZF2jr&LQBSa2Rk#Zg|%*!6d+FF}j(%w3=*MW^x z?}9a^UV7Vnb9Hk|tEjkQRCU#h5A1N&*vP6m=(BwVYWs!MHTbl)?F};ve&) zqeq~qUwIj92aoVX;#y0H0}TQ*-^_2{*1AG7l$CrEwm-owr13h0<7XZ}7QL`iMa^2a zV?Byo(uFNw(_Q;t+Gh#ch=;aeG&G*NyRT@52$Qf*?|H6(*X(J`y5D=rjkW{&wxFQ< z{P|(MY|I39{!PGhZF}p1%aYH=2w#J`;v(JzOky!jKG1)}vcT zW4($|!lC;C+m02QW1M_^MzNo1JibDziV|6T(#PknBl>n6gNl`LHw$6Z<+Dt+O5q5& zHFqG)kvL>mf8Ljt76|SG=J5aT2S^McE&bU0$Xg4c?K;$l+=xd$7MC)x&$4pS@1POX zr%I4D5+OGs#3*#lxbmYUB*m};id6RK+AGw0w5T6q6OlVVSNxKJLD$E#1QD1vu(#Ii zqM_4eLeHHsI4@}E;V~C`@NMep^)x1grDtU9fQ^w8#f5}#hY&*7jG;GYz~Jn6cpIRW zIQHSWK?fM6luFO0DU8wX#AjyCB7*(m*rP-BWX_8c>2?~+=D&#y!~((q0az3E*|w0y zh4C}*J|rZhZnUC~A08Mc5SFeLbmtyI&mbwf9YC!9sYSRd`45h{0|!I^hPEOFn%G;r zC9)5nc@I)em~jUMJ^so~EbhgKLk0&uddSt-Uu9+OR!r3~l%gAo+m88B*Nctj@i$+1 zd%8&+Bf#UGK3x*Fh-JNYrk5}~*SfW}^4DU7m{Dj5xd`NkHO62d0-QHxLO4z~C+GO( z_d27tZf^fsJYR7JUw5eFQKI7Q6ei zz;F5~eVNPjgzEDgq_%kvVP}jI33bRp{SJ&3X&V~oLCCOj7%(X$vu%ScgcuFNd-|aQ zEa5rHcjCke*b!gI8fDu%BB^WU)FIRt@ypK{?uNhK$wbqQI@Mr1!gd-dw;Mls;zZzg z^A&4yBS^$kn+)b@F>C=Mr0|&e2J~6rq+JbF$xef{jgHPcaZcF+nZ!fkNftg|*nMkG z7s*?7@RkXU1R1_hQHP` zFLt-{qXWFOy&XW|CJm5FP%7W3-`VKH)?^bjGU;q^>{#vvDZK8>f zDa?`Ac_7UcK2^{y&-CrsNvOWTkqlz`R-IYydtbCpoB@BVJv}E3@O)TiqB8&%S+8Gr z#P&;|{Mf$)l=>@oM}1zNI56?9gsL>5|4{0GP$yYf4L;bv`kaO4BmM6#63MS96mFPZ zM=837Ar0BbC^rdmMc}A=_WrxR_*h{QmsUbM@~>+FE6Wl|gLoT2=91~2kd#EYe*Jmg zj3b_FyHeBJl}P<{1XV>+0F$M}4X`slxKK*$JB*s}D-*)ht3tWa%gld!Mf|h{(YorQNX>e+%x|c4u zWYUzc7Qtlj6!d8-t{1!G2!zU>oM+DvKMeaK65u$>t7SAoydyH9EISv@AbH-WbhNb> z>uGfpnm&H4o0!M~Tk3z?n9Dew($(Mp2zck3y283J+?N-eOLL;mB05(9lOu=(HxC4P z^9D@w6vxMNX|KMh01SbFkuir@hi&Eh0P-l!<(D}kVq)#S=kU6r6Z7*j6%x&K1p_K@ zXQF8_htPV&Ez7pNQJM;Jw{3?-KLK=PdrMEjufI#M*a65CbOhkTvmIn_JE>Na2p$fC z`1+4JxV1j0VK{>O}>u$$r3!->KFEq2n~XS{gi; zTE_5n&UT!Qe!`-Q?haBBv|j#yO?`Pdm233&iy|2}GL)#DF(fJTRBejPLS`XS5}AkC zQrYHqNC<_^i85!FZ5~RZfimqfB^#T_T;F>8{l4$|&Ohh6&MDsad7o!JYu)Q!_YLX; zw_sCCyi{j=YGCdK+j6u`7$f?;yIU8iWH`&u%tBJ9;b&BGUWXnD$f2N@P2_^y zId}~hRP<6mfoCIvp#Lw6!X?M#sOEW?!PZu6j7v;}ECFW;yiu<+uzPv$_Ks;|vCc?X zZXqn=COkaMe|}E6!9zTXo=KbwH{Q809e|DX&=KVri4@2`a~o{yqr2=eU!2g|96r2R z*68ryU=&njP^e^~FTl!ZE;@>2mGJ-$V#=S<7(1tnob}`bvvO|;eDJ620e(HMqEdL2 z`S3EL1rN=^xv{NGf}2B7HJ{4FZNkNT?+mJ1U;LJXQqK#OAgn~g4->l{ZC?PFP4~wX zKr)>C=n>S>yqI}j$fEvz4s6oDg2~W;RP!$tueZltL(7-o3_>z}puM@7g~fefEL?)N zPFXUflxRy5bnyrN_f^v08?Jr+yqscO2z~d@2CG>J?Zv-L>iZDYF4Tyg-?v^#RrO6p zh?y<0A^`umD0&)MLlJD>^1mGTxz7v$eyav7w4ur9d0get{irCIZ6Qe&?WRew!0ijk z_k98c{1=6CHAcB23Mwiw&_n*ez#PWpCq#N;be9s|p$O8hjJi9`hhRN@ugu}{#^6-v zBt@UsJjr4RKh_%n0%+Y_E!!P=8nxGn(9*Kpyoqf|IS5OEi(S-^NO(UxH+L4`v34aF zcyDMl3^s)!Kk%-DGH^O;9V-+SvhD!6z-=u*A1FYuHH!4;|DEh{GbB0I5Vl(FD}IxQ zDlkV0V1nP3lISjz{|VFs@q~+XLwSV|QqCw*>xSh;pA(4~C}Dr}2z}4FfW=ztd;IRe z6ZEwzM1{Zz(C857@wA_}&2j1Ixw>@(TYjRqCQl$riarLpAee&v*`tGOcl}TB z@b%Z^`OpaQ{mFvCX-2cg8^~vCG$^^>=^Y!|IEV*uGeQghp;0owriTAt(rblqpbh9e z5$qP7+RMVU^z;^(_^&-YY5oP?)m_B_-l)3d~tpgX47kzQo9VYxW^V85n0zwRQKXNwqGDZ&7 z`ANYA>PM;v`p023Kft)ZTR6ziPw~CEcPY5BZu+)7K9!mWHMDPw>rD@Jwx{FsSXh*( z=Bg2SM5r7NNf)?J(~YobeL1KCBlW={A@g^!IYG*xodLtuzE^G_uO+d9!sv6CYvHcZ zc^IRI`4q(BoWepq%im84yU=ybS%II=s$CEwY>wuf@JG5K7bxCGR6rko`V6jzZd>1+ zo0YNb_Ll!H`cN^jGXX^6U)!>mzQzW3Uj_%i0@8+wFywS~ECU0>f5jG_H8@@JQ}^*b z6!U-4wYT)kfsiI9O0byqAx_8QhjSdg~jmkfx6&-21B$s&ff-dkNWRsM~`8R zhPu#GfGb!*U>DuDLCo%)P%5exBG3IRF!noD3?^tIUh352{H(X?6m>zctYvYnNG~7F zoAl_RQB2A9^#kf)f2N5c?h$|1hhch(uuL2J1+aYBoEyKr>lBs2*Dg0dU&qovn^2>$ z1<3#}3?4Uduqye@%?*F_z@2V=RT!CBIAgrh$K0Mkoh1+{M--i( z|0-B}W5}}Vp4)5U9nb?(c`58}SL=U85ab*|wba(nkVhO4CPr<=Nf%JlhI1gq=7mQ}BJigpbSZlpX*N;?jf?~Bt=x8_TELu8RQLefjSUBhnRJeb>KEa)x@Bf zPg-WeO(vGTOTN49$>-{}5vFc0CNAzLTD)e0o4ls)XW}EQF9#rLq}Qd5;_U2z+1b+) z`*(n;18G<(f7Emf<;55mGAx68);6;#-ID!g$(bg70=kL)P5`%R{MqyL!@QFO&-X8l z%tygxJIwJ-Hr66qh2eG)$uri3f z#U~dxH=jC1IvB5c_WZf7`WjrC*>;XtPTOaOfX6*sHiBVsKAUQRf;jdf=&knw2m zCoa>QU*2VRF69O_pTJ_NdYImW8I8dSJ)IgOwXYq)YWKdkgeSI?=G%y<;auu`dyOv_?`a^93*v{K{aB;Hlh8zhvOyk%%!Hxc5@fo*;lH0y%{hq$bZW&#C{PP(KQba% zJp%yY#JCv{V@ArGemm!`%<_%q`I(8xUi+NK#$#_166!Rvj9CTLww7eY8kG@S%gDrp zgq#;I-Z}}I$|)5o8sUBeu%ml+LmDo{(5qS6HTUF2MHNCOuFK&4{~DS?_L^ckNxp!y1cl7f$H{|d)>T=*DD}+W*KLaZP)Q42p zT^`>TyT^fZj_to=?CiIqiUTPuygBXSSTp0$zqDV+G$7#lihga?l=O=}f8gYYMrE*y z1=BIe_F>ZgQ;K(Cs$sruTi5B5bxOq^5*8k=%cZUwM59|<1$pDe0WJaXAJ|gs0*HdOh*{+w;qBZunDse^QrbJuw^eiZ zV3ruxxsdtG+Ms0uYMbJgIo$4``p=0P%;?T4;Q{b;_TVP*05K}Ac|pwMH&<`=+zu6P zmc*oJ24Ga2$JOjr>Wzjj1K-6{us!!M2HkaOK+*Y^y{oJ9!+-%6syxZ)9CdP z&er^KgB=-cRqgEPP#QgOj?ry3SbOq8ymYcH)U#UE!+vl_%Lnp)bJ&aSH}2L8i1p~P z`>mH@0d?WZE>@P?L(#q9EwOQu!E1- z<{l$8Ce=5@O{`&_q#1OCOzd9TQg^C2@qZ6v)mTFka3ZYjzukd_GvYaRB69c5?I8V= z;XRqxPOi09cEJVWC!11R!AS>AJ#O+Z6tU?z2ORrux0${hF3L_gf14Zpu$zv|w}V3 z>9GwL+S&>84X8g>Rlt9IJ)p0NGiSpr;g5jFS-sdU%CCgn^cBOVyH1Da0qpG#Icg2u zk=57xZXXl7HL)qpwfEk7#g`M2*}dW(lZUK5pnIB-`c!G_!H34*Kpy*Hq5m74DB#Ly zu&rK#QW9irs?$VY3XG`vyded}815h2t5vVI3}POrvQePDAn6rk6{w=ykJ(u{Nl>vo zg$X*=QD$majEa*ay%fCL^zccvA`@S4C}Zg;WAlb@morz+0-RLmvbh@ms6P-NArZ;6zEL?Po)4mX~o@EYh% z1$GpO0s~LX?U~w9EY;${yVIYvTceLz?_!j-P7b%s6T@~wZ<=v<6hPB&pDZ2Phxij! z3syb=DwXOGCeYgYb)uNgk)C`5yQ~xEOp0tQgl_UzdElaMZTEdsSLnY}J2uD|d0I%^ zSyxH$vwGwT!+)ho_aAo2($#q0JE@_O0Oen-_do{p9ip`F#vdN|lGx>GP|)UDq8`@d zQ2nFs$+-rTayVyY4fO4MZ4KAHI(xC}$>PVM^6i%4CNig(6_%9DC`a(#J#%=p_m>2o z9$2c6pyXcXVXbNaaH2)3q@m@=UWu1*mV2+1Hwu#rwC~olOW*mCXOlU+3dY(Hbz`Fk zQk&0IGXS%Pq%?)?fbDOiJYWqcV__W_Pd-!8P(IWl5&^!zmiu=1^=?2Cq31V%22!x) zL}CIA;l34{=^DVI@wG8(>d+oh890LHee2ff%nbl}D+qB$d02>bu<3r%FI;l{~%TML) zka^PG+iI-8W~i_4RX0cvX!8#Cq4NI8mYc=>7v=s50P-s0r9hB=xt|zEZ^1DXthWyB;gfD}R})kCeC8P^Xi}PsMpA$&!Xk!k zMQr(4H+I54eya8K*NSxn$Q&K-JaNqPnHd++6Y2Z!8qChVn;qka#+w#+!xvNe6v~Zz zj5T7>xI1?41NObNTBeRmluiyBPVUGl*H^=E3>1`)xJaqrzFmf<;m;lr0qOx6WW#bg z{G8BX$ht;UPTU7j#aGd%0#UGuNm9reP8_9=O#Kb03TnOta*_WH|4fj0&dR|tF_r3| z12FDvstf)4#g*N{G8(cFm7TUWH{bX7UzRi6gcvnoPz~nkucxrqf^!%7YAGkHfNL$M zs#gjXi9QZ*D#v7hkKvPOp_f zpzW6V_fbAvM{Em}1O0DO`Sn+`@Y7Tt6qk|5SC14kMVQUM)Cpn;}=hMlicMYQC(;v0Oo{A9AWqdzAepAvUn1BC%U_1Eb$nK4v z*4sFqosm&wvtjUyvbJU5`Di0Y^nOl!QB%Y>mTfdXf{M17Y!`vc^bjZc_3f?F8^a18 zeg)A{5|M@P?+#l?Bz^}(e~Mj2j*C4CZmQAIY}rc$-e>|Vw){a?O7rCY-m`XgnVwsN z-d=8pPyE`0*R77#$WH7uZo@Y(1=Zv3+9{_e&d|NBv7=I{IOwCbE6ui38Z{yDehwkC z#-7P|5hV}D%A(w+4QV^#7pVyeN<>AFU#fB4d?2;y0qZMDVh7 zL$0R6fO~)?9(aVtN_~=i&dSF2jg)GiK6jy-Nqh2d^s(Q${M^L-$hpztD1e=n$lnGo z!OOH$pw*RUk4t9S1PJ{JODCTR zXee!NRvrk4xHK(VS2V(F-V3T3ev1X}Lg*MWOWA!#FIE8)taF9pX>Et~(UL--Vae6f z9cGA(jQp>#5Lo0Q*y5rR!o?gn4FG#k*0+;Wi)qeB#pA~vZ{6>I<54<<+1YMjA^Ob5 zXW629mq1mxeD$hC{ucu8Ss*}xeX43|NQD3b_wC=f%hD>Z3e5RH>lwxeouw{({Ri?= ztBH{ky@HJI3#FAphYTN{gi?k8htp0VfH$W$kmt$5D`4(rfPxSoM(B|~%d=44+4=dY zpbJqcnXq^s~#Se$r7&&3=Pw% zWdBRJ8WOiatRel;jI*#Ue3Uir+CtQxOL)t|aLHMmjdT>#&G=NU`X~H4+SU$)rPpZJ zwEDUtr|CkDRx;}l)xfd;lSt3(7+?q*k7eG!fB&F+<1$Cgp7HhmJB0I6cyaSnP)@5Q zhG0;~#D4{&m#>XbLxX|2S{ zOJ}NpK8p9UBOQt=b5!*r_T@?U2gqj;TaFhjQLSQbYQ!@DMgVB%du1N1Zx6HA60+X> zprfgNDCvIl=AUHp=g*%XPhn4B_UuAu=unw#AFHSGY?ONJtyDry);B|cLf0IUJnRlg zj`x8LZP3TP15G`^*`Y;E%KbqU)in9U%F6oI?;BTU8*NV-pLX7ej*y-VbdLc!19RO4 zH9b?1LTjcxoh#Z8wDn2?ZgLOapfDZs6rd4Y(o3!bK)QcqPoUIL~tyc#Qa4PNMSiKF2yS?{(P^s^r-KZl^Jw5q@KR6s?EilTyR&_?4Ki@5ms%it|IRO5X_M@MIRnREcLmBoJb7A}`kv0Du>Hvlod z{ri(O%XObYKOW=*>P{5riI-1w67=&kANI|h^*omNFf$Rvl#-r5JEwiuIrZdC>jB2W5-N5El~;3SG^P6XQlUQ<_PXCYe}#Q(Z4vUbxt6Z# zo#UIeddxV$!$)>ePJJjr5r4RpXyBjZ)6CtjFTvBYG|{<4t>{ZvlX(4s2Oo|hg3YR$ z&eGLyVeDEP{^-o_$=~gG{lxgG>-)E7I+{Y*3nb@)EBB6f(cPpJ?~g7G(L`Ce`^e?=;H= znIly;gflL?Ga7ZVDnT?Ga|6hjx)t1c{8oOLj9$FSjbDBA-8daXe2FB7L_>G*?yz!S zGd#LM)s^r_*%a9ect3Oisjlv1 z{RG0*bK%ifmvGf_p|hUYD@pX_6UX425~L_6`~hEK@wt2ZjX7#kTScDp$;;Nzh~X$5^FE^ybmYO!^< z^)g5kN7__LKvV6XNo=`4?S*}e9H#s1q`s$RSd%aa+kUUiIpg3U>`xuKeCxwj4dE&W zdtTcitTv*tm%@y6u-pXRa2XgCp&n1R5~qEDmVjQ+AYk9&Qh-ldrC(1cY)2B-oj4Pn zMC#JF>XGCl$AK{^#`d;SQCZo2)5j`EM*!3Qw)MS-!#EhMfmaYD)lX1UFbfM;(@jm~ zQ#d)=5u~;74s3O(S}wu=MLjy7OAhw3k9AZ8ci$J*s;IYu?m%}{n>)0@89~7M?D79@ z$FCfZYLNH@xStEoAFDtQCo;B`)!8g7T)>uCye@) zWILH!6{;Z&Y5djl#%l+#vQ-iwa`2e2%}I_&rhN&15P7mf7L-x36%FAn&ph;%!Tyxb zK9CNytsl>Ayf};)G1OS&0=LgT&l(`B^j}B3j+>{ zvrGlk@R*prEYS~*oSk!LS=KCZBJ!VuBz6erHCg>%eg!MYQ2*z;EWq3Nhsn+J61iBs zB>;nx$Di}_gZ~WevGNBJL5;v7=NGalzDI|5y6S)ake&|8*bcwvD(Y!z$%kG8l7`C? z($XitOxNut|AGy18fQ%x%9ie)11Y*8ccuuWH*nCzR0Z^rb<>r0UeRYi7=HI6f7tL# zHJ;E)14Y5Cu+4vf88p7+-kk526l4c;C$2vAH9=@?<*Sy=U0kLMwZ(xsbz>D!xrYU!Pp&*V~IM}z%MIjpR%?SbE;qoXk5AgMu%;22^_C99~a zqUJhx&-7${WyQrlr+)4B|* zlH`<>=Eb$X)oOl5+F9hU7&eq83#%s77dJRJx|`yO%+!6l*LoS=i_#-i(gw4&^SkxE zfB`2qtQbYN-ALfEiP6S^z~dqvsEpE1SEs{t2S|$o!F4}sAkpZ>4C5_a5LR0!C-aIc z8OUzhcim!)-v+7)1vZW;LNB-dcI^1#Ik*tfVFDl?yM=9sX@bVKWBJwxBW;(0czqlK zt8mA@BtRUmirD^FcmqhRP}C*_G|t)iw}?gnHvtuSUKyNIyac{BxJKm+a(r$^dfwca(Mi32HIhBI~WN&*0?fI+4Kxm zpw-ih^Ecbn^}Q8gdbe|834Y>B`w~SKqPxSyCl08zVLAl%d<%_s{8$ZzUg6q+KJ*%) zp9x#=J{+Ya=Yxj4jIofF>#LM6eFI4-1dtd$^LXgh7ew`bS;p&3r}Z{tZDU2pzU2y5J7bxpM-7B{-zh>ZyHYo z8-a9>=s5<|6b<`&y`&w6WT?CKj{}1eaL2}O zPmN9_G-_?rC8Kj94Oqn?oK|hbZx(#8YhXkS5S;z!6j>vl`6wL7oy}FT`r5`r*aTR# zB{D>C@ZS1@XfRT)ncIO@AP|=mQlu=uGCy(zFcrORl2VPVkUwdPK0LDhsWR0Jm%dp* zN8JR>3Rz?uj?0LL0?7q(a(3XRq$ii`BW8D~aJQfH?f>9|&=XGHjvv>6D-UHqLhBmv z5GAlN_kO25(PNB&R9c{#MhW92<=|0+5;m}s3p&zuqk(IEf)eNH@sFr0|GN`(XqF_D>J!q&|ykI~t z?@Fs8KdTc+V*Q{}Kc*xwK!>4UXj7MmMJUm&ZEjWQtjp4@fTtf|$xy+z#O#xzWvw-I z+LL#2vqzx|0?eT~y&}5M_K790Emu7&2%lvE9$#+p;nxbUL3Zv1X*Nj44+@#;r9Xa5 zuJfDIxA;wSP52I9{w;j@+sJD=vfs#Yp1q*6A~w(Kjoz*B{UB_glI+{%Roywk+@Y|q zJqi9Dtd`|$U$VxA4AR$x7B_Xmt3vYI4-Mh9*K3XLny0d_xw^TT0K*lM0I*N~>=Kn| zmWI&ek-t3ZZkYr4&IxzeI->-S+-cwu^TOQdrPra#WAL}F^iq}nw+{*#u-&W#Hpw%R z*9p@v2)f9!HR%Cfu)^r_J|s*GE8C{R-QNv%>EPPl zNKcM3_k(@FFC^3f&~By$+c<<8GzS|gLC66$j*;JGHY;lPlE?}VtAr7-iYQN4fo<3|_MKOESH z2j}#0CpCo<3r6NvvtqkEC5y+huyyO-W0%I(jt*uN+yzxs^u~|SBIj-I!)$oJkjK;l z+hz-6mFHhpSlrYk43IkN1wHURNE}AhK|E5d_YgAFJlZT_0-s6+wm3b7rS2=Vi6Mie z#7GsB2S~-Lbp^@r0YC{4Zr*{s?Z5kox69t)cy4<9&d2|-@G&RouOA&h25JD8X9q}< zO1~=v&uX#2{rM6Khhl-)MkbJw=7P1+G4286; zm&%Nk3hV&y#HD0BYz5Nb+ztbt4rlSIFGxQC5NgP?q3v=Hsx26ws17vbYIfuVrVHru zN&r(J3Lg4HYK1rp!w?-MoY0jdlo-WwZs76YoH{+_D3nsyLV|i}!&*Q75j6Fe@1!beSLA6({|-Ta{z?El%RlCwaCt@i;D0U* zHL>-3P1Ljl_71aA43U}V*HC~Ju3rH3Pmub)XFYk+25qVS_wOA|vp6U>)rI;Rs_3yX zyAwN;W>$my7--CP^Kkq?`oe2tDz#u+`E|3S*Rl99P%?s*lE6qaM+K?57s4>T5Hk1F zeH{*j#>!#T;!iTj%o42lfDe0w|77qxYZl61UQE~#;}&m0HyM2R$>%#1CQU;9LY>sq zyZ3Nf34%vlMF?HH#xmsMaJnttPp%+;n8|LoV;WSLi5JRS(jM4;+Qr!EeT<2+`>Fb`**h%9$d*l;pOV!UD`2!6b-Mssif+(8yiqiv-@*x}pq$`? zPG3t;zE=UM7TT0-xIp+4pN?iv(mR;xK)E&mTO}>KBEsCnGX74i-X7Sf{4h*=?>vK9 zJ>Q$26e(|*wj#X^Qib6Ll!5^G%516!omVx%YgI-x?+4rHO}fy*yg&b*<@NGDTd(Hb zWo)j^Fm4B0eR!beeUKH{`8!;{_CsmmCw{`=WsaqY4ab`*tkol#+{Y3Rw_~`^rw`q7 zQdQggao7H*GWX1$-h1z(M|m>MRLRpa)o*61cj2&s^_lbZl57Uft%nnCESGX+D3xt@ zaEK^Be0WLrT$Rg93#rw_@daydYM_6BtD9wAw$C(GY%Hxd{HniO*xFh>Uyb>j^8b9I z&$Cx?{60vwd8ll0h!eG51wOW^rKRGHo+Bifk2?&9qjRh=Bvb6km_C0XC#Q4VgzUjn zjfC^eG&9G>q&_Bky7Y_;tJ+&r$%PFl8C57~5(8IzW@rPOs@K<9?@M};e9`B3moi^k zm_%*UUf5K7>glV@Q>iX+_ZixBf}>f(xe%n#_Xzgp?ksv&2Wka0j)9OAk)ED6-I<>P zL%-lqJN*HCwP$N4frQL7Pt`nS-RY|9M=qQPFbutRp%DH)TNcC`qON+aB{yFjU{BB8 ztj5{YC17(sFDXHJv=iZH<+YUrr$LAmbit#5jxMK><;yb4=_O2KbAG#+lxS?=N7TLK zz$CbKOG-xO3g|z792DPqHBkhub`%+aoaKpqNM~8J{f(lh7a5^6qgS;MydSat&L)YlzX-_X|`(j*$L?XJE?~S&fH1 zDrj@I0Ax+Q>nC@`M81yg!%sJT+--VnU2 z^V_h*R@0*|ua*bIa1s0H{!UI-0NVpZI^y8YhWb}6;jJAg0fG5jDd3@jTxVs-*0&|h z(r`G9J=^?VzC^h3y+OX@)>MjR@PWs%;7!p7ugEVLS5((?6K@kIcY6#+%HNOaZh3cq z2V43v-Wk;GAPAB{K^$PNuJx$@d_iCXMuEgZ6Vg9d-{&G{4d41BxsD}iIZ_U8l9zH0 zKXHEyCNTwidwZoXm6D4})u5yM2C$SLKW{a7y=1v`CC>ZfdrQfbEG+lbAGY=654S;m zw-fYT(^LbB10rfA_*p@@3}v}yRz<{Y?x(GNW!EFAVrm(OW1Q|M_4Q?IW@p@dt$d{D z-a+-?Wbk*AKYpCzo2o(Gi4lvusDmM{lFuqV?!uVc+Z%$MK6K#5c25(b?f@q$o<8~# z4aayn*TksHk|~c)eo?)rZL$7`h4P>MSx~rF9Q8Hp5QzexrH+Y2O`*PcK{EGH+k@O- z4m}b#rdB90S;_MU%rT>WqZMurmBC3jL6rh^?Sb-eX(3qAD+AU-p945P!~g~d#$x}u z$53oT(gb`ZAT6_jp!KK8n}R^O4}^~Z{riH(#!N8biH7}J&92G1&AEO~=0pBNQ7ju- zmQbM<3QZSSNde*>X!onFjrag4-H_$1--I~FY}_5#*W8)>U%Fp8fgdn{G}%p<>^4D`TFM<-GrYmu**O~bZIilsH1{B zC*rTY=*e-`?C1@5*qYqLS3}DeL=D%hYP=4El#@J(l<-Se^3?I;WNst1X5jIEA6-30 zVfmVCd8PdPVx|Kf%aetqVp1j(6Vn&yD*z-y2pBx>RWCPPFLalb|E%lg)I8Z*@X^^@ zM>RFGLF}>fVH2o%d;g-D0n+3kT|3<0-`v$j7%uqHTs*mxAeY}cA6YajHPE?snsRVy zVHh?r*nCQWe}!hL2o%Wnf!}tc0mR@AwX@3ny(vM@Kn}Zn*h8D4lY<+aL_nSw8n!xp z-}P`niUv>9Z@{QnTG%|HpuA$k(PG+ThCRWNL9?6o&4&4z&m|5<03`)Ph+g*fO@=D$ z5ey_B8%xirm zoayP)xIC`)Jv2A1NWK|xM0pDJ$0NjW#(d3Q`S#{XpmPDMp&#T`pPLmlC>iV-x_j{P zACd)PY#+U{w6oufiQTeM`-+>JnS(BjgLU#HsM zVut$^9YmM3xuOJ+5`gtkypgJ_75iJa*#ooXrXR#Ulk~I4mh)t+>>Q_9HGYEGAW)t? z3&Gn1?j7*uf+m(pHLy(J{p_p3^ZL$f4WsNP`!=ya;gcOA+gEZgHsD+se&6gfF>3Hh zdGrYUl5lXYdn~tvpg9X^GjwBkjG`;M7w9!^#t8%{pCnxb=v@lmiPv6QP<8HLMQ4=< zNEalfeMgqnBmwM&3H22CP~s{u4I=L`JAIeq>BTxuqBNz5oceuKQBg4!ifWj3ezfe* z11xv2H&#qrx@$tI=y+Uj$8PYv@(}iVX^J|xb)l-yv_9N~(5ci9BEh(@A?Ve%g3BQ@ zV`Q}8cOl9LmFTmQ2EPrZun!~Wzip*PQ-xt;KY#wLt}Y$GbHLy17Mp!GSKJ|#0`UcG zGb2+S3=2tP@*~I~ zN(+T55c&x177L&%Ag_@{yVbW_tG2} zxql6$v0nGpWl}*C8&qdrA1=!v_mFXod)1H;x417!g-J*Kj$vw9*Z0!A4}B90fbJ+L zU6|U~T!45E+(*xQd!E?Y3EN%HZu_|zMp1Bnd-mA7))YKBPn7BBgbD$#^(1?nD!J1CxtYZ79lL`m<2BGWa&~iztSa~?OW z`ov{r6JZb(vZ?XXqmn#(F|3Rrgc5xN#v99am0TbRhLn+37EX2i5UxpI#ukM(v4*mr z55(0xeToO=csUp{7_u-mUCt+^1AGbne`t?&a;X6h+pzhwC_4F~${;`OEJdpBj1h?e zlg1p1WUM!4DP5?l6IE5sfQBe6nS`X|VR(v>k&&TCNl89qYtZqwz14m1m(eq00yNYB zuYh8>?8b8Vxp8>M&f*}x;#FLnlO^vc({>L(cYr}4se#M=8J2wCB{3y4(;z5F85#+1 z(@(`(Rk)f!cA%@qlNIP5RDd zVPTg0S6{vH^M-2e0E&k*lzUfrCp<1R)> z_~p1+FL+{ued*N9d|V!4Q#HjRLKLif4C=R#96Bl9(tX?JgN-v_z1_{9^pVG&bb?4| z*r!Ulm6Xxh+tUM{HmT4s0f8rhg9k<7i|@TR5{T44h#yK@u`ahFk-8{d**aA!;kn+& zg4JgxCm$pwwGItkk|t&xUc1}p%-z7@t|^qVfSbHA^v6mu(0lvEix)$86X4gkeDk6P z|E6Wt^VO%Xx!GrGNv2p~uTU51ZEt8~Wo91Cc&{KQhsj7*P?)y*b864F{GqSmwg%$F zYnR7&A8k}Qw-U_0i+)#RGj3R?Z9>xHi|fnH+;vn#6nq=6K%kbT&P_vU5%#f`uDC@uYL)3`WRBASt+ zB4zq@V-erf-1=$jSg?T24a&I=exnPB#MG!d9f~fWzl2W zV>uw9gpQOdPnO3$tS85A))(iWz;rWd5Hha@G0cWE-R279T(A1wVyf4E0cXLG-JT%( zq-mo#yaR7QR_}Nm59@wWg^Re4T!81G)7FUkq7mzGvr1*pgW$j zT~KDO`1(qm^7nw_dRO)u{4V^(MZ8BA$0rgsif(hztd(9 zJWzV*UI7QgRvAN!rd)@I(YM-^^m1&4?_bV9MTMn|7;|l=QmM5pL}3moH+9qQsei|G zjOpjQAUnimi`Ulaog>r5aK10>+T(L^A*3;h<%3=fRpsvt$a_r#Mmd}<2XE4aWnx@{)_xAO43fb zWt7+U>($Ws?16|PT|3np4E3hpiq+)Rn0G}Vlk>Xm_wixUVhuKRs*Aiz@-FcGo|dVb zAFw~57jyx4w3Ol4T!iS**uif0<4yMs7N`tWm$y8BQiDiboN2!1Z|1n+jy7Mnp|6Vd zd00z)*~*`*_*hT$JWj7vbo^%9AB91N3R|U>T{N!W>7xld@ne_c1pK}Y=}AsKCY+>O zq_?1p*CMe*&9{U#&9|`c*L02(@c&k$_uPljMH=dMPwHeEtx2FWq-(k|u>LebTp~pU zYkD`65X|i#(Y3E;Fy^SreVTMtj&bZUMw1@(bgH>7hUh^>KB{?WzBhg9GO+c7x9yRq zx Date: Fri, 5 Jul 2024 10:18:57 +0200 Subject: [PATCH 06/71] Adds a first version of the PgVectorConnection. --- config-schema.json | 2 +- settings.gradle | 2 +- .../database/AbstractConnectionProvider.kt | 5 +- .../descriptors/AbstractDescriptorWriter.kt | 2 +- vitrivr-engine-module-pgvector/build.gradle | 62 +++++ .../engine/database/pgvector/Constants.kt | 34 +++ .../database/pgvector/PgVectorConnection.kt | 94 +++++++ .../pgvector/PgVectorConnectionProvider.kt | 106 ++++++++ .../AbstractDescriptorInitializer.kt | 50 ++++ .../descriptor/AbstractDescriptorReader.kt | 243 ++++++++++++++++++ .../descriptor/AbstractDescriptorWriter.kt | 58 +++++ .../pgvector/descriptor/model/PgVector.kt | 124 +++++++++ .../struct/StructDescriptorInitializer.kt | 53 ++++ .../struct/StructDescriptorProvider.kt | 20 ++ .../struct/StructDescriptorReader.kt | 88 +++++++ .../struct/StructDescriptorWriter.kt | 158 ++++++++++++ .../vector/VectorDescriptorInitializer.kt | 33 +++ .../vector/VectorDescriptorProvider.kt | 19 ++ .../vector/VectorDescriptorReader.kt | 143 +++++++++++ .../vector/VectorDescriptorWriter.kt | 79 ++++++ .../retrievable/RetrievableInitializer.kt | 74 ++++++ .../pgvector/retrievable/RetrievableReader.kt | 136 ++++++++++ .../pgvector/retrievable/RetrievableWriter.kt | 182 +++++++++++++ vitrivr-engine-server/build.gradle | 1 + 24 files changed, 1764 insertions(+), 4 deletions(-) create mode 100644 vitrivr-engine-module-pgvector/build.gradle create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Constants.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorProvider.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorProvider.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt diff --git a/config-schema.json b/config-schema.json index ce812ba0..fc7f88b9 100755 --- a/config-schema.json +++ b/config-schema.json @@ -5,7 +5,7 @@ "connection": { "database": "CottontailConnectionProvider", "parameters": { - "Host": "127.0.0.1", + "host": "127.0.0.1", "port": "1865" } }, diff --git a/settings.gradle b/settings.gradle index db29d6cb..0ad7ee17 100644 --- a/settings.gradle +++ b/settings.gradle @@ -8,8 +8,8 @@ include 'vitrivr-engine-index' include 'vitrivr-engine-query' include 'vitrivr-engine-server' include 'vitrivr-engine-module-cottontaildb' +include 'vitrivr-engine-module-pgvector' include 'vitrivr-engine-module-features' include 'vitrivr-engine-module-m3d' include 'vitrivr-engine-module-fes' include 'vitrivr-engine-module-statistics' - diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/AbstractConnectionProvider.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/AbstractConnectionProvider.kt index 231dbb5b..a20582a2 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/AbstractConnectionProvider.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/AbstractConnectionProvider.kt @@ -43,5 +43,8 @@ abstract class AbstractConnectionProvider: ConnectionProvider { * @return The registered [DescriptorProvider] . */ @Suppress("UNCHECKED_CAST") - override fun obtain(descriptorClass: KClass): DescriptorProvider = this.registered[descriptorClass] as DescriptorProvider + override fun obtain(descriptorClass: KClass): DescriptorProvider { + val provider = this.registered[descriptorClass] ?: throw IllegalStateException("No DescriptorProvider registered for $descriptorClass.") + return provider as DescriptorProvider + } } \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt index 3e7e030c..fa6ab4da 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt @@ -20,7 +20,7 @@ private val logger: KLogger = KotlinLogging.logger {} /** * An abstract implementation of a [DescriptorWriter] for Cottontail DB. - + * * @author Ralph Gasser * @version 1.0.0 */ diff --git a/vitrivr-engine-module-pgvector/build.gradle b/vitrivr-engine-module-pgvector/build.gradle new file mode 100644 index 00000000..ba047ee8 --- /dev/null +++ b/vitrivr-engine-module-pgvector/build.gradle @@ -0,0 +1,62 @@ +plugins { + id 'maven-publish' + id 'signing' +} + +repositories { + mavenCentral() +} + +dependencies { + api project(':vitrivr-engine-core') + + /** PostgreSQL driver. */ + implementation group: 'org.postgresql', name: 'postgresql', version: '42.7.3' +} + +/* Publication of vitrivr engine query to Maven Central. */ +publishing { + publications { + mavenJava(MavenPublication) { + groupId = 'org.vitrivr' + artifactId = 'vitrivr-engine-module-pgvector' + version = System.getenv().getOrDefault("MAVEN_PUBLICATION_VERSION", version.toString()) + from components.java + pom { + name = 'vitrivr Engine pgVector Plugin' + description = 'A plugin that adds connectivity to the PostgreSQL with pgVector database.' + url = 'https://github.com/vitrivr/vitrivr-engine/' + licenses { + license { + name = 'MIT License' + } + } + developers { + developer { + id = 'ppanopticon' + name = 'Ralph Gasser' + email = 'ralph.gasser@unibas.ch' + } + } + scm { + connection = 'scm:git:https://github.com/vitrivr/vitrivr-engine.git' + url = 'https://github.com/vitrivr/vitrivr-engine/' + } + } + } + } + repositories { + repositories { + maven { + def releasesRepoUrl = 'https://oss.sonatype.org/service/local/staging/deploy/maven2/' + def snapshotsRepoUrl = 'https://oss.sonatype.org/content/repositories/snapshots/' + name = "OSSRH" + url = version.endsWith('SNAPSHOT') ? snapshotsRepoUrl : releasesRepoUrl + credentials { + username = System.getenv("MAVEN_USERNAME") + password = System.getenv("MAVEN_PASSWORD") + } + } + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Constants.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Constants.kt new file mode 100644 index 00000000..3a05ef6d --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Constants.kt @@ -0,0 +1,34 @@ +package org.vitrivr.engine.database.pgvector + +/** The name of the retrievable entity. */ +const val RETRIEVABLE_ENTITY_NAME = "retrievable" + +/** The column name of a retrievable ID. */ +const val RETRIEVABLE_ID_COLUMN_NAME = "retrievableId" + +/** The column name of a retrievable ID. */ +const val RETRIEVABLE_TYPE_COLUMN_NAME = "type" + +/** The name of the retrievable entity. */ +const val RELATIONSHIP_ENTITY_NAME = "relationships" + +/** The column name of a retrievable ID. */ +const val SUBJECT_ID_COLUMN_NAME = "subjectId" + +/** The column name of a retrievable ID. */ +const val OBJECT_ID_COLUMN_NAME = "objectId" + +/** The column name of a retrievable ID. */ +const val PREDICATE_COLUMN_NAME = "predicate" + +/** The prefix for descriptor entities. */ +const val DESCRIPTOR_ENTITY_PREFIX = "descriptor" + +/** The column name of a descriptor ID. */ +const val DESCRIPTOR_ID_COLUMN_NAME = "descriptorId" + +/** The column name of a descriptor ID. */ +const val DESCRIPTOR_COLUMN_NAME = "descriptor" + +/** The column name used to describe a distance.*/ +const val DISTANCE_COLUMN_NAME = "distance" \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt new file mode 100644 index 00000000..59717d23 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt @@ -0,0 +1,94 @@ +package org.vitrivr.engine.database.pgvector + +import io.github.oshai.kotlinlogging.KLogger +import io.github.oshai.kotlinlogging.KotlinLogging.logger +import org.postgresql.PGConnection +import org.vitrivr.engine.core.database.AbstractConnection +import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer +import org.vitrivr.engine.core.database.retrievable.RetrievableReader +import org.vitrivr.engine.core.database.retrievable.RetrievableWriter +import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector +import java.sql.Connection +import java.sql.SQLException + + +/** Defines [KLogger] of the class. */ +internal val LOGGER: KLogger = logger("org.vitrivr.engine.database.pgvector.PgVectorConnection") + +/** + * A [AbstractConnection] to connect to a PostgreSQL instance with the pgVector extension. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: String, internal val connection: Connection): AbstractConnection(schemaName, provider) { + + init { + /* Make sure that the pg_vector extension is installed. */ + try { + this.connection.prepareStatement("CREATE EXTENSION IF NOT EXISTS vector;").use { + it.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to create extension pg_vector due to exception." } + throw e + } + + /* Register the vector data type. */ + this.connection.unwrap(PGConnection::class.java).addDataType("vector", PgVector::class.java) + + /* Create necessary database. */ + try { + this.connection.prepareStatement("CREATE DATABASE $schemaName;").use { + it.execute() + } + } catch (e: SQLException) { + if (e.sqlState == "42P04") { + LOGGER.info { "Database '$schemaName' already exists." } + } else { + LOGGER.error(e) { "Failed to create database '$schemaName' due to exception." } + throw e + } + } + } + + /** + * Generates and returns a [RetrievableInitializer] for this [PgVectorConnection]. + * + * @return [RetrievableInitializer] + */ + override fun getRetrievableInitializer(): RetrievableInitializer + = org.vitrivr.engine.database.pgvector.retrievable.RetrievableInitializer(this.connection) + + /** + * Generates and returns a [RetrievableWriter] for this [PgVectorConnection]. + * + * @return [RetrievableWriter] + */ + override fun getRetrievableWriter(): RetrievableWriter + = org.vitrivr.engine.database.pgvector.retrievable.RetrievableWriter(this.connection) + + /** + * Generates and returns a [RetrievableWriter] for this [PgVectorConnection]. + * + * @return [RetrievableReader] + */ + override fun getRetrievableReader(): RetrievableReader + = org.vitrivr.engine.database.pgvector.retrievable.RetrievableReader(this.connection) + + /** + * Returns the human-readable description of this [PgVectorConnection]. + */ + override fun description(): String = this.connection.toString() + + /** + * Closes this [PgVectorConnection] + */ + override fun close() { + try { + this.connection.close() + } catch (e: SQLException) { + LOGGER.warn(e) { "Failed to close database connection due to exception." } + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt new file mode 100644 index 00000000..b41be7aa --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt @@ -0,0 +1,106 @@ +package org.vitrivr.engine.database.pgvector + +import org.vitrivr.engine.core.database.AbstractConnectionProvider +import org.vitrivr.engine.core.database.Connection +import org.vitrivr.engine.core.database.ConnectionProvider +import org.vitrivr.engine.core.database.descriptor.DescriptorProvider +import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.RasterDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.SkeletonDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.metadata.MediaDimensionsDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.metadata.Rectangle2DMetadataDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.metadata.TemporalMetadataDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.FileSourceMetadataDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.MapStructDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.VideoSourceMetadataDescriptor +import org.vitrivr.engine.core.model.descriptor.vector.* +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.database.pgvector.descriptor.struct.StructDescriptorProvider +import org.vitrivr.engine.database.pgvector.descriptor.vector.VectorDescriptorProvider +import java.sql.DriverManager +import java.util.* + + +/** + * Implementation of the [ConnectionProvider] interface for PostgreSQL with the PGVector extension. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class PgVectorConnectionProvider: AbstractConnectionProvider() { + + companion object { + /** Name of the host parameter. */ + const val PARAMETER_NAME_HOST = "host" + + /** Name of the host parameter. */ + const val PARAMETER_DEFAULT_HOST = "127.0.0.1" + + /** Name of the port parameter. */ + const val PARAMETER_NAME_PORT = "port" + + /** Name of the host parameter. */ + const val PARAMETER_DEFAULT_PORT = 5432 + + /** Name of the host parameter. */ + const val PARAMETER_NAME_USERNAME = "username" + + /** Name of the host parameter. */ + const val PARAMETER_NAME_PASSWORD = "password" + + /** Name of the host parameter. */ + const val PARAMETER_NAME_SSL = "ssl" + } + + /** The name of this [PgVectorConnectionProvider]. */ + override val databaseName: String = "PostgreSQL with pgVector" + + /** The version of this [PgVectorConnectionProvider]. */ + override val version: String = "1.0.0" + + /** + * This method is called during initialization of the [PgVectorConnectionProvider] and can be used to register [DescriptorProvider]s. + */ + override fun initialize() { + /* Vector descriptors. */ + this.register(BooleanVectorDescriptor::class, VectorDescriptorProvider) + this.register(IntVectorDescriptor::class, VectorDescriptorProvider) + this.register(LongVectorDescriptor::class, VectorDescriptorProvider) + this.register(FloatVectorDescriptor::class, VectorDescriptorProvider) + this.register(DoubleVectorDescriptor::class, VectorDescriptorProvider) + + /* Struct descriptor. */ + this.register(LabelDescriptor::class, StructDescriptorProvider) + this.register(FileSourceMetadataDescriptor::class, StructDescriptorProvider) + this.register(VideoSourceMetadataDescriptor::class, StructDescriptorProvider) + this.register(TemporalMetadataDescriptor::class, StructDescriptorProvider) + this.register(Rectangle2DMetadataDescriptor::class, StructDescriptorProvider) + this.register(MediaDimensionsDescriptor::class, StructDescriptorProvider) + this.register(SkeletonDescriptor::class, StructDescriptorProvider) + this.register(RasterDescriptor::class, StructDescriptorProvider) + this.register(MapStructDescriptor::class, StructDescriptorProvider) + } + + /** + * Opens a new [PgVectorConnection] for the given [Schema]. + * + * @param parameters The optional parameters. + * @return [Connection] + */ + override fun openConnection(schemaName: String, parameters: Map): Connection { + + /* Prepare connection URL. */ + val host = parameters.getOrDefault(PARAMETER_NAME_HOST, PARAMETER_DEFAULT_HOST) + val port = parameters[PARAMETER_NAME_PORT]?.toInt() ?: PARAMETER_DEFAULT_PORT + val url = "jdbc:postgresql://${host}:${port}/" + + /* Prepare properties (optional). */ + val props = Properties() + parameters[PARAMETER_NAME_USERNAME]?.let { props.setProperty("user", it) } + parameters[PARAMETER_NAME_PASSWORD]?.let { props.setProperty("password", it) } + parameters[PARAMETER_NAME_SSL]?.let { props.setProperty("ssl", it) } + + /* Open JDBC connection and return PgVectorConnection. */ + return PgVectorConnection(this, schemaName, DriverManager.getConnection(url, props)) + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt new file mode 100644 index 00000000..adca6fe4 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt @@ -0,0 +1,50 @@ +package org.vitrivr.engine.database.pgvector.descriptor + +import org.vitrivr.engine.core.database.descriptor.DescriptorInitializer +import org.vitrivr.engine.core.model.descriptor.Descriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ENTITY_PREFIX +import org.vitrivr.engine.database.pgvector.LOGGER +import java.sql.Connection +import java.sql.SQLException + +/** + * An abstract implementation of a [DescriptorInitializer] for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractDescriptorInitializer(final override val field: Schema.Field<*, D>, protected val connection: Connection): DescriptorInitializer { + + /** The name of the table backing this [AbstractDescriptorInitializer]. */ + protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" + + /** + * Checks if the schema for this [AbstractDescriptorInitializer] has been properly initialized. + * + * @return True if entity has been initialized, false otherwise. + */ + override fun isInitialized(): Boolean { + try { + this.connection.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $tableName").use { + it.execute() + } + } catch (e: SQLException) { + return false + } + return true + } + + /** + * Truncates the table backing this [AbstractDescriptorInitializer]. + */ + override fun truncate() { + try { + this.connection.prepareStatement(/* sql = postgres */ "TRUNCATE $tableName").use { + it.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to truncate entities due to exception." } + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt new file mode 100644 index 00000000..4f0c2a3c --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt @@ -0,0 +1,243 @@ +package org.vitrivr.engine.database.pgvector.descriptor + +import org.vitrivr.engine.core.database.descriptor.DescriptorReader +import org.vitrivr.engine.core.model.descriptor.Descriptor +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.query.Query +import org.vitrivr.engine.core.model.retrievable.RetrievableId +import org.vitrivr.engine.core.model.retrievable.Retrieved +import org.vitrivr.engine.database.pgvector.* +import java.sql.PreparedStatement +import java.sql.ResultSet +import java.util.* + +/** + * An abstract implementation of a [DescriptorReader] for Cottontail DB. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractDescriptorReader(final override val field: Schema.Field<*, D>, protected val connection: PgVectorConnection) : DescriptorReader { + + /** The name of the table backing this [AbstractDescriptorReader]. */ + protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" + + + /** The [Descriptor] prototype handled by this [AbstractDescriptorReader]. */ + protected val prototype: D= this.field.analyser.prototype(this.field) + + /** + * Returns a single [Descriptor]s of type [D] that has the provided [DescriptorId]. + * + * @param descriptorId The [DescriptorId], i.e., the [UUID] of the [Descriptor] to return. + * @return [Sequence] of all [Descriptor]s. + */ + override fun get(descriptorId: DescriptorId): D? { + try { + this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?").use { stmt -> + stmt.setObject(1, descriptorId) + stmt.executeQuery().use { res -> + if (res.next()) { + return this.rowToDescriptor(res) + } else { + return null + } + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to check fetch descriptor $descriptorId from '$tableName' due to SQL error." } + return null + } + } + + /** + * Returns the [Descriptor]s of type [D] that belong to the provided [RetrievableId]. + * + * @param retrievableId The [RetrievableId] to search for. + * @return [Sequence] of [Descriptor] of type [D] + */ + override fun getFor(retrievableId: RetrievableId): Sequence { + try { + this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> + stmt.setObject(1, retrievableId) + val result = stmt.executeQuery() + return sequence { + result.use { + while (result.next()) { + yield(this@AbstractDescriptorReader.rowToDescriptor(result)) + } + } + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to fetch descriptor for retrievable $retrievableId from '$tableName' due to SQL error." } + return emptySequence() + } + } + + /** + * Checks whether a [Descriptor] of type [D] with the provided [UUID] exists. + * + * @param descriptorId The [DescriptorId], i.e., the [UUID] of the [Descriptor] to check for. + * @return True if descriptor exsits, false otherwise + */ + override fun exists(descriptorId: DescriptorId): Boolean { + try { + this.connection.connection.prepareStatement("SELECT count(*) FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?").use { stmt -> + stmt.setObject(1, descriptorId) + stmt.executeQuery().use { res -> + res.next() + return res.getLong(1) > 0L + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to check for descriptor $descriptorId in '$tableName' due to SQL error." } + return false + } + } + + /** + * Returns all [Descriptor]s of type [D] as a [Sequence]. + * + * @return [Sequence] of all [Descriptor]s. + */ + override fun getAll(): Sequence { + try { + this.connection.connection.prepareStatement("SELECT * FROM $tableName").use { stmt -> + val result = stmt.executeQuery() + return sequence { + result.use { + while (result.next()) { + yield(this@AbstractDescriptorReader.rowToDescriptor(result)) + } + } + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to fetch descriptors from '$tableName' due to SQL error." } + return emptySequence() + } + } + + /** + * Returns a [Sequence] of all [Descriptor]s whose [DescriptorId] is contained in the provided [Iterable]. + * + * @param descriptorIds A [Iterable] of [DescriptorId]s to return. + * @return [Sequence] of [Descriptor] of type [D] + */ + override fun getAll(descriptorIds: Iterable): Sequence { + try { + this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ANY (?)").use { stmt -> + val values = descriptorIds.map { it }.toTypedArray() + stmt.setArray(1, this.connection.connection.createArrayOf("uuid", values)) + val result = stmt.executeQuery() + return sequence { + result.use { + while (result.next()) { + yield(this@AbstractDescriptorReader.rowToDescriptor(result)) + } + } + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to fetch descriptors from '$tableName' due to SQL error." } + return emptySequence() + } + } + + /** + * Returns a [Sequence] of all [Descriptor] whose [RetrievableId] is contained in the provided [Iterable]. + * + * @param retrievableIds A [Iterable] of [RetrievableId]s to return [Descriptor]s for + * @return [Sequence] of [Descriptor] of type [D] + */ + override fun getAllFor(retrievableIds: Iterable): Sequence { + try { + this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?)").use { stmt -> + val values = retrievableIds.map { it }.toTypedArray() + stmt.setArray(1, this.connection.connection.createArrayOf("uuid", values)) + val result = stmt.executeQuery() + return sequence { + result.use { + while (result.next()) { + yield(this@AbstractDescriptorReader.rowToDescriptor(result)) + } + } + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to fetch descriptors from '$tableName' due to SQL error." } + return emptySequence() + } + } + + /** + * Returns the number of [Descriptor]s contained in the entity managed by this [AbstractDescriptorReader] + * + * @return [Sequence] of all [Descriptor]s. + */ + override fun count(): Long { + try { + this.connection.connection.prepareStatement("SELECT COUNT(*) FROM $tableName;").use { stmt -> + stmt.executeQuery().use { result -> + result.next() + return result.getLong(1) + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to count retrievable due to SQL error." } + return 0L + } + } + + /** + * Returns a [Sequence] of all [Retrieved]s that match the given [Query]. + * + * Implicitly, this methods executes a [query] and then JOINS the result with the [Retrieved]s. + * + * @param query The [Query] that should be executed. + * @return [Sequence] of [Retrieved]. + */ + override fun queryAndJoin(query: Query): Sequence { + val descriptors = query(query).toList() + if (descriptors.isEmpty()) return emptySequence() + + /* Fetch retrievable ids. */ + val retrievables = this.connection.getRetrievableReader().getAll(descriptors.mapNotNull { it.retrievableId }.toSet()).map { it.id to it }.toMap() + return descriptors.asSequence().mapNotNull { descriptor -> + val retrievable = retrievables[descriptor.retrievableId] + if (retrievable != null) { + retrievable.addDescriptor(descriptor) + retrievable as Retrieved + } else { + null + } + } + } + + + /** + * Converts a [PreparedStatement] to a [Sequence] of [Descriptor] of type [D]. + * + * @return [Sequence] of [Descriptor]s of type [D] + */ + protected fun PreparedStatement.executeAndStream(): Sequence = this.use { + val result = this.executeQuery() + return sequence { + result.use { + while (result.next()) { + yield(this@AbstractDescriptorReader.rowToDescriptor(result)) + } + } + } + } + + /** + * Converts a [ResultSet] to a [Descriptor] of type [D]. + * + * @param result The [ResultSet] to convert. + * @return [Descriptor] of type [D] + */ + abstract fun rowToDescriptor(result: ResultSet): D +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt new file mode 100644 index 00000000..6d295d5f --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt @@ -0,0 +1,58 @@ +package org.vitrivr.engine.database.pgvector.descriptor + +import org.vitrivr.engine.core.database.descriptor.DescriptorWriter +import org.vitrivr.engine.core.model.descriptor.Descriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ENTITY_PREFIX +import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.LOGGER +import org.vitrivr.engine.database.pgvector.PgVectorConnection +import java.sql.SQLException + +/** + * An abstract implementation of a [DescriptorWriter] for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractDescriptorWriter(final override val field: Schema.Field<*, D>, protected val connection: PgVectorConnection): DescriptorWriter { + /** The name of the table backing this [AbstractDescriptorInitializer]. */ + protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" + + /** + * Deletes (writes) a [Descriptor] of type [D] using this [AbstractDescriptorWriter]. + * + * @param item A [Descriptor]s to delete. + * @return True on success, false otherwise. + */ + override fun delete(item: D): Boolean { + try { + this.connection.connection.prepareStatement("DELETE FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?;").use { stmt -> + stmt.setObject(1, item.id) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to delete descriptor ${item.id} due to SQL error." } + return false + } + } + + /** + * Deletes (writes) [Descriptor]s of type [D] using this [AbstractDescriptorWriter]. + * + * @param items A [Iterable] of [Descriptor]s to delete. + * @return True on success, false otherwise. + */ + override fun deleteAll(items: Iterable): Boolean { + try { + this.connection.connection.prepareStatement("DELETE FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ANY (?);").use { stmt -> + val values = items.map { it.id }.toTypedArray() + stmt.setArray(1, this.connection.connection.createArrayOf("uuid", values)) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to delete descriptors due to SQL error." } + return false + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt new file mode 100644 index 00000000..a26db32c --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt @@ -0,0 +1,124 @@ +package org.vitrivr.engine.database.pgvector.descriptor.model + +import org.postgresql.util.ByteConverter +import org.postgresql.util.PGBinaryObject +import org.postgresql.util.PGobject +import org.vitrivr.engine.core.model.types.Value +import java.io.Serializable +import java.sql.SQLException +import java.util.* + +/** + * The [PgVector] class represents a vector in PostgreSQL. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class PgVector(private var vec: FloatArray? = null) : PGobject(), PGBinaryObject, Serializable { + + init { + this.type = "vector" + } + + /** + * Constructor + * + * @param number + * @param v list of numbers + */ + constructor(v: List) : this(FloatArray(v.size) { + v[it].toFloat() + }) + + /** + * Constructor + * + * @param number + * @param v list of numbers + */ + constructor(vector: List>) : this(FloatArray(vector.size) { + when (val v = vector[it]) { + is Value.Float-> v.value + is Value.Double-> v.value.toFloat() + is Value.Int-> v.value.toFloat() + is Value.Long-> v.value.toFloat() + else -> throw IllegalArgumentException("Could not convert $v to float.") + } + }) + + /** + * Constructor + * + * @param s text representation of a vector + * @throws SQLException exception + */ + constructor(s: String?) : this() { + setValue(s) + } + + /** + * Sets the value from a text representation of a vector + */ + @Throws(SQLException::class) + override fun setValue(s: String?) { + if (s == null) { + this.vec = null + } else { + val sp = s.substring(1, s.length - 1).split(",".toRegex()).dropLastWhile { it.isEmpty() }.toTypedArray() + this.vec = FloatArray(sp.size) + for (i in sp.indices) { + vec!![i] = sp[i].toFloat() + } + } + } + + /** + * Returns the text representation of a vector + */ + override fun getValue(): String? = this.vec?.contentToString()?.replace(" ", "") + + /** + * Returns the number of bytes for the binary representation + */ + override fun lengthInBytes(): Int = if (vec == null) 0 else 4 + vec!!.size * 4 + + /** + * Sets the value from a binary representation of a vector + */ + @Throws(SQLException::class) + override fun setByteValue(value: ByteArray, offset: Int) { + val dim: Short = ByteConverter.int2(value, offset) + val unused: Short = ByteConverter.int2(value, offset + 2) + if (unused != 0.toShort()) { + throw SQLException("expected unused to be 0") + } + + this.vec = FloatArray(dim.toInt()) { + ByteConverter.float4(value, offset + 4 + it * 4) + } + } + + /** + * Writes the binary representation of a vector + */ + override fun toBytes(bytes: ByteArray, offset: Int) { + if (vec == null) { + return + } + + // server will error on overflow due to unconsumed buffer + // could set to Short.MAX_VALUE for friendlier error message + ByteConverter.int2(bytes, offset, vec!!.size) + ByteConverter.int2(bytes, offset + 2, 0) + for (i in this.vec!!.indices) { + ByteConverter.float4(bytes, offset + 4 + i * 4, vec!![i]) + } + } + + /** + * Returns an array + * + * @return an array + */ + fun toArray(): FloatArray? = this.vec +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt new file mode 100644 index 00000000..c58569bb --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt @@ -0,0 +1,53 @@ +package org.vitrivr.engine.database.pgvector.descriptor.struct + +import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.types.Type +import org.vitrivr.engine.database.pgvector.* +import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorInitializer +import java.sql.SQLException + +/** + * A [AbstractDescriptorInitializer] implementation for [StructDescriptor]s. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, connection: PgVectorConnection) : AbstractDescriptorInitializer(field, connection.connection) { + override fun initialize() { + val statement = StringBuilder("CREATE TABLE IF NOT EXISTS $tableName(") + statement.append("$DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, ") + statement.append("$RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, ") + + /* Add columns for each field in the struct. */ + for (field in this.field.analyser.prototype(this.field).schema()) { + require(field.dimensions.size <= 1) { "Cottontail DB currently doesn't support tensor types."} + when (field.type) { + Type.STRING -> statement.append("\"${field.name}\" varchar(255), ") + Type.BOOLEAN -> statement.append("\"${field.name}\" boolean, ") + Type.BYTE -> statement.append("$\"{field.name}\" smallint, ") + Type.SHORT -> statement.append("\"${field.name}\" smallint, ") + Type.INT -> statement.append("\"${field.name}\" integer, ") + Type.LONG -> statement.append("\"${field.name}\" bigint, ") + Type.FLOAT -> statement.append("\"${field.name}\" real, ") + Type.DOUBLE -> statement.append("\"${field.name}\" double precision, ") + Type.DATETIME -> statement.append("\"${field.name}\" datetime, ") + } + } + + /* Finalize statement*/ + statement.append("PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), ") + statement.append("FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + + try { + /* Create 'retrievable' entity. */ + if (!this.connection.prepareStatement(/* sql = postgres */ statement.toString()).use { + it.execute() + }) { + LOGGER.warn { "Failed to initialize entity '$tableName'. It probably exists." } + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to initialize entity '$tableName' due to exception." } + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorProvider.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorProvider.kt new file mode 100644 index 00000000..478580bf --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorProvider.kt @@ -0,0 +1,20 @@ +package org.vitrivr.engine.database.pgvector.descriptor.struct + +import org.vitrivr.engine.core.database.Connection +import org.vitrivr.engine.core.database.descriptor.DescriptorProvider +import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.database.pgvector.PgVectorConnection + +/** + * A [DescriptorProvider] for [LabelDescriptor]s. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +object StructDescriptorProvider : DescriptorProvider { + override fun newInitializer(connection: Connection, field: Schema.Field<*, StructDescriptor>) = StructDescriptorInitializer(field, connection as PgVectorConnection) + override fun newReader(connection: Connection, field: Schema.Field<*, StructDescriptor>) = StructDescriptorReader(field, connection as PgVectorConnection) + override fun newWriter(connection: Connection, field: Schema.Field<*, StructDescriptor>) = StructDescriptorWriter(field, connection as PgVectorConnection) +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt new file mode 100644 index 00000000..393498f9 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt @@ -0,0 +1,88 @@ +package org.vitrivr.engine.database.pgvector.descriptor.struct + +import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.query.Query +import org.vitrivr.engine.core.model.query.bool.SimpleBooleanQuery +import org.vitrivr.engine.core.model.query.fulltext.SimpleFulltextQuery +import org.vitrivr.engine.core.model.retrievable.Retrieved +import org.vitrivr.engine.core.model.types.Type +import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.LOGGER +import org.vitrivr.engine.database.pgvector.PgVectorConnection +import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorReader +import java.sql.ResultSet +import java.util.* +import kotlin.reflect.full.primaryConstructor + +/** + * An [AbstractDescriptorReader] for [LabelDescriptor]s. + * + * @author Ralph Gasser + * @version 1.1.0 + */ +class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connection: PgVectorConnection) : AbstractDescriptorReader(field, connection) { + /** + * Executes the provided [Query] and returns a [Sequence] of [Retrieved]s that match it. + * + * @param query The [Query] to execute. + * @return [Sequence] of [StructDescriptor]s that match the query. + */ + override fun query(query: Query): Sequence { + try { + /* Prepare statement based on query. */ + val statement = when (query) { + is SimpleFulltextQuery -> TODO() + is SimpleBooleanQuery<*> -> { + require(query.attributeName != null) { "Boolean query on a struct field requires specification of a field's attribute name." } + this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE ${query.attributeName} = ?;").apply { + setString(1, query.value.toString()) + } + } + else -> throw IllegalArgumentException("Query of typ ${query::class} is not supported by StringDescriptorReader.") + } + + /* Execute statement and return it. */ + return statement.executeAndStream() + } catch (e: Exception) { + LOGGER.error(e) { "Failed to execute query on '$tableName' due to SQL error." } + return emptySequence() + } + } + + /** + * Converts the provided [ResultSet] to a [StructDescriptor]. + * + * @param result The [ResultSet] to convert. + * @return The resulting [StructDescriptor]. + */ + override fun rowToDescriptor(result: ResultSet): StructDescriptor { + val constructor = this.field.analyser.descriptorClass.primaryConstructor ?: throw IllegalStateException("Provided type ${this.field.analyser.descriptorClass} does not have a primary constructor.") + val parameters: MutableList = mutableListOf( + result.getObject(DESCRIPTOR_ID_COLUMN_NAME, UUID::class.java) ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'."), + result.getObject(DESCRIPTOR_ID_COLUMN_NAME, UUID::class.java) ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'."), + ) + + /* Append dynamic parameters of struct. */ + for (field in this.prototype.schema()) { + parameters.add( + when(field.type) { + Type.STRING -> result.getString(field.name) + Type.BOOLEAN -> result.getBoolean(field.name) + Type.BYTE -> result.getByte(field.name) + Type.SHORT -> result.getShort(field.name) + Type.INT -> result.getInt(field.name) + Type.LONG -> result.getLong(field.name) + Type.FLOAT -> result.getFloat(field.name) + Type.DOUBLE -> result.getDouble(field.name) + Type.DATETIME -> result.getDate(field.name).toLocalDate() + } + ) + } + + /* Call constructor. */ + return constructor.call(*parameters.toTypedArray()) + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt new file mode 100644 index 00000000..3f672349 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt @@ -0,0 +1,158 @@ +package org.vitrivr.engine.database.pgvector.descriptor.struct + +import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.types.Value +import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.LOGGER +import org.vitrivr.engine.database.pgvector.PgVectorConnection +import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorWriter +import java.sql.Date +import java.sql.PreparedStatement +import java.sql.SQLException + +/** + * An [AbstractDescriptorWriter] for [StructDescriptor]s. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connection: PgVectorConnection) : AbstractDescriptorWriter(field, connection) { + + /** + * Adds (writes) a single [StructDescriptor] using this [StructDescriptorWriter]. + * + * @param item The [StructDescriptor] to write. + * @return True on success, false otherwise. + */ + override fun add(item: StructDescriptor): Boolean { + try { + this.prepareInsertStatement().use { stmt -> + stmt.setAny(1, item.id) + stmt.setAny(2, item.retrievableId) + for ((i, v) in item.values().withIndex()) { + stmt.setAny(3 + i, v) + } + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to INSERT descriptor ${item.id} into '$tableName' due to SQL error." } + return false + } + } + + /** + * Adds (writes) a batch of [StructDescriptor] using this [StructDescriptorWriter]. + * + * @param items A [Iterable] of [StructDescriptor]s to write. + * @return True on success, false otherwise. + */ + override fun addAll(items: Iterable): Boolean { + try { + this.prepareInsertStatement().use { stmt -> + for (item in items) { + stmt.setAny(1, item.id) + stmt.setAny(2, item.retrievableId) + for ((i, v) in item.values().withIndex()) { + stmt.setAny(3 + i, v) + } + stmt.addBatch() + } + return stmt.executeBatch().all { it == 1 } + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to INSERT descriptors into '$tableName' due to SQL error." } + return false + } + } + + /** + * Updates a specific [StructDescriptor] using this [StructDescriptorWriter]. + * + * @param item A [StructDescriptor]s to update. + * @return True on success, false otherwise. + */ + override fun update(item: StructDescriptor): Boolean { + try { + this.prepareUpdateStatement().use { stmt -> + val values = item.values() + stmt.setObject(1, item.retrievableId) + for ((i, v) in values.withIndex()) { + stmt.setAny(2 + i, v) + } + stmt.setObject(1 + values.size, item.id) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to UPDATE descriptors in '$tableName' due to SQL error." } + return false + } + } + + /** + * Sets a value of [Value] type in a [PreparedStatement]. + */ + private fun PreparedStatement.setValue(index: Int, value: Value<*>) { + when(value) { + is Value.Boolean -> this.setBoolean(index, value.value) + is Value.Byte -> this.setByte(index, value.value) + is Value.DateTime -> this.setDate(index, Date(value.value.toInstant().toEpochMilli())) + is Value.Double -> this.setDouble(index, value.value) + is Value.Float -> this.setFloat(index, value.value) + is Value.Int -> this.setInt(index, value.value) + is Value.Long -> this.setLong(index, value.value) + is Value.Short -> this.setShort(index, value.value) + is Value.String -> this.setString(index, value.value) + } + } + + /** + * Sets a value of [Any] type in a [PreparedStatement]. + */ + private fun PreparedStatement.setAny(index: Int, value: Any?) { + when (value) { + is String -> this.setString(index, value) + is Int -> this.setInt(index, value) + is Long -> this.setLong(index, value) + is Float -> this.setFloat(index, value) + is Double -> this.setDouble(index, value) + is Boolean -> this.setBoolean(index, value) + is ByteArray -> this.setBytes(index, value) + is Value<*> -> this.setValue(index, value) + else -> this.setObject(index, value) + } + } + + /** + * Prepares an INSERT statement for this [StructDescriptorWriter]. + * + * @return [PreparedStatement] + */ + private fun prepareUpdateStatement(): PreparedStatement { + val statement = StringBuilder("UPDATE $tableName SET $RETRIEVABLE_ID_COLUMN_NAME = ?") + for (field in this.field.analyser.prototype(this.field).schema()) { + statement.append(", \"${field.name}\" = ?") + } + statement.append("WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?;") + return this.connection.connection.prepareStatement(statement.toString()) + } + + /** + * Prepares an INSERT statement for this [StructDescriptorWriter]. + * + * @return [PreparedStatement] + */ + private fun prepareInsertStatement(): PreparedStatement { + val statement = StringBuilder("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME") + for (field in this.field.analyser.prototype(this.field).schema()) { + statement.append(", \"${field.name}\"") + } + statement.append(") VALUES (?, ?") + for (field in this.field.analyser.prototype(this.field).schema()) { + statement.append(", ?") + } + statement.append(");") + return this.connection.connection.prepareStatement(statement.toString()) + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt new file mode 100644 index 00000000..e8dfa895 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt @@ -0,0 +1,33 @@ +package org.vitrivr.engine.database.pgvector.descriptor.vector + +import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.database.pgvector.* +import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorInitializer +import java.sql.SQLException + +/** + * An [AbstractDescriptorInitializer] implementation for [VectorDescriptor]s. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class VectorDescriptorInitializer(field: Schema.Field<*, VectorDescriptor<*>>, connection: PgVectorConnection): AbstractDescriptorInitializer>(field, connection.connection) { + /** + * Initializes the [RetrievableInitializer]. + */ + override fun initialize() { + val type = this.field.analyser.prototype(this.field) + try { + /* Create 'retrievable' entity. */ + if (!this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME vector(${type.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));").use { + it.execute() + }) { + LOGGER.warn { "Failed to initialize entity '$tableName'. It probably exists." } + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to initialize entity due to exception." } + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorProvider.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorProvider.kt new file mode 100644 index 00000000..62cf473c --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorProvider.kt @@ -0,0 +1,19 @@ +package org.vitrivr.engine.database.pgvector.descriptor.vector + +import org.vitrivr.engine.core.database.Connection +import org.vitrivr.engine.core.database.descriptor.DescriptorProvider +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.database.pgvector.PgVectorConnection + +/** + * A [DescriptorProvider] for [VectorDescriptor]s. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +object VectorDescriptorProvider: DescriptorProvider> { + override fun newInitializer(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = VectorDescriptorInitializer(field, connection as PgVectorConnection) + override fun newReader(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = VectorDescriptorReader(field, connection as PgVectorConnection) + override fun newWriter(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = VectorDescriptorWriter(field, connection as PgVectorConnection) +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt new file mode 100644 index 00000000..92042598 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt @@ -0,0 +1,143 @@ +package org.vitrivr.engine.database.pgvector.descriptor.vector + +import org.vitrivr.engine.core.database.descriptor.DescriptorReader +import org.vitrivr.engine.core.model.descriptor.vector.* +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.query.Query +import org.vitrivr.engine.core.model.query.basics.Distance +import org.vitrivr.engine.core.model.query.basics.Distance.* +import org.vitrivr.engine.core.model.query.proximity.ProximityQuery +import org.vitrivr.engine.core.model.retrievable.Retrieved +import org.vitrivr.engine.core.model.retrievable.attributes.DistanceAttribute +import org.vitrivr.engine.core.model.types.toValue +import org.vitrivr.engine.database.pgvector.* +import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorReader +import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector +import java.sql.ResultSet +import java.util.* + +/** + * An abstract implementation of a [DescriptorReader] for Cottontail DB. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connection: PgVectorConnection) : AbstractDescriptorReader>(field, connection) { + /** + * Executes the provided [Query] and returns a [Sequence] of [Retrieved]s that match it. + * + * @param query The [Query] to execute. + */ + override fun query(query: Query): Sequence> { + try { + val statement = when (query) { + is ProximityQuery<*> -> this.connection.connection.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").apply { + setObject(1, PgVector(query.value)) + } + else -> throw UnsupportedOperationException("Query of typ ${query::class} is not supported by VectorDescriptorReader.") + } + return statement.executeAndStream() + } catch (e: Exception) { + LOGGER.error(e) { "Failed to execute query on '$tableName' due to SQL error." } + return emptySequence() + } + } + + /** + * Returns a [Sequence] of all [Retrieved]s that match the given [Query]. + * + * Implicitly, this methods executes a [query] and then JOINS the result with the [Retrieved]s. + * + * @param query The [Query] that should be executed. + * @return [Sequence] of [Retrieved]. + */ + override fun queryAndJoin(query: Query): Sequence { + try { + return when (query) { + is ProximityQuery<*> -> { + val descriptors = mutableListOf, Float>>() + this.connection.connection.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").use { stmt -> + stmt.setObject(1, PgVector(query.value)) + stmt.executeQuery().use { result -> + while (result.next()) { + descriptors.add(this@VectorDescriptorReader.rowToDescriptor(result) to result.getFloat(DISTANCE_COLUMN_NAME)) + } + } + } + + /* Fetch retrievable ids. */ + val retrievables = this.connection.getRetrievableReader().getAll(descriptors.mapNotNull { it.first.retrievableId }.toSet()).map { it.id to it }.toMap() + descriptors.asSequence().mapNotNull { (descriptor, distance) -> + val retrievable = retrievables[descriptor.retrievableId] + if (retrievable != null) { + retrievable.addDescriptor(descriptor) + retrievable.addAttribute(DistanceAttribute(distance)) + retrievable as Retrieved + } else { + null + } + } + } + else -> super.queryAndJoin(query) + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to execute query on '$tableName' due to SQL error." } + return emptySequence() + } + } + + /** + * Converts a [ResultSet] to a [VectorDescriptor]. + * + * @param result [ResultSet] to convert. + * @return [VectorDescriptor] + */ + override fun rowToDescriptor(result: ResultSet): VectorDescriptor<*> { + val descriptorId = result.getObject(DESCRIPTOR_ID_COLUMN_NAME, UUID::class.java) + val retrievableId = result.getObject(RETRIEVABLE_ID_COLUMN_NAME, UUID::class.java) + return when (this.prototype) { + is FloatVectorDescriptor -> FloatVectorDescriptor( + descriptorId, + retrievableId, + result.getObject(DESCRIPTOR_COLUMN_NAME, PgVector::class.java).let { vector -> + vector.toArray()?.map { it.toValue() } ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + } + ) + + is DoubleVectorDescriptor -> DoubleVectorDescriptor( + descriptorId, + retrievableId, + result.getObject(DESCRIPTOR_COLUMN_NAME, PgVector::class.java).let { vector -> + vector.toArray()?.map { it.toDouble().toValue() } ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + } + ) + + is IntVectorDescriptor -> IntVectorDescriptor( + descriptorId, + retrievableId, + result.getObject(DESCRIPTOR_COLUMN_NAME, PgVector::class.java).let { vector -> + vector.toArray()?.map { it.toInt().toValue() } ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + } + ) + + is LongVectorDescriptor -> LongVectorDescriptor( + descriptorId, + retrievableId, + result.getObject(DESCRIPTOR_COLUMN_NAME, PgVector::class.java).let { vector -> + vector.toArray()?.map { it.toLong().toValue() } ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + } + ) + + is BooleanVectorDescriptor -> TODO() + } + } + + /** + * Converts a [Distance] to a pgVector distance operator. + */ + private fun Distance.operator() = when(this) { + MANHATTAN -> "<+>" + EUCLIDEAN -> "<->" + COSINE -> "<=>" + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt new file mode 100644 index 00000000..9d62212a --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt @@ -0,0 +1,79 @@ +package org.vitrivr.engine.database.pgvector.descriptor.vector + +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.database.pgvector.* +import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorWriter +import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector +import java.sql.SQLException + +/** + * An [AbstractDescriptorWriter] for [VectorDescriptor]s. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connection: PgVectorConnection) : AbstractDescriptorWriter>(field, connection) { + /** + * Adds (writes) a single [VectorDescriptor] using this [VectorDescriptorWriter]. + * + * @param item The [VectorDescriptor] to write. + * @return True on success, false otherwise. + */ + override fun add(item: VectorDescriptor<*>): Boolean { + try { + this.connection.connection.prepareStatement("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $DESCRIPTOR_COLUMN_NAME) VALUES (?, ?, ?);").use { stmt -> + stmt.setObject(1, item.id) + stmt.setObject(2, item.retrievableId) + stmt.setObject(3, PgVector(item.vector)) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to persist descriptor ${item.id} due to SQL error." } + return false + } + } + + /** + * Adds (writes) a batch of [VectorDescriptor] using this [VectorDescriptorWriter]. + * + * @param items A [Iterable] of [VectorDescriptor]s to write. + * @return True on success, false otherwise. + */ + override fun addAll(items: Iterable>): Boolean { + try { + this.connection.connection.prepareStatement("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $DESCRIPTOR_COLUMN_NAME) VALUES (?, ?, ?);").use { stmt -> + for (item in items) { + stmt.setObject(1, item.id) + stmt.setObject(2, item.retrievableId) + stmt.setObject(3, PgVector(item.vector)) + stmt.addBatch() + } + return stmt.executeBatch().all { it == 1 } + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to persist descriptors due to SQL error." } + return false + } + } + + /** + * Updates a specific [VectorDescriptor] using this [VectorDescriptorWriter]. + * + * @param item A [VectorDescriptor]s to update. + * @return True on success, false otherwise. + */ + override fun update(item: VectorDescriptor<*>): Boolean { + try { + this.connection.connection.prepareStatement("UPDATE $tableName SET $RETRIEVABLE_ID_COLUMN_NAME = ?, $DESCRIPTOR_COLUMN_NAME = ? WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?;").use { stmt -> + stmt.setObject(1, item.retrievableId) + stmt.setObject(2, PgVector(item.vector)) + stmt.setObject(3, item.id) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to persist descriptors due to SQL error." } + return false + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt new file mode 100644 index 00000000..03dee51f --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt @@ -0,0 +1,74 @@ +package org.vitrivr.engine.database.pgvector.retrievable + +import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer +import org.vitrivr.engine.core.model.retrievable.Retrievable +import org.vitrivr.engine.database.pgvector.* +import java.sql.Connection +import java.sql.SQLException + +/** + * A [RetrievableInitializer] implementation for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +internal class RetrievableInitializer(private val connection: Connection): RetrievableInitializer { + /** + * Initializes the [RetrievableInitializer]. + */ + override fun initialize() { + try { + /* Create 'retrievable' entity. */ + if (!this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, type VARCHAR(100), PRIMARY KEY ($RETRIEVABLE_ID_COLUMN_NAME));").use { + it.execute() + }) { + LOGGER.warn { "Failed to initialize entity 'retrievable'. It probably exists." } + } + + /* Create 'relationship' entity. */ + if (!this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));").use { + it.execute() + }) { + LOGGER.warn { "Failed to initialize entity 'relationships'. It probably exists." } + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to initialize entity due to exception." } + } + } + + /** + * Checks if the schema for this [RetrievableInitializer] has been properly initialized. + * + * @return True if entity has been initialized, false otherwise. + */ + override fun isInitialized(): Boolean { + try { + this.connection.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $RETRIEVABLE_ENTITY_NAME").use { + it.execute() + } + } catch (e: SQLException) { + return false + } + try { + this.connection.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $RELATIONSHIP_ENTITY_NAME").use { + it.execute() + } + } catch (e: SQLException) { + return false + } + return true + } + + /** + * Truncates the entity that is used to store [Retrievable]s and Relationships in PostgreSQL with pgVector. + */ + override fun truncate() { + try { + this.connection.prepareStatement(/* sql = postgres */ "TRUNCATE $RETRIEVABLE_ENTITY_NAME, $RELATIONSHIP_ENTITY_NAME").use { + it.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to truncate entities due to exception." } + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt new file mode 100644 index 00000000..04156a8b --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt @@ -0,0 +1,136 @@ +package org.vitrivr.engine.database.pgvector.retrievable + +import org.vitrivr.engine.core.database.retrievable.RetrievableReader +import org.vitrivr.engine.core.model.retrievable.Retrievable +import org.vitrivr.engine.core.model.retrievable.RetrievableId +import org.vitrivr.engine.core.model.retrievable.Retrieved +import org.vitrivr.engine.database.pgvector.LOGGER +import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ENTITY_NAME +import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.RETRIEVABLE_TYPE_COLUMN_NAME +import java.sql.Connection +import java.util.* + +/** + * A [RetrievableReader] implementation for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class RetrievableReader(private val connection: Connection): RetrievableReader { + /** + * Returns the [Retrievable]s that matches the provided [RetrievableId] + */ + override fun get(id: RetrievableId): Retrievable? { + try { + this.connection.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> + stmt.setObject(1, id) + stmt.executeQuery().use { res -> + if (res.next() ) { + return Retrieved(res.getObject(RETRIEVABLE_ID_COLUMN_NAME, UUID::class.java), res.getString(RETRIEVABLE_TYPE_COLUMN_NAME), false) + } else { + return null + } + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to check for retrievable $id due to SQL error." } + return null + } + } + + /** + * Checks whether a [Retrievable] with the provided [RetrievableId] exists. + * + * @param id The [RetrievableId], i.e., the [UUID] of the [Retrievable] to check for. + * @return True if descriptor exists, false otherwise + */ + override fun exists(id: RetrievableId): Boolean { + try { + this.connection.prepareStatement("SELECT count(*) FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> + stmt.setObject(1, id) + stmt.executeQuery().use { res -> + res.next() + return res.getLong(1) > 0L + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to check for retrievable $id due to SQL error." } + return false + } + } + + /** + * Returns all [Retrievable]s that match any of the provided [RetrievableId] + * + * @param ids A [Iterable] of [RetrievableId]s to return. + * @return A [Sequence] of all [Retrievable]. + */ + override fun getAll(ids: Iterable): Sequence { + try { + this.connection.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?)").use { stmt -> + val values = ids.map { it }.toTypedArray() + stmt.setArray(1, connection.createArrayOf("uuid", values)) + val result = stmt.executeQuery() + return sequence { + while (result.next()) { + yield(Retrieved(result.getObject(RETRIEVABLE_ID_COLUMN_NAME, UUID::class.java), result.getString(RETRIEVABLE_TYPE_COLUMN_NAME), false)) + } + result.close() + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to check for retrievables due to SQL error." } + return emptySequence() + } + } + + /** + * Returns all [Retrievable]s stored by the database. + * + * @return A [Sequence] of all [Retrievable]s in the database. + */ + override fun getAll(): Sequence { + try { + this.connection.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME").use { stmt -> + val result = stmt.executeQuery() + return sequence { + while (result.next()) { + yield(Retrieved(result.getObject(RETRIEVABLE_ID_COLUMN_NAME, UUID::class.java), result.getString(RETRIEVABLE_TYPE_COLUMN_NAME), false)) + } + result.close() + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to check for retrievables due to SQL error." } + return emptySequence() + } + } + + override fun getConnections( + subjectIds: Collection, + predicates: Collection, + objectIds: Collection + ): Sequence> { + TODO("Not yet implemented") + } + + /** + * Counts the number of [Retrievable] stored by the database. + * + * @return The number of [Retrievable]s. + */ + override fun count(): Long { + try { + this.connection.prepareStatement("SELECT COUNT(*) FROM $RETRIEVABLE_ENTITY_NAME;").use { stmt -> + stmt.executeQuery().use { result -> + result.next() + return result.getLong(1) + } + } + } catch (e: Exception) { + LOGGER.error(e) { "Failed to count retrievable due to SQL error." } + return 0L + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt new file mode 100644 index 00000000..70708a34 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt @@ -0,0 +1,182 @@ +package org.vitrivr.engine.database.pgvector.retrievable + +import org.vitrivr.engine.core.database.retrievable.RetrievableWriter +import org.vitrivr.engine.core.model.relationship.Relationship +import org.vitrivr.engine.core.model.retrievable.Retrievable +import org.vitrivr.engine.database.pgvector.* +import java.sql.Connection +import java.sql.SQLException + +/** + * A [RetrievableWriter] implementation for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +internal class RetrievableWriter(private val connection: Connection): RetrievableWriter { + /** + * Adds a new [Retrievable] to the database using this [RetrievableWriter] instance. + * + * @param item [Retrievable] to add. + */ + override fun add(item: Retrievable): Boolean { + try { + this.connection.prepareStatement("INSERT INTO $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME, $RETRIEVABLE_TYPE_COLUMN_NAME) VALUES (?, ?);").use { stmt -> + stmt.setObject(1, item.id) + stmt.setString(2, item.type) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to persist retrievable ${item.id} due to SQL error." } + return false + } + } + + /** + * Adds new [Retrievable]s to the database using this [RetrievableWriter] instance. + * + * @param items An [Iterable] of [Retrievable]s to add. + */ + override fun addAll(items: Iterable): Boolean { + try { + this.connection.prepareStatement("INSERT INTO $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME, $RETRIEVABLE_TYPE_COLUMN_NAME) VALUES (?, ?);").use { stmt -> + for (item in items) { + stmt.setObject(1, item.id) + stmt.setString(2, item.type) + stmt.addBatch() + } + return stmt.executeBatch().all { it == 1 } + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to persist retrievables due to SQL error." } + return false + } + } + + /** + * Updates a specific [Retrievable] using this [RetrievableWriter]. + * + * @param item A [Retrievable]s to update. + * @return True on success, false otherwise. + */ + override fun update(item: Retrievable): Boolean { + try { + this.connection.prepareStatement("UPDATE $RETRIEVABLE_ENTITY_NAME SET $RETRIEVABLE_TYPE_COLUMN_NAME = ? WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> + stmt.setString(1, item.type) + stmt.setObject(2, item.id) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to update retrievable ${item.id} due to SQL error." } + return false + } + } + + /** + * Deletes (writes) a [Retrievable] using this [RetrievableWriter]. + * + * @param item A [Retrievable]s to delete. + * @return True on success, false otherwise. + */ + override fun delete(item: Retrievable): Boolean { + try { + this.connection.prepareStatement("DELETE FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?;").use { stmt -> + stmt.setObject(1, item.id) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to delete retrievable ${item.id} due to SQL error." } + return false + } + } + + /** + * Deletes (writes) a iterable of [Retrievable]s using this [RetrievableWriter]. + * + * @param items A [Iterable] of [Retrievable]s to delete. + * @return True on success, false otherwise. + */ + override fun deleteAll(items: Iterable): Boolean { + try { + this.connection.prepareStatement("DELETE FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?);").use { stmt -> + val values = items.map { it.id }.toTypedArray() + stmt.setArray(1, connection.createArrayOf("uuid", values)) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to delete retrievables due to SQL error." } + return false + } + } + + /** + * Persists a [Relationship]. + * + * @param relationship [Relationship] to persist + * @return True on success, false otherwise. + */ + override fun connect(relationship: Relationship): Boolean { + try { + this.connection.prepareStatement("INSERT INTO $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME,$PREDICATE_COLUMN_NAME,$SUBJECT_ID_COLUMN_NAME) VALUES (?,?,?)").use { stmt -> + stmt.setObject(1, relationship.objectId) + stmt.setString(2, relationship.predicate) + stmt.setObject(3, relationship.subjectId) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to insert relationship ${relationship.objectId} -(${relationship.predicate}-> ${relationship.subjectId} due to SQL error." } + return false + } + } + + /** + * Persists an [Iterable] of [Relationship]s. + * + * @param relationships An [Iterable] of [Relationship]s to persist. + * @return True on success, false otherwise. + */ + override fun connectAll(relationships: Iterable): Boolean { + try { + this.connection.prepareStatement("INSERT INTO $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME,$PREDICATE_COLUMN_NAME,$SUBJECT_ID_COLUMN_NAME) VALUES (?,?,?)").use { stmt -> + for (relationship in relationships) { + stmt.setObject(1, relationship.objectId) + stmt.setString(2, relationship.predicate) + stmt.setObject(3, relationship.subjectId) + stmt.addBatch() + } + return stmt.executeBatch().all { it == 1 } + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to insert relationships due to SQL error." } + return false + } + } + + /** + * Severs the specified connection between two [Retrievable]s. + * + * @param relationship [Relationship] to delete + * @return True on success, false otherwise. + */ + override fun disconnect(relationship: Relationship): Boolean { + try { + this.connection.prepareStatement("DELETE FROM $RELATIONSHIP_ENTITY_NAME WHERE $OBJECT_ID_COLUMN_NAME = ? AND $PREDICATE_COLUMN_NAME = ? AND $SUBJECT_ID_COLUMN_NAME = ?").use { stmt -> + stmt.setObject(1, relationship.objectId) + stmt.setString(2, relationship.predicate) + stmt.setObject(3, relationship.subjectId) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to delete relationship ${relationship.objectId} -(${relationship.predicate}-> ${relationship.subjectId} due to SQL error." } + return false + } + } + + /** + * Deletes all [Relationship]s + * + * @param relationships An [Iterable] of [Relationship] to delete. + * @return True on success, false otherwise. + */ + override fun disconnectAll(relationships: Iterable): Boolean = relationships.map { this.disconnect(it) }.all { it } +} \ No newline at end of file diff --git a/vitrivr-engine-server/build.gradle b/vitrivr-engine-server/build.gradle index b2959089..f050825d 100644 --- a/vitrivr-engine-server/build.gradle +++ b/vitrivr-engine-server/build.gradle @@ -10,6 +10,7 @@ dependencies { api project(':vitrivr-engine-query') api project(':vitrivr-engine-module-features') /* TODO: This dependency is not necessary and only here to facilitate easy testing. */ api project(':vitrivr-engine-module-cottontaildb') /* TODO: This dependency is not necessary and only here to facilitate easy testing. */ + api project(':vitrivr-engine-module-pgvector') /* TODO: This dependency is not necessary and only here to facilitate easy testing. */ api project(':vitrivr-engine-module-fes') /* TODO: This dependency is not necessary and only here to facilitate easy testing. */ /** Clikt & JLine */ From 8fe416c60826854fd97652dee5bdc329766378c8 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 8 Jul 2024 07:26:38 +0200 Subject: [PATCH 07/71] Removes check for success (which is not necessary). Signed-off-by: Ralph Gasser --- .../descriptor/struct/StructDescriptorInitializer.kt | 6 ++---- .../descriptor/vector/VectorDescriptorInitializer.kt | 5 ++--- .../pgvector/retrievable/RetrievableInitializer.kt | 9 +++------ 3 files changed, 7 insertions(+), 13 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt index c58569bb..ff967d0d 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt @@ -41,10 +41,8 @@ class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, conn try { /* Create 'retrievable' entity. */ - if (!this.connection.prepareStatement(/* sql = postgres */ statement.toString()).use { - it.execute() - }) { - LOGGER.warn { "Failed to initialize entity '$tableName'. It probably exists." } + this.connection.prepareStatement(/* sql = postgres */ statement.toString()).use { + it.execute() } } catch (e: SQLException) { LOGGER.error(e) { "Failed to initialize entity '$tableName' due to exception." } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt index e8dfa895..d14bdc9e 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt @@ -21,10 +21,9 @@ class VectorDescriptorInitializer(field: Schema.Field<*, VectorDescriptor<*>>, c val type = this.field.analyser.prototype(this.field) try { /* Create 'retrievable' entity. */ - if (!this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME vector(${type.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));").use { + this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME vector(${type.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + .use { it.execute() - }) { - LOGGER.warn { "Failed to initialize entity '$tableName'. It probably exists." } } } catch (e: SQLException) { LOGGER.error(e) { "Failed to initialize entity due to exception." } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt index 03dee51f..d2a393c5 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt @@ -19,17 +19,14 @@ internal class RetrievableInitializer(private val connection: Connection): Retri override fun initialize() { try { /* Create 'retrievable' entity. */ - if (!this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, type VARCHAR(100), PRIMARY KEY ($RETRIEVABLE_ID_COLUMN_NAME));").use { + this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, type VARCHAR(100), PRIMARY KEY ($RETRIEVABLE_ID_COLUMN_NAME));").use { it.execute() - }) { - LOGGER.warn { "Failed to initialize entity 'retrievable'. It probably exists." } } /* Create 'relationship' entity. */ - if (!this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));").use { + this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + .use { it.execute() - }) { - LOGGER.warn { "Failed to initialize entity 'relationships'. It probably exists." } } } catch (e: SQLException) { LOGGER.error(e) { "Failed to initialize entity due to exception." } From 9cca35eea07583349518da3165d3fe8d78fb0a20 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 8 Jul 2024 07:26:47 +0200 Subject: [PATCH 08/71] Fixes compilation error. Signed-off-by: Ralph Gasser --- .../database/pgvector/descriptor/model/PgVector.kt | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt index a26db32c..f36713ab 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt @@ -26,18 +26,9 @@ class PgVector(private var vec: FloatArray? = null) : PGobject(), PGBinaryObject * @param number * @param v list of numbers */ - constructor(v: List) : this(FloatArray(v.size) { - v[it].toFloat() - }) - - /** - * Constructor - * - * @param number - * @param v list of numbers - */ - constructor(vector: List>) : this(FloatArray(vector.size) { + constructor(vector: List) : this(FloatArray(vector.size) { when (val v = vector[it]) { + is Number -> v.toFloat() is Value.Float-> v.value is Value.Double-> v.value.toFloat() is Value.Int-> v.value.toFloat() From 35f7dbf67f196728324f00764212df375f0e1867 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 8 Jul 2024 08:05:09 +0200 Subject: [PATCH 09/71] Fixes a bug. Signed-off-by: Ralph Gasser --- .../pgvector/descriptor/struct/StructDescriptorWriter.kt | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt index 3f672349..32ab002f 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt @@ -11,6 +11,7 @@ import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorWriter import java.sql.Date import java.sql.PreparedStatement import java.sql.SQLException +import java.util.* /** * An [AbstractDescriptorWriter] for [StructDescriptor]s. @@ -55,7 +56,7 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio stmt.setAny(1, item.id) stmt.setAny(2, item.retrievableId) for ((i, v) in item.values().withIndex()) { - stmt.setAny(3 + i, v) + stmt.setAny(3 + i, v.second) } stmt.addBatch() } @@ -119,6 +120,7 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio is Double -> this.setDouble(index, value) is Boolean -> this.setBoolean(index, value) is ByteArray -> this.setBytes(index, value) + is UUID -> this.setObject(index, value) is Value<*> -> this.setValue(index, value) else -> this.setObject(index, value) } From 1e492ae1b5a71251835fb38f5db23fdf403bf291 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 8 Jul 2024 15:31:42 +0200 Subject: [PATCH 10/71] Adds a PgBitVector for use with boolean data. --- .../database/pgvector/PgVectorConnection.kt | 2 + .../pgvector/descriptor/model/PgBitVector.kt | 148 ++++++++++++++++++ .../pgvector/descriptor/model/PgVector.kt | 2 + .../vector/VectorDescriptorReader.kt | 7 +- 4 files changed, 158 insertions(+), 1 deletion(-) create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt index 59717d23..f4bae0b4 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt @@ -7,6 +7,7 @@ import org.vitrivr.engine.core.database.AbstractConnection import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer import org.vitrivr.engine.core.database.retrievable.RetrievableReader import org.vitrivr.engine.core.database.retrievable.RetrievableWriter +import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector import java.sql.Connection import java.sql.SQLException @@ -36,6 +37,7 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin /* Register the vector data type. */ this.connection.unwrap(PGConnection::class.java).addDataType("vector", PgVector::class.java) + this.connection.unwrap(PGConnection::class.java).addDataType("bit", PgBitVector::class.java) /* Create necessary database. */ try { diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt new file mode 100644 index 00000000..0aaa7e5f --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt @@ -0,0 +1,148 @@ +package org.vitrivr.engine.database.pgvector.descriptor.model + +import org.postgresql.util.ByteConverter +import org.postgresql.util.PGBinaryObject +import org.postgresql.util.PGobject +import java.io.Serializable +import java.sql.SQLException + + +/** + * The [PgBitVector] class represents a bit vector in PostgreSQL. + * + * @see https://github.com/pgvector/pgvector-java/blob/master/src/main/java/com/pgvector/PGbit.java + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class PgBitVector () : PGobject(), PGBinaryObject, Serializable { + + /** The [ByteArray] backing this [PgBitVector]. */ + private var vec: ByteArray? = null + + /** The length of this [PgBitVector]. */ + private var length = 0 + + /** + * Constructor + */ + init { + this.type = "bit" + } + + /** + * Constructor + * + * @param v boolean array + */ + constructor(v: BooleanArray) : this() { + val vec = ByteArray((v.size + 7) / 8) + for (i in v.indices) { + vec[i / 8] = (vec[i / 8].toInt() or ((if (v[i]) 1 else 0) shl (7 - (i % 8)))).toByte() + } + + /* Update local state. */ + this.vec = vec + this.length = v.size + } + + /** + * Constructor + * + * @param s text representation of a bit string + * @throws SQLException exception + */ + constructor(s: String?) : this() { + setValue(s) + } + + /** + * Sets the value from a text representation of a bit string + */ + @Throws(SQLException::class) + override fun setValue(s: String?) { + if (s == null) { + this.vec = null + this.length = 0 + } else { + val data = ByteArray((s.length + 7) / 8) + for (i in s.indices) { + data[i / 8] = (data[i / 8].toInt() or ((if (s[i] != '0') 1 else 0) shl (7 - (i % 8)))).toByte() + } + /* Update local state. */ + this.vec = data + this.length = s.length + } + } + + /** + * Returns the text representation of a bit string + */ + override fun getValue(): String? { + val data = this.vec ?: return null + val sb = StringBuilder(length) + for (i in 0 until length) { + sb.append(if (((data[i / 8].toInt() shr (7 - (i % 8))) and 1) == 1) '1' else '0') + } + return sb.toString() + } + + /** + * Returns the number of bytes for the binary representation + */ + override fun lengthInBytes(): Int = this.vec?.size?.plus(4) ?: 0 + + /** + * Sets the value from a binary representation of a bit string + */ + @Throws(SQLException::class) + override fun setByteValue(value: ByteArray, offset: Int) { + val length = ByteConverter.int4(value, offset) + val data = ByteArray((length + 7) / 8) + for (i in data.indices) { + data[i] = value[offset + 4 + i] + } + + /* Update local state. */ + this.vec = data + this.length = length + } + + /** + * Writes the binary representation of a bit string + */ + override fun toBytes(bytes: ByteArray, offset: Int) { + val data = this.vec ?: return + ByteConverter.int4(bytes, offset, length) + for (i in data.indices) { + bytes[offset + 4 + i] = data[i] + } + } + + /** + * Returns the length + * + * @return an array + */ + fun length(): Int = this.length + + /** + * Returns a byte array + * + * @return an array + */ + fun toByteArray(): ByteArray? = this.vec + + /** + * Returns an array + * + * @return an array + */ + fun toArray(): BooleanArray { + val bits = BooleanArray(this.length) + for (i in 0 until this.length) { + bits[i] = ((this.vec!![i / 8].toInt() shr (7 - (i % 8))) and 1) == 1 + } + return bits + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt index f36713ab..8d31e0ee 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt @@ -11,6 +11,8 @@ import java.util.* /** * The [PgVector] class represents a vector in PostgreSQL. * + * @see https://github.com/pgvector/pgvector-java/blob/master/src/main/java/com/pgvector/PGvector.java + * * @author Ralph Gasser * @version 1.0.0 */ diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt index 92042598..0fd51122 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt @@ -12,6 +12,7 @@ import org.vitrivr.engine.core.model.retrievable.attributes.DistanceAttribute import org.vitrivr.engine.core.model.types.toValue import org.vitrivr.engine.database.pgvector.* import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorReader +import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector import java.sql.ResultSet import java.util.* @@ -128,7 +129,11 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec } ) - is BooleanVectorDescriptor -> TODO() + is BooleanVectorDescriptor -> BooleanVectorDescriptor( + descriptorId, + retrievableId, + result.getObject(DESCRIPTOR_COLUMN_NAME, PgBitVector::class.java).let { vector -> vector.toArray().map { it.toValue() } } + ) } } From 5ddd71ea2a92de9957952e4d0e43fefe58458b8b Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 8 Jul 2024 15:33:36 +0200 Subject: [PATCH 11/71] Adds missing Hamming and Jaccard distances. --- .../org/vitrivr/engine/core/model/query/basics/Distance.kt | 4 +++- .../pgvector/descriptor/vector/VectorDescriptorReader.kt | 2 ++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/basics/Distance.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/basics/Distance.kt index 5c90b1bb..3ceb2f37 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/basics/Distance.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/basics/Distance.kt @@ -11,5 +11,7 @@ import org.vitrivr.engine.core.model.query.proximity.ProximityQuery enum class Distance { MANHATTAN, EUCLIDEAN, - COSINE; + COSINE, + HAMMING, + JACCARD; } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt index 0fd51122..f3243285 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt @@ -144,5 +144,7 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec MANHATTAN -> "<+>" EUCLIDEAN -> "<->" COSINE -> "<=>" + HAMMING -> "<~>" + JACCARD -> "<%>" } } \ No newline at end of file From d6622802d88c57842470c1464dde1615b128d88e Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 8 Jul 2024 15:43:20 +0200 Subject: [PATCH 12/71] Writer and Reader now expose owning connection (in preparation of transaction support). --- .../vitrivr/engine/core/database/Reader.kt | 3 +++ .../vitrivr/engine/core/database/Writer.kt | 3 +++ .../descriptors/AbstractDescriptorReader.kt | 2 +- .../descriptors/AbstractDescriptorWriter.kt | 2 +- .../retrievable/RetrievableReader.kt | 2 +- .../retrievable/RetrievableWriter.kt | 2 +- .../database/string/StringConnection.kt | 4 ++-- .../string/writer/StringDescriptorWriter.kt | 5 ++--- .../string/writer/StringRetrievableWriter.kt | 3 ++- .../database/string/writer/StringWriter.kt | 3 ++- .../database/pgvector/PgVectorConnection.kt | 20 +++++++++--------- .../descriptor/AbstractDescriptorReader.kt | 20 +++++++++--------- .../descriptor/AbstractDescriptorWriter.kt | 8 +++---- .../struct/StructDescriptorInitializer.kt | 2 +- .../struct/StructDescriptorReader.kt | 2 +- .../struct/StructDescriptorWriter.kt | 4 ++-- .../vector/VectorDescriptorInitializer.kt | 2 +- .../vector/VectorDescriptorReader.kt | 4 ++-- .../vector/VectorDescriptorWriter.kt | 6 +++--- .../retrievable/RetrievableInitializer.kt | 13 ++++++------ .../pgvector/retrievable/RetrievableReader.kt | 20 +++++++----------- .../pgvector/retrievable/RetrievableWriter.kt | 21 +++++++++---------- 22 files changed, 76 insertions(+), 75 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Reader.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Reader.kt index abba9526..4fe3c5a0 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Reader.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Reader.kt @@ -13,6 +13,9 @@ import java.util.* * @version 1.1.0 */ interface Reader { + /** The [Connection] used by this [Writer]. */ + val connection: Connection + /** * Returns a [Sequence] of all [Persistable] accessible by this [Reader]. * diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Writer.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Writer.kt index cf915455..35c8a1c0 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Writer.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Writer.kt @@ -15,6 +15,9 @@ import java.util.* */ interface Writer { + /** The [Connection] used by this [Writer]. */ + val connection: Connection + /** * Adds (and typically persists) a single [Persistable] of type [T] through this [Writer]. * diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt index b2676ba2..e2909540 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt @@ -27,7 +27,7 @@ private val logger: KLogger = KotlinLogging.logger {} * @author Ralph Gasser * @version 1.0.0 */ -abstract class AbstractDescriptorReader(final override val field: Schema.Field<*, D>, protected val connection: CottontailConnection) : DescriptorReader { +abstract class AbstractDescriptorReader(final override val field: Schema.Field<*, D>, override val connection: CottontailConnection) : DescriptorReader { /** The [Name.EntityName] used by this [Descriptor]. */ protected val entityName: Name.EntityName = Name.EntityName.create(this.field.schema.name, "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName.lowercase()}") diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt index fa6ab4da..ae5a74d2 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt @@ -24,7 +24,7 @@ private val logger: KLogger = KotlinLogging.logger {} * @author Ralph Gasser * @version 1.0.0 */ -abstract class AbstractDescriptorWriter(final override val field: Schema.Field<*, D>, protected val connection: CottontailConnection) : DescriptorWriter { +abstract class AbstractDescriptorWriter(final override val field: Schema.Field<*, D>, override val connection: CottontailConnection) : DescriptorWriter { /** The [Name.EntityName] used by this [Descriptor]. */ protected val entityName: Name.EntityName = Name.EntityName.create(this.field.schema.name, "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName.lowercase()}") diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableReader.kt index 2cee1e9b..45bba869 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableReader.kt @@ -28,7 +28,7 @@ private val logger: KLogger = KotlinLogging.logger {} * @author Ralph Gasser * @version 1.0.0 */ -internal class RetrievableReader(private val connection: CottontailConnection) : RetrievableReader { +internal class RetrievableReader(override val connection: CottontailConnection) : RetrievableReader { /** The [Name.EntityName] for this [RetrievableInitializer]. */ private val entityName: Name.EntityName = Name.EntityName.create(this.connection.schemaName, RETRIEVABLE_ENTITY_NAME) diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableWriter.kt index fffc53d0..6fd0c94d 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableWriter.kt @@ -28,7 +28,7 @@ private val logger: KLogger = KotlinLogging.logger {} * @author Ralph Gasser * @version 1.0.0 */ -internal class RetrievableWriter(private val connection: CottontailConnection) : RetrievableWriter { +internal class RetrievableWriter(override val connection: CottontailConnection) : RetrievableWriter { /** The [Name.EntityName] of the retrievable entity. */ private val entityName: Name.EntityName = Name.EntityName.create(this.connection.schemaName, RETRIEVABLE_ENTITY_NAME) diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt index 1299e3da..b961ef59 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt @@ -1,19 +1,19 @@ package org.vitrivr.engine.module.features.database.string -import org.vitrivr.engine.module.features.database.string.writer.StringRetrievableWriter import org.vitrivr.engine.core.database.AbstractConnection import org.vitrivr.engine.core.database.retrievable.NoRetrievableInitializer import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer import org.vitrivr.engine.core.database.retrievable.RetrievableReader import org.vitrivr.engine.core.database.retrievable.RetrievableWriter import org.vitrivr.engine.core.model.Persistable +import org.vitrivr.engine.module.features.database.string.writer.StringRetrievableWriter class StringConnection(override val provider: StringConnectionProvider, schemaName: String, internal val stringify: (Persistable) -> String) : AbstractConnection(schemaName, provider) { override fun getRetrievableInitializer(): RetrievableInitializer = NoRetrievableInitializer() - override fun getRetrievableWriter(): RetrievableWriter = StringRetrievableWriter(provider.targetStream, stringify) + override fun getRetrievableWriter(): RetrievableWriter = StringRetrievableWriter(this, provider.targetStream, stringify) override fun getRetrievableReader(): RetrievableReader { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringDescriptorWriter.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringDescriptorWriter.kt index 87be81de..66c63105 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringDescriptorWriter.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringDescriptorWriter.kt @@ -1,9 +1,8 @@ package org.vitrivr.engine.module.features.database.string.writer -import org.vitrivr.engine.module.features.database.string.StringConnection import org.vitrivr.engine.core.database.descriptor.DescriptorWriter import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.module.features.database.string.StringConnection -class StringDescriptorWriter(connection: StringConnection, override val field: Schema.Field<*,D>) : DescriptorWriter, StringWriter(connection.provider.targetStream, connection.stringify) { -} \ No newline at end of file +class StringDescriptorWriter(connection: StringConnection, override val field: Schema.Field<*,D>) : DescriptorWriter, StringWriter(connection, connection.provider.targetStream, connection.stringify) \ No newline at end of file diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringRetrievableWriter.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringRetrievableWriter.kt index 21c4d5e3..4dbed8d2 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringRetrievableWriter.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringRetrievableWriter.kt @@ -4,9 +4,10 @@ import org.vitrivr.engine.core.database.retrievable.RetrievableWriter import org.vitrivr.engine.core.model.Persistable import org.vitrivr.engine.core.model.relationship.Relationship import org.vitrivr.engine.core.model.retrievable.Retrievable +import org.vitrivr.engine.module.features.database.string.StringConnection import java.io.OutputStream -class StringRetrievableWriter(outputStream: OutputStream, stringify: (Persistable) -> String) : StringWriter(outputStream, stringify), RetrievableWriter { +class StringRetrievableWriter(connection: StringConnection, outputStream: OutputStream, stringify: (Persistable) -> String) : StringWriter(connection, outputStream, stringify), RetrievableWriter { override fun connect(relationship: Relationship): Boolean { TODO("Not yet implemented") } diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringWriter.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringWriter.kt index 4e8d7ab9..3dea5891 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringWriter.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/writer/StringWriter.kt @@ -2,11 +2,12 @@ package org.vitrivr.engine.module.features.database.string.writer import org.vitrivr.engine.core.database.Writer import org.vitrivr.engine.core.model.Persistable +import org.vitrivr.engine.module.features.database.string.StringConnection import java.io.IOException import java.io.OutputStream import java.io.PrintWriter -open class StringWriter(outputStream: OutputStream, private val stringify: (Persistable) -> String) : Writer { +open class StringWriter(override val connection: StringConnection, outputStream: OutputStream, private val stringify: (Persistable) -> String) : Writer { private val writer = PrintWriter(outputStream) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt index f4bae0b4..e92d2f28 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt @@ -22,12 +22,12 @@ internal val LOGGER: KLogger = logger("org.vitrivr.engine.database.pgvector.PgVe * @author Ralph Gasser * @version 1.0.0 */ -class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: String, internal val connection: Connection): AbstractConnection(schemaName, provider) { +class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: String, internal val jdbc: Connection): AbstractConnection(schemaName, provider) { init { /* Make sure that the pg_vector extension is installed. */ try { - this.connection.prepareStatement("CREATE EXTENSION IF NOT EXISTS vector;").use { + this.jdbc.prepareStatement("CREATE EXTENSION IF NOT EXISTS vector;").use { it.execute() } } catch (e: SQLException) { @@ -36,12 +36,12 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin } /* Register the vector data type. */ - this.connection.unwrap(PGConnection::class.java).addDataType("vector", PgVector::class.java) - this.connection.unwrap(PGConnection::class.java).addDataType("bit", PgBitVector::class.java) + this.jdbc.unwrap(PGConnection::class.java).addDataType("vector", PgVector::class.java) + this.jdbc.unwrap(PGConnection::class.java).addDataType("bit", PgBitVector::class.java) /* Create necessary database. */ try { - this.connection.prepareStatement("CREATE DATABASE $schemaName;").use { + this.jdbc.prepareStatement("CREATE DATABASE $schemaName;").use { it.execute() } } catch (e: SQLException) { @@ -60,7 +60,7 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin * @return [RetrievableInitializer] */ override fun getRetrievableInitializer(): RetrievableInitializer - = org.vitrivr.engine.database.pgvector.retrievable.RetrievableInitializer(this.connection) + = org.vitrivr.engine.database.pgvector.retrievable.RetrievableInitializer(this) /** * Generates and returns a [RetrievableWriter] for this [PgVectorConnection]. @@ -68,7 +68,7 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin * @return [RetrievableWriter] */ override fun getRetrievableWriter(): RetrievableWriter - = org.vitrivr.engine.database.pgvector.retrievable.RetrievableWriter(this.connection) + = org.vitrivr.engine.database.pgvector.retrievable.RetrievableWriter(this) /** * Generates and returns a [RetrievableWriter] for this [PgVectorConnection]. @@ -76,19 +76,19 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin * @return [RetrievableReader] */ override fun getRetrievableReader(): RetrievableReader - = org.vitrivr.engine.database.pgvector.retrievable.RetrievableReader(this.connection) + = org.vitrivr.engine.database.pgvector.retrievable.RetrievableReader(this) /** * Returns the human-readable description of this [PgVectorConnection]. */ - override fun description(): String = this.connection.toString() + override fun description(): String = this.jdbc.toString() /** * Closes this [PgVectorConnection] */ override fun close() { try { - this.connection.close() + this.jdbc.close() } catch (e: SQLException) { LOGGER.warn(e) { "Failed to close database connection due to exception." } } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt index 4f0c2a3c..a44e287b 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt @@ -18,7 +18,7 @@ import java.util.* * @author Ralph Gasser * @version 1.0.0 */ -abstract class AbstractDescriptorReader(final override val field: Schema.Field<*, D>, protected val connection: PgVectorConnection) : DescriptorReader { +abstract class AbstractDescriptorReader(final override val field: Schema.Field<*, D>, override val connection: PgVectorConnection) : DescriptorReader { /** The name of the table backing this [AbstractDescriptorReader]. */ protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" @@ -35,7 +35,7 @@ abstract class AbstractDescriptorReader(final override val field */ override fun get(descriptorId: DescriptorId): D? { try { - this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT * FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?").use { stmt -> stmt.setObject(1, descriptorId) stmt.executeQuery().use { res -> if (res.next()) { @@ -59,7 +59,7 @@ abstract class AbstractDescriptorReader(final override val field */ override fun getFor(retrievableId: RetrievableId): Sequence { try { - this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT * FROM $tableName WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> stmt.setObject(1, retrievableId) val result = stmt.executeQuery() return sequence { @@ -84,7 +84,7 @@ abstract class AbstractDescriptorReader(final override val field */ override fun exists(descriptorId: DescriptorId): Boolean { try { - this.connection.connection.prepareStatement("SELECT count(*) FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT count(*) FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?").use { stmt -> stmt.setObject(1, descriptorId) stmt.executeQuery().use { res -> res.next() @@ -104,7 +104,7 @@ abstract class AbstractDescriptorReader(final override val field */ override fun getAll(): Sequence { try { - this.connection.connection.prepareStatement("SELECT * FROM $tableName").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT * FROM $tableName").use { stmt -> val result = stmt.executeQuery() return sequence { result.use { @@ -128,9 +128,9 @@ abstract class AbstractDescriptorReader(final override val field */ override fun getAll(descriptorIds: Iterable): Sequence { try { - this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ANY (?)").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT * FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ANY (?)").use { stmt -> val values = descriptorIds.map { it }.toTypedArray() - stmt.setArray(1, this.connection.connection.createArrayOf("uuid", values)) + stmt.setArray(1, this.connection.jdbc.createArrayOf("uuid", values)) val result = stmt.executeQuery() return sequence { result.use { @@ -154,9 +154,9 @@ abstract class AbstractDescriptorReader(final override val field */ override fun getAllFor(retrievableIds: Iterable): Sequence { try { - this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?)").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT * FROM $tableName WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?)").use { stmt -> val values = retrievableIds.map { it }.toTypedArray() - stmt.setArray(1, this.connection.connection.createArrayOf("uuid", values)) + stmt.setArray(1, this.connection.jdbc.createArrayOf("uuid", values)) val result = stmt.executeQuery() return sequence { result.use { @@ -179,7 +179,7 @@ abstract class AbstractDescriptorReader(final override val field */ override fun count(): Long { try { - this.connection.connection.prepareStatement("SELECT COUNT(*) FROM $tableName;").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT COUNT(*) FROM $tableName;").use { stmt -> stmt.executeQuery().use { result -> result.next() return result.getLong(1) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt index 6d295d5f..65a30142 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt @@ -15,7 +15,7 @@ import java.sql.SQLException * @author Ralph Gasser * @version 1.0.0 */ -abstract class AbstractDescriptorWriter(final override val field: Schema.Field<*, D>, protected val connection: PgVectorConnection): DescriptorWriter { +abstract class AbstractDescriptorWriter(final override val field: Schema.Field<*, D>, override val connection: PgVectorConnection): DescriptorWriter { /** The name of the table backing this [AbstractDescriptorInitializer]. */ protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" @@ -27,7 +27,7 @@ abstract class AbstractDescriptorWriter(final override val field */ override fun delete(item: D): Boolean { try { - this.connection.connection.prepareStatement("DELETE FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?;").use { stmt -> + this.connection.jdbc.prepareStatement("DELETE FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?;").use { stmt -> stmt.setObject(1, item.id) return stmt.execute() } @@ -45,9 +45,9 @@ abstract class AbstractDescriptorWriter(final override val field */ override fun deleteAll(items: Iterable): Boolean { try { - this.connection.connection.prepareStatement("DELETE FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ANY (?);").use { stmt -> + this.connection.jdbc.prepareStatement("DELETE FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ANY (?);").use { stmt -> val values = items.map { it.id }.toTypedArray() - stmt.setArray(1, this.connection.connection.createArrayOf("uuid", values)) + stmt.setArray(1, this.connection.jdbc.createArrayOf("uuid", values)) return stmt.execute() } } catch (e: SQLException) { diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt index ff967d0d..58c5db4f 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt @@ -13,7 +13,7 @@ import java.sql.SQLException * @author Ralph Gasser * @version 1.0.0 */ -class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, connection: PgVectorConnection) : AbstractDescriptorInitializer(field, connection.connection) { +class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, connection: PgVectorConnection) : AbstractDescriptorInitializer(field, connection.jdbc) { override fun initialize() { val statement = StringBuilder("CREATE TABLE IF NOT EXISTS $tableName(") statement.append("$DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, ") diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt index 393498f9..44e452c9 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt @@ -37,7 +37,7 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio is SimpleFulltextQuery -> TODO() is SimpleBooleanQuery<*> -> { require(query.attributeName != null) { "Boolean query on a struct field requires specification of a field's attribute name." } - this.connection.connection.prepareStatement("SELECT * FROM $tableName WHERE ${query.attributeName} = ?;").apply { + this.connection.jdbc.prepareStatement("SELECT * FROM $tableName WHERE ${query.attributeName} = ?;").apply { setString(1, query.value.toString()) } } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt index 32ab002f..114979cb 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt @@ -137,7 +137,7 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio statement.append(", \"${field.name}\" = ?") } statement.append("WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?;") - return this.connection.connection.prepareStatement(statement.toString()) + return this.connection.jdbc.prepareStatement(statement.toString()) } /** @@ -155,6 +155,6 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio statement.append(", ?") } statement.append(");") - return this.connection.connection.prepareStatement(statement.toString()) + return this.connection.jdbc.prepareStatement(statement.toString()) } } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt index d14bdc9e..4694847a 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt @@ -13,7 +13,7 @@ import java.sql.SQLException * @author Ralph Gasser * @version 1.0.0 */ -class VectorDescriptorInitializer(field: Schema.Field<*, VectorDescriptor<*>>, connection: PgVectorConnection): AbstractDescriptorInitializer>(field, connection.connection) { +class VectorDescriptorInitializer(field: Schema.Field<*, VectorDescriptor<*>>, connection: PgVectorConnection): AbstractDescriptorInitializer>(field, connection.jdbc) { /** * Initializes the [RetrievableInitializer]. */ diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt index f3243285..7ec7e4fa 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt @@ -32,7 +32,7 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec override fun query(query: Query): Sequence> { try { val statement = when (query) { - is ProximityQuery<*> -> this.connection.connection.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").apply { + is ProximityQuery<*> -> this.connection.jdbc.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").apply { setObject(1, PgVector(query.value)) } else -> throw UnsupportedOperationException("Query of typ ${query::class} is not supported by VectorDescriptorReader.") @@ -57,7 +57,7 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec return when (query) { is ProximityQuery<*> -> { val descriptors = mutableListOf, Float>>() - this.connection.connection.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").use { stmt -> stmt.setObject(1, PgVector(query.value)) stmt.executeQuery().use { result -> while (result.next()) { diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt index 9d62212a..ea2d424b 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt @@ -22,7 +22,7 @@ class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connec */ override fun add(item: VectorDescriptor<*>): Boolean { try { - this.connection.connection.prepareStatement("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $DESCRIPTOR_COLUMN_NAME) VALUES (?, ?, ?);").use { stmt -> + this.connection.jdbc.prepareStatement("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $DESCRIPTOR_COLUMN_NAME) VALUES (?, ?, ?);").use { stmt -> stmt.setObject(1, item.id) stmt.setObject(2, item.retrievableId) stmt.setObject(3, PgVector(item.vector)) @@ -42,7 +42,7 @@ class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connec */ override fun addAll(items: Iterable>): Boolean { try { - this.connection.connection.prepareStatement("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $DESCRIPTOR_COLUMN_NAME) VALUES (?, ?, ?);").use { stmt -> + this.connection.jdbc.prepareStatement("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $DESCRIPTOR_COLUMN_NAME) VALUES (?, ?, ?);").use { stmt -> for (item in items) { stmt.setObject(1, item.id) stmt.setObject(2, item.retrievableId) @@ -65,7 +65,7 @@ class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connec */ override fun update(item: VectorDescriptor<*>): Boolean { try { - this.connection.connection.prepareStatement("UPDATE $tableName SET $RETRIEVABLE_ID_COLUMN_NAME = ?, $DESCRIPTOR_COLUMN_NAME = ? WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?;").use { stmt -> + this.connection.jdbc.prepareStatement("UPDATE $tableName SET $RETRIEVABLE_ID_COLUMN_NAME = ?, $DESCRIPTOR_COLUMN_NAME = ? WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?;").use { stmt -> stmt.setObject(1, item.retrievableId) stmt.setObject(2, PgVector(item.vector)) stmt.setObject(3, item.id) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt index d2a393c5..8abd804b 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt @@ -3,7 +3,6 @@ package org.vitrivr.engine.database.pgvector.retrievable import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer import org.vitrivr.engine.core.model.retrievable.Retrievable import org.vitrivr.engine.database.pgvector.* -import java.sql.Connection import java.sql.SQLException /** @@ -12,19 +11,19 @@ import java.sql.SQLException * @author Ralph Gasser * @version 1.0.0 */ -internal class RetrievableInitializer(private val connection: Connection): RetrievableInitializer { +internal class RetrievableInitializer(private val connection: PgVectorConnection): RetrievableInitializer { /** * Initializes the [RetrievableInitializer]. */ override fun initialize() { try { /* Create 'retrievable' entity. */ - this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, type VARCHAR(100), PRIMARY KEY ($RETRIEVABLE_ID_COLUMN_NAME));").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, type VARCHAR(100), PRIMARY KEY ($RETRIEVABLE_ID_COLUMN_NAME));").use { it.execute() } /* Create 'relationship' entity. */ - this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") .use { it.execute() } @@ -40,14 +39,14 @@ internal class RetrievableInitializer(private val connection: Connection): Retri */ override fun isInitialized(): Boolean { try { - this.connection.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $RETRIEVABLE_ENTITY_NAME").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $RETRIEVABLE_ENTITY_NAME").use { it.execute() } } catch (e: SQLException) { return false } try { - this.connection.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $RELATIONSHIP_ENTITY_NAME").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $RELATIONSHIP_ENTITY_NAME").use { it.execute() } } catch (e: SQLException) { @@ -61,7 +60,7 @@ internal class RetrievableInitializer(private val connection: Connection): Retri */ override fun truncate() { try { - this.connection.prepareStatement(/* sql = postgres */ "TRUNCATE $RETRIEVABLE_ENTITY_NAME, $RELATIONSHIP_ENTITY_NAME").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "TRUNCATE $RETRIEVABLE_ENTITY_NAME, $RELATIONSHIP_ENTITY_NAME").use { it.execute() } } catch (e: SQLException) { diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt index 04156a8b..4fdc9d8b 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt @@ -4,11 +4,7 @@ import org.vitrivr.engine.core.database.retrievable.RetrievableReader import org.vitrivr.engine.core.model.retrievable.Retrievable import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.retrievable.Retrieved -import org.vitrivr.engine.database.pgvector.LOGGER -import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ENTITY_NAME -import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME -import org.vitrivr.engine.database.pgvector.RETRIEVABLE_TYPE_COLUMN_NAME -import java.sql.Connection +import org.vitrivr.engine.database.pgvector.* import java.util.* /** @@ -17,13 +13,13 @@ import java.util.* * @author Ralph Gasser * @version 1.0.0 */ -class RetrievableReader(private val connection: Connection): RetrievableReader { +class RetrievableReader(override val connection: PgVectorConnection): RetrievableReader { /** * Returns the [Retrievable]s that matches the provided [RetrievableId] */ override fun get(id: RetrievableId): Retrievable? { try { - this.connection.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> stmt.setObject(1, id) stmt.executeQuery().use { res -> if (res.next() ) { @@ -47,7 +43,7 @@ class RetrievableReader(private val connection: Connection): RetrievableReader { */ override fun exists(id: RetrievableId): Boolean { try { - this.connection.prepareStatement("SELECT count(*) FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT count(*) FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> stmt.setObject(1, id) stmt.executeQuery().use { res -> res.next() @@ -68,9 +64,9 @@ class RetrievableReader(private val connection: Connection): RetrievableReader { */ override fun getAll(ids: Iterable): Sequence { try { - this.connection.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?)").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?)").use { stmt -> val values = ids.map { it }.toTypedArray() - stmt.setArray(1, connection.createArrayOf("uuid", values)) + stmt.setArray(1, this.connection.jdbc.createArrayOf("uuid", values)) val result = stmt.executeQuery() return sequence { while (result.next()) { @@ -92,7 +88,7 @@ class RetrievableReader(private val connection: Connection): RetrievableReader { */ override fun getAll(): Sequence { try { - this.connection.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME").use { stmt -> val result = stmt.executeQuery() return sequence { while (result.next()) { @@ -122,7 +118,7 @@ class RetrievableReader(private val connection: Connection): RetrievableReader { */ override fun count(): Long { try { - this.connection.prepareStatement("SELECT COUNT(*) FROM $RETRIEVABLE_ENTITY_NAME;").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT COUNT(*) FROM $RETRIEVABLE_ENTITY_NAME;").use { stmt -> stmt.executeQuery().use { result -> result.next() return result.getLong(1) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt index 70708a34..4592097a 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt @@ -4,7 +4,6 @@ import org.vitrivr.engine.core.database.retrievable.RetrievableWriter import org.vitrivr.engine.core.model.relationship.Relationship import org.vitrivr.engine.core.model.retrievable.Retrievable import org.vitrivr.engine.database.pgvector.* -import java.sql.Connection import java.sql.SQLException /** @@ -13,7 +12,7 @@ import java.sql.SQLException * @author Ralph Gasser * @version 1.0.0 */ -internal class RetrievableWriter(private val connection: Connection): RetrievableWriter { +internal class RetrievableWriter(override val connection: PgVectorConnection): RetrievableWriter { /** * Adds a new [Retrievable] to the database using this [RetrievableWriter] instance. * @@ -21,7 +20,7 @@ internal class RetrievableWriter(private val connection: Connection): Retrievabl */ override fun add(item: Retrievable): Boolean { try { - this.connection.prepareStatement("INSERT INTO $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME, $RETRIEVABLE_TYPE_COLUMN_NAME) VALUES (?, ?);").use { stmt -> + this.connection.jdbc.prepareStatement("INSERT INTO $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME, $RETRIEVABLE_TYPE_COLUMN_NAME) VALUES (?, ?);").use { stmt -> stmt.setObject(1, item.id) stmt.setString(2, item.type) return stmt.execute() @@ -39,7 +38,7 @@ internal class RetrievableWriter(private val connection: Connection): Retrievabl */ override fun addAll(items: Iterable): Boolean { try { - this.connection.prepareStatement("INSERT INTO $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME, $RETRIEVABLE_TYPE_COLUMN_NAME) VALUES (?, ?);").use { stmt -> + this.connection.jdbc.prepareStatement("INSERT INTO $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME, $RETRIEVABLE_TYPE_COLUMN_NAME) VALUES (?, ?);").use { stmt -> for (item in items) { stmt.setObject(1, item.id) stmt.setString(2, item.type) @@ -61,7 +60,7 @@ internal class RetrievableWriter(private val connection: Connection): Retrievabl */ override fun update(item: Retrievable): Boolean { try { - this.connection.prepareStatement("UPDATE $RETRIEVABLE_ENTITY_NAME SET $RETRIEVABLE_TYPE_COLUMN_NAME = ? WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> + this.connection.jdbc.prepareStatement("UPDATE $RETRIEVABLE_ENTITY_NAME SET $RETRIEVABLE_TYPE_COLUMN_NAME = ? WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> stmt.setString(1, item.type) stmt.setObject(2, item.id) return stmt.execute() @@ -80,7 +79,7 @@ internal class RetrievableWriter(private val connection: Connection): Retrievabl */ override fun delete(item: Retrievable): Boolean { try { - this.connection.prepareStatement("DELETE FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?;").use { stmt -> + this.connection.jdbc.prepareStatement("DELETE FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?;").use { stmt -> stmt.setObject(1, item.id) return stmt.execute() } @@ -98,9 +97,9 @@ internal class RetrievableWriter(private val connection: Connection): Retrievabl */ override fun deleteAll(items: Iterable): Boolean { try { - this.connection.prepareStatement("DELETE FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?);").use { stmt -> + this.connection.jdbc.prepareStatement("DELETE FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?);").use { stmt -> val values = items.map { it.id }.toTypedArray() - stmt.setArray(1, connection.createArrayOf("uuid", values)) + stmt.setArray(1, this.connection.jdbc.createArrayOf("uuid", values)) return stmt.execute() } } catch (e: SQLException) { @@ -117,7 +116,7 @@ internal class RetrievableWriter(private val connection: Connection): Retrievabl */ override fun connect(relationship: Relationship): Boolean { try { - this.connection.prepareStatement("INSERT INTO $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME,$PREDICATE_COLUMN_NAME,$SUBJECT_ID_COLUMN_NAME) VALUES (?,?,?)").use { stmt -> + this.connection.jdbc.prepareStatement("INSERT INTO $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME,$PREDICATE_COLUMN_NAME,$SUBJECT_ID_COLUMN_NAME) VALUES (?,?,?)").use { stmt -> stmt.setObject(1, relationship.objectId) stmt.setString(2, relationship.predicate) stmt.setObject(3, relationship.subjectId) @@ -137,7 +136,7 @@ internal class RetrievableWriter(private val connection: Connection): Retrievabl */ override fun connectAll(relationships: Iterable): Boolean { try { - this.connection.prepareStatement("INSERT INTO $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME,$PREDICATE_COLUMN_NAME,$SUBJECT_ID_COLUMN_NAME) VALUES (?,?,?)").use { stmt -> + this.connection.jdbc.prepareStatement("INSERT INTO $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME,$PREDICATE_COLUMN_NAME,$SUBJECT_ID_COLUMN_NAME) VALUES (?,?,?)").use { stmt -> for (relationship in relationships) { stmt.setObject(1, relationship.objectId) stmt.setString(2, relationship.predicate) @@ -160,7 +159,7 @@ internal class RetrievableWriter(private val connection: Connection): Retrievabl */ override fun disconnect(relationship: Relationship): Boolean { try { - this.connection.prepareStatement("DELETE FROM $RELATIONSHIP_ENTITY_NAME WHERE $OBJECT_ID_COLUMN_NAME = ? AND $PREDICATE_COLUMN_NAME = ? AND $SUBJECT_ID_COLUMN_NAME = ?").use { stmt -> + this.connection.jdbc.prepareStatement("DELETE FROM $RELATIONSHIP_ENTITY_NAME WHERE $OBJECT_ID_COLUMN_NAME = ? AND $PREDICATE_COLUMN_NAME = ? AND $SUBJECT_ID_COLUMN_NAME = ?").use { stmt -> stmt.setObject(1, relationship.objectId) stmt.setString(2, relationship.predicate) stmt.setObject(3, relationship.subjectId) From a93aa9b50b443d71d0e83a3bea04c3f1f3992414 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 8 Jul 2024 15:52:30 +0200 Subject: [PATCH 13/71] Adds experimental (and very simple) transaction support to Connection. --- .../engine/core/database/Connection.kt | 7 ++++++ .../cottontaildb/CottontailConnection.kt | 24 +++++++++++++++++++ .../database/pgvector/PgVectorConnection.kt | 21 ++++++++++++++++ 3 files changed, 52 insertions(+) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Connection.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Connection.kt index 0ee9c29d..e1e039d9 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Connection.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Connection.kt @@ -25,6 +25,13 @@ sealed interface Connection: Closeable { /** The name of the [Schema] managed by this [Connection]. */ val schemaName: String + /** + * Executes the provided action within a transaction (if supported by the database). + * + * @param action The action to execute within the transaction. + */ + fun withTransaction(action: (Unit) -> T): T + /** * Initializes the database layer with the [Schema] used by this [Connection]. */ diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnection.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnection.kt index ba533b71..2e5b0929 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnection.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnection.kt @@ -1,5 +1,7 @@ package org.vitrivr.engine.plugin.cottontaildb +import io.github.oshai.kotlinlogging.KLogger +import io.github.oshai.kotlinlogging.KotlinLogging.logger import io.grpc.ManagedChannel import io.grpc.ManagedChannelBuilder import org.vitrivr.cottontail.client.SimpleClient @@ -8,6 +10,9 @@ import org.vitrivr.engine.plugin.cottontaildb.retrievable.RetrievableInitializer import org.vitrivr.engine.plugin.cottontaildb.retrievable.RetrievableReader import org.vitrivr.engine.plugin.cottontaildb.retrievable.RetrievableWriter +/** Defines [KLogger] of the class. */ +internal val LOGGER: KLogger = logger("org.vitrivr.engine.plugin.cottontaildb.CottontailConnection") + /** * A [AbstractConnection] to connect to a Cottontail DB instance. * @@ -22,6 +27,25 @@ class CottontailConnection(provider: CottontailConnectionProvider, schemaName: S /** The [SimpleClient] instance used by this [CottontailConnection]. */ internal val client = SimpleClient(this.channel) + /** + * Tries to execute a given action within a database transaction. + * + * @param action The action to execute within the transaction. + */ + @Synchronized + override fun withTransaction(action: (Unit) -> T): T { + val transactionId = this.client.begin() + try { + val ret = action.invoke(Unit) + this.client.commit(transactionId) + return ret + } catch (e: Throwable) { + this.client.rollback(transactionId) + LOGGER.error(e) { "Failed to execute action in transaction due to erro.." } + throw e + } + } + /** * Generates and returns a [RetrievableInitializer] for this [CottontailConnection]. * diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt index e92d2f28..86984529 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt @@ -54,6 +54,27 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin } } + /** + * Tries to execute a given action within a database transaction. + * + * @param action The action to execute within the transaction. + */ + @Synchronized + override fun withTransaction(action: (Unit) -> T): T { + try { + this.jdbc.autoCommit = false + val ret = action.invoke(Unit) + this.jdbc.commit() + return ret + } catch (e: Throwable) { + LOGGER.error(e) { "Failed to execute transaction due to exception." } + this.jdbc.rollback() + throw e + } finally { + this.jdbc.autoCommit = true + } + } + /** * Generates and returns a [RetrievableInitializer] for this [PgVectorConnection]. * From e426c9b49d6b59079ef29741a02916c1738c3755 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 8 Jul 2024 23:01:15 +0200 Subject: [PATCH 14/71] Updates type system: - Vectors now also have basic types. - Descriptors now always encode values of those basic types following a defined schema. - Cleans StructDescriptor vs. MapStructDescriptor; all StructDescriptors inherit from a map-based implementation. Signed-off-by: Ralph Gasser --- .../metadata/source/exif/ExifMetadata.kt | 18 +- .../source/exif/ExifMetadataExtractor.kt | 70 ++++---- .../source/exif/ExifMetadataRetriever.kt | 2 +- .../file/FileSourceMetadataExtractor.kt | 8 +- .../video/VideoSourceMetadataExtractor.kt | 16 +- .../temporal/TemporalMetadataExtractor.kt | 6 +- .../model/color/RGBFloatColorContainer.kt | 2 +- .../engine/core/model/descriptor/Attribute.kt | 15 ++ .../core/model/descriptor/Descriptor.kt | 9 +- .../core/model/descriptor/FieldSchema.kt | 12 -- .../descriptor/scalar/BooleanDescriptor.kt | 10 +- .../descriptor/scalar/DoubleDescriptor.kt | 10 +- .../descriptor/scalar/FloatDescriptor.kt | 10 +- .../model/descriptor/scalar/IntDescriptor.kt | 10 +- .../model/descriptor/scalar/LongDescriptor.kt | 10 +- .../descriptor/scalar/ScalarDescriptor.kt | 5 +- .../descriptor/scalar/StringDescriptor.kt | 10 +- .../descriptor/struct/LabelDescriptor.kt | 30 ++-- .../descriptor/struct/MapStructDescriptor.kt | 37 ++++ .../descriptor/struct/RasterDescriptor.kt | 33 ---- .../descriptor/struct/SkeletonDescriptor.kt | 35 ---- .../metadata/MediaDimensionsDescriptor.kt | 56 +++--- .../metadata/Rectangle2DMetadataDescriptor.kt | 63 +++---- .../metadata/TemporalMetadataDescriptor.kt | 38 ++-- .../source/FileSourceMetadataDescriptor.kt | 42 ++--- .../metadata/source/MapStructDescriptor.kt | 65 ------- .../source/VideoSourceMetadataDescriptor.kt | 79 +++++---- .../vector/BooleanVectorDescriptor.kt | 14 +- .../vector/DoubleVectorDescriptor.kt | 14 +- .../vector/FloatVectorDescriptor.kt | 14 +- .../descriptor/vector/IntVectorDescriptor.kt | 14 +- .../descriptor/vector/LongVectorDescriptor.kt | 14 +- .../descriptor/vector/VectorDescriptor.kt | 9 +- .../model/query/proximity/ProximityQuery.kt | 4 +- .../vitrivr/engine/core/model/types/Type.kt | 162 ++++++++++++++++-- .../vitrivr/engine/core/model/types/Value.kt | 58 +++++++ .../engine/plugin/cottontaildb/Common.kt | 62 +++---- .../CottontailConnectionProvider.kt | 8 +- .../struct/StructDescriptorInitializer.kt | 56 +----- .../struct/StructDescriptorReader.kt | 107 +++--------- .../struct/StructDescriptorWriter.kt | 10 +- .../vector/VectorDescriptorReader.kt | 22 +-- .../database/string/StringConnection.kt | 3 + .../feature/averagecolor/AverageColor.kt | 8 +- .../averagecolor/AverageColorRetriever.kt | 4 +- .../AverageColorRaster.kt | 5 +- .../averagecolorraster/RasterDescriptor.kt | 37 ++++ .../feature/external/ExternalAnalyser.kt | 2 +- .../external/implementations/clip/CLIP.kt | 8 +- .../external/implementations/dino/DINO.kt | 6 +- .../feature/migration/AverageColorGrid8.kt | 2 +- .../migration/AverageColorGrid8Reduced15.kt | 2 +- .../feature/migration/AverageFuzzyHist.kt | 2 +- .../migration/AverageFuzzyHistNormalized.kt | 2 +- .../module/features/feature/migration/CLD.kt | 2 +- .../feature/migration/CLDReduced15.kt | 2 +- .../feature/migration/ConceptMasksADE20k.kt | 2 +- .../feature/migration/DominantEdgeGrid.kt | 2 +- .../feature/migration/DominantEdgeGrid16.kt | 2 +- .../feature/migration/DominantEdgeGrid8.kt | 2 +- .../module/features/feature/migration/EHD.kt | 2 +- .../features/feature/migration/EdgeARP88.kt | 2 +- .../features/feature/migration/EdgeGrid16.kt | 2 +- .../features/feature/migration/HOGMF25k512.kt | 2 +- .../feature/migration/HueHistogram.kt | 2 +- .../feature/migration/InceptionResNetV2.kt | 2 +- .../features/feature/migration/MedianColor.kt | 2 +- .../feature/migration/MedianFuzzyHist.kt | 2 +- .../features/feature/migration/OpenCLIP.kt | 2 +- .../feature/migration/SURFMF25K512.kt | 2 +- .../migration/VisualTextCoEmbedding.kt | 2 +- .../feature/skeleton/SkeletonDescriptor.kt | 37 ++++ .../{migration => skeleton}/SkeletonPose.kt | 19 +- ...trivr.engine.core.model.metamodel.Analyser | 4 +- .../implementations/DenseEmbedding.kt | 10 +- .../implementations/ImageClassification.kt | 8 +- .../pgvector/PgVectorConnectionProvider.kt | 6 +- .../pgvector/descriptor/model/PgBitVector.kt | 3 +- .../struct/StructDescriptorInitializer.kt | 24 +-- .../struct/StructDescriptorReader.kt | 23 ++- .../struct/StructDescriptorWriter.kt | 103 ++++++----- .../vector/VectorDescriptorReader.kt | 44 ++--- .../vector/VectorDescriptorWriter.kt | 21 ++- .../transform/lookup/FieldLookupFactory.kt | 2 +- .../engine/query/parsing/QueryParser.kt | 20 +-- .../engine/tools/CineastMigrationTool.kt | 40 +++-- 86 files changed, 915 insertions(+), 833 deletions(-) create mode 100644 vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Attribute.kt delete mode 100644 vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/FieldSchema.kt create mode 100644 vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/MapStructDescriptor.kt delete mode 100644 vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/RasterDescriptor.kt delete mode 100644 vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/SkeletonDescriptor.kt delete mode 100644 vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/MapStructDescriptor.kt rename vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/{migration => averagecolorraster}/AverageColorRaster.kt (86%) create mode 100644 vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolorraster/RasterDescriptor.kt create mode 100644 vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/skeleton/SkeletonDescriptor.kt rename vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/{migration => skeleton}/SkeletonPose.kt (84%) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadata.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadata.kt index f001c534..f4279824 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadata.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadata.kt @@ -1,16 +1,16 @@ package org.vitrivr.engine.core.features.metadata.source.exif -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging import org.vitrivr.engine.core.context.IndexContext import org.vitrivr.engine.core.context.QueryContext import org.vitrivr.engine.core.model.content.element.ContentElement -import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.MapStructDescriptor +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.metamodel.Analyser import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query import org.vitrivr.engine.core.model.query.bool.SimpleBooleanQuery import org.vitrivr.engine.core.model.retrievable.Retrievable +import org.vitrivr.engine.core.model.types.Type import org.vitrivr.engine.core.operators.Operator import org.vitrivr.engine.core.operators.ingest.Extractor import org.vitrivr.engine.core.operators.retrieve.Retriever @@ -18,11 +18,17 @@ import java.util.* class ExifMetadata : Analyser, MapStructDescriptor> { - private val logger: KLogger = KotlinLogging.logger {} - override val contentClasses = setOf(ContentElement::class) override val descriptorClass = MapStructDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = MapStructDescriptor.prototype(field.parameters) + override fun prototype(field: Schema.Field<*, *>): MapStructDescriptor { + val parameters = field.parameters.map { (k, v) -> Attribute(k, Type.valueOf(v)) } + return MapStructDescriptor( + UUID.randomUUID(), + UUID.randomUUID(), + parameters, + parameters.associate { it.name to it.type.defaultValue() }, + ) + } override fun newExtractor( name: String, input: Operator, diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataExtractor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataExtractor.kt index 0a1f5f80..a215a7c4 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataExtractor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataExtractor.kt @@ -9,7 +9,9 @@ import io.github.oshai.kotlinlogging.KLogger import io.github.oshai.kotlinlogging.KotlinLogging import org.vitrivr.engine.core.features.AbstractExtractor import org.vitrivr.engine.core.model.content.element.ContentElement -import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.MapStructDescriptor +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.Retrievable import org.vitrivr.engine.core.model.retrievable.attributes.SourceAttribute @@ -45,33 +47,32 @@ private fun convertDate(date: String): Date? { } -private fun convertType(directory: Directory, tagType: Int, type: Type): Value<*>? = - when (type) { - Type.STRING -> Value.String(directory.getString(tagType)) - Type.BOOLEAN -> Value.Boolean(directory.getBoolean(tagType)) - Type.BYTE -> Value.Byte(directory.getObject(tagType) as Byte) - Type.SHORT -> Value.Short(directory.getObject(tagType) as Short) - Type.INT -> Value.Int(directory.getInt(tagType)) - Type.LONG -> Value.Long(directory.getLong(tagType)) - Type.FLOAT -> Value.Float(directory.getFloat(tagType)) - Type.DOUBLE -> Value.Double(directory.getDouble(tagType)) - Type.DATETIME -> convertDate(directory.getString(tagType))?.let { Value.DateTime(it) } - } +private fun convertType(directory: Directory, tagType: Int, type: Type): Value<*>? = when (type) { + Type.Boolean -> Value.Boolean(directory.getBoolean(tagType)) + Type.Byte -> Value.Byte(directory.getObject(tagType) as Byte) + Type.Datetime -> convertDate(directory.getString(tagType))?.let { Value.DateTime(it) } + Type.Double -> Value.Double(directory.getDouble(tagType)) + Type.Float -> Value.Float(directory.getFloat(tagType)) + Type.Int -> Value.Int(directory.getInt(tagType)) + Type.Long -> Value.Long(directory.getLong(tagType)) + Type.Short -> Value.Short(directory.getObject(tagType) as Short) + Type.String -> Value.String(directory.getString(tagType)) + else -> throw IllegalArgumentException("Unsupported type: $type") +} private fun JsonElement.convertType(type: Type): Value<*>? { - if (this.isJsonNull) { - return null - } + if (this.isJsonNull) return null return when (type) { - Type.STRING -> Value.String(this.asString) - Type.BOOLEAN -> Value.Boolean(this.asBoolean) - Type.BYTE -> Value.Byte(this.asByte) - Type.SHORT -> Value.Short(this.asShort) - Type.INT -> Value.Int(this.asInt) - Type.LONG -> Value.Long(this.asLong) - Type.FLOAT -> Value.Float(this.asFloat) - Type.DOUBLE -> Value.Double(this.asDouble) - Type.DATETIME -> convertDate(this.asString)?.let { Value.DateTime(it) } + Type.String -> Value.String(this.asString) + Type.Boolean -> Value.Boolean(this.asBoolean) + Type.Byte -> Value.Byte(this.asByte) + Type.Short -> Value.Short(this.asShort) + Type.Int -> Value.Int(this.asInt) + Type.Long -> Value.Long(this.asLong) + Type.Float -> Value.Float(this.asFloat) + Type.Double -> Value.Double(this.asDouble) + Type.Datetime -> convertDate(this.asString)?.let { Value.DateTime(it) } + else -> throw IllegalArgumentException("Unsupported type: $type") } } @@ -86,25 +87,23 @@ class ExifMetadataExtractor( override fun extract(retrievable: Retrievable): List { val metadata = ImageMetadataReader.readMetadata((retrievable.filteredAttribute(SourceAttribute::class.java)?.source as FileSource).path.toFile()) - val columnValues = mutableMapOf>() - - val subfields = this.field!!.parameters - + val columnValues = mutableMapOf>() + val attributes = this.field?.parameters?.map { (k, v) -> k to Attribute(k, Type.valueOf(v)) }?.toMap() ?: emptyMap() for (directory in metadata.directories) { for (tag in directory.tags) { val tagname = tag.tagName.replace(NON_ALPHANUMERIC_REGEX, "") val fullname = "${directory.name.replace(NON_ALPHANUMERIC_REGEX, "")}_$tagname" if (fullname == "ExifSubIFD_UserComment" || fullname == "JpegComment_JPEGComment") { - if (fullname in subfields){ + if (fullname in attributes) { columnValues[fullname] = Value.String(tag.description) } try { val json = JsonParser.parseString(tag.description).asJsonObject json.entrySet().forEach { (key, value) -> - subfields[key]?.let { typeString -> - value.convertType(Type.valueOf(typeString))?.let { converted -> + attributes[key]?.let { attribute -> + value.convertType(attribute.type)?.let { converted -> columnValues[key] = converted } } @@ -113,17 +112,16 @@ class ExifMetadataExtractor( logger.warn { "Failed to parse JSON from $fullname: ${tag.description}" } } } else { - subfields[fullname]?.let { typeString -> - convertType(directory, tag.tagType, Type.valueOf(typeString))?.let { converted -> + attributes[fullname]?.let { attribute -> + convertType(directory, tag.tagType, attribute.type)?.let { converted -> columnValues[fullname] = converted } } - } } } logger.info { "Extracted fields: ${columnValues.entries.joinToString { (key, value) -> "$key = ${value.value}" }}" } - return listOf(MapStructDescriptor(UUID.randomUUID(), retrievable.id, subfields, columnValues.mapValues { it.value.value }, field = this.field)) + return listOf(MapStructDescriptor(UUID.randomUUID(), retrievable.id, attributes.values.toList(), columnValues.mapValues { it.value }, field = this.field)) } } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataRetriever.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataRetriever.kt index 1ba09ad8..3312c140 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataRetriever.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataRetriever.kt @@ -3,7 +3,7 @@ package org.vitrivr.engine.core.features.metadata.source.exif import org.vitrivr.engine.core.context.QueryContext import org.vitrivr.engine.core.features.AbstractRetriever import org.vitrivr.engine.core.model.content.element.ContentElement -import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.MapStructDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/file/FileSourceMetadataExtractor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/file/FileSourceMetadataExtractor.kt index 3abb8a02..48c46890 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/file/FileSourceMetadataExtractor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/file/FileSourceMetadataExtractor.kt @@ -20,7 +20,7 @@ import kotlin.io.path.absolutePathString * An [Extractor] that extracts [FileSourceMetadataDescriptor]s from [Ingested] objects. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ class FileSourceMetadataExtractor(input: Operator, field: Schema.Field, FileSourceMetadataDescriptor>?) : AbstractExtractor, FileSourceMetadataDescriptor>(input, field) { /** @@ -47,8 +47,10 @@ class FileSourceMetadataExtractor(input: Operator, field: Schema.Fi FileSourceMetadataDescriptor( id = UUID.randomUUID(), retrievableId = retrievable.id, - path = Value.String(source.path.absolutePathString()), - size = Value.Long(Files.size(source.path)), + mapOf( + "path" to Value.String(source.path.absolutePathString()), + "size" to Value.Long(Files.size(source.path)) + ), this@FileSourceMetadataExtractor.field ) ) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/video/VideoSourceMetadataExtractor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/video/VideoSourceMetadataExtractor.kt index d8321e1f..8360bc1f 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/video/VideoSourceMetadataExtractor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/video/VideoSourceMetadataExtractor.kt @@ -19,7 +19,7 @@ import java.util.* * An [Extractor] that extracts [VideoSourceMetadataDescriptor]s from [Ingested] objects. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ class VideoSourceMetadataExtractor(input: Operator, field: Schema.Field, VideoSourceMetadataDescriptor>?) : AbstractExtractor, VideoSourceMetadataDescriptor>(input, field) { /** @@ -46,12 +46,14 @@ class VideoSourceMetadataExtractor(input: Operator, field: Schema.F VideoSourceMetadataDescriptor( id = UUID.randomUUID(), retrievableId = retrievable.id, - width = Value.Int(source.width() ?: 0), - height = Value.Int(source.height() ?: 0), - fps = Value.Double(source.fps() ?: 0.0), - channels = Value.Int(source.channels() ?: 0), - sampleRate = Value.Int(source.sampleRate() ?: 0), - sampleSize = Value.Int(source.sampleSize() ?: 0), + mapOf( + "width" to Value.Int(source.width() ?: 0), + "height" to Value.Int(source.height() ?: 0), + "fps" to Value.Double(source.fps() ?: 0.0), + "channels" to Value.Int(source.channels() ?: 0), + "sampleRate" to Value.Int(source.sampleRate() ?: 0), + "sampleSize" to Value.Int(source.sampleSize() ?: 0) + ), this@VideoSourceMetadataExtractor.field ) ) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/temporal/TemporalMetadataExtractor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/temporal/TemporalMetadataExtractor.kt index 5d9f0b15..f4dee4fe 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/temporal/TemporalMetadataExtractor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/temporal/TemporalMetadataExtractor.kt @@ -18,7 +18,7 @@ import java.util.* * An [Extractor] that extracts [TemporalMetadataDescriptor]s from [Ingested] objects. * * @author Ralph Gasser - * @version 1.2.0 + * @version 1.3.0 */ class TemporalMetadataExtractor(input: Operator, field: Schema.Field, TemporalMetadataDescriptor>?) : AbstractExtractor, TemporalMetadataDescriptor>(input, field) { @@ -33,10 +33,10 @@ class TemporalMetadataExtractor(input: Operator, field: Schema.Fiel override fun extract(retrievable: Retrievable): List { if (retrievable.hasAttribute(TimePointAttribute::class.java)) { val timestamp = retrievable.filteredAttribute(TimePointAttribute::class.java)!! - return listOf(TemporalMetadataDescriptor(UUID.randomUUID(), retrievable.id, Value.Long(timestamp.timepointNs), Value.Long(timestamp.timepointNs), this@TemporalMetadataExtractor.field)) + return listOf(TemporalMetadataDescriptor(UUID.randomUUID(), retrievable.id, mapOf("start" to Value.Long(timestamp.timepointNs), "end" to Value.Long(timestamp.timepointNs)), this@TemporalMetadataExtractor.field)) } else if (retrievable.hasAttribute(TimeRangeAttribute::class.java)) { val span = retrievable.filteredAttribute(TimeRangeAttribute::class.java)!! - return listOf(TemporalMetadataDescriptor(UUID.randomUUID(), retrievable.id, Value.Long(span.startNs), Value.Long(span.endNs), this@TemporalMetadataExtractor.field)) + return listOf(TemporalMetadataDescriptor(UUID.randomUUID(), retrievable.id, mapOf("start" to Value.Long(span.startNs), "end" to Value.Long(span.endNs)), this@TemporalMetadataExtractor.field)) } return emptyList() } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/color/RGBFloatColorContainer.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/color/RGBFloatColorContainer.kt index fe6c35ef..3218bc69 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/color/RGBFloatColorContainer.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/color/RGBFloatColorContainer.kt @@ -67,7 +67,7 @@ open class RGBFloatColorContainer(open val red: Float, open val green: Float, op * * @return [List] of [Value.Float]s */ - fun toValueList() = listOf(Value.Float(this.red), Value.Float(this.green), Value.Float(this.blue)) + fun toVector() = Value.FloatVector(floatArrayOf(this.red, this.green, this.blue)) fun distanceTo(other: RGBFloatColorContainer): Float = sqrt( (this.red - other.red) * (this.red - other.red) + (this.green - other.green) * (this.green - other.green) + (this.blue - other.blue) * (this.blue - other.blue) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Attribute.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Attribute.kt new file mode 100644 index 00000000..ebe0e895 --- /dev/null +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Attribute.kt @@ -0,0 +1,15 @@ +package org.vitrivr.engine.core.model.descriptor + +import org.vitrivr.engine.core.model.types.Type + +/** The name of an attribute. */ +typealias AttributeName = String + +/** + * A [Attribute] describes a field in a [Descriptor]. This is required for a [Descriptor]'s ability to + * describe its own schema. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +data class Attribute(val name: AttributeName, val type: Type, val nullable: Boolean = false, val indexed: Boolean = false) \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt index 73a953cb..465968df 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt @@ -4,6 +4,7 @@ import org.vitrivr.engine.core.model.Persistable import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.Retrievable import org.vitrivr.engine.core.model.retrievable.RetrievableId +import org.vitrivr.engine.core.model.types.Value import java.util.* /** A typealias to identify the [UUID] identifying a [Descriptor]. */ @@ -31,16 +32,16 @@ interface Descriptor : Persistable { get() = this.field != null /** - * Returns a [FieldSchema] [List] for this [Descriptor]. + * Returns a [Attribute] [List] for this [Descriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - fun schema(): List + fun schema(): List /** * Returns the fields and its values of this [Descriptor] as a [Map]. * * @return A [Map] of this [Descriptor]'s fields (without the IDs). */ - fun values(): List> + fun values(): Map?> } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/FieldSchema.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/FieldSchema.kt deleted file mode 100644 index baa6529a..00000000 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/FieldSchema.kt +++ /dev/null @@ -1,12 +0,0 @@ -package org.vitrivr.engine.core.model.descriptor - -import org.vitrivr.engine.core.model.types.Type - -/** - * A [FieldSchema] describes a field in a [Descriptor]. This is required for a [Descriptor]'s ability to - * describe its own schema. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -data class FieldSchema(val name: String, val type: Type, val dimensions: IntArray = intArrayOf(), val nullable: Boolean = false) \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt index e1334a2a..1e1f71d9 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt @@ -1,7 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.scalar import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,13 +21,13 @@ data class BooleanDescriptor( override val field: Schema.Field<*, BooleanDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(FieldSchema("value", Type.BOOLEAN)) + private val SCHEMA = listOf(Attribute("value", Type.Boolean)) } /** - * Returns the [FieldSchema] [List ]of this [BooleanDescriptor]. + * Returns the [Attribute] [List ]of this [BooleanDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun schema(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt index 8cb8e234..7196d5ee 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt @@ -1,7 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.scalar import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,13 +21,13 @@ data class DoubleDescriptor( override val field: Schema.Field<*, DoubleDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(FieldSchema("value", Type.DOUBLE)) + private val SCHEMA = listOf(Attribute("value", Type.Double)) } /** - * Returns the [FieldSchema] [List] of this [DoubleDescriptor]. + * Returns the [Attribute] [List] of this [DoubleDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun schema(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt index f0f0bf7f..68159726 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt @@ -1,7 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.scalar import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,13 +21,13 @@ data class FloatDescriptor( override val field: Schema.Field<*, FloatDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(FieldSchema("value", Type.FLOAT)) + private val SCHEMA = listOf(Attribute("value", Type.Float)) } /** - * Returns the [FieldSchema] [List] of this [FloatDescriptor]. + * Returns the [Attribute] [List] of this [FloatDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun schema(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt index cbfb3ba8..9184a2eb 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt @@ -1,7 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.scalar import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,13 +21,13 @@ data class IntDescriptor( override val field: Schema.Field<*, IntDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(FieldSchema("value", Type.INT)) + private val SCHEMA = listOf(Attribute("value", Type.Int)) } /** - * Returns the [FieldSchema] [List] of this [IntDescriptor]. + * Returns the [Attribute] [List] of this [IntDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun schema(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt index a9a56df2..f8dff29d 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt @@ -1,7 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.scalar import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,13 +21,13 @@ data class LongDescriptor( override val field: Schema.Field<*, LongDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(FieldSchema("value", Type.LONG)) + private val SCHEMA = listOf(Attribute("value", Type.Long)) } /** - * Returns the [FieldSchema] [List] of this [LongDescriptor]. + * Returns the [Attribute] [List] of this [LongDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun schema(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/ScalarDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/ScalarDescriptor.kt index 6be50862..23728002 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/ScalarDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/ScalarDescriptor.kt @@ -1,5 +1,6 @@ package org.vitrivr.engine.core.model.descriptor.scalar +import org.vitrivr.engine.core.model.descriptor.AttributeName import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.types.Value @@ -7,7 +8,7 @@ import org.vitrivr.engine.core.model.types.Value * A [Descriptor] with a scalar value [T]. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ sealed interface ScalarDescriptor> : Descriptor { @@ -19,5 +20,5 @@ sealed interface ScalarDescriptor> : Descriptor { * * @return A [Map] of this [ScalarDescriptor]'s fields (without the IDs). */ - override fun values(): List> = listOf("value" to this.value) + override fun values(): Map = mapOf("value" to this.value) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt index e6615886..0f050b7e 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt @@ -1,7 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.scalar import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,13 +21,13 @@ data class StringDescriptor( override val field: Schema.Field<*, StringDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(FieldSchema("value", Type.STRING)) + private val SCHEMA = listOf(Attribute("value", Type.String)) } /** - * Returns the [FieldSchema] [List ]of this [StringDescriptor]. + * Returns the [Attribute] [List ]of this [StringDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun schema(): List = SCHEMA } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/LabelDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/LabelDescriptor.kt index 125b5c6b..33e7896e 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/LabelDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/LabelDescriptor.kt @@ -1,7 +1,8 @@ package org.vitrivr.engine.core.model.descriptor.struct import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -11,31 +12,24 @@ import org.vitrivr.engine.core.model.types.Value * A [StructDescriptor] used to store a string label and a confidence for that label. * * @author Ralph Gasser - * @version 1.0.0 + * @version 2.0.0 */ -data class LabelDescriptor( +class LabelDescriptor( override var id: DescriptorId, override var retrievableId: RetrievableId?, - val label: Value.String, - val confidence: Value.Float, + values: Map?>, override val field: Schema.Field<*, LabelDescriptor>? = null -) : StructDescriptor { +) : MapStructDescriptor(id, retrievableId, SCHEMA, values, field) { companion object { private val SCHEMA = listOf( - FieldSchema("label", Type.STRING), - FieldSchema("confidence", Type.FLOAT), + Attribute("label", Type.String), + Attribute("confidence", Type.Float), ) } - /** - * Returns the [FieldSchema] [List ]of this [LabelDescriptor]. - * - * @return [List] of [FieldSchema] - */ - override fun schema(): List = SCHEMA + /** The stored label. */ + val label: Value.String by this.values - override fun values(): List> = listOf( - "label" to this.label, - "confidence" to this.confidence - ) + /** The associated confidence. */ + val confidence: Value.Float by this.values } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/MapStructDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/MapStructDescriptor.kt new file mode 100644 index 00000000..d3ecd7f0 --- /dev/null +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/MapStructDescriptor.kt @@ -0,0 +1,37 @@ +package org.vitrivr.engine.core.model.descriptor.struct + +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.Descriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.retrievable.RetrievableId +import org.vitrivr.engine.core.model.types.Value + +/** + * An abstract [StructDescriptor] implementation that is backed by a [Map] of [Value]s. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +open class MapStructDescriptor( + override var id: DescriptorId, + override var retrievableId: RetrievableId?, + protected val layout: List, + protected val values: Map?>, + override val field: Schema.Field<*, out StructDescriptor>? = null +) : StructDescriptor { + /** + * Returns a [Attribute] [List] for this [Descriptor]. + * + * @return [List] of [Attribute] + */ + final override fun schema(): List = this.layout + + /** + * Returns the fields and its values of this [Descriptor] as a [Map]. + * + * @return A [Map] of this [Descriptor]'s fields (without the IDs). + */ + final override fun values(): Map?> = this.values +} \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/RasterDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/RasterDescriptor.kt deleted file mode 100644 index 72b89c5a..00000000 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/RasterDescriptor.kt +++ /dev/null @@ -1,33 +0,0 @@ -package org.vitrivr.engine.core.model.descriptor.struct - -import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.core.model.retrievable.RetrievableId -import org.vitrivr.engine.core.model.types.Type -import org.vitrivr.engine.core.model.types.Value - -data class RasterDescriptor( - override var id: DescriptorId, - override var retrievableId: RetrievableId?, - val hist: List, - val raster: List, - override val field: Schema.Field<*, RasterDescriptor>? = null -) : StructDescriptor { - - - /** - * Returns the [FieldSchema] [List ]of this [RasterDescriptor]. - * - * @return [List] of [FieldSchema] - */ - override fun schema(): List = listOf( - FieldSchema("hist", Type.FLOAT, intArrayOf(this.hist.size)), - FieldSchema("raster", Type.FLOAT, intArrayOf(this.raster.size)) - ) - - override fun values(): List> = listOf( - "hist" to this.hist, - "raster" to this.raster - ) -} \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/SkeletonDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/SkeletonDescriptor.kt deleted file mode 100644 index 8d3ec7d9..00000000 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/SkeletonDescriptor.kt +++ /dev/null @@ -1,35 +0,0 @@ -package org.vitrivr.engine.core.model.descriptor.struct - -import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.core.model.retrievable.RetrievableId -import org.vitrivr.engine.core.model.types.Type -import org.vitrivr.engine.core.model.types.Value - -data class SkeletonDescriptor( - override var id: DescriptorId, - override var retrievableId: RetrievableId?, - val person: Value.Int, - val skeleton: List, - val weights: List, - override val field: Schema.Field<*, SkeletonDescriptor>? = null -) : StructDescriptor { - - /** - * Returns the [FieldSchema] [List ]of this [SkeletonDescriptor]. - * - * @return [List] of [FieldSchema] - */ - override fun schema(): List = listOf( - FieldSchema("person", Type.INT), - FieldSchema("skeleton", Type.FLOAT, intArrayOf(this.skeleton.size)), - FieldSchema("weights", Type.FLOAT, intArrayOf(this.weights.size)) - ) - - override fun values(): List> = listOf( - "person" to this.person, - "skeleton" to this.skeleton, - "weights" to this.weights - ) -} \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/MediaDimensionsDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/MediaDimensionsDescriptor.kt index de28fa22..f38028ba 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/MediaDimensionsDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/MediaDimensionsDescriptor.kt @@ -1,7 +1,9 @@ package org.vitrivr.engine.core.model.descriptor.struct.metadata import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.VideoSourceMetadataDescriptor import org.vitrivr.engine.core.model.metamodel.Schema @@ -14,41 +16,35 @@ import java.util.* * A [StructDescriptor] used to store metadata about a 2D raster graphic. * * @author Ralph Gasser - * @version 1.0.0 + * @version 2.0.0 */ -data class MediaDimensionsDescriptor( +class MediaDimensionsDescriptor( override var id: DescriptorId, override var retrievableId: RetrievableId?, - val width: Value.Int, - val height: Value.Int, + values: Map?>, override val field: Schema.Field<*, MediaDimensionsDescriptor>? = null -) : StructDescriptor { +) : MapStructDescriptor(id, retrievableId, SCHEMA, values, field) { + companion object { + /** The field schema associated with a [VideoSourceMetadataDescriptor]. */ + private val SCHEMA = listOf( + Attribute("width", Type.Int), + Attribute("height", Type.Int), + ) - companion object { - /** The field schema associated with a [VideoSourceMetadataDescriptor]. */ - private val SCHEMA = listOf( - FieldSchema("width", Type.INT), - FieldSchema("height", Type.INT), + /** The prototype [MediaDimensionsDescriptor]. */ + val PROTOTYPE = MediaDimensionsDescriptor( + UUID.randomUUID(), + UUID.randomUUID(), + mapOf( + "width" to Value.Int(0), + "height" to Value.Int(0) ) + ) + } - /** The prototype [MediaDimensionsDescriptor]. */ - val PROTOTYPE = MediaDimensionsDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.Int(0), Value.Int(0)) - } - - /** - * Returns the [FieldSchema] [List ]of this [StructDescriptor]. - * - * @return [List] of [FieldSchema] - */ - override fun schema(): List = SCHEMA + /** The width of this image. */ + val width: Value.Int by this.values - /** - * Returns the fields and its values of this [MediaDimensionsDescriptor] as a [Map]. - * - * @return A [Map] of this [MediaDimensionsDescriptor]'s fields (without the IDs). - */ - override fun values(): List> = listOf( - "width" to this.width, - "height" to this.height - ) + /** The height of this image. */ + val height: Value.Int by this.values } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/Rectangle2DMetadataDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/Rectangle2DMetadataDescriptor.kt index 70c4fa32..acba69ce 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/Rectangle2DMetadataDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/Rectangle2DMetadataDescriptor.kt @@ -1,7 +1,9 @@ package org.vitrivr.engine.core.model.descriptor.struct.metadata import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId @@ -13,47 +15,46 @@ import java.util.* * A [StructDescriptor] used to store spatial metadata in a 2D raster graphic. * * @author Ralph Gasser - * @version 1.0.0 + * @version 2.0.0 */ -data class Rectangle2DMetadataDescriptor( +class Rectangle2DMetadataDescriptor( override var id: DescriptorId, override var retrievableId: RetrievableId?, - val leftX: Value.Int, - val leftY: Value.Int, - val width: Value.Int, - val height: Value.Int, + values: Map?>, override val field: Schema.Field<*, Rectangle2DMetadataDescriptor>? = null -) : StructDescriptor { +) : MapStructDescriptor(id, retrievableId, SCHEMA, values, field) { companion object { /** The field schema associated with a [Rectangle2DMetadataDescriptor]. */ private val SCHEMA = listOf( - FieldSchema("leftX", Type.INT), - FieldSchema("leftY", Type.INT), - FieldSchema("width", Type.INT), - FieldSchema("height", Type.INT), + Attribute("leftX", Type.Int), + Attribute("leftY", Type.Int), + Attribute("width", Type.Int), + Attribute("height", Type.Int), ) /** The prototype [Rectangle2DMetadataDescriptor]. */ - val PROTOTYPE = Rectangle2DMetadataDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.Int(0), Value.Int(0), Value.Int(0), Value.Int(0)) + val PROTOTYPE = Rectangle2DMetadataDescriptor( + UUID.randomUUID(), + UUID.randomUUID(), + mapOf( + "leftX" to Value.Int(0), + "leftY" to Value.Int(0), + "width" to Value.Int(0), + "height" to Value.Int(0) + ) + ) } - /** - * Returns the [FieldSchema] [List ]of this [StructDescriptor]. - * - * @return [List] of [FieldSchema] - */ - override fun schema(): List = SCHEMA - - /** - * Returns the fields and its values of this [Rectangle2DMetadataDescriptor] as a [Map]. - * - * @return A [Map] of this [Rectangle2DMetadataDescriptor]'s fields (without the IDs). - */ - override fun values(): List> = listOf( - "leftX" to this.leftX, - "leftY" to this.leftY, - "width" to this.width, - "height" to this.height - ) + /** The top left pixel of the rectangle (X-component). */ + val leftX: Value.Int by this.values + + /** The top left pixel of the rectangle (Y-component). */ + val leftY: Value.Int by this.values + + /** The width of the rectangle. */ + val width: Value.Int by this.values + + /** The height of the rectangle. */ + val height: Value.Int by this.values } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/TemporalMetadataDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/TemporalMetadataDescriptor.kt index fd3b36f0..8f027226 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/TemporalMetadataDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/TemporalMetadataDescriptor.kt @@ -1,7 +1,9 @@ package org.vitrivr.engine.core.model.descriptor.struct.metadata import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId @@ -13,41 +15,29 @@ import java.util.* * A [StructDescriptor] used to store temporal metadata. * * @author Ralph Gasser - * @version 1.0.0 + * @version 2.0.0 */ -data class TemporalMetadataDescriptor( +class TemporalMetadataDescriptor( override var id: DescriptorId, override var retrievableId: RetrievableId?, //retrievable Id must come first, due to reflection - val startNs: Value.Long, - val endNs: Value.Long, + values: Map?>, override val field: Schema.Field<*, TemporalMetadataDescriptor>? = null -) : StructDescriptor { +) : MapStructDescriptor(id, retrievableId, SCHEMA, values, field) { companion object { /** The field schema associated with a [TemporalMetadataDescriptor]. */ private val SCHEMA = listOf( - FieldSchema("start", Type.LONG), - FieldSchema("end", Type.LONG), + Attribute("start", Type.Long), + Attribute("end", Type.Long), ) /** The prototype [TemporalMetadataDescriptor]. */ - val PROTOTYPE = TemporalMetadataDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.Long(0L), Value.Long(0L)) + val PROTOTYPE = TemporalMetadataDescriptor(UUID.randomUUID(), UUID.randomUUID(), mapOf("start" to Value.Long(0L), "end" to Value.Long(0L))) } - /** - * Returns the [FieldSchema] [List ]of this [TemporalMetadataDescriptor]. - * - * @return [List] of [FieldSchema] - */ - override fun schema(): List = SCHEMA + /** The start timestamp in nanoseconds. */ + val start: Value.Long by this.values - /** - * Returns the fields and its values of this [TemporalMetadataDescriptor] as a [Map]. - * - * @return A [Map] of this [TemporalMetadataDescriptor]'s fields (without the IDs). - */ - override fun values(): List> = listOf( - "start" to this.startNs, - "end" to this.endNs - ) + /** The end timestamp in nanoseconds. */ + val end: Value.Long by this.values } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/FileSourceMetadataDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/FileSourceMetadataDescriptor.kt index 1de8e8c1..664298a3 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/FileSourceMetadataDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/FileSourceMetadataDescriptor.kt @@ -1,7 +1,9 @@ package org.vitrivr.engine.core.model.descriptor.struct.metadata.source import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId @@ -13,41 +15,29 @@ import java.util.* * A [StructDescriptor] used to store metadata about a file. * * @author Ralph Gasser - * @version 1.0.0 + * @version 2.0.0 */ -data class FileSourceMetadataDescriptor( +class FileSourceMetadataDescriptor( override var id: DescriptorId, override var retrievableId: RetrievableId?, - val path: Value.String, - val size: Value.Long, + values: Map?>, override val field: Schema.Field<*, FileSourceMetadataDescriptor>? = null -) : StructDescriptor { +) : MapStructDescriptor(id, retrievableId, SCHEMA, values, field) { + + /** The path to the file. */ + val path: Value.String by this.values + + /** The size of the file in bytes. */ + val size: Value.Long by this.values companion object { /** The field schema associated with a [FileSourceMetadataDescriptor]. */ private val SCHEMA = listOf( - FieldSchema("path", Type.STRING), - FieldSchema("size", Type.LONG), + Attribute("path", Type.String), + Attribute("size", Type.Long), ) /** The prototype [FileSourceMetadataDescriptor]. */ - val PROTOTYPE = FileSourceMetadataDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.String(""), Value.Long(0L)) + val PROTOTYPE = FileSourceMetadataDescriptor(UUID.randomUUID(), UUID.randomUUID(), mapOf("path" to Value.String(""), "size" to Value.Long(0L))) } - - /** - * Returns the [FieldSchema] [List ]of this [StructDescriptor]. - * - * @return [List] of [FieldSchema] - */ - override fun schema(): List = SCHEMA - - /** - * Returns the fields and its values of this [FileSourceMetadataDescriptor] as a [Map]. - * - * @return A [Map] of this [FileSourceMetadataDescriptor]'s fields (without the IDs). - */ - override fun values(): List> = listOf( - "path" to this.path, - "size" to this.size - ) } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/MapStructDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/MapStructDescriptor.kt deleted file mode 100644 index 53e94935..00000000 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/MapStructDescriptor.kt +++ /dev/null @@ -1,65 +0,0 @@ -package org.vitrivr.engine.core.model.descriptor.struct.metadata.source - -import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema -import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.core.model.retrievable.RetrievableId -import org.vitrivr.engine.core.model.types.Type -import org.vitrivr.engine.core.model.types.Value -import java.util.* - -data class MapStructDescriptor( - override var id: DescriptorId, - override var retrievableId: RetrievableId?, - val columnTypes: Map, - val columnValues: Map, - override val transient: Boolean = false, - override val field: Schema.Field<*, MapStructDescriptor>? = null -) : StructDescriptor { - - companion object{ - fun prototype(columnTypes: Map): MapStructDescriptor { - val columnValues = columnTypes.mapValues { (_, type) -> - when (Type.valueOf(type)) { - Type.STRING -> "" - Type.BOOLEAN -> false - Type.BYTE -> 0.toByte() - Type.SHORT -> 0.toShort() - Type.INT -> 0 - Type.LONG -> 0L - Type.FLOAT -> 0.0f - Type.DOUBLE -> 0.0 - Type.DATETIME -> Date() - } - } - return MapStructDescriptor(UUID.randomUUID(), UUID.randomUUID(), columnTypes, columnValues) - - } - } - - override fun schema(): List { - return this.columnTypes.map { (key, type) -> - FieldSchema(key, Type.valueOf(type), nullable=true) - } - } - - override fun values(): List> { - return this.columnTypes.map { (key, type) -> - val value = this.columnValues[key] // This will be null if key is not present in columnValues - val pairedValue = when (Type.valueOf(type)) { - Type.STRING -> (value as? Value.String) - Type.BOOLEAN -> (value as? Value.Boolean) - Type.BYTE -> (value as? Value.Byte) - Type.SHORT -> (value as? Value.Short) - Type.INT -> (value as? Value.Int) - Type.LONG -> (value as? Value.Long) - Type.FLOAT -> (value as? Value.Float) - Type.DOUBLE -> (value as? Value.Double) - Type.DATETIME -> (value as? Value.DateTime) - } - Pair(key, pairedValue) - } - - } -} diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/VideoSourceMetadataDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/VideoSourceMetadataDescriptor.kt index 9b226f10..248741d7 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/VideoSourceMetadataDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/VideoSourceMetadataDescriptor.kt @@ -1,7 +1,9 @@ package org.vitrivr.engine.core.model.descriptor.struct.metadata.source import org.vitrivr.engine.core.model.descriptor.DescriptorId -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId @@ -13,53 +15,56 @@ import java.util.* * A [StructDescriptor] used to store metadata about a video source (e.g., a file). * * @author Ralph Gasser - * @version 1.0.0 + * @version 2.0.0 */ class VideoSourceMetadataDescriptor( override var id: DescriptorId, override var retrievableId: RetrievableId?, - val width: Value.Int, - val height: Value.Int, - val fps: Value.Double, - val channels: Value.Int, - val sampleRate: Value.Int, - val sampleSize: Value.Int, + values: Map?>, override val field: Schema.Field<*, VideoSourceMetadataDescriptor>? = null -) : StructDescriptor { +) : MapStructDescriptor(id, retrievableId, SCHEMA, values, field) { + + /** The width of the video source in pixels. */ + val width: Value.Int by this.values + + /** The width of the video source in pixels. */ + val height: Value.Int by this.values + + /** The number of visual frames per seconds. */ + val fps: Value.Double by this.values + + /** The number of audio channels. */ + val channels: Value.Int by this.values + + /** The sample rate in kHz. */ + val sampleRate: Value.Int by this.values + + /** The sample size in bytes. */ + val sampleSize: Value.Int by this.values companion object { /** The field schema associated with a [VideoSourceMetadataDescriptor]. */ private val SCHEMA = listOf( - FieldSchema("width", Type.INT), - FieldSchema("height", Type.INT), - FieldSchema("fps", Type.DOUBLE), - FieldSchema("channels", Type.INT), - FieldSchema("sampleRate", Type.INT), - FieldSchema("sampleSize", Type.INT), + Attribute("width", Type.Int), + Attribute("height", Type.Int), + Attribute("fps", Type.Double), + Attribute("channels", Type.Int), + Attribute("sampleRate", Type.Int), + Attribute("sampleSize", Type.Int), ) /** The prototype [VideoSourceMetadataDescriptor]. */ - val PROTOTYPE = VideoSourceMetadataDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.Int(0), Value.Int(0), Value.Double(0.0), Value.Int(0), Value.Int(0), Value.Int(0)) + val PROTOTYPE = VideoSourceMetadataDescriptor( + UUID.randomUUID(), + UUID.randomUUID(), + mapOf( + "width" to Value.Int(0), + "height" to Value.Int(0), + "fps" to Value.Double(0.0), + "channels" to Value.Int(0), + "sampleRate" to Value.Int(0), + "sampleSize" to Value.Int(0) + ) + ) } - - /** - * Returns the [FieldSchema] [List ]of this [StructDescriptor]. - * - * @return [List] of [FieldSchema] - */ - override fun schema(): List = SCHEMA - - /** - * Returns the fields and its values of this [FileSourceMetadataDescriptor] as a [Map]. - * - * @return A [Map] of this [FileSourceMetadataDescriptor]'s fields (without the IDs). - */ - override fun values(): List> = listOf( - "width" to this.width, - "height" to this.height, - "fps" to this.fps, - "channels" to this.channels, - "sampleRate" to this.sampleRate, - "sampleSize" to this.sampleSize - ) } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt index d5dcc468..be823b66 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt @@ -1,6 +1,6 @@ package org.vitrivr.engine.core.model.descriptor.vector -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -12,19 +12,19 @@ import java.util.* * * @author Luca Rossetto * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ data class BooleanVectorDescriptor( override var id: UUID = UUID.randomUUID(), override var retrievableId: RetrievableId? = null, - override val vector: List, + override val vector: Value.BooleanVector, override val field: Schema.Field<*, BooleanVectorDescriptor>? = null -) : VectorDescriptor { +) : VectorDescriptor { /** - * Returns the [FieldSchema] [List ]of this [BooleanVectorDescriptor]. + * Returns the [Attribute] [List ]of this [BooleanVectorDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = listOf(FieldSchema("vector", Type.BOOLEAN, intArrayOf(this.dimensionality))) + override fun schema(): List = listOf(Attribute("vector", Type.BooleanVector(this.dimensionality))) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt index 46ec0081..0c3902a1 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt @@ -1,6 +1,6 @@ package org.vitrivr.engine.core.model.descriptor.vector -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -12,19 +12,19 @@ import java.util.* * * @author Luca Rossetto * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ data class DoubleVectorDescriptor( override var id: UUID = UUID.randomUUID(), override var retrievableId: RetrievableId? = null, - override val vector: List, + override val vector: Value.DoubleVector, override val field: Schema.Field<*, DoubleVectorDescriptor>? = null -) : VectorDescriptor { +) : VectorDescriptor { /** - * Returns the [FieldSchema] [List ]of this [DoubleVectorDescriptor]. + * Returns the [Attribute] [List ]of this [DoubleVectorDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = listOf(FieldSchema("vector", Type.DOUBLE, intArrayOf(this.dimensionality))) + override fun schema(): List = listOf(Attribute("vector", Type.DoubleVector(this.dimensionality))) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt index 6e9f4ace..06f6481f 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt @@ -1,6 +1,6 @@ package org.vitrivr.engine.core.model.descriptor.vector -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -13,19 +13,19 @@ import java.util.* * * @author Luca Rossetto * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ data class FloatVectorDescriptor( override var id: UUID = UUID.randomUUID(), override var retrievableId: RetrievableId? = null, - override val vector: List, + override val vector: Value.FloatVector, override val field: Schema.Field<*, FloatVectorDescriptor>? = null -) : VectorDescriptor { +) : VectorDescriptor { /** - * Returns the [FieldSchema] [List ]of this [FloatVectorDescriptor]. + * Returns the [Attribute] [List ]of this [FloatVectorDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = listOf(FieldSchema("vector", Type.FLOAT, intArrayOf(this.dimensionality))) + override fun schema(): List = listOf(Attribute("vector", Type.FloatVector(this.dimensionality))) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt index 4b3b42ee..fde0e8d3 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt @@ -1,6 +1,6 @@ package org.vitrivr.engine.core.model.descriptor.vector -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -12,19 +12,19 @@ import java.util.* * * @author Luca Rossetto * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ data class IntVectorDescriptor( override var id: UUID = UUID.randomUUID(), override var retrievableId: RetrievableId? = null, - override val vector: List, + override val vector: Value.IntVector, override val field: Schema.Field<*, IntVectorDescriptor>? = null -) : VectorDescriptor { +) : VectorDescriptor { /** - * Returns the [FieldSchema] [List ]of this [IntVectorDescriptor]. + * Returns the [Attribute] [List ]of this [IntVectorDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = listOf(FieldSchema("vector", Type.INT, intArrayOf(this.dimensionality))) + override fun schema(): List = listOf(Attribute("vector", Type.Int)) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt index 74d41884..2de95bcc 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt @@ -1,6 +1,6 @@ package org.vitrivr.engine.core.model.descriptor.vector -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -12,19 +12,19 @@ import java.util.* * * @author Luca Rossetto * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ data class LongVectorDescriptor( override var id: UUID = UUID.randomUUID(), override var retrievableId: RetrievableId? = null, - override val vector: List, + override val vector: Value.LongVector, override val field: Schema.Field<*, LongVectorDescriptor>? = null -) : VectorDescriptor { +) : VectorDescriptor { /** - * Returns the [FieldSchema] [List ]of this [LongVectorDescriptor]. + * Returns the [Attribute] [List ]of this [LongVectorDescriptor]. * - * @return [List] of [FieldSchema] + * @return [List] of [Attribute] */ - override fun schema(): List = listOf(FieldSchema("vector", Type.LONG, intArrayOf(this.dimensionality))) + override fun schema(): List = listOf(Attribute("vector", Type.Long)) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/VectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/VectorDescriptor.kt index ffbcc1be..a0594ace 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/VectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/VectorDescriptor.kt @@ -1,5 +1,6 @@ package org.vitrivr.engine.core.model.descriptor.vector +import org.vitrivr.engine.core.model.descriptor.AttributeName import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor import org.vitrivr.engine.core.model.types.Value @@ -9,20 +10,20 @@ import org.vitrivr.engine.core.model.types.Value * * @author Luca Rossetto * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ -sealed interface VectorDescriptor> : Descriptor { +sealed interface VectorDescriptor> : Descriptor { /** The size of this [VectorDescriptor]. */ val dimensionality: Int get() = this.vector.size /** The [List] of values [T]. */ - val vector: List + val vector: T /** * Returns the fields and its values of this [ScalarDescriptor] as a [Map]. * * @return A [Map] of this [ScalarDescriptor]'s fields (without the IDs). */ - override fun values(): List?>> = listOf("vector" to this.vector) + override fun values(): Map = mapOf("vector" to this.vector) } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/proximity/ProximityQuery.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/proximity/ProximityQuery.kt index 8088a933..8456ad87 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/proximity/ProximityQuery.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/proximity/ProximityQuery.kt @@ -17,9 +17,9 @@ import org.vitrivr.engine.core.model.types.Value * @version 1.0.0 */ -data class ProximityQuery>( +data class ProximityQuery>( /** The [VectorDescriptor] being used; specifies both the query field and the comparison value. */ - val value: List, + val value: T, /** The [Distance] used for the comparison. */ val distance: Distance = Distance.EUCLIDEAN, diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt index 131f7ce8..fad4c49a 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt @@ -1,20 +1,158 @@ package org.vitrivr.engine.core.model.types +import java.util.* + /** - * An enumeration of supported [Type]s + * The [Type]s supported by vitrivr-engine. * * @author Ralph Gasser * @author Loris Sauter - * @version 1.1.0 + * @version 2.0.0 */ -enum class Type { - STRING, - BOOLEAN, - BYTE, - SHORT, - INT, - LONG, - FLOAT, - DOUBLE, - DATETIME +sealed interface Type { + + companion object { + /** + * Returns the [Type] for the given [String] representation. + * + * @param type [String] representation of the [Type] + * @return [Type] + */ + fun valueOf(type: kotlin.String, dimensions: kotlin.Int = 0): Type { + return when (type.uppercase()) { + "STRING" -> String + "BOOLEAN" -> Boolean + "BYTE" -> Byte + "SHORT" -> Short + "INT" -> Int + "LONG" -> Long + "FLOAT" -> Float + "DOUBLE" -> Double + "DATETIME" -> Datetime + "BOOLEANVECTOR" -> BooleanVector(dimensions) + "INTVECTOR" -> IntVector(dimensions) + "LONGVECTOR" -> LongVector(dimensions) + "FLOATVECTOR" -> FloatVector(dimensions) + "DOUBLEVECTOR" -> DoubleVector(dimensions) + else -> throw IllegalArgumentException("Type $type is not supported!") + } + } + } + + + /** The number of dimensions for this [Type]. */ + val dimensions: kotlin.Int + + + /** + * + */ + fun defaultValue(): Value<*> + + /** + * A [Type] that represents a String. + */ + data object String : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.String("") + } + + /** + * A [Type] that represents a [Boolean] value. + */ + data object Boolean : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.Boolean(false) + } + + /** + * A [Type] that represents a [Byte] value. + */ + data object Byte : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.Byte(0) + } + + /** + * A [Type] that represents a [Short] value. + */ + data object Short : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.Short(0) + } + + /** + * A [Type] that represents a [Int] value. + */ + data object Int : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.Int(0) + } + + /** + * A [Type] that represents a [Long] value. + */ + data object Long : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.Long(0L) + } + + /** + * A [Type] that represents a [Float] value. + */ + data object Float : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.Float(0.0f) + } + + /** + * A [Type] that represents a [Double] value. + */ + data object Double : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.Double(0.0) + } + + /** + * A [Type] that represents a [Datetime] value. + */ + data object Datetime : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.DateTime(Date()) + } + + /** + * A [Type] that represents a [BooleanVector] value. + */ + data class BooleanVector(override val dimensions: kotlin.Int) : Type { + override fun defaultValue(): Value<*> = Value.BooleanVector(BooleanArray(this.dimensions)) + } + + /** + * A [Type] that represents a [IntVector] value. + */ + data class IntVector(override val dimensions: kotlin.Int) : Type { + override fun defaultValue(): Value<*> = Value.IntVector(IntArray(this.dimensions)) + } + + /** + * A [Type] that represents a [LongVector] value. + */ + data class LongVector(override val dimensions: kotlin.Int) : Type { + override fun defaultValue(): Value<*> = Value.LongVector(LongArray(this.dimensions)) + } + + /** + * A [Type] that represents a [FloatVector] value. + */ + data class FloatVector(override val dimensions: kotlin.Int) : Type { + override fun defaultValue(): Value<*> = Value.FloatVector(FloatArray(this.dimensions)) + } + + /** + * A [Type] that represents a [DoubleVector] value. + */ + data class DoubleVector(override val dimensions: kotlin.Int) : Type { + override fun defaultValue(): Value<*> = Value.DoubleVector(DoubleArray(this.dimensions)) + } } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt index a0e3f37e..3bd60409 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt @@ -27,6 +27,7 @@ sealed interface Value { } } + /** The actual, primitive value. */ val value: T @JvmInline @@ -55,4 +56,61 @@ sealed interface Value { @JvmInline value class DateTime(override val value: Date) : Value + + /** + * A [Vector] in vitrivr-engine maps primitive data types. + * + * Part of the vitrivr-engine's internal type system. + * + * @param T The type of the elements in the [Vector]. + * @property size The size of the [Vector]. + * @property value The actual, primitive value. + * @constructor Creates a new [Vector] with the given [value]. + */ + sealed interface Vector : Value { + val size: kotlin.Int + } + + + @JvmInline + value class BooleanVector(override val value: BooleanArray) : Vector { + constructor(size: kotlin.Int, init: (kotlin.Int) -> kotlin.Boolean = { false }) : this(BooleanArray(size, init)) + + override val size: kotlin.Int + get() = this.value.size + } + + @JvmInline + value class IntVector(override val value: IntArray) : Vector { + constructor(size: kotlin.Int, init: (kotlin.Int) -> kotlin.Int = { 0 }) : this(IntArray(size, init)) + + override val size: kotlin.Int + get() = this.value.size + } + + + @JvmInline + value class LongVector(override val value: LongArray) : Vector { + constructor(size: kotlin.Int, init: (kotlin.Int) -> kotlin.Long = { 0L }) : this(LongArray(size, init)) + + override val size: kotlin.Int + get() = this.value.size + } + + + @JvmInline + value class FloatVector(override val value: FloatArray) : Vector { + constructor(size: kotlin.Int, init: (kotlin.Int) -> kotlin.Float = { 0.0f }) : this(FloatArray(size, init)) + + override val size: kotlin.Int + get() = this.value.size + } + + @JvmInline + value class DoubleVector(override val value: DoubleArray) : Vector { + constructor(size: kotlin.Int, init: (kotlin.Int) -> kotlin.Double = { 0.0 }) : this(DoubleArray(size, init)) + + override val size: kotlin.Int + get() = this.value.size + } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt index 356a6f73..1158b204 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt @@ -3,7 +3,7 @@ package org.vitrivr.engine.plugin.cottontaildb import org.vitrivr.cottontail.client.language.basics.predicate.Compare import org.vitrivr.cottontail.core.types.Types import org.vitrivr.cottontail.core.values.* -import org.vitrivr.engine.core.model.descriptor.FieldSchema +import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.descriptor.scalar.* import org.vitrivr.engine.core.model.descriptor.vector.* import org.vitrivr.engine.core.model.query.basics.ComparisonOperator @@ -49,48 +49,25 @@ const val DISTANCE_COLUMN_NAME = "distance" const val SCORE_COLUMN_NAME = "score" /** - * Converts a vitrivr-engine [FieldSchema] to a Cottontail DB [Types]. + * Converts a vitrivr-engine [Attribute] to a Cottontail DB [Types]. * * @return [Compare.Operator] used for this [SimpleBooleanQuery] */ -internal fun FieldSchema.toCottontailType(): Types<*> { - val vector = this.dimensions.size == 1 - return when (this.type) { - Type.STRING -> Types.String - Type.BYTE -> Types.Byte - Type.SHORT -> Types.Short - Type.BOOLEAN -> if (vector) { - Types.BooleanVector(this.dimensions[0]) - } else { - Types.Boolean - } - - Type.INT -> if (vector) { - Types.IntVector(this.dimensions[0]) - } else { - Types.Int - } - - Type.LONG -> if (vector) { - Types.LongVector(this.dimensions[0]) - } else { - Types.Long - } - - Type.FLOAT -> if (vector) { - Types.FloatVector(this.dimensions[0]) - } else { - Types.Float - } - - Type.DOUBLE -> if (vector) { - Types.DoubleVector(this.dimensions[0]) - } else { - Types.Double - } - - Type.DATETIME -> Types.Date - } +internal fun Type.toCottontailType(): Types<*> = when (this) { + Type.String -> Types.String + Type.Byte -> Types.Byte + Type.Short -> Types.Short + Type.Boolean -> Types.Boolean + Type.Int -> Types.Int + Type.Long -> Types.Long + Type.Float -> Types.Float + Type.Double -> Types.Double + Type.Datetime -> Types.Date + is Type.BooleanVector -> Types.BooleanVector(this.dimensions) + is Type.DoubleVector -> Types.DoubleVector(this.dimensions) + is Type.FloatVector -> Types.FloatVector(this.dimensions) + is Type.IntVector -> Types.IntVector(this.dimensions) + is Type.LongVector -> Types.LongVector(this.dimensions) } /** @@ -151,6 +128,11 @@ internal fun Value<*>.toCottontailValue(): PublicValue = when (this) { is Value.Short -> ShortValue(this.value) is Value.String -> StringValue(this.value) is Value.DateTime -> DateValue(this.value) + is Value.BooleanVector -> BooleanVectorValue(this.value) + is Value.DoubleVector -> DoubleVectorValue(this.value) + is Value.FloatVector -> FloatVectorValue(this.value) + is Value.IntVector -> IntVectorValue(this.value) + is Value.LongVector -> LongVectorValue(this.value) } /** diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt index 415a02aa..53b73327 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt @@ -6,13 +6,10 @@ import org.vitrivr.engine.core.database.ConnectionProvider import org.vitrivr.engine.core.database.descriptor.DescriptorProvider import org.vitrivr.engine.core.model.descriptor.scalar.* import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.RasterDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.SkeletonDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.MediaDimensionsDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.Rectangle2DMetadataDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.TemporalMetadataDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.FileSourceMetadataDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.VideoSourceMetadataDescriptor import org.vitrivr.engine.core.model.descriptor.vector.* import org.vitrivr.engine.core.model.metamodel.Schema @@ -25,7 +22,7 @@ import org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescripto * Implementation of the [ConnectionProvider] interface for Cottontail DB. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ class CottontailConnectionProvider: AbstractConnectionProvider() { @@ -77,9 +74,6 @@ class CottontailConnectionProvider: AbstractConnectionProvider() { this.register(TemporalMetadataDescriptor::class, StructDescriptorProvider) this.register(Rectangle2DMetadataDescriptor::class, StructDescriptorProvider) this.register(MediaDimensionsDescriptor::class, StructDescriptorProvider) - this.register(SkeletonDescriptor::class, StructDescriptorProvider) - this.register(RasterDescriptor::class, StructDescriptorProvider) - this.register(MapStructDescriptor::class, StructDescriptorProvider) } /** diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorInitializer.kt index 4a46065c..77895435 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorInitializer.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorInitializer.kt @@ -1,26 +1,21 @@ package org.vitrivr.engine.plugin.cottontaildb.descriptors.struct -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging import io.grpc.StatusRuntimeException import org.vitrivr.cottontail.client.language.ddl.CreateEntity import org.vitrivr.cottontail.core.database.Name import org.vitrivr.cottontail.core.types.Types import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.core.model.types.Type -import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection -import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ID_COLUMN_NAME -import org.vitrivr.engine.plugin.cottontaildb.RETRIEVABLE_ID_COLUMN_NAME +import org.vitrivr.engine.plugin.cottontaildb.* +import org.vitrivr.engine.plugin.cottontaildb.LOGGER import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorInitializer - -private val logger: KLogger = KotlinLogging.logger {} +import org.vitrivr.engine.plugin.cottontaildb.toCottontailType /** * A [AbstractDescriptorInitializer] implementation for [StructDescriptor]s. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, connection: CottontailConnection) : AbstractDescriptorInitializer(field, connection) { /** @@ -32,54 +27,15 @@ class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, conn .column(Name.ColumnName.create(DESCRIPTOR_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = true, autoIncrement = false) .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) - for (field in this.field.analyser.prototype(this.field).schema()) { - require(field.dimensions.size <= 1) { "Cottontail DB currently doesn't support tensor types."} - val vector = field.dimensions.size == 1 - val type = when (field.type) { - Type.STRING -> Types.String - Type.BYTE -> Types.Byte - Type.SHORT -> Types.Short - Type.BOOLEAN -> if (vector) { - Types.BooleanVector(field.dimensions[0]) - } else { - Types.Boolean - } - - Type.INT -> if (vector) { - Types.IntVector(field.dimensions[0]) - } else { - Types.Int - } - - Type.LONG -> if (vector) { - Types.LongVector(field.dimensions[0]) - } else { - Types.Long - } - - Type.FLOAT -> if (vector) { - Types.FloatVector(field.dimensions[0]) - } else { - Types.Float - } - - Type.DOUBLE -> if (vector) { - Types.DoubleVector(field.dimensions[0]) - } else { - Types.Double - } - - Type.DATETIME -> Types.Date - } - create.column(Name.ColumnName.create(field.name), type, nullable = field.nullable, primaryKey = false, autoIncrement = false) + create.column(Name.ColumnName.create(field.name), field.type.toCottontailType(), nullable = field.nullable, primaryKey = false, autoIncrement = false) } try { /* Try to create entity. */ this.connection.client.create(create) } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } } } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt index 2206d8b8..cdf7237e 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt @@ -5,9 +5,10 @@ import org.vitrivr.cottontail.client.language.basics.expression.Literal import org.vitrivr.cottontail.client.language.basics.predicate.Compare import org.vitrivr.cottontail.core.tuple.Tuple import org.vitrivr.cottontail.core.types.Types +import org.vitrivr.engine.core.model.descriptor.AttributeName import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.MapStructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query import org.vitrivr.engine.core.model.query.bool.SimpleBooleanQuery @@ -32,8 +33,7 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio init { val prototype = this.field.analyser.prototype(this.field) for (f in prototype.schema()) { - require(f.dimensions.size <= 1) { "Cottontail DB currently doesn't support tensor types."} - this.fieldMap.add(f.name to f.toCottontailType()) + this.fieldMap.add(f.name to f.type.toCottontailType()) } } @@ -79,42 +79,33 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio * @return The resulting [StructDescriptor]. */ override fun tupleToDescriptor(tuple: Tuple): StructDescriptor { - return if (this.field.analyser.descriptorClass == MapStructDescriptor::class) { - this.tupleToMapStructDescriptor(tuple) - }else{ - this.tupleToStructDescriptor(tuple) - } - } - - private fun tupleToStructDescriptor(tuple: Tuple): StructDescriptor{ val constructor = this.field.analyser.descriptorClass.primaryConstructor ?: throw IllegalStateException("Provided type ${this.field.analyser.descriptorClass} does not have a primary constructor.") + val valueMap = mutableMapOf>() val parameters: MutableList = mutableListOf( tuple.asUuidValue(DESCRIPTOR_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'."), tuple.asUuidValue(RETRIEVABLE_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'."), + valueMap ) /* Append dynamic parameters of struct. */ for ((name, type) in this.fieldMap) { - parameters.add( - when(type) { - Types.Boolean -> tuple.asBoolean(name)?.let { Value.Boolean(it) } - Types.Date -> tuple.asDate(name) - Types.Byte -> tuple.asByte(name)?.let { Value.Byte(it) } - Types.Double -> tuple.asDouble(name)?.let { Value.Double(it) } - Types.Float -> tuple.asFloat(name)?.let { Value.Float(it) } - Types.Int -> tuple.asInt(name)?.let { Value.Int(it) } - Types.Long -> tuple.asLong(name)?.let { Value.Long(it) } - Types.Short -> tuple.asShort(name)?.let { Value.Short(it) } - Types.String -> tuple.asString(name)?.let { Value.String(it) } - Types.Uuid -> UUID.fromString(tuple.asString(name)) - is Types.BooleanVector -> tuple.asBooleanVector(name)?.let { List(it.size) { i -> it[i] } } - is Types.DoubleVector -> tuple.asBooleanVector(name)?.let { List(it.size) { i -> it[i] } } - is Types.FloatVector -> tuple.asBooleanVector(name)?.let { List(it.size) { i -> it[i] } } - is Types.IntVector -> tuple.asIntVector(name)?.let { List(it.size) { i -> it[i] } } - is Types.LongVector -> tuple.asLongVector(name)?.let { List(it.size) { i -> it[i] } } - else -> throw IllegalArgumentException("Type $type is not supported by StructDescriptorReader.") - } - ) + valueMap[name] = when (type) { + Types.Boolean -> tuple.asBoolean(name)?.let { Value.Boolean(it) } + Types.Date -> tuple.asDate(name)?.let { Value.DateTime(it) } + Types.Byte -> tuple.asByte(name)?.let { Value.Byte(it) } + Types.Double -> tuple.asDouble(name)?.let { Value.Double(it) } + Types.Float -> tuple.asFloat(name)?.let { Value.Float(it) } + Types.Int -> tuple.asInt(name)?.let { Value.Int(it) } + Types.Long -> tuple.asLong(name)?.let { Value.Long(it) } + Types.Short -> tuple.asShort(name)?.let { Value.Short(it) } + Types.String -> tuple.asString(name)?.let { Value.String(it) } + is Types.BooleanVector -> tuple.asBooleanVector(name)?.let { Value.BooleanVector(it) } + is Types.DoubleVector -> tuple.asDoubleVector(name)?.let { Value.DoubleVector(it) } + is Types.FloatVector -> tuple.asFloatVector(name)?.let { Value.FloatVector(it) } + is Types.IntVector -> tuple.asIntVector(name)?.let { Value.IntVector(it) } + is Types.LongVector -> tuple.asLongVector(name)?.let { Value.LongVector(it) } + else -> throw IllegalArgumentException("Type $type is not supported by StructDescriptorReader.") + } as Value<*> } parameters.add(field) //add field information, as this is for all StructDescriptors the last constructor argument. @@ -122,58 +113,4 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio /* Call constructor. */ return constructor.call(*parameters.toTypedArray()) } - - private fun tupleToMapStructDescriptor(tuple: Tuple): MapStructDescriptor{ - require(this.field.analyser.descriptorClass == MapStructDescriptor::class){"Must be MapStructDescriptor, but '${this.field.analyser.descriptorClass}' given"} - val constructor = this.field.analyser.descriptorClass.primaryConstructor ?: throw IllegalStateException("Provided type ${this.field.analyser.descriptorClass} does not have a primary constructor.") - val parameters: MutableList = mutableListOf( - tuple.asUuidValue(DESCRIPTOR_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'."), - tuple.asUuidValue(RETRIEVABLE_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'."), - ) - - /* Third argument: columnTypes */ - parameters.add(this.field.parameters) - - /* Fourth argument: columnValues */ - val values: MutableList = mutableListOf() - /* Append dynamic parameters of struct. */ - for ((name, type) in this.fieldMap) { - values.add( - when(type) { - Types.Boolean -> tuple.asBoolean(name)?.let { Value.Boolean(it) } - Types.Date -> tuple.asDate(name) - Types.Byte -> tuple.asByte(name)?.let { Value.Byte(it) } - Types.Double -> tuple.asDouble(name)?.let { Value.Double(it) } - Types.Float -> tuple.asFloat(name)?.let { Value.Float(it) } - Types.Int -> tuple.asInt(name)?.let { Value.Int(it) } - Types.Long -> tuple.asLong(name)?.let { Value.Long(it) } - Types.Short -> tuple.asShort(name)?.let { Value.Short(it) } - Types.String -> tuple.asString(name)?.let { Value.String(it) } - Types.Uuid -> UUID.fromString(tuple.asString(name)) - is Types.BooleanVector -> tuple.asBooleanVector(name)?.let { List(it.size) { i -> it[i] } } - is Types.DoubleVector -> tuple.asBooleanVector(name)?.let { List(it.size) { i -> it[i] } } - is Types.FloatVector -> tuple.asBooleanVector(name)?.let { List(it.size) { i -> it[i] } } - is Types.IntVector -> tuple.asIntVector(name)?.let { List(it.size) { i -> it[i] } } - is Types.LongVector -> tuple.asLongVector(name)?.let { List(it.size) { i -> it[i] } } - else -> throw IllegalArgumentException("Type $type is not supported by StructDescriptorReader.") - } - ) - } - require(values.size == this.field.parameters.size){"Some data was missing from the DB. Field definition and values from DB sizes differ!"} - val columnValues: MutableMap = mutableMapOf() - val columnKeys = this.field.parameters.keys.toTypedArray() - for(i in 0 until values.size){ - columnValues[columnKeys[i]] = values[i] - } - parameters.add(columnValues) - - /* Fifth argument: transient */ - parameters.add(false) //add 'transient' flag to false, since the results were actually retrieved - - /* Sixth argument: field */ - parameters.add(this.field) - - /* Call constructor. */ - return constructor.call(*parameters.toTypedArray()) as MapStructDescriptor - } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorWriter.kt index 3eff0a45..d5e689e6 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorWriter.kt @@ -68,22 +68,22 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio /* Insert values. */ for (item in items) { - val values = item.values() + val attributes = item.schema() if (index == 0) { - val columns = Array(values.size + 2) { + val columns = Array(attributes.size + 2) { when (it) { 0 -> DESCRIPTOR_ID_COLUMN_NAME 1 -> RETRIEVABLE_ID_COLUMN_NAME - else -> values[it - 2].first + else -> attributes[it - 2].name } } insert.columns(*columns) } - val inserts: Array = Array(values.size + 2) { + val inserts: Array = Array(attributes.size + 2) { when (it) { 0 -> UuidValue(item.id) 1 -> item.retrievableId?.let { v -> UuidValue(v) } - else -> values[it - 2].second?.toCottontailValue() + else -> item.values()[attributes[it - 2].name]?.toCottontailValue() } } insert.any(*inserts) diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt index 53508600..94279c84 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt @@ -9,6 +9,7 @@ import org.vitrivr.engine.core.model.query.Query import org.vitrivr.engine.core.model.query.proximity.ProximityQuery import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.core.model.retrievable.attributes.DistanceAttribute +import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.core.model.types.toValue import org.vitrivr.engine.plugin.cottontaildb.* import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorReader @@ -17,7 +18,7 @@ import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorRead * An [AbstractDescriptorReader] for [FloatVectorDescriptor]s. * * @author Ralph Gasser - * @version 1.2.0 + * @version 1.3.0 */ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connection: CottontailConnection) : AbstractDescriptorReader>(field, connection) { @@ -114,43 +115,38 @@ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*> */ override fun tupleToDescriptor(tuple: Tuple): VectorDescriptor<*> { val descriptorId = tuple.asUuidValue(DESCRIPTOR_ID_COLUMN_NAME)?.value - ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'.") + ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'.") val retrievableId = tuple.asUuidValue(RETRIEVABLE_ID_COLUMN_NAME)?.value - ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'.") + ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'.") return when (this.prototype) { is BooleanVectorDescriptor -> BooleanVectorDescriptor( descriptorId, retrievableId, - tuple.asBooleanVector(DESCRIPTOR_COLUMN_NAME)?.map { it.toValue() } - ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + Value.BooleanVector(tuple.asBooleanVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) ) is FloatVectorDescriptor -> FloatVectorDescriptor( descriptorId, retrievableId, - tuple.asFloatVector(DESCRIPTOR_COLUMN_NAME)?.map { it.toValue() } - ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + Value.FloatVector(tuple.asFloatVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) ) is DoubleVectorDescriptor -> DoubleVectorDescriptor( descriptorId, retrievableId, - tuple.asDoubleVector(DESCRIPTOR_COLUMN_NAME)?.map { it.toValue() } - ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + Value.DoubleVector(tuple.asDoubleVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) ) is IntVectorDescriptor -> IntVectorDescriptor( descriptorId, retrievableId, - tuple.asIntVector(DESCRIPTOR_COLUMN_NAME)?.map { it.toValue() } - ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + Value.IntVector(tuple.asIntVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) ) is LongVectorDescriptor -> LongVectorDescriptor( descriptorId, retrievableId, - tuple.asLongVector(DESCRIPTOR_COLUMN_NAME)?.map { it.toValue() } - ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + Value.LongVector(tuple.asLongVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) ) } } diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt index b961ef59..6a0c9d51 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt @@ -10,6 +10,9 @@ import org.vitrivr.engine.module.features.database.string.writer.StringRetrievab class StringConnection(override val provider: StringConnectionProvider, schemaName: String, internal val stringify: (Persistable) -> String) : AbstractConnection(schemaName, provider) { + override fun withTransaction(action: (Unit) -> T): T { + throw UnsupportedOperationException("Transactions are not supported by StringConnection.") + } override fun getRetrievableInitializer(): RetrievableInitializer = NoRetrievableInitializer() diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt index 2510922b..4f9f7329 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt @@ -41,7 +41,7 @@ class AverageColor : Analyser { * @param field [Schema.Field] to create the prototype for. * @return [FloatVectorDescriptor] */ - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), listOf(Value.Float(0.0f), Value.Float(0.0f), Value.Float(0.0f))) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(3)) /** * Generates and returns a new [AverageColorExtractor] instance for this [AverageColor]. @@ -78,9 +78,9 @@ class AverageColor : Analyser { */ override fun newRetrieverForQuery(field: Schema.Field, query: Query, context: QueryContext): AverageColorRetriever { require(field.analyser == this) { "The field '${field.fieldName}' analyser does not correspond with this analyser. This is a programmer's error!" } - require(query is ProximityQuery<*> && query.value.first() is Value.Float) { "The query is not a ProximityQuery." } + require(query is ProximityQuery<*> && query.value is Value.FloatVector) { "The query is not a ProximityQuery." } @Suppress("UNCHECKED_CAST") - return AverageColorRetriever(field, query as ProximityQuery) + return AverageColorRetriever(field, query as ProximityQuery) } /** @@ -130,6 +130,6 @@ class AverageColor : Analyser { /* Generate descriptor. */ val averageColor = RGBFloatColorContainer(color.red / rgb.size, color.green / rgb.size, color.blue / rgb.size) - FloatVectorDescriptor(UUID.randomUUID(), null, averageColor.toValueList()) + FloatVectorDescriptor(UUID.randomUUID(), null, averageColor.toVector()) } } diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorRetriever.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorRetriever.kt index b6d35114..551fd034 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorRetriever.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorRetriever.kt @@ -20,11 +20,11 @@ import org.vitrivr.engine.core.operators.retrieve.Retriever * @see [AverageColor] * * @author Luca Rossetto - * @version 1.0.0 + * @version 1.1.0 */ class AverageColorRetriever( override val field: Schema.Field, - private val query: ProximityQuery + private val query: ProximityQuery ) : Retriever { private val logger: KLogger = KotlinLogging.logger {} diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorRaster.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolorraster/AverageColorRaster.kt similarity index 86% rename from vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorRaster.kt rename to vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolorraster/AverageColorRaster.kt index c7c58cd1..e20dc179 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorRaster.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolorraster/AverageColorRaster.kt @@ -1,9 +1,8 @@ -package org.vitrivr.engine.module.features.feature.migration +package org.vitrivr.engine.module.features.feature.averagecolorraster import org.vitrivr.engine.core.context.IndexContext import org.vitrivr.engine.core.context.QueryContext import org.vitrivr.engine.core.model.content.element.ContentElement -import org.vitrivr.engine.core.model.descriptor.struct.RasterDescriptor import org.vitrivr.engine.core.model.metamodel.Analyser import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query @@ -18,7 +17,7 @@ class AverageColorRaster: Analyser, RasterDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = RasterDescriptor::class override fun prototype(field: Schema.Field<*, *>): RasterDescriptor = - RasterDescriptor(id = UUID.randomUUID(), retrievableId = UUID.randomUUID(), hist = List(15) { Value.Float(0.0f) }, raster = List(64) { Value.Float(0.0f) }) // should transient be false? what is transient? + RasterDescriptor(id = UUID.randomUUID(), retrievableId = UUID.randomUUID(), mapOf("hist" to Value.FloatVector(FloatArray(15)), "raster" to Value.FloatVector(FloatArray(64)))) override fun newRetrieverForContent(field: Schema.Field, RasterDescriptor>, content: Collection>, context: QueryContext): Retriever, RasterDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolorraster/RasterDescriptor.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolorraster/RasterDescriptor.kt new file mode 100644 index 00000000..486919e8 --- /dev/null +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolorraster/RasterDescriptor.kt @@ -0,0 +1,37 @@ +package org.vitrivr.engine.module.features.feature.averagecolorraster + +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.retrievable.RetrievableId +import org.vitrivr.engine.core.model.types.Type +import org.vitrivr.engine.core.model.types.Value + +/** + * A struct vector descriptor that stores the average color and raster of an image. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class RasterDescriptor( + override var id: DescriptorId, + override var retrievableId: RetrievableId?, + values: Map?>, + override val field: Schema.Field<*, RasterDescriptor>? = null +) : MapStructDescriptor(id, retrievableId, LAYOUT, values, field) { + + companion object { + private val LAYOUT = listOf( + Attribute("hist", Type.FloatVector(15)), + Attribute("raster", Type.FloatVector(4)) + ) + } + + /** The histogram vector. */ + val hist: Value.FloatVector by this.values + + /** The raster vector. */ + val raster: Value.FloatVector by this.values +} \ No newline at end of file diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/ExternalAnalyser.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/ExternalAnalyser.kt index 30828d28..686ad68c 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/ExternalAnalyser.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/ExternalAnalyser.kt @@ -83,7 +83,7 @@ abstract class ExternalAnalyser, U : Descriptor> : Analyse if (responseCode != HttpURLConnection.HTTP_OK) return null return connection.inputStream.use { stream -> when (U::class) { - FloatVectorDescriptor::class -> FloatVectorDescriptor(UUID.randomUUID(), null, Json.decodeFromStream(stream).map { Value.Float(it) }) + FloatVectorDescriptor::class -> FloatVectorDescriptor(UUID.randomUUID(), null, Value.FloatVector(Json.decodeFromStream(stream))) else -> null } as U? } diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/implementations/clip/CLIP.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/implementations/clip/CLIP.kt index fe1995e3..247b3984 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/implementations/clip/CLIP.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/implementations/clip/CLIP.kt @@ -27,7 +27,7 @@ import java.util.* * Implementation of the [CLIP] [ExternalAnalyser], which derives the CLIP feature from an [ImageContent] or [TextContent] as [FloatVectorDescriptor]. * * @author Rahel Arnold - * @version 1.2.0 + * @version 1.3.0 */ class CLIP : ExternalAnalyser, FloatVectorDescriptor>() { @@ -64,7 +64,7 @@ class CLIP : ExternalAnalyser, FloatVectorDescriptor>() { * * @return [FloatVectorDescriptor] */ - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(512) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(512)) /** * Generates and returns a new [Extractor] instance for this [CLIP]. @@ -108,9 +108,9 @@ class CLIP : ExternalAnalyser, FloatVectorDescriptor>() { */ override fun newRetrieverForQuery(field: Schema.Field, FloatVectorDescriptor>, query: Query, context: QueryContext): DenseRetriever> { require(field.analyser == this) { "The field '${field.fieldName}' analyser does not correspond with this analyser. This is a programmer's error!" } - require(query is ProximityQuery<*> && query.value.first() is Value.Float) { "The query is not a ProximityQuery." } + require(query is ProximityQuery<*> && query.value is Value.FloatVector) { "The query is not a ProximityQuery." } @Suppress("UNCHECKED_CAST") - return DenseRetriever(field, query as ProximityQuery, context) + return DenseRetriever(field, query as ProximityQuery, context) } /** diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/implementations/dino/DINO.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/implementations/dino/DINO.kt index 65e78566..1473f344 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/implementations/dino/DINO.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/external/implementations/dino/DINO.kt @@ -55,7 +55,7 @@ class DINO : ExternalAnalyser() { * * @return [FloatVectorDescriptor] */ - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(384) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(384)) /** * Generates and returns a new [Extractor] instance for this [DINO]. @@ -99,9 +99,9 @@ class DINO : ExternalAnalyser() { */ override fun newRetrieverForQuery(field: Schema.Field, query: Query, context: QueryContext): DenseRetriever { require(field.analyser == this) { "The field '${field.fieldName}' analyser does not correspond with this analyser. This is a programmer's error!" } - require(query is ProximityQuery<*> && query.value.first() is Value.Float) { "The query is not a ProximityQuery." } + require(query is ProximityQuery<*> && query.value is Value.FloatVector) { "The query is not a ProximityQuery." } @Suppress("UNCHECKED_CAST") - return DenseRetriever(field, query as ProximityQuery, context) + return DenseRetriever(field, query as ProximityQuery, context) } /** diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorGrid8.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorGrid8.kt index c719d255..d83590ec 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorGrid8.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorGrid8.kt @@ -17,7 +17,7 @@ import java.util.* class AverageColorGrid8 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(192) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(192)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorGrid8Reduced15.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorGrid8Reduced15.kt index 6d1a2ea7..f0779e2e 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorGrid8Reduced15.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageColorGrid8Reduced15.kt @@ -17,7 +17,7 @@ import java.util.* class AverageColorGrid8Reduced15 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(192) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(192)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageFuzzyHist.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageFuzzyHist.kt index 65fff5ef..765f50fa 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageFuzzyHist.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageFuzzyHist.kt @@ -17,7 +17,7 @@ import java.util.* class AverageFuzzyHist : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(15) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(15)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageFuzzyHistNormalized.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageFuzzyHistNormalized.kt index 5bf8dccf..e2ee9aec 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageFuzzyHistNormalized.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/AverageFuzzyHistNormalized.kt @@ -17,7 +17,7 @@ import java.util.* class AverageFuzzyHistNormalized : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(15) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(15)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/CLD.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/CLD.kt index 88ee2dcb..caf1090f 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/CLD.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/CLD.kt @@ -17,7 +17,7 @@ import java.util.* class CLD : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(12) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(12)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/CLDReduced15.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/CLDReduced15.kt index 223d36ad..e28484d9 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/CLDReduced15.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/CLDReduced15.kt @@ -17,7 +17,7 @@ import java.util.* class CLDReduced15 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(12) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(12)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/ConceptMasksADE20k.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/ConceptMasksADE20k.kt index 86e0f1b6..d8ea266f 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/ConceptMasksADE20k.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/ConceptMasksADE20k.kt @@ -17,7 +17,7 @@ import java.util.* class ConceptMasksADE20k : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(2048) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(2048)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid.kt index c2160e8a..89c050aa 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid.kt @@ -17,7 +17,7 @@ import java.util.* class DominantEdgeGrid : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(64) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(64)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid16.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid16.kt index dfed671a..f6cd65a9 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid16.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid16.kt @@ -17,7 +17,7 @@ import java.util.* class DominantEdgeGrid16 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(256) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(256)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid8.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid8.kt index 7d113eec..6c50f83d 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid8.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/DominantEdgeGrid8.kt @@ -17,7 +17,7 @@ import java.util.* class DominantEdgeGrid8 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(64) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(64)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EHD.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EHD.kt index cc6b7d0f..6e2ebad5 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EHD.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EHD.kt @@ -17,7 +17,7 @@ import java.util.* class EHD : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(80) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(80)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EdgeARP88.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EdgeARP88.kt index fffa3ee9..342328fb 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EdgeARP88.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EdgeARP88.kt @@ -17,7 +17,7 @@ import java.util.* class EdgeARP88 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(64) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(64)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EdgeGrid16.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EdgeGrid16.kt index 130a9c27..8158b51f 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EdgeGrid16.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/EdgeGrid16.kt @@ -17,7 +17,7 @@ import java.util.* class EdgeGrid16 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(256) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(256)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/HOGMF25k512.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/HOGMF25k512.kt index bbd42660..f20686f7 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/HOGMF25k512.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/HOGMF25k512.kt @@ -17,7 +17,7 @@ import java.util.* class HOGMF25k512 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(512) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(512)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/HueHistogram.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/HueHistogram.kt index 0b13a5f9..2cf63783 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/HueHistogram.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/HueHistogram.kt @@ -17,7 +17,7 @@ import java.util.* class HueHistogram : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(16) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(16)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/InceptionResNetV2.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/InceptionResNetV2.kt index 33c366c4..866516d3 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/InceptionResNetV2.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/InceptionResNetV2.kt @@ -17,7 +17,7 @@ import java.util.* class InceptionResNetV2 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(1536) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(1536)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/MedianColor.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/MedianColor.kt index abfc77dd..19ed1bf9 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/MedianColor.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/MedianColor.kt @@ -17,7 +17,7 @@ import java.util.* class MedianColor : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(3) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(3)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/MedianFuzzyHist.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/MedianFuzzyHist.kt index bcec984f..6486f1a7 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/MedianFuzzyHist.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/MedianFuzzyHist.kt @@ -17,7 +17,7 @@ import java.util.* class MedianFuzzyHist : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(15) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(15)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/OpenCLIP.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/OpenCLIP.kt index cd355315..5edb0124 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/OpenCLIP.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/OpenCLIP.kt @@ -17,7 +17,7 @@ import java.util.* class OpenCLIP : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(512) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(512)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/SURFMF25K512.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/SURFMF25K512.kt index 2301a03b..b1367835 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/SURFMF25K512.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/SURFMF25K512.kt @@ -17,7 +17,7 @@ import java.util.* class SURFMF25K512 : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(512) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(512)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/VisualTextCoEmbedding.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/VisualTextCoEmbedding.kt index 02bcd4f7..eeced2b9 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/VisualTextCoEmbedding.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/VisualTextCoEmbedding.kt @@ -17,7 +17,7 @@ import java.util.* class VisualTextCoEmbedding : Analyser, FloatVectorDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = FloatVectorDescriptor::class - override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(256) { Value.Float(0.0f) }) + override fun prototype(field: Schema.Field<*, *>) = FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(256)) override fun newRetrieverForContent(field: Schema.Field, FloatVectorDescriptor>, content: Collection>, context: QueryContext): Retriever, FloatVectorDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/skeleton/SkeletonDescriptor.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/skeleton/SkeletonDescriptor.kt new file mode 100644 index 00000000..4a6231c7 --- /dev/null +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/skeleton/SkeletonDescriptor.kt @@ -0,0 +1,37 @@ +package org.vitrivr.engine.module.features.feature.skeleton + +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.retrievable.RetrievableId +import org.vitrivr.engine.core.model.types.Type +import org.vitrivr.engine.core.model.types.Value + +class SkeletonDescriptor( + override var id: DescriptorId, + override var retrievableId: RetrievableId?, + values: Map?>, + override val field: Schema.Field<*, SkeletonDescriptor>? = null +) : MapStructDescriptor(id, retrievableId, LAYOUT, values, field) { + + companion object { + private val LAYOUT = listOf( + Attribute("person", Type.Int), + Attribute("skeleton", Type.FloatVector(12)), + Attribute("weights", Type.FloatVector(12)) + ) + } + + /** The person index. */ + val person: Value.Int by this.values + + /** The vector describing the skeleton. */ + val skeleton: List by this.values + + /** The vector describing the skeleto weights. */ + val weights: List by this.values + + +} \ No newline at end of file diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/SkeletonPose.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/skeleton/SkeletonPose.kt similarity index 84% rename from vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/SkeletonPose.kt rename to vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/skeleton/SkeletonPose.kt index 337d9d66..9d696cf3 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/migration/SkeletonPose.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/skeleton/SkeletonPose.kt @@ -1,9 +1,8 @@ -package org.vitrivr.engine.module.features.feature.migration +package org.vitrivr.engine.module.features.feature.skeleton import org.vitrivr.engine.core.context.IndexContext import org.vitrivr.engine.core.context.QueryContext import org.vitrivr.engine.core.model.content.element.ContentElement -import org.vitrivr.engine.core.model.descriptor.struct.SkeletonDescriptor import org.vitrivr.engine.core.model.metamodel.Analyser import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query @@ -13,16 +12,24 @@ import org.vitrivr.engine.core.operators.Operator import org.vitrivr.engine.core.operators.ingest.Extractor import org.vitrivr.engine.core.operators.retrieve.Retriever +/** + * A descriptor used to store a skeleton pose. + * + * @author Ralph Gasser + * @version 1.0.0 + */ class SkeletonPose : Analyser, SkeletonDescriptor> { override val contentClasses = setOf(ContentElement::class) override val descriptorClass = SkeletonDescriptor::class override fun prototype(field: Schema.Field<*, *>): SkeletonDescriptor = SkeletonDescriptor( id = java.util.UUID.randomUUID(), retrievableId = java.util.UUID.randomUUID(), - person = Value.Int(0), - skeleton = List(12) { Value.Float(0.0f) }, - weights = List(12) { Value.Float(0.0f) } - ) // should transient be false? what is transient? + mapOf( + "person" to Value.Int(0), + "skeleton" to Value.FloatVector(FloatArray(12)), + "weights" to Value.FloatVector(FloatArray(12)) + ) + ) override fun newRetrieverForContent(field: Schema.Field, SkeletonDescriptor>, content: Collection>, context: QueryContext): Retriever, SkeletonDescriptor> { TODO("Not yet implemented") diff --git a/vitrivr-engine-module-features/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser b/vitrivr-engine-module-features/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser index b7f83405..bfbe95c4 100644 --- a/vitrivr-engine-module-features/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser +++ b/vitrivr-engine-module-features/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser @@ -4,7 +4,7 @@ org.vitrivr.engine.module.features.feature.external.implementations.clip.CLIP org.vitrivr.engine.module.features.feature.migration.WhisperASR org.vitrivr.engine.module.features.feature.migration.ProvidedOCR org.vitrivr.engine.module.features.feature.migration.AudioTranscription -org.vitrivr.engine.module.features.feature.migration.SkeletonPose +org.vitrivr.engine.module.features.feature.skeleton.SkeletonPose org.vitrivr.engine.module.features.feature.migration.MedianColor org.vitrivr.engine.module.features.feature.migration.CLD org.vitrivr.engine.module.features.feature.migration.CLDReduced15 @@ -24,7 +24,7 @@ org.vitrivr.engine.module.features.feature.migration.HOGMF25k512 org.vitrivr.engine.module.features.feature.migration.SURFMF25K512 org.vitrivr.engine.module.features.feature.migration.InceptionResNetV2 org.vitrivr.engine.module.features.feature.migration.ConceptMasksADE20k -org.vitrivr.engine.module.features.feature.migration.AverageColorRaster +org.vitrivr.engine.module.features.feature.averagecolorraster.AverageColorRaster org.vitrivr.engine.module.features.feature.migration.EdgeARP88 org.vitrivr.engine.module.features.feature.migration.DominantColor org.vitrivr.engine.module.features.feature.migration.DominantEdgeGrid8 \ No newline at end of file diff --git a/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/DenseEmbedding.kt b/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/DenseEmbedding.kt index d510689f..1ad33ea8 100644 --- a/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/DenseEmbedding.kt +++ b/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/DenseEmbedding.kt @@ -63,11 +63,11 @@ companion object { when (element) { is ImageContent -> { val index = imageContents.indexOf(element) - listOf(FloatVectorDescriptor(UUID.randomUUID(), null, imageResults!![index].map { Value.Float(it) }, null)) + listOf(FloatVectorDescriptor(UUID.randomUUID(), null, Value.FloatVector(imageResults!![index].toFloatArray()), null)) } is TextContent -> { val index = textContents.indexOf(element) - listOf(FloatVectorDescriptor(UUID.randomUUID(), null, textResults!![index].map { Value.Float(it) }, null)) + listOf(FloatVectorDescriptor(UUID.randomUUID(), null, Value.FloatVector(textResults!![index].toFloatArray()), null)) } else -> throw(IllegalArgumentException("Content type not supported")) } @@ -86,7 +86,7 @@ companion object { override fun prototype(field: Schema.Field<*, *>) : FloatVectorDescriptor { //convert to integer val length = field.parameters[LENGTH_PARAMETER_NAME]?.toIntOrNull() ?: LENGTH_PARAMETER_DEFAULT - return FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(length) { Value.Float(0.0f) }) + return FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(length)) } /** @@ -100,9 +100,9 @@ companion object { */ override fun newRetrieverForQuery(field: Schema.Field, FloatVectorDescriptor>, query: Query, context: QueryContext): Retriever, FloatVectorDescriptor> { require(field.analyser == this) { "The field '${field.fieldName}' analyser does not correspond with this analyser. This is a programmer's error!" } - require(query is ProximityQuery<*> && query.value.first() is Value.Float) { "The query is not a ProximityQuery." } + require(query is ProximityQuery<*> && query.value is Value.FloatVector) { "The query is not a ProximityQuery." } @Suppress("UNCHECKED_CAST") - return DenseRetriever(field, query as ProximityQuery, context) + return DenseRetriever(field, query as ProximityQuery, context) } /** diff --git a/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/ImageClassification.kt b/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/ImageClassification.kt index 34d5af6b..4d2f902b 100644 --- a/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/ImageClassification.kt +++ b/vitrivr-engine-module-fes/src/main/kotlin/org/vitrivr/engine/base/features/external/implementations/ImageClassification.kt @@ -52,7 +52,7 @@ class ImageClassification : ExternalFesAnalyser() for (result in results) { val filteredResults = result - .mapIndexed { index, score -> LabelDescriptor(UUID.randomUUID(), null, Value.String(classes[index]), Value.Float(score), null) } + .mapIndexed { index, score -> LabelDescriptor(UUID.randomUUID(), null, mapOf("label" to Value.String(classes[index]), "confidence" to Value.Float(score)), null) } .filter { it.confidence.value >= threshold } .sortedByDescending { it.confidence.value } .take(topk) @@ -74,7 +74,7 @@ class ImageClassification : ExternalFesAnalyser() * @return [LabelDescriptor] */ override fun prototype(field: Schema.Field<*,*>): LabelDescriptor { - return LabelDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.String(""), Value.Float(0.0f)) + return LabelDescriptor(UUID.randomUUID(), UUID.randomUUID(), mapOf("label" to Value.String(""), "confidence" to Value.Float(0.0f))) } override fun newRetrieverForContent(field: Schema.Field, content: Collection, context: QueryContext): Retriever { @@ -102,7 +102,7 @@ class ImageClassification : ExternalFesAnalyser() val batchSize = context.getProperty(name, BATCHSIZE_PARAMETER_NAME)?.toIntOrNull() ?: BATCHSIZE_PARAMETER_DEFAULT.toInt() return object : FesExtractor(input, null, batchSize) { override fun assignRetrievableId(descriptor: LabelDescriptor, retrievableId: RetrievableId): LabelDescriptor { - return descriptor.copy(retrievableId = retrievableId) + return LabelDescriptor(descriptor.id, retrievableId, descriptor.values(), descriptor.field) } } } @@ -124,7 +124,7 @@ class ImageClassification : ExternalFesAnalyser() val batchSize = context.getProperty(field.fieldName, BATCHSIZE_PARAMETER_NAME)?.toIntOrNull() ?: BATCHSIZE_PARAMETER_DEFAULT.toInt() return object : FesExtractor(input, field, batchSize) { override fun assignRetrievableId(descriptor: LabelDescriptor, retrievableId: RetrievableId): LabelDescriptor { - return descriptor.copy(retrievableId = retrievableId, field = field) + return LabelDescriptor(descriptor.id, retrievableId, descriptor.values(), field) } } } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt index b41be7aa..d22d3d47 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt @@ -5,13 +5,11 @@ import org.vitrivr.engine.core.database.Connection import org.vitrivr.engine.core.database.ConnectionProvider import org.vitrivr.engine.core.database.descriptor.DescriptorProvider import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.RasterDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.SkeletonDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.MediaDimensionsDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.Rectangle2DMetadataDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.TemporalMetadataDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.FileSourceMetadataDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.VideoSourceMetadataDescriptor import org.vitrivr.engine.core.model.descriptor.vector.* import org.vitrivr.engine.core.model.metamodel.Schema @@ -76,8 +74,6 @@ class PgVectorConnectionProvider: AbstractConnectionProvider() { this.register(TemporalMetadataDescriptor::class, StructDescriptorProvider) this.register(Rectangle2DMetadataDescriptor::class, StructDescriptorProvider) this.register(MediaDimensionsDescriptor::class, StructDescriptorProvider) - this.register(SkeletonDescriptor::class, StructDescriptorProvider) - this.register(RasterDescriptor::class, StructDescriptorProvider) this.register(MapStructDescriptor::class, StructDescriptorProvider) } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt index 0aaa7e5f..7d869164 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt @@ -138,7 +138,8 @@ class PgBitVector () : PGobject(), PGBinaryObject, Serializable { * * @return an array */ - fun toArray(): BooleanArray { + fun toArray(): BooleanArray? { + if (this.vec == null) return null val bits = BooleanArray(this.length) for (i in 0 until this.length) { bits[i] = ((this.vec!![i / 8].toInt() shr (7 - (i % 8))) and 1) == 1 diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt index 58c5db4f..85a898c8 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt @@ -21,17 +21,21 @@ class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, conn /* Add columns for each field in the struct. */ for (field in this.field.analyser.prototype(this.field).schema()) { - require(field.dimensions.size <= 1) { "Cottontail DB currently doesn't support tensor types."} when (field.type) { - Type.STRING -> statement.append("\"${field.name}\" varchar(255), ") - Type.BOOLEAN -> statement.append("\"${field.name}\" boolean, ") - Type.BYTE -> statement.append("$\"{field.name}\" smallint, ") - Type.SHORT -> statement.append("\"${field.name}\" smallint, ") - Type.INT -> statement.append("\"${field.name}\" integer, ") - Type.LONG -> statement.append("\"${field.name}\" bigint, ") - Type.FLOAT -> statement.append("\"${field.name}\" real, ") - Type.DOUBLE -> statement.append("\"${field.name}\" double precision, ") - Type.DATETIME -> statement.append("\"${field.name}\" datetime, ") + Type.String -> statement.append("\"${field.name}\" varchar(255), ") + Type.Boolean -> statement.append("\"${field.name}\" boolean, ") + Type.Byte -> statement.append("$\"{field.name}\" smallint, ") + Type.Short -> statement.append("\"${field.name}\" smallint, ") + Type.Int -> statement.append("\"${field.name}\" integer, ") + Type.Long -> statement.append("\"${field.name}\" bigint, ") + Type.Float -> statement.append("\"${field.name}\" real, ") + Type.Double -> statement.append("\"${field.name}\" double precision, ") + Type.Datetime -> statement.append("\"${field.name}\" datetime, ") + is Type.BooleanVector -> statement.append("\"${field.name}\" vector(${field.type.dimensions}), ") + is Type.DoubleVector -> statement.append("\"${field.name}\" vector(${field.type.dimensions}), ") + is Type.FloatVector -> statement.append("\"${field.name}\" vector(${field.type.dimensions}), ") + is Type.IntVector -> statement.append("\"${field.name}\" vector(${field.type.dimensions}), ") + is Type.LongVector -> statement.append("\"${field.name}\" vector(${field.type.dimensions}), ") } } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt index 44e452c9..45b9c3af 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt @@ -13,6 +13,8 @@ import org.vitrivr.engine.database.pgvector.LOGGER import org.vitrivr.engine.database.pgvector.PgVectorConnection import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorReader +import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector +import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector import java.sql.ResultSet import java.util.* import kotlin.reflect.full.primaryConstructor @@ -69,15 +71,18 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio for (field in this.prototype.schema()) { parameters.add( when(field.type) { - Type.STRING -> result.getString(field.name) - Type.BOOLEAN -> result.getBoolean(field.name) - Type.BYTE -> result.getByte(field.name) - Type.SHORT -> result.getShort(field.name) - Type.INT -> result.getInt(field.name) - Type.LONG -> result.getLong(field.name) - Type.FLOAT -> result.getFloat(field.name) - Type.DOUBLE -> result.getDouble(field.name) - Type.DATETIME -> result.getDate(field.name).toLocalDate() + Type.String -> result.getString(field.name) + Type.Boolean -> result.getBoolean(field.name) + Type.Byte -> result.getByte(field.name) + Type.Short -> result.getShort(field.name) + Type.Int -> result.getInt(field.name) + Type.Long -> result.getLong(field.name) + Type.Float -> result.getFloat(field.name) + Type.Double -> result.getDouble(field.name) + Type.Datetime -> result.getDate(field.name).toLocalDate() + is Type.BooleanVector -> result.getObject(field.name, PgBitVector::class.java) + is Type.FloatVector -> result.getObject(field.name, PgVector::class.java) + else -> throw IllegalArgumentException("Unsupported type ${field.type} in struct descriptor.") } ) } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt index 114979cb..af3a8a8c 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt @@ -2,15 +2,20 @@ package org.vitrivr.engine.database.pgvector.descriptor.struct import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.types.Type import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME import org.vitrivr.engine.database.pgvector.LOGGER import org.vitrivr.engine.database.pgvector.PgVectorConnection import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorWriter +import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector +import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector import java.sql.Date +import java.sql.JDBCType import java.sql.PreparedStatement import java.sql.SQLException +import java.sql.SQLType import java.util.* /** @@ -30,10 +35,16 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio override fun add(item: StructDescriptor): Boolean { try { this.prepareInsertStatement().use { stmt -> - stmt.setAny(1, item.id) - stmt.setAny(2, item.retrievableId) - for ((i, v) in item.values().withIndex()) { - stmt.setAny(3 + i, v) + stmt.setObject(1, item.id) + stmt.setObject(2, item.retrievableId) + var i = 3 + for (attribute in item.schema()) { + val value = item.values()[attribute.name] + if (value != null) { + stmt.setValue(i++, value) + } else { + stmt.setNull(i++, attribute.type.toSqlType()) + } } return stmt.execute() } @@ -53,10 +64,16 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio try { this.prepareInsertStatement().use { stmt -> for (item in items) { - stmt.setAny(1, item.id) - stmt.setAny(2, item.retrievableId) - for ((i, v) in item.values().withIndex()) { - stmt.setAny(3 + i, v.second) + stmt.setObject(1, item.id) + stmt.setObject(2, item.retrievableId) + var i = 3 + for (attribute in item.schema()) { + val value = item.values()[attribute.name] + if (value != null) { + stmt.setValue(i++, value) + } else { + stmt.setNull(i++, attribute.type.toSqlType()) + } } stmt.addBatch() } @@ -77,12 +94,16 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio override fun update(item: StructDescriptor): Boolean { try { this.prepareUpdateStatement().use { stmt -> - val values = item.values() stmt.setObject(1, item.retrievableId) - for ((i, v) in values.withIndex()) { - stmt.setAny(2 + i, v) + var i = 2 + for (attribute in item.schema()) { + val value = item.values()[attribute.name] + if (value != null) { + stmt.setValue(i++, value) + } else { + stmt.setNull(i++, attribute.type.toSqlType()) + } } - stmt.setObject(1 + values.size, item.id) return stmt.execute() } } catch (e: SQLException) { @@ -94,38 +115,40 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio /** * Sets a value of [Value] type in a [PreparedStatement]. */ - private fun PreparedStatement.setValue(index: Int, value: Value<*>) { - when(value) { - is Value.Boolean -> this.setBoolean(index, value.value) - is Value.Byte -> this.setByte(index, value.value) - is Value.DateTime -> this.setDate(index, Date(value.value.toInstant().toEpochMilli())) - is Value.Double -> this.setDouble(index, value.value) - is Value.Float -> this.setFloat(index, value.value) - is Value.Int -> this.setInt(index, value.value) - is Value.Long -> this.setLong(index, value.value) - is Value.Short -> this.setShort(index, value.value) - is Value.String -> this.setString(index, value.value) - } + private fun PreparedStatement.setValue(index: Int, value: Value<*>) = when (value) { + is Value.Boolean -> this.setBoolean(index, value.value) + is Value.Byte -> this.setByte(index, value.value) + is Value.DateTime -> this.setDate(index, Date(value.value.toInstant().toEpochMilli())) + is Value.Double -> this.setDouble(index, value.value) + is Value.Float -> this.setFloat(index, value.value) + is Value.Int -> this.setInt(index, value.value) + is Value.Long -> this.setLong(index, value.value) + is Value.Short -> this.setShort(index, value.value) + is Value.String -> this.setString(index, value.value) + is Value.BooleanVector -> this.setObject(index, PgBitVector(value.value)) + is Value.FloatVector -> this.setObject(index, PgVector(value.value)) + else -> throw IllegalArgumentException("Unsupported value type for vector value.") } /** - * Sets a value of [Any] type in a [PreparedStatement]. + * Converts a [Type] to a [SQLType]. */ - private fun PreparedStatement.setAny(index: Int, value: Any?) { - when (value) { - is String -> this.setString(index, value) - is Int -> this.setInt(index, value) - is Long -> this.setLong(index, value) - is Float -> this.setFloat(index, value) - is Double -> this.setDouble(index, value) - is Boolean -> this.setBoolean(index, value) - is ByteArray -> this.setBytes(index, value) - is UUID -> this.setObject(index, value) - is Value<*> -> this.setValue(index, value) - else -> this.setObject(index, value) - } - } - + private fun Type.toSqlType(): Int = when (this) { + Type.Boolean -> JDBCType.BOOLEAN + Type.Byte -> JDBCType.TINYINT + Type.Short -> JDBCType.SMALLINT + Type.Int -> JDBCType.INTEGER + Type.Long -> JDBCType.BIGINT + Type.Float -> JDBCType.REAL + Type.Double -> JDBCType.DOUBLE + Type.Datetime -> JDBCType.DATE + Type.String -> JDBCType.VARCHAR + is Type.BooleanVector -> JDBCType.ARRAY + is Type.DoubleVector -> JDBCType.ARRAY + is Type.FloatVector -> JDBCType.ARRAY + is Type.IntVector -> JDBCType.ARRAY + is Type.LongVector -> JDBCType.ARRAY + }.ordinal /** * Prepares an INSERT statement for this [StructDescriptorWriter]. * diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt index 7ec7e4fa..67694a3d 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt @@ -9,7 +9,7 @@ import org.vitrivr.engine.core.model.query.basics.Distance.* import org.vitrivr.engine.core.model.query.proximity.ProximityQuery import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.core.model.retrievable.attributes.DistanceAttribute -import org.vitrivr.engine.core.model.types.toValue +import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.database.pgvector.* import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorReader import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector @@ -33,7 +33,11 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec try { val statement = when (query) { is ProximityQuery<*> -> this.connection.jdbc.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").apply { - setObject(1, PgVector(query.value)) + when (val vector = query.value) { + is Value.FloatVector -> this.setObject(1, PgVector(vector.value)) + is Value.BooleanVector -> this.setObject(1, PgBitVector(vector.value)) + else -> throw IllegalArgumentException("Unsupported value type ${query.value::class}.") + } } else -> throw UnsupportedOperationException("Query of typ ${query::class} is not supported by VectorDescriptorReader.") } @@ -58,7 +62,11 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec is ProximityQuery<*> -> { val descriptors = mutableListOf, Float>>() this.connection.jdbc.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").use { stmt -> - stmt.setObject(1, PgVector(query.value)) + when (val vector = query.value) { + is Value.FloatVector -> stmt.setObject(1, PgVector(vector.value)) + is Value.BooleanVector -> stmt.setObject(1, PgBitVector(vector.value)) + else -> throw IllegalArgumentException("Unsupported value type ${query.value::class}.") + } stmt.executeQuery().use { result -> while (result.next()) { descriptors.add(this@VectorDescriptorReader.rowToDescriptor(result) to result.getFloat(DISTANCE_COLUMN_NAME)) @@ -101,39 +109,19 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec descriptorId, retrievableId, result.getObject(DESCRIPTOR_COLUMN_NAME, PgVector::class.java).let { vector -> - vector.toArray()?.map { it.toValue() } ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") - } - ) - - is DoubleVectorDescriptor -> DoubleVectorDescriptor( - descriptorId, - retrievableId, - result.getObject(DESCRIPTOR_COLUMN_NAME, PgVector::class.java).let { vector -> - vector.toArray()?.map { it.toDouble().toValue() } ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") - } - ) - - is IntVectorDescriptor -> IntVectorDescriptor( - descriptorId, - retrievableId, - result.getObject(DESCRIPTOR_COLUMN_NAME, PgVector::class.java).let { vector -> - vector.toArray()?.map { it.toInt().toValue() } ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + Value.FloatVector(vector.toArray() ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) } ) - is LongVectorDescriptor -> LongVectorDescriptor( + is BooleanVectorDescriptor -> BooleanVectorDescriptor( descriptorId, retrievableId, - result.getObject(DESCRIPTOR_COLUMN_NAME, PgVector::class.java).let { vector -> - vector.toArray()?.map { it.toLong().toValue() } ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + result.getObject(DESCRIPTOR_COLUMN_NAME, PgBitVector::class.java).let { vector -> + Value.BooleanVector(vector.toArray() ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) } ) - is BooleanVectorDescriptor -> BooleanVectorDescriptor( - descriptorId, - retrievableId, - result.getObject(DESCRIPTOR_COLUMN_NAME, PgBitVector::class.java).let { vector -> vector.toArray().map { it.toValue() } } - ) + else -> throw IllegalArgumentException("Unsupported descriptor type ${this.prototype::class}.") } } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt index ea2d424b..7037859e 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt @@ -1,9 +1,12 @@ package org.vitrivr.engine.database.pgvector.descriptor.vector +import org.vitrivr.engine.core.model.descriptor.vector.FloatVectorDescriptor import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.database.pgvector.* import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorWriter +import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector import java.sql.SQLException @@ -25,7 +28,11 @@ class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connec this.connection.jdbc.prepareStatement("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $DESCRIPTOR_COLUMN_NAME) VALUES (?, ?, ?);").use { stmt -> stmt.setObject(1, item.id) stmt.setObject(2, item.retrievableId) - stmt.setObject(3, PgVector(item.vector)) + when (val vector = item.vector) { + is Value.FloatVector -> stmt.setObject(3, PgVector(vector.value)) + is Value.BooleanVector -> stmt.setObject(3, PgBitVector(vector.value)) + else -> throw IllegalArgumentException("Unsupported vector type ${vector::class.simpleName}") + } return stmt.execute() } } catch (e: SQLException) { @@ -46,7 +53,11 @@ class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connec for (item in items) { stmt.setObject(1, item.id) stmt.setObject(2, item.retrievableId) - stmt.setObject(3, PgVector(item.vector)) + when (val vector = item.vector) { + is Value.FloatVector -> stmt.setObject(3, PgVector(vector.value)) + is Value.BooleanVector -> stmt.setObject(3, PgBitVector(vector.value)) + else -> throw IllegalArgumentException("Unsupported vector type ${vector::class.simpleName}") + } stmt.addBatch() } return stmt.executeBatch().all { it == 1 } @@ -67,7 +78,11 @@ class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connec try { this.connection.jdbc.prepareStatement("UPDATE $tableName SET $RETRIEVABLE_ID_COLUMN_NAME = ?, $DESCRIPTOR_COLUMN_NAME = ? WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?;").use { stmt -> stmt.setObject(1, item.retrievableId) - stmt.setObject(2, PgVector(item.vector)) + when (val vector = item.vector) { + is Value.FloatVector -> stmt.setObject(2, PgVector(vector.value)) + is Value.BooleanVector -> stmt.setObject(2, PgBitVector(vector.value)) + else -> throw IllegalArgumentException("Unsupported vector type ${vector::class.simpleName}") + } stmt.setObject(3, item.id) return stmt.execute() } diff --git a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookupFactory.kt b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookupFactory.kt index c2bef129..d2c2d62c 100644 --- a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookupFactory.kt +++ b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookupFactory.kt @@ -17,7 +17,7 @@ class FieldLookupFactory() : TransformerFactory { if(providedKeys.length == 1 && providedKeys == "*") { schemaField.analyser.prototype(schemaField).schema().map { it.name } }else{ - providedKeys.split(",").map { s -> s.trim() } ?: emptyList() + providedKeys.split(",").map { s -> s.trim() } } }else{ emptyList() diff --git a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt index 828dc844..97779d7b 100644 --- a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt +++ b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt @@ -92,7 +92,7 @@ class QueryParser(val schema: Schema) { val descriptor = reader.getFor(id).firstOrNull() ?: throw IllegalArgumentException("No retrievable with id '$id' present in ${field.fieldName}") field.getRetrieverForDescriptor(descriptor, description.context) } - is VectorInputData -> field.getRetrieverForDescriptor(FloatVectorDescriptor(vector = input.data.map { Value.Float(it) }), description.context) + is VectorInputData -> field.getRetrieverForDescriptor(FloatVectorDescriptor(vector = Value.FloatVector(input.data.toFloatArray())), description.context) else -> { /* Is this a boolean sub-field query ? */ if(fieldAndAttributeName.second != null && input.comparison != null){ @@ -101,26 +101,26 @@ class QueryParser(val schema: Schema) { /* For now, we support not all input data */ val value = when(input){ is TextInputData -> { - require(subfield.type == Type.STRING){"The given sub-field ${fieldAndAttributeName.first}.${fieldAndAttributeName.second}'s type is ${subfield.type}, which is not the expexted ${Type.STRING}"} + require(subfield.type == Type.String) { "The given sub-field ${fieldAndAttributeName.first}.${fieldAndAttributeName.second}'s type is ${subfield.type}, which is not the expected ${Type.String}" } Value.String(input.data) } is BooleanInputData -> { - require(subfield.type == Type.BOOLEAN){"The given sub-field ${fieldAndAttributeName.first}.${fieldAndAttributeName.second}'s type is ${subfield.type}, which is not the expexted ${Type.BOOLEAN}"} + require(subfield.type == Type.Boolean) { "The given sub-field ${fieldAndAttributeName.first}.${fieldAndAttributeName.second}'s type is ${subfield.type}, which is not the expected ${Type.Boolean}" } Value.Boolean(input.data) } is NumericInputData -> { when(subfield.type){ - Type.DOUBLE -> Value.Double(input.data) - Type.INT -> Value.Int(input.data.toInt()) - Type.LONG -> Value.Long(input.data.toLong()) - Type.SHORT -> Value.Short(input.data.toInt().toShort()) - Type.BYTE -> Value.Byte(input.data.toInt().toByte()) - Type.FLOAT -> Value.Float(input.data.toFloat()) + Type.Double -> Value.Double(input.data) + Type.Float -> Value.Float(input.data.toFloat()) + Type.Long -> Value.Long(input.data.toLong()) + Type.Int -> Value.Int(input.data.toInt()) + Type.Short -> Value.Short(input.data.toInt().toShort()) + Type.Byte -> Value.Byte(input.data.toInt().toByte()) else -> throw IllegalArgumentException("Cannot work with NumericInputData $input but non-numerical sub-field $subfield") } } is DateInputData -> { - require(subfield.type == Type.DATETIME){"The given sub-field ${fieldAndAttributeName.first}.${fieldAndAttributeName.second}'s type is ${subfield.type}, which is not the expexted ${Type.DATETIME}"} + require(subfield.type == Type.Datetime) { "The given sub-field ${fieldAndAttributeName.first}.${fieldAndAttributeName.second}'s type is ${subfield.type}, which is not the expected ${Type.Datetime}" } Value.DateTime(input.parseDate()) } else -> throw UnsupportedOperationException("Subfield query for $input is currently not supported") diff --git a/vitrivr-engine-server/src/main/kotlin/org/vitrivr/engine/tools/CineastMigrationTool.kt b/vitrivr-engine-server/src/main/kotlin/org/vitrivr/engine/tools/CineastMigrationTool.kt index f52ff85c..0cf7cc89 100644 --- a/vitrivr-engine-server/src/main/kotlin/org/vitrivr/engine/tools/CineastMigrationTool.kt +++ b/vitrivr-engine-server/src/main/kotlin/org/vitrivr/engine/tools/CineastMigrationTool.kt @@ -14,8 +14,6 @@ import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.scalar.FloatDescriptor import org.vitrivr.engine.core.model.descriptor.scalar.StringDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.RasterDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.SkeletonDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.MediaDimensionsDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.TemporalMetadataDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.FileSourceMetadataDescriptor @@ -27,6 +25,8 @@ import org.vitrivr.engine.core.model.retrievable.Ingested import org.vitrivr.engine.core.model.retrievable.Retrievable import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Value +import org.vitrivr.engine.module.features.feature.averagecolorraster.RasterDescriptor +import org.vitrivr.engine.module.features.feature.skeleton.SkeletonDescriptor import java.io.File import java.io.FileReader import java.nio.file.Files @@ -142,7 +142,7 @@ data class CineastVectorFeature(override val id: String, val feature: List, return RasterDescriptor( id = DescriptorId.randomUUID(), retrievableId = RetrievableId.fromString(idmap[id]), - hist = hist.map { Value.Float(it) }, - raster = raster.map { Value.Float(it) }, + mapOf( + "hist" to Value.FloatVector(hist.toFloatArray()), + "raster" to Value.FloatVector(raster.toFloatArray()) + ) ) } } @@ -330,8 +334,10 @@ class CineastMigrationTool(val migrationconfigpath: String, val schemaconfigpath val fileMetadataDescriptor = FileSourceMetadataDescriptor( id = DescriptorId.randomUUID(), retrievableId = objectRetrievable.id, - path = Value.String(mobject.path), - size = Value.Long(size), + mapOf( + "path" to Value.String(mobject.path), + "size" to Value.Long(size), + ) ) filemetadatawriter.add(fileMetadataDescriptor) retrievableWriter.add(objectRetrievable) @@ -382,8 +388,11 @@ class CineastMigrationTool(val migrationconfigpath: String, val schemaconfigpath id = DescriptorId.randomUUID(), retrievableId = RetrievableId.fromString(retrievableId) ?: throw IllegalArgumentException("Could not find retrievable id for object ${mobjectmetadata.objectid}"), - width = Value.Int(width), - height = Value.Int(height) + mapOf( + "width" to Value.Int(width), + "height" to Value.Int(height) + ) + ) mediadimensionswriter.add(dimensionsDescriptor) } @@ -443,8 +452,11 @@ class CineastMigrationTool(val migrationconfigpath: String, val schemaconfigpath val temporalMetadataDescriptor = TemporalMetadataDescriptor( id = DescriptorId.randomUUID(), retrievableId = ingested.id, - startNs = Value.Long(segment.segmentstartabs.toLong() * 1000 * 1000 * 1000), - endNs = Value.Long(segment.segmentendabs.toLong() * 1000 * 1000 * 1000), + mapOf( + "start" to Value.Long(segment.segmentstartabs.toLong() * 1000 * 1000 * 1000), + "end" to Value.Long(segment.segmentendabs.toLong() * 1000 * 1000 * 1000), + ) + ) ingestedList.add(ingested) From d06fa57ca96f9cc2b3cd286dc94305e2d12ba90c Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Tue, 9 Jul 2024 06:34:04 +0200 Subject: [PATCH 15/71] Minor optimisations. Signed-off-by: Ralph Gasser --- .../AbstractDescriptorInitializer.kt | 3 + .../descriptor/AbstractDescriptorWriter.kt | 85 +++++++++++++++++-- .../struct/StructDescriptorInitializer.kt | 2 +- .../struct/StructDescriptorWriter.kt | 81 +----------------- .../vector/VectorDescriptorInitializer.kt | 3 +- .../vector/VectorDescriptorWriter.kt | 1 - 6 files changed, 86 insertions(+), 89 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt index adca6fe4..78f39892 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt @@ -19,6 +19,9 @@ abstract class AbstractDescriptorInitializer(final override val /** The name of the table backing this [AbstractDescriptorInitializer]. */ protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" + /** The [Descriptor] prototype for this [AbstractDescriptorWriter]. */ + protected val prototype = this.field.analyser.prototype(this.field) + /** * Checks if the schema for this [AbstractDescriptorInitializer] has been properly initialized. * diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt index 65a30142..954db10c 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt @@ -3,11 +3,13 @@ package org.vitrivr.engine.database.pgvector.descriptor import org.vitrivr.engine.core.database.descriptor.DescriptorWriter import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ENTITY_PREFIX -import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME -import org.vitrivr.engine.database.pgvector.LOGGER -import org.vitrivr.engine.database.pgvector.PgVectorConnection -import java.sql.SQLException +import org.vitrivr.engine.core.model.types.Type +import org.vitrivr.engine.core.model.types.Value +import org.vitrivr.engine.database.pgvector.* +import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector +import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector +import org.vitrivr.engine.database.pgvector.descriptor.struct.StructDescriptorWriter +import java.sql.* /** * An abstract implementation of a [DescriptorWriter] for PostgreSQL with pgVector. @@ -19,6 +21,9 @@ abstract class AbstractDescriptorWriter(final override val field /** The name of the table backing this [AbstractDescriptorInitializer]. */ protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" + /** The [Descriptor] prototype for this [AbstractDescriptorWriter]. */ + protected val prototype = this.field.analyser.prototype(this.field) + /** * Deletes (writes) a [Descriptor] of type [D] using this [AbstractDescriptorWriter]. * @@ -55,4 +60,74 @@ abstract class AbstractDescriptorWriter(final override val field return false } } + + /** + * Sets a value of [Value] type in a [PreparedStatement]. + */ + protected fun PreparedStatement.setValue(index: Int, value: Value<*>) = when (value) { + is Value.Boolean -> this.setBoolean(index, value.value) + is Value.Byte -> this.setByte(index, value.value) + is Value.DateTime -> this.setDate(index, Date(value.value.toInstant().toEpochMilli())) + is Value.Double -> this.setDouble(index, value.value) + is Value.Float -> this.setFloat(index, value.value) + is Value.Int -> this.setInt(index, value.value) + is Value.Long -> this.setLong(index, value.value) + is Value.Short -> this.setShort(index, value.value) + is Value.String -> this.setString(index, value.value) + is Value.BooleanVector -> this.setObject(index, PgBitVector(value.value)) + is Value.FloatVector -> this.setObject(index, PgVector(value.value)) + else -> throw IllegalArgumentException("Unsupported value type for vector value.") + } + + /** + * Converts a [Type] to a [SQLType]. + */ + protected fun Type.toSqlType(): Int = when (this) { + Type.Boolean -> JDBCType.BOOLEAN + Type.Byte -> JDBCType.TINYINT + Type.Short -> JDBCType.SMALLINT + Type.Int -> JDBCType.INTEGER + Type.Long -> JDBCType.BIGINT + Type.Float -> JDBCType.REAL + Type.Double -> JDBCType.DOUBLE + Type.Datetime -> JDBCType.DATE + Type.String -> JDBCType.VARCHAR + is Type.BooleanVector -> JDBCType.ARRAY + is Type.DoubleVector -> JDBCType.ARRAY + is Type.FloatVector -> JDBCType.ARRAY + is Type.IntVector -> JDBCType.ARRAY + is Type.LongVector -> JDBCType.ARRAY + }.ordinal + + /** + * Prepares an INSERT statement for this [StructDescriptorWriter]. + * + * @return [PreparedStatement] + */ + protected fun prepareUpdateStatement(): PreparedStatement { + val statement = StringBuilder("UPDATE $tableName SET $RETRIEVABLE_ID_COLUMN_NAME = ?") + for (field in this.prototype.schema()) { + statement.append(", \"${field.name}\" = ?") + } + statement.append("WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?;") + return this.connection.jdbc.prepareStatement(statement.toString()) + } + + /** + * Prepares an INSERT statement for this [StructDescriptorWriter]. + * + * @return [PreparedStatement] + */ + protected fun prepareInsertStatement(): PreparedStatement { + val statement = StringBuilder("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME") + for (field in this.prototype.schema()) { + statement.append(", \"${field.name}\"") + } + statement.append(") VALUES (?, ?") + for (field in this.field.analyser.prototype(this.field).schema()) { + statement.append(", ?") + } + statement.append(");") + return this.connection.jdbc.prepareStatement(statement.toString()) + } } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt index 85a898c8..025778fd 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt @@ -20,7 +20,7 @@ class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, conn statement.append("$RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, ") /* Add columns for each field in the struct. */ - for (field in this.field.analyser.prototype(this.field).schema()) { + for (field in this.prototype.schema()) { when (field.type) { Type.String -> statement.append("\"${field.name}\" varchar(255), ") Type.Boolean -> statement.append("\"${field.name}\" boolean, ") diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt index af3a8a8c..ebc40cd7 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt @@ -2,21 +2,10 @@ package org.vitrivr.engine.database.pgvector.descriptor.struct import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.core.model.types.Type -import org.vitrivr.engine.core.model.types.Value -import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME import org.vitrivr.engine.database.pgvector.LOGGER import org.vitrivr.engine.database.pgvector.PgVectorConnection -import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorWriter -import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector -import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector -import java.sql.Date -import java.sql.JDBCType -import java.sql.PreparedStatement import java.sql.SQLException -import java.sql.SQLType -import java.util.* /** * An [AbstractDescriptorWriter] for [StructDescriptor]s. @@ -104,6 +93,7 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio stmt.setNull(i++, attribute.type.toSqlType()) } } + stmt.setObject(i, item.id) return stmt.execute() } } catch (e: SQLException) { @@ -111,73 +101,4 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio return false } } - - /** - * Sets a value of [Value] type in a [PreparedStatement]. - */ - private fun PreparedStatement.setValue(index: Int, value: Value<*>) = when (value) { - is Value.Boolean -> this.setBoolean(index, value.value) - is Value.Byte -> this.setByte(index, value.value) - is Value.DateTime -> this.setDate(index, Date(value.value.toInstant().toEpochMilli())) - is Value.Double -> this.setDouble(index, value.value) - is Value.Float -> this.setFloat(index, value.value) - is Value.Int -> this.setInt(index, value.value) - is Value.Long -> this.setLong(index, value.value) - is Value.Short -> this.setShort(index, value.value) - is Value.String -> this.setString(index, value.value) - is Value.BooleanVector -> this.setObject(index, PgBitVector(value.value)) - is Value.FloatVector -> this.setObject(index, PgVector(value.value)) - else -> throw IllegalArgumentException("Unsupported value type for vector value.") - } - - /** - * Converts a [Type] to a [SQLType]. - */ - private fun Type.toSqlType(): Int = when (this) { - Type.Boolean -> JDBCType.BOOLEAN - Type.Byte -> JDBCType.TINYINT - Type.Short -> JDBCType.SMALLINT - Type.Int -> JDBCType.INTEGER - Type.Long -> JDBCType.BIGINT - Type.Float -> JDBCType.REAL - Type.Double -> JDBCType.DOUBLE - Type.Datetime -> JDBCType.DATE - Type.String -> JDBCType.VARCHAR - is Type.BooleanVector -> JDBCType.ARRAY - is Type.DoubleVector -> JDBCType.ARRAY - is Type.FloatVector -> JDBCType.ARRAY - is Type.IntVector -> JDBCType.ARRAY - is Type.LongVector -> JDBCType.ARRAY - }.ordinal - /** - * Prepares an INSERT statement for this [StructDescriptorWriter]. - * - * @return [PreparedStatement] - */ - private fun prepareUpdateStatement(): PreparedStatement { - val statement = StringBuilder("UPDATE $tableName SET $RETRIEVABLE_ID_COLUMN_NAME = ?") - for (field in this.field.analyser.prototype(this.field).schema()) { - statement.append(", \"${field.name}\" = ?") - } - statement.append("WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?;") - return this.connection.jdbc.prepareStatement(statement.toString()) - } - - /** - * Prepares an INSERT statement for this [StructDescriptorWriter]. - * - * @return [PreparedStatement] - */ - private fun prepareInsertStatement(): PreparedStatement { - val statement = StringBuilder("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME") - for (field in this.field.analyser.prototype(this.field).schema()) { - statement.append(", \"${field.name}\"") - } - statement.append(") VALUES (?, ?") - for (field in this.field.analyser.prototype(this.field).schema()) { - statement.append(", ?") - } - statement.append(");") - return this.connection.jdbc.prepareStatement(statement.toString()) - } } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt index 4694847a..72b84a63 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt @@ -18,10 +18,9 @@ class VectorDescriptorInitializer(field: Schema.Field<*, VectorDescriptor<*>>, c * Initializes the [RetrievableInitializer]. */ override fun initialize() { - val type = this.field.analyser.prototype(this.field) try { /* Create 'retrievable' entity. */ - this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME vector(${type.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME vector(${this.prototype.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") .use { it.execute() } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt index 7037859e..63ec2fed 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt @@ -1,6 +1,5 @@ package org.vitrivr.engine.database.pgvector.descriptor.vector -import org.vitrivr.engine.core.model.descriptor.vector.FloatVectorDescriptor import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.types.Value From f280c73d55e70b53c26dbb21758e575afbe272c6 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Tue, 9 Jul 2024 06:38:35 +0200 Subject: [PATCH 16/71] Fixes handling of BooleanVectors. Signed-off-by: Ralph Gasser --- .../descriptor/struct/StructDescriptorInitializer.kt | 2 +- .../descriptor/vector/VectorDescriptorInitializer.kt | 10 ++++++---- 2 files changed, 7 insertions(+), 5 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt index 025778fd..d5b4f2ca 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt @@ -31,7 +31,7 @@ class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, conn Type.Float -> statement.append("\"${field.name}\" real, ") Type.Double -> statement.append("\"${field.name}\" double precision, ") Type.Datetime -> statement.append("\"${field.name}\" datetime, ") - is Type.BooleanVector -> statement.append("\"${field.name}\" vector(${field.type.dimensions}), ") + is Type.BooleanVector -> statement.append("\"${field.name}\" bit(${field.type.dimensions}), ") is Type.DoubleVector -> statement.append("\"${field.name}\" vector(${field.type.dimensions}), ") is Type.FloatVector -> statement.append("\"${field.name}\" vector(${field.type.dimensions}), ") is Type.IntVector -> statement.append("\"${field.name}\" vector(${field.type.dimensions}), ") diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt index 72b84a63..903625e4 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt @@ -3,6 +3,7 @@ package org.vitrivr.engine.database.pgvector.descriptor.vector import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.database.pgvector.* import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorInitializer import java.sql.SQLException @@ -19,11 +20,12 @@ class VectorDescriptorInitializer(field: Schema.Field<*, VectorDescriptor<*>>, c */ override fun initialize() { try { - /* Create 'retrievable' entity. */ - this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME vector(${this.prototype.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") - .use { - it.execute() + val stmt = if (this.prototype.vector is Value.BooleanVector) { + this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME bit(${this.prototype.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + } else { + this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME vector(${this.prototype.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") } + stmt.use { it.execute() } } catch (e: SQLException) { LOGGER.error(e) { "Failed to initialize entity due to exception." } } From 013e2deaee9793ad0e723d4ff25651d34ecd0358 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Tue, 9 Jul 2024 06:54:02 +0200 Subject: [PATCH 17/71] Minor simplification. Signed-off-by: Ralph Gasser --- .../kotlin/org/vitrivr/engine/core/model/types/Type.kt | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt index fad4c49a..ec835f0e 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt @@ -125,34 +125,34 @@ sealed interface Type { * A [Type] that represents a [BooleanVector] value. */ data class BooleanVector(override val dimensions: kotlin.Int) : Type { - override fun defaultValue(): Value<*> = Value.BooleanVector(BooleanArray(this.dimensions)) + override fun defaultValue(): Value<*> = Value.BooleanVector(this.dimensions) } /** * A [Type] that represents a [IntVector] value. */ data class IntVector(override val dimensions: kotlin.Int) : Type { - override fun defaultValue(): Value<*> = Value.IntVector(IntArray(this.dimensions)) + override fun defaultValue(): Value<*> = Value.IntVector(this.dimensions) } /** * A [Type] that represents a [LongVector] value. */ data class LongVector(override val dimensions: kotlin.Int) : Type { - override fun defaultValue(): Value<*> = Value.LongVector(LongArray(this.dimensions)) + override fun defaultValue(): Value<*> = Value.LongVector(this.dimensions) } /** * A [Type] that represents a [FloatVector] value. */ data class FloatVector(override val dimensions: kotlin.Int) : Type { - override fun defaultValue(): Value<*> = Value.FloatVector(FloatArray(this.dimensions)) + override fun defaultValue(): Value<*> = Value.FloatVector(this.dimensions) } /** * A [Type] that represents a [DoubleVector] value. */ data class DoubleVector(override val dimensions: kotlin.Int) : Type { - override fun defaultValue(): Value<*> = Value.DoubleVector(DoubleArray(this.dimensions)) + override fun defaultValue(): Value<*> = Value.DoubleVector(this.dimensions) } } From 375e3d953b7cfd82cde89176ac70652a06bd447e Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Tue, 9 Jul 2024 09:25:23 +0200 Subject: [PATCH 18/71] StringConnection now indicates that it does not support transaction when using the respective method. --- .../features/database/string/StringConnection.kt | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt index b961ef59..110c3464 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt @@ -1,5 +1,7 @@ package org.vitrivr.engine.module.features.database.string +import io.github.oshai.kotlinlogging.KLogger +import io.github.oshai.kotlinlogging.KotlinLogging.logger import org.vitrivr.engine.core.database.AbstractConnection import org.vitrivr.engine.core.database.retrievable.NoRetrievableInitializer import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer @@ -9,8 +11,20 @@ import org.vitrivr.engine.core.model.Persistable import org.vitrivr.engine.module.features.database.string.writer.StringRetrievableWriter + +/** Defines [KLogger] of the class. */ +internal val LOGGER: KLogger = logger("org.vitrivr.engine.module.database.string.StringConnection") + class StringConnection(override val provider: StringConnectionProvider, schemaName: String, internal val stringify: (Persistable) -> String) : AbstractConnection(schemaName, provider) { + /** + * [StringConnection] does not support transactions. + */ + override fun withTransaction(action: (Unit) -> T): T { + LOGGER.warn { "Transactions are not supported by the StringConnection." } + return action.invoke(Unit) + } + override fun getRetrievableInitializer(): RetrievableInitializer = NoRetrievableInitializer() override fun getRetrievableWriter(): RetrievableWriter = StringRetrievableWriter(this, provider.targetStream, stringify) From 1888f9c12bcfe72bbdda9d7a20d154d08f388b72 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Tue, 9 Jul 2024 09:25:37 +0200 Subject: [PATCH 19/71] PersistingSink now uses transactions. --- .../core/operators/persistence/PersistingSink.kt | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/persistence/PersistingSink.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/persistence/PersistingSink.kt index e84a0466..9fb7c824 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/persistence/PersistingSink.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/operators/persistence/PersistingSink.kt @@ -61,11 +61,13 @@ class PersistingSink(override val input: Operator, val context: Ind collect(retrievable, Triple(retrievables, relationships, descriptors)) /* Write entities to database. */ - this.writer.addAll(retrievables) - this.writer.connectAll(relationships) - for ((f, d) in descriptors) { - val writer = f.let { field -> this.descriptorWriters.computeIfAbsent(field) { it.getWriter() } } as? DescriptorWriter - writer?.addAll(d) + this.writer.connection.withTransaction { + this.writer.addAll(retrievables) + this.writer.connectAll(relationships) + for ((f, d) in descriptors) { + val writer = f.let { field -> this.descriptorWriters.computeIfAbsent(field) { it.getWriter() } } as? DescriptorWriter + writer?.addAll(d) + } } } From 8c2106c302c44f5ca60722d217284817df8658c4 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Tue, 9 Jul 2024 09:33:30 +0200 Subject: [PATCH 20/71] Makes when exhaustive as suggested by @lucaro. --- .../features/metadata/source/exif/ExifMetadataExtractor.kt | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataExtractor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataExtractor.kt index a215a7c4..c24eee29 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataExtractor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/metadata/source/exif/ExifMetadataExtractor.kt @@ -57,7 +57,12 @@ private fun convertType(directory: Directory, tagType: Int, type: Type): Value<* Type.Long -> Value.Long(directory.getLong(tagType)) Type.Short -> Value.Short(directory.getObject(tagType) as Short) Type.String -> Value.String(directory.getString(tagType)) - else -> throw IllegalArgumentException("Unsupported type: $type") + Type.Text -> Value.String(directory.getString(tagType)) + is Type.BooleanVector -> throw IllegalArgumentException("Unsupported type: $type") + is Type.DoubleVector -> throw IllegalArgumentException("Unsupported type: $type") + is Type.FloatVector -> throw IllegalArgumentException("Unsupported type: $type") + is Type.IntVector -> throw IllegalArgumentException("Unsupported type: $type") + is Type.LongVector -> throw IllegalArgumentException("Unsupported type: $type") } private fun JsonElement.convertType(type: Type): Value<*>? { From b8aab1fbf0a3e387dd9f1153cf428f4ac51c9953 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Tue, 9 Jul 2024 09:33:54 +0200 Subject: [PATCH 21/71] Adds an additional TEXT type (VARCHAR vs TEXT in relational databases). --- .../org/vitrivr/engine/core/model/types/Type.kt | 15 ++++++++++++++- .../vitrivr/engine/plugin/cottontaildb/Common.kt | 1 + .../features/database/string/StringConnection.kt | 4 ---- .../descriptor/AbstractDescriptorWriter.kt | 1 + .../struct/StructDescriptorInitializer.kt | 1 + 5 files changed, 17 insertions(+), 5 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt index ec835f0e..8f3928aa 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Type.kt @@ -20,6 +20,7 @@ sealed interface Type { */ fun valueOf(type: kotlin.String, dimensions: kotlin.Int = 0): Type { return when (type.uppercase()) { + "TEXT" -> Text "STRING" -> String "BOOLEAN" -> Boolean "BYTE" -> Byte @@ -50,7 +51,19 @@ sealed interface Type { fun defaultValue(): Value<*> /** - * A [Type] that represents a String. + * A [Type] that represents a (longer) text. + * + * Primarily needed, since simple strings and longer texts are treated differently by certain databases. + */ + data object Text : Type { + override val dimensions: kotlin.Int = 1 + override fun defaultValue(): Value<*> = Value.String("") + } + + /** + * A [Type] that represents a simple String. + * + * Primarily needed, since simple strings and longer texts are treated differently by certain databases. */ data object String : Type { override val dimensions: kotlin.Int = 1 diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt index 1158b204..6de6c88f 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt @@ -55,6 +55,7 @@ const val SCORE_COLUMN_NAME = "score" */ internal fun Type.toCottontailType(): Types<*> = when (this) { Type.String -> Types.String + Type.Text -> Types.String Type.Byte -> Types.Byte Type.Short -> Types.Short Type.Boolean -> Types.Boolean diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt index e97490b1..816f80df 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/database/string/StringConnection.kt @@ -16,10 +16,6 @@ import org.vitrivr.engine.module.features.database.string.writer.StringRetrievab internal val LOGGER: KLogger = logger("org.vitrivr.engine.module.database.string.StringConnection") class StringConnection(override val provider: StringConnectionProvider, schemaName: String, internal val stringify: (Persistable) -> String) : AbstractConnection(schemaName, provider) { - override fun withTransaction(action: (Unit) -> T): T { - throw UnsupportedOperationException("Transactions are not supported by StringConnection.") - } - /** * [StringConnection] does not support transactions. */ diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt index 954db10c..c86ced8a 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt @@ -92,6 +92,7 @@ abstract class AbstractDescriptorWriter(final override val field Type.Double -> JDBCType.DOUBLE Type.Datetime -> JDBCType.DATE Type.String -> JDBCType.VARCHAR + Type.Text -> JDBCType.CLOB is Type.BooleanVector -> JDBCType.ARRAY is Type.DoubleVector -> JDBCType.ARRAY is Type.FloatVector -> JDBCType.ARRAY diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt index d5b4f2ca..5ba455b3 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt @@ -23,6 +23,7 @@ class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, conn for (field in this.prototype.schema()) { when (field.type) { Type.String -> statement.append("\"${field.name}\" varchar(255), ") + Type.Text -> statement.append("\"${field.name}\" text, ") Type.Boolean -> statement.append("\"${field.name}\" boolean, ") Type.Byte -> statement.append("$\"{field.name}\" smallint, ") Type.Short -> statement.append("\"${field.name}\" smallint, ") From 6f7459999419513a833593b987bb91f69c9a8cb8 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Tue, 9 Jul 2024 10:53:51 +0200 Subject: [PATCH 22/71] Adds ScalarDescriptorProvider and refactors the components w.r.t. how columns are named. --- .../descriptor/scalar/BooleanDescriptor.kt | 5 +- .../descriptor/scalar/DoubleDescriptor.kt | 5 +- .../descriptor/scalar/FloatDescriptor.kt | 5 +- .../model/descriptor/scalar/IntDescriptor.kt | 5 +- .../model/descriptor/scalar/LongDescriptor.kt | 5 +- .../descriptor/scalar/ScalarDescriptor.kt | 6 +- .../descriptor/scalar/StringDescriptor.kt | 5 +- .../vector/BooleanVectorDescriptor.kt | 3 +- .../vector/DoubleVectorDescriptor.kt | 3 +- .../vector/FloatVectorDescriptor.kt | 3 +- .../descriptor/vector/IntVectorDescriptor.kt | 3 +- .../descriptor/vector/LongVectorDescriptor.kt | 3 +- .../descriptor/vector/VectorDescriptor.kt | 6 +- .../vitrivr/engine/core/model/types/Value.kt | 26 +++-- .../engine/plugin/cottontaildb/Common.kt | 4 +- .../scalar/ScalarDescriptorInitializer.kt | 11 +- .../scalar/ScalarDescriptorReader.kt | 18 +-- .../scalar/ScalarDescriptorWriter.kt | 19 ++-- .../string/StringDescriptorInitializer.kt | 6 +- .../string/StringDescriptorReader.kt | 13 ++- .../string/StringDescriptorWriter.kt | 12 +- .../vector/VectorDescriptorInitializer.kt | 10 +- .../vector/VectorDescriptorReader.kt | 20 ++-- .../vector/VectorDescriptorWriter.kt | 12 +- .../engine/database/pgvector/Constants.kt | 3 - .../pgvector/PgVectorConnectionProvider.kt | 10 ++ .../AbstractDescriptorInitializer.kt | 53 --------- ...tializer.kt => PgDescriptorInitializer.kt} | 53 +++++++-- ...criptorWriter.kt => PgDescriptorWriter.kt} | 99 ++++++++++++++++- .../pgvector/descriptor/model/PgBitVector.kt | 9 +- .../pgvector/descriptor/model/PgVector.kt | 27 ++++- .../scalar/ScalarDescriptorProvider.kt | 22 ++++ .../scalar/ScalarDescriptorReader.kt | 54 +++++++++ .../struct/StructDescriptorProvider.kt | 6 +- .../struct/StructDescriptorReader.kt | 42 +++---- .../struct/StructDescriptorWriter.kt | 104 ------------------ .../vector/VectorDescriptorInitializer.kt | 33 ------ .../vector/VectorDescriptorProvider.kt | 6 +- .../vector/VectorDescriptorReader.kt | 31 ++++-- .../vector/VectorDescriptorWriter.kt | 93 ---------------- .../retrievable/RetrievableInitializer.kt | 7 +- .../pgvector/retrievable/RetrievableReader.kt | 18 +-- 42 files changed, 436 insertions(+), 442 deletions(-) delete mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt rename vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/{struct/StructDescriptorInitializer.kt => PgDescriptorInitializer.kt} (56%) rename vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/{AbstractDescriptorWriter.kt => PgDescriptorWriter.kt} (57%) create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorProvider.kt create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt delete mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt delete mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt delete mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt index 1e1f71d9..f013e0e5 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt @@ -1,7 +1,8 @@ package org.vitrivr.engine.core.model.descriptor.scalar -import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,7 +22,7 @@ data class BooleanDescriptor( override val field: Schema.Field<*, BooleanDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(Attribute("value", Type.Boolean)) + private val SCHEMA = listOf(Attribute(VALUE_ATTRIBUTE_NAME, Type.Boolean)) } /** diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt index 7196d5ee..856ebf24 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt @@ -1,7 +1,8 @@ package org.vitrivr.engine.core.model.descriptor.scalar -import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,7 +22,7 @@ data class DoubleDescriptor( override val field: Schema.Field<*, DoubleDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(Attribute("value", Type.Double)) + private val SCHEMA = listOf(Attribute(VALUE_ATTRIBUTE_NAME, Type.Double)) } /** diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt index 68159726..2272344a 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt @@ -1,7 +1,8 @@ package org.vitrivr.engine.core.model.descriptor.scalar -import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,7 +22,7 @@ data class FloatDescriptor( override val field: Schema.Field<*, FloatDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(Attribute("value", Type.Float)) + private val SCHEMA = listOf(Attribute(VALUE_ATTRIBUTE_NAME, Type.Float)) } /** diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt index 9184a2eb..34c29f0f 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt @@ -1,7 +1,8 @@ package org.vitrivr.engine.core.model.descriptor.scalar -import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,7 +22,7 @@ data class IntDescriptor( override val field: Schema.Field<*, IntDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(Attribute("value", Type.Int)) + private val SCHEMA = listOf(Attribute(VALUE_ATTRIBUTE_NAME, Type.Int)) } /** diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt index f8dff29d..42bc3739 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt @@ -1,7 +1,8 @@ package org.vitrivr.engine.core.model.descriptor.scalar -import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,7 +22,7 @@ data class LongDescriptor( override val field: Schema.Field<*, LongDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(Attribute("value", Type.Long)) + private val SCHEMA = listOf(Attribute(VALUE_ATTRIBUTE_NAME, Type.Long)) } /** diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/ScalarDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/ScalarDescriptor.kt index 23728002..452a369f 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/ScalarDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/ScalarDescriptor.kt @@ -10,7 +10,11 @@ import org.vitrivr.engine.core.model.types.Value * @author Ralph Gasser * @version 1.1.0 */ -sealed interface ScalarDescriptor> : Descriptor { +sealed interface ScalarDescriptor> : Descriptor { + + companion object { + const val VALUE_ATTRIBUTE_NAME = "value" + } /** The [Value] held by this [ScalarDescriptor]. */ val value: T diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt index 0f050b7e..e91b3635 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt @@ -1,7 +1,8 @@ package org.vitrivr.engine.core.model.descriptor.scalar -import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.DescriptorId +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -21,7 +22,7 @@ data class StringDescriptor( override val field: Schema.Field<*, StringDescriptor>? = null ) : ScalarDescriptor { companion object { - private val SCHEMA = listOf(Attribute("value", Type.String)) + private val SCHEMA = listOf(Attribute(VALUE_ATTRIBUTE_NAME, Type.String)) } /** diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt index be823b66..2ffd580a 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt @@ -1,6 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.vector import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -26,5 +27,5 @@ data class BooleanVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute("vector", Type.BooleanVector(this.dimensionality))) + override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.BooleanVector(this.dimensionality))) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt index 0c3902a1..1aa1ce38 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt @@ -1,6 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.vector import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -26,5 +27,5 @@ data class DoubleVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute("vector", Type.DoubleVector(this.dimensionality))) + override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.DoubleVector(this.dimensionality))) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt index 06f6481f..2173b30e 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt @@ -1,6 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.vector import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -27,5 +28,5 @@ data class FloatVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute("vector", Type.FloatVector(this.dimensionality))) + override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.FloatVector(this.dimensionality))) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt index fde0e8d3..d7876a37 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt @@ -1,6 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.vector import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -26,5 +27,5 @@ data class IntVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute("vector", Type.Int)) + override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.Int)) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt index 2de95bcc..88f7742b 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt @@ -1,6 +1,7 @@ package org.vitrivr.engine.core.model.descriptor.vector import org.vitrivr.engine.core.model.descriptor.Attribute +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Type @@ -26,5 +27,5 @@ data class LongVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute("vector", Type.Long)) + override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.Long)) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/VectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/VectorDescriptor.kt index a0594ace..bb02b8a4 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/VectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/VectorDescriptor.kt @@ -13,6 +13,10 @@ import org.vitrivr.engine.core.model.types.Value * @version 1.1.0 */ sealed interface VectorDescriptor> : Descriptor { + companion object { + const val VECTOR_ATTRIBUTE_NAME = "vector" + } + /** The size of this [VectorDescriptor]. */ val dimensionality: Int get() = this.vector.size @@ -25,5 +29,5 @@ sealed interface VectorDescriptor> : Descriptor { * * @return A [Map] of this [ScalarDescriptor]'s fields (without the IDs). */ - override fun values(): Map = mapOf("vector" to this.vector) + override fun values(): Map = mapOf(VECTOR_ATTRIBUTE_NAME to this.vector) } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt index 3bd60409..b84c8fdf 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt @@ -1,6 +1,6 @@ package org.vitrivr.engine.core.model.types -import java.util.Date +import java.util.* /** * A [Value] in vitrivr-engine maps primitive data types. @@ -30,32 +30,38 @@ sealed interface Value { /** The actual, primitive value. */ val value: T + + sealed interface ScalarValue: Value + + @JvmInline + value class String(override val value: kotlin.String) : ScalarValue + @JvmInline - value class String(override val value: kotlin.String) : Value + value class Text(override val value: kotlin.String) : ScalarValue @JvmInline - value class Boolean(override val value: kotlin.Boolean) : Value + value class Boolean(override val value: kotlin.Boolean) : ScalarValue @JvmInline - value class Byte(override val value: kotlin.Byte) : Value + value class Byte(override val value: kotlin.Byte) : ScalarValue @JvmInline - value class Short(override val value: kotlin.Short) : Value + value class Short(override val value: kotlin.Short) : ScalarValue @JvmInline - value class Int(override val value: kotlin.Int) : Value + value class Int(override val value: kotlin.Int) : ScalarValue @JvmInline - value class Long(override val value: kotlin.Long) : Value + value class Long(override val value: kotlin.Long) : ScalarValue @JvmInline - value class Float(override val value: kotlin.Float) : Value + value class Float(override val value: kotlin.Float) : ScalarValue @JvmInline - value class Double(override val value: kotlin.Double) : Value + value class Double(override val value: kotlin.Double) : ScalarValue @JvmInline - value class DateTime(override val value: Date) : Value + value class DateTime(override val value: Date) : ScalarValue /** * A [Vector] in vitrivr-engine maps primitive data types. diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt index 6de6c88f..310ffae0 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/Common.kt @@ -39,9 +39,6 @@ const val DESCRIPTOR_ENTITY_PREFIX = "descriptor" /** The column name of a descriptor ID. */ const val DESCRIPTOR_ID_COLUMN_NAME = "descriptorId" -/** The column name of a descriptor ID. */ -const val DESCRIPTOR_COLUMN_NAME = "descriptor" - /** The column name used to describe a distance.*/ const val DISTANCE_COLUMN_NAME = "distance" @@ -128,6 +125,7 @@ internal fun Value<*>.toCottontailValue(): PublicValue = when (this) { is Value.Long -> LongValue(this.value) is Value.Short -> ShortValue(this.value) is Value.String -> StringValue(this.value) + is Value.Text -> StringValue(this.value) is Value.DateTime -> DateValue(this.value) is Value.BooleanVector -> BooleanVectorValue(this.value) is Value.DoubleVector -> DoubleVectorValue(this.value) diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorInitializer.kt index b0591df1..718a3fc3 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorInitializer.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorInitializer.kt @@ -1,23 +1,20 @@ package org.vitrivr.engine.plugin.cottontaildb.descriptors.scalar -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging import io.grpc.StatusRuntimeException import org.vitrivr.cottontail.client.language.ddl.CreateEntity import org.vitrivr.cottontail.core.database.Name import org.vitrivr.cottontail.core.types.Types import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.* import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorInitializer -private val logger: KLogger = KotlinLogging.logger {} - /** * A [AbstractDescriptorInitializer] implementation for [ScalarDescriptor]s. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.0.1 */ class ScalarDescriptorInitializer>(field: Schema.Field<*, T>, connection: CottontailConnection) : AbstractDescriptorInitializer(field, connection) { /** @@ -31,13 +28,13 @@ class ScalarDescriptorInitializer>(field: Schema.Field<* val create = CreateEntity(this.entityName) .column(Name.ColumnName.create(DESCRIPTOR_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = true, autoIncrement = false) .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) - .column(Name.ColumnName.create(DESCRIPTOR_COLUMN_NAME), type, nullable = false, primaryKey = false, autoIncrement = false) + .column(Name.ColumnName.create(VALUE_ATTRIBUTE_NAME), type, nullable = false, primaryKey = false, autoIncrement = false) try { /* Try to create entity. */ this.connection.client.create(create) } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } } } } \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorReader.kt index 70e952e2..26497147 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorReader.kt @@ -5,6 +5,8 @@ import org.vitrivr.cottontail.client.language.basics.expression.Literal import org.vitrivr.cottontail.client.language.basics.predicate.Compare import org.vitrivr.cottontail.core.tuple.Tuple import org.vitrivr.engine.core.model.descriptor.scalar.* +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query import org.vitrivr.engine.core.model.query.bool.SimpleBooleanQuery @@ -36,8 +38,8 @@ class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connec val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) .select(RETRIEVABLE_ID_COLUMN_NAME) .select(DESCRIPTOR_ID_COLUMN_NAME) - .select(DESCRIPTOR_COLUMN_NAME) - .where(Compare(Column(this.entityName.column(DESCRIPTOR_COLUMN_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) + .select(VALUE_ATTRIBUTE_NAME) + .where(Compare(Column(this.entityName.column(VALUE_ATTRIBUTE_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) /* Execute query. */ return this.connection.client.query(cottontailQuery).asSequence().map { @@ -64,12 +66,12 @@ class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connec val retrievableId = tuple.asUuidValue(RETRIEVABLE_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'.") val descriptorId = tuple.asUuidValue(DESCRIPTOR_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'.") return when (this.prototype) { - is BooleanDescriptor -> BooleanDescriptor(retrievableId, descriptorId, tuple.asBoolean(DESCRIPTOR_COLUMN_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) - is DoubleDescriptor -> DoubleDescriptor(retrievableId, descriptorId, tuple.asDouble(DESCRIPTOR_COLUMN_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) - is FloatDescriptor -> FloatDescriptor(retrievableId, descriptorId, tuple.asFloat(DESCRIPTOR_COLUMN_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) - is IntDescriptor -> IntDescriptor(retrievableId, descriptorId, tuple.asInt(DESCRIPTOR_COLUMN_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) - is LongDescriptor -> LongDescriptor(retrievableId, descriptorId, tuple.asLong(DESCRIPTOR_COLUMN_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) - is StringDescriptor -> StringDescriptor(retrievableId, descriptorId, tuple.asString(DESCRIPTOR_COLUMN_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) + is BooleanDescriptor -> BooleanDescriptor(retrievableId, descriptorId, tuple.asBoolean(VALUE_ATTRIBUTE_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) + is DoubleDescriptor -> DoubleDescriptor(retrievableId, descriptorId, tuple.asDouble(VALUE_ATTRIBUTE_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) + is FloatDescriptor -> FloatDescriptor(retrievableId, descriptorId, tuple.asFloat(VALUE_ATTRIBUTE_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) + is IntDescriptor -> IntDescriptor(retrievableId, descriptorId, tuple.asInt(VALUE_ATTRIBUTE_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) + is LongDescriptor -> LongDescriptor(retrievableId, descriptorId, tuple.asLong(VALUE_ATTRIBUTE_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) + is StringDescriptor -> StringDescriptor(retrievableId, descriptorId, tuple.asString(VALUE_ATTRIBUTE_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) } } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorWriter.kt index 261d6c83..63e83778 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorWriter.kt @@ -1,7 +1,5 @@ package org.vitrivr.engine.plugin.cottontaildb.descriptors.scalar -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging import io.grpc.StatusRuntimeException import org.vitrivr.cottontail.client.language.basics.expression.Column import org.vitrivr.cottontail.client.language.basics.expression.Literal @@ -11,17 +9,16 @@ import org.vitrivr.cottontail.client.language.dml.Insert import org.vitrivr.cottontail.client.language.dml.Update import org.vitrivr.cottontail.core.values.UuidValue import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.* import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorWriter -private val logger: KLogger = KotlinLogging.logger {} - /** * An [AbstractDescriptorWriter] for [ScalarDescriptor]s. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.0.1 */ class ScalarDescriptorWriter(field: Schema.Field<*, ScalarDescriptor<*>>, connection: CottontailConnection) : AbstractDescriptorWriter>(field, connection) { @@ -35,14 +32,14 @@ class ScalarDescriptorWriter(field: Schema.Field<*, ScalarDescriptor<*>>, connec val insert = Insert(this.entityName).values( DESCRIPTOR_ID_COLUMN_NAME to UuidValue(item.id), RETRIEVABLE_ID_COLUMN_NAME to UuidValue(item.retrievableId ?: throw IllegalArgumentException("A scalar descriptor must be associated with a retrievable ID.")), - DESCRIPTOR_COLUMN_NAME to item.toCottontailValue() + VALUE_ATTRIBUTE_NAME to item.toCottontailValue() ) return try { this.connection.client.insert(insert).use { it.hasNext() } } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to persist descriptor ${item.id} due to exception." } + LOGGER.error(e) { "Failed to persist descriptor ${item.id} due to exception." } false } } @@ -56,7 +53,7 @@ class ScalarDescriptorWriter(field: Schema.Field<*, ScalarDescriptor<*>>, connec override fun addAll(items: Iterable>): Boolean { /* Prepare insert query. */ var size = 0 - val insert = BatchInsert(this.entityName).columns(DESCRIPTOR_ID_COLUMN_NAME, RETRIEVABLE_ID_COLUMN_NAME, DESCRIPTOR_COLUMN_NAME) + val insert = BatchInsert(this.entityName).columns(DESCRIPTOR_ID_COLUMN_NAME, RETRIEVABLE_ID_COLUMN_NAME, VALUE_ATTRIBUTE_NAME) for (item in items) { size += 1 insert.values(UuidValue(item.id.toString()), UuidValue(item.retrievableId ?: throw IllegalArgumentException("A scalar descriptor must be associated with a retrievable ID.")), item.toCottontailValue()) @@ -68,7 +65,7 @@ class ScalarDescriptorWriter(field: Schema.Field<*, ScalarDescriptor<*>>, connec it.hasNext() } } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to persist $size scalar descriptors due to exception." } + LOGGER.error(e) { "Failed to persist $size scalar descriptors due to exception." } false } } @@ -86,14 +83,14 @@ class ScalarDescriptorWriter(field: Schema.Field<*, ScalarDescriptor<*>>, connec Compare.Operator.EQUAL, Literal(UuidValue(item.id)) ) - ).values(DESCRIPTOR_COLUMN_NAME to item.toCottontailValue()) + ).values(VALUE_ATTRIBUTE_NAME to item.toCottontailValue()) /* Delete values. */ return try { this.connection.client.update(update) true } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to update descriptor due to exception." } + LOGGER.error(e) { "Failed to update descriptor due to exception." } false } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorInitializer.kt index e3005032..52350e75 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorInitializer.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorInitializer.kt @@ -8,10 +8,10 @@ import org.vitrivr.cottontail.client.language.ddl.CreateIndex import org.vitrivr.cottontail.core.database.Name import org.vitrivr.cottontail.core.types.Types import org.vitrivr.cottontail.grpc.CottontailGrpc +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.descriptor.scalar.StringDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection -import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_COLUMN_NAME import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ID_COLUMN_NAME import org.vitrivr.engine.plugin.cottontaildb.RETRIEVABLE_ID_COLUMN_NAME import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorInitializer @@ -33,7 +33,7 @@ class StringDescriptorInitializer(field: Schema.Field<*, StringDescriptor>, conn val create = CreateEntity(this.entityName) .column(Name.ColumnName.create(DESCRIPTOR_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = true, autoIncrement = false) .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) - .column(Name.ColumnName.create(DESCRIPTOR_COLUMN_NAME), Types.String, nullable = false, primaryKey = false, autoIncrement = false) + .column(Name.ColumnName.create(VALUE_ATTRIBUTE_NAME), Types.String, nullable = false, primaryKey = false, autoIncrement = false) try { /* Try to create entity. */ @@ -42,7 +42,7 @@ class StringDescriptorInitializer(field: Schema.Field<*, StringDescriptor>, conn /* Create entity if necessary. */ if (this.field.parameters.containsKey("index")) { val createIndex = CreateIndex(this.entityName, CottontailGrpc.IndexType.LUCENE) - .column(this.entityName.column(DESCRIPTOR_COLUMN_NAME)) + .column(this.entityName.column(VALUE_ATTRIBUTE_NAME)) this.connection.client.create(createIndex) } } catch (e: StatusRuntimeException) { diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorReader.kt index 365c19ec..f9090635 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorReader.kt @@ -4,6 +4,7 @@ import org.vitrivr.cottontail.client.language.basics.expression.Column import org.vitrivr.cottontail.client.language.basics.expression.Literal import org.vitrivr.cottontail.client.language.basics.predicate.Compare import org.vitrivr.cottontail.core.tuple.Tuple +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.descriptor.scalar.StringDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query @@ -36,7 +37,7 @@ class StringDescriptorReader(field: Schema.Field<*, StringDescriptor>, connectio is SimpleFulltextQuery -> { cottontailQuery .select("*") - .fulltext(DESCRIPTOR_COLUMN_NAME, query.value.value, "score") + .fulltext(VALUE_ATTRIBUTE_NAME, query.value.value, "score") if (query.limit < Long.MAX_VALUE) { cottontailQuery.limit(query.limit) } @@ -44,7 +45,7 @@ class StringDescriptorReader(field: Schema.Field<*, StringDescriptor>, connectio is SimpleBooleanQuery<*> -> { require(query.value is Value.String) { "StringDescriptorReader can only perform comparisons to string values." } - cottontailQuery.where(Compare(Column(this.entityName.column(DESCRIPTOR_COLUMN_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) + cottontailQuery.where(Compare(Column(this.entityName.column(VALUE_ATTRIBUTE_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) } else -> throw IllegalArgumentException("Query of typ ${query::class} is not supported by StringDescriptorReader.") } @@ -69,9 +70,9 @@ class StringDescriptorReader(field: Schema.Field<*, StringDescriptor>, connectio when (query) { is SimpleFulltextQuery -> { cottontailQuery - .select(DESCRIPTOR_COLUMN_NAME) + .select(VALUE_ATTRIBUTE_NAME) .select(DESCRIPTOR_ID_COLUMN_NAME) - .fulltext(DESCRIPTOR_COLUMN_NAME, query.value.value, "score") + .fulltext(VALUE_ATTRIBUTE_NAME, query.value.value, "score") if (query.limit < Long.MAX_VALUE) { cottontailQuery.limit(query.limit) @@ -80,7 +81,7 @@ class StringDescriptorReader(field: Schema.Field<*, StringDescriptor>, connectio is SimpleBooleanQuery<*> -> { require(query.value is Value.String) { "StringDescriptorReader can only perform comparisons to string values." } - cottontailQuery.where(Compare(Column(this.entityName.column(DESCRIPTOR_COLUMN_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) + cottontailQuery.where(Compare(Column(this.entityName.column(VALUE_ATTRIBUTE_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) } else -> throw IllegalArgumentException("Query of typ ${query::class} is not supported by StringDescriptorReader.") @@ -112,7 +113,7 @@ class StringDescriptorReader(field: Schema.Field<*, StringDescriptor>, connectio override fun tupleToDescriptor(tuple: Tuple): StringDescriptor { val retrievableId = tuple.asUuidValue(RETRIEVABLE_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'.") val descriptorId = tuple.asUuidValue(DESCRIPTOR_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'.") - val value = tuple.asString(DESCRIPTOR_COLUMN_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.") + val value = tuple.asString(VALUE_ATTRIBUTE_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VALUE_ATTRIBUTE_NAME'.") return StringDescriptor(descriptorId, retrievableId, value) } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorWriter.kt index ad3d7c3f..042f6287 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorWriter.kt @@ -10,11 +10,15 @@ import org.vitrivr.cottontail.client.language.dml.BatchInsert import org.vitrivr.cottontail.client.language.dml.Insert import org.vitrivr.cottontail.client.language.dml.Update import org.vitrivr.cottontail.core.values.UuidValue +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.descriptor.scalar.StringDescriptor import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.* +import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection +import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ID_COLUMN_NAME +import org.vitrivr.engine.plugin.cottontaildb.RETRIEVABLE_ID_COLUMN_NAME import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorWriter import org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescriptorWriter +import org.vitrivr.engine.plugin.cottontaildb.toCottontailValue private val logger: KLogger = KotlinLogging.logger {} @@ -36,7 +40,7 @@ class StringDescriptorWriter(field: Schema.Field<*, StringDescriptor>, connectio val insert = Insert(this.entityName).values( DESCRIPTOR_ID_COLUMN_NAME to UuidValue(item.id), RETRIEVABLE_ID_COLUMN_NAME to UuidValue(item.retrievableId ?: throw IllegalArgumentException("A string descriptor must be associated with a retrievable ID.")), - DESCRIPTOR_COLUMN_NAME to item.toCottontailValue() + VALUE_ATTRIBUTE_NAME to item.toCottontailValue() ) return try { this.connection.client.insert(insert).use { @@ -57,7 +61,7 @@ class StringDescriptorWriter(field: Schema.Field<*, StringDescriptor>, connectio override fun addAll(items: Iterable): Boolean { /* Prepare insert query. */ var size = 0 - val insert = BatchInsert(this.entityName).columns(DESCRIPTOR_ID_COLUMN_NAME, RETRIEVABLE_ID_COLUMN_NAME, DESCRIPTOR_COLUMN_NAME) + val insert = BatchInsert(this.entityName).columns(DESCRIPTOR_ID_COLUMN_NAME, RETRIEVABLE_ID_COLUMN_NAME, VALUE_ATTRIBUTE_NAME) for (item in items) { size += 1 insert.values(UuidValue(item.id.toString()), UuidValue(item.retrievableId ?: throw IllegalArgumentException("A string descriptor must be associated with a retrievable ID.")), item.toCottontailValue()) @@ -87,7 +91,7 @@ class StringDescriptorWriter(field: Schema.Field<*, StringDescriptor>, connectio Compare.Operator.EQUAL, Literal(UuidValue(item.id)) ) - ).values(DESCRIPTOR_COLUMN_NAME to item.toCottontailValue()) + ).values(VALUE_ATTRIBUTE_NAME to item.toCottontailValue()) /* Updates values. */ return try { diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorInitializer.kt index 8f1e1d60..2c042560 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorInitializer.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorInitializer.kt @@ -1,19 +1,15 @@ package org.vitrivr.engine.plugin.cottontaildb.descriptors.vector -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging import io.grpc.StatusRuntimeException import org.vitrivr.cottontail.client.language.ddl.CreateEntity import org.vitrivr.cottontail.core.database.Name import org.vitrivr.cottontail.core.types.Types import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.* import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorInitializer -private val logger: KLogger = KotlinLogging.logger {} - - /** * A [AbstractDescriptorInitializer] implementation for [VectorDescriptor]s. * @@ -29,12 +25,12 @@ internal class VectorDescriptorInitializer(field: Schema.Field<*, VectorDescript val create = CreateEntity(this.entityName) .column(Name.ColumnName.create(DESCRIPTOR_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = true, autoIncrement = false) .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) - .column(Name.ColumnName.create(DESCRIPTOR_COLUMN_NAME), type, nullable = false, primaryKey = false, autoIncrement = false) + .column(Name.ColumnName.create(VECTOR_ATTRIBUTE_NAME), type, nullable = false, primaryKey = false, autoIncrement = false) try { this.connection.client.create(create).close() } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } } } } \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt index 94279c84..a2b6c171 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt @@ -4,13 +4,13 @@ import org.vitrivr.cottontail.client.language.basics.Direction import org.vitrivr.cottontail.client.language.basics.Distances import org.vitrivr.cottontail.core.tuple.Tuple import org.vitrivr.engine.core.model.descriptor.vector.* +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query import org.vitrivr.engine.core.model.query.proximity.ProximityQuery import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.core.model.retrievable.attributes.DistanceAttribute import org.vitrivr.engine.core.model.types.Value -import org.vitrivr.engine.core.model.types.toValue import org.vitrivr.engine.plugin.cottontaildb.* import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorReader @@ -35,7 +35,7 @@ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*> val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) .select(RETRIEVABLE_ID_COLUMN_NAME) .distance( - DESCRIPTOR_COLUMN_NAME, + VECTOR_ATTRIBUTE_NAME, query.value.toCottontailValue(), Distances.valueOf(query.distance.toString()), DISTANCE_COLUMN_NAME @@ -44,7 +44,7 @@ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*> .limit(query.k) if (query.fetchVector) { - cottontailQuery.select(DESCRIPTOR_COLUMN_NAME) + cottontailQuery.select(VECTOR_ATTRIBUTE_NAME) cottontailQuery.select(DESCRIPTOR_ID_COLUMN_NAME) } @@ -68,11 +68,11 @@ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*> when (query) { is ProximityQuery<*> -> { val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) - .select(DESCRIPTOR_COLUMN_NAME) + .select(VECTOR_ATTRIBUTE_NAME) .select(DESCRIPTOR_ID_COLUMN_NAME) .select(RETRIEVABLE_ID_COLUMN_NAME) .distance( - DESCRIPTOR_COLUMN_NAME, + VECTOR_ATTRIBUTE_NAME, query.value.toCottontailValue(), Distances.valueOf(query.distance.toString()), DISTANCE_COLUMN_NAME @@ -122,31 +122,31 @@ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*> is BooleanVectorDescriptor -> BooleanVectorDescriptor( descriptorId, retrievableId, - Value.BooleanVector(tuple.asBooleanVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) + Value.BooleanVector(tuple.asBooleanVector(VECTOR_ATTRIBUTE_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) ) is FloatVectorDescriptor -> FloatVectorDescriptor( descriptorId, retrievableId, - Value.FloatVector(tuple.asFloatVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) + Value.FloatVector(tuple.asFloatVector(VECTOR_ATTRIBUTE_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) ) is DoubleVectorDescriptor -> DoubleVectorDescriptor( descriptorId, retrievableId, - Value.DoubleVector(tuple.asDoubleVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) + Value.DoubleVector(tuple.asDoubleVector(VECTOR_ATTRIBUTE_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) ) is IntVectorDescriptor -> IntVectorDescriptor( descriptorId, retrievableId, - Value.IntVector(tuple.asIntVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) + Value.IntVector(tuple.asIntVector(VECTOR_ATTRIBUTE_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) ) is LongVectorDescriptor -> LongVectorDescriptor( descriptorId, retrievableId, - Value.LongVector(tuple.asLongVector(DESCRIPTOR_COLUMN_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) + Value.LongVector(tuple.asLongVector(VECTOR_ATTRIBUTE_NAME) ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) ) } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorWriter.kt index 9512f3a2..c225e07c 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorWriter.kt @@ -11,9 +11,13 @@ import org.vitrivr.cottontail.client.language.dml.Insert import org.vitrivr.cottontail.client.language.dml.Update import org.vitrivr.cottontail.core.values.UuidValue import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.* +import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection +import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ID_COLUMN_NAME +import org.vitrivr.engine.plugin.cottontaildb.RETRIEVABLE_ID_COLUMN_NAME import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorWriter +import org.vitrivr.engine.plugin.cottontaildb.toCottontailValue private val logger: KLogger = KotlinLogging.logger {} @@ -35,7 +39,7 @@ class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connec val insert = Insert(this.entityName).values( DESCRIPTOR_ID_COLUMN_NAME to UuidValue(item.id.toString()), RETRIEVABLE_ID_COLUMN_NAME to UuidValue(item.retrievableId ?: throw IllegalArgumentException("A vector descriptor must be associated with a retrievable ID.")), - DESCRIPTOR_COLUMN_NAME to item.toCottontailValue() + VECTOR_ATTRIBUTE_NAME to item.toCottontailValue() ) return try { this.connection.client.insert(insert).use { @@ -56,7 +60,7 @@ class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connec override fun addAll(items: Iterable>): Boolean { /* Prepare insert query. */ var size = 0 - val insert = BatchInsert(this.entityName).columns(DESCRIPTOR_ID_COLUMN_NAME, RETRIEVABLE_ID_COLUMN_NAME, DESCRIPTOR_COLUMN_NAME) + val insert = BatchInsert(this.entityName).columns(DESCRIPTOR_ID_COLUMN_NAME, RETRIEVABLE_ID_COLUMN_NAME, VECTOR_ATTRIBUTE_NAME) for (item in items) { size += 1 insert.values(UuidValue(item.id), UuidValue(item.retrievableId ?: throw IllegalArgumentException("A vector descriptor must be associated with a retrievable ID.")), item.toCottontailValue()) @@ -85,7 +89,7 @@ class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connec Compare.Operator.EQUAL, Literal(UuidValue(item.id)) ) - ).values(DESCRIPTOR_COLUMN_NAME to item.toCottontailValue()) + ).values(VECTOR_ATTRIBUTE_NAME to item.toCottontailValue()) /* Delete values. */ return try { diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Constants.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Constants.kt index 3a05ef6d..735a41cd 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Constants.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Constants.kt @@ -27,8 +27,5 @@ const val DESCRIPTOR_ENTITY_PREFIX = "descriptor" /** The column name of a descriptor ID. */ const val DESCRIPTOR_ID_COLUMN_NAME = "descriptorId" -/** The column name of a descriptor ID. */ -const val DESCRIPTOR_COLUMN_NAME = "descriptor" - /** The column name used to describe a distance.*/ const val DISTANCE_COLUMN_NAME = "distance" \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt index d22d3d47..78fd06e0 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt @@ -4,6 +4,7 @@ import org.vitrivr.engine.core.database.AbstractConnectionProvider import org.vitrivr.engine.core.database.Connection import org.vitrivr.engine.core.database.ConnectionProvider import org.vitrivr.engine.core.database.descriptor.DescriptorProvider +import org.vitrivr.engine.core.model.descriptor.scalar.* import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.metadata.MediaDimensionsDescriptor @@ -13,6 +14,7 @@ import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.FileSourc import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.VideoSourceMetadataDescriptor import org.vitrivr.engine.core.model.descriptor.vector.* import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.database.pgvector.descriptor.scalar.ScalarDescriptorProvider import org.vitrivr.engine.database.pgvector.descriptor.struct.StructDescriptorProvider import org.vitrivr.engine.database.pgvector.descriptor.vector.VectorDescriptorProvider import java.sql.DriverManager @@ -60,6 +62,14 @@ class PgVectorConnectionProvider: AbstractConnectionProvider() { * This method is called during initialization of the [PgVectorConnectionProvider] and can be used to register [DescriptorProvider]s. */ override fun initialize() { + /* Scalar descriptors. */ + this.register(BooleanDescriptor::class, ScalarDescriptorProvider) + this.register(IntDescriptor::class, ScalarDescriptorProvider) + this.register(LongDescriptor::class, ScalarDescriptorProvider) + this.register(FloatDescriptor::class, ScalarDescriptorProvider) + this.register(DoubleDescriptor::class, ScalarDescriptorProvider) + this.register(StringDescriptor::class, ScalarDescriptorProvider) + /* Vector descriptors. */ this.register(BooleanVectorDescriptor::class, VectorDescriptorProvider) this.register(IntVectorDescriptor::class, VectorDescriptorProvider) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt deleted file mode 100644 index 78f39892..00000000 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorInitializer.kt +++ /dev/null @@ -1,53 +0,0 @@ -package org.vitrivr.engine.database.pgvector.descriptor - -import org.vitrivr.engine.core.database.descriptor.DescriptorInitializer -import org.vitrivr.engine.core.model.descriptor.Descriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ENTITY_PREFIX -import org.vitrivr.engine.database.pgvector.LOGGER -import java.sql.Connection -import java.sql.SQLException - -/** - * An abstract implementation of a [DescriptorInitializer] for PostgreSQL with pgVector. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -abstract class AbstractDescriptorInitializer(final override val field: Schema.Field<*, D>, protected val connection: Connection): DescriptorInitializer { - - /** The name of the table backing this [AbstractDescriptorInitializer]. */ - protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" - - /** The [Descriptor] prototype for this [AbstractDescriptorWriter]. */ - protected val prototype = this.field.analyser.prototype(this.field) - - /** - * Checks if the schema for this [AbstractDescriptorInitializer] has been properly initialized. - * - * @return True if entity has been initialized, false otherwise. - */ - override fun isInitialized(): Boolean { - try { - this.connection.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $tableName").use { - it.execute() - } - } catch (e: SQLException) { - return false - } - return true - } - - /** - * Truncates the table backing this [AbstractDescriptorInitializer]. - */ - override fun truncate() { - try { - this.connection.prepareStatement(/* sql = postgres */ "TRUNCATE $tableName").use { - it.execute() - } - } catch (e: SQLException) { - LOGGER.error(e) { "Failed to truncate entities due to exception." } - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt similarity index 56% rename from vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt rename to vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index 5ba455b3..d9b326b5 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -1,19 +1,29 @@ -package org.vitrivr.engine.database.pgvector.descriptor.struct +package org.vitrivr.engine.database.pgvector.descriptor -import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor +import org.vitrivr.engine.core.database.descriptor.DescriptorInitializer +import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.types.Type import org.vitrivr.engine.database.pgvector.* -import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorInitializer import java.sql.SQLException /** - * A [AbstractDescriptorInitializer] implementation for [StructDescriptor]s. + * An abstract implementation of a [DescriptorInitializer] for PostgreSQL with pgVector. * * @author Ralph Gasser * @version 1.0.0 */ -class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, connection: PgVectorConnection) : AbstractDescriptorInitializer(field, connection.jdbc) { +open class PgDescriptorInitializer(final override val field: Schema.Field<*, D>, protected val connection: PgVectorConnection): DescriptorInitializer { + + /** The name of the table backing this [PgDescriptorInitializer]. */ + protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" + + /** The [Descriptor] prototype for this [PgDescriptorWriter]. */ + protected val prototype = this.field.analyser.prototype(this.field) + + /** + * + */ override fun initialize() { val statement = StringBuilder("CREATE TABLE IF NOT EXISTS $tableName(") statement.append("$DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, ") @@ -25,7 +35,7 @@ class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, conn Type.String -> statement.append("\"${field.name}\" varchar(255), ") Type.Text -> statement.append("\"${field.name}\" text, ") Type.Boolean -> statement.append("\"${field.name}\" boolean, ") - Type.Byte -> statement.append("$\"{field.name}\" smallint, ") + Type.Byte -> statement.append("\"${field.name}\" smallint, ") Type.Short -> statement.append("\"${field.name}\" smallint, ") Type.Int -> statement.append("\"${field.name}\" integer, ") Type.Long -> statement.append("\"${field.name}\" bigint, ") @@ -46,11 +56,40 @@ class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, conn try { /* Create 'retrievable' entity. */ - this.connection.prepareStatement(/* sql = postgres */ statement.toString()).use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ statement.toString()).use { it.execute() } } catch (e: SQLException) { LOGGER.error(e) { "Failed to initialize entity '$tableName' due to exception." } } } + + /** + * Checks if the schema for this [PgDescriptorInitializer] has been properly initialized. + * + * @return True if entity has been initialized, false otherwise. + */ + override fun isInitialized(): Boolean { + try { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $tableName").use { + it.execute() + } + } catch (e: SQLException) { + return false + } + return true + } + + /** + * Truncates the table backing this [PgDescriptorInitializer]. + */ + override fun truncate() { + try { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "TRUNCATE $tableName").use { + it.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to truncate entities due to exception." } + } + } } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt similarity index 57% rename from vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt rename to vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt index c86ced8a..27218854 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt @@ -2,13 +2,13 @@ package org.vitrivr.engine.database.pgvector.descriptor import org.vitrivr.engine.core.database.descriptor.DescriptorWriter import org.vitrivr.engine.core.model.descriptor.Descriptor +import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.types.Type import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.database.pgvector.* import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector -import org.vitrivr.engine.database.pgvector.descriptor.struct.StructDescriptorWriter import java.sql.* /** @@ -17,15 +17,102 @@ import java.sql.* * @author Ralph Gasser * @version 1.0.0 */ -abstract class AbstractDescriptorWriter(final override val field: Schema.Field<*, D>, override val connection: PgVectorConnection): DescriptorWriter { - /** The name of the table backing this [AbstractDescriptorInitializer]. */ +open class PgDescriptorWriter(final override val field: Schema.Field<*, D>, override val connection: PgVectorConnection): DescriptorWriter { + /** The name of the table backing this [PgDescriptorInitializer]. */ protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" - /** The [Descriptor] prototype for this [AbstractDescriptorWriter]. */ + /** The [Descriptor] prototype for this [PgDescriptorWriter]. */ protected val prototype = this.field.analyser.prototype(this.field) /** - * Deletes (writes) a [Descriptor] of type [D] using this [AbstractDescriptorWriter]. + * Adds (writes) a single [StructDescriptor] using this [StructDescriptorWriter]. + * + * @param item The [StructDescriptor] to write. + * @return True on success, false otherwise. + */ + override fun add(item: D): Boolean { + try { + this.prepareInsertStatement().use { stmt -> + stmt.setObject(1, item.id) + stmt.setObject(2, item.retrievableId) + var i = 3 + for (attribute in item.schema()) { + val value = item.values()[attribute.name] + if (value != null) { + stmt.setValue(i++, value) + } else { + stmt.setNull(i++, attribute.type.toSqlType()) + } + } + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to INSERT descriptor ${item.id} into '$tableName' due to SQL error." } + return false + } + } + + /** + * Adds (writes) a batch of [Descriptor] of type [D] using this [StructDescriptorWriter]. + * + * @param items A [Iterable] of [Descriptor]s to write. + * @return True on success, false otherwise. + */ + override fun addAll(items: Iterable): Boolean { + try { + this.prepareInsertStatement().use { stmt -> + for (item in items) { + stmt.setObject(1, item.id) + stmt.setObject(2, item.retrievableId) + var i = 3 + for (attribute in item.schema()) { + val value = item.values()[attribute.name] + if (value != null) { + stmt.setValue(i++, value) + } else { + stmt.setNull(i++, attribute.type.toSqlType()) + } + } + stmt.addBatch() + } + return stmt.executeBatch().all { it == 1 } + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to INSERT descriptors into '$tableName' due to SQL error." } + return false + } + } + + /** + * Updates a specific [Descriptor] of type [D] using this [StructDescriptorWriter]. + * + * @param item A [Descriptor]s to update. + * @return True on success, false otherwise. + */ + override fun update(item: D): Boolean { + try { + this.prepareUpdateStatement().use { stmt -> + stmt.setObject(1, item.retrievableId) + var i = 2 + for (attribute in item.schema()) { + val value = item.values()[attribute.name] + if (value != null) { + stmt.setValue(i++, value) + } else { + stmt.setNull(i++, attribute.type.toSqlType()) + } + } + stmt.setObject(i, item.id) + return stmt.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to UPDATE descriptors in '$tableName' due to SQL error." } + return false + } + } + + /** + * Deletes (writes) a [Descriptor] of type [D] using this [PgDescriptorWriter]. * * @param item A [Descriptor]s to delete. * @return True on success, false otherwise. @@ -43,7 +130,7 @@ abstract class AbstractDescriptorWriter(final override val field } /** - * Deletes (writes) [Descriptor]s of type [D] using this [AbstractDescriptorWriter]. + * Deletes (writes) [Descriptor]s of type [D] using this [PgDescriptorWriter]. * * @param items A [Iterable] of [Descriptor]s to delete. * @return True on success, false otherwise. diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt index 7d869164..11e04df8 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt @@ -3,6 +3,7 @@ package org.vitrivr.engine.database.pgvector.descriptor.model import org.postgresql.util.ByteConverter import org.postgresql.util.PGBinaryObject import org.postgresql.util.PGobject +import org.vitrivr.engine.core.model.types.Value import java.io.Serializable import java.sql.SQLException @@ -134,16 +135,16 @@ class PgBitVector () : PGobject(), PGBinaryObject, Serializable { fun toByteArray(): ByteArray? = this.vec /** - * Returns an array + * Returns a [Value.BooleanVector] representation of this [PgBitVector]. * - * @return an array + * @return [Value.BooleanVector] */ - fun toArray(): BooleanArray? { + fun toBooleanVector(): Value.BooleanVector? { if (this.vec == null) return null val bits = BooleanArray(this.length) for (i in 0 until this.length) { bits[i] = ((this.vec!![i / 8].toInt() shr (7 - (i % 8))) and 1) == 1 } - return bits + return Value.BooleanVector(bits) } } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt index 8d31e0ee..f8b57cb9 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt @@ -109,9 +109,30 @@ class PgVector(private var vec: FloatArray? = null) : PGobject(), PGBinaryObject } /** - * Returns an array + * Returns an [Value.FloatVector] representation of this [PgVector]. * - * @return an array + * @return [Value.FloatVector] */ - fun toArray(): FloatArray? = this.vec + fun toFloatVector(): Value.FloatVector? = this.vec?.let { Value.FloatVector(it) } + + /** + * Returns an [Value.DoubleVector] representation of this [PgVector]. + * + * @return [Value.DoubleVector] + */ + fun toDoubleVector(): Value.DoubleVector? = this.vec?.let { vec -> Value.DoubleVector(DoubleArray(vec.size) { i -> vec[i].toDouble()}) } + + /** + * Returns an [Value.IntVector] representation of this [PgVector]. + * + * @return [Value.IntVector] + */ + fun toIntVector(): Value.IntVector? = this.vec?.let { vec -> Value.IntVector(IntArray(vec.size) { i -> vec[i].toInt()}) } + + /** + * Returns an [Value.LongVector] representation of this [PgVector]. + * + * @return [Value.LongVector] + */ + fun toLongVector(): Value.LongVector? = this.vec?.let { vec -> Value.LongVector(LongArray(vec.size) { i -> vec[i].toLong()}) } } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorProvider.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorProvider.kt new file mode 100644 index 00000000..8fb2ff59 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorProvider.kt @@ -0,0 +1,22 @@ +package org.vitrivr.engine.database.pgvector.descriptor.scalar + +import org.vitrivr.engine.core.database.Connection +import org.vitrivr.engine.core.database.descriptor.DescriptorProvider +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor +import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.database.pgvector.PgVectorConnection +import org.vitrivr.engine.database.pgvector.descriptor.PgDescriptorInitializer +import org.vitrivr.engine.database.pgvector.descriptor.PgDescriptorWriter + +/** + * A [DescriptorProvider] for [StructDescriptor]s. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +object ScalarDescriptorProvider: DescriptorProvider> { + override fun newInitializer(connection: Connection, field: Schema.Field<*, ScalarDescriptor<*>>) = PgDescriptorInitializer(field, connection as PgVectorConnection) + override fun newReader(connection: Connection, field: Schema.Field<*, ScalarDescriptor<*>>) = ScalarDescriptorReader(field, connection as PgVectorConnection) + override fun newWriter(connection: Connection, field: Schema.Field<*, ScalarDescriptor<*>>) = PgDescriptorWriter(field, connection as PgVectorConnection) +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt new file mode 100644 index 00000000..3ddbb003 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt @@ -0,0 +1,54 @@ +package org.vitrivr.engine.database.pgvector.descriptor.scalar + +import org.vitrivr.engine.core.model.descriptor.scalar.* +import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME +import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor +import org.vitrivr.engine.core.model.descriptor.vector.* +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.query.Query +import org.vitrivr.engine.core.model.retrievable.Retrieved +import org.vitrivr.engine.core.model.types.Value +import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.PgVectorConnection +import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorReader +import java.sql.ResultSet +import java.util.* + +/** + * A [DescriptorReader] for [ScalarDescriptor]s. + * + * @author Ralph Gasser + * @version 1.1.0 + */ +class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connection: PgVectorConnection) : AbstractDescriptorReader>(field, connection) { + + /** + * Executes the provided [Query] and returns a [Sequence] of [Retrieved]s that match it. + * + * @param query The [Query] to execute. + * @return [Sequence] of [StructDescriptor]s that match the query. + */ + override fun query(query: Query): Sequence> { + TODO("Not yet implemented") + } + + /** + * Converts the provided [ResultSet] to a [VectorDescriptor]. + * + * @param result The [ResultSet] to convert. + * @return The resulting [VectorDescriptor]. + */ + override fun rowToDescriptor(result: ResultSet): ScalarDescriptor<*> { + val descriptorId = result.getObject(DESCRIPTOR_ID_COLUMN_NAME, UUID::class.java) + val retrievableId = result.getObject(RETRIEVABLE_ID_COLUMN_NAME, UUID::class.java) + return when (this.prototype) { + is BooleanDescriptor -> BooleanDescriptor(descriptorId, retrievableId, Value.Boolean(result.getBoolean(VALUE_ATTRIBUTE_NAME))) + is IntDescriptor -> IntDescriptor(descriptorId, retrievableId, Value.Int(result.getInt(VALUE_ATTRIBUTE_NAME))) + is LongDescriptor -> LongDescriptor(descriptorId, retrievableId, Value.Long(result.getLong(VALUE_ATTRIBUTE_NAME))) + is FloatDescriptor -> FloatDescriptor(descriptorId, retrievableId, Value.Float(result.getFloat(VALUE_ATTRIBUTE_NAME))) + is DoubleDescriptor -> DoubleDescriptor(descriptorId, retrievableId, Value.Double(result.getDouble(VALUE_ATTRIBUTE_NAME))) + is StringDescriptor -> StringDescriptor(descriptorId, retrievableId, Value.String(result.getString(VALUE_ATTRIBUTE_NAME))) + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorProvider.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorProvider.kt index 478580bf..edbccb8c 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorProvider.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorProvider.kt @@ -6,6 +6,8 @@ import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.database.pgvector.PgVectorConnection +import org.vitrivr.engine.database.pgvector.descriptor.PgDescriptorInitializer +import org.vitrivr.engine.database.pgvector.descriptor.PgDescriptorWriter /** * A [DescriptorProvider] for [LabelDescriptor]s. @@ -14,7 +16,7 @@ import org.vitrivr.engine.database.pgvector.PgVectorConnection * @version 1.0.0 */ object StructDescriptorProvider : DescriptorProvider { - override fun newInitializer(connection: Connection, field: Schema.Field<*, StructDescriptor>) = StructDescriptorInitializer(field, connection as PgVectorConnection) + override fun newInitializer(connection: Connection, field: Schema.Field<*, StructDescriptor>) = PgDescriptorInitializer(field, connection as PgVectorConnection) override fun newReader(connection: Connection, field: Schema.Field<*, StructDescriptor>) = StructDescriptorReader(field, connection as PgVectorConnection) - override fun newWriter(connection: Connection, field: Schema.Field<*, StructDescriptor>) = StructDescriptorWriter(field, connection as PgVectorConnection) + override fun newWriter(connection: Connection, field: Schema.Field<*, StructDescriptor>) = PgDescriptorWriter(field, connection as PgVectorConnection) } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt index 45b9c3af..bdfd6405 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt @@ -1,6 +1,6 @@ package org.vitrivr.engine.database.pgvector.descriptor.struct -import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor +import org.vitrivr.engine.core.model.descriptor.AttributeName import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query @@ -8,6 +8,7 @@ import org.vitrivr.engine.core.model.query.bool.SimpleBooleanQuery import org.vitrivr.engine.core.model.query.fulltext.SimpleFulltextQuery import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.core.model.types.Type +import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME import org.vitrivr.engine.database.pgvector.LOGGER import org.vitrivr.engine.database.pgvector.PgVectorConnection @@ -20,7 +21,7 @@ import java.util.* import kotlin.reflect.full.primaryConstructor /** - * An [AbstractDescriptorReader] for [LabelDescriptor]s. + * An [AbstractDescriptorReader] for [StructDescriptor]s. * * @author Ralph Gasser * @version 1.1.0 @@ -62,29 +63,32 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio */ override fun rowToDescriptor(result: ResultSet): StructDescriptor { val constructor = this.field.analyser.descriptorClass.primaryConstructor ?: throw IllegalStateException("Provided type ${this.field.analyser.descriptorClass} does not have a primary constructor.") + val values = TreeMap?>() val parameters: MutableList = mutableListOf( result.getObject(DESCRIPTOR_ID_COLUMN_NAME, UUID::class.java) ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'."), - result.getObject(DESCRIPTOR_ID_COLUMN_NAME, UUID::class.java) ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'."), + result.getObject(RETRIEVABLE_ID_COLUMN_NAME, UUID::class.java) ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'."), + values ) /* Append dynamic parameters of struct. */ for (field in this.prototype.schema()) { - parameters.add( - when(field.type) { - Type.String -> result.getString(field.name) - Type.Boolean -> result.getBoolean(field.name) - Type.Byte -> result.getByte(field.name) - Type.Short -> result.getShort(field.name) - Type.Int -> result.getInt(field.name) - Type.Long -> result.getLong(field.name) - Type.Float -> result.getFloat(field.name) - Type.Double -> result.getDouble(field.name) - Type.Datetime -> result.getDate(field.name).toLocalDate() - is Type.BooleanVector -> result.getObject(field.name, PgBitVector::class.java) - is Type.FloatVector -> result.getObject(field.name, PgVector::class.java) - else -> throw IllegalArgumentException("Unsupported type ${field.type} in struct descriptor.") - } - ) + values[field.name] = when(field.type) { + Type.String -> result.getString(field.name)?.let { Value.String(it) } + Type.Text -> result.getString(field.name)?.let { Value.Text(it) } + Type.Boolean -> result.getBoolean(field.name).let { Value.Boolean(it) } + Type.Byte -> result.getByte(field.name).let { Value.Byte(it) } + Type.Short -> result.getShort(field.name).let { Value.Short(it) } + Type.Int -> result.getInt(field.name).let { Value.Int(it) } + Type.Long -> result.getLong(field.name).let { Value.Long(it) } + Type.Float -> result.getFloat(field.name).let { Value.Float(it) } + Type.Double -> result.getDouble(field.name).let { Value.Double(it) } + Type.Datetime -> result.getDate(field.name).toInstant().let { Value.DateTime(Date(it.toEpochMilli())) } + is Type.BooleanVector -> result.getObject(field.name, PgBitVector::class.java).toBooleanVector() + is Type.IntVector -> result.getObject(field.name, PgVector::class.java)?.toIntVector() + is Type.LongVector -> result.getObject(field.name, PgVector::class.java)?.toLongVector() + is Type.FloatVector -> result.getObject(field.name, PgVector::class.java)?.toFloatVector() + is Type.DoubleVector -> result.getObject(field.name, PgVector::class.java)?.toDoubleVector() + } as Value<*>? } /* Call constructor. */ diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt deleted file mode 100644 index ebc40cd7..00000000 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorWriter.kt +++ /dev/null @@ -1,104 +0,0 @@ -package org.vitrivr.engine.database.pgvector.descriptor.struct - -import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.database.pgvector.LOGGER -import org.vitrivr.engine.database.pgvector.PgVectorConnection -import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorWriter -import java.sql.SQLException - -/** - * An [AbstractDescriptorWriter] for [StructDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connection: PgVectorConnection) : AbstractDescriptorWriter(field, connection) { - - /** - * Adds (writes) a single [StructDescriptor] using this [StructDescriptorWriter]. - * - * @param item The [StructDescriptor] to write. - * @return True on success, false otherwise. - */ - override fun add(item: StructDescriptor): Boolean { - try { - this.prepareInsertStatement().use { stmt -> - stmt.setObject(1, item.id) - stmt.setObject(2, item.retrievableId) - var i = 3 - for (attribute in item.schema()) { - val value = item.values()[attribute.name] - if (value != null) { - stmt.setValue(i++, value) - } else { - stmt.setNull(i++, attribute.type.toSqlType()) - } - } - return stmt.execute() - } - } catch (e: SQLException) { - LOGGER.error(e) { "Failed to INSERT descriptor ${item.id} into '$tableName' due to SQL error." } - return false - } - } - - /** - * Adds (writes) a batch of [StructDescriptor] using this [StructDescriptorWriter]. - * - * @param items A [Iterable] of [StructDescriptor]s to write. - * @return True on success, false otherwise. - */ - override fun addAll(items: Iterable): Boolean { - try { - this.prepareInsertStatement().use { stmt -> - for (item in items) { - stmt.setObject(1, item.id) - stmt.setObject(2, item.retrievableId) - var i = 3 - for (attribute in item.schema()) { - val value = item.values()[attribute.name] - if (value != null) { - stmt.setValue(i++, value) - } else { - stmt.setNull(i++, attribute.type.toSqlType()) - } - } - stmt.addBatch() - } - return stmt.executeBatch().all { it == 1 } - } - } catch (e: SQLException) { - LOGGER.error(e) { "Failed to INSERT descriptors into '$tableName' due to SQL error." } - return false - } - } - - /** - * Updates a specific [StructDescriptor] using this [StructDescriptorWriter]. - * - * @param item A [StructDescriptor]s to update. - * @return True on success, false otherwise. - */ - override fun update(item: StructDescriptor): Boolean { - try { - this.prepareUpdateStatement().use { stmt -> - stmt.setObject(1, item.retrievableId) - var i = 2 - for (attribute in item.schema()) { - val value = item.values()[attribute.name] - if (value != null) { - stmt.setValue(i++, value) - } else { - stmt.setNull(i++, attribute.type.toSqlType()) - } - } - stmt.setObject(i, item.id) - return stmt.execute() - } - } catch (e: SQLException) { - LOGGER.error(e) { "Failed to UPDATE descriptors in '$tableName' due to SQL error." } - return false - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt deleted file mode 100644 index 903625e4..00000000 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializer.kt +++ /dev/null @@ -1,33 +0,0 @@ -package org.vitrivr.engine.database.pgvector.descriptor.vector - -import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer -import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.core.model.types.Value -import org.vitrivr.engine.database.pgvector.* -import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorInitializer -import java.sql.SQLException - -/** - * An [AbstractDescriptorInitializer] implementation for [VectorDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -class VectorDescriptorInitializer(field: Schema.Field<*, VectorDescriptor<*>>, connection: PgVectorConnection): AbstractDescriptorInitializer>(field, connection.jdbc) { - /** - * Initializes the [RetrievableInitializer]. - */ - override fun initialize() { - try { - val stmt = if (this.prototype.vector is Value.BooleanVector) { - this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME bit(${this.prototype.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") - } else { - this.connection.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $tableName ($DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, $RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, $DESCRIPTOR_COLUMN_NAME vector(${this.prototype.dimensionality}) NOT NULL, PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") - } - stmt.use { it.execute() } - } catch (e: SQLException) { - LOGGER.error(e) { "Failed to initialize entity due to exception." } - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorProvider.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorProvider.kt index 62cf473c..57b7a00a 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorProvider.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorProvider.kt @@ -5,6 +5,8 @@ import org.vitrivr.engine.core.database.descriptor.DescriptorProvider import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.database.pgvector.PgVectorConnection +import org.vitrivr.engine.database.pgvector.descriptor.PgDescriptorInitializer +import org.vitrivr.engine.database.pgvector.descriptor.PgDescriptorWriter /** * A [DescriptorProvider] for [VectorDescriptor]s. @@ -13,7 +15,7 @@ import org.vitrivr.engine.database.pgvector.PgVectorConnection * @version 1.0.0 */ object VectorDescriptorProvider: DescriptorProvider> { - override fun newInitializer(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = VectorDescriptorInitializer(field, connection as PgVectorConnection) + override fun newInitializer(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = PgDescriptorInitializer(field, connection as PgVectorConnection) override fun newReader(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = VectorDescriptorReader(field, connection as PgVectorConnection) - override fun newWriter(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = VectorDescriptorWriter(field, connection as PgVectorConnection) + override fun newWriter(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = PgDescriptorWriter(field, connection as PgVectorConnection) } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt index 67694a3d..17ed3b77 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt @@ -2,6 +2,7 @@ package org.vitrivr.engine.database.pgvector.descriptor.vector import org.vitrivr.engine.core.database.descriptor.DescriptorReader import org.vitrivr.engine.core.model.descriptor.vector.* +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query import org.vitrivr.engine.core.model.query.basics.Distance @@ -32,7 +33,7 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec override fun query(query: Query): Sequence> { try { val statement = when (query) { - is ProximityQuery<*> -> this.connection.jdbc.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").apply { + is ProximityQuery<*> -> this.connection.jdbc.prepareStatement("SELECT *, $VECTOR_ATTRIBUTE_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").apply { when (val vector = query.value) { is Value.FloatVector -> this.setObject(1, PgVector(vector.value)) is Value.BooleanVector -> this.setObject(1, PgBitVector(vector.value)) @@ -61,7 +62,7 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec return when (query) { is ProximityQuery<*> -> { val descriptors = mutableListOf, Float>>() - this.connection.jdbc.prepareStatement("SELECT *, $DESCRIPTOR_COLUMN_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").use { stmt -> + this.connection.jdbc.prepareStatement("SELECT *, $VECTOR_ATTRIBUTE_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").use { stmt -> when (val vector = query.value) { is Value.FloatVector -> stmt.setObject(1, PgVector(vector.value)) is Value.BooleanVector -> stmt.setObject(1, PgBitVector(vector.value)) @@ -108,17 +109,31 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec is FloatVectorDescriptor -> FloatVectorDescriptor( descriptorId, retrievableId, - result.getObject(DESCRIPTOR_COLUMN_NAME, PgVector::class.java).let { vector -> - Value.FloatVector(vector.toArray() ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) - } + result.getObject(VECTOR_ATTRIBUTE_NAME, PgVector::class.java)?.toFloatVector() ?: throw IllegalArgumentException("The provided vector value is missing the required field '$VECTOR_ATTRIBUTE_NAME'.") + ) + + is DoubleVectorDescriptor -> DoubleVectorDescriptor( + descriptorId, + retrievableId, + result.getObject(VECTOR_ATTRIBUTE_NAME, PgVector::class.java)?.toDoubleVector() ?: throw IllegalArgumentException("The provided vector value is missing the required field '$VECTOR_ATTRIBUTE_NAME'.") + ) + + is IntVectorDescriptor -> IntVectorDescriptor( + descriptorId, + retrievableId, + result.getObject(VECTOR_ATTRIBUTE_NAME, PgVector::class.java)?.toIntVector() ?: throw IllegalArgumentException("The provided vector value is missing the required field '$VECTOR_ATTRIBUTE_NAME'.") + ) + + is LongVectorDescriptor -> LongVectorDescriptor( + descriptorId, + retrievableId, + result.getObject(VECTOR_ATTRIBUTE_NAME, PgVector::class.java)?.toLongVector() ?: throw IllegalArgumentException("The provided vector value is missing the required field '$VECTOR_ATTRIBUTE_NAME'.") ) is BooleanVectorDescriptor -> BooleanVectorDescriptor( descriptorId, retrievableId, - result.getObject(DESCRIPTOR_COLUMN_NAME, PgBitVector::class.java).let { vector -> - Value.BooleanVector(vector.toArray() ?: throw IllegalArgumentException("The provided vector value is missing the required field '$DESCRIPTOR_COLUMN_NAME'.")) - } + result.getObject(VECTOR_ATTRIBUTE_NAME, PgBitVector::class.java)?.toBooleanVector() ?: throw IllegalArgumentException("The provided vector value is missing the required field '$VECTOR_ATTRIBUTE_NAME'.") ) else -> throw IllegalArgumentException("Unsupported descriptor type ${this.prototype::class}.") diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt deleted file mode 100644 index 63ec2fed..00000000 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorWriter.kt +++ /dev/null @@ -1,93 +0,0 @@ -package org.vitrivr.engine.database.pgvector.descriptor.vector - -import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.core.model.types.Value -import org.vitrivr.engine.database.pgvector.* -import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorWriter -import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector -import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector -import java.sql.SQLException - -/** - * An [AbstractDescriptorWriter] for [VectorDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connection: PgVectorConnection) : AbstractDescriptorWriter>(field, connection) { - /** - * Adds (writes) a single [VectorDescriptor] using this [VectorDescriptorWriter]. - * - * @param item The [VectorDescriptor] to write. - * @return True on success, false otherwise. - */ - override fun add(item: VectorDescriptor<*>): Boolean { - try { - this.connection.jdbc.prepareStatement("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $DESCRIPTOR_COLUMN_NAME) VALUES (?, ?, ?);").use { stmt -> - stmt.setObject(1, item.id) - stmt.setObject(2, item.retrievableId) - when (val vector = item.vector) { - is Value.FloatVector -> stmt.setObject(3, PgVector(vector.value)) - is Value.BooleanVector -> stmt.setObject(3, PgBitVector(vector.value)) - else -> throw IllegalArgumentException("Unsupported vector type ${vector::class.simpleName}") - } - return stmt.execute() - } - } catch (e: SQLException) { - LOGGER.error(e) { "Failed to persist descriptor ${item.id} due to SQL error." } - return false - } - } - - /** - * Adds (writes) a batch of [VectorDescriptor] using this [VectorDescriptorWriter]. - * - * @param items A [Iterable] of [VectorDescriptor]s to write. - * @return True on success, false otherwise. - */ - override fun addAll(items: Iterable>): Boolean { - try { - this.connection.jdbc.prepareStatement("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $DESCRIPTOR_COLUMN_NAME) VALUES (?, ?, ?);").use { stmt -> - for (item in items) { - stmt.setObject(1, item.id) - stmt.setObject(2, item.retrievableId) - when (val vector = item.vector) { - is Value.FloatVector -> stmt.setObject(3, PgVector(vector.value)) - is Value.BooleanVector -> stmt.setObject(3, PgBitVector(vector.value)) - else -> throw IllegalArgumentException("Unsupported vector type ${vector::class.simpleName}") - } - stmt.addBatch() - } - return stmt.executeBatch().all { it == 1 } - } - } catch (e: SQLException) { - LOGGER.error(e) { "Failed to persist descriptors due to SQL error." } - return false - } - } - - /** - * Updates a specific [VectorDescriptor] using this [VectorDescriptorWriter]. - * - * @param item A [VectorDescriptor]s to update. - * @return True on success, false otherwise. - */ - override fun update(item: VectorDescriptor<*>): Boolean { - try { - this.connection.jdbc.prepareStatement("UPDATE $tableName SET $RETRIEVABLE_ID_COLUMN_NAME = ?, $DESCRIPTOR_COLUMN_NAME = ? WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?;").use { stmt -> - stmt.setObject(1, item.retrievableId) - when (val vector = item.vector) { - is Value.FloatVector -> stmt.setObject(2, PgVector(vector.value)) - is Value.BooleanVector -> stmt.setObject(2, PgBitVector(vector.value)) - else -> throw IllegalArgumentException("Unsupported vector type ${vector::class.simpleName}") - } - stmt.setObject(3, item.id) - return stmt.execute() - } - } catch (e: SQLException) { - LOGGER.error(e) { "Failed to persist descriptors due to SQL error." } - return false - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt index 8abd804b..9228b042 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt @@ -17,15 +17,14 @@ internal class RetrievableInitializer(private val connection: PgVectorConnection */ override fun initialize() { try { - /* Create 'retrievable' entity. */ + /* Create 'retrievable' entity and index. */ this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, type VARCHAR(100), PRIMARY KEY ($RETRIEVABLE_ID_COLUMN_NAME));").use { it.execute() } /* Create 'relationship' entity. */ - this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") - .use { - it.execute() + this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));").use { + it.execute() } } catch (e: SQLException) { LOGGER.error(e) { "Failed to initialize entity due to exception." } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt index 4fdc9d8b..2d479018 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableReader.kt @@ -64,16 +64,16 @@ class RetrievableReader(override val connection: PgVectorConnection): Retrievabl */ override fun getAll(ids: Iterable): Sequence { try { - this.connection.jdbc.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?)").use { stmt -> - val values = ids.map { it }.toTypedArray() - stmt.setArray(1, this.connection.jdbc.createArrayOf("uuid", values)) - val result = stmt.executeQuery() - return sequence { - while (result.next()) { - yield(Retrieved(result.getObject(RETRIEVABLE_ID_COLUMN_NAME, UUID::class.java), result.getString(RETRIEVABLE_TYPE_COLUMN_NAME), false)) - } - result.close() + val statement = this.connection.jdbc.prepareStatement("SELECT * FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?)") + val values = ids.map { it }.toTypedArray() + statement.setArray(1, this.connection.jdbc.createArrayOf("uuid", values)) + return sequence { + val result = statement.executeQuery() + while (result.next()) { + yield(Retrieved(result.getObject(RETRIEVABLE_ID_COLUMN_NAME, UUID::class.java), result.getString(RETRIEVABLE_TYPE_COLUMN_NAME), false)) } + result.close() + statement.close() } } catch (e: Exception) { LOGGER.error(e) { "Failed to check for retrievables due to SQL error." } From cd027206a222ae3d72a11d982ab395adf0116170 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Wed, 10 Jul 2024 13:22:32 +0200 Subject: [PATCH 23/71] ADds means to specify indexes on schema creation. --- .../engine/core/config/schema/FieldConfig.kt | 3 +- .../engine/core/config/schema/IndexConfig.kt | 16 +++++++++ .../engine/core/config/schema/IndexType.kt | 15 ++++++++ .../engine/core/model/metamodel/Schema.kt | 18 +++++----- .../core/model/metamodel/SchemaManager.kt | 4 +-- .../descriptor/PgDescriptorInitializer.kt | 35 +++++++++++++++++-- 6 files changed, 77 insertions(+), 14 deletions(-) create mode 100644 vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexConfig.kt create mode 100644 vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexType.kt diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/FieldConfig.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/FieldConfig.kt index d3c2b4ce..f20245e4 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/FieldConfig.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/FieldConfig.kt @@ -11,6 +11,5 @@ import org.vitrivr.engine.core.model.metamodel.Schema * @author Ralph Gasser * @version 1.0.0 */ - @Serializable -data class FieldConfig(val name: String, val factory: String, val parameters: Map = emptyMap()) +data class FieldConfig(val name: String, val factory: String, val parameters: Map = emptyMap(), val indexes: List = emptyList()) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexConfig.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexConfig.kt new file mode 100644 index 00000000..76cc3e12 --- /dev/null +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexConfig.kt @@ -0,0 +1,16 @@ +package org.vitrivr.engine.core.config.schema + +import kotlinx.serialization.Serializable +import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.metamodel.Schema + +/** + * A serializable configuration object to configure indexes for a [Schema.Field] within a [Schema]. + * + * @see [Schema.Field] + * + * @author Ralph Gasser + * @version 1.0.0 + */ +@Serializable +data class IndexConfig(val attributes: List, val type: IndexType, val parameters: Map = emptyMap()) \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexType.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexType.kt new file mode 100644 index 00000000..ea6aabf2 --- /dev/null +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexType.kt @@ -0,0 +1,15 @@ +package org.vitrivr.engine.core.config.schema + +/** + * The type of index that can be created on a [Schema.Field]. + */ +enum class IndexType { + /** A B-tree based index. Well-suited for point-lookups and Boolean search on scalar values. */ + SCALAR, + + /** A fulltext index. Well-suited for fulltext-queries. */ + FULLTEXT, + + /** Index suited for NNS. */ + NNS +} \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Schema.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Schema.kt index 68add8d6..828b88ef 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Schema.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Schema.kt @@ -2,6 +2,7 @@ package org.vitrivr.engine.core.model.metamodel import org.vitrivr.engine.core.config.ingest.IngestionConfig import org.vitrivr.engine.core.config.ingest.IngestionPipelineBuilder +import org.vitrivr.engine.core.config.schema.IndexConfig import org.vitrivr.engine.core.config.schema.SchemaConfig import org.vitrivr.engine.core.context.IndexContext import org.vitrivr.engine.core.context.QueryContext @@ -29,7 +30,7 @@ typealias FieldName = String * A [Schema] that defines a particular vitrivr instance's meta data model. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ open class Schema(val name: String = "vitrivr", val connection: Connection) : Closeable { @@ -49,13 +50,13 @@ open class Schema(val name: String = "vitrivr", val connection: Connection) : Cl * Adds a new [Field] to this [Schema]. * * @param name The name of the new [Field]. Must be unique. + * @param analyser The [Analyser] to use with the new [Field]. + * @param parameters The (optional) parameters used to configure the [Field]. + * @param indexes List of [IndexConfig]s that can be used to configure indexes on the [Field]. + * @return [Field] instance. */ - fun addField( - name: String, - analyser: Analyser, Descriptor>, - parameters: Map = emptyMap() - ) { - this.fields.add(Field(name, analyser, parameters)) + fun addField(name: String, analyser: Analyser, Descriptor>, parameters: Map = emptyMap(), indexes: List) { + this.fields.add(Field(name, analyser, parameters, indexes)) } /** @@ -157,7 +158,8 @@ open class Schema(val name: String = "vitrivr", val connection: Connection) : Cl inner class Field, D : Descriptor>( val fieldName: FieldName, val analyser: Analyser, - val parameters: Map = emptyMap() + val parameters: Map = emptyMap(), + val indexes: List = emptyList() ) { /** Pointer to the [Schema] this [Field] belongs to.*/ val schema: Schema diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/SchemaManager.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/SchemaManager.kt index 7c994c0e..cc221d86 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/SchemaManager.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/SchemaManager.kt @@ -21,7 +21,7 @@ import kotlin.concurrent.write * The [SchemaManager] maps [Schema] definitions to database [Connection] objects. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.0.1 */ class SchemaManager { /** An internal [HashMap] of all open [Connection]. */ @@ -55,7 +55,7 @@ class SchemaManager { throw IllegalArgumentException("Field names must not have a dot (.) in their name.") } @Suppress("UNCHECKED_CAST") - schema.addField(it.name, analyser as Analyser, Descriptor>, it.parameters) + schema.addField(it.name, analyser as Analyser, Descriptor>, it.parameters, it.indexes) } config.resolvers.map { schema.addResolver(it.key, (loadServiceForName(it.value.factory) ?: throw IllegalArgumentException("Failed to find resolver factory implementation for '${it.value.factory}'.")).newResolver(schema, it.value.parameters)) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index d9b326b5..416f9411 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -1,8 +1,10 @@ package org.vitrivr.engine.database.pgvector.descriptor +import org.vitrivr.engine.core.config.schema.IndexType import org.vitrivr.engine.core.database.descriptor.DescriptorInitializer import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.query.basics.Distance import org.vitrivr.engine.core.model.types.Type import org.vitrivr.engine.database.pgvector.* import java.sql.SQLException @@ -22,7 +24,7 @@ open class PgDescriptorInitializer(final override val field: Sch protected val prototype = this.field.analyser.prototype(this.field) /** - * + * Initializes the PostgreSQL table entity backing this [PgDescriptorInitializer]. */ override fun initialize() { val statement = StringBuilder("CREATE TABLE IF NOT EXISTS $tableName(") @@ -54,14 +56,32 @@ open class PgDescriptorInitializer(final override val field: Sch statement.append("PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), ") statement.append("FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + /* Create entity. */ try { - /* Create 'retrievable' entity. */ this.connection.jdbc.prepareStatement(/* sql = postgres */ statement.toString()).use { it.execute() } } catch (e: SQLException) { LOGGER.error(e) { "Failed to initialize entity '$tableName' due to exception." } } + + /* Create indexes (optional). */ + for (index in this.field.indexes) { + try { + val indexStatement = when (index.type) { + IndexType.SCALAR -> "CREATE INDEX IF NOT EXISTS ON $tableName (${index.attributes.joinToString(",")});" + IndexType.FULLTEXT -> "CREATE INDEX IF NOT EXISTS ON $tableName USING GIN(${index.attributes.joinToString(",")});" + IndexType.NNS -> { + require(index.attributes.size == 1) { "NNS index can only be created on a single attribute." } + val distance = index.parameters["distance"]?.let { Distance.valueOf(it.uppercase()) } ?: Distance.EUCLIDEAN + "CREATE INDEX IF NOT EXISTS ON $tableName USING HNSW(${index.attributes.first()} ${distance.toIndexName()});" + } + } + this.connection.jdbc.prepareStatement(/* sql = postgres */ indexStatement).use { it.execute() } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to create index ${index.type} for entity '$tableName' due to exception." } + } + } } /** @@ -92,4 +112,15 @@ open class PgDescriptorInitializer(final override val field: Sch LOGGER.error(e) { "Failed to truncate entities due to exception." } } } + + /** + * Closes the [PgDescriptorInitializer]. + */ + private fun Distance.toIndexName() = when (this) { + Distance.MANHATTAN -> "vector_l1_ops" + Distance.EUCLIDEAN -> "sparsevec_l2_ops" + Distance.COSINE -> "vector_cosine_ops" + Distance.HAMMING -> "bit_hamming_ops" + Distance.JACCARD -> "bit_jaccard_ops" + } } \ No newline at end of file From 947570d2c8f89f4c636ba367dd09e184539f8965 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Wed, 10 Jul 2024 13:52:47 +0200 Subject: [PATCH 24/71] Adds support for fulltext and boolean queries and performs some cleanup. --- .../engine/database/pgvector/Utilities.kt | 60 +++++++++ .../descriptor/AbstractDescriptorReader.kt | 18 --- .../pgvector/descriptor/PgDescriptorWriter.kt | 21 --- .../pgvector/descriptor/model/PgVector.kt | 27 ++++ .../scalar/ScalarDescriptorReader.kt | 58 ++++++-- .../struct/StructDescriptorReader.kt | 73 +++++++---- .../vector/VectorDescriptorReader.kt | 124 +++++++++--------- 7 files changed, 245 insertions(+), 136 deletions(-) create mode 100644 vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Utilities.kt diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Utilities.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Utilities.kt new file mode 100644 index 00000000..92ed71c7 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Utilities.kt @@ -0,0 +1,60 @@ +package org.vitrivr.engine.database.pgvector + +import org.vitrivr.engine.core.model.query.basics.ComparisonOperator +import org.vitrivr.engine.core.model.query.basics.Distance +import org.vitrivr.engine.core.model.query.basics.Distance.* +import org.vitrivr.engine.core.model.types.Value +import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector +import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector +import java.sql.Date +import java.sql.PreparedStatement + +/** + * Sets a value of [Value] type in a [PreparedStatement]. + * + * @param index The index to set. + * @param value The [Value] to set. + */ +internal fun PreparedStatement.setValue(index: Int, value: Value<*>) = when (value) { + is Value.Boolean -> this.setBoolean(index, value.value) + is Value.Byte -> this.setByte(index, value.value) + is Value.DateTime -> this.setDate(index, Date(value.value.toInstant().toEpochMilli())) + is Value.Double -> this.setDouble(index, value.value) + is Value.Float -> this.setFloat(index, value.value) + is Value.Int -> this.setInt(index, value.value) + is Value.Long -> this.setLong(index, value.value) + is Value.Short -> this.setShort(index, value.value) + is Value.String -> this.setString(index, value.value) + is Value.FloatVector -> this.setObject(index, PgVector(value.value)) + is Value.DoubleVector -> this.setObject(index, PgVector(value.value)) + is Value.IntVector -> this.setObject(index, PgVector(value.value)) + is Value.LongVector -> this.setObject(index, PgVector(value.value)) + is Value.BooleanVector -> this.setObject(index, PgBitVector(value.value)) + else -> throw IllegalArgumentException("Unsupported value type for vector value.") +} + +/** + * Converts a [Distance] to a pgVector distance operator. + */ +fun Distance.toSql() = when(this) { + MANHATTAN -> "<+>" + EUCLIDEAN -> "<->" + COSINE -> "<=>" + HAMMING -> "<~>" + JACCARD -> "<%>" +} + +/** + * Converts a [ComparisonOperator] to a SQL operator. + * + * @return SQL comparison operator. + */ +internal fun ComparisonOperator.toSql(): String = when (this){ + ComparisonOperator.EQ -> "=" + ComparisonOperator.NEQ -> "!=" + ComparisonOperator.LE -> "<" + ComparisonOperator.GR -> ">" + ComparisonOperator.LEQ -> ">=" + ComparisonOperator.GEQ -> "<=" + ComparisonOperator.LIKE -> "LIKE" +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt index a44e287b..49c557f9 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt @@ -8,7 +8,6 @@ import org.vitrivr.engine.core.model.query.Query import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.database.pgvector.* -import java.sql.PreparedStatement import java.sql.ResultSet import java.util.* @@ -216,23 +215,6 @@ abstract class AbstractDescriptorReader(final override val field } } - - /** - * Converts a [PreparedStatement] to a [Sequence] of [Descriptor] of type [D]. - * - * @return [Sequence] of [Descriptor]s of type [D] - */ - protected fun PreparedStatement.executeAndStream(): Sequence = this.use { - val result = this.executeQuery() - return sequence { - result.use { - while (result.next()) { - yield(this@AbstractDescriptorReader.rowToDescriptor(result)) - } - } - } - } - /** * Converts a [ResultSet] to a [Descriptor] of type [D]. * diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt index 27218854..ee05d5cd 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt @@ -5,10 +5,7 @@ import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.types.Type -import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.database.pgvector.* -import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector -import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector import java.sql.* /** @@ -148,24 +145,6 @@ open class PgDescriptorWriter(final override val field: Schema.F } } - /** - * Sets a value of [Value] type in a [PreparedStatement]. - */ - protected fun PreparedStatement.setValue(index: Int, value: Value<*>) = when (value) { - is Value.Boolean -> this.setBoolean(index, value.value) - is Value.Byte -> this.setByte(index, value.value) - is Value.DateTime -> this.setDate(index, Date(value.value.toInstant().toEpochMilli())) - is Value.Double -> this.setDouble(index, value.value) - is Value.Float -> this.setFloat(index, value.value) - is Value.Int -> this.setInt(index, value.value) - is Value.Long -> this.setLong(index, value.value) - is Value.Short -> this.setShort(index, value.value) - is Value.String -> this.setString(index, value.value) - is Value.BooleanVector -> this.setObject(index, PgBitVector(value.value)) - is Value.FloatVector -> this.setObject(index, PgVector(value.value)) - else -> throw IllegalArgumentException("Unsupported value type for vector value.") - } - /** * Converts a [Type] to a [SQLType]. */ diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt index f8b57cb9..10f2bd55 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt @@ -39,6 +39,33 @@ class PgVector(private var vec: FloatArray? = null) : PGobject(), PGBinaryObject } }) + /** + * Constructor + * + * @param vector [DoubleArray] + */ + constructor(vector: DoubleArray) : this(FloatArray(vector.size) { + vector[it].toFloat() + }) + + /** + * Constructor + * + * @param vector [IntArray] + */ + constructor(vector: IntArray) : this(FloatArray(vector.size) { + vector[it].toFloat() + }) + + /** + * Constructor + * + * @param vector [LongArray] + */ + constructor(vector: LongArray) : this(FloatArray(vector.size) { + vector[it].toFloat() + }) + /** * Constructor * diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt index 3ddbb003..c25de5ab 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt @@ -3,23 +3,23 @@ package org.vitrivr.engine.database.pgvector.descriptor.scalar import org.vitrivr.engine.core.model.descriptor.scalar.* import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor -import org.vitrivr.engine.core.model.descriptor.vector.* +import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query +import org.vitrivr.engine.core.model.query.bool.SimpleBooleanQuery +import org.vitrivr.engine.core.model.query.fulltext.SimpleFulltextQuery import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.core.model.types.Value -import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME -import org.vitrivr.engine.database.pgvector.PgVectorConnection -import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.* import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorReader import java.sql.ResultSet import java.util.* /** - * A [DescriptorReader] for [ScalarDescriptor]s. + * A [AbstractDescriptorReader] for [ScalarDescriptor]s. * * @author Ralph Gasser - * @version 1.1.0 + * @version 1.0.0 */ class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connection: PgVectorConnection) : AbstractDescriptorReader>(field, connection) { @@ -29,8 +29,10 @@ class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connec * @param query The [Query] to execute. * @return [Sequence] of [StructDescriptor]s that match the query. */ - override fun query(query: Query): Sequence> { - TODO("Not yet implemented") + override fun query(query: Query): Sequence> = when (query) { + is SimpleFulltextQuery -> queryFulltext(query) + is SimpleBooleanQuery<*> -> queryBoolean(query) + else -> throw IllegalArgumentException("Query of type ${query::class} is not supported by ScalarDescriptorReader.") } /** @@ -51,4 +53,44 @@ class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connec is StringDescriptor -> StringDescriptor(descriptorId, retrievableId, Value.String(result.getString(VALUE_ATTRIBUTE_NAME))) } } + + /** + * Executes a [SimpleFulltextQuery] and returns a [Sequence] of [ScalarDescriptor]s. + * + * @param query The [SimpleFulltextQuery] to execute. + * @return [Sequence] of [ScalarDescriptor]s. + */ + private fun queryFulltext(query: SimpleFulltextQuery): Sequence> { + val statement = "SELECT * FROM $tableName WHERE $VALUE_ATTRIBUTE_NAME @@ to_tsquery(?)" + return sequence { + this@ScalarDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> + stmt.setString(1, query.value.value) + stmt.executeQuery().use { result -> + while (result.next()) { + yield(rowToDescriptor(result)) + } + } + } + } + } + + /** + * Executes a [SimpleBooleanQuery] and returns a [Sequence] of [ScalarDescriptor]s. + * + * @param query The [SimpleBooleanQuery] to execute. + * @return [Sequence] of [ScalarDescriptor]s. + */ + private fun queryBoolean(query: SimpleBooleanQuery<*>): Sequence> { + val statement = "SELECT * FROM $tableName WHERE $VALUE_ATTRIBUTE_NAME ${query.comparison.toSql()} ?" + return sequence { + this@ScalarDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> + stmt.setValue(1, query.value) + stmt.executeQuery().use { result -> + while (result.next()) { + yield(rowToDescriptor(result)) + } + } + } + } + } } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt index bdfd6405..8d650c7f 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt @@ -9,10 +9,7 @@ import org.vitrivr.engine.core.model.query.fulltext.SimpleFulltextQuery import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.core.model.types.Type import org.vitrivr.engine.core.model.types.Value -import org.vitrivr.engine.database.pgvector.DESCRIPTOR_ID_COLUMN_NAME -import org.vitrivr.engine.database.pgvector.LOGGER -import org.vitrivr.engine.database.pgvector.PgVectorConnection -import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ID_COLUMN_NAME +import org.vitrivr.engine.database.pgvector.* import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorReader import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector @@ -24,7 +21,7 @@ import kotlin.reflect.full.primaryConstructor * An [AbstractDescriptorReader] for [StructDescriptor]s. * * @author Ralph Gasser - * @version 1.1.0 + * @version 1.0.0 */ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connection: PgVectorConnection) : AbstractDescriptorReader(field, connection) { /** @@ -33,26 +30,10 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio * @param query The [Query] to execute. * @return [Sequence] of [StructDescriptor]s that match the query. */ - override fun query(query: Query): Sequence { - try { - /* Prepare statement based on query. */ - val statement = when (query) { - is SimpleFulltextQuery -> TODO() - is SimpleBooleanQuery<*> -> { - require(query.attributeName != null) { "Boolean query on a struct field requires specification of a field's attribute name." } - this.connection.jdbc.prepareStatement("SELECT * FROM $tableName WHERE ${query.attributeName} = ?;").apply { - setString(1, query.value.toString()) - } - } - else -> throw IllegalArgumentException("Query of typ ${query::class} is not supported by StringDescriptorReader.") - } - - /* Execute statement and return it. */ - return statement.executeAndStream() - } catch (e: Exception) { - LOGGER.error(e) { "Failed to execute query on '$tableName' due to SQL error." } - return emptySequence() - } + override fun query(query: Query): Sequence = when (query) { + is SimpleFulltextQuery -> queryFulltext(query) + is SimpleBooleanQuery<*> -> queryBoolean(query) + else -> throw IllegalArgumentException("Query of typ ${query::class} is not supported by StructDescriptorReader.") } /** @@ -94,4 +75,46 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio /* Call constructor. */ return constructor.call(*parameters.toTypedArray()) } + + /** + * Executes a [SimpleFulltextQuery] and returns a [Sequence] of [StructDescriptor]s. + * + * @param query The [SimpleFulltextQuery] to execute. + * @return [Sequence] of [StructDescriptor]s. + */ + private fun queryFulltext(query: SimpleFulltextQuery): Sequence { + require(query.attributeName != null) { "Query attribute must not be null for a fulltext query on a struct descriptor." } + val statement = "SELECT * FROM $tableName WHERE ${query.attributeName} @@ to_tsquery(?)" + return sequence { + this@StructDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> + stmt.setString(1, query.value.value) + stmt.executeQuery().use { result -> + while (result.next()) { + yield(rowToDescriptor(result)) + } + } + } + } + } + + /** + * Executes a [SimpleBooleanQuery] and returns a [Sequence] of [StructDescriptor]s. + * + * @param query The [SimpleBooleanQuery] to execute. + * @return [Sequence] of [StructDescriptor]s. + */ + private fun queryBoolean(query: SimpleBooleanQuery<*>): Sequence { + require(query.attributeName != null) { "Query attribute must not be null for a fulltext query on a struct descriptor." } + val statement = "SELECT * FROM $tableName WHERE ${query.value} ${query.comparison.toSql()} ?" + return sequence { + this@StructDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> + stmt.setValue(1, query.value) + stmt.executeQuery().use { result -> + while (result.next()) { + yield(rowToDescriptor(result)) + } + } + } + } + } } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt index 17ed3b77..789324d7 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt @@ -5,12 +5,9 @@ import org.vitrivr.engine.core.model.descriptor.vector.* import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query -import org.vitrivr.engine.core.model.query.basics.Distance -import org.vitrivr.engine.core.model.query.basics.Distance.* import org.vitrivr.engine.core.model.query.proximity.ProximityQuery import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.core.model.retrievable.attributes.DistanceAttribute -import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.database.pgvector.* import org.vitrivr.engine.database.pgvector.descriptor.AbstractDescriptorReader import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector @@ -30,23 +27,9 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec * * @param query The [Query] to execute. */ - override fun query(query: Query): Sequence> { - try { - val statement = when (query) { - is ProximityQuery<*> -> this.connection.jdbc.prepareStatement("SELECT *, $VECTOR_ATTRIBUTE_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").apply { - when (val vector = query.value) { - is Value.FloatVector -> this.setObject(1, PgVector(vector.value)) - is Value.BooleanVector -> this.setObject(1, PgBitVector(vector.value)) - else -> throw IllegalArgumentException("Unsupported value type ${query.value::class}.") - } - } - else -> throw UnsupportedOperationException("Query of typ ${query::class} is not supported by VectorDescriptorReader.") - } - return statement.executeAndStream() - } catch (e: Exception) { - LOGGER.error(e) { "Failed to execute query on '$tableName' due to SQL error." } - return emptySequence() - } + override fun query(query: Query): Sequence> = when (query) { + is ProximityQuery<*> -> queryProximity(query) + else -> throw UnsupportedOperationException("Query of typ ${query::class} is not supported by VectorDescriptorReader.") } /** @@ -57,43 +40,9 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec * @param query The [Query] that should be executed. * @return [Sequence] of [Retrieved]. */ - override fun queryAndJoin(query: Query): Sequence { - try { - return when (query) { - is ProximityQuery<*> -> { - val descriptors = mutableListOf, Float>>() - this.connection.jdbc.prepareStatement("SELECT *, $VECTOR_ATTRIBUTE_NAME ${query.distance.operator()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME LIMIT ${query.k}").use { stmt -> - when (val vector = query.value) { - is Value.FloatVector -> stmt.setObject(1, PgVector(vector.value)) - is Value.BooleanVector -> stmt.setObject(1, PgBitVector(vector.value)) - else -> throw IllegalArgumentException("Unsupported value type ${query.value::class}.") - } - stmt.executeQuery().use { result -> - while (result.next()) { - descriptors.add(this@VectorDescriptorReader.rowToDescriptor(result) to result.getFloat(DISTANCE_COLUMN_NAME)) - } - } - } - - /* Fetch retrievable ids. */ - val retrievables = this.connection.getRetrievableReader().getAll(descriptors.mapNotNull { it.first.retrievableId }.toSet()).map { it.id to it }.toMap() - descriptors.asSequence().mapNotNull { (descriptor, distance) -> - val retrievable = retrievables[descriptor.retrievableId] - if (retrievable != null) { - retrievable.addDescriptor(descriptor) - retrievable.addAttribute(DistanceAttribute(distance)) - retrievable as Retrieved - } else { - null - } - } - } - else -> super.queryAndJoin(query) - } - } catch (e: Exception) { - LOGGER.error(e) { "Failed to execute query on '$tableName' due to SQL error." } - return emptySequence() - } + override fun queryAndJoin(query: Query): Sequence = when (query) { + is ProximityQuery<*> -> queryAndJoinProximity(query) + else -> super.queryAndJoin(query) } /** @@ -141,13 +90,60 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec } /** - * Converts a [Distance] to a pgVector distance operator. + * Executes a [ProximityQuery] and returns a [Sequence] of [VectorDescriptor]s. + * + * @param query The [ProximityQuery] to execute. + * @return [Sequence] of [VectorDescriptor]s. */ - private fun Distance.operator() = when(this) { - MANHATTAN -> "<+>" - EUCLIDEAN -> "<->" - COSINE -> "<=>" - HAMMING -> "<~>" - JACCARD -> "<%>" + private fun queryProximity(query: ProximityQuery<*>): Sequence> = sequence { + val statement = if (query.fetchVector) { + "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" + } else { + "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" + } + this@VectorDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> + stmt.setValue(1, query.value) + stmt.executeQuery().use { result -> + while (result.next()) { + yield(rowToDescriptor(result)) + } + } + } + } + + /** + * Executes a [ProximityQuery] and returns a [Sequence] of [VectorDescriptor]s. + * + * @param query The [ProximityQuery] to execute. + * @return [Sequence] of [VectorDescriptor]s. + */ + private fun queryAndJoinProximity(query: ProximityQuery<*>): Sequence { + val descriptors = mutableListOf, Float>>() + val statement = if (query.fetchVector) { + "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" + } else { + "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" + } + this@VectorDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> + stmt.setValue(1, query.value) + stmt.executeQuery().use { result -> + while (result.next()) { + descriptors.add(this@VectorDescriptorReader.rowToDescriptor(result) to result.getFloat(DISTANCE_COLUMN_NAME)) + } + } + + /* Fetch retrievable ids. */ + val retrievables = this.connection.getRetrievableReader().getAll(descriptors.mapNotNull { it.first.retrievableId }.toSet()).map { it.id to it }.toMap() + return descriptors.asSequence().mapNotNull { (descriptor, distance) -> + val retrievable = retrievables[descriptor.retrievableId] + if (retrievable != null) { + retrievable.addDescriptor(descriptor) + retrievable.addAttribute(DistanceAttribute(distance)) + retrievable as Retrieved + } else { + null + } + } + } } } \ No newline at end of file From 6bb4532c6ccaf43a4613f1346734f5e89c4585e0 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Wed, 10 Jul 2024 15:19:05 +0200 Subject: [PATCH 25/71] Removes 'IF NOT EXISTS' from PgDescriptorInitializer. --- .../database/pgvector/descriptor/PgDescriptorInitializer.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index 416f9411..b30c22ff 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -74,7 +74,7 @@ open class PgDescriptorInitializer(final override val field: Sch IndexType.NNS -> { require(index.attributes.size == 1) { "NNS index can only be created on a single attribute." } val distance = index.parameters["distance"]?.let { Distance.valueOf(it.uppercase()) } ?: Distance.EUCLIDEAN - "CREATE INDEX IF NOT EXISTS ON $tableName USING HNSW(${index.attributes.first()} ${distance.toIndexName()});" + "CREATE INDEX ON $tableName USING hnsw (${index.attributes.first()} ${distance.toIndexName()});" } } this.connection.jdbc.prepareStatement(/* sql = postgres */ indexStatement).use { it.execute() } From 93657462f4c1389cc7e909ba7c78c78d85688caa Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Wed, 10 Jul 2024 16:01:03 +0200 Subject: [PATCH 26/71] PgVector and PgBitVector are now Cloneable. --- .../pgvector/descriptor/model/PgBitVector.kt | 16 ++++++++++++++-- .../pgvector/descriptor/model/PgVector.kt | 11 +++++++++-- 2 files changed, 23 insertions(+), 4 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt index 11e04df8..6a608820 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgBitVector.kt @@ -11,12 +11,12 @@ import java.sql.SQLException /** * The [PgBitVector] class represents a bit vector in PostgreSQL. * - * @see https://github.com/pgvector/pgvector-java/blob/master/src/main/java/com/pgvector/PGbit.java + * @link https://github.com/pgvector/pgvector-java/blob/master/src/main/java/com/pgvector/PGbit.java * * @author Ralph Gasser * @version 1.0.0 */ -class PgBitVector () : PGobject(), PGBinaryObject, Serializable { +class PgBitVector() : PGobject(), PGBinaryObject, Serializable, Cloneable { /** The [ByteArray] backing this [PgBitVector]. */ private var vec: ByteArray? = null @@ -120,6 +120,18 @@ class PgBitVector () : PGobject(), PGBinaryObject, Serializable { } } + /** + * Clones this [PgBitVector]. + * + * @return Copy of this [PgBitVector]. + */ + override fun clone(): PgBitVector { + val clone = PgBitVector() + clone.vec = this.vec?.copyOf() + clone.length = this.length + return clone + } + /** * Returns the length * diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt index 10f2bd55..dc458c5a 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/model/PgVector.kt @@ -11,12 +11,12 @@ import java.util.* /** * The [PgVector] class represents a vector in PostgreSQL. * - * @see https://github.com/pgvector/pgvector-java/blob/master/src/main/java/com/pgvector/PGvector.java + * @link https://github.com/pgvector/pgvector-java/blob/master/src/main/java/com/pgvector/PGvector.java * * @author Ralph Gasser * @version 1.0.0 */ -class PgVector(private var vec: FloatArray? = null) : PGobject(), PGBinaryObject, Serializable { +class PgVector(private var vec: FloatArray? = null) : PGobject(), PGBinaryObject, Serializable, Cloneable { init { this.type = "vector" @@ -135,6 +135,13 @@ class PgVector(private var vec: FloatArray? = null) : PGobject(), PGBinaryObject } } + /** + * Clones this [PgVector]. + * + * @return Copy of this [PgVector]. + */ + override fun clone(): PgVector = PgVector(this.vec?.copyOf()) + /** * Returns an [Value.FloatVector] representation of this [PgVector]. * From 470fc86f92a606f7d9e62962729a7093c1fb8aed Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Wed, 10 Jul 2024 16:14:40 +0200 Subject: [PATCH 27/71] Fixes a bug in StructDescriptorReader. --- .../pgvector/descriptor/struct/StructDescriptorReader.kt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt index 8d650c7f..0977c70f 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt @@ -48,7 +48,8 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio val parameters: MutableList = mutableListOf( result.getObject(DESCRIPTOR_ID_COLUMN_NAME, UUID::class.java) ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'."), result.getObject(RETRIEVABLE_ID_COLUMN_NAME, UUID::class.java) ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'."), - values + values, + this.field ) /* Append dynamic parameters of struct. */ @@ -105,7 +106,7 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio */ private fun queryBoolean(query: SimpleBooleanQuery<*>): Sequence { require(query.attributeName != null) { "Query attribute must not be null for a fulltext query on a struct descriptor." } - val statement = "SELECT * FROM $tableName WHERE ${query.value} ${query.comparison.toSql()} ?" + val statement = "SELECT * FROM $tableName WHERE ${query.attributeName} ${query.comparison.toSql()} ?" return sequence { this@StructDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> stmt.setValue(1, query.value) From a17beead8ac6f4b405ab820605041bcb23b1c55e Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 10:23:10 +0200 Subject: [PATCH 28/71] Simplifies Cottontail DB database driver based on recent refactoring activities. Signed-off-by: Ralph Gasser --- gradle.properties | 1 + .../CottontailConnectionProvider.kt | 5 +- .../AbstractDescriptorInitializer.kt | 49 ------ .../descriptors/AbstractDescriptorReader.kt | 20 +-- .../descriptors/AbstractDescriptorWriter.kt | 83 ----------- .../CottontailDescriptorInitializer.kt | 73 +++++++++ ...riter.kt => CottontailDescriptorWriter.kt} | 101 +++++++++---- .../scalar/ScalarDescriptorInitializer.kt | 40 ----- .../scalar/ScalarDescriptorProvider.kt | 8 +- .../scalar/ScalarDescriptorReader.kt | 69 ++++++--- .../scalar/ScalarDescriptorWriter.kt | 97 ------------ .../string/StringDescriptorInitializer.kt | 52 ------- .../string/StringDescriptorProvider.kt | 19 --- .../string/StringDescriptorReader.kt | 119 --------------- .../string/StringDescriptorWriter.kt | 105 ------------- .../struct/StructDescriptorInitializer.kt | 41 ----- .../struct/StructDescriptorProvider.kt | 8 +- .../struct/StructDescriptorReader.kt | 76 ++++++---- .../vector/VectorDescriptorInitializer.kt | 36 ----- .../vector/VectorDescriptorProvider.kt | 6 +- .../vector/VectorDescriptorReader.kt | 140 ++++++++++-------- .../vector/VectorDescriptorWriter.kt | 103 ------------- vitrivr-engine-module-pgvector/build.gradle | 2 +- .../descriptor/PgDescriptorInitializer.kt | 2 + 24 files changed, 345 insertions(+), 910 deletions(-) delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorInitializer.kt delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt create mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt rename vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/{struct/StructDescriptorWriter.kt => CottontailDescriptorWriter.kt} (50%) delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorInitializer.kt delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorWriter.kt delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorInitializer.kt delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorProvider.kt delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorReader.kt delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorWriter.kt delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorInitializer.kt delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorInitializer.kt delete mode 100644 vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorWriter.kt diff --git a/gradle.properties b/gradle.properties index b5dd6299..bd2eeb1b 100644 --- a/gradle.properties +++ b/gradle.properties @@ -7,6 +7,7 @@ version_javacv=1.5.9 version_javalin=6.1.3 version_javalinopenapi=6.1.3 version_javalinssl=6.1.3 +version_jdbc_postgres=42.7.3 version_jline=3.23.0 version_junit=5.10.1 version_junit_platform=1.10.1 diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt index 53b73327..44625cb4 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnectionProvider.kt @@ -14,7 +14,6 @@ import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.VideoSour import org.vitrivr.engine.core.model.descriptor.vector.* import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.descriptors.scalar.ScalarDescriptorProvider -import org.vitrivr.engine.plugin.cottontaildb.descriptors.string.StringDescriptorProvider import org.vitrivr.engine.plugin.cottontaildb.descriptors.struct.StructDescriptorProvider import org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescriptorProvider @@ -56,9 +55,7 @@ class CottontailConnectionProvider: AbstractConnectionProvider() { this.register(LongDescriptor::class, ScalarDescriptorProvider) this.register(FloatDescriptor::class, ScalarDescriptorProvider) this.register(DoubleDescriptor::class, ScalarDescriptorProvider) - - /* String descriptor. */ - this.register(StringDescriptor::class, StringDescriptorProvider) + this.register(StringDescriptor::class, ScalarDescriptorProvider) /* Vector descriptors. */ this.register(BooleanVectorDescriptor::class, VectorDescriptorProvider) diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorInitializer.kt deleted file mode 100644 index c0d6ed83..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorInitializer.kt +++ /dev/null @@ -1,49 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors - -import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.ddl.ListEntities -import org.vitrivr.cottontail.client.language.ddl.TruncateEntity -import org.vitrivr.cottontail.core.database.Name -import org.vitrivr.engine.core.database.descriptor.DescriptorInitializer -import org.vitrivr.engine.core.model.descriptor.Descriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection -import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ENTITY_PREFIX - -/** - * An abstract implementation of a [DescriptorInitializer] for Cottontail DB. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -abstract class AbstractDescriptorInitializer(final override val field: Schema.Field<*, D>, protected val connection: CottontailConnection) : DescriptorInitializer { - /** The [Name.EntityName] used by this [Descriptor]. */ - protected val entityName: Name.EntityName = Name.EntityName.create(this.field.schema.name, "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName.lowercase()}") - - /** - * Checks if the schema for this [AbstractDescriptorInitializer] has been properly initialized. - * - * @return True if entity has been initialized, false otherwise. - */ - override fun isInitialized(): Boolean = try { - this.connection.client.list(ListEntities(this.entityName.schema)).asSequence().any { - Name.EntityName.parse(it.asString(0)!!) == this.entityName - } - } catch (e: StatusRuntimeException) { - false - } - - /** - * Truncates the entity backing this [AbstractDescriptorInitializer]. - */ - override fun truncate() { - val truncate = TruncateEntity(this.entityName) - try { - this.connection.client.truncate(truncate).use { - it.hasNext() - } - } catch (e: StatusRuntimeException) { - /* TODO: Log. */ - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt index e2909540..03c8d662 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt @@ -1,7 +1,5 @@ package org.vitrivr.engine.plugin.cottontaildb.descriptors -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging import io.grpc.StatusRuntimeException import org.vitrivr.cottontail.client.language.basics.expression.Column import org.vitrivr.cottontail.client.language.basics.expression.Literal @@ -19,13 +17,11 @@ import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.plugin.cottontaildb.* import java.util.* -private val logger: KLogger = KotlinLogging.logger {} - /** * An abstract implementation of a [DescriptorReader] for Cottontail DB. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.0.1 */ abstract class AbstractDescriptorReader(final override val field: Schema.Field<*, D>, override val connection: CottontailConnection) : DescriptorReader { @@ -51,7 +47,7 @@ abstract class AbstractDescriptorReader(final override val field ret } } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to retrieve descriptor $descriptorId due to exception." } + LOGGER.error(e) { "Failed to retrieve descriptor $descriptorId due to exception." } null } } @@ -68,7 +64,7 @@ abstract class AbstractDescriptorReader(final override val field val result = this.connection.client.query(query) result.asSequence().map { this.tupleToDescriptor(it) } } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to retrieve descriptors for retrievable $retrievableId due to exception." } + LOGGER.error(e) { "Failed to retrieve descriptors for retrievable $retrievableId due to exception." } emptySequence() } } @@ -86,7 +82,7 @@ abstract class AbstractDescriptorReader(final override val field val result = this.connection.client.query(query) result.next().asBoolean(0) ?: false } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to retrieve descriptor $descriptorId due to exception." } + LOGGER.error(e) { "Failed to retrieve descriptor $descriptorId due to exception." } false } } @@ -102,7 +98,7 @@ abstract class AbstractDescriptorReader(final override val field val result = this.connection.client.query(query) result.asSequence().map { this.tupleToDescriptor(it) } } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to retrieve descriptors due to exception." } + LOGGER.error(e) { "Failed to retrieve descriptors due to exception." } emptySequence() } } @@ -120,7 +116,7 @@ abstract class AbstractDescriptorReader(final override val field val result = this.connection.client.query(query) result.asSequence().map { this.tupleToDescriptor(it) } } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to retrieve descriptors for provided descriptor IDs due to exception." } + LOGGER.error(e) { "Failed to retrieve descriptors for provided descriptor IDs due to exception." } emptySequence() } } @@ -138,7 +134,7 @@ abstract class AbstractDescriptorReader(final override val field val result = this.connection.client.query(query) result.asSequence().map { this.tupleToDescriptor(it) } } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to retrieve descriptors for provided retrievable IDs due to exception." } + LOGGER.error(e) { "Failed to retrieve descriptors for provided retrievable IDs due to exception." } emptySequence() } } @@ -160,7 +156,7 @@ abstract class AbstractDescriptorReader(final override val field result.close() ret } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to count descriptors due to exception." } + LOGGER.error(e) { "Failed to count descriptors due to exception." } 0L } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt deleted file mode 100644 index ae5a74d2..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorWriter.kt +++ /dev/null @@ -1,83 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors - -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging -import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.basics.expression.Column -import org.vitrivr.cottontail.client.language.basics.expression.Literal -import org.vitrivr.cottontail.client.language.basics.predicate.Compare -import org.vitrivr.cottontail.client.language.dml.Delete -import org.vitrivr.cottontail.core.database.Name -import org.vitrivr.cottontail.core.values.UuidValue -import org.vitrivr.engine.core.database.descriptor.DescriptorWriter -import org.vitrivr.engine.core.model.descriptor.Descriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection -import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ENTITY_PREFIX -import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ID_COLUMN_NAME - -private val logger: KLogger = KotlinLogging.logger {} - -/** - * An abstract implementation of a [DescriptorWriter] for Cottontail DB. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -abstract class AbstractDescriptorWriter(final override val field: Schema.Field<*, D>, override val connection: CottontailConnection) : DescriptorWriter { - /** The [Name.EntityName] used by this [Descriptor]. */ - protected val entityName: Name.EntityName = Name.EntityName.create(this.field.schema.name, "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName.lowercase()}") - - /** - * Deletes (writes) a [Descriptor] of type [D] using this [AbstractDescriptorWriter]. - * - * @param item A [Descriptor]s to delete. - * @return True on success, false otherwise. - */ - override fun delete(item: D): Boolean { - val delete = Delete(this.entityName).where( - Compare( - Column(this.entityName.column(DESCRIPTOR_ID_COLUMN_NAME)), - Compare.Operator.EQUAL, - Literal(UuidValue(item.id)) - ) - ) - - /* Delete values. */ - return try { - this.connection.client.delete(delete).use { - it.hasNext() - } - } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to delete scalar descriptor due to exception." } - false - } - } - - /** - * Deletes (writes) [Descriptor]s of type [D] using this [AbstractDescriptorWriter]. - * - * @param items A [Iterable] of [Descriptor]s to delete. - * @return True on success, false otherwise. - */ - override fun deleteAll(items: Iterable): Boolean { - val ids = items.map { UuidValue(it.id) } - val delete = Delete(this.entityName).where( - Compare( - Column(this.entityName.column(DESCRIPTOR_ID_COLUMN_NAME)), - Compare.Operator.IN, - org.vitrivr.cottontail.client.language.basics.expression.List(ids.toTypedArray()) - ) - ) - - /* Delete values. */ - return try { - this.connection.client.delete(delete).use { - it.hasNext() - } - } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to delete descriptor due to exception." } - false - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt new file mode 100644 index 00000000..66ce7540 --- /dev/null +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt @@ -0,0 +1,73 @@ +package org.vitrivr.engine.plugin.cottontaildb.descriptors + +import io.grpc.StatusRuntimeException +import org.vitrivr.cottontail.client.language.ddl.CreateEntity +import org.vitrivr.cottontail.client.language.ddl.ListEntities +import org.vitrivr.cottontail.client.language.ddl.TruncateEntity +import org.vitrivr.cottontail.core.database.Name +import org.vitrivr.cottontail.core.types.Types +import org.vitrivr.engine.core.database.descriptor.DescriptorInitializer +import org.vitrivr.engine.core.model.descriptor.Descriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.plugin.cottontaildb.* + +/** + * An abstract implementation of a [DescriptorInitializer] for Cottontail DB. + * + * @author Ralph Gasser + * @version 1.1.0 + */ +open class CottontailDescriptorInitializer(final override val field: Schema.Field<*, D>, protected val connection: CottontailConnection) : DescriptorInitializer { + /** The [Name.EntityName] used by this [Descriptor]. */ + protected val entityName: Name.EntityName = Name.EntityName.create(this.field.schema.name, "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName.lowercase()}") + + /** + * Initializes the Cottontail DB entity backing this [CottontailDescriptorInitializer]. + */ + override fun initialize() { + /* Prepare query. */ + val create = CreateEntity(this.entityName) + .column(Name.ColumnName.create(DESCRIPTOR_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = true, autoIncrement = false) + .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) + + /* Append fields. */ + for (field in this.field.analyser.prototype(this.field).schema()) { + create.column(Name.ColumnName.create(field.name), field.type.toCottontailType(), nullable = field.nullable, primaryKey = false, autoIncrement = false) + } + + try { + /* Try to create entity. */ + this.connection.client.create(create) + } catch (e: StatusRuntimeException) { + LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + } + } + + /** + * Checks if the schema for this [CottontailDescriptorInitializer] has been properly initialized. + * + * @return True if entity has been initialized, false otherwise. + */ + override fun isInitialized(): Boolean = try { + this.connection.client.list(ListEntities(this.entityName.schema)).asSequence().any { + Name.EntityName.parse(it.asString(0)!!) == this.entityName + } + } catch (e: StatusRuntimeException) { + LOGGER.error(e) { "Failed to check initialization status of entity '${this.entityName}' due to exception." } + false + } + + /** + * Truncates the entity backing this [CottontailDescriptorInitializer]. + */ + override fun truncate() { + val truncate = TruncateEntity(this.entityName) + try { + this.connection.client.truncate(truncate).use { + it.hasNext() + } + } catch (e: StatusRuntimeException) { + LOGGER.error(e) { "Failed to truncate entity '${this.entityName}' due to exception." } + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorWriter.kt similarity index 50% rename from vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorWriter.kt rename to vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorWriter.kt index d5e689e6..24fbdaed 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorWriter.kt @@ -1,48 +1,46 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.struct +package org.vitrivr.engine.plugin.cottontaildb.descriptors -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging import io.grpc.StatusRuntimeException import org.vitrivr.cottontail.client.language.basics.expression.Column import org.vitrivr.cottontail.client.language.basics.expression.Literal import org.vitrivr.cottontail.client.language.basics.predicate.Compare import org.vitrivr.cottontail.client.language.dml.BatchInsert +import org.vitrivr.cottontail.client.language.dml.Delete import org.vitrivr.cottontail.client.language.dml.Insert import org.vitrivr.cottontail.client.language.dml.Update +import org.vitrivr.cottontail.core.database.Name import org.vitrivr.cottontail.core.values.UuidValue +import org.vitrivr.engine.core.database.descriptor.DescriptorWriter +import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection -import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ID_COLUMN_NAME -import org.vitrivr.engine.plugin.cottontaildb.RETRIEVABLE_ID_COLUMN_NAME -import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorWriter -import org.vitrivr.engine.plugin.cottontaildb.toCottontailValue - -private val logger: KLogger = KotlinLogging.logger {} +import org.vitrivr.engine.plugin.cottontaildb.* /** - * An [AbstractDescriptorWriter] for [StructDescriptor]s. + * An abstract implementation of a [DescriptorWriter] for Cottontail DB. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ -class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connection: CottontailConnection) : AbstractDescriptorWriter(field, connection) { +open class CottontailDescriptorWriter(final override val field: Schema.Field<*, D>, override val connection: CottontailConnection) : DescriptorWriter { + /** The [Name.EntityName] used by this [Descriptor]. */ + protected val entityName: Name.EntityName = Name.EntityName.create(this.field.schema.name, "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName.lowercase()}") /** - * Adds (writes) a single [StructDescriptor] using this [StructDescriptorWriter]. + * Adds (writes) a single [StructDescriptor] using this [CottontailDescriptorWriter]. * * @param item The [StructDescriptor] to write. * @return True on success, false otherwise. */ - override fun add(item: StructDescriptor): Boolean { + override fun add(item: D): Boolean { val insert = Insert(this.entityName).values( DESCRIPTOR_ID_COLUMN_NAME to UuidValue(item.id), RETRIEVABLE_ID_COLUMN_NAME to UuidValue(item.retrievableId ?: throw IllegalArgumentException("A struct descriptor must be associated with a retrievable ID.")), ) /* Append fields. */ - for ((field, value) in item.values()) { - insert.value(field, value?.toCottontailValue()) + for ((attribute, value) in item.values()) { + insert.value(attribute, value?.toCottontailValue()) } return try { @@ -50,18 +48,18 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio it.hasNext() } } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to persist descriptor ${item.id} due to exception." } + LOGGER.error(e) { "Failed to persist descriptor ${item.id} due to exception." } false } } /** - * Adds (writes) a batch of [StructDescriptor] using this [StructDescriptorWriter]. + * Adds (writes) a batch of [StructDescriptor] using this [CottontailDescriptorWriter]. * * @param items A [Iterable] of [StructDescriptor]s to write. * @return True on success, false otherwise. */ - override fun addAll(items: Iterable): Boolean { + override fun addAll(items: Iterable): Boolean { /* Prepare insert query. */ var index = 0 val insert = BatchInsert(this.entityName) @@ -96,18 +94,18 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio it.hasNext() } } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to persist ${index + 1} scalar descriptors due to exception." } + LOGGER.error(e) { "Failed to persist ${index + 1} scalar descriptors due to exception." } false } } /** - * Updates a specific [StructDescriptor] using this [StructDescriptorWriter]. + * Updates a specific [Descriptor] of type [D] using this [CottontailDescriptorWriter]. * - * @param item A [StructDescriptor]s to update. + * @param item A [Descriptor]s to update. * @return True on success, false otherwise. */ - override fun update(item: StructDescriptor): Boolean { + override fun update(item: D): Boolean { val update = Update(this.entityName).where( Compare( Column(this.entityName.column(DESCRIPTOR_ID_COLUMN_NAME)), @@ -126,7 +124,60 @@ class StructDescriptorWriter(field: Schema.Field<*, StructDescriptor>, connectio this.connection.client.update(update) true } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to update descriptor due to exception." } + LOGGER.error(e) { "Failed to update descriptor due to exception." } + false + } + } + + /** + * Deletes (writes) a [Descriptor] of type [D] using this [CottontailDescriptorWriter]. + * + * @param item A [Descriptor]s to delete. + * @return True on success, false otherwise. + */ + override fun delete(item: D): Boolean { + val delete = Delete(this.entityName).where( + Compare( + Column(this.entityName.column(DESCRIPTOR_ID_COLUMN_NAME)), + Compare.Operator.EQUAL, + Literal(UuidValue(item.id)) + ) + ) + + /* Delete values. */ + return try { + this.connection.client.delete(delete).use { + it.hasNext() + } + } catch (e: StatusRuntimeException) { + LOGGER.error(e) { "Failed to delete scalar descriptor due to exception." } + false + } + } + + /** + * Deletes (writes) [Descriptor]s of type [D] using this [CottontailDescriptorWriter]. + * + * @param items A [Iterable] of [Descriptor]s to delete. + * @return True on success, false otherwise. + */ + override fun deleteAll(items: Iterable): Boolean { + val ids = items.map { UuidValue(it.id) } + val delete = Delete(this.entityName).where( + Compare( + Column(this.entityName.column(DESCRIPTOR_ID_COLUMN_NAME)), + Compare.Operator.IN, + org.vitrivr.cottontail.client.language.basics.expression.List(ids.toTypedArray()) + ) + ) + + /* Delete values. */ + return try { + this.connection.client.delete(delete).use { + it.hasNext() + } + } catch (e: StatusRuntimeException) { + LOGGER.error(e) { "Failed to delete descriptor due to exception." } false } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorInitializer.kt deleted file mode 100644 index 718a3fc3..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorInitializer.kt +++ /dev/null @@ -1,40 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.scalar - -import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.ddl.CreateEntity -import org.vitrivr.cottontail.core.database.Name -import org.vitrivr.cottontail.core.types.Types -import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor -import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.* -import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorInitializer - -/** - * A [AbstractDescriptorInitializer] implementation for [ScalarDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.1 - */ -class ScalarDescriptorInitializer>(field: Schema.Field<*, T>, connection: CottontailConnection) : AbstractDescriptorInitializer(field, connection) { - /** - * Initializes the Cottontail DB entity backing this [AbstractDescriptorInitializer]. - */ - override fun initialize() { - /* Determine type based on prototype for Schema.Field. */ - val type = this.field.analyser.prototype(this.field).toType() - - /* Prepare query. */ - val create = CreateEntity(this.entityName) - .column(Name.ColumnName.create(DESCRIPTOR_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = true, autoIncrement = false) - .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) - .column(Name.ColumnName.create(VALUE_ATTRIBUTE_NAME), type, nullable = false, primaryKey = false, autoIncrement = false) - - try { - /* Try to create entity. */ - this.connection.client.create(create) - } catch (e: StatusRuntimeException) { - LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorProvider.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorProvider.kt index ce6b262e..b27c560c 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorProvider.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorProvider.kt @@ -5,15 +5,17 @@ import org.vitrivr.engine.core.database.descriptor.DescriptorProvider import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection +import org.vitrivr.engine.plugin.cottontaildb.descriptors.CottontailDescriptorInitializer +import org.vitrivr.engine.plugin.cottontaildb.descriptors.CottontailDescriptorWriter /** * A [DescriptorProvider] for [ScalarDescriptor]s. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ internal object ScalarDescriptorProvider : DescriptorProvider> { - override fun newInitializer(connection: Connection, field: Schema.Field<*, ScalarDescriptor<*>>) = ScalarDescriptorInitializer(field, connection as CottontailConnection) + override fun newInitializer(connection: Connection, field: Schema.Field<*, ScalarDescriptor<*>>) = CottontailDescriptorInitializer(field, connection as CottontailConnection) override fun newReader(connection: Connection, field: Schema.Field<*, ScalarDescriptor<*>>) = ScalarDescriptorReader(field, connection as CottontailConnection) - override fun newWriter(connection: Connection, field: Schema.Field<*, ScalarDescriptor<*>>) = ScalarDescriptorWriter(field, connection as CottontailConnection) + override fun newWriter(connection: Connection, field: Schema.Field<*, ScalarDescriptor<*>>) = CottontailDescriptorWriter(field, connection as CottontailConnection) } \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorReader.kt index 26497147..46db246b 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorReader.kt @@ -10,6 +10,7 @@ import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companio import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query import org.vitrivr.engine.core.model.query.bool.SimpleBooleanQuery +import org.vitrivr.engine.core.model.query.fulltext.SimpleFulltextQuery import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.core.model.types.toValue import org.vitrivr.engine.plugin.cottontaildb.* @@ -31,29 +32,10 @@ class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connec * * @param query The [Query] to execute. */ - override fun query(query: Query): Sequence> { - require(query is SimpleBooleanQuery<*>) { "Query of type ${query::class} is not supported by ScalarDescriptorReader." } - - /* Prepare query. */ - val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) - .select(RETRIEVABLE_ID_COLUMN_NAME) - .select(DESCRIPTOR_ID_COLUMN_NAME) - .select(VALUE_ATTRIBUTE_NAME) - .where(Compare(Column(this.entityName.column(VALUE_ATTRIBUTE_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) - - /* Execute query. */ - return this.connection.client.query(cottontailQuery).asSequence().map { - this.tupleToDescriptor(it) - } - } - - /** - * Executes the provided [Query] and returns a [Sequence] of [Retrieved]s that match it. - * - * @param query The [Query] to execute. - */ - override fun queryAndJoin(query: Query): Sequence { - TODO("Not yet implemented") + override fun query(query: Query): Sequence> = when (query) { + is SimpleFulltextQuery -> this.queryFulltext(query) + is SimpleBooleanQuery<*> -> this.queryBoolean(query) + else -> throw UnsupportedOperationException("The provided query type ${query::class.simpleName} is not supported by this reader.") } /** @@ -74,4 +56,45 @@ class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connec is StringDescriptor -> StringDescriptor(retrievableId, descriptorId, tuple.asString(VALUE_ATTRIBUTE_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VECTOR_ATTRIBUTE_NAME'.")) } } + + /** + * Executes a [SimpleFulltextQuery] and returns a [Sequence] of [ScalarDescriptor]s. + * + * @param query The [SimpleFulltextQuery] to execute. + * @return [Sequence] of [ScalarDescriptor]s. + */ + private fun queryFulltext(query: SimpleFulltextQuery): Sequence> { + val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) + .select("*") + .fulltext(VALUE_ATTRIBUTE_NAME, query.value.value, "score") + + if (query.limit < Long.MAX_VALUE) { + cottontailQuery.limit(query.limit) + } + + /* Execute query. */ + return this.connection.client.query(cottontailQuery).asSequence().map { + this.tupleToDescriptor(it) + } + } + + /** + * Executes a [SimpleBooleanQuery] and returns a [Sequence] of [ScalarDescriptor]s. + * + * @param query The [SimpleBooleanQuery] to execute. + * @return [Sequence] of [ScalarDescriptor]s. + */ + private fun queryBoolean(query: SimpleBooleanQuery<*>): Sequence> { + /* Prepare query. */ + val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) + .select(RETRIEVABLE_ID_COLUMN_NAME) + .select(DESCRIPTOR_ID_COLUMN_NAME) + .select(VALUE_ATTRIBUTE_NAME) + .where(Compare(Column(this.entityName.column(VALUE_ATTRIBUTE_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) + + /* Execute query. */ + return this.connection.client.query(cottontailQuery).asSequence().map { + this.tupleToDescriptor(it) + } + } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorWriter.kt deleted file mode 100644 index 63e83778..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/scalar/ScalarDescriptorWriter.kt +++ /dev/null @@ -1,97 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.scalar - -import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.basics.expression.Column -import org.vitrivr.cottontail.client.language.basics.expression.Literal -import org.vitrivr.cottontail.client.language.basics.predicate.Compare -import org.vitrivr.cottontail.client.language.dml.BatchInsert -import org.vitrivr.cottontail.client.language.dml.Insert -import org.vitrivr.cottontail.client.language.dml.Update -import org.vitrivr.cottontail.core.values.UuidValue -import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor -import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.* -import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorWriter - -/** - * An [AbstractDescriptorWriter] for [ScalarDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.1 - */ -class ScalarDescriptorWriter(field: Schema.Field<*, ScalarDescriptor<*>>, connection: CottontailConnection) : AbstractDescriptorWriter>(field, connection) { - - /** - * Adds (writes) a single [ScalarDescriptor] using this [ScalarDescriptorWriter]. - * - * @param item The [ScalarDescriptor] to write. - * @return True on success, false otherwise. - */ - override fun add(item: ScalarDescriptor<*>): Boolean { - val insert = Insert(this.entityName).values( - DESCRIPTOR_ID_COLUMN_NAME to UuidValue(item.id), - RETRIEVABLE_ID_COLUMN_NAME to UuidValue(item.retrievableId ?: throw IllegalArgumentException("A scalar descriptor must be associated with a retrievable ID.")), - VALUE_ATTRIBUTE_NAME to item.toCottontailValue() - ) - return try { - this.connection.client.insert(insert).use { - it.hasNext() - } - } catch (e: StatusRuntimeException) { - LOGGER.error(e) { "Failed to persist descriptor ${item.id} due to exception." } - false - } - } - - /** - * Adds (writes) a batch of [ScalarDescriptor] using this [ScalarDescriptorWriter]. - * - * @param items A [Iterable] of [ScalarDescriptor]s to write. - * @return True on success, false otherwise. - */ - override fun addAll(items: Iterable>): Boolean { - /* Prepare insert query. */ - var size = 0 - val insert = BatchInsert(this.entityName).columns(DESCRIPTOR_ID_COLUMN_NAME, RETRIEVABLE_ID_COLUMN_NAME, VALUE_ATTRIBUTE_NAME) - for (item in items) { - size += 1 - insert.values(UuidValue(item.id.toString()), UuidValue(item.retrievableId ?: throw IllegalArgumentException("A scalar descriptor must be associated with a retrievable ID.")), item.toCottontailValue()) - } - - /* Insert values. */ - return try { - this.connection.client.insert(insert).use { - it.hasNext() - } - } catch (e: StatusRuntimeException) { - LOGGER.error(e) { "Failed to persist $size scalar descriptors due to exception." } - false - } - } - - /** - * Updates a specific [ScalarDescriptor] using this [ScalarDescriptorWriter]. - * - * @param item A [ScalarDescriptor]s to update. - * @return True on success, false otherwise. - */ - override fun update(item: ScalarDescriptor<*>): Boolean { - val update = Update(this.entityName).where( - Compare( - Column(this.entityName.column(DESCRIPTOR_ID_COLUMN_NAME)), - Compare.Operator.EQUAL, - Literal(UuidValue(item.id)) - ) - ).values(VALUE_ATTRIBUTE_NAME to item.toCottontailValue()) - - /* Delete values. */ - return try { - this.connection.client.update(update) - true - } catch (e: StatusRuntimeException) { - LOGGER.error(e) { "Failed to update descriptor due to exception." } - false - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorInitializer.kt deleted file mode 100644 index 52350e75..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorInitializer.kt +++ /dev/null @@ -1,52 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.string - -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging -import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.ddl.CreateEntity -import org.vitrivr.cottontail.client.language.ddl.CreateIndex -import org.vitrivr.cottontail.core.database.Name -import org.vitrivr.cottontail.core.types.Types -import org.vitrivr.cottontail.grpc.CottontailGrpc -import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME -import org.vitrivr.engine.core.model.descriptor.scalar.StringDescriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection -import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ID_COLUMN_NAME -import org.vitrivr.engine.plugin.cottontaildb.RETRIEVABLE_ID_COLUMN_NAME -import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorInitializer - -private val logger: KLogger = KotlinLogging.logger {} - -/** - * A [AbstractDescriptorInitializer] implementation for [StringDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -class StringDescriptorInitializer(field: Schema.Field<*, StringDescriptor>, connection: CottontailConnection) : AbstractDescriptorInitializer(field, connection) { - /** - * Initializes the Cottontail DB entity backing this [AbstractDescriptorInitializer]. - */ - override fun initialize() { - /* Prepare to create entity. */ - val create = CreateEntity(this.entityName) - .column(Name.ColumnName.create(DESCRIPTOR_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = true, autoIncrement = false) - .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) - .column(Name.ColumnName.create(VALUE_ATTRIBUTE_NAME), Types.String, nullable = false, primaryKey = false, autoIncrement = false) - - try { - /* Try to create entity. */ - this.connection.client.create(create) - - /* Create entity if necessary. */ - if (this.field.parameters.containsKey("index")) { - val createIndex = CreateIndex(this.entityName, CottontailGrpc.IndexType.LUCENE) - .column(this.entityName.column(VALUE_ATTRIBUTE_NAME)) - this.connection.client.create(createIndex) - } - } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorProvider.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorProvider.kt deleted file mode 100644 index 13f347eb..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorProvider.kt +++ /dev/null @@ -1,19 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.string - -import org.vitrivr.engine.core.database.Connection -import org.vitrivr.engine.core.database.descriptor.DescriptorProvider -import org.vitrivr.engine.core.model.descriptor.scalar.StringDescriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection - -/** - * A [DescriptorProvider] for [StringDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -object StringDescriptorProvider : DescriptorProvider { - override fun newInitializer(connection: Connection, field: Schema.Field<*, StringDescriptor>) = StringDescriptorInitializer(field, connection as CottontailConnection) - override fun newReader(connection: Connection, field: Schema.Field<*, StringDescriptor>) = StringDescriptorReader(field, connection as CottontailConnection) - override fun newWriter(connection: Connection, field: Schema.Field<*, StringDescriptor>) = StringDescriptorWriter(field, connection as CottontailConnection) -} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorReader.kt deleted file mode 100644 index f9090635..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorReader.kt +++ /dev/null @@ -1,119 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.string - -import org.vitrivr.cottontail.client.language.basics.expression.Column -import org.vitrivr.cottontail.client.language.basics.expression.Literal -import org.vitrivr.cottontail.client.language.basics.predicate.Compare -import org.vitrivr.cottontail.core.tuple.Tuple -import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME -import org.vitrivr.engine.core.model.descriptor.scalar.StringDescriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.core.model.query.Query -import org.vitrivr.engine.core.model.query.bool.SimpleBooleanQuery -import org.vitrivr.engine.core.model.query.fulltext.SimpleFulltextQuery -import org.vitrivr.engine.core.model.retrievable.Retrieved -import org.vitrivr.engine.core.model.retrievable.attributes.ScoreAttribute -import org.vitrivr.engine.core.model.types.Value -import org.vitrivr.engine.core.model.types.toValue -import org.vitrivr.engine.plugin.cottontaildb.* -import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorReader - - -/** - * An [AbstractDescriptorReader] for [StringDescriptor]s. - * - * @author Ralph Gasser - * @version 2.0.0 - */ -class StringDescriptorReader(field: Schema.Field<*, StringDescriptor>, connection: CottontailConnection) : AbstractDescriptorReader(field, connection) { - /** - * Executes the provided [Query] and returns a [Sequence] of [Retrieved]s that match it. - * - * @param query The [Query] to execute. - */ - override fun query(query: Query): Sequence { - /* Prepare query. */ - val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName).select(RETRIEVABLE_ID_COLUMN_NAME) - when (query) { - is SimpleFulltextQuery -> { - cottontailQuery - .select("*") - .fulltext(VALUE_ATTRIBUTE_NAME, query.value.value, "score") - if (query.limit < Long.MAX_VALUE) { - cottontailQuery.limit(query.limit) - } - } - - is SimpleBooleanQuery<*> -> { - require(query.value is Value.String) { "StringDescriptorReader can only perform comparisons to string values." } - cottontailQuery.where(Compare(Column(this.entityName.column(VALUE_ATTRIBUTE_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) - } - else -> throw IllegalArgumentException("Query of typ ${query::class} is not supported by StringDescriptorReader.") - } - - /* Execute query. */ - return this.connection.client.query(cottontailQuery).asSequence().map { tuple -> - this.tupleToDescriptor(tuple) - } - } - - /** - * Returns a [Sequence] of all [Retrieved]s that match the given [Query]. - * - * Implicitly, this methods executes a [query] and then JOINS the result with the [Retrieved]s. - * - * @param query The [Query] that should be executed. - * @return [Sequence] of [Retrieved]. - */ - override fun queryAndJoin(query: Query): Sequence { - /* Prepare query. */ - val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName).select(RETRIEVABLE_ID_COLUMN_NAME) - when (query) { - is SimpleFulltextQuery -> { - cottontailQuery - .select(VALUE_ATTRIBUTE_NAME) - .select(DESCRIPTOR_ID_COLUMN_NAME) - .fulltext(VALUE_ATTRIBUTE_NAME, query.value.value, "score") - - if (query.limit < Long.MAX_VALUE) { - cottontailQuery.limit(query.limit) - } - } - - is SimpleBooleanQuery<*> -> { - require(query.value is Value.String) { "StringDescriptorReader can only perform comparisons to string values." } - cottontailQuery.where(Compare(Column(this.entityName.column(VALUE_ATTRIBUTE_NAME)), query.operator(), Literal(query.value.toCottontailValue()))) - } - - else -> throw IllegalArgumentException("Query of typ ${query::class} is not supported by StringDescriptorReader.") - } - - /* Execute query. */ - val descriptors = this.connection.client.query(cottontailQuery).asSequence().map { tuple -> - val scoreIndex = tuple.indexOf(SCORE_COLUMN_NAME) - tupleToDescriptor(tuple) to if (scoreIndex > -1) { - tuple.asDouble(SCORE_COLUMN_NAME)?.let { ScoreAttribute.Unbound(it.toFloat()) } - } else { - null - } - }.toList() - if (descriptors.isEmpty()) return emptySequence() - - /* Fetch retrievable ids. */ - val retrievables = this.fetchRetrievable(descriptors.mapNotNull { it.first.retrievableId }.toSet()) - return descriptors.asSequence().mapNotNull { descriptor -> - val retrievable = retrievables[descriptor.first.retrievableId] ?: return@mapNotNull null - - /* Append descriptor and score attribute. */ - retrievable.addDescriptor(descriptor.first) - descriptor.second?.let { retrievable.addAttribute(it) } - retrievable - } - } - - override fun tupleToDescriptor(tuple: Tuple): StringDescriptor { - val retrievableId = tuple.asUuidValue(RETRIEVABLE_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${RETRIEVABLE_ID_COLUMN_NAME}'.") - val descriptorId = tuple.asUuidValue(DESCRIPTOR_ID_COLUMN_NAME)?.value ?: throw IllegalArgumentException("The provided tuple is missing the required field '${DESCRIPTOR_ID_COLUMN_NAME}'.") - val value = tuple.asString(VALUE_ATTRIBUTE_NAME)?.toValue() ?: throw IllegalArgumentException("The provided tuple is missing the required field '$VALUE_ATTRIBUTE_NAME'.") - return StringDescriptor(descriptorId, retrievableId, value) - } -} diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorWriter.kt deleted file mode 100644 index 042f6287..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/string/StringDescriptorWriter.kt +++ /dev/null @@ -1,105 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.string - -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging -import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.basics.expression.Column -import org.vitrivr.cottontail.client.language.basics.expression.Literal -import org.vitrivr.cottontail.client.language.basics.predicate.Compare -import org.vitrivr.cottontail.client.language.dml.BatchInsert -import org.vitrivr.cottontail.client.language.dml.Insert -import org.vitrivr.cottontail.client.language.dml.Update -import org.vitrivr.cottontail.core.values.UuidValue -import org.vitrivr.engine.core.model.descriptor.scalar.ScalarDescriptor.Companion.VALUE_ATTRIBUTE_NAME -import org.vitrivr.engine.core.model.descriptor.scalar.StringDescriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection -import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ID_COLUMN_NAME -import org.vitrivr.engine.plugin.cottontaildb.RETRIEVABLE_ID_COLUMN_NAME -import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorWriter -import org.vitrivr.engine.plugin.cottontaildb.descriptors.vector.VectorDescriptorWriter -import org.vitrivr.engine.plugin.cottontaildb.toCottontailValue - -private val logger: KLogger = KotlinLogging.logger {} - -/** - * An [AbstractDescriptorWriter] for [StringDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -class StringDescriptorWriter(field: Schema.Field<*, StringDescriptor>, connection: CottontailConnection) : AbstractDescriptorWriter(field, connection) { - - /** - * Adds (writes) a single [StringDescriptor] using this [StringDescriptorWriter]. - * - * @param item The [StringDescriptor] to write. - * @return True on success, false otherwise. - */ - override fun add(item: StringDescriptor): Boolean { - val insert = Insert(this.entityName).values( - DESCRIPTOR_ID_COLUMN_NAME to UuidValue(item.id), - RETRIEVABLE_ID_COLUMN_NAME to UuidValue(item.retrievableId ?: throw IllegalArgumentException("A string descriptor must be associated with a retrievable ID.")), - VALUE_ATTRIBUTE_NAME to item.toCottontailValue() - ) - return try { - this.connection.client.insert(insert).use { - it.hasNext() - } - } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to persist descriptor ${item.id} due to exception." } - false - } - } - - /** - * Adds (writes) a batch of [StringDescriptor] using this [StringDescriptorWriter]. - * - * @param items A [Iterable] of [StringDescriptor]s to write. - * @return True on success, false otherwise. - */ - override fun addAll(items: Iterable): Boolean { - /* Prepare insert query. */ - var size = 0 - val insert = BatchInsert(this.entityName).columns(DESCRIPTOR_ID_COLUMN_NAME, RETRIEVABLE_ID_COLUMN_NAME, VALUE_ATTRIBUTE_NAME) - for (item in items) { - size += 1 - insert.values(UuidValue(item.id.toString()), UuidValue(item.retrievableId ?: throw IllegalArgumentException("A string descriptor must be associated with a retrievable ID.")), item.toCottontailValue()) - } - - /* Insert values. */ - return try { - this.connection.client.insert(insert).use { - it.hasNext() - } - } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to persist $size string descriptors due to exception." } - false - } - } - - /** - * Updates a specific [StringDescriptor] using this [VectorDescriptorWriter]. - * - * @param item A [StringDescriptor]s to update. - * @return True on success, false otherwise. - */ - override fun update(item: StringDescriptor): Boolean { - val update = Update(this.entityName).where( - Compare( - Column(this.entityName.column(DESCRIPTOR_ID_COLUMN_NAME)), - Compare.Operator.EQUAL, - Literal(UuidValue(item.id)) - ) - ).values(VALUE_ATTRIBUTE_NAME to item.toCottontailValue()) - - /* Updates values. */ - return try { - this.connection.client.update(update) - true - } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to update descriptor due to exception." } - false - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorInitializer.kt deleted file mode 100644 index 77895435..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorInitializer.kt +++ /dev/null @@ -1,41 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.struct - -import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.ddl.CreateEntity -import org.vitrivr.cottontail.core.database.Name -import org.vitrivr.cottontail.core.types.Types -import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.* -import org.vitrivr.engine.plugin.cottontaildb.LOGGER -import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorInitializer -import org.vitrivr.engine.plugin.cottontaildb.toCottontailType - -/** - * A [AbstractDescriptorInitializer] implementation for [StructDescriptor]s. - * - * @author Ralph Gasser - * @version 1.1.0 - */ -class StructDescriptorInitializer(field: Schema.Field<*, StructDescriptor>, connection: CottontailConnection) : AbstractDescriptorInitializer(field, connection) { - /** - * Initializes the Cottontail DB entity backing this [AbstractDescriptorInitializer]. - */ - override fun initialize() { - /* Prepare query. */ - val create = CreateEntity(this.entityName) - .column(Name.ColumnName.create(DESCRIPTOR_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = true, autoIncrement = false) - .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) - - for (field in this.field.analyser.prototype(this.field).schema()) { - create.column(Name.ColumnName.create(field.name), field.type.toCottontailType(), nullable = field.nullable, primaryKey = false, autoIncrement = false) - } - - try { - /* Try to create entity. */ - this.connection.client.create(create) - } catch (e: StatusRuntimeException) { - LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } - } - } -} diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorProvider.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorProvider.kt index 68022c70..7629868e 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorProvider.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorProvider.kt @@ -6,15 +6,17 @@ import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection +import org.vitrivr.engine.plugin.cottontaildb.descriptors.CottontailDescriptorInitializer +import org.vitrivr.engine.plugin.cottontaildb.descriptors.CottontailDescriptorWriter /** * A [DescriptorProvider] for [LabelDescriptor]s. * * @author Ralph Gasser - * @version 1.0.0 + * @version 1.1.0 */ object StructDescriptorProvider : DescriptorProvider { - override fun newInitializer(connection: Connection, field: Schema.Field<*, StructDescriptor>) = StructDescriptorInitializer(field, connection as CottontailConnection) + override fun newInitializer(connection: Connection, field: Schema.Field<*, StructDescriptor>) = CottontailDescriptorInitializer(field, connection as CottontailConnection) override fun newReader(connection: Connection, field: Schema.Field<*, StructDescriptor>) = StructDescriptorReader(field, connection as CottontailConnection) - override fun newWriter(connection: Connection, field: Schema.Field<*, StructDescriptor>) = StructDescriptorWriter(field, connection as CottontailConnection) + override fun newWriter(connection: Connection, field: Schema.Field<*, StructDescriptor>) = CottontailDescriptorWriter(field, connection as CottontailConnection) } \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt index cdf7237e..3e34b99b 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt @@ -7,7 +7,6 @@ import org.vitrivr.cottontail.core.tuple.Tuple import org.vitrivr.cottontail.core.types.Types import org.vitrivr.engine.core.model.descriptor.AttributeName import org.vitrivr.engine.core.model.descriptor.struct.LabelDescriptor -import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.Query @@ -17,7 +16,6 @@ import org.vitrivr.engine.core.model.retrievable.Retrieved import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.plugin.cottontaildb.* import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorReader -import java.util.* import kotlin.reflect.full.primaryConstructor /** @@ -43,33 +41,10 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio * @param query The [Query] to execute. * @return [Sequence] of [StructDescriptor]s that match the query. */ - override fun query(query: Query): Sequence { - /* Prepare query. */ - val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName).select(RETRIEVABLE_ID_COLUMN_NAME).select(DESCRIPTOR_ID_COLUMN_NAME) - for ((name, _) in this.fieldMap) { - cottontailQuery.select(name) - } - - when (query) { - is SimpleFulltextQuery -> { - require(query.attributeName != null) { "Fulltext query on a struct field requires specification of a field's attribute name." } - cottontailQuery.fulltext(query.attributeName!!, query.value.value, SCORE_COLUMN_NAME) - if (query.limit < Long.MAX_VALUE) { - cottontailQuery.limit(query.limit) - } - } - - is SimpleBooleanQuery<*> -> { - require(query.attributeName != null){"Boolean query on a struct field requires specification of a field's attribute name."} - cottontailQuery.where(Compare(Column(query.attributeName!!), query.operator(), Literal(query.value.toCottontailValue()))) - } - else -> throw IllegalArgumentException("Query of typ ${query::class} is not supported by StringDescriptorReader.") - } - - /* Execute query. */ - return this.connection.client.query(cottontailQuery).asSequence().map { tuple -> - this.tupleToDescriptor(tuple) - } + override fun query(query: Query): Sequence = when (query) { + is SimpleFulltextQuery -> this.queryFulltext(query) + is SimpleBooleanQuery<*> -> this.queryBoolean(query) + else -> throw UnsupportedOperationException("The provided query type ${query::class.simpleName} is not supported by this reader.") } /** @@ -113,4 +88,47 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio /* Call constructor. */ return constructor.call(*parameters.toTypedArray()) } + + /** + * Executes a [SimpleFulltextQuery] and returns a [Sequence] of [StructDescriptor]s. + * + * @param query The [SimpleFulltextQuery] to execute. + * @return [Sequence] of [StructDescriptor]s. + */ + private fun queryFulltext(query: SimpleFulltextQuery): Sequence { + require(query.attributeName != null) { "Fulltext query on a struct field requires specification of a field's attribute name." } + val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName).select(RETRIEVABLE_ID_COLUMN_NAME).select(DESCRIPTOR_ID_COLUMN_NAME) + for ((name, _) in this.fieldMap) { + cottontailQuery.select(name) + } + cottontailQuery.fulltext(query.attributeName!!, query.value.value, SCORE_COLUMN_NAME) + if (query.limit < Long.MAX_VALUE) { + cottontailQuery.limit(query.limit) + } + + /* Execute query. */ + return this.connection.client.query(cottontailQuery).asSequence().map { + this.tupleToDescriptor(it) + } + } + + /** + * Executes a [SimpleBooleanQuery] and returns a [Sequence] of [StructDescriptor]s. + * + * @param query The [SimpleBooleanQuery] to execute. + * @return [Sequence] of [StructDescriptor]s. + */ + private fun queryBoolean(query: SimpleBooleanQuery<*>): Sequence { + require(query.attributeName != null) { "Boolean query on a struct field requires specification of a field's attribute name." } + val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName).select(RETRIEVABLE_ID_COLUMN_NAME).select(DESCRIPTOR_ID_COLUMN_NAME) + for ((name, _) in this.fieldMap) { + cottontailQuery.select(name) + } + cottontailQuery.where(Compare(Column(query.attributeName!!), query.operator(), Literal(query.value.toCottontailValue()))) + + /* Execute query. */ + return this.connection.client.query(cottontailQuery).asSequence().map { + this.tupleToDescriptor(it) + } + } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorInitializer.kt deleted file mode 100644 index 2c042560..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorInitializer.kt +++ /dev/null @@ -1,36 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.vector - -import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.ddl.CreateEntity -import org.vitrivr.cottontail.core.database.Name -import org.vitrivr.cottontail.core.types.Types -import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor -import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.* -import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorInitializer - -/** - * A [AbstractDescriptorInitializer] implementation for [VectorDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -internal class VectorDescriptorInitializer(field: Schema.Field<*, VectorDescriptor<*>>, connection: CottontailConnection) : AbstractDescriptorInitializer>(field, connection) { - /** - * Initializes the Cottontail DB entity backing this [AbstractDescriptorInitializer]. - */ - override fun initialize() { - val type = this.field.analyser.prototype(this.field).toType() - val create = CreateEntity(this.entityName) - .column(Name.ColumnName.create(DESCRIPTOR_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = true, autoIncrement = false) - .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) - .column(Name.ColumnName.create(VECTOR_ATTRIBUTE_NAME), type, nullable = false, primaryKey = false, autoIncrement = false) - - try { - this.connection.client.create(create).close() - } catch (e: StatusRuntimeException) { - LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorProvider.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorProvider.kt index 6d2b8a55..ec87c41e 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorProvider.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorProvider.kt @@ -5,6 +5,8 @@ import org.vitrivr.engine.core.database.descriptor.DescriptorProvider import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection +import org.vitrivr.engine.plugin.cottontaildb.descriptors.CottontailDescriptorInitializer +import org.vitrivr.engine.plugin.cottontaildb.descriptors.CottontailDescriptorWriter /** * A [DescriptorProvider] for [VectorDescriptor]s. @@ -13,7 +15,7 @@ import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection * @version 1.0.0 */ internal object VectorDescriptorProvider : DescriptorProvider> { - override fun newInitializer(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = VectorDescriptorInitializer(field, connection as CottontailConnection) + override fun newInitializer(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = CottontailDescriptorInitializer(field, connection as CottontailConnection) override fun newReader(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = VectorDescriptorReader(field, connection as CottontailConnection) - override fun newWriter(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = VectorDescriptorWriter(field, connection as CottontailConnection) + override fun newWriter(connection: Connection, field: Schema.Field<*, VectorDescriptor<*>>) = CottontailDescriptorWriter(field, connection as CottontailConnection) } \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt index a2b6c171..65f3a690 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt @@ -18,7 +18,7 @@ import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorRead * An [AbstractDescriptorReader] for [FloatVectorDescriptor]s. * * @author Ralph Gasser - * @version 1.3.0 + * @version 1.3.1 */ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connection: CottontailConnection) : AbstractDescriptorReader>(field, connection) { @@ -31,28 +31,7 @@ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*> * @param query The [Query] to execute. */ override fun query(query: Query): Sequence> = when (query) { - is ProximityQuery<*> -> { - val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) - .select(RETRIEVABLE_ID_COLUMN_NAME) - .distance( - VECTOR_ATTRIBUTE_NAME, - query.value.toCottontailValue(), - Distances.valueOf(query.distance.toString()), - DISTANCE_COLUMN_NAME - ) - .order(DISTANCE_COLUMN_NAME, Direction.valueOf(query.order.name)) - .limit(query.k) - - if (query.fetchVector) { - cottontailQuery.select(VECTOR_ATTRIBUTE_NAME) - cottontailQuery.select(DESCRIPTOR_ID_COLUMN_NAME) - } - - this.connection.client.query(cottontailQuery).asSequence().map { - tupleToDescriptor(it) - } - } - + is ProximityQuery<*> -> queryProximity(query) else -> throw UnsupportedOperationException("Query of typ ${query::class} is not supported by FloatVectorDescriptorReader.") } @@ -64,47 +43,9 @@ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*> * @param query The [Query] that should be executed. * @return [Sequence] of [Retrieved]. */ - override fun queryAndJoin(query: Query): Sequence { - when (query) { - is ProximityQuery<*> -> { - val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) - .select(VECTOR_ATTRIBUTE_NAME) - .select(DESCRIPTOR_ID_COLUMN_NAME) - .select(RETRIEVABLE_ID_COLUMN_NAME) - .distance( - VECTOR_ATTRIBUTE_NAME, - query.value.toCottontailValue(), - Distances.valueOf(query.distance.toString()), - DISTANCE_COLUMN_NAME - ) - .order(DISTANCE_COLUMN_NAME, Direction.valueOf(query.order.name)) - .limit(query.k) - - /* Fetch descriptors */ - val descriptors = this.connection.client.query(cottontailQuery).asSequence().map { tuple -> - val scoreIndex = tuple.indexOf(DISTANCE_COLUMN_NAME) - tupleToDescriptor(tuple) to if (scoreIndex > -1) { - tuple.asFloat(DISTANCE_COLUMN_NAME)?.let { DistanceAttribute(it) } - } else { - null - } - }.toList() - if (descriptors.isEmpty()) return emptySequence() - - /* Fetch retrievable ids. */ - val retrievables = this.fetchRetrievable(descriptors.mapNotNull { it.first.retrievableId }.toSet()) - return descriptors.asSequence().mapNotNull { descriptor -> - val retrievable = retrievables[descriptor.first.retrievableId] ?: return@mapNotNull null - - /* Append descriptor and distance attribute. */ - retrievable.addDescriptor(descriptor.first) - descriptor.second?.let { retrievable.addAttribute(it) } - retrievable - } - } - - else -> throw UnsupportedOperationException("Query of typ ${query::class} is not supported by FloatVectorDescriptorReader.") - } + override fun queryAndJoin(query: Query): Sequence = when (query) { + is ProximityQuery<*> -> queryAndJoinProximity(query) + else -> super.queryAndJoin(query) } /** @@ -150,4 +91,75 @@ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*> ) } } + + /** + * Executes a [ProximityQuery] and returns a [Sequence] of [VectorDescriptor]s. + * + * @param query The [ProximityQuery] to execute. + * @return [Sequence] of [VectorDescriptor]s. + */ + private fun queryAndJoinProximity(query: ProximityQuery<*>): Sequence { + val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) + .select(VECTOR_ATTRIBUTE_NAME) + .select(DESCRIPTOR_ID_COLUMN_NAME) + .select(RETRIEVABLE_ID_COLUMN_NAME) + .distance( + VECTOR_ATTRIBUTE_NAME, + query.value.toCottontailValue(), + Distances.valueOf(query.distance.toString()), + DISTANCE_COLUMN_NAME + ) + .order(DISTANCE_COLUMN_NAME, Direction.valueOf(query.order.name)) + .limit(query.k) + + /* Fetch descriptors */ + val descriptors = this.connection.client.query(cottontailQuery).asSequence().map { tuple -> + val scoreIndex = tuple.indexOf(DISTANCE_COLUMN_NAME) + tupleToDescriptor(tuple) to if (scoreIndex > -1) { + tuple.asFloat(DISTANCE_COLUMN_NAME)?.let { DistanceAttribute(it) } + } else { + null + } + }.toList() + if (descriptors.isEmpty()) return emptySequence() + + /* Fetch retrievable ids. */ + val retrievables = this.fetchRetrievable(descriptors.mapNotNull { it.first.retrievableId }.toSet()) + return descriptors.asSequence().mapNotNull { descriptor -> + val retrievable = retrievables[descriptor.first.retrievableId] ?: return@mapNotNull null + + /* Append descriptor and distance attribute. */ + retrievable.addDescriptor(descriptor.first) + descriptor.second?.let { retrievable.addAttribute(it) } + retrievable + } + } + + /** + * Executes a [ProximityQuery] and returns a [Sequence] of [VectorDescriptor]s. + * + * @param query The [ProximityQuery] to execute. + * @return [Sequence] of [VectorDescriptor]s. + */ + private fun queryProximity(query: ProximityQuery<*>): Sequence> { + val cottontailQuery = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) + .select(RETRIEVABLE_ID_COLUMN_NAME) + .distance( + VECTOR_ATTRIBUTE_NAME, + query.value.toCottontailValue(), + Distances.valueOf(query.distance.toString()), + DISTANCE_COLUMN_NAME + ) + .order(DISTANCE_COLUMN_NAME, Direction.valueOf(query.order.name)) + .limit(query.k) + + if (query.fetchVector) { + cottontailQuery.select(VECTOR_ATTRIBUTE_NAME) + cottontailQuery.select(DESCRIPTOR_ID_COLUMN_NAME) + } + + return this.connection.client.query(cottontailQuery).asSequence().map { + tupleToDescriptor(it) + } + } } \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorWriter.kt deleted file mode 100644 index c225e07c..00000000 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorWriter.kt +++ /dev/null @@ -1,103 +0,0 @@ -package org.vitrivr.engine.plugin.cottontaildb.descriptors.vector - -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging -import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.basics.expression.Column -import org.vitrivr.cottontail.client.language.basics.expression.Literal -import org.vitrivr.cottontail.client.language.basics.predicate.Compare -import org.vitrivr.cottontail.client.language.dml.BatchInsert -import org.vitrivr.cottontail.client.language.dml.Insert -import org.vitrivr.cottontail.client.language.dml.Update -import org.vitrivr.cottontail.core.values.UuidValue -import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor -import org.vitrivr.engine.core.model.descriptor.vector.VectorDescriptor.Companion.VECTOR_ATTRIBUTE_NAME -import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.plugin.cottontaildb.CottontailConnection -import org.vitrivr.engine.plugin.cottontaildb.DESCRIPTOR_ID_COLUMN_NAME -import org.vitrivr.engine.plugin.cottontaildb.RETRIEVABLE_ID_COLUMN_NAME -import org.vitrivr.engine.plugin.cottontaildb.descriptors.AbstractDescriptorWriter -import org.vitrivr.engine.plugin.cottontaildb.toCottontailValue - -private val logger: KLogger = KotlinLogging.logger {} - -/** - * An [AbstractDescriptorWriter] for [VectorDescriptor]s. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -class VectorDescriptorWriter(field: Schema.Field<*, VectorDescriptor<*>>, connection: CottontailConnection) : AbstractDescriptorWriter>(field, connection) { - - /** - * Adds (writes) a single [VectorDescriptor] using this [VectorDescriptorWriter]. - * - * @param item The [VectorDescriptor] to write. - * @return True on success, false otherwise. - */ - override fun add(item: VectorDescriptor<*>): Boolean { - val insert = Insert(this.entityName).values( - DESCRIPTOR_ID_COLUMN_NAME to UuidValue(item.id.toString()), - RETRIEVABLE_ID_COLUMN_NAME to UuidValue(item.retrievableId ?: throw IllegalArgumentException("A vector descriptor must be associated with a retrievable ID.")), - VECTOR_ATTRIBUTE_NAME to item.toCottontailValue() - ) - return try { - this.connection.client.insert(insert).use { - it.hasNext() - } - } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to persist descriptor ${item.id} due to exception." } - false - } - } - - /** - * Adds (writes) a batch of [VectorDescriptor] using this [VectorDescriptorWriter]. - * - * @param items A [Iterable] of [VectorDescriptor]s to write. - * @return True on success, false otherwise. - */ - override fun addAll(items: Iterable>): Boolean { - /* Prepare insert query. */ - var size = 0 - val insert = BatchInsert(this.entityName).columns(DESCRIPTOR_ID_COLUMN_NAME, RETRIEVABLE_ID_COLUMN_NAME, VECTOR_ATTRIBUTE_NAME) - for (item in items) { - size += 1 - insert.values(UuidValue(item.id), UuidValue(item.retrievableId ?: throw IllegalArgumentException("A vector descriptor must be associated with a retrievable ID.")), item.toCottontailValue()) - } - - /* Insert values. */ - return try { - this.connection.client.insert(insert) - true - } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to persist $size descriptors due to exception." } - false - } - } - - /** - * Updates a specific [VectorDescriptor] using this [VectorDescriptorWriter]. - * - * @param item A [VectorDescriptor]s to update. - * @return True on success, false otherwise. - */ - override fun update(item: VectorDescriptor<*>): Boolean { - val update = Update(this.entityName).where( - Compare( - Column(this.entityName.column(DESCRIPTOR_ID_COLUMN_NAME)), - Compare.Operator.EQUAL, - Literal(UuidValue(item.id)) - ) - ).values(VECTOR_ATTRIBUTE_NAME to item.toCottontailValue()) - - /* Delete values. */ - return try { - this.connection.client.update(update) - true - } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to update descriptor due to exception." } - false - } - } -} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/build.gradle b/vitrivr-engine-module-pgvector/build.gradle index ba047ee8..98d76110 100644 --- a/vitrivr-engine-module-pgvector/build.gradle +++ b/vitrivr-engine-module-pgvector/build.gradle @@ -11,7 +11,7 @@ dependencies { api project(':vitrivr-engine-core') /** PostgreSQL driver. */ - implementation group: 'org.postgresql', name: 'postgresql', version: '42.7.3' + implementation group: 'org.postgresql', name: 'postgresql', version: version_jdbc_postgres } /* Publication of vitrivr engine query to Maven Central. */ diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index b30c22ff..7c00db8e 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -63,6 +63,7 @@ open class PgDescriptorInitializer(final override val field: Sch } } catch (e: SQLException) { LOGGER.error(e) { "Failed to initialize entity '$tableName' due to exception." } + throw e } /* Create indexes (optional). */ @@ -80,6 +81,7 @@ open class PgDescriptorInitializer(final override val field: Sch this.connection.jdbc.prepareStatement(/* sql = postgres */ indexStatement).use { it.execute() } } catch (e: SQLException) { LOGGER.error(e) { "Failed to create index ${index.type} for entity '$tableName' due to exception." } + throw e } } } From 0e8a64382d7c1571d8ab689a1a8e497ba4a2f768 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 11:32:15 +0200 Subject: [PATCH 29/71] Fixes two potential bugs with naming of schemas and search path. Signed-off-by: Ralph Gasser --- .../database/pgvector/PgVectorConnection.kt | 17 +++++++++++++---- 1 file changed, 13 insertions(+), 4 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt index 86984529..1fe1bed6 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt @@ -22,7 +22,7 @@ internal val LOGGER: KLogger = logger("org.vitrivr.engine.database.pgvector.PgVe * @author Ralph Gasser * @version 1.0.0 */ -class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: String, internal val jdbc: Connection): AbstractConnection(schemaName, provider) { +class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: String, val jdbc: Connection) : AbstractConnection(schemaName, provider) { init { /* Make sure that the pg_vector extension is installed. */ @@ -41,17 +41,26 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin /* Create necessary database. */ try { - this.jdbc.prepareStatement("CREATE DATABASE $schemaName;").use { + this.jdbc.prepareStatement("CREATE SCHEMA \"${schemaName}\";").use { it.execute() } } catch (e: SQLException) { if (e.sqlState == "42P04") { - LOGGER.info { "Database '$schemaName' already exists." } + LOGGER.info { "Schema '$schemaName' already exists." } } else { - LOGGER.error(e) { "Failed to create database '$schemaName' due to exception." } + LOGGER.error(e) { "Failed to create schema '$schemaName' due to exception." } throw e } } + + try { + this.jdbc.prepareStatement("SET search_path TO \"$schemaName\";").use { + it.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to set search path '$schemaName' due to exception." } + throw e + } } /** From ed8bb30fab873f846ae6f85fea300609e6606bf8 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 14:40:42 +0200 Subject: [PATCH 30/71] One can now specify database to use for a PgVectorConnection. Signed-off-by: Ralph Gasser --- .../pgvector/PgVectorConnectionProvider.kt | 11 +++++++++-- .../descriptor/PgDescriptorInitializer.kt | 16 +++++++++++++++- 2 files changed, 24 insertions(+), 3 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt index 78fd06e0..1ed12572 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnectionProvider.kt @@ -42,6 +42,12 @@ class PgVectorConnectionProvider: AbstractConnectionProvider() { /** Name of the host parameter. */ const val PARAMETER_DEFAULT_PORT = 5432 + /** Name of the port parameter. */ + const val PARAMETER_NAME_DATABASE = "database" + + /** Name of the port parameter. */ + const val PARAMETER_DEFAULT_DATABASE = "postgres" + /** Name of the host parameter. */ const val PARAMETER_NAME_USERNAME = "username" @@ -50,6 +56,7 @@ class PgVectorConnectionProvider: AbstractConnectionProvider() { /** Name of the host parameter. */ const val PARAMETER_NAME_SSL = "ssl" + } /** The name of this [PgVectorConnectionProvider]. */ @@ -94,11 +101,11 @@ class PgVectorConnectionProvider: AbstractConnectionProvider() { * @return [Connection] */ override fun openConnection(schemaName: String, parameters: Map): Connection { - /* Prepare connection URL. */ val host = parameters.getOrDefault(PARAMETER_NAME_HOST, PARAMETER_DEFAULT_HOST) val port = parameters[PARAMETER_NAME_PORT]?.toInt() ?: PARAMETER_DEFAULT_PORT - val url = "jdbc:postgresql://${host}:${port}/" + val database = parameters[PARAMETER_NAME_DATABASE]?.toInt() ?: PARAMETER_DEFAULT_DATABASE + val url = "jdbc:postgresql://${host}:${port}/${database}" /* Prepare properties (optional). */ val props = Properties() diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index 7c00db8e..4c88a538 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -27,7 +27,7 @@ open class PgDescriptorInitializer(final override val field: Sch * Initializes the PostgreSQL table entity backing this [PgDescriptorInitializer]. */ override fun initialize() { - val statement = StringBuilder("CREATE TABLE IF NOT EXISTS $tableName(") + val statement = StringBuilder("CREATE TABLE IF NOT EXISTS \"${tableName}\" (") statement.append("$DESCRIPTOR_ID_COLUMN_NAME uuid NOT NULL, ") statement.append("$RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, ") @@ -86,6 +86,20 @@ open class PgDescriptorInitializer(final override val field: Sch } } + /** + * De-initializes the PostgreSQL table entity backing this [PgDescriptorInitializer]. + */ + override fun deinitialize() { + try { + /* Create 'retrievable' entity and index. */ + this.connection.jdbc.prepareStatement(/* sql = postgres */ "DROP TABLE IF EXISTS \"${tableName}\";").use { + it.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to de-initialize entity '$tableName' due to exception." } + } + } + /** * Checks if the schema for this [PgDescriptorInitializer] has been properly initialized. * From dcf39298a4ccdeb5debb1e63b1e3d7acddaca7f0 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 14:50:43 +0200 Subject: [PATCH 31/71] Initializer now has a method deinitialize(). Signed-off-by: Ralph Gasser --- .../engine/core/database/Initializer.kt | 7 +++- .../descriptor/NoDescriptorInitializer.kt | 9 ++++-- .../retrievable/NoRetrievableInitializer.kt | 4 +++ .../CottontailDescriptorInitializer.kt | 15 ++++++++- .../retrievable/RetrievableInitializer.kt | 30 +++++++++++------ .../database/pgvector/PgVectorConnection.kt | 2 +- .../descriptor/PgDescriptorInitializer.kt | 2 +- .../retrievable/RetrievableInitializer.kt | 32 +++++++++++++++---- 8 files changed, 80 insertions(+), 21 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Initializer.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Initializer.kt index 553ebe8c..a31aa2f4 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Initializer.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Initializer.kt @@ -8,7 +8,7 @@ import org.vitrivr.engine.core.model.Persistable * * @author Ralph Gasser * @author Luca Rossetto - * @version 1.0.0 + * @version 1.1.0 */ interface Initializer { /** @@ -16,6 +16,11 @@ interface Initializer { */ fun initialize() + /** + * De-initializes the (persistent) data structures required by the [Persistable] of type [T]. + */ + fun deinitialize() + /** * Returns true if the structures created by this [Initializer] have already been created and setup. * diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/descriptor/NoDescriptorInitializer.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/descriptor/NoDescriptorInitializer.kt index 6ca7a468..773d1798 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/descriptor/NoDescriptorInitializer.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/descriptor/NoDescriptorInitializer.kt @@ -12,10 +12,15 @@ import org.vitrivr.engine.core.model.metamodel.Schema * @version 1.0.0 */ class NoDescriptorInitializer(override val field: Schema.Field<*,D>) : DescriptorInitializer { - override fun initialize() { /* No op. */ + override fun initialize() { + /* No op. */ } + override fun deinitialize() { + /* No op. */ + } override fun isInitialized(): Boolean = false - override fun truncate() { /* No op. */ + override fun truncate() { + /* No op. */ } } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/retrievable/NoRetrievableInitializer.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/retrievable/NoRetrievableInitializer.kt index dafae4a6..f40c7616 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/retrievable/NoRetrievableInitializer.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/retrievable/NoRetrievableInitializer.kt @@ -12,6 +12,10 @@ class NoRetrievableInitializer : RetrievableInitializer { override fun initialize() { /* No op. */ } + override fun deinitialize() { + TODO("Not yet implemented") + } + override fun isInitialized(): Boolean = false override fun truncate() { /* No op. */ } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt index 66ce7540..e5858746 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt @@ -2,6 +2,7 @@ package org.vitrivr.engine.plugin.cottontaildb.descriptors import io.grpc.StatusRuntimeException import org.vitrivr.cottontail.client.language.ddl.CreateEntity +import org.vitrivr.cottontail.client.language.ddl.DropEntity import org.vitrivr.cottontail.client.language.ddl.ListEntities import org.vitrivr.cottontail.client.language.ddl.TruncateEntity import org.vitrivr.cottontail.core.database.Name @@ -39,7 +40,19 @@ open class CottontailDescriptorInitializer(final override val fi /* Try to create entity. */ this.connection.client.create(create) } catch (e: StatusRuntimeException) { - LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + LOGGER.error(e) { "Failed to initialize entity '${this.entityName}' due to exception." } + } + } + + /** + * Initializes the Cottontail DB entity backing this [CottontailDescriptorInitializer]. + */ + override fun deinitialize() { + try { + /* Try to drop entity. */ + this.connection.client.drop(DropEntity(this.entityName)) + } catch (e: StatusRuntimeException) { + LOGGER.error(e) { "Failed to de-initialize entity '${this.entityName}' due to exception." } } } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableInitializer.kt index 71e3f3ac..71b73260 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableInitializer.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableInitializer.kt @@ -1,7 +1,5 @@ package org.vitrivr.engine.plugin.cottontaildb.retrievable -import io.github.oshai.kotlinlogging.KLogger -import io.github.oshai.kotlinlogging.KotlinLogging import io.grpc.StatusRuntimeException import org.vitrivr.cottontail.client.language.ddl.* import org.vitrivr.cottontail.core.database.Name @@ -11,9 +9,6 @@ import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer import org.vitrivr.engine.core.model.retrievable.Retrievable import org.vitrivr.engine.plugin.cottontaildb.* -/** Defines [KLogger] of the class. */ -private val logger: KLogger = KotlinLogging.logger {} - /** * A [RetrievableInitializer] implementation for Cottontail DB. * @@ -37,7 +32,7 @@ internal class RetrievableInitializer(private val connection: CottontailConnecti try { this.connection.client.create(createSchema) } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } } try { @@ -55,7 +50,7 @@ internal class RetrievableInitializer(private val connection: CottontailConnecti .column(this.entityName.column(RETRIEVABLE_TYPE_COLUMN_NAME)) ) } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } } try { @@ -82,7 +77,24 @@ internal class RetrievableInitializer(private val connection: CottontailConnecti .column(this.relationshipEntityName.column(PREDICATE_COLUMN_NAME)) ) } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + } + } + + /** + * De-initializes the entities that is used to store [Retrievable]s in Cottontail DB. + */ + override fun deinitialize() { + try { + this.connection.client.drop(DropEntity(this.entityName)) + } catch (e: StatusRuntimeException) { + LOGGER.error(e) { "Failed to initialize entity ${this.entityName} due to exception." } + } + + try { + this.connection.client.drop(DropEntity(this.relationshipEntityName)) + } catch (e: StatusRuntimeException) { + LOGGER.error(e) { "Failed to initialize entity '${this.entityName}' due to exception." } } } @@ -107,7 +119,7 @@ internal class RetrievableInitializer(private val connection: CottontailConnecti try { this.connection.client.truncate(truncate) } catch (e: StatusRuntimeException) { - logger.error(e) { "Failed to truncate entity ${this.entityName} due to exception." } + LOGGER.error(e) { "Failed to truncate entity '${this.entityName}' due to exception." } } } } \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt index 1fe1bed6..4697150e 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt @@ -45,7 +45,7 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin it.execute() } } catch (e: SQLException) { - if (e.sqlState == "42P04") { + if (e.sqlState == "42P06") { LOGGER.info { "Schema '$schemaName' already exists." } } else { LOGGER.error(e) { "Failed to create schema '$schemaName' due to exception." } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index 4c88a538..45892f00 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -92,7 +92,7 @@ open class PgDescriptorInitializer(final override val field: Sch override fun deinitialize() { try { /* Create 'retrievable' entity and index. */ - this.connection.jdbc.prepareStatement(/* sql = postgres */ "DROP TABLE IF EXISTS \"${tableName}\";").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "DROP TABLE IF EXISTS \"${tableName}\" CASCADE;").use { it.execute() } } catch (e: SQLException) { diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt index 9228b042..cab2c530 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt @@ -13,17 +13,37 @@ import java.sql.SQLException */ internal class RetrievableInitializer(private val connection: PgVectorConnection): RetrievableInitializer { /** - * Initializes the [RetrievableInitializer]. + * Initializes the data structures backing this [RetrievableInitializer]. */ override fun initialize() { try { /* Create 'retrievable' entity and index. */ - this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, type VARCHAR(100), PRIMARY KEY ($RETRIEVABLE_ID_COLUMN_NAME));").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS \"${RETRIEVABLE_ENTITY_NAME}\" ($RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, type VARCHAR(100), PRIMARY KEY ($RETRIEVABLE_ID_COLUMN_NAME));").use { it.execute() } /* Create 'relationship' entity. */ - this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS $RELATIONSHIP_ENTITY_NAME ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS \"${RELATIONSHIP_ENTITY_NAME}\" ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + .use { + it.execute() + } + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to initialize entity due to exception." } + } + } + + /** + * De-initializes the data structures backing this [RetrievableInitializer]. + */ + override fun deinitialize() { + try { + /* Create 'retrievable' entity and index. */ + this.connection.jdbc.prepareStatement(/* sql = postgres */ "DROP TABLE IF EXISTS \"${RETRIEVABLE_ENTITY_NAME}\" CASCADE;").use { + it.execute() + } + + /* Create 'relationship' entity. */ + this.connection.jdbc.prepareStatement(/* sql = postgres */ "DROP TABLE IF EXISTS \"${RELATIONSHIP_ENTITY_NAME}\" CASCADE;").use { it.execute() } } catch (e: SQLException) { @@ -38,14 +58,14 @@ internal class RetrievableInitializer(private val connection: PgVectorConnection */ override fun isInitialized(): Boolean { try { - this.connection.jdbc.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $RETRIEVABLE_ENTITY_NAME").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM \"${RETRIEVABLE_ENTITY_NAME}\";").use { it.execute() } } catch (e: SQLException) { return false } try { - this.connection.jdbc.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM $RELATIONSHIP_ENTITY_NAME").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "SELECT count(*) FROM \"$RELATIONSHIP_ENTITY_NAME\";").use { it.execute() } } catch (e: SQLException) { @@ -59,7 +79,7 @@ internal class RetrievableInitializer(private val connection: PgVectorConnection */ override fun truncate() { try { - this.connection.jdbc.prepareStatement(/* sql = postgres */ "TRUNCATE $RETRIEVABLE_ENTITY_NAME, $RELATIONSHIP_ENTITY_NAME").use { + this.connection.jdbc.prepareStatement(/* sql = postgres */ "TRUNCATE \"${RETRIEVABLE_ENTITY_NAME}\", \"${RELATIONSHIP_ENTITY_NAME}\"").use { it.execute() } } catch (e: SQLException) { From 247a6281360627fd3bb9a536beba1fb956bcd434 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 14:51:16 +0200 Subject: [PATCH 32/71] The AverageColor analyser is now part of the core module and serves as and a simple example and base facility to build test cases. Signed-off-by: Ralph Gasser --- .../engine/core/features}/averagecolor/AverageColor.kt | 5 ++++- .../core/features}/averagecolor/AverageColorExtractor.kt | 2 +- .../core/features}/averagecolor/AverageColorRetriever.kt | 2 +- .../org.vitrivr.engine.core.model.metamodel.Analyser | 1 + .../org.vitrivr.engine.core.model.metamodel.Analyser | 1 - 5 files changed, 7 insertions(+), 4 deletions(-) rename {vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature => vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features}/averagecolor/AverageColor.kt (96%) rename {vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature => vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features}/averagecolor/AverageColorExtractor.kt (96%) rename {vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature => vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features}/averagecolor/AverageColorRetriever.kt (96%) diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColor.kt similarity index 96% rename from vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt rename to vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColor.kt index 4f9f7329..7af3f02d 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColor.kt @@ -1,4 +1,4 @@ -package org.vitrivr.engine.module.features.feature.averagecolor +package org.vitrivr.engine.core.features.averagecolor import io.github.oshai.kotlinlogging.KLogger import io.github.oshai.kotlinlogging.KotlinLogging @@ -25,6 +25,9 @@ import java.util.* /** * Implementation of the [AverageColor] [Analyser], which derives the average color from an [ImageContent] as [FloatVectorDescriptor]. * + * This [Analyser] has little practical relevance these days but acts as a simple example for how to create a custom [Analyser] that uses vectors. + * Furthermore, it can be used to implement test cases. + * * @author Ralph Gasser * @version 1.0.0 */ diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorExtractor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorExtractor.kt similarity index 96% rename from vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorExtractor.kt rename to vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorExtractor.kt index 4bf0afcc..1d6f663f 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorExtractor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorExtractor.kt @@ -1,4 +1,4 @@ -package org.vitrivr.engine.module.features.feature.averagecolor +package org.vitrivr.engine.core.features.averagecolor import org.vitrivr.engine.core.features.AbstractExtractor import org.vitrivr.engine.core.features.metadata.source.file.FileSourceMetadataExtractor diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorRetriever.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorRetriever.kt similarity index 96% rename from vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorRetriever.kt rename to vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorRetriever.kt index 551fd034..02097f2b 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorRetriever.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorRetriever.kt @@ -1,4 +1,4 @@ -package org.vitrivr.engine.module.features.feature.averagecolor +package org.vitrivr.engine.core.features.averagecolor import io.github.oshai.kotlinlogging.KLogger import io.github.oshai.kotlinlogging.KotlinLogging diff --git a/vitrivr-engine-core/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser b/vitrivr-engine-core/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser index a8bdfd01..736805a4 100644 --- a/vitrivr-engine-core/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser +++ b/vitrivr-engine-core/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser @@ -1,3 +1,4 @@ +org.vitrivr.engine.core.features.averagecolor.AverageColor org.vitrivr.engine.core.features.metadata.source.file.FileSourceMetadata org.vitrivr.engine.core.features.metadata.source.exif.ExifMetadata org.vitrivr.engine.core.features.metadata.source.video.VideoSourceMetadata diff --git a/vitrivr-engine-module-features/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser b/vitrivr-engine-module-features/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser index bfbe95c4..a8c5e73b 100644 --- a/vitrivr-engine-module-features/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser +++ b/vitrivr-engine-module-features/src/main/resources/META-INF/services/org.vitrivr.engine.core.model.metamodel.Analyser @@ -1,4 +1,3 @@ -org.vitrivr.engine.module.features.feature.averagecolor.AverageColor org.vitrivr.engine.module.features.feature.external.implementations.dino.DINO org.vitrivr.engine.module.features.feature.external.implementations.clip.CLIP org.vitrivr.engine.module.features.feature.migration.WhisperASR From a74dc3a5a6bd5a18d20c50aa29d45d264b57d7f7 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 14:51:24 +0200 Subject: [PATCH 33/71] Fixes compilation errors. Signed-off-by: Ralph Gasser --- .../org/vitrivr/engine/core/database/MockConnection.kt | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/vitrivr-engine-core/src/test/kotlin/org/vitrivr/engine/core/database/MockConnection.kt b/vitrivr-engine-core/src/test/kotlin/org/vitrivr/engine/core/database/MockConnection.kt index b3b37405..6d682b49 100755 --- a/vitrivr-engine-core/src/test/kotlin/org/vitrivr/engine/core/database/MockConnection.kt +++ b/vitrivr-engine-core/src/test/kotlin/org/vitrivr/engine/core/database/MockConnection.kt @@ -9,12 +9,14 @@ import org.vitrivr.engine.core.database.retrievable.RetrievableWriter import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.metamodel.Schema -class MockConnection( - schemaName: String, - provider: ConnectionProvider = MockConnectionProvider(schemaName) -) : AbstractConnection(schemaName, provider) { +class MockConnection(schemaName: String, provider: ConnectionProvider = MockConnectionProvider(schemaName)) : AbstractConnection(schemaName, provider) { override val provider: ConnectionProvider get() = TODO("Not yet implemented") + + override fun withTransaction(action: (Unit) -> T): T { + TODO("Not yet implemented") + } + override val schemaName: String get() = TODO("Not yet implemented") From 7ff7d5f73c8d9470c2f52b4e823662a6c6a34b0c Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 14:52:19 +0200 Subject: [PATCH 34/71] Adds basic CI facilities to build Unit Tests: - GitHub Action for pushes / PRs to dev and main - Service containers for PostgreSQL and Cottontail DB - Scaffold for test case on database layer (to be extended). Signed-off-by: Ralph Gasser --- .github/workflows/ci.yml | 50 ++++++++++++++++++ build.gradle | 3 +- vitrivr-engine-core/build.gradle | 4 ++ .../engine/core/config/schema/SchemaConfig.kt | 23 ++++++++- .../core/database/AbstractDatabaseTest.kt | 31 +++++++++++ .../AbstractRetrievableInitializerTest.kt | 36 +++++++++++++ vitrivr-engine-module-pgvector/build.gradle | 3 ++ .../pgvector/RetrievableInitializerTest.kt | 51 +++++++++++++++++++ .../test/resources/test-schema-postgres.json | 31 +++++++++++ 9 files changed, 230 insertions(+), 2 deletions(-) create mode 100644 .github/workflows/ci.yml create mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/AbstractDatabaseTest.kt create mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt create mode 100644 vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt create mode 100644 vitrivr-engine-module-pgvector/src/test/resources/test-schema-postgres.json diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml new file mode 100644 index 00000000..cb18e2a1 --- /dev/null +++ b/.github/workflows/ci.yml @@ -0,0 +1,50 @@ +name: Java CI with Gradle + +on: + push: + branches: [ main, dev ] + pull_request: + branches: [ main, dev ] + +jobs: + build: + runs-on: ubuntu-latest + + # Setup Cottontail DB and PostgreSQL service container + services: + cottontail: + image: vitrivr/cottontaildb:0.16.6 + ports: + - 1865:1865 + options: -it + pgvector: + image: pgvector/pgvector:pg16 + ports: + - 5432:5432 + env: + POSTGRES_PASSWORD: vitrivr + + # Start actual job. + steps: + - uses: actions/checkout@v2 + - name: Cottontail DB connection test + run: nc -zv 127.0.0.1 1865 + - name: PostgreSQL connection test + run: nc -zv 127.0.0.1 5432 + - name: Set up JDK + uses: actions/setup-java@v4 + with: + java-version: '21' + distribution: 'temurin' + - name: Cache Gradle packages + uses: actions/cache@v4 + with: + path: ~/.gradle/caches + key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle') }} + restore-keys: ${{ runner.os }}-gradle + - name: Test with gradle ubuntu + if: matrix.os == 'ubuntu-latest' + run: ./gradlew test --info + - name: Test with gradle windows + if: matrix.os == 'windows-latest' + run: ./gradlew test --info \ No newline at end of file diff --git a/build.gradle b/build.gradle index 1e168a18..e1d10211 100644 --- a/build.gradle +++ b/build.gradle @@ -47,9 +47,10 @@ subprojects { } /* Common plugins. */ - apply plugin: 'java' apply plugin: 'idea' apply plugin: 'kotlin' + apply plugin: 'java' + apply plugin: 'java-test-fixtures' /* Java Version */ sourceCompatibility = JavaVersion.VERSION_21 diff --git a/vitrivr-engine-core/build.gradle b/vitrivr-engine-core/build.gradle index 40db6ce5..569a4813 100755 --- a/vitrivr-engine-core/build.gradle +++ b/vitrivr-engine-core/build.gradle @@ -7,10 +7,14 @@ plugins { dependencies { /** JOML dependencies for 3D mesh support. */ implementation group: 'org.joml', name: 'joml', version: version_joml + /** dependencies for exif metadata extraction. */ implementation 'com.drewnoakes:metadata-extractor:2.19.0' implementation 'com.google.code.gson:gson:2.8.9' implementation group: 'io.javalin.community.openapi', name: 'javalin-openapi-plugin', version: version_javalinopenapi + + /* Test Fixtures from Cottontail DB core. .*/ + testFixturesImplementation group: 'org.junit.jupiter', name: 'junit-jupiter-api', version: version_junit } /* Publication of vitrivr engine core to Maven Central. */ diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/SchemaConfig.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/SchemaConfig.kt index 77a42d55..de075a6f 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/SchemaConfig.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/SchemaConfig.kt @@ -1,9 +1,12 @@ package org.vitrivr.engine.core.config.schema import kotlinx.serialization.Serializable +import kotlinx.serialization.json.Json import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.operators.general.Exporter import org.vitrivr.engine.core.resolver.Resolver +import java.nio.file.Files +import java.nio.file.Paths /** @@ -43,4 +46,22 @@ data class SchemaConfig( * List of [PipelineConfig]s that are part of this [SchemaConfig]. */ val extractionPipelines: List = emptyList() -) +) { + + companion object { + + /** + * Tries to load a [SchemaConfig] from the resources. + * + * @param resourcePath Path to the resource. + * @return [SchemaConfig] + */ + fun loadFromResource(resourcePath: String): SchemaConfig { + val json = Json { ignoreUnknownKeys = true } // Configure Json to ignore unknown keys + val uri = this::class.java.classLoader.resources("test-schema-postgres.json").findFirst().orElseThrow { IllegalArgumentException("Resource '$resourcePath' not found!") }.toURI() + val path = Paths.get(uri) + val jsonString = Files.readString(path) + return json.decodeFromString(jsonString) + } + } +} diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/AbstractDatabaseTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/AbstractDatabaseTest.kt new file mode 100644 index 00000000..7d05373a --- /dev/null +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/AbstractDatabaseTest.kt @@ -0,0 +1,31 @@ +package org.vitrivr.engine.core.database + +import org.vitrivr.engine.core.config.schema.SchemaConfig +import org.vitrivr.engine.core.model.metamodel.SchemaManager + +/** + * Abstract base class for database tests. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractDatabaseTest(schemaPath: String) { + + companion object { + val SCHEMA_NAME = "vitrivr-test" + } + + + /** [SchemaManager] for this [AbstractDatabaseTest]. */ + protected val manager = SchemaManager() + + init { + /* Loads schema. */ + val schema = SchemaConfig.loadFromResource(schemaPath) + this.manager.load(schema) + } + + + /** */ + protected val schema = this.manager.getSchema("vitrivr-test") ?: throw IllegalArgumentException("Schema 'vitrivr-test' not found!") +} \ No newline at end of file diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt new file mode 100644 index 00000000..66275ba7 --- /dev/null +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt @@ -0,0 +1,36 @@ +package org.vitrivr.engine.core.database.retrievable + +import org.junit.jupiter.api.AfterEach +import org.vitrivr.engine.core.database.AbstractDatabaseTest + +/** + * An abstract set of test cases to test the proper functioning of [RetrievableInitializer] implementations. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractRetrievableInitializerTest(schemaPath: String) : AbstractDatabaseTest(schemaPath) { + /** + * Tests if the [RetrievableInitializer] can be initialized without throwing an exception. Furthermore, + * the test should check if the necessary tables have been created. + */ + abstract fun testIsInitializedWithoutInitialization() + + /** + * Tests if the [RetrievableInitializer] can be initialized without throwing an exception. Furthermore, + * the test should check if the necessary tables have been created. + */ + abstract fun testInitializeEntities() + + /** + * Cleans up the database after each test. + * + */ + @AfterEach + open fun cleanup() { + this.schema.connection.getRetrievableInitializer().deinitialize() + for (field in this.schema.fields()) { + field.getInitializer().deinitialize() + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/build.gradle b/vitrivr-engine-module-pgvector/build.gradle index 98d76110..44412dd3 100644 --- a/vitrivr-engine-module-pgvector/build.gradle +++ b/vitrivr-engine-module-pgvector/build.gradle @@ -12,6 +12,9 @@ dependencies { /** PostgreSQL driver. */ implementation group: 'org.postgresql', name: 'postgresql', version: version_jdbc_postgres + + /** Vitrivr engine Core is required for running tests. */ + testImplementation(testFixtures(project(':vitrivr-engine-core'))) } /* Publication of vitrivr engine query to Maven Central. */ diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt new file mode 100644 index 00000000..57808060 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt @@ -0,0 +1,51 @@ +package org.vitrivr.engine.database.pgvector + +import org.junit.jupiter.api.Assertions +import org.junit.jupiter.api.Test +import org.vitrivr.engine.core.database.retrievable.AbstractRetrievableInitializerTest + +/** + * An [AbstractRetrievableInitializerTest] for the [PgVectorConnection]. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class RetrievableInitializerTest : AbstractRetrievableInitializerTest("test-schema-postgres.json") { + + /** Internal [PgVectorConnection] object. */ + private val connection = this.schema.connection as? PgVectorConnection ?: throw IllegalArgumentException("Schema 'vitrivr-test' not found!") + + @Test + override fun testIsInitializedWithoutInitialization() { + Assertions.assertFalse(this.connection.getRetrievableInitializer().isInitialized()) + } + + @Test + override fun testInitializeEntities() { + /* Check initialization status (should be false). */ + Assertions.assertFalse(this.connection.getRetrievableInitializer().isInitialized()) + + /* Initialize basic tables. */ + this.connection.getRetrievableInitializer().initialize() + + /* Check initialization status (should be true). */ + Assertions.assertTrue(this.connection.getRetrievableInitializer().isInitialized()) + + /* Check for existence of tables. */ + this.connection.jdbc.prepareStatement("SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_schema = ? AND table_name = ?)").use { statement -> + /* Check existence of retrievable table. */ + statement.setString(1, SCHEMA_NAME) + statement.setString(2, RETRIEVABLE_ENTITY_NAME) + statement.executeQuery().use { result -> + Assertions.assertTrue(result.next() && result.getBoolean(1)) + } + + /* Check existence of relationship table. */ + statement.setString(1, SCHEMA_NAME) + statement.setString(2, RELATIONSHIP_ENTITY_NAME) + statement.executeQuery().use { result -> + Assertions.assertTrue(result.next() && result.getBoolean(1)) + } + } + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/test/resources/test-schema-postgres.json b/vitrivr-engine-module-pgvector/src/test/resources/test-schema-postgres.json new file mode 100644 index 00000000..f91f48e9 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/test/resources/test-schema-postgres.json @@ -0,0 +1,31 @@ +{ + "name": "vitrivr-test", + "connection": { + "database": "PgVectorConnectionProvider", + "parameters": { + "host": "127.0.0.1", + "port": "5432", + "username": "postgres", + "password": "vitrivr" + } + }, + "fields": [ + { + "name": "averagecolor", + "factory": "AverageColor" + }, + { + "name": "file", + "factory": "FileSourceMetadata" + }, + { + "name": "time", + "factory": "TemporalMetadata" + }, + { + "name": "video", + "factory": "VideoSourceMetadata" + } + ], + "exporters": [] +} From 5f2b4e06fbe9d4a951253f73b2189e40d839527f Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 14:59:19 +0200 Subject: [PATCH 35/71] Adds missing types. Signed-off-by: Ralph Gasser --- .../org/vitrivr/engine/core/model/types/Value.kt | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt index b84c8fdf..5bd1dbbb 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/types/Value.kt @@ -13,6 +13,12 @@ import java.util.* sealed interface Value { companion object { + /** + * Converts [Any] value to a [Value]. + * + * @param value The value to convert. + * @return [Value] + */ fun of(value: Any): Value<*> = when (value) { is kotlin.String -> String(value) is kotlin.Boolean -> Boolean(value) @@ -22,6 +28,11 @@ sealed interface Value { is kotlin.Long -> Long(value) is kotlin.Float -> Float(value) is kotlin.Double -> Double(value) + is BooleanArray -> BooleanVector(value) + is DoubleArray -> DoubleVector(value) + is FloatArray -> FloatVector(value) + is LongArray -> LongVector(value) + is IntArray -> IntVector(value) is Date -> DateTime(value) else -> throw IllegalArgumentException("Unsupported data type.") } From 23ceb65deb493ba2c8917bcbb36099befc7863ce Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 15:00:08 +0200 Subject: [PATCH 36/71] Updates actions/checkou to @v4 Signed-off-by: Ralph Gasser --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cb18e2a1..6a7583a4 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -26,7 +26,7 @@ jobs: # Start actual job. steps: - - uses: actions/checkout@v2 + - uses: actions/checkout@v4 - name: Cottontail DB connection test run: nc -zv 127.0.0.1 1865 - name: PostgreSQL connection test From f99ecdb131282082996e9b906f5a3bbab3da1571 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 15:09:08 +0200 Subject: [PATCH 37/71] Refactors tests slightly. Signed-off-by: Ralph Gasser --- .../core/database/AbstractDatabaseTest.kt | 9 ++-- .../AbstractRetrievableInitializerTest.kt | 53 +++++++++++++++++-- .../pgvector/RetrievableInitializerTest.kt | 39 +++++--------- 3 files changed, 68 insertions(+), 33 deletions(-) diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/AbstractDatabaseTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/AbstractDatabaseTest.kt index 7d05373a..2952a11a 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/AbstractDatabaseTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/AbstractDatabaseTest.kt @@ -1,6 +1,7 @@ package org.vitrivr.engine.core.database import org.vitrivr.engine.core.config.schema.SchemaConfig +import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.metamodel.SchemaManager /** @@ -10,12 +11,10 @@ import org.vitrivr.engine.core.model.metamodel.SchemaManager * @version 1.0.0 */ abstract class AbstractDatabaseTest(schemaPath: String) { - companion object { val SCHEMA_NAME = "vitrivr-test" } - /** [SchemaManager] for this [AbstractDatabaseTest]. */ protected val manager = SchemaManager() @@ -25,7 +24,9 @@ abstract class AbstractDatabaseTest(schemaPath: String) { this.manager.load(schema) } + /** The test [Schema]. */ + protected val testSchema: Schema = this.manager.getSchema("vitrivr-test") ?: throw IllegalArgumentException("Schema 'vitrivr-test' not found!") - /** */ - protected val schema = this.manager.getSchema("vitrivr-test") ?: throw IllegalArgumentException("Schema 'vitrivr-test' not found!") + /** The [Connection] object backing the [Schema]. */ + protected val testConnection = this.testSchema.connection } \ No newline at end of file diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt index 66275ba7..385420ed 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt @@ -1,6 +1,8 @@ package org.vitrivr.engine.core.database.retrievable import org.junit.jupiter.api.AfterEach +import org.junit.jupiter.api.Assertions +import org.junit.jupiter.api.Test import org.vitrivr.engine.core.database.AbstractDatabaseTest /** @@ -10,17 +12,60 @@ import org.vitrivr.engine.core.database.AbstractDatabaseTest * @version 1.0.0 */ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : AbstractDatabaseTest(schemaPath) { + /** + * Tests if the [RetrievableInitializer.isInitialized] works as expected., + */ + @Test + fun testIsInitializedWithoutInitialization() { + Assertions.assertFalse(this.testConnection.getRetrievableInitializer().isInitialized()) + } + /** * Tests if the [RetrievableInitializer] can be initialized without throwing an exception. Furthermore, * the test should check if the necessary tables have been created. */ - abstract fun testIsInitializedWithoutInitialization() + @Test + fun testInitializeEntities() { + /* Check initialization status (should be false). */ + Assertions.assertFalse(this.testConnection.getRetrievableInitializer().isInitialized()) + + /* Initialize basic tables. */ + this.testConnection.getRetrievableInitializer().initialize() + + /* Check initialization status (should be true). */ + Assertions.assertTrue(this.testConnection.getRetrievableInitializer().isInitialized()) + Assertions.assertTrue(this.checkTablesExist()) + } /** * Tests if the [RetrievableInitializer] can be initialized without throwing an exception. Furthermore, * the test should check if the necessary tables have been created. */ - abstract fun testInitializeEntities() + @Test + fun testDeInitializeEntities() { + /* Check initialization status (should be false). */ + Assertions.assertFalse(this.testConnection.getRetrievableInitializer().isInitialized()) + + /* Initialize basic tables. */ + this.testConnection.getRetrievableInitializer().initialize() + + /* Check initialization status (should be true). */ + Assertions.assertTrue(this.testConnection.getRetrievableInitializer().isInitialized()) + Assertions.assertTrue(this.checkTablesExist()) + + this.testConnection.getRetrievableInitializer().deinitialize() + Assertions.assertFalse(this.testConnection.getRetrievableInitializer().isInitialized()) + Assertions.assertFalse(this.checkTablesExist()) + } + + /** + * Checks if tables initialized actually exists. + * + * This check is database specific and therefore abstract. + * + * @return True if tables exist, false otherwise. + */ + protected abstract fun checkTablesExist(): Boolean /** * Cleans up the database after each test. @@ -28,8 +73,8 @@ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : Abstract */ @AfterEach open fun cleanup() { - this.schema.connection.getRetrievableInitializer().deinitialize() - for (field in this.schema.fields()) { + this.testSchema.connection.getRetrievableInitializer().deinitialize() + for (field in this.testSchema.fields()) { field.getInitializer().deinitialize() } } diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt index 57808060..229572a4 100644 --- a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt +++ b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt @@ -1,7 +1,5 @@ package org.vitrivr.engine.database.pgvector -import org.junit.jupiter.api.Assertions -import org.junit.jupiter.api.Test import org.vitrivr.engine.core.database.retrievable.AbstractRetrievableInitializerTest /** @@ -11,41 +9,32 @@ import org.vitrivr.engine.core.database.retrievable.AbstractRetrievableInitializ * @version 1.0.0 */ class RetrievableInitializerTest : AbstractRetrievableInitializerTest("test-schema-postgres.json") { - - /** Internal [PgVectorConnection] object. */ - private val connection = this.schema.connection as? PgVectorConnection ?: throw IllegalArgumentException("Schema 'vitrivr-test' not found!") - - @Test - override fun testIsInitializedWithoutInitialization() { - Assertions.assertFalse(this.connection.getRetrievableInitializer().isInitialized()) - } - - @Test - override fun testInitializeEntities() { - /* Check initialization status (should be false). */ - Assertions.assertFalse(this.connection.getRetrievableInitializer().isInitialized()) - - /* Initialize basic tables. */ - this.connection.getRetrievableInitializer().initialize() - - /* Check initialization status (should be true). */ - Assertions.assertTrue(this.connection.getRetrievableInitializer().isInitialized()) - + /** + * Checks if tables initialized actually exists. + * + * @return True if tables exist, false otherwise. + */ + override fun checkTablesExist(): Boolean { /* Check for existence of tables. */ - this.connection.jdbc.prepareStatement("SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_schema = ? AND table_name = ?)").use { statement -> + (this.testConnection as PgVectorConnection).jdbc.prepareStatement("SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_schema = ? AND table_name = ?)").use { statement -> /* Check existence of retrievable table. */ statement.setString(1, SCHEMA_NAME) statement.setString(2, RETRIEVABLE_ENTITY_NAME) statement.executeQuery().use { result -> - Assertions.assertTrue(result.next() && result.getBoolean(1)) + if (!result.next() || !result.getBoolean(1)) { + return false + } } /* Check existence of relationship table. */ statement.setString(1, SCHEMA_NAME) statement.setString(2, RELATIONSHIP_ENTITY_NAME) statement.executeQuery().use { result -> - Assertions.assertTrue(result.next() && result.getBoolean(1)) + if (!result.next() || !result.getBoolean(1)) { + return false + } } } + return true } } \ No newline at end of file From b80868440b4f3306bbe883fe14862409e5658a23 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 15:29:56 +0200 Subject: [PATCH 38/71] Adds more test cases for RetrievableWriter. Signed-off-by: Ralph Gasser --- .../AbstractRetrievableInitializerTest.kt | 12 ++- .../AbstractRetrievableWriterTest.kt | 100 ++++++++++++++++++ .../retrievable/RetrievableInitializer.kt | 2 +- .../pgvector/retrievable/RetrievableWriter.kt | 6 +- .../RetrievableInitializerTest.kt | 5 +- .../retrievable/RetrievableWriterTest.kt | 13 +++ 6 files changed, 130 insertions(+), 8 deletions(-) create mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt rename vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/{ => retrievable}/RetrievableInitializerTest.kt (85%) create mode 100644 vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriterTest.kt diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt index 385420ed..ac25b0f7 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt @@ -2,6 +2,7 @@ package org.vitrivr.engine.core.database.retrievable import org.junit.jupiter.api.AfterEach import org.junit.jupiter.api.Assertions +import org.junit.jupiter.api.BeforeEach import org.junit.jupiter.api.Test import org.vitrivr.engine.core.database.AbstractDatabaseTest @@ -67,6 +68,14 @@ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : Abstract */ protected abstract fun checkTablesExist(): Boolean + /** + * Cleans up the database after each test. + */ + @BeforeEach + open fun prepare() { + this.testSchema.connection.getRetrievableInitializer().initialize() + } + /** * Cleans up the database after each test. * @@ -74,8 +83,5 @@ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : Abstract @AfterEach open fun cleanup() { this.testSchema.connection.getRetrievableInitializer().deinitialize() - for (field in this.testSchema.fields()) { - field.getInitializer().deinitialize() - } } } \ No newline at end of file diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt new file mode 100644 index 00000000..ecc256d1 --- /dev/null +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt @@ -0,0 +1,100 @@ +package org.vitrivr.engine.core.database.retrievable + +import org.junit.jupiter.api.AfterEach +import org.junit.jupiter.api.Assertions +import org.junit.jupiter.api.BeforeEach +import org.junit.jupiter.api.Test +import org.vitrivr.engine.core.database.AbstractDatabaseTest +import org.vitrivr.engine.core.model.retrievable.Ingested +import java.util.* + +/** + * An abstract set of test cases to test the proper functioning of [RetrievableWriter] implementations. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractRetrievableWriterTest(schemaPath: String) : AbstractDatabaseTest(schemaPath) { + /** + * Tests if the [RetrievableWriter.add] works as expected. + */ + @Test + fun testAdd() { + val writer = this.testConnection.getRetrievableWriter() + val reader = this.testConnection.getRetrievableReader() + + /* Create and add retrievable. */ + val id = UUID.randomUUID() + Assertions.assertTrue(writer.add(Ingested(id, "INGESTED:TEST", false))) + + /* Check if retrievable can be read. */ + Assertions.assertEquals(1L, reader.count()) + Assertions.assertEquals("INGESTED:TEST", reader[id]?.type) + } + + /** + * Tests if the [RetrievableWriter.add] works as expected. + */ + @Test + fun testAddAll() { + val writer = this.testConnection.getRetrievableWriter() + val reader = this.testConnection.getRetrievableReader() + val size = Random().nextInt(500, 5000) + + /* Create and add retrievable. */ + val ids = (0 until size).map { UUID.randomUUID() } + val ingested = ids.map { Ingested(it, "INGESTED:TEST", false) } + Assertions.assertTrue(writer.addAll(ingested)) + + /* Check if retrievable can be read. */ + Assertions.assertEquals(ids.size.toLong(), reader.count()) + reader.getAll(ids).forEachIndexed { i, it -> + Assertions.assertEquals(ids[i], it.id) + Assertions.assertEquals("INGESTED:TEST", it.type) + } + } + + /** + * Tests if the [RetrievableWriter.add] works as expected. + */ + @Test + fun testDelete() { + val writer = this.testConnection.getRetrievableWriter() + val reader = this.testConnection.getRetrievableReader() + val size = Random().nextInt(500, 5000) + + /* Create and add retrievable. */ + val ids = (0 until size).map { UUID.randomUUID() } + val ingested = ids.map { Ingested(it, "INGESTED:TEST", false) } + val delete = ingested[Random().nextInt(0, ingested.size)] + + /* Execute actions. */ + Assertions.assertFalse(writer.delete(delete)) + Assertions.assertTrue(writer.addAll(ingested)) + Assertions.assertTrue(writer.delete(delete)) + + /* Check if retrievable can be read. */ + Assertions.assertEquals(ids.size.toLong() - 1L, reader.count()) + reader.getAll(ids).forEachIndexed() { i, it -> + Assertions.assertNotEquals(delete.id, it.id) + Assertions.assertEquals("INGESTED:TEST", it.type) + } + } + + /** + * Cleans up the database after each test. + */ + @BeforeEach + open fun prepare() { + this.testSchema.connection.getRetrievableInitializer().initialize() + } + + /** + * Cleans up the database after each test. + * + */ + @AfterEach + open fun cleanup() { + this.testSchema.connection.getRetrievableInitializer().deinitialize() + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt index cab2c530..afd8408f 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializer.kt @@ -23,7 +23,7 @@ internal class RetrievableInitializer(private val connection: PgVectorConnection } /* Create 'relationship' entity. */ - this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS \"${RELATIONSHIP_ENTITY_NAME}\" ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME), FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + this.connection.jdbc.prepareStatement(/* sql = postgres */ "CREATE TABLE IF NOT EXISTS \"${RELATIONSHIP_ENTITY_NAME}\" ($OBJECT_ID_COLUMN_NAME uuid NOT NULL, $PREDICATE_COLUMN_NAME VARCHAR(100) NOT NULL, $SUBJECT_ID_COLUMN_NAME uuid NOT NULL, PRIMARY KEY ($OBJECT_ID_COLUMN_NAME, $PREDICATE_COLUMN_NAME, $SUBJECT_ID_COLUMN_NAME), FOREIGN KEY($OBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME) ON DELETE CASCADE, FOREIGN KEY($SUBJECT_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME) ON DELETE CASCADE);") .use { it.execute() } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt index 4592097a..e3ba8fcf 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt @@ -81,7 +81,7 @@ internal class RetrievableWriter(override val connection: PgVectorConnection): R try { this.connection.jdbc.prepareStatement("DELETE FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?;").use { stmt -> stmt.setObject(1, item.id) - return stmt.execute() + return stmt.executeUpdate() > 0 } } catch (e: SQLException) { LOGGER.error(e) { "Failed to delete retrievable ${item.id} due to SQL error." } @@ -100,10 +100,10 @@ internal class RetrievableWriter(override val connection: PgVectorConnection): R this.connection.jdbc.prepareStatement("DELETE FROM $RETRIEVABLE_ENTITY_NAME WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?);").use { stmt -> val values = items.map { it.id }.toTypedArray() stmt.setArray(1, this.connection.jdbc.createArrayOf("uuid", values)) - return stmt.execute() + return stmt.executeUpdate() > 0 } } catch (e: SQLException) { - LOGGER.error(e) { "Failed to delete retrievables due to SQL error." } + LOGGER.error(e) { "Failed to delete retrievable due to SQL error." } return false } } diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializerTest.kt similarity index 85% rename from vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt rename to vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializerTest.kt index 229572a4..ce2780dd 100644 --- a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/RetrievableInitializerTest.kt +++ b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializerTest.kt @@ -1,6 +1,9 @@ -package org.vitrivr.engine.database.pgvector +package org.vitrivr.engine.database.pgvector.retrievable import org.vitrivr.engine.core.database.retrievable.AbstractRetrievableInitializerTest +import org.vitrivr.engine.database.pgvector.PgVectorConnection +import org.vitrivr.engine.database.pgvector.RELATIONSHIP_ENTITY_NAME +import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ENTITY_NAME /** * An [AbstractRetrievableInitializerTest] for the [PgVectorConnection]. diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriterTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriterTest.kt new file mode 100644 index 00000000..c6326f32 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriterTest.kt @@ -0,0 +1,13 @@ +package org.vitrivr.engine.database.pgvector.retrievable + +import org.vitrivr.engine.core.database.retrievable.AbstractRetrievableWriterTest + +/** + * An [AbstractRetrievableWriterTest] for the [PgVectorConnection]. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class RetrievableWriterTest : AbstractRetrievableWriterTest("test-schema-postgres.json") { + +} \ No newline at end of file From 07a8181a8d3d4731b8d6d2db38782d2936d3ffda Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 15:30:11 +0200 Subject: [PATCH 39/71] Fixes few edge case issues. Signed-off-by: Ralph Gasser --- .../engine/database/pgvector/Utilities.kt | 24 ++++++++++++++ .../descriptor/PgDescriptorInitializer.kt | 2 +- .../pgvector/descriptor/PgDescriptorWriter.kt | 33 ++++--------------- 3 files changed, 31 insertions(+), 28 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Utilities.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Utilities.kt index 92ed71c7..4c620286 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Utilities.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/Utilities.kt @@ -3,11 +3,14 @@ package org.vitrivr.engine.database.pgvector import org.vitrivr.engine.core.model.query.basics.ComparisonOperator import org.vitrivr.engine.core.model.query.basics.Distance import org.vitrivr.engine.core.model.query.basics.Distance.* +import org.vitrivr.engine.core.model.types.Type import org.vitrivr.engine.core.model.types.Value import org.vitrivr.engine.database.pgvector.descriptor.model.PgBitVector import org.vitrivr.engine.database.pgvector.descriptor.model.PgVector import java.sql.Date +import java.sql.JDBCType import java.sql.PreparedStatement +import java.sql.SQLType /** * Sets a value of [Value] type in a [PreparedStatement]. @@ -33,6 +36,27 @@ internal fun PreparedStatement.setValue(index: Int, value: Value<*>) = when (val else -> throw IllegalArgumentException("Unsupported value type for vector value.") } +/** + * Converts a [Type] to a [SQLType]. + */ +internal fun Type.toSql(): Int = when (this) { + Type.Boolean -> JDBCType.BOOLEAN + Type.Byte -> JDBCType.TINYINT + Type.Short -> JDBCType.SMALLINT + Type.Int -> JDBCType.INTEGER + Type.Long -> JDBCType.BIGINT + Type.Float -> JDBCType.REAL + Type.Double -> JDBCType.DOUBLE + Type.Datetime -> JDBCType.DATE + Type.String -> JDBCType.VARCHAR + Type.Text -> JDBCType.CLOB + is Type.BooleanVector -> JDBCType.ARRAY + is Type.DoubleVector -> JDBCType.ARRAY + is Type.FloatVector -> JDBCType.ARRAY + is Type.IntVector -> JDBCType.ARRAY + is Type.LongVector -> JDBCType.ARRAY +}.ordinal + /** * Converts a [Distance] to a pgVector distance operator. */ diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index 45892f00..1d23f1c5 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -54,7 +54,7 @@ open class PgDescriptorInitializer(final override val field: Sch /* Finalize statement*/ statement.append("PRIMARY KEY ($DESCRIPTOR_ID_COLUMN_NAME), ") - statement.append("FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME));") + statement.append("FOREIGN KEY ($RETRIEVABLE_ID_COLUMN_NAME) REFERENCES $RETRIEVABLE_ENTITY_NAME($RETRIEVABLE_ID_COLUMN_NAME) ON DELETE CASCADE);") /* Create entity. */ try { diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt index ee05d5cd..844cc62c 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt @@ -4,7 +4,6 @@ import org.vitrivr.engine.core.database.descriptor.DescriptorWriter import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema -import org.vitrivr.engine.core.model.types.Type import org.vitrivr.engine.database.pgvector.* import java.sql.* @@ -38,10 +37,10 @@ open class PgDescriptorWriter(final override val field: Schema.F if (value != null) { stmt.setValue(i++, value) } else { - stmt.setNull(i++, attribute.type.toSqlType()) + stmt.setNull(i++, attribute.type.toSql()) } } - return stmt.execute() + return stmt.executeUpdate() == 1 } } catch (e: SQLException) { LOGGER.error(e) { "Failed to INSERT descriptor ${item.id} into '$tableName' due to SQL error." } @@ -67,7 +66,7 @@ open class PgDescriptorWriter(final override val field: Schema.F if (value != null) { stmt.setValue(i++, value) } else { - stmt.setNull(i++, attribute.type.toSqlType()) + stmt.setNull(i++, attribute.type.toSql()) } } stmt.addBatch() @@ -96,7 +95,7 @@ open class PgDescriptorWriter(final override val field: Schema.F if (value != null) { stmt.setValue(i++, value) } else { - stmt.setNull(i++, attribute.type.toSqlType()) + stmt.setNull(i++, attribute.type.toSql()) } } stmt.setObject(i, item.id) @@ -118,7 +117,7 @@ open class PgDescriptorWriter(final override val field: Schema.F try { this.connection.jdbc.prepareStatement("DELETE FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?;").use { stmt -> stmt.setObject(1, item.id) - return stmt.execute() + return stmt.executeUpdate() == 1 } } catch (e: SQLException) { LOGGER.error(e) { "Failed to delete descriptor ${item.id} due to SQL error." } @@ -137,7 +136,7 @@ open class PgDescriptorWriter(final override val field: Schema.F this.connection.jdbc.prepareStatement("DELETE FROM $tableName WHERE $DESCRIPTOR_ID_COLUMN_NAME = ANY (?);").use { stmt -> val values = items.map { it.id }.toTypedArray() stmt.setArray(1, this.connection.jdbc.createArrayOf("uuid", values)) - return stmt.execute() + return stmt.executeUpdate() > 0 } } catch (e: SQLException) { LOGGER.error(e) { "Failed to delete descriptors due to SQL error." } @@ -145,26 +144,6 @@ open class PgDescriptorWriter(final override val field: Schema.F } } - /** - * Converts a [Type] to a [SQLType]. - */ - protected fun Type.toSqlType(): Int = when (this) { - Type.Boolean -> JDBCType.BOOLEAN - Type.Byte -> JDBCType.TINYINT - Type.Short -> JDBCType.SMALLINT - Type.Int -> JDBCType.INTEGER - Type.Long -> JDBCType.BIGINT - Type.Float -> JDBCType.REAL - Type.Double -> JDBCType.DOUBLE - Type.Datetime -> JDBCType.DATE - Type.String -> JDBCType.VARCHAR - Type.Text -> JDBCType.CLOB - is Type.BooleanVector -> JDBCType.ARRAY - is Type.DoubleVector -> JDBCType.ARRAY - is Type.FloatVector -> JDBCType.ARRAY - is Type.IntVector -> JDBCType.ARRAY - is Type.LongVector -> JDBCType.ARRAY - }.ordinal /** * Prepares an INSERT statement for this [StructDescriptorWriter]. From a792898d7302d0c16a1eb674a10b1a7825d23b5c Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 15:31:57 +0200 Subject: [PATCH 40/71] Bugfix. Signed-off-by: Ralph Gasser --- .../retrievable/AbstractRetrievableInitializerTest.kt | 9 --------- 1 file changed, 9 deletions(-) diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt index ac25b0f7..a71cf49c 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt @@ -2,7 +2,6 @@ package org.vitrivr.engine.core.database.retrievable import org.junit.jupiter.api.AfterEach import org.junit.jupiter.api.Assertions -import org.junit.jupiter.api.BeforeEach import org.junit.jupiter.api.Test import org.vitrivr.engine.core.database.AbstractDatabaseTest @@ -68,14 +67,6 @@ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : Abstract */ protected abstract fun checkTablesExist(): Boolean - /** - * Cleans up the database after each test. - */ - @BeforeEach - open fun prepare() { - this.testSchema.connection.getRetrievableInitializer().initialize() - } - /** * Cleans up the database after each test. * From e8d026bb8831aa2d464f39943e9e54a250515da0 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 15:33:37 +0200 Subject: [PATCH 41/71] And another bugfix. Signed-off-by: Ralph Gasser --- .../engine/database/pgvector/retrievable/RetrievableWriter.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt index e3ba8fcf..33c7f8cf 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt @@ -23,7 +23,7 @@ internal class RetrievableWriter(override val connection: PgVectorConnection): R this.connection.jdbc.prepareStatement("INSERT INTO $RETRIEVABLE_ENTITY_NAME ($RETRIEVABLE_ID_COLUMN_NAME, $RETRIEVABLE_TYPE_COLUMN_NAME) VALUES (?, ?);").use { stmt -> stmt.setObject(1, item.id) stmt.setString(2, item.type) - return stmt.execute() + return stmt.executeUpdate() == 1 } } catch (e: SQLException) { LOGGER.error(e) { "Failed to persist retrievable ${item.id} due to SQL error." } From 3d83f88fc35d312c89943131736310e085e6be30 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 16:09:09 +0200 Subject: [PATCH 42/71] Adds remaining base operations to AbstractRetrievableWriterTest. Signed-off-by: Ralph Gasser --- .../AbstractRetrievableWriterTest.kt | 54 +++++++++++++++++++ .../pgvector/retrievable/RetrievableWriter.kt | 2 +- 2 files changed, 55 insertions(+), 1 deletion(-) diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt index ecc256d1..4701ceeb 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt @@ -81,6 +81,60 @@ abstract class AbstractRetrievableWriterTest(schemaPath: String) : AbstractDatab } } + /** + * Tests if the [RetrievableWriter.add] works as expected. + */ + @Test + fun testDeleteAll() { + val writer = this.testConnection.getRetrievableWriter() + val reader = this.testConnection.getRetrievableReader() + val size = Random().nextInt(500, 5000) + + /* Create and add retrievable. */ + val ids = (0 until size).map { UUID.randomUUID() } + val ingested = ids.map { Ingested(it, "INGESTED:TEST", false) } + + /* Execute actions. */ + Assertions.assertTrue(writer.addAll(ingested)) + + /* Check if retrievable can be read. */ + Assertions.assertEquals(ids.size.toLong(), reader.count()) + reader.getAll(ids).forEachIndexed() { i, it -> + Assertions.assertEquals("INGESTED:TEST", it.type) + } + + /* Execute actions. */ + Assertions.assertTrue(writer.deleteAll(ingested)) + Assertions.assertEquals(0L, reader.count()) + } + + + /** + * Tests if the [RetrievableWriter.add] works as expected. + */ + @Test + fun testUpdate() { + val writer = this.testConnection.getRetrievableWriter() + val reader = this.testConnection.getRetrievableReader() + val size = Random().nextInt(500, 5000) + + /* Create and add retrievable. */ + val ids = (0 until size).map { UUID.randomUUID() } + val ingested = ids.map { Ingested(it, "INGESTED:TEST", false) } + val update = ingested[Random().nextInt(0, ingested.size)] + + /* Execute actions. */ + Assertions.assertTrue(writer.addAll(ingested)) + + /* Check if retrievable can be read. */ + Assertions.assertEquals(ids.size.toLong(), reader.count()) + Assertions.assertTrue(writer.update(update.copy(type = "INGESTED:TEST2"))) + + /* Execute actions. */ + Assertions.assertEquals(ids.size.toLong(), reader.count()) + Assertions.assertEquals("INGESTED:TEST2", reader[update.id]?.type) + } + /** * Cleans up the database after each test. */ diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt index 33c7f8cf..5de14b54 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableWriter.kt @@ -63,7 +63,7 @@ internal class RetrievableWriter(override val connection: PgVectorConnection): R this.connection.jdbc.prepareStatement("UPDATE $RETRIEVABLE_ENTITY_NAME SET $RETRIEVABLE_TYPE_COLUMN_NAME = ? WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> stmt.setString(1, item.type) stmt.setObject(2, item.id) - return stmt.execute() + return stmt.executeUpdate() == 1 } } catch (e: SQLException) { LOGGER.error(e) { "Failed to update retrievable ${item.id} due to SQL error." } From e9b5551514e742831d7f8bc46c9c6e097f6c119e Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 16:37:09 +0200 Subject: [PATCH 43/71] Fixes another minor PostgreSQL glitch. Signed-off-by: Ralph Gasser --- .../engine/database/pgvector/PgVectorConnection.kt | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt index 4697150e..1b256dc4 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/PgVectorConnection.kt @@ -35,11 +35,7 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin throw e } - /* Register the vector data type. */ - this.jdbc.unwrap(PGConnection::class.java).addDataType("vector", PgVector::class.java) - this.jdbc.unwrap(PGConnection::class.java).addDataType("bit", PgBitVector::class.java) - - /* Create necessary database. */ + /* Create necessary schema. */ try { this.jdbc.prepareStatement("CREATE SCHEMA \"${schemaName}\";").use { it.execute() @@ -54,13 +50,17 @@ class PgVectorConnection(provider: PgVectorConnectionProvider, schemaName: Strin } try { - this.jdbc.prepareStatement("SET search_path TO \"$schemaName\";").use { + this.jdbc.prepareStatement("SET search_path TO \"$schemaName\", public;").use { it.execute() } } catch (e: SQLException) { LOGGER.error(e) { "Failed to set search path '$schemaName' due to exception." } throw e } + + /* Register the vector data type. */ + this.jdbc.unwrap(PGConnection::class.java).addDataType("vector", PgVector::class.java) + this.jdbc.unwrap(PGConnection::class.java).addDataType("bit", PgBitVector::class.java) } /** From 1f0fc477e5aae34e50af8ad2ff8d2146a16e64af Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Thu, 11 Jul 2024 16:37:29 +0200 Subject: [PATCH 44/71] Adds unit tests for DescriptorInitializer. Signed-off-by: Ralph Gasser --- .../AbstractDescriptorInitializerTest.kt | 70 +++++++++++++++++++ ...AbstractVectorDescriptorInitializerTest.kt | 14 ++++ .../vector/VectorDescriptorInitializerTest.kt | 10 +++ 3 files changed, 94 insertions(+) create mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt create mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractVectorDescriptorInitializerTest.kt create mode 100644 vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializerTest.kt diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt new file mode 100644 index 00000000..886443b5 --- /dev/null +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt @@ -0,0 +1,70 @@ +package org.vitrivr.engine.core.database.descriptor + +import org.junit.jupiter.api.AfterEach +import org.junit.jupiter.api.Assertions +import org.junit.jupiter.api.Test +import org.vitrivr.engine.core.database.AbstractDatabaseTest +import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer +import org.vitrivr.engine.core.model.metamodel.Schema + +/** + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractDescriptorInitializerTest(schemaPath: String) : AbstractDatabaseTest(schemaPath) { + /** The [Schema.Field] used for this [AbstractVectorDescriptorInitializerTest]. */ + protected abstract val field: Schema.Field<*, *> + + /** + * Tests if the [DescriptorInitializer.isInitialized] works as expected., + */ + @Test + fun testIsInitializedWithoutInitialization() { + Assertions.assertFalse(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + } + + /** + * Tests if the [RetrievableInitializer] can be initialized without throwing an exception. Furthermore, + * the test should check if the necessary tables have been created. + */ + @Test + fun testInitializeEntities() { + /* Check initialization status (should be false). */ + Assertions.assertFalse(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + + /* Initialize basic tables. */ + this.testConnection.getDescriptorInitializer(this.field).initialize() + + /* Check initialization status (should be true). */ + Assertions.assertTrue(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + } + + /** + * Tests if the [RetrievableInitializer] can be initialized without throwing an exception. Furthermore, + * the test should check if the necessary tables have been created. + */ + @Test + fun testDeInitializeEntities() { + /* Check initialization status (should be false). */ + Assertions.assertFalse(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + + /* Initialize basic tables. */ + this.testConnection.getDescriptorInitializer(this.field).initialize() + + /* Check initialization status (should be true). */ + Assertions.assertTrue(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + + this.testConnection.getDescriptorInitializer(this.field).deinitialize() + Assertions.assertFalse(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + } + + /** + * Cleans up the database after each test. + * + */ + @AfterEach + open fun cleanup() { + this.testSchema.connection.getDescriptorInitializer(this.field).deinitialize() + } +} \ No newline at end of file diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractVectorDescriptorInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractVectorDescriptorInitializerTest.kt new file mode 100644 index 00000000..8d9d39d1 --- /dev/null +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractVectorDescriptorInitializerTest.kt @@ -0,0 +1,14 @@ +package org.vitrivr.engine.core.database.descriptor.vector + +import org.vitrivr.engine.core.database.descriptor.AbstractDescriptorInitializerTest +import org.vitrivr.engine.core.model.metamodel.Schema + +/** + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractVectorDescriptorInitializerTest(schemaPath: String) : AbstractDescriptorInitializerTest(schemaPath) { + /** The [Schema.Field] used for this [AbstractVectorDescriptorInitializerTest]. */ + override val field: Schema.Field<*, *> = this.testSchema["averagecolor"]!! +} \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializerTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializerTest.kt new file mode 100644 index 00000000..7f6501d4 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializerTest.kt @@ -0,0 +1,10 @@ +package org.vitrivr.engine.database.pgvector.descriptor.vector + +import org.vitrivr.engine.core.database.descriptor.vector.AbstractVectorDescriptorInitializerTest + +/** + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class VectorDescriptorInitializerTest : AbstractVectorDescriptorInitializerTest("test-schema-postgres.json") \ No newline at end of file From db4d3607e7b3c7bd9d456e9219ec7eeb92d27578 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 13:33:19 +0200 Subject: [PATCH 45/71] Adds distance implementations to Distance enumeration. Signed-off-by: Ralph Gasser --- .../core/model/query/basics/Distance.kt | 111 +++++++++++++++++- 1 file changed, 106 insertions(+), 5 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/basics/Distance.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/basics/Distance.kt index 3ceb2f37..4e6c5156 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/basics/Distance.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/query/basics/Distance.kt @@ -1,6 +1,10 @@ package org.vitrivr.engine.core.model.query.basics import org.vitrivr.engine.core.model.query.proximity.ProximityQuery +import org.vitrivr.engine.core.model.types.Value +import kotlin.math.abs +import kotlin.math.pow +import kotlin.math.sqrt /** * Enumeration of [Distance] functions supported by [ProximityQuery]. @@ -9,9 +13,106 @@ import org.vitrivr.engine.core.model.query.proximity.ProximityQuery * @version 1.0.0 */ enum class Distance { - MANHATTAN, - EUCLIDEAN, - COSINE, - HAMMING, - JACCARD; + MANHATTAN { + override fun invoke(v1: Value.FloatVector, v2: Value.FloatVector): Float { + var sum = 0.0f + for (i in v1.value.indices) { + sum += abs(v1.value[i] - v2.value[i]) + } + return sum + } + + override fun invoke(v1: Value.DoubleVector, v2: Value.DoubleVector): Double { + var sum = 0.0 + for (i in v1.value.indices) { + sum += abs(v1.value[i] - v2.value[i]) + } + return sum + } + }, + EUCLIDEAN { + override fun invoke(v1: Value.FloatVector, v2: Value.FloatVector): Float { + var sum = 0.0f + for (i in v1.value.indices) { + sum += (v1.value[i] - v2.value[i]).pow(2) + } + return sqrt(sum) + } + + override fun invoke(v1: Value.DoubleVector, v2: Value.DoubleVector): Double { + var sum = 0.0 + for (i in v1.value.indices) { + sum += (v1.value[i] - v2.value[i]).pow(2) + } + return sqrt(sum) + } + }, + COSINE { + override fun invoke(v1: Value.FloatVector, v2: Value.FloatVector): Float { + var dotProduct = 0.0f + var normV1 = 0.0f + var normV2 = 0.0f + for (i in v1.value.indices) { + dotProduct += v1.value[i] * v2.value[i] + normV1 += v1.value[i].pow(2) + normV2 += v2.value[i].pow(2) + } + normV1 = sqrt(normV1) + normV2 = sqrt(normV2) + return -(dotProduct / (normV1 * normV2)) + } + + override fun invoke(v1: Value.DoubleVector, v2: Value.DoubleVector): Double { + var dotProduct = 0.0 + var normV1 = 0.0 + var normV2 = 0.0 + for (i in v1.value.indices) { + dotProduct += v1.value[i] * v2.value[i] + normV1 += v1.value[i].pow(2) + normV2 += v2.value[i].pow(2) + } + normV1 = sqrt(normV1) + normV2 = sqrt(normV2) + return -(dotProduct / (normV1 * normV2)) + } + }, + HAMMING { + override fun invoke(v1: Value.FloatVector, v2: Value.FloatVector): Float { + var sum = 0.0f + for (i in v1.value.indices) { + sum += if (v1.value[i] != v2.value[i]) 1.0f else 0.0f + } + return sum + } + + override fun invoke(v1: Value.DoubleVector, v2: Value.DoubleVector): Double { + var sum = 0.0 + for (i in v1.value.indices) { + sum += if (v1.value[i] != v2.value[i]) 1.0 else 0.0 + } + return sum + } + }, + JACCARD { + override fun invoke(v1: Value.FloatVector, v2: Value.FloatVector): Float = throw UnsupportedOperationException("Jaccard distance is not supported for float vectors.") + override fun invoke(v1: Value.DoubleVector, v2: Value.DoubleVector): Double = throw UnsupportedOperationException("Jaccard distance is not supported for float vectors.") + }; + + /** + * Calculates this [Distance] between two [Value.FloatVector]. + * + * @param v1 [Value.FloatVector] First vector for distance calculation. + * @param v2 [Value.FloatVector] Second vector for distance calculation. + * @return [Float] + */ + abstract operator fun invoke(v1: Value.FloatVector, v2: Value.FloatVector): Float + + /** + * Calculates this [Distance] between two [Value.DoubleVector]. + * + * @param v1 [Value.DoubleVector] First vector for distance calculation. + * @param v2 [Value.DoubleVector] Second vector for distance calculation. + * @return [Double] + */ + abstract operator fun invoke(v1: Value.DoubleVector, v2: Value.DoubleVector): Double } \ No newline at end of file From 37264be9e1ae0c22652e03069baa0eb761dd1836 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 13:58:06 +0200 Subject: [PATCH 46/71] Adds more test cases for database connection. Signed-off-by: Ralph Gasser --- vitrivr-engine-core/build.gradle | 1 + .../engine/core/config/schema/SchemaConfig.kt | 2 +- .../AbstractDescriptorInitializerTest.kt | 54 +++++--- .../descriptor/DescriptorInitializerTest.kt | 12 ++ ...AbstractFloatVectorDescriptorReaderTest.kt | 126 ++++++++++++++++++ ...AbstractVectorDescriptorInitializerTest.kt | 14 -- .../build.gradle | 3 + .../descriptor/DescriptorInitializerTest.kt | 11 ++ .../resources/test-schema-cottontaildb.json | 29 ++++ .../descriptor/DescriptorInitializerTest.kt | 11 ++ .../vector/FloatVectorDescriptorReaderTest.kt | 11 ++ .../vector/VectorDescriptorInitializerTest.kt | 10 -- 12 files changed, 239 insertions(+), 45 deletions(-) create mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorInitializerTest.kt create mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractFloatVectorDescriptorReaderTest.kt delete mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractVectorDescriptorInitializerTest.kt create mode 100644 vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/DescriptorInitializerTest.kt create mode 100644 vitrivr-engine-module-cottontaildb/src/test/resources/test-schema-cottontaildb.json create mode 100644 vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/DescriptorInitializerTest.kt create mode 100644 vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/FloatVectorDescriptorReaderTest.kt delete mode 100644 vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializerTest.kt diff --git a/vitrivr-engine-core/build.gradle b/vitrivr-engine-core/build.gradle index 569a4813..8ad619c8 100755 --- a/vitrivr-engine-core/build.gradle +++ b/vitrivr-engine-core/build.gradle @@ -15,6 +15,7 @@ dependencies { /* Test Fixtures from Cottontail DB core. .*/ testFixturesImplementation group: 'org.junit.jupiter', name: 'junit-jupiter-api', version: version_junit + testFixturesImplementation group: 'org.junit.jupiter', name: 'junit-jupiter-params', version: version_junit } /* Publication of vitrivr engine core to Maven Central. */ diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/SchemaConfig.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/SchemaConfig.kt index de075a6f..1350706c 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/SchemaConfig.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/SchemaConfig.kt @@ -58,7 +58,7 @@ data class SchemaConfig( */ fun loadFromResource(resourcePath: String): SchemaConfig { val json = Json { ignoreUnknownKeys = true } // Configure Json to ignore unknown keys - val uri = this::class.java.classLoader.resources("test-schema-postgres.json").findFirst().orElseThrow { IllegalArgumentException("Resource '$resourcePath' not found!") }.toURI() + val uri = this::class.java.classLoader.resources(resourcePath).findFirst().orElseThrow { IllegalArgumentException("Resource '$resourcePath' not found!") }.toURI() val path = Paths.get(uri) val jsonString = Files.readString(path) return json.decodeFromString(jsonString) diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt index 886443b5..723cede3 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt @@ -2,61 +2,66 @@ package org.vitrivr.engine.core.database.descriptor import org.junit.jupiter.api.AfterEach import org.junit.jupiter.api.Assertions -import org.junit.jupiter.api.Test +import org.junit.jupiter.api.TestInstance +import org.junit.jupiter.params.ParameterizedTest +import org.junit.jupiter.params.provider.MethodSource import org.vitrivr.engine.core.database.AbstractDatabaseTest import org.vitrivr.engine.core.database.retrievable.RetrievableInitializer import org.vitrivr.engine.core.model.metamodel.Schema +import java.util.stream.Stream /** + * An abstract test to test the functionality of the [DescriptorInitializer]. * * @author Ralph Gasser * @version 1.0.0 */ +@TestInstance(TestInstance.Lifecycle.PER_CLASS) abstract class AbstractDescriptorInitializerTest(schemaPath: String) : AbstractDatabaseTest(schemaPath) { - /** The [Schema.Field] used for this [AbstractVectorDescriptorInitializerTest]. */ - protected abstract val field: Schema.Field<*, *> - /** * Tests if the [DescriptorInitializer.isInitialized] works as expected., */ - @Test - fun testIsInitializedWithoutInitialization() { - Assertions.assertFalse(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + @ParameterizedTest + @MethodSource("getFields") + fun testIsInitializedWithoutInitialization(field: Schema.Field<*, *>) { + Assertions.assertFalse(this.testConnection.getDescriptorInitializer(field).isInitialized()) } /** * Tests if the [RetrievableInitializer] can be initialized without throwing an exception. Furthermore, * the test should check if the necessary tables have been created. */ - @Test - fun testInitializeEntities() { + @ParameterizedTest + @MethodSource("getFields") + fun testInitializeEntities(field: Schema.Field<*, *>) { /* Check initialization status (should be false). */ - Assertions.assertFalse(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + Assertions.assertFalse(this.testConnection.getDescriptorInitializer(field).isInitialized()) /* Initialize basic tables. */ - this.testConnection.getDescriptorInitializer(this.field).initialize() + this.testConnection.getDescriptorInitializer(field).initialize() /* Check initialization status (should be true). */ - Assertions.assertTrue(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + Assertions.assertTrue(this.testConnection.getDescriptorInitializer(field).isInitialized()) } /** * Tests if the [RetrievableInitializer] can be initialized without throwing an exception. Furthermore, * the test should check if the necessary tables have been created. */ - @Test - fun testDeInitializeEntities() { + @ParameterizedTest + @MethodSource("getFields") + fun testDeInitializeEntities(field: Schema.Field<*, *>) { /* Check initialization status (should be false). */ - Assertions.assertFalse(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + Assertions.assertFalse(this.testConnection.getDescriptorInitializer(field).isInitialized()) /* Initialize basic tables. */ - this.testConnection.getDescriptorInitializer(this.field).initialize() + this.testConnection.getDescriptorInitializer(field).initialize() /* Check initialization status (should be true). */ - Assertions.assertTrue(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + Assertions.assertTrue(this.testConnection.getDescriptorInitializer(field).isInitialized()) - this.testConnection.getDescriptorInitializer(this.field).deinitialize() - Assertions.assertFalse(this.testConnection.getDescriptorInitializer(this.field).isInitialized()) + this.testConnection.getDescriptorInitializer(field).deinitialize() + Assertions.assertFalse(this.testConnection.getDescriptorInitializer(field).isInitialized()) } /** @@ -65,6 +70,15 @@ abstract class AbstractDescriptorInitializerTest(schemaPath: String) : AbstractD */ @AfterEach open fun cleanup() { - this.testSchema.connection.getDescriptorInitializer(this.field).deinitialize() + for (field in this.testSchema.fields()) { + this.testSchema.connection.getDescriptorInitializer(field).deinitialize() + } } + + /** + * Returns a [Stream] of [Schema.Field]s that are part of the test schema. + * + * @return [Stream] of [Schema.Field]s + */ + fun getFields(): Stream> = this.testSchema.fields().map { it as Schema.Field<*, *> }.stream() } \ No newline at end of file diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorInitializerTest.kt new file mode 100644 index 00000000..1e83c423 --- /dev/null +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorInitializerTest.kt @@ -0,0 +1,12 @@ +package org.vitrivr.engine.core.database.descriptor + +import org.vitrivr.engine.core.database.descriptor.AbstractDescriptorInitializerTest +import org.vitrivr.engine.core.model.metamodel.Schema + +/** + * An [AbstractDescriptorInitializerTest] implementation for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class DescriptorInitializerTest(schemaPath: String) : AbstractDescriptorInitializerTest(schemaPath) \ No newline at end of file diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractFloatVectorDescriptorReaderTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractFloatVectorDescriptorReaderTest.kt new file mode 100644 index 00000000..08199132 --- /dev/null +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractFloatVectorDescriptorReaderTest.kt @@ -0,0 +1,126 @@ +package org.vitrivr.engine.core.database.descriptor.vector + +import org.junit.jupiter.api.AfterEach +import org.junit.jupiter.api.Assertions +import org.junit.jupiter.api.BeforeEach +import org.junit.jupiter.api.Test +import org.junit.jupiter.params.ParameterizedTest +import org.junit.jupiter.params.provider.EnumSource +import org.vitrivr.engine.core.database.AbstractDatabaseTest +import org.vitrivr.engine.core.model.descriptor.vector.FloatVectorDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.query.basics.Distance +import org.vitrivr.engine.core.model.query.basics.SortOrder +import org.vitrivr.engine.core.model.query.proximity.ProximityQuery +import org.vitrivr.engine.core.model.retrievable.Ingested +import org.vitrivr.engine.core.model.types.Value +import java.util.* + +/** + * A series of test cases the test the functionality of the [VectorDescriptorReader]. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractFloatVectorDescriptorReaderTest(schemaPath: String) : AbstractDatabaseTest(schemaPath) { + + /** The [Schema.Field] used for this [DescriptorInitializerTest]. */ + private val field: Schema.Field<*, FloatVectorDescriptor> = this.testSchema["averagecolor"]!! as Schema.Field<*, FloatVectorDescriptor> + + /** + * Tests nearest neighbour search through the [VectorDescriptorReader.query] method. + */ + @ParameterizedTest + @EnumSource(mode = EnumSource.Mode.EXCLUDE, names = ["JACCARD", "HAMMING"]) + fun testNearestNeighbourSearch(distance: Distance) { + val writer = this.testConnection.getDescriptorWriter(this.field) + val reader = this.testConnection.getDescriptorReader(this.field) + val random = SplittableRandom() + val size = random.nextInt(500, 5000) + + /* Generate and store test data. */ + val retrievables = (0 until size).map { + Ingested(UUID.randomUUID(), "SOURCE:TEST", true) + } + Assertions.assertTrue(this.testConnection.getRetrievableWriter().addAll(retrievables)) + + val descriptors = retrievables.map { + FloatVectorDescriptor(UUID.randomUUID(), it.id, Value.FloatVector(FloatArray(3) { random.nextFloat() })) + } + Assertions.assertTrue(writer.addAll(descriptors)) + + /* Perform nearest neighbour search. */ + val query = ProximityQuery( + Value.FloatVector(FloatArray(3) { random.nextFloat() }), + distance, + SortOrder.ASC, + 100, + fetchVector = true + ) + val result = reader.query(query).toList() + + /* Make manual query and compare. */ + val manual = descriptors.sortedBy { distance(it.vector, query.value) }.take(100) + result.zip(manual).forEach { + Assertions.assertEquals(it.first.id, it.second.id) + } + } + + /** + * Tests farthest neighbour search through the [VectorDescriptorReader.query] method. + */ + @ParameterizedTest + @EnumSource(mode = EnumSource.Mode.EXCLUDE, names = ["JACCARD", "HAMMING"]) + fun testFarthestNeighbourSearch(distance: Distance) { + val writer = this.testConnection.getDescriptorWriter(this.field) + val reader = this.testConnection.getDescriptorReader(this.field) + val random = SplittableRandom() + val size = random.nextInt(500, 5000) + + /* Generate and store test data. */ + val retrievables = (0 until size).map { + Ingested(UUID.randomUUID(), "SOURCE:TEST", true) + } + Assertions.assertTrue(this.testConnection.getRetrievableWriter().addAll(retrievables)) + + val descriptors = retrievables.map { + FloatVectorDescriptor(UUID.randomUUID(), it.id, Value.FloatVector(FloatArray(3) { random.nextFloat() })) + } + Assertions.assertTrue(writer.addAll(descriptors)) + + /* Perform nearest neighbour search. */ + val query = ProximityQuery( + Value.FloatVector(FloatArray(3) { random.nextFloat() }), + distance, + SortOrder.DESC, + 100, + fetchVector = true + ) + val result = reader.query(query).toList() + + /* Make manual query and compare. */ + val manual = descriptors.sortedByDescending { distance(it.vector, query.value) }.take(100) + result.zip(manual).forEach { + Assertions.assertEquals(it.first.id, it.second.id) + } + } + + /** + * Cleans up the database after each test. + */ + @BeforeEach + open fun prepare() { + this.testConnection.getRetrievableInitializer().initialize() + this.testConnection.getDescriptorInitializer(this.field).initialize() + } + + /** + * Cleans up the database after each test. + * + */ + @AfterEach + open fun cleanup() { + this.testConnection.getRetrievableInitializer().deinitialize() + this.testConnection.getDescriptorInitializer(this.field).deinitialize() + } +} \ No newline at end of file diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractVectorDescriptorInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractVectorDescriptorInitializerTest.kt deleted file mode 100644 index 8d9d39d1..00000000 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractVectorDescriptorInitializerTest.kt +++ /dev/null @@ -1,14 +0,0 @@ -package org.vitrivr.engine.core.database.descriptor.vector - -import org.vitrivr.engine.core.database.descriptor.AbstractDescriptorInitializerTest -import org.vitrivr.engine.core.model.metamodel.Schema - -/** - * - * @author Ralph Gasser - * @version 1.0.0 - */ -abstract class AbstractVectorDescriptorInitializerTest(schemaPath: String) : AbstractDescriptorInitializerTest(schemaPath) { - /** The [Schema.Field] used for this [AbstractVectorDescriptorInitializerTest]. */ - override val field: Schema.Field<*, *> = this.testSchema["averagecolor"]!! -} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/build.gradle b/vitrivr-engine-module-cottontaildb/build.gradle index 32b8331a..da3bdd7c 100644 --- a/vitrivr-engine-module-cottontaildb/build.gradle +++ b/vitrivr-engine-module-cottontaildb/build.gradle @@ -16,6 +16,9 @@ dependencies { /** gRPC and Protobuf. */ implementation group: 'io.grpc', name: 'grpc-all', version: version_grpc implementation group: 'com.google.protobuf', name: 'protobuf-java', version: version_protobuf + + /** vitrivr engine Core is required for running tests. */ + testImplementation(testFixtures(project(':vitrivr-engine-core'))) } /* Publication of vitrivr engine query to Maven Central. */ diff --git a/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/DescriptorInitializerTest.kt b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/DescriptorInitializerTest.kt new file mode 100644 index 00000000..b62c7b01 --- /dev/null +++ b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/DescriptorInitializerTest.kt @@ -0,0 +1,11 @@ +package org.vitrivr.engine.database.cottontaildb.descriptor + +import org.vitrivr.engine.core.database.descriptor.AbstractDescriptorInitializerTest + +/** + * An [AbstractDescriptorInitializerTest] for Cottontail DB. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class DescriptorInitializerTest : AbstractDescriptorInitializerTest("test-schema-cottontaildb.json") \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/test/resources/test-schema-cottontaildb.json b/vitrivr-engine-module-cottontaildb/src/test/resources/test-schema-cottontaildb.json new file mode 100644 index 00000000..6802bf37 --- /dev/null +++ b/vitrivr-engine-module-cottontaildb/src/test/resources/test-schema-cottontaildb.json @@ -0,0 +1,29 @@ +{ + "name": "vitrivr-test", + "connection": { + "database": "CottontailConnectionProvider", + "parameters": { + "host": "127.0.0.1", + "port": "1865" + } + }, + "fields": [ + { + "name": "averagecolor", + "factory": "AverageColor" + }, + { + "name": "file", + "factory": "FileSourceMetadata" + }, + { + "name": "time", + "factory": "TemporalMetadata" + }, + { + "name": "video", + "factory": "VideoSourceMetadata" + } + ], + "exporters": [] +} diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/DescriptorInitializerTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/DescriptorInitializerTest.kt new file mode 100644 index 00000000..92a9d757 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/DescriptorInitializerTest.kt @@ -0,0 +1,11 @@ +package org.vitrivr.engine.database.pgvector.descriptor + +import org.vitrivr.engine.core.database.descriptor.AbstractDescriptorInitializerTest + +/** + * An [AbstractDescriptorInitializerTest] implementation for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class DescriptorInitializerTest : AbstractDescriptorInitializerTest("test-schema-postgres.json") \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/FloatVectorDescriptorReaderTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/FloatVectorDescriptorReaderTest.kt new file mode 100644 index 00000000..6586ee9e --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/FloatVectorDescriptorReaderTest.kt @@ -0,0 +1,11 @@ +package org.vitrivr.engine.database.pgvector.descriptor.vector + +import org.vitrivr.engine.core.database.descriptor.vector.AbstractFloatVectorDescriptorReaderTest + +/** + * An [AbstractFloatVectorDescriptorReaderTest] implementation for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class FloatVectorDescriptorReaderTest : AbstractFloatVectorDescriptorReaderTest("test-schema-postgres.json") \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializerTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializerTest.kt deleted file mode 100644 index 7f6501d4..00000000 --- a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorInitializerTest.kt +++ /dev/null @@ -1,10 +0,0 @@ -package org.vitrivr.engine.database.pgvector.descriptor.vector - -import org.vitrivr.engine.core.database.descriptor.vector.AbstractVectorDescriptorInitializerTest - -/** - * - * @author Ralph Gasser - * @version 1.0.0 - */ -class VectorDescriptorInitializerTest : AbstractVectorDescriptorInitializerTest("test-schema-postgres.json") \ No newline at end of file From 08daf5940f4986ed128efc7c71725423ecf500dc Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 13:58:25 +0200 Subject: [PATCH 47/71] CottontailConnection now creates schema upon initializing the connection. Signed-off-by: Ralph Gasser --- .../plugin/cottontaildb/CottontailConnection.kt | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnection.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnection.kt index 2e5b0929..76eeab3c 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnection.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/CottontailConnection.kt @@ -4,7 +4,9 @@ import io.github.oshai.kotlinlogging.KLogger import io.github.oshai.kotlinlogging.KotlinLogging.logger import io.grpc.ManagedChannel import io.grpc.ManagedChannelBuilder +import io.grpc.StatusRuntimeException import org.vitrivr.cottontail.client.SimpleClient +import org.vitrivr.cottontail.client.language.ddl.CreateSchema import org.vitrivr.engine.core.database.AbstractConnection import org.vitrivr.engine.plugin.cottontaildb.retrievable.RetrievableInitializer import org.vitrivr.engine.plugin.cottontaildb.retrievable.RetrievableReader @@ -27,6 +29,15 @@ class CottontailConnection(provider: CottontailConnectionProvider, schemaName: S /** The [SimpleClient] instance used by this [CottontailConnection]. */ internal val client = SimpleClient(this.channel) + + init { + try { + this.client.create(CreateSchema(this.schemaName).ifNotExists()) + } catch (e: StatusRuntimeException) { + LOGGER.error(e) { "Failed to create schema '${this.schemaName}' due to exception." } + } + } + /** * Tries to execute a given action within a database transaction. * From bd67ea591929c6f71643b5dd58d408e1e51a64ca Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 15:26:32 +0200 Subject: [PATCH 48/71] Slight adjustments to existing test cases. Signed-off-by: Ralph Gasser --- .../AbstractDescriptorInitializerTest.kt | 2 +- ...AbstractFloatVectorDescriptorReaderTest.kt | 76 ++++++++++++++----- .../AbstractRetrievableInitializerTest.kt | 12 --- 3 files changed, 57 insertions(+), 33 deletions(-) diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt index 723cede3..28018570 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt @@ -80,5 +80,5 @@ abstract class AbstractDescriptorInitializerTest(schemaPath: String) : AbstractD * * @return [Stream] of [Schema.Field]s */ - fun getFields(): Stream> = this.testSchema.fields().map { it as Schema.Field<*, *> }.stream() + private fun getFields(): Stream> = this.testSchema.fields().map { it as Schema.Field<*, *> }.stream() } \ No newline at end of file diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractFloatVectorDescriptorReaderTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractFloatVectorDescriptorReaderTest.kt index 08199132..9b80665e 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractFloatVectorDescriptorReaderTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/vector/AbstractFloatVectorDescriptorReaderTest.kt @@ -7,12 +7,16 @@ import org.junit.jupiter.api.Test import org.junit.jupiter.params.ParameterizedTest import org.junit.jupiter.params.provider.EnumSource import org.vitrivr.engine.core.database.AbstractDatabaseTest +import org.vitrivr.engine.core.database.descriptor.DescriptorWriter +import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.descriptor.vector.FloatVectorDescriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.query.basics.Distance import org.vitrivr.engine.core.model.query.basics.SortOrder import org.vitrivr.engine.core.model.query.proximity.ProximityQuery import org.vitrivr.engine.core.model.retrievable.Ingested +import org.vitrivr.engine.core.model.retrievable.attributes.DistanceAttribute +import org.vitrivr.engine.core.model.retrievable.attributes.ScoreAttribute import org.vitrivr.engine.core.model.types.Value import java.util.* @@ -28,26 +32,48 @@ abstract class AbstractFloatVectorDescriptorReaderTest(schemaPath: String) : Abs private val field: Schema.Field<*, FloatVectorDescriptor> = this.testSchema["averagecolor"]!! as Schema.Field<*, FloatVectorDescriptor> /** - * Tests nearest neighbour search through the [VectorDescriptorReader.query] method. + * Tests [VectorDescriptorReader.queryAndJoin] method. */ @ParameterizedTest @EnumSource(mode = EnumSource.Mode.EXCLUDE, names = ["JACCARD", "HAMMING"]) - fun testNearestNeighbourSearch(distance: Distance) { + fun testQueryAndJoin(distance: Distance) { val writer = this.testConnection.getDescriptorWriter(this.field) val reader = this.testConnection.getDescriptorReader(this.field) val random = SplittableRandom() - val size = random.nextInt(500, 5000) /* Generate and store test data. */ - val retrievables = (0 until size).map { - Ingested(UUID.randomUUID(), "SOURCE:TEST", true) - } - Assertions.assertTrue(this.testConnection.getRetrievableWriter().addAll(retrievables)) + val descriptors = this.initialize(writer, random) - val descriptors = retrievables.map { - FloatVectorDescriptor(UUID.randomUUID(), it.id, Value.FloatVector(FloatArray(3) { random.nextFloat() })) + /* Perform nearest neighbour search. */ + val query = ProximityQuery( + Value.FloatVector(FloatArray(3) { random.nextFloat() }), + distance, + SortOrder.ASC, + 100, + fetchVector = true + ) + val result = reader.queryAndJoin(query).toList() + + /* Make manual query and compare. */ + val manual = descriptors.sortedBy { distance(it.vector, query.value) }.take(100) + result.zip(manual).forEach { + Assertions.assertEquals(it.first.id, it.second.retrievableId) + Assertions.assertTrue(it.first.hasAttribute(DistanceAttribute::class.java)) } - Assertions.assertTrue(writer.addAll(descriptors)) + } + + /** + * Tests nearest neighbour search through the [VectorDescriptorReader.query] method. + */ + @ParameterizedTest + @EnumSource(mode = EnumSource.Mode.EXCLUDE, names = ["JACCARD", "HAMMING"]) + fun testNearestNeighbourSearch(distance: Distance) { + val writer = this.testConnection.getDescriptorWriter(this.field) + val reader = this.testConnection.getDescriptorReader(this.field) + val random = SplittableRandom() + + /* Generate and store test data. */ + val descriptors = this.initialize(writer, random) /* Perform nearest neighbour search. */ val query = ProximityQuery( @@ -75,18 +101,9 @@ abstract class AbstractFloatVectorDescriptorReaderTest(schemaPath: String) : Abs val writer = this.testConnection.getDescriptorWriter(this.field) val reader = this.testConnection.getDescriptorReader(this.field) val random = SplittableRandom() - val size = random.nextInt(500, 5000) /* Generate and store test data. */ - val retrievables = (0 until size).map { - Ingested(UUID.randomUUID(), "SOURCE:TEST", true) - } - Assertions.assertTrue(this.testConnection.getRetrievableWriter().addAll(retrievables)) - - val descriptors = retrievables.map { - FloatVectorDescriptor(UUID.randomUUID(), it.id, Value.FloatVector(FloatArray(3) { random.nextFloat() })) - } - Assertions.assertTrue(writer.addAll(descriptors)) + val descriptors = this.initialize(writer, random) /* Perform nearest neighbour search. */ val query = ProximityQuery( @@ -105,6 +122,25 @@ abstract class AbstractFloatVectorDescriptorReaderTest(schemaPath: String) : Abs } } + /** + * Initializes the test data. + */ + private fun initialize(writer: DescriptorWriter, random: SplittableRandom): List { + val size = random.nextInt(500, 5000) + + /* Generate and store test data. */ + val retrievables = (0 until size).map { + Ingested(UUID.randomUUID(), "SOURCE:TEST", true) + } + Assertions.assertTrue(this.testConnection.getRetrievableWriter().addAll(retrievables)) + + val descriptors = retrievables.map { + FloatVectorDescriptor(UUID.randomUUID(), it.id, Value.FloatVector(FloatArray(3) { random.nextFloat() })) + } + Assertions.assertTrue(writer.addAll(descriptors)) + return descriptors + } + /** * Cleans up the database after each test. */ diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt index a71cf49c..56f17a17 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt @@ -34,7 +34,6 @@ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : Abstract /* Check initialization status (should be true). */ Assertions.assertTrue(this.testConnection.getRetrievableInitializer().isInitialized()) - Assertions.assertTrue(this.checkTablesExist()) } /** @@ -51,22 +50,11 @@ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : Abstract /* Check initialization status (should be true). */ Assertions.assertTrue(this.testConnection.getRetrievableInitializer().isInitialized()) - Assertions.assertTrue(this.checkTablesExist()) this.testConnection.getRetrievableInitializer().deinitialize() Assertions.assertFalse(this.testConnection.getRetrievableInitializer().isInitialized()) - Assertions.assertFalse(this.checkTablesExist()) } - /** - * Checks if tables initialized actually exists. - * - * This check is database specific and therefore abstract. - * - * @return True if tables exist, false otherwise. - */ - protected abstract fun checkTablesExist(): Boolean - /** * Cleans up the database after each test. * From 44be403fae91095b560897d8070fc212990b9a6b Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 15:27:09 +0200 Subject: [PATCH 49/71] Fixes an issue in VectorDescriptorReader for Cottontail DB which led to missing DistanceAttribute. Signed-off-by: Ralph Gasser --- .../core/features/averagecolor/AverageColorRetriever.kt | 4 ++-- .../core/model/retrievable/attributes/DistanceAttribute.kt | 6 ++++++ .../descriptors/vector/VectorDescriptorReader.kt | 2 +- 3 files changed, 9 insertions(+), 3 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorRetriever.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorRetriever.kt index 02097f2b..48a788b2 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorRetriever.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColorRetriever.kt @@ -32,8 +32,8 @@ class AverageColorRetriever( companion object { private const val MAXIMUM_DISTANCE = 3f fun scoringFunction(retrieved: Retrieved): Float { - val distance = retrieved.filteredAttribute()?.distance ?: return 0f - return 1f - (distance / MAXIMUM_DISTANCE) + val distance = retrieved.filteredAttribute()?.distance ?: return 0.0f + return 1.0f - (distance / MAXIMUM_DISTANCE) } } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/retrievable/attributes/DistanceAttribute.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/retrievable/attributes/DistanceAttribute.kt index 9298841a..4531ea05 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/retrievable/attributes/DistanceAttribute.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/retrievable/attributes/DistanceAttribute.kt @@ -2,6 +2,12 @@ package org.vitrivr.engine.core.model.retrievable.attributes import kotlin.math.min +/** + * A [MergingRetrievableAttribute] that contains a distance value. + * + * @author Luca Rossetto + * @version 1.1.0 + */ data class DistanceAttribute(val distance: Float) : MergingRetrievableAttribute { override fun merge(other: MergingRetrievableAttribute): DistanceAttribute = DistanceAttribute( min(this.distance, (other as? DistanceAttribute)?.distance ?: Float.POSITIVE_INFINITY) diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt index 65f3a690..50157301 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/vector/VectorDescriptorReader.kt @@ -116,7 +116,7 @@ internal class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*> val descriptors = this.connection.client.query(cottontailQuery).asSequence().map { tuple -> val scoreIndex = tuple.indexOf(DISTANCE_COLUMN_NAME) tupleToDescriptor(tuple) to if (scoreIndex > -1) { - tuple.asFloat(DISTANCE_COLUMN_NAME)?.let { DistanceAttribute(it) } + tuple.asDouble(DISTANCE_COLUMN_NAME)?.let { DistanceAttribute(it.toFloat()) } } else { null } From c1fa48262a44148215894b25f7600a7b2b3ece18 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 15:27:22 +0200 Subject: [PATCH 50/71] Adds more Cottontail DB related test cases. Signed-off-by: Ralph Gasser --- .../vector/FloatVectorDescriptorReaderTest.kt | 11 +++++++++++ .../retrievable/RetrievableInitializerTest.kt | 11 +++++++++++ .../cottontaildb/retrievable/RetrievableWriterTest.kt | 11 +++++++++++ 3 files changed, 33 insertions(+) create mode 100644 vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/vector/FloatVectorDescriptorReaderTest.kt create mode 100644 vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/retrievable/RetrievableInitializerTest.kt create mode 100644 vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/retrievable/RetrievableWriterTest.kt diff --git a/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/vector/FloatVectorDescriptorReaderTest.kt b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/vector/FloatVectorDescriptorReaderTest.kt new file mode 100644 index 00000000..bec913ea --- /dev/null +++ b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/vector/FloatVectorDescriptorReaderTest.kt @@ -0,0 +1,11 @@ +package org.vitrivr.engine.database.cottontaildb.descriptor.vector + +import org.vitrivr.engine.core.database.descriptor.vector.AbstractFloatVectorDescriptorReaderTest + +/** + * An [AbstractFloatVectorDescriptorReaderTest] for Cottontail DB. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class FloatVectorDescriptorReaderTest : AbstractFloatVectorDescriptorReaderTest("test-schema-cottontaildb.json") \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/retrievable/RetrievableInitializerTest.kt b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/retrievable/RetrievableInitializerTest.kt new file mode 100644 index 00000000..97362cc0 --- /dev/null +++ b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/retrievable/RetrievableInitializerTest.kt @@ -0,0 +1,11 @@ +package org.vitrivr.engine.database.cottontaildb.retrievable + +import org.vitrivr.engine.core.database.retrievable.AbstractRetrievableInitializerTest + +/** + * An [AbstractRetrievableInitializerTest] implementation for Cottontail DB. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class RetrievableInitializerTest : AbstractRetrievableInitializerTest("test-schema-cottontaildb.json") \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/retrievable/RetrievableWriterTest.kt b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/retrievable/RetrievableWriterTest.kt new file mode 100644 index 00000000..c4385483 --- /dev/null +++ b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/retrievable/RetrievableWriterTest.kt @@ -0,0 +1,11 @@ +package org.vitrivr.engine.database.cottontaildb.retrievable + +import org.vitrivr.engine.core.database.retrievable.AbstractRetrievableWriterTest + +/** + * An [AbstractRetrievableWriterTest] for Cottontail DB. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class RetrievableWriterTest : AbstractRetrievableWriterTest("test-schema-cottontaildb.json") \ No newline at end of file From 50550df285c7ddc3cb3db832b3019f4358417db7 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 15:31:20 +0200 Subject: [PATCH 51/71] Fixes a missing check in RetrievableWriter for Cottontail DB. Signed-off-by: Ralph Gasser --- .../plugin/cottontaildb/retrievable/RetrievableWriter.kt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableWriter.kt index 6fd0c94d..c8145ab1 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/retrievable/RetrievableWriter.kt @@ -122,8 +122,8 @@ internal class RetrievableWriter(override val connection: CottontailConnection) /* Delete values. */ return try { - this.connection.client.delete(delete) - true + val result = this.connection.client.delete(delete) + return result.hasNext() && result.next().asLong(0)!! > 0L } catch (e: StatusRuntimeException) { logger.error(e) { "Failed to delete retrievable due to exception." } false From 399f7d2f41b0f3393956c3f1d3231c9cd036e4af Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 15:31:49 +0200 Subject: [PATCH 52/71] Fixes unit test (which cannot assume the order of returned elements to correspond to insertion order). Signed-off-by: Ralph Gasser --- .../core/database/retrievable/AbstractRetrievableWriterTest.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt index 4701ceeb..ee153526 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableWriterTest.kt @@ -49,7 +49,7 @@ abstract class AbstractRetrievableWriterTest(schemaPath: String) : AbstractDatab /* Check if retrievable can be read. */ Assertions.assertEquals(ids.size.toLong(), reader.count()) reader.getAll(ids).forEachIndexed { i, it -> - Assertions.assertEquals(ids[i], it.id) + Assertions.assertTrue(ids.contains(it.id)) Assertions.assertEquals("INGESTED:TEST", it.type) } } From 059823da3ae39959ba901e6f885324082d2923b5 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 15:34:32 +0200 Subject: [PATCH 53/71] Removes superfluous files. Signed-off-by: Ralph Gasser --- .../database/descriptor/DescriptorInitializerTest.kt | 12 ------------ 1 file changed, 12 deletions(-) delete mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorInitializerTest.kt diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorInitializerTest.kt deleted file mode 100644 index 1e83c423..00000000 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorInitializerTest.kt +++ /dev/null @@ -1,12 +0,0 @@ -package org.vitrivr.engine.core.database.descriptor - -import org.vitrivr.engine.core.database.descriptor.AbstractDescriptorInitializerTest -import org.vitrivr.engine.core.model.metamodel.Schema - -/** - * An [AbstractDescriptorInitializerTest] implementation for PostgreSQL with pgVector. - * - * @author Ralph Gasser - * @version 1.0.0 - */ -class DescriptorInitializerTest(schemaPath: String) : AbstractDescriptorInitializerTest(schemaPath) \ No newline at end of file From 12125dab96743db9b92bd9d6832f36c222997f97 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 15:42:15 +0200 Subject: [PATCH 54/71] Fixes various minor issues. Signed-off-by: Ralph Gasser --- .../scalar/ScalarDescriptorReader.kt | 4 ++-- .../struct/StructDescriptorReader.kt | 4 ++-- .../vector/VectorDescriptorReader.kt | 18 +++++++----------- 3 files changed, 11 insertions(+), 15 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt index c25de5ab..82d3f39b 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/scalar/ScalarDescriptorReader.kt @@ -61,7 +61,7 @@ class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connec * @return [Sequence] of [ScalarDescriptor]s. */ private fun queryFulltext(query: SimpleFulltextQuery): Sequence> { - val statement = "SELECT * FROM $tableName WHERE $VALUE_ATTRIBUTE_NAME @@ to_tsquery(?)" + val statement = "SELECT * FROM \"$tableName\" WHERE $VALUE_ATTRIBUTE_NAME @@ plainto_tsquery(?)" return sequence { this@ScalarDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> stmt.setString(1, query.value.value) @@ -81,7 +81,7 @@ class ScalarDescriptorReader(field: Schema.Field<*, ScalarDescriptor<*>>, connec * @return [Sequence] of [ScalarDescriptor]s. */ private fun queryBoolean(query: SimpleBooleanQuery<*>): Sequence> { - val statement = "SELECT * FROM $tableName WHERE $VALUE_ATTRIBUTE_NAME ${query.comparison.toSql()} ?" + val statement = "SELECT * FROM \"$tableName\" WHERE $VALUE_ATTRIBUTE_NAME ${query.comparison.toSql()} ?" return sequence { this@ScalarDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> stmt.setValue(1, query.value) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt index 0977c70f..b3598394 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt @@ -85,7 +85,7 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio */ private fun queryFulltext(query: SimpleFulltextQuery): Sequence { require(query.attributeName != null) { "Query attribute must not be null for a fulltext query on a struct descriptor." } - val statement = "SELECT * FROM $tableName WHERE ${query.attributeName} @@ to_tsquery(?)" + val statement = "SELECT * FROM \"$tableName\" WHERE ${query.attributeName} @@ plainto_tsquery(?)" return sequence { this@StructDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> stmt.setString(1, query.value.value) @@ -106,7 +106,7 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio */ private fun queryBoolean(query: SimpleBooleanQuery<*>): Sequence { require(query.attributeName != null) { "Query attribute must not be null for a fulltext query on a struct descriptor." } - val statement = "SELECT * FROM $tableName WHERE ${query.attributeName} ${query.comparison.toSql()} ?" + val statement = "SELECT * FROM \"$tableName\" WHERE ${query.attributeName} ${query.comparison.toSql()} ?" return sequence { this@StructDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> stmt.setValue(1, query.value) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt index 789324d7..6d69d35b 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/vector/VectorDescriptorReader.kt @@ -96,11 +96,8 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec * @return [Sequence] of [VectorDescriptor]s. */ private fun queryProximity(query: ProximityQuery<*>): Sequence> = sequence { - val statement = if (query.fetchVector) { - "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" - } else { - "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" - } + val statement = + "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM \"${tableName}\" ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" this@VectorDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> stmt.setValue(1, query.value) stmt.executeQuery().use { result -> @@ -119,11 +116,8 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec */ private fun queryAndJoinProximity(query: ProximityQuery<*>): Sequence { val descriptors = mutableListOf, Float>>() - val statement = if (query.fetchVector) { - "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" - } else { - "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM $tableName ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" - } + val statement = + "SELECT $DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME, $VECTOR_ATTRIBUTE_NAME, $VECTOR_ATTRIBUTE_NAME ${query.distance.toSql()} ? AS $DISTANCE_COLUMN_NAME FROM \"${tableName}\" ORDER BY $DISTANCE_COLUMN_NAME ${query.order} LIMIT ${query.k}" this@VectorDescriptorReader.connection.jdbc.prepareStatement(statement).use { stmt -> stmt.setValue(1, query.value) stmt.executeQuery().use { result -> @@ -137,7 +131,9 @@ class VectorDescriptorReader(field: Schema.Field<*, VectorDescriptor<*>>, connec return descriptors.asSequence().mapNotNull { (descriptor, distance) -> val retrievable = retrievables[descriptor.retrievableId] if (retrievable != null) { - retrievable.addDescriptor(descriptor) + if (query.fetchVector) { + retrievable.addDescriptor(descriptor) + } retrievable.addAttribute(DistanceAttribute(distance)) retrievable as Retrieved } else { From 277d12b6d76fc9bf35408b7edb6b1271a369656a Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 16:35:50 +0200 Subject: [PATCH 55/71] Fixes index related issues in PgDescriptorInitializer. Signed-off-by: Ralph Gasser --- .../descriptor/PgDescriptorInitializer.kt | 6 +++--- .../src/test/resources/test-schema-postgres.json | 16 +++++++++++++++- 2 files changed, 18 insertions(+), 4 deletions(-) diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index 1d23f1c5..f4b2efbc 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -70,12 +70,12 @@ open class PgDescriptorInitializer(final override val field: Sch for (index in this.field.indexes) { try { val indexStatement = when (index.type) { - IndexType.SCALAR -> "CREATE INDEX IF NOT EXISTS ON $tableName (${index.attributes.joinToString(",")});" - IndexType.FULLTEXT -> "CREATE INDEX IF NOT EXISTS ON $tableName USING GIN(${index.attributes.joinToString(",")});" + IndexType.SCALAR -> "CREATE INDEX ON $tableName (${index.attributes.joinToString(",")});" + IndexType.FULLTEXT -> "CREATE INDEX ON $tableName USING gin(${index.attributes.joinToString(",") { "to_tsvector('${index.parameters["language"] ?: "english"}', $it)" }});" IndexType.NNS -> { require(index.attributes.size == 1) { "NNS index can only be created on a single attribute." } val distance = index.parameters["distance"]?.let { Distance.valueOf(it.uppercase()) } ?: Distance.EUCLIDEAN - "CREATE INDEX ON $tableName USING hnsw (${index.attributes.first()} ${distance.toIndexName()});" + "CREATE INDEX ON $tableName USING hnsw(${index.attributes.first()} ${distance.toIndexName()});" } } this.connection.jdbc.prepareStatement(/* sql = postgres */ indexStatement).use { it.execute() } diff --git a/vitrivr-engine-module-pgvector/src/test/resources/test-schema-postgres.json b/vitrivr-engine-module-pgvector/src/test/resources/test-schema-postgres.json index f91f48e9..7e69198d 100644 --- a/vitrivr-engine-module-pgvector/src/test/resources/test-schema-postgres.json +++ b/vitrivr-engine-module-pgvector/src/test/resources/test-schema-postgres.json @@ -16,7 +16,21 @@ }, { "name": "file", - "factory": "FileSourceMetadata" + "factory": "FileSourceMetadata", + "indexes": [ + { + "attributes": [ + "path" + ], + "type": "FULLTEXT" + }, + { + "attributes": [ + "size" + ], + "type": "SCALAR" + } + ] }, { "name": "time", From d02e1c64f44f56d31cf04d2034b7d7da4fc3f79a Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 16:36:13 +0200 Subject: [PATCH 56/71] Some refactoring. Signed-off-by: Ralph Gasser --- .../descriptor/AbstractDescriptorInitializerTest.kt | 11 ++++++++++- .../AbstractRetrievableInitializerTest.kt | 6 +++--- .../retrievable/RetrievableInitializerTest.kt | 12 +++++++++++- 3 files changed, 24 insertions(+), 5 deletions(-) diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt index 28018570..cf1570e1 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/AbstractDescriptorInitializerTest.kt @@ -2,6 +2,7 @@ package org.vitrivr.engine.core.database.descriptor import org.junit.jupiter.api.AfterEach import org.junit.jupiter.api.Assertions +import org.junit.jupiter.api.BeforeEach import org.junit.jupiter.api.TestInstance import org.junit.jupiter.params.ParameterizedTest import org.junit.jupiter.params.provider.MethodSource @@ -64,12 +65,20 @@ abstract class AbstractDescriptorInitializerTest(schemaPath: String) : AbstractD Assertions.assertFalse(this.testConnection.getDescriptorInitializer(field).isInitialized()) } + /** + * Prepares the database before each test. + */ + @BeforeEach + open fun prepare() { + this.testSchema.connection.getRetrievableInitializer().initialize() + } + /** * Cleans up the database after each test. - * */ @AfterEach open fun cleanup() { + this.testSchema.connection.getRetrievableInitializer().deinitialize() for (field in this.testSchema.fields()) { this.testSchema.connection.getDescriptorInitializer(field).deinitialize() } diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt index 56f17a17..6c11b001 100644 --- a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/retrievable/AbstractRetrievableInitializerTest.kt @@ -16,7 +16,7 @@ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : Abstract * Tests if the [RetrievableInitializer.isInitialized] works as expected., */ @Test - fun testIsInitializedWithoutInitialization() { + open fun testIsInitializedWithoutInitialization() { Assertions.assertFalse(this.testConnection.getRetrievableInitializer().isInitialized()) } @@ -25,7 +25,7 @@ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : Abstract * the test should check if the necessary tables have been created. */ @Test - fun testInitializeEntities() { + open fun testInitializeEntities() { /* Check initialization status (should be false). */ Assertions.assertFalse(this.testConnection.getRetrievableInitializer().isInitialized()) @@ -41,7 +41,7 @@ abstract class AbstractRetrievableInitializerTest(schemaPath: String) : Abstract * the test should check if the necessary tables have been created. */ @Test - fun testDeInitializeEntities() { + open fun testDeInitializeEntities() { /* Check initialization status (should be false). */ Assertions.assertFalse(this.testConnection.getRetrievableInitializer().isInitialized()) diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializerTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializerTest.kt index ce2780dd..35be2e7e 100644 --- a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializerTest.kt +++ b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/retrievable/RetrievableInitializerTest.kt @@ -1,5 +1,6 @@ package org.vitrivr.engine.database.pgvector.retrievable +import org.junit.jupiter.api.Assertions import org.vitrivr.engine.core.database.retrievable.AbstractRetrievableInitializerTest import org.vitrivr.engine.database.pgvector.PgVectorConnection import org.vitrivr.engine.database.pgvector.RELATIONSHIP_ENTITY_NAME @@ -12,12 +13,21 @@ import org.vitrivr.engine.database.pgvector.RETRIEVABLE_ENTITY_NAME * @version 1.0.0 */ class RetrievableInitializerTest : AbstractRetrievableInitializerTest("test-schema-postgres.json") { + + /** + * + */ + override fun testInitializeEntities() { + super.testInitializeEntities() + Assertions.assertTrue(checkTablesExist()) + } + /** * Checks if tables initialized actually exists. * * @return True if tables exist, false otherwise. */ - override fun checkTablesExist(): Boolean { + private fun checkTablesExist(): Boolean { /* Check for existence of tables. */ (this.testConnection as PgVectorConnection).jdbc.prepareStatement("SELECT EXISTS (SELECT FROM information_schema.tables WHERE table_schema = ? AND table_name = ?)").use { statement -> /* Check existence of retrievable table. */ From d3eadfb9e467d3f902c549b2197010aa26c171fe Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 12 Jul 2024 16:37:13 +0200 Subject: [PATCH 57/71] Adds index creation to Cottontail DB related tests. Signed-off-by: Ralph Gasser --- .../test/resources/test-schema-cottontaildb.json | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/vitrivr-engine-module-cottontaildb/src/test/resources/test-schema-cottontaildb.json b/vitrivr-engine-module-cottontaildb/src/test/resources/test-schema-cottontaildb.json index 6802bf37..f39f59b6 100644 --- a/vitrivr-engine-module-cottontaildb/src/test/resources/test-schema-cottontaildb.json +++ b/vitrivr-engine-module-cottontaildb/src/test/resources/test-schema-cottontaildb.json @@ -14,7 +14,21 @@ }, { "name": "file", - "factory": "FileSourceMetadata" + "factory": "FileSourceMetadata", + "indexes": [ + { + "attributes": [ + "path" + ], + "type": "FULLTEXT" + }, + { + "attributes": [ + "size" + ], + "type": "SCALAR" + } + ] }, { "name": "time", From 2b40d96f99575a003e221d949308c6c19122cf58 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Sat, 13 Jul 2024 09:53:04 +0200 Subject: [PATCH 58/71] Adds unit tests for boolean and fulltext queries on FileMetadataDescriptors. Signed-off-by: Ralph Gasser --- .../source/FileSourceMetadataDescriptor.kt | 5 +- ...bstractFileMetadataDescriptorReaderTest.kt | 254 ++++++++++++++++++ .../FileMetadataDescriptorReaderTest.kt | 11 + .../FileMetadataDescriptorReaderTest.kt | 11 + 4 files changed, 280 insertions(+), 1 deletion(-) create mode 100644 vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/struct/AbstractFileMetadataDescriptorReaderTest.kt create mode 100644 vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/struct/FileMetadataDescriptorReaderTest.kt create mode 100644 vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/FileMetadataDescriptorReaderTest.kt diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/FileSourceMetadataDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/FileSourceMetadataDescriptor.kt index 664298a3..273f5159 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/FileSourceMetadataDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/metadata/source/FileSourceMetadataDescriptor.kt @@ -1,8 +1,8 @@ package org.vitrivr.engine.core.model.descriptor.struct.metadata.source -import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.descriptor.AttributeName +import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.struct.MapStructDescriptor import org.vitrivr.engine.core.model.descriptor.struct.StructDescriptor import org.vitrivr.engine.core.model.metamodel.Schema @@ -24,6 +24,9 @@ class FileSourceMetadataDescriptor( override val field: Schema.Field<*, FileSourceMetadataDescriptor>? = null ) : MapStructDescriptor(id, retrievableId, SCHEMA, values, field) { + constructor(id: DescriptorId, retrievableId: RetrievableId?, path: Value.String, size: Value.Long, field: Schema.Field<*, FileSourceMetadataDescriptor>) : + this(id, retrievableId, mapOf("path" to path, "size" to size), field) + /** The path to the file. */ val path: Value.String by this.values diff --git a/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/struct/AbstractFileMetadataDescriptorReaderTest.kt b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/struct/AbstractFileMetadataDescriptorReaderTest.kt new file mode 100644 index 00000000..b4504161 --- /dev/null +++ b/vitrivr-engine-core/src/testFixtures/kotlin/org/vitrivr/engine/core/database/descriptor/struct/AbstractFileMetadataDescriptorReaderTest.kt @@ -0,0 +1,254 @@ +package org.vitrivr.engine.core.database.descriptor.struct + +import org.junit.jupiter.api.AfterEach +import org.junit.jupiter.api.Assertions +import org.junit.jupiter.api.BeforeEach +import org.junit.jupiter.api.Test +import org.vitrivr.engine.core.database.AbstractDatabaseTest +import org.vitrivr.engine.core.database.descriptor.DescriptorWriter +import org.vitrivr.engine.core.model.descriptor.struct.metadata.source.FileSourceMetadataDescriptor +import org.vitrivr.engine.core.model.metamodel.Schema +import org.vitrivr.engine.core.model.query.basics.ComparisonOperator +import org.vitrivr.engine.core.model.query.bool.SimpleBooleanQuery +import org.vitrivr.engine.core.model.query.fulltext.SimpleFulltextQuery +import org.vitrivr.engine.core.model.retrievable.Ingested +import org.vitrivr.engine.core.model.types.Value +import java.nio.file.Paths +import java.util.* + +/** + * An [AbstractDatabaseTest] that tests for basic boolean queries on [FileSourceMetadataDescriptor]. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +abstract class AbstractFileMetadataDescriptorReaderTest(schemaPath: String) : AbstractDatabaseTest(schemaPath) { + + companion object { + /* Define a list of possible root directories. */ + val ROOTS = arrayOf( + "/home", + "/home/test", + "/home/user/", + "/home/admin/", + "/tmp/", + "/var/log/", + "/var/", + "/usr/local/bin/", + "/var/opt", + "/etc", + "/tank", + "/mnt/tank", + "/usr", + "/usr/lib" + ) + + /* Define a list of possible root directories. */ + val FILENAMES = listOf( + "file1.txt", + "test.txt", + "document.pdf", + "image.png", + "image.jpg", + "image.jpeg", + "notes.docx", + "video.mp4", + "audio.mp3", + "test.jpg", + "test.png" + ) + } + + + /** The [Schema.Field] used for this [AbstractFileMetadataDescriptorReaderTest]. */ + private val field: Schema.Field<*, FileSourceMetadataDescriptor> = this.testSchema["file"]!! as Schema.Field<*, FileSourceMetadataDescriptor> + + /** + * Tests for equals comparison. + */ + @Test + fun testBooleanQueryEquals() { + val writer = this.testConnection.getDescriptorWriter(this.field) + val reader = this.testConnection.getDescriptorReader(this.field) + val random = SplittableRandom() + + /* Generate and store test data. */ + val descriptors = this.initialize(writer, random) + val d = descriptors[random.nextInt(0, descriptors.size)] + + /* Prepare and execute query. */ + val query = SimpleBooleanQuery( + d.path, + ComparisonOperator.EQ, + "path" + ) + + /* Check results. */ + val result = reader.query(query).toList() + Assertions.assertTrue(result.isNotEmpty()) + for (r in result) { + Assertions.assertEquals(d.path, r.path) + } + } + + /** + * Tests for LIKE comparison. + */ + @Test + fun testBooleanQueryLike() { + val writer = this.testConnection.getDescriptorWriter(this.field) + val reader = this.testConnection.getDescriptorReader(this.field) + val random = SplittableRandom() + + /* Generate and store test data. */ + this.initialize(writer, random) + + /* Prepare and execute query. */ + val query = SimpleBooleanQuery( + Value.String("%.jpg"), + ComparisonOperator.LIKE, + "path" + ) + + /* Check results. */ + val result = reader.query(query).toList() + for (r in result) { + Assertions.assertTrue(r.path.value.endsWith(".jpg")) + } + } + + /** + * Tests for greater-than comparison. + */ + @Test + fun testBooleanQueryGreater() { + val writer = this.testConnection.getDescriptorWriter(this.field) + val reader = this.testConnection.getDescriptorReader(this.field) + val random = SplittableRandom() + + /* Generate and store test data. */ + this.initialize(writer, random) + + /* Prepare and execute query. */ + val size = Value.Long(random.nextLong(0, 100_000_000L)) + val query = SimpleBooleanQuery( + size, + ComparisonOperator.GR, + "size" + ) + + /* Check results. */ + val result = reader.query(query).toList() + for (r in result) { + Assertions.assertTrue(r.size.value > size.value) + } + } + + /** + * Tests for less-than comparison. + */ + @Test + fun testBooleanQueryLess() { + val writer = this.testConnection.getDescriptorWriter(this.field) + val reader = this.testConnection.getDescriptorReader(this.field) + val random = SplittableRandom() + + /* Generate and store test data. */ + this.initialize(writer, random) + + /* Prepare and execute query. */ + val size = Value.Long(random.nextLong(0, 100_000_000L)) + val query = SimpleBooleanQuery( + size, + ComparisonOperator.LE, + "size" + ) + + /* Check results. */ + val result = reader.query(query).toList() + for (r in result) { + Assertions.assertTrue(r.size.value < size.value) + } + } + + /** + * Tests for less-than comparison. + */ + @Test + fun testFulltextQuery() { + val writer = this.testConnection.getDescriptorWriter(this.field) + val reader = this.testConnection.getDescriptorReader(this.field) + val random = SplittableRandom() + + /* Generate and store test data. */ + this.initialize(writer, random) + + /* Prepare and execute query. */ + val query = SimpleFulltextQuery( + Value.String("var"), + "path" + ) + + /* Check results. */ + val result = reader.query(query).toList() + for (r in result) { + Assertions.assertTrue(r.path.value.contains("var")) + } + } + + /** + * Initializes the test data. + */ + private fun initialize(writer: DescriptorWriter, random: SplittableRandom): List { + val size = random.nextInt(500, 5000) + + /* Generate and store test data. */ + val retrievables = (0 until size).map { + Ingested(UUID.randomUUID(), "SOURCE:TEST", true) + } + Assertions.assertTrue(this.testConnection.getRetrievableWriter().addAll(retrievables)) + + val descriptors = retrievables.map { + FileSourceMetadataDescriptor(UUID.randomUUID(), it.id, this.generateRandomPath(random), Value.Long(random.nextLong(0, 100_000_000L)), this.field) + } + Assertions.assertTrue(writer.addAll(descriptors)) + return descriptors + } + + /** + * Generates a random path and wraps it in a Value.String. + * + * @return A Value.String representing a random path. + */ + private fun generateRandomPath(random: SplittableRandom): Value.String { + /* Define a list of possible root directories. */ + val root = ROOTS[random.nextInt(ROOTS.size)] + val fileName = FILENAMES[random.nextInt(FILENAMES.size)] + + /* Define a list of possible root directories. */ + val path = Paths.get(root, fileName).toString() + + /* Define a list of possible root directories. */ + return Value.String(path) + } + + + /** + * Cleans up the database after each test. + */ + @BeforeEach + open fun prepare() { + this.testConnection.getRetrievableInitializer().initialize() + this.testConnection.getDescriptorInitializer(this.field).initialize() + } + + /** + * Cleans up the database after each test. + * + */ + @AfterEach + open fun cleanup() { + this.testConnection.getRetrievableInitializer().deinitialize() + this.testConnection.getDescriptorInitializer(this.field).deinitialize() + } +} \ No newline at end of file diff --git a/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/struct/FileMetadataDescriptorReaderTest.kt b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/struct/FileMetadataDescriptorReaderTest.kt new file mode 100644 index 00000000..e0ceecdd --- /dev/null +++ b/vitrivr-engine-module-cottontaildb/src/test/kotlin/org/vitrivr/engine/database/cottontaildb/descriptor/struct/FileMetadataDescriptorReaderTest.kt @@ -0,0 +1,11 @@ +package org.vitrivr.engine.database.cottontaildb.descriptor.struct + +import org.vitrivr.engine.core.database.descriptor.struct.AbstractFileMetadataDescriptorReaderTest + +/** + * An [AbstractFileMetadataDescriptorReaderTest] implementation for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class FileMetadataDescriptorReaderTest : AbstractFileMetadataDescriptorReaderTest("test-schema-cottontaildb.json") \ No newline at end of file diff --git a/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/FileMetadataDescriptorReaderTest.kt b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/FileMetadataDescriptorReaderTest.kt new file mode 100644 index 00000000..20155d50 --- /dev/null +++ b/vitrivr-engine-module-pgvector/src/test/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/FileMetadataDescriptorReaderTest.kt @@ -0,0 +1,11 @@ +package org.vitrivr.engine.database.pgvector.descriptor.struct + +import org.vitrivr.engine.core.database.descriptor.struct.AbstractFileMetadataDescriptorReaderTest + +/** + * An [AbstractFileMetadataDescriptorReaderTest] implementation for PostgreSQL with pgVector. + * + * @author Ralph Gasser + * @version 1.0.0 + */ +class FileMetadataDescriptorReaderTest : AbstractFileMetadataDescriptorReaderTest("test-schema-postgres.json") \ No newline at end of file From 464c005a9762fa08b5d4e2f0e0aa8c4ca1f1a750 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Sat, 13 Jul 2024 14:29:18 +0200 Subject: [PATCH 59/71] Adds index support to CottontailDescriptorInitializer. Signed-off-by: Ralph Gasser --- .../CottontailDescriptorInitializer.kt | 24 +++++++++++++++---- 1 file changed, 20 insertions(+), 4 deletions(-) diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt index e5858746..d1658fc8 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt @@ -1,16 +1,16 @@ package org.vitrivr.engine.plugin.cottontaildb.descriptors import io.grpc.StatusRuntimeException -import org.vitrivr.cottontail.client.language.ddl.CreateEntity -import org.vitrivr.cottontail.client.language.ddl.DropEntity -import org.vitrivr.cottontail.client.language.ddl.ListEntities -import org.vitrivr.cottontail.client.language.ddl.TruncateEntity +import org.vitrivr.cottontail.client.language.ddl.* import org.vitrivr.cottontail.core.database.Name import org.vitrivr.cottontail.core.types.Types +import org.vitrivr.cottontail.grpc.CottontailGrpc +import org.vitrivr.engine.core.config.schema.IndexType import org.vitrivr.engine.core.database.descriptor.DescriptorInitializer import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.* +import java.sql.SQLException /** * An abstract implementation of a [DescriptorInitializer] for Cottontail DB. @@ -42,6 +42,22 @@ open class CottontailDescriptorInitializer(final override val fi } catch (e: StatusRuntimeException) { LOGGER.error(e) { "Failed to initialize entity '${this.entityName}' due to exception." } } + + /* Create indexes (optional). */ + for (index in this.field.indexes) { + require(index.attributes.size == 1) { "Cottontail DB currently only supports single-column indexes." } + try { + val createIndex = when (index.type) { + IndexType.SCALAR -> CreateIndex(this.entityName, CottontailGrpc.IndexType.BTREE).column(index.attributes.first()) + IndexType.FULLTEXT -> CreateIndex(this.entityName, CottontailGrpc.IndexType.LUCENE).column(index.attributes.first()) + IndexType.NNS -> CreateIndex(this.entityName, CottontailGrpc.IndexType.PQ).column(index.attributes.first()) + } + this.connection.client.create(createIndex) + } catch (e: SQLException) { + LOGGER.error(e) { "Failed to create index ${index.type} for entity '$entityName' due to exception." } + throw e + } + } } /** From 495c774993206778d6f3b493f260b8bbfe6fe137 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 15 Jul 2024 09:02:26 +0200 Subject: [PATCH 60/71] Updates Cottontail DB version. Signed-off-by: Ralph Gasser --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 6a7583a4..cd17b680 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -13,7 +13,7 @@ jobs: # Setup Cottontail DB and PostgreSQL service container services: cottontail: - image: vitrivr/cottontaildb:0.16.6 + image: vitrivr/cottontaildb:0.16.7 ports: - 1865:1865 options: -it From bf0f2a21bb91398e9db1559b4a4ee1402fbde57e Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 15 Jul 2024 10:55:48 +0200 Subject: [PATCH 61/71] Adds sanity check upon initialization as requested by @lucaro. Signed-off-by: Ralph Gasser --- .../org/vitrivr/engine/core/config/schema/IndexConfig.kt | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexConfig.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexConfig.kt index 76cc3e12..8eef81e3 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexConfig.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/config/schema/IndexConfig.kt @@ -13,4 +13,8 @@ import org.vitrivr.engine.core.model.metamodel.Schema * @version 1.0.0 */ @Serializable -data class IndexConfig(val attributes: List, val type: IndexType, val parameters: Map = emptyMap()) \ No newline at end of file +data class IndexConfig(val attributes: List, val type: IndexType, val parameters: Map = emptyMap()) { + init { + require(attributes.isNotEmpty()) { "Cannot define index on empty list of attributes." } + } +} \ No newline at end of file From 00bbc151a7873cf6f0314dc5262699d8a4845762 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 15 Jul 2024 10:56:54 +0200 Subject: [PATCH 62/71] Removes erroneous TODO(). Signed-off-by: Ralph Gasser --- .../database/retrievable/NoRetrievableInitializer.kt | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/retrievable/NoRetrievableInitializer.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/retrievable/NoRetrievableInitializer.kt index f40c7616..a1e3be93 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/retrievable/NoRetrievableInitializer.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/retrievable/NoRetrievableInitializer.kt @@ -9,14 +9,17 @@ package org.vitrivr.engine.core.database.retrievable * @version 1.0.0 */ class NoRetrievableInitializer : RetrievableInitializer { - override fun initialize() { /* No op. */ + override fun initialize() { + /* No op. */ } override fun deinitialize() { - TODO("Not yet implemented") + /* No op. */ } - override fun isInitialized(): Boolean = false - override fun truncate() { /* No op. */ + override fun truncate() { + /* No op. */ } + + override fun isInitialized(): Boolean = false } \ No newline at end of file From 15cd6a0c1a908e794094d91168becd3e9676dbec Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 15 Jul 2024 11:10:54 +0200 Subject: [PATCH 63/71] Renames methods in DescriptorReader interface. Signed-off-by: Ralph Gasser --- .../database/descriptor/DescriptorReader.kt | 28 +++++++++---------- .../descriptors/AbstractDescriptorReader.kt | 4 +-- .../descriptor/AbstractDescriptorReader.kt | 4 +-- .../operators/transform/lookup/FieldLookup.kt | 2 +- .../transform/lookup/ObjectFieldLookup.kt | 2 +- .../engine/query/parsing/QueryParser.kt | 2 +- 6 files changed, 21 insertions(+), 21 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorReader.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorReader.kt index 53837a0b..5df0973d 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorReader.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/descriptor/DescriptorReader.kt @@ -17,21 +17,13 @@ import org.vitrivr.engine.core.model.retrievable.Retrieved * * @author Luca Rossetto * @author Ralph Gasser - * @version 2.0.0 + * @version 2.0.1 */ interface DescriptorReader : Reader { /** The [Analyser] this [DescriptorReader] belongs to. */ val field: Schema.Field<*,D> - /** - * Returns the [Descriptor] of type [D] that corresponds to the provided [DescriptorId]. - * - * @param descriptorId The [DescriptorId] to return. - * @return [Descriptor] of type [D] or null - */ - operator fun get(descriptorId: DescriptorId): D? - /** * Checks if a [Descriptor] of type [D] for provided [DescriptorId] exists. * @@ -41,12 +33,12 @@ interface DescriptorReader : Reader { fun exists(descriptorId: DescriptorId): Boolean /** - * Returns the [Descriptor]s of type [D] that belong to the provided [RetrievableId]. + * Returns the [Descriptor] of type [D] that corresponds to the provided [DescriptorId]. * - * @param retrievableId The [RetrievableId] to search for. - * @return [Sequence] of [Descriptor] of type [D] + * @param descriptorId The [DescriptorId] to return. + * @return [Descriptor] of type [D] or null */ - fun getFor(retrievableId: RetrievableId): Sequence + operator fun get(descriptorId: DescriptorId): D? /** * Returns a [Sequence] of all [Descriptor] whose [DescriptorId] is contained in the provided [Iterable]. @@ -56,13 +48,21 @@ interface DescriptorReader : Reader { */ fun getAll(descriptorIds: Iterable): Sequence + /** + * Returns the [Descriptor]s of type [D] that belong to the provided [RetrievableId]. + * + * @param retrievableId The [RetrievableId] to search for. + * @return [Sequence] of [Descriptor] of type [D] + */ + fun getForRetrievable(retrievableId: RetrievableId): Sequence + /** * Returns a [Sequence] of all [Descriptor] whose [RetrievableId] is contained in the provided [Iterable]. * * @param retrievableIds A [Iterable] of [RetrievableId]s to return [Descriptor]s for * @return [Sequence] of [Descriptor] of type [D] */ - fun getAllFor(retrievableIds: Iterable): Sequence + fun getAllForRetrievable(retrievableIds: Iterable): Sequence /** * Returns a [Sequence] of all [Descriptor]s [D]s that match the given [Query]. diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt index 03c8d662..bf4805ef 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/AbstractDescriptorReader.kt @@ -58,7 +58,7 @@ abstract class AbstractDescriptorReader(final override val field * @param retrievableId The [RetrievableId] to search for. * @return [Sequence] of [Descriptor] of type [D] */ - override fun getFor(retrievableId: RetrievableId): Sequence { + override fun getForRetrievable(retrievableId: RetrievableId): Sequence { val query = org.vitrivr.cottontail.client.language.dql.Query(this.entityName).where(Compare(Column(this.entityName.column(RETRIEVABLE_ID_COLUMN_NAME)), Compare.Operator.EQUAL, Literal(UuidValue(retrievableId)))) return try { val result = this.connection.client.query(query) @@ -127,7 +127,7 @@ abstract class AbstractDescriptorReader(final override val field * @param retrievableIds A [Iterable] of [RetrievableId]s to return [Descriptor]s for * @return [Sequence] of [Descriptor] of type [D] */ - override fun getAllFor(retrievableIds: Iterable): Sequence { + override fun getAllForRetrievable(retrievableIds: Iterable): Sequence { val query = org.vitrivr.cottontail.client.language.dql.Query(this.entityName) .where(Compare(Column(this.entityName.column(RETRIEVABLE_ID_COLUMN_NAME)), Compare.Operator.IN, org.vitrivr.cottontail.client.language.basics.expression.List(retrievableIds.map { UuidValue(it) }.toTypedArray()))) return try { diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt index 49c557f9..dddc0a71 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/AbstractDescriptorReader.kt @@ -56,7 +56,7 @@ abstract class AbstractDescriptorReader(final override val field * @param retrievableId The [RetrievableId] to search for. * @return [Sequence] of [Descriptor] of type [D] */ - override fun getFor(retrievableId: RetrievableId): Sequence { + override fun getForRetrievable(retrievableId: RetrievableId): Sequence { try { this.connection.jdbc.prepareStatement("SELECT * FROM $tableName WHERE $RETRIEVABLE_ID_COLUMN_NAME = ?").use { stmt -> stmt.setObject(1, retrievableId) @@ -151,7 +151,7 @@ abstract class AbstractDescriptorReader(final override val field * @param retrievableIds A [Iterable] of [RetrievableId]s to return [Descriptor]s for * @return [Sequence] of [Descriptor] of type [D] */ - override fun getAllFor(retrievableIds: Iterable): Sequence { + override fun getAllForRetrievable(retrievableIds: Iterable): Sequence { try { this.connection.jdbc.prepareStatement("SELECT * FROM $tableName WHERE $RETRIEVABLE_ID_COLUMN_NAME = ANY (?)").use { stmt -> val values = retrievableIds.map { it }.toTypedArray() diff --git a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookup.kt b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookup.kt index 06b7d1d4..9b14d214 100644 --- a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookup.kt +++ b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookup.kt @@ -35,7 +35,7 @@ class FieldLookup( val descriptors = if (ids.isEmpty()) { emptyMap() } else { - this@FieldLookup.reader.getAllFor(ids).associateBy { it.retrievableId!! } + this@FieldLookup.reader.getAllForRetrievable(ids).associateBy { it.retrievableId!! } } /* Emit retrievable with added attribute. */ diff --git a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/ObjectFieldLookup.kt b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/ObjectFieldLookup.kt index c8aff560..cda01633 100644 --- a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/ObjectFieldLookup.kt +++ b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/ObjectFieldLookup.kt @@ -38,7 +38,7 @@ class ObjectFieldLookup(override val input: Operator, private v /* Fetch descriptors for retrievables that should be enriched. */ val descriptors = if (enrich.isNotEmpty()) { - this@ObjectFieldLookup.reader.getAllFor(enrich.keys).associateBy { it.retrievableId!! } + this@ObjectFieldLookup.reader.getAllForRetrievable(enrich.keys).associateBy { it.retrievableId!! } } else { emptyMap() } diff --git a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt index 97779d7b..fbf13b3b 100644 --- a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt +++ b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt @@ -89,7 +89,7 @@ class QueryParser(val schema: Schema) { is RetrievableIdInputData -> { val id = UUID.fromString(input.id) val reader = field.getReader() - val descriptor = reader.getFor(id).firstOrNull() ?: throw IllegalArgumentException("No retrievable with id '$id' present in ${field.fieldName}") + val descriptor = reader.getForRetrievable(id).firstOrNull() ?: throw IllegalArgumentException("No retrievable with id '$id' present in ${field.fieldName}") field.getRetrieverForDescriptor(descriptor, description.context) } is VectorInputData -> field.getRetrieverForDescriptor(FloatVectorDescriptor(vector = Value.FloatVector(input.data.toFloatArray())), description.context) From 2eaa4935fcc2ebaa843d9b5c39810c7d9c231c21 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 15 Jul 2024 11:22:26 +0200 Subject: [PATCH 64/71] Adds ability to specify index structures as proposed by @lucaro. Signed-off-by: Ralph Gasser --- .../engine/core/database/Initializer.kt | 10 +++++++++ .../descriptor/PgDescriptorInitializer.kt | 22 ++++++++++++++++--- 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Initializer.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Initializer.kt index a31aa2f4..e7651418 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Initializer.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/database/Initializer.kt @@ -1,6 +1,7 @@ package org.vitrivr.engine.core.database import org.vitrivr.engine.core.model.Persistable +import org.vitrivr.engine.core.model.query.basics.Distance /** * The [Initializer] is part of the data persistence layer of vitrivr and can be used to encapsulate DDL operations @@ -11,6 +12,15 @@ import org.vitrivr.engine.core.model.Persistable * @version 1.1.0 */ interface Initializer { + + companion object { + /** The column name used to specify type of index. This is database specific.*/ + const val INDEX_TYPE_PARAMETER_NAME = "type" + + /** The parameter name used to specify the distance. Used upon index creation. References [Distance].*/ + const val DISTANCE_PARAMETER_NAME = "distance" + } + /** * Initializes the (persistent) data structures required by the [Persistable] of type [T]. */ diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index f4b2efbc..46dc6481 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -1,6 +1,8 @@ package org.vitrivr.engine.database.pgvector.descriptor import org.vitrivr.engine.core.config.schema.IndexType +import org.vitrivr.engine.core.database.Initializer.Companion.DISTANCE_PARAMETER_NAME +import org.vitrivr.engine.core.database.Initializer.Companion.INDEX_TYPE_PARAMETER_NAME import org.vitrivr.engine.core.database.descriptor.DescriptorInitializer import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.metamodel.Schema @@ -17,6 +19,14 @@ import java.sql.SQLException */ open class PgDescriptorInitializer(final override val field: Schema.Field<*, D>, protected val connection: PgVectorConnection): DescriptorInitializer { + companion object { + /** Set of scalar index structures supported by PostgreSQL. */ + private val INDEXES_SCALAR = setOf("btree", "brin", "hash") + + /** Set of NNS index structures supported by PostgreSQL. */ + private val INDEXES_NNS = setOf("hnsw", "ivfflat") + } + /** The name of the table backing this [PgDescriptorInitializer]. */ protected val tableName: String = "${DESCRIPTOR_ENTITY_PREFIX}_${this.field.fieldName}" @@ -70,12 +80,18 @@ open class PgDescriptorInitializer(final override val field: Sch for (index in this.field.indexes) { try { val indexStatement = when (index.type) { - IndexType.SCALAR -> "CREATE INDEX ON $tableName (${index.attributes.joinToString(",")});" + IndexType.SCALAR -> { + val type = index.parameters[INDEX_TYPE_PARAMETER_NAME]?.lowercase() ?: "btree" + require(type in INDEXES_SCALAR) { "Index type '$type' is not supported by PostgreSQL." } + "CREATE INDEX ON $tableName USING $type(${index.attributes.joinToString(",")});" + } IndexType.FULLTEXT -> "CREATE INDEX ON $tableName USING gin(${index.attributes.joinToString(",") { "to_tsvector('${index.parameters["language"] ?: "english"}', $it)" }});" IndexType.NNS -> { require(index.attributes.size == 1) { "NNS index can only be created on a single attribute." } - val distance = index.parameters["distance"]?.let { Distance.valueOf(it.uppercase()) } ?: Distance.EUCLIDEAN - "CREATE INDEX ON $tableName USING hnsw(${index.attributes.first()} ${distance.toIndexName()});" + val type = index.parameters[INDEX_TYPE_PARAMETER_NAME]?.lowercase() ?: "hnsw" + val distance = index.parameters[DISTANCE_PARAMETER_NAME]?.let { Distance.valueOf(it.uppercase()) } ?: Distance.EUCLIDEAN + require(type in INDEXES_NNS) { "Index type '$type' is not supported by PostgreSQL." } + "CREATE INDEX ON $tableName USING $type(${index.attributes.first()} ${distance.toIndexName()});" } } this.connection.jdbc.prepareStatement(/* sql = postgres */ indexStatement).use { it.execute() } From 2861b8e0ce119e3ed39dec72ee1e6a20e222d6ce Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 15 Jul 2024 11:25:43 +0200 Subject: [PATCH 65/71] Removes superfluous parameter. Signed-off-by: Ralph Gasser --- .../org/vitrivr/engine/core/model/descriptor/Attribute.kt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Attribute.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Attribute.kt index ebe0e895..df1a6930 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Attribute.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Attribute.kt @@ -12,4 +12,4 @@ typealias AttributeName = String * @author Ralph Gasser * @version 1.0.0 */ -data class Attribute(val name: AttributeName, val type: Type, val nullable: Boolean = false, val indexed: Boolean = false) \ No newline at end of file +data class Attribute(val name: AttributeName, val type: Type, val nullable: Boolean = false) \ No newline at end of file From 6e3aa0577164c95d9ed9c16c71bf2c6abaa61205 Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Mon, 15 Jul 2024 11:57:04 +0200 Subject: [PATCH 66/71] Renames Descriptor.schema() to Descriptor.layout(). Signed-off-by: Ralph Gasser --- .../engine/core/model/descriptor/Descriptor.kt | 2 +- .../model/descriptor/scalar/BooleanDescriptor.kt | 2 +- .../core/model/descriptor/scalar/DoubleDescriptor.kt | 2 +- .../core/model/descriptor/scalar/FloatDescriptor.kt | 2 +- .../core/model/descriptor/scalar/IntDescriptor.kt | 2 +- .../core/model/descriptor/scalar/LongDescriptor.kt | 2 +- .../core/model/descriptor/scalar/StringDescriptor.kt | 2 +- .../model/descriptor/struct/MapStructDescriptor.kt | 4 ++-- .../descriptor/vector/BooleanVectorDescriptor.kt | 2 +- .../descriptor/vector/DoubleVectorDescriptor.kt | 2 +- .../model/descriptor/vector/FloatVectorDescriptor.kt | 2 +- .../model/descriptor/vector/IntVectorDescriptor.kt | 2 +- .../model/descriptor/vector/LongVectorDescriptor.kt | 2 +- .../descriptors/CottontailDescriptorInitializer.kt | 2 +- .../descriptors/CottontailDescriptorWriter.kt | 2 +- .../descriptors/struct/StructDescriptorReader.kt | 2 +- .../pgvector/descriptor/PgDescriptorInitializer.kt | 2 +- .../pgvector/descriptor/PgDescriptorWriter.kt | 12 ++++++------ .../descriptor/struct/StructDescriptorReader.kt | 2 +- .../operators/transform/lookup/FieldLookupFactory.kt | 2 +- .../org/vitrivr/engine/query/parsing/QueryParser.kt | 3 ++- 21 files changed, 28 insertions(+), 27 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt index 465968df..bfbe2939 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/Descriptor.kt @@ -36,7 +36,7 @@ interface Descriptor : Persistable { * * @return [List] of [Attribute] */ - fun schema(): List + fun layout(): List /** * Returns the fields and its values of this [Descriptor] as a [Map]. diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt index f013e0e5..356b582a 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/BooleanDescriptor.kt @@ -30,5 +30,5 @@ data class BooleanDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun layout(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt index 856ebf24..828408f4 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/DoubleDescriptor.kt @@ -30,5 +30,5 @@ data class DoubleDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun layout(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt index 2272344a..b87fa80c 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/FloatDescriptor.kt @@ -30,5 +30,5 @@ data class FloatDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun layout(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt index 34c29f0f..b56daff3 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/IntDescriptor.kt @@ -30,5 +30,5 @@ data class IntDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun layout(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt index 42bc3739..a472150d 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/LongDescriptor.kt @@ -30,5 +30,5 @@ data class LongDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun layout(): List = SCHEMA } \ No newline at end of file diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt index e91b3635..3a1bfa07 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/scalar/StringDescriptor.kt @@ -30,5 +30,5 @@ data class StringDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = SCHEMA + override fun layout(): List = SCHEMA } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/MapStructDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/MapStructDescriptor.kt index d3ecd7f0..bf12ba56 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/MapStructDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/struct/MapStructDescriptor.kt @@ -1,9 +1,9 @@ package org.vitrivr.engine.core.model.descriptor.struct -import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.descriptor.Attribute import org.vitrivr.engine.core.model.descriptor.AttributeName import org.vitrivr.engine.core.model.descriptor.Descriptor +import org.vitrivr.engine.core.model.descriptor.DescriptorId import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.core.model.retrievable.RetrievableId import org.vitrivr.engine.core.model.types.Value @@ -26,7 +26,7 @@ open class MapStructDescriptor( * * @return [List] of [Attribute] */ - final override fun schema(): List = this.layout + final override fun layout(): List = this.layout /** * Returns the fields and its values of this [Descriptor] as a [Map]. diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt index 2ffd580a..52b29910 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/BooleanVectorDescriptor.kt @@ -27,5 +27,5 @@ data class BooleanVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.BooleanVector(this.dimensionality))) + override fun layout(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.BooleanVector(this.dimensionality))) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt index 1aa1ce38..a5c1f028 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/DoubleVectorDescriptor.kt @@ -27,5 +27,5 @@ data class DoubleVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.DoubleVector(this.dimensionality))) + override fun layout(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.DoubleVector(this.dimensionality))) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt index 2173b30e..d1205f52 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/FloatVectorDescriptor.kt @@ -28,5 +28,5 @@ data class FloatVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.FloatVector(this.dimensionality))) + override fun layout(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.FloatVector(this.dimensionality))) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt index d7876a37..dbea9d17 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/IntVectorDescriptor.kt @@ -27,5 +27,5 @@ data class IntVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.Int)) + override fun layout(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.Int)) } diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt index 88f7742b..825f0226 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/descriptor/vector/LongVectorDescriptor.kt @@ -27,5 +27,5 @@ data class LongVectorDescriptor( * * @return [List] of [Attribute] */ - override fun schema(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.Long)) + override fun layout(): List = listOf(Attribute(VECTOR_ATTRIBUTE_NAME, Type.Long)) } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt index d1658fc8..31a6e614 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt @@ -32,7 +32,7 @@ open class CottontailDescriptorInitializer(final override val fi .column(Name.ColumnName.create(RETRIEVABLE_ID_COLUMN_NAME), Types.Uuid, nullable = false, primaryKey = false, autoIncrement = false) /* Append fields. */ - for (field in this.field.analyser.prototype(this.field).schema()) { + for (field in this.field.analyser.prototype(this.field).layout()) { create.column(Name.ColumnName.create(field.name), field.type.toCottontailType(), nullable = field.nullable, primaryKey = false, autoIncrement = false) } diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorWriter.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorWriter.kt index 24fbdaed..f62f22a5 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorWriter.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorWriter.kt @@ -66,7 +66,7 @@ open class CottontailDescriptorWriter(final override val field: /* Insert values. */ for (item in items) { - val attributes = item.schema() + val attributes = item.layout() if (index == 0) { val columns = Array(attributes.size + 2) { when (it) { diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt index 3e34b99b..c0df79ec 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/struct/StructDescriptorReader.kt @@ -30,7 +30,7 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio private val fieldMap = mutableListOf>>() init { val prototype = this.field.analyser.prototype(this.field) - for (f in prototype.schema()) { + for (f in prototype.layout()) { this.fieldMap.add(f.name to f.type.toCottontailType()) } } diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt index 46dc6481..827a39a7 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorInitializer.kt @@ -42,7 +42,7 @@ open class PgDescriptorInitializer(final override val field: Sch statement.append("$RETRIEVABLE_ID_COLUMN_NAME uuid NOT NULL, ") /* Add columns for each field in the struct. */ - for (field in this.prototype.schema()) { + for (field in this.prototype.layout()) { when (field.type) { Type.String -> statement.append("\"${field.name}\" varchar(255), ") Type.Text -> statement.append("\"${field.name}\" text, ") diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt index 844cc62c..b2300d00 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/PgDescriptorWriter.kt @@ -32,7 +32,7 @@ open class PgDescriptorWriter(final override val field: Schema.F stmt.setObject(1, item.id) stmt.setObject(2, item.retrievableId) var i = 3 - for (attribute in item.schema()) { + for (attribute in item.layout()) { val value = item.values()[attribute.name] if (value != null) { stmt.setValue(i++, value) @@ -61,7 +61,7 @@ open class PgDescriptorWriter(final override val field: Schema.F stmt.setObject(1, item.id) stmt.setObject(2, item.retrievableId) var i = 3 - for (attribute in item.schema()) { + for (attribute in item.layout()) { val value = item.values()[attribute.name] if (value != null) { stmt.setValue(i++, value) @@ -90,7 +90,7 @@ open class PgDescriptorWriter(final override val field: Schema.F this.prepareUpdateStatement().use { stmt -> stmt.setObject(1, item.retrievableId) var i = 2 - for (attribute in item.schema()) { + for (attribute in item.layout()) { val value = item.values()[attribute.name] if (value != null) { stmt.setValue(i++, value) @@ -152,7 +152,7 @@ open class PgDescriptorWriter(final override val field: Schema.F */ protected fun prepareUpdateStatement(): PreparedStatement { val statement = StringBuilder("UPDATE $tableName SET $RETRIEVABLE_ID_COLUMN_NAME = ?") - for (field in this.prototype.schema()) { + for (field in this.prototype.layout()) { statement.append(", \"${field.name}\" = ?") } statement.append("WHERE $DESCRIPTOR_ID_COLUMN_NAME = ?;") @@ -166,11 +166,11 @@ open class PgDescriptorWriter(final override val field: Schema.F */ protected fun prepareInsertStatement(): PreparedStatement { val statement = StringBuilder("INSERT INTO $tableName ($DESCRIPTOR_ID_COLUMN_NAME, $RETRIEVABLE_ID_COLUMN_NAME") - for (field in this.prototype.schema()) { + for (field in this.prototype.layout()) { statement.append(", \"${field.name}\"") } statement.append(") VALUES (?, ?") - for (field in this.field.analyser.prototype(this.field).schema()) { + for (field in this.field.analyser.prototype(this.field).layout()) { statement.append(", ?") } statement.append(");") diff --git a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt index b3598394..d9870e62 100644 --- a/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt +++ b/vitrivr-engine-module-pgvector/src/main/kotlin/org/vitrivr/engine/database/pgvector/descriptor/struct/StructDescriptorReader.kt @@ -53,7 +53,7 @@ class StructDescriptorReader(field: Schema.Field<*, StructDescriptor>, connectio ) /* Append dynamic parameters of struct. */ - for (field in this.prototype.schema()) { + for (field in this.prototype.layout()) { values[field.name] = when(field.type) { Type.String -> result.getString(field.name)?.let { Value.String(it) } Type.Text -> result.getString(field.name)?.let { Value.Text(it) } diff --git a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookupFactory.kt b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookupFactory.kt index d2c2d62c..0e1cd0c0 100644 --- a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookupFactory.kt +++ b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/operators/transform/lookup/FieldLookupFactory.kt @@ -15,7 +15,7 @@ class FieldLookupFactory() : TransformerFactory { val providedKeys = context[name, "keys"] val keys = if(providedKeys?.isNotBlank() == true){ if(providedKeys.length == 1 && providedKeys == "*") { - schemaField.analyser.prototype(schemaField).schema().map { it.name } + schemaField.analyser.prototype(schemaField).layout().map { it.name } }else{ providedKeys.split(",").map { s -> s.trim() } } diff --git a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt index fbf13b3b..ea3f4319 100644 --- a/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt +++ b/vitrivr-engine-query/src/main/kotlin/org/vitrivr/engine/query/parsing/QueryParser.kt @@ -97,7 +97,8 @@ class QueryParser(val schema: Schema) { /* Is this a boolean sub-field query ? */ if(fieldAndAttributeName.second != null && input.comparison != null){ /* yes */ - val subfield = field.analyser.prototype(field).schema().find { it.name == fieldAndAttributeName.second } ?: throw IllegalArgumentException("Field '${field.fieldName}' does not have a subfield with name '${fieldAndAttributeName.second}'") + val subfield = + field.analyser.prototype(field).layout().find { it.name == fieldAndAttributeName.second } ?: throw IllegalArgumentException("Field '${field.fieldName}' does not have a subfield with name '${fieldAndAttributeName.second}'") /* For now, we support not all input data */ val value = when(input){ is TextInputData -> { From 32ae241316a982df01a151cca2de3deb094c463a Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Wed, 24 Jul 2024 10:16:33 +0200 Subject: [PATCH 67/71] Adds a sanity check for Schema.Field names. --- .../kotlin/org/vitrivr/engine/core/model/metamodel/Schema.kt | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Schema.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Schema.kt index 828b88ef..a0dacb11 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Schema.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/metamodel/Schema.kt @@ -161,6 +161,11 @@ open class Schema(val name: String = "vitrivr", val connection: Connection) : Cl val parameters: Map = emptyMap(), val indexes: List = emptyList() ) { + + init { + require(this.fieldName.matches(Regex("^[a-zA-Z0-9]+$"))) { "Field name can only contain alpha-numeric characters and numbers." } + } + /** Pointer to the [Schema] this [Field] belongs to.*/ val schema: Schema get() = this@Schema From db1f0e120bb61d905ad8d1f8d9527932e57cabfe Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Wed, 24 Jul 2024 10:25:14 +0200 Subject: [PATCH 68/71] It's now also possible to parametrize Cottontail DB index structures. --- .../CottontailDescriptorInitializer.kt | 22 +++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt index 31a6e614..3e8727e7 100644 --- a/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt +++ b/vitrivr-engine-module-cottontaildb/src/main/kotlin/org/vitrivr/engine/plugin/cottontaildb/descriptors/CottontailDescriptorInitializer.kt @@ -6,11 +6,11 @@ import org.vitrivr.cottontail.core.database.Name import org.vitrivr.cottontail.core.types.Types import org.vitrivr.cottontail.grpc.CottontailGrpc import org.vitrivr.engine.core.config.schema.IndexType +import org.vitrivr.engine.core.database.Initializer.Companion.INDEX_TYPE_PARAMETER_NAME import org.vitrivr.engine.core.database.descriptor.DescriptorInitializer import org.vitrivr.engine.core.model.descriptor.Descriptor import org.vitrivr.engine.core.model.metamodel.Schema import org.vitrivr.engine.plugin.cottontaildb.* -import java.sql.SQLException /** * An abstract implementation of a [DescriptorInitializer] for Cottontail DB. @@ -48,12 +48,26 @@ open class CottontailDescriptorInitializer(final override val fi require(index.attributes.size == 1) { "Cottontail DB currently only supports single-column indexes." } try { val createIndex = when (index.type) { - IndexType.SCALAR -> CreateIndex(this.entityName, CottontailGrpc.IndexType.BTREE).column(index.attributes.first()) + IndexType.SCALAR -> { + val type = CottontailGrpc.IndexType.valueOf(index.parameters[INDEX_TYPE_PARAMETER_NAME]?.uppercase() ?: "BTREE") + require(type in setOf(CottontailGrpc.IndexType.BTREE, CottontailGrpc.IndexType.BTREE_UQ)) { "Index type '$type' is not supported for scalar search." } + CreateIndex(this.entityName, CottontailGrpc.IndexType.BTREE).column(index.attributes.first()) + } IndexType.FULLTEXT -> CreateIndex(this.entityName, CottontailGrpc.IndexType.LUCENE).column(index.attributes.first()) - IndexType.NNS -> CreateIndex(this.entityName, CottontailGrpc.IndexType.PQ).column(index.attributes.first()) + IndexType.NNS -> { + val type = CottontailGrpc.IndexType.valueOf(index.parameters[INDEX_TYPE_PARAMETER_NAME]?.uppercase() ?: "PQ") + require(type in setOf(CottontailGrpc.IndexType.VAF, CottontailGrpc.IndexType.PQ, CottontailGrpc.IndexType.LSH, CottontailGrpc.IndexType.IVFPQ)) { "Index type '$type' is not supported for vector search." } + CreateIndex(this.entityName, type).column(index.attributes.first()).apply { + for ((k,v,) in index.parameters) { + if (k != INDEX_TYPE_PARAMETER_NAME) { + this.param(k, v) + } + } + } + } } this.connection.client.create(createIndex) - } catch (e: SQLException) { + } catch (e: StatusRuntimeException) { LOGGER.error(e) { "Failed to create index ${index.type} for entity '$entityName' due to exception." } throw e } From 593f7dd2704246883b21ac7b0f8de881710a7d61 Mon Sep 17 00:00:00 2001 From: Luca Rossetto Date: Thu, 25 Jul 2024 11:30:46 +0200 Subject: [PATCH 69/71] Made AverageColor.analyze static --- .../feature/averagecolor/AverageColor.kt | 33 ++++++++++--------- .../averagecolor/AverageColorExtractor.kt | 2 +- 2 files changed, 19 insertions(+), 16 deletions(-) diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt index 2510922b..7228dcd7 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColor.kt @@ -114,22 +114,25 @@ class AverageColor : Analyser { * @param context The [QueryContext] to use with the [Retriever] */ override fun newRetrieverForContent(field: Schema.Field, content: Collection, context: QueryContext): AverageColorRetriever = - this.newRetrieverForDescriptors(field, this.analyse(content), context) + this.newRetrieverForDescriptors(field, analyse(content), context) - /** - * Performs the [AverageColor] analysis on the provided [List] of [ImageContent] elements. - * - * @param content The [List] of [ImageContent] elements. - * @return [List] of [FloatVectorDescriptor]s. - */ - fun analyse(content: Collection): List = content.map { - logger.trace{"Analysing"} - val color = MutableRGBFloatColorContainer() - val rgb = it.content.getRGBArray() - rgb.forEach { c -> color += RGBByteColorContainer.fromRGB(c) } + companion object { + + /** + * Performs the [AverageColor] analysis on the provided [List] of [ImageContent] elements. + * + * @param content The [List] of [ImageContent] elements. + * @return [List] of [FloatVectorDescriptor]s. + */ + fun analyse(content: Collection): List = content.map { + val color = MutableRGBFloatColorContainer() + val rgb = it.content.getRGBArray() + rgb.forEach { c -> color += RGBByteColorContainer.fromRGB(c) } - /* Generate descriptor. */ - val averageColor = RGBFloatColorContainer(color.red / rgb.size, color.green / rgb.size, color.blue / rgb.size) - FloatVectorDescriptor(UUID.randomUUID(), null, averageColor.toValueList()) + /* Generate descriptor. */ + val averageColor = + RGBFloatColorContainer(color.red / rgb.size, color.green / rgb.size, color.blue / rgb.size) + FloatVectorDescriptor(UUID.randomUUID(), null, averageColor.toValueList()) + } } } diff --git a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorExtractor.kt b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorExtractor.kt index 4bf0afcc..04d6ff9e 100644 --- a/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorExtractor.kt +++ b/vitrivr-engine-module-features/src/main/kotlin/org/vitrivr/engine/module/features/feature/averagecolor/AverageColorExtractor.kt @@ -39,6 +39,6 @@ class AverageColorExtractor(input: Operator, field: Schema.Field { val content = retrievable.content.filterIsInstance() - return AverageColor().analyse(content).map { it.copy(retrievableId = retrievable.id, field = this@AverageColorExtractor.field) } + return AverageColor.analyse(content).map { it.copy(retrievableId = retrievable.id, field = this@AverageColorExtractor.field) } } } \ No newline at end of file From 6cac8e13a6dcd9bb2ea89534c7f88a1139959c4a Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 26 Jul 2024 09:27:31 +0200 Subject: [PATCH 70/71] Fixes compilation issue in M3D module. Signed-off-by: Ralph Gasser --- .../org/vitrivr/engine/model3d/ModelHandler.kt | 5 ----- .../sphericalharmonics/SphericalHarmonics.kt | 14 +++++++------- 2 files changed, 7 insertions(+), 12 deletions(-) diff --git a/vitrivr-engine-module-m3d/src/main/kotlin/org/vitrivr/engine/model3d/ModelHandler.kt b/vitrivr-engine-module-m3d/src/main/kotlin/org/vitrivr/engine/model3d/ModelHandler.kt index d322f20f..db1fd7bb 100644 --- a/vitrivr-engine-module-m3d/src/main/kotlin/org/vitrivr/engine/model3d/ModelHandler.kt +++ b/vitrivr-engine-module-m3d/src/main/kotlin/org/vitrivr/engine/model3d/ModelHandler.kt @@ -6,7 +6,6 @@ import org.joml.Vector4f import org.lwjgl.BufferUtils import org.lwjgl.assimp.* import org.lwjgl.assimp.Assimp.* - import org.lwjgl.system.MemoryStack import org.vitrivr.engine.core.model.mesh.* import java.io.File @@ -15,7 +14,6 @@ import java.io.InputStream import java.nio.ByteBuffer import java.nio.IntBuffer import java.util.* -import kotlin.collections.ArrayList class ModelHandler { @@ -95,9 +93,6 @@ class ModelHandler { LOGGER.trace("Try loading model {} from InputStream", modelId) val aiScene = loadAIScene(modelId, inputStream) - - val modelDir = File("").absolutePath // You may need to adjust this to the appropriate directory - val materialList: MutableList = ArrayList() val defaultMaterial = Material(mutableListOf()) // Define a default material val aiMeshes = aiScene.mMeshes() diff --git a/vitrivr-engine-module-m3d/src/main/kotlin/org/vitrivr/engine/model3d/features/sphericalharmonics/SphericalHarmonics.kt b/vitrivr-engine-module-m3d/src/main/kotlin/org/vitrivr/engine/model3d/features/sphericalharmonics/SphericalHarmonics.kt index f90c4137..e6bcd440 100644 --- a/vitrivr-engine-module-m3d/src/main/kotlin/org/vitrivr/engine/model3d/features/sphericalharmonics/SphericalHarmonics.kt +++ b/vitrivr-engine-module-m3d/src/main/kotlin/org/vitrivr/engine/model3d/features/sphericalharmonics/SphericalHarmonics.kt @@ -94,12 +94,12 @@ class SphericalHarmonics: Analyser { val numberOfCoefficients: Int = SphericalHarmonicsFunction.numberOfCoefficients(maxL, true) - SphericalHarmonicsFunction.numberOfCoefficients(minL - 1, true) /* Prepares an empty array for the feature vector. */ - val feature = Array((R - cap) * numberOfCoefficients) { _ -> Value.Float(0f) } + val feature = FloatArray((R - cap) * numberOfCoefficients) { _ -> 0f } /* Voxelizes the grid from the mesh. If the resulting grid is invisible, the method returns immediately. */ val grid: VoxelModel = voxelizer.voxelize(mesh, gridSize + 1, gridSize + 1, gridSize + 1) if (!grid.isVisible()) { - return FloatVectorDescriptor(UUID.randomUUID(), null, feature.toList()) + return FloatVectorDescriptor(UUID.randomUUID(), null, Value.FloatVector(feature)) } val descriptors: MutableList> = LinkedList() @@ -144,12 +144,12 @@ class SphericalHarmonics: Analyser { var index = 0 for (radius in descriptors) { for (descriptor in radius) { - feature[index++] = Value.Float(descriptor.abs().toFloat()) + feature[index++] = descriptor.abs().toFloat() } } /* Returns the normalized vector. */ - return FloatVectorDescriptor(UUID.randomUUID(), null, MathHelper.normalizeL2(feature).toList()) + return FloatVectorDescriptor(UUID.randomUUID(), null, Value.FloatVector(MathHelper.normalizeL2(feature))) } } @@ -162,16 +162,16 @@ class SphericalHarmonics: Analyser { val maxL = field.parameters[MAXL_PARAMETER_NAME]?.toIntOrNull() ?: MAXL_PARAMETER_DEFAULT val numberOfCoefficients: Int = SphericalHarmonicsFunction.numberOfCoefficients(maxL, true) - SphericalHarmonicsFunction.numberOfCoefficients(minL - 1, true) val vectorSize = ((gridSize / 2) - cap) * numberOfCoefficients - return FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), List(vectorSize) { Value.Float(0f) }) + return FloatVectorDescriptor(UUID.randomUUID(), UUID.randomUUID(), Value.FloatVector(FloatArray(vectorSize))) } override fun newRetrieverForQuery(field: Schema.Field, query: Query, context: QueryContext): Retriever { require(field.analyser == this) { "The field '${field.fieldName}' analyser does not correspond with this analyser. This is a programmer's error!" } - require(query is ProximityQuery<*> && query.value.first() is Value.Float) { } + require(query is ProximityQuery<*> && query.value is Value.FloatVector) { } /* Construct query. */ @Suppress("UNCHECKED_CAST") - return SphericalHarmonicsRetriever(field, query as ProximityQuery, context) + return SphericalHarmonicsRetriever(field, query as ProximityQuery, context) } override fun newRetrieverForDescriptors(field: Schema.Field, descriptors: Collection, context: QueryContext): Retriever { From c62af4d273bc5dae6ea66b681ec289200de029cc Mon Sep 17 00:00:00 2001 From: Ralph Gasser Date: Fri, 26 Jul 2024 10:43:52 +0200 Subject: [PATCH 71/71] Changes RGBByteColorContainer to a value class to me more memory efficient (i.e., prevent millions of allocations in AverageColor.analyse()). Signed-off-by: Ralph Gasser --- .../features/averagecolor/AverageColor.kt | 2 +- .../core/model/color/RGBByteColorContainer.kt | 48 ++++++++++++++----- .../image/AverageImageContentAggregator.kt | 2 +- .../RepresentativeImageContentAggregator.kt | 4 +- 4 files changed, 41 insertions(+), 15 deletions(-) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColor.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColor.kt index 4a1e3816..9d9769e3 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColor.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/features/averagecolor/AverageColor.kt @@ -44,7 +44,7 @@ class AverageColor : Analyser { fun analyse(content: Collection): List = content.map { val color = MutableRGBFloatColorContainer() val rgb = it.content.getRGBArray() - rgb.forEach { c -> color += RGBByteColorContainer.fromRGB(c) } + rgb.forEach { c -> color += RGBByteColorContainer(c) } /* Generate descriptor. */ val averageColor = RGBFloatColorContainer(color.red / rgb.size, color.green / rgb.size, color.blue / rgb.size) diff --git a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/color/RGBByteColorContainer.kt b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/color/RGBByteColorContainer.kt index 5bdc74f3..8db668c1 100644 --- a/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/color/RGBByteColorContainer.kt +++ b/vitrivr-engine-core/src/main/kotlin/org/vitrivr/engine/core/model/color/RGBByteColorContainer.kt @@ -1,19 +1,45 @@ package org.vitrivr.engine.core.model.color -data class RGBByteColorContainer(val red: UByte, val green: UByte, val blue: UByte) { +/** + * A container for RGB colors. + * + * @version 1.1.0 + * @author Luca Rossetto + * @author Ralph Gasser + */ +@JvmInline +value class RGBByteColorContainer(private val rgb: Int) { + /** + * Constructor for creating a [RGBByteColorContainer] from RGB [UByte] values. + * + * @param red Red component. + * @param green Green component. + * @param blue Blue component. + */ + constructor(red: UByte, green: UByte, blue: UByte) : this(blue.toInt() and 0xFF or ((green.toInt() and 0xFF) shl 8) or ((red.toInt() and 0xFF) shl 16)) - companion object { - fun fromRGB(rgb: Int): RGBByteColorContainer = - RGBByteColorContainer( - (rgb shr 16 and 0xFF).toUByte(), - (rgb shr 8 and 0xFF).toUByte(), - (rgb and 0xFF).toUByte() - ) - } + /** Red component of this [RGBByteColorContainer]. */ + val red: UByte + get() = (this.rgb shr 16 and 0xFF).toUByte() + /** Green component of this [RGBByteColorContainer]. */ + val green: UByte + get() = (this.rgb shr 8 and 0xFF).toUByte() + + /** Blue component of this [RGBByteColorContainer]. */ + val blue: UByte + get() = (this.rgb and 0xFF).toUByte() + + /** + * + */ fun toFloatContainer(): RGBFloatColorContainer = RGBFloatColorContainer(this.red.toFloat() / 255f, this.green.toFloat() / 255f, this.blue.toFloat() / 255f) - fun toRGBInt(): Int = this.blue.toInt() and 0xFF or ((this.green.toInt() and 0xFF) shl 8) or ((this.red.toInt() and 0xFF) shl 16) - + /** + * Converts this [RGBByteColorContainer] to an [Int] representation. + * + * @return [Int] representation of this [RGBByteColorContainer]. + */ + fun toRGBInt(): Int = this.rgb } diff --git a/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/aggregators/image/AverageImageContentAggregator.kt b/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/aggregators/image/AverageImageContentAggregator.kt index 12b0be36..72d0d09f 100644 --- a/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/aggregators/image/AverageImageContentAggregator.kt +++ b/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/aggregators/image/AverageImageContentAggregator.kt @@ -56,7 +56,7 @@ class AverageImageContentAggregator : TransformerFactory { images.forEach { imageContent -> require(imageContent.height == height && imageContent.width == width) { "Unable to aggregate images! All images must have same dimension." } imageContent.content.getRGBArray().forEachIndexed { index, color -> - colors[index] += RGBByteColorContainer.fromRGB(color) + colors[index] += RGBByteColorContainer(color) } } diff --git a/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/aggregators/image/RepresentativeImageContentAggregator.kt b/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/aggregators/image/RepresentativeImageContentAggregator.kt index 04bf7cb6..e91c9ccf 100644 --- a/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/aggregators/image/RepresentativeImageContentAggregator.kt +++ b/vitrivr-engine-index/src/main/kotlin/org/vitrivr/engine/index/aggregators/image/RepresentativeImageContentAggregator.kt @@ -56,7 +56,7 @@ class RepresentativeImageContentAggregator : TransformerFactory { images.forEach { imageContent -> require(imageContent.height == height && imageContent.width == width) { "Unable to aggregate images! All images must have same dimension." } imageContent.content.getRGBArray().forEachIndexed { index, color -> - colors[index] += RGBByteColorContainer.fromRGB(color) + colors[index] += RGBByteColorContainer(color) } } @@ -68,7 +68,7 @@ class RepresentativeImageContentAggregator : TransformerFactory { val mostRepresentative = images.minBy { imageContent -> imageContent.content.getRGBArray().mapIndexed { index, color -> - RGBByteColorContainer.fromRGB(color).toFloatContainer().distanceTo(colors[index]) + RGBByteColorContainer(color).toFloatContainer().distanceTo(colors[index]) }.sum() }