[DERCBOT-1267] Gen AI - Document Compressor #1788

assouktim · 2024-11-18T15:16:06Z

This pull request adds support for a new Document Compressor feature in the bot administration service. The changes include adding new data models, services, and API endpoints to manage the Document Compressor configuration. Additionally, some existing code has been updated to integrate this new functionality.

Document Compressor Feature:

bot/admin/server/src/main/kotlin/BotAdminService.kt: Added BotDocumentCompressorConfigurationDAO and implemented methods to delete the Document Compressor configuration. [1] [2] [3]
bot/admin/server/src/main/kotlin/BotAdminVerticle.kt: Added new API endpoints to handle CRUD operations for the Document Compressor configuration.
bot/admin/server/src/main/kotlin/model/BotDocumentCompressorConfigurationDTO.kt: Created a new data transfer object for the Document Compressor configuration.
bot/admin/server/src/main/kotlin/service/DocumentCompressorService.kt: Implemented the service to manage Document Compressor configurations, including validation and error handling.
bot/admin/server/src/main/kotlin/service/DocumentCompressorValidationService.kt: Added a validation service to check the Document Compressor configuration settings.

Other Changes:

bot/connector-rest-client/src/main/kotlin/model/ClientFootnote.kt and bot/connector-web-model/src/main/kotlin/ai/tock/bot/connector/web/send/Footnote.kt: Added a new score field to the Footnote data class. [1] [2]
bot/connector-web/src/main/kotlin/WebMessageProcessor.kt and bot/connector-web/src/test/kotlin/WebConnectorResponseTest.kt: Updated the WebMessageProcessor and its test to handle the new score field in Footnote. [1] [2]
bot/engine/src/main/kotlin/admin/bot/compressor/BotDocumentCompressorConfiguration.kt and bot/engine/src/main/kotlin/admin/bot/compressor/BotDocumentCompressorConfigurationDAO.kt: Added new data models and DAO interfaces for the Document Compressor configuration. [1] [2]
bot/engine/src/main/kotlin/definition/BotDefinition.kt and bot/engine/src/main/kotlin/definition/BotDefinitionBase.kt: Updated the BotDefinition and BotDefinitionBase to include the Document Compressor configuration. [1] [2] [3] [4]
bot/engine/src/main/kotlin/engine/BotRepository.kt: Added import for BotDocumentCompressorConfigurationMonitor.

Benvii

Thanks for adding all the check document compressor settings to the orchestrator 👍️, juste a small comment about an exception raising that could be improved.

We begin to have a lot of Gen AI related configuration directly on the Bot Definition, I'm not sure but in a futur refactory I might be a good option to group them especially if we want to expose the to python scripted stories (tock-py) and also kotlin scripted stories. It would be great to have access to some BotDefinition configurations inside buildin sotries (stories behind the Bot API). Maybe we should create an issue about that.
My idea is in a near futur to expose Gen AI settings to build in stories (especially python, tock-py ones) so that python developpers could easily build there own langchain chains using the bot settings and the gen-ai-orchestrator factories provided in a separated python package.

We also have a lot of gen ai related configuration endpoint directly on the bot admin verticle we should in the futur group them in a different verticle.

Benvii · 2025-01-10T11:01:09Z

gen-ai/orchestrator-server/src/main/python/server/.run/Gen AI Orchestrator.run.xml

@@ -6,6 +6,7 @@
    <option name="PARENT_ENVS" value="true" />
    <envs>
      <env name="PYTHONUNBUFFERED" value="1" />
+      <env name="REQUESTS_CA_BUNDLE" value="/etc/ssl/certs/ca-certificates.crt" />


This shouldn't be committed here, put it in a dot env file instead.

Benvii · 2025-01-10T12:27:49Z

...server/src/gen_ai_orchestrator/services/langchain/impls/document_compressor/bloomz_rerank.py

+                    cause=f'Response: {response.text}, Reason: {response.reason}',
+                    request=f'[POST] {url}',
+                ))
+        except Exception as exc:


Here you will catch the already formatted exception inside the try block at L80 as GenAIDocumentCompressorErrorException inherits from Exception. This will lead to a wrapping of the exception inside an other GenAIDocumentCompressorErrorException object which is a bit strange and will make it difficult to handle it.

Instead you can do the following :

except GenAIDocumentCompressorErrorException: # Re-raise GenAIDocumentCompressorErrorException without modification raise except Exception as exc: # Catch all other exceptions logger.error(f'Unknown error ! {exc}') raise GenAIDocumentCompressorErrorException(ErrorInfo( error=exc.__class__.__name__, cause=str(exc), request=f'[POST] {url}', ))

yes, absolutely

Benvii · 2025-01-10T12:43:34Z

...hon/server/src/gen_ai_orchestrator/models/document_compressor/document_compressor_setting.py

+class BaseDocumentCompressorSetting(BaseModel):
+    provider: DocumentCompressorProvider = Field(
+        description='The document compressor provider.',
+        examples=[DocumentCompressorProvider.BLOOMZ],


It's a bit late for this comment but I think that all document compressor will have max_documents and min_score but we only have one kind of compressor for now so can't be sure this refactoring could still be done in the futur if we add other kind of compressors.

Benvii · 2025-01-10T12:44:53Z

...n/kotlin/ai/tock/genai/orchestratorcore/models/compressor/BloomzDocumentCompressorSetting.kt

+package ai.tock.genai.orchestratorcore.models.compressor
+
+data class BloomzDocumentCompressorSetting(
+    val maxDocuments: Int,


It's optional on the orchestrator side but we can make it required at the bot api / bot admin level. Same for label.

it is already required for Tock Studio, botApi and botAdmin

Benvii · 2025-01-10T12:46:49Z

gen-ai/orchestrator-core/src/main/kotlin/ai/tock/genai/orchestratorcore/models/Constants.kt

@@ -35,6 +37,7 @@ object Constants {

    const val GEN_AI_COMPLETION_SENTENCE_GENERATION="$GEN_AI_COMPLETION/sentenceGeneration"

+    const val GEN_AI_COMPRESSION="$GEN_AI/COMPRESSION"


Is it used somewhere ? Or is it for futur use in case we have a compressor with secrets ?

yes, I'm going to delete it, it's not in use now.

Benvii · 2025-01-10T12:48:22Z

gen-ai/orchestrator-core/src/main/kotlin/ai/tock/genai/orchestratorcore/models/Constants.kt

@@ -26,6 +26,8 @@ object Constants {
    const val OPEN_SEARCH = "OpenSearch"
    const val PG_VECTOR = "PGVector"

+    const val BLOOMZ = "BloomzRerank"


Bloomz can also be used a retriever / embedding model and also as guardrail model, this contant should have a more explicit name indicating that this is the compressor type.
Please don't apply the suggestion do a refactoring using an IDE.

Suggested change

const val BLOOMZ = "BloomzRerank"

const val BLOOMZ_COMPRESSOR = "BloomzRerank"

bot/admin/server/src/main/kotlin/service/ObservabilityValidationService.kt

assouktim · 2025-01-10T15:37:36Z

Thanks for adding all the check document compressor settings to the orchestrator 👍️, juste a small comment about an exception raising that could be improved.

We begin to have a lot of Gen AI related configuration directly on the Bot Definition, I'm not sure but in a futur refactory I might be a good option to group them especially if we want to expose the to python scripted stories (tock-py) and also kotlin scripted stories. It would be great to have access to some BotDefinition configurations inside buildin sotries (stories behind the Bot API). Maybe we should create an issue about that. My idea is in a near futur to expose Gen AI settings to build in stories (especially python, tock-py ones) so that python developpers could easily build there own langchain chains using the bot settings and the gen-ai-orchestrator factories provided in a separated python package.

We also have a lot of gen ai related configuration endpoint directly on the bot admin verticle we should in the futur group them in a different verticle.

Yes, we need to include it in the analysis when we refactor the orchestrator code. We need to put this in a separate ticket.

assouktim self-assigned this Nov 18, 2024

assouktim added the enhancement label Nov 18, 2024

assouktim force-pushed the feature/dercbot-1267 branch 2 times, most recently from cd83089 to 84e3c63 Compare November 25, 2024 13:18

rkuffer mentioned this pull request Nov 26, 2024

[Tock Studio] Documents compressor settings + score display #1795

Open

Benvii requested changes Jan 10, 2025

View reviewed changes

assouktim force-pushed the feature/dercbot-1267 branch from 84e3c63 to ea378c3 Compare January 10, 2025 14:41

assouktim added 2 commits January 10, 2025 16:50

[DERCBOT-1267] Gen AI - Document Compressor

a2b2244

[DERCBOT-1267] Reviews

a66b280

assouktim force-pushed the feature/dercbot-1267 branch from 98d7c19 to c9f9ccb Compare January 10, 2025 15:51

[DERCBOT-1267] poetry update

1a917bf

assouktim force-pushed the feature/dercbot-1267 branch from c9f9ccb to 1a917bf Compare January 13, 2025 08:57

[DERCBOT-1274] Kapt - Generated sources for the known intent threshold.

0eaa363

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DERCBOT-1267] Gen AI - Document Compressor #1788

[DERCBOT-1267] Gen AI - Document Compressor #1788

assouktim commented Nov 18, 2024 •

edited by Benvii

Loading

Benvii left a comment

Benvii Jan 10, 2025

Benvii Jan 10, 2025

assouktim Jan 10, 2025

Benvii Jan 10, 2025

Benvii Jan 10, 2025

assouktim Jan 10, 2025

Benvii Jan 10, 2025

assouktim Jan 10, 2025

Benvii Jan 10, 2025

assouktim commented Jan 10, 2025

		@@ -35,6 +37,7 @@ object Constants {

		const val GEN_AI_COMPLETION_SENTENCE_GENERATION="$GEN_AI_COMPLETION/sentenceGeneration"

		const val GEN_AI_COMPRESSION="$GEN_AI/COMPRESSION"

	const val BLOOMZ = "BloomzRerank"
	const val BLOOMZ_COMPRESSOR = "BloomzRerank"

[DERCBOT-1267] Gen AI - Document Compressor #1788

Are you sure you want to change the base?

[DERCBOT-1267] Gen AI - Document Compressor #1788

Conversation

assouktim commented Nov 18, 2024 • edited by Benvii Loading

Document Compressor Feature:

Other Changes:

Benvii left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

assouktim commented Jan 10, 2025

assouktim commented Nov 18, 2024 •

edited by Benvii

Loading