feat: add support to pgvector #267

kzamlynska · 2025-01-08T08:30:50Z

No description provided.

github-actions · 2025-01-08T08:32:26Z

Code Coverage Summary

Filename                                                                                                      Stmts    Miss  Cover    Missing
----------------------------------------------------------------------------------------------------------  -------  ------  -------  ------------------------------------------------------
packages/__init__.py                                                                                              0       0  100.00%
packages/ragbits-cli/src/ragbits/cli/__init__.py                                                                 31       4  87.10%   73-74, 81-82
packages/ragbits-cli/src/ragbits/cli/_utils.py                                                                   23       4  82.61%   45, 62-64
packages/ragbits-cli/src/ragbits/cli/state.py                                                                    55       8  85.45%   48-49, 61, 64-65, 108, 115-116
packages/ragbits-conversations/src/ragbits/conversations/__init__.py                                              0       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/__init__.py                                      0       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/compressors/__init__.py                          3       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/compressors/base.py                             10       0  100.00%
packages/ragbits-conversations/src/ragbits/conversations/history/compressors/llm.py                              29       1  96.55%   79
packages/ragbits-conversations/tests/unit/history/test_llm_compressor.py                                         64       0  100.00%
packages/ragbits-core/src/ragbits/core/__init__.py                                                                6       2  66.67%   8-9
packages/ragbits-core/src/ragbits/core/cli.py                                                                     6       0  100.00%
packages/ragbits-core/src/ragbits/core/config.py                                                                 17       0  100.00%
packages/ragbits-core/src/ragbits/core/options.py                                                                17       0  100.00%
packages/ragbits-core/src/ragbits/core/types.py                                                                   9       0  100.00%
packages/ragbits-core/src/ragbits/core/audit/__init__.py                                                         74       5  93.24%   42-45, 52-54
packages/ragbits-core/src/ragbits/core/audit/base.py                                                             44       1  97.73%   119
packages/ragbits-core/src/ragbits/core/audit/cli.py                                                              69       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/__init__.py                                                     4       0  100.00%
packages/ragbits-core/src/ragbits/core/embeddings/base.py                                                        20       2  90.00%   56, 69
packages/ragbits-core/src/ragbits/core/embeddings/exceptions.py                                                  17       7  58.82%   7-8, 17, 26-27, 36, 45
packages/ragbits-core/src/ragbits/core/embeddings/litellm.py                                                     38      18  52.63%   78-112
packages/ragbits-core/src/ragbits/core/embeddings/noop.py                                                         8       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/__init__.py                                                           4       0  100.00%
packages/ragbits-core/src/ragbits/core/llms/base.py                                                              66       8  87.88%   61, 73, 174, 193-201, 205
packages/ragbits-core/src/ragbits/core/llms/exceptions.py                                                        17       7  58.82%   7-8, 17, 26-27, 36, 45
packages/ragbits-core/src/ragbits/core/llms/factory.py                                                           12       3  75.00%   30, 41, 51
packages/ragbits-core/src/ragbits/core/llms/litellm.py                                                           95      29  69.47%   94, 100-121, 165, 175, 204-229, 251-256, 267
packages/ragbits-core/src/ragbits/core/llms/local.py                                                             56      27  51.79%   9-12, 66-74, 86-87, 108-119, 140-156
packages/ragbits-core/src/ragbits/core/llms/mock.py                                                              29       9  68.97%   63-74
packages/ragbits-core/src/ragbits/core/metadata_stores/__init__.py                                                3       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/base.py                                                   11       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/exceptions.py                                              4       0  100.00%
packages/ragbits-core/src/ragbits/core/metadata_stores/in_memory.py                                              16       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/__init__.py                                                         2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/_cli.py                                                            44      21  52.27%   25-33, 47-49, 63-65, 73-75, 89-97
packages/ragbits-core/src/ragbits/core/prompt/base.py                                                            20       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/parsers.py                                                         35       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/prompt.py                                                         126       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/__init__.py                                               2       0  100.00%
packages/ragbits-core/src/ragbits/core/prompt/discovery/prompt_discovery.py                                      33       2  93.94%   54-55
packages/ragbits-core/src/ragbits/core/utils/__init__.py                                                          0       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/_pyproject.py                                                       38       1  97.37%   113
packages/ragbits-core/src/ragbits/core/utils/config_handling.py                                                  72       8  88.89%   16, 54-55, 62-63, 152-154
packages/ragbits-core/src/ragbits/core/utils/decorators.py                                                       29       0  100.00%
packages/ragbits-core/src/ragbits/core/utils/dict_transformations.py                                             72       3  95.83%   24, 27, 108
packages/ragbits-core/src/ragbits/core/vector_stores/__init__.py                                                  3       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/_cli.py                                                     51       4  92.16%   63, 85, 91, 121
packages/ragbits-core/src/ragbits/core/vector_stores/base.py                                                     40       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/chroma.py                                                   59       1  98.31%   77
packages/ragbits-core/src/ragbits/core/vector_stores/in_memory.py                                                38       0  100.00%
packages/ragbits-core/src/ragbits/core/vector_stores/pgvector.py                                                124      12  90.32%   173-181, 215-217, 289
packages/ragbits-core/src/ragbits/core/vector_stores/qdrant.py                                                   66       1  98.48%   81
packages/ragbits-core/tests/cli/__init__.py                                                                       0       0  100.00%
packages/ragbits-core/tests/cli/test_cli_trace_handler.py                                                        48       3  93.75%   30, 43, 56
packages/ragbits-core/tests/cli/test_vector_store.py                                                            105       0  100.00%
packages/ragbits-core/tests/integration/vector_stores/test_vector_store.py                                       31       0  100.00%
packages/ragbits-core/tests/unit/__init__.py                                                                      0       0  100.00%
packages/ragbits-core/tests/unit/test_options.py                                                                 21       0  100.00%
packages/ragbits-core/tests/unit/audit/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/audit/test_cli.py                                                              102       0  100.00%
packages/ragbits-core/tests/unit/audit/test_trace.py                                                             93       3  96.77%   14, 17, 20
packages/ragbits-core/tests/unit/embeddings/test_from_config.py                                                  14       0  100.00%
packages/ragbits-core/tests/unit/llms/__init__.py                                                                 0       0  100.00%
packages/ragbits-core/tests/unit/llms/test_from_config.py                                                        16       0  100.00%
packages/ragbits-core/tests/unit/llms/test_litellm.py                                                            70       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/__init__.py                                                         0       0  100.00%
packages/ragbits-core/tests/unit/llms/factory/test_get_default_llm.py                                            12       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/__init__.py                                                      0       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/test_from_config.py                                             11       0  100.00%
packages/ragbits-core/tests/unit/metadata_stores/test_in_memory.py                                               22       0  100.00%
packages/ragbits-core/tests/unit/prompts/__init__.py                                                              0       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_parsers.py                                                         65       0  100.00%
packages/ragbits-core/tests/unit/prompts/test_prompt.py                                                         165       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/__init__.py                                                    0       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/prompt_classes_for_tests.py                                   30       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/test_prompt_discovery.py                                      18       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/__init__.py                     2       1  50.00%   3
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/__init__.py             3       2  33.33%   2-4
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt1.py        14       0  100.00%
packages/ragbits-core/tests/unit/prompts/discovery/ragbits_tests_pkg_with_prompts/prompts/temp_prompt2.py        14       0  100.00%
packages/ragbits-core/tests/unit/utils/__init__.py                                                                0       0  100.00%
packages/ragbits-core/tests/unit/utils/test_config_handling.py                                                   65       2  96.92%   27-28
packages/ragbits-core/tests/unit/utils/test_decorators.py                                                        26       2  92.31%   17, 39
packages/ragbits-core/tests/unit/utils/test_dict_transformations.py                                              69       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_find.py                                                    13       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_config.py                                               9       0  100.00%
packages/ragbits-core/tests/unit/utils/pyproject/test_get_instace.py                                             37       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/__init__.py                                                        0       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_chroma.py                                                    61       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_from_config.py                                               42       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_in_memory.py                                                 84       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_pgvector.py                                                 173       0  100.00%
packages/ragbits-core/tests/unit/vector_stores/test_qdrant.py                                                    40       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/__init__.py                                          2       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/_main.py                                           119       4  96.64%   266-273, 276
packages/ragbits-document-search/src/ragbits/document_search/cli.py                                              28       1  96.43%   78
packages/ragbits-document-search/src/ragbits/document_search/documents/__init__.py                                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/documents/document.py                               61       2  96.72%   99, 146
packages/ragbits-document-search/src/ragbits/document_search/documents/element.py                                78      12  84.62%   87, 167-174, 183-185
packages/ragbits-document-search/src/ragbits/document_search/documents/exceptions.py                             11       5  54.55%   7-8, 17, 26-27
packages/ragbits-document-search/src/ragbits/document_search/documents/source_resolver.py                        20       5  75.00%   5, 51-52, 55-56
packages/ragbits-document-search/src/ragbits/document_search/documents/sources.py                               180      15  91.67%   166, 204, 246, 383-386, 390-391, 421, 428-429, 443-445
packages/ragbits-document-search/src/ragbits/document_search/ingestion/__init__.py                                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/document_processor.py                     33       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/__init__.py           5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/base.py              25       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/batched.py           18       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/distributed.py       30       8  73.33%   8-9, 36, 64-71
packages/ragbits-document-search/src/ragbits/document_search/ingestion/processor_strategies/sequential.py        13       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/__init__.py                      3       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/base.py                         19       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/dummy.py                        20       7  65.00%   33, 54-60
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/__init__.py         4       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/default.py         46       4  91.30%   98, 103-104, 137
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/images.py          50      19  62.00%   73-80, 87-99, 111, 124
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/pdf.py             19       6  68.42%   23, 35-43
packages/ragbits-document-search/src/ragbits/document_search/ingestion/providers/unstructured/utils.py           38      11  71.05%   71, 82-83, 98-101, 110, 121-123
packages/ragbits-document-search/src/ragbits/document_search/retrieval/__init__.py                                0       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/__init__.py                     5       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/base.py                         9       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/llm.py                         25       4  84.00%   47-50
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/noop.py                         6       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rephrasers/prompts.py                     16       1  93.75%   52
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/__init__.py                      3       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/base.py                         17       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/litellm.py                      16       0  100.00%
packages/ragbits-document-search/src/ragbits/document_search/retrieval/rerankers/noop.py                          9       0  100.00%
packages/ragbits-document-search/tests/__init__.py                                                                0       0  100.00%
packages/ragbits-document-search/tests/helpers.py                                                                 3       0  100.00%
packages/ragbits-document-search/tests/cli/test_search.py                                                        56       0  100.00%
packages/ragbits-document-search/tests/integration/__init__.py                                                    0       0  100.00%
packages/ragbits-document-search/tests/integration/test_rerankers.py                                             15       6  60.00%   18-38
packages/ragbits-document-search/tests/integration/test_sources.py                                               23      10  56.52%   22-32, 40-45
packages/ragbits-document-search/tests/integration/test_unstructured.py                                          48      10  79.17%   52-58, 71-77
packages/ragbits-document-search/tests/unit/test_config.py                                                       57       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_processor.py                                           17       0  100.00%
packages/ragbits-document-search/tests/unit/test_document_search.py                                             233       1  99.57%   466
packages/ragbits-document-search/tests/unit/test_documents.py                                                    13       0  100.00%
packages/ragbits-document-search/tests/unit/test_elements.py                                                     19       0  100.00%
packages/ragbits-document-search/tests/unit/test_local_file_source.py                                            13       0  100.00%
packages/ragbits-document-search/tests/unit/test_processing_strategies.py                                        25       0  100.00%
packages/ragbits-document-search/tests/unit/test_providers.py                                                    41       0  100.00%
packages/ragbits-document-search/tests/unit/test_rephrasers.py                                                   26       0  100.00%
packages/ragbits-document-search/tests/unit/test_rerankers.py                                                    51       1  98.04%   23
packages/ragbits-document-search/tests/unit/test_source_discriminator.py                                         35       0  100.00%
packages/ragbits-document-search/tests/unit/test_sources.py                                                      52       8  84.62%   20-21, 56-61
packages/ragbits-document-search/tests/unit/testprojects/project_with_instance_factory/__init__.py                0       0  100.00%
packages/ragbits-document-search/tests/unit/testprojects/project_with_instance_factory/factories.py              20       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/__init__.py                                                    0       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/base.py                                                       15       0  100.00%
packages/ragbits-guardrails/src/ragbits/guardrails/openai_moderation.py                                          19       5  73.68%   29-33
packages/ragbits-guardrails/tests/unit/test_openai_moderation.py                                                 35       0  100.00%
TOTAL                                                                                                          4930     346  92.98%

Diff against main

Filename                                                            Stmts    Miss  Cover
----------------------------------------------------------------  -------  ------  --------
packages/ragbits-core/src/ragbits/core/vector_stores/pgvector.py     +124     +12  +90.32%
packages/ragbits-core/tests/unit/vector_stores/test_pgvector.py      +173       0  +100.00%
TOTAL                                                                +297     +12  +0.19%

Results for commit: 7c8a95f

Minimum allowed coverage is 60%

♻️ This comment has been updated with latest results

github-actions · 2025-01-08T08:35:24Z

Trivy scanning results.

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: JWT (jwt-token)
════════════════════════════════════════
JWT token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA:80
────────────────────────────────────────
78 >>> encoded = jwt.encode({"some": "payload"}, "secret", algorithm="HS256")
79 >>> print(encoded)
80 [ *********************************************************************************************************
81 >>> jwt.decode(encoded, "secret", algorithms=["HS256"])
────────────────────────────────────────

.venv/lib/python3.10/site-packages/litellm/llms/huggingface/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

Total: 1 (MEDIUM: 0, HIGH: 0, CRITICAL: 1)

CRITICAL: HuggingFace (hugging-face-access-token)
════════════════════════════════════════
Hugging Face Access Token
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/llms/huggingface/huggingface_llms_metadata/hf_text_generation_models.txt:36162
────────────────────────────────────────
36160 mncai/Llama2-7B-Active_3rd-floor-LoRA-dim64_epoch4
36161 ajcdp/CM
36162 [ Nagharjun17/*************************************
36163 BigSalmon/InformalToFormalLincoln114Paraphrase
────────────────────────────────────────

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

Total: 1 (MEDIUM: 1, HIGH: 0, CRITICAL: 0)

MEDIUM: Slack (slack-web-hook)
════════════════════════════════════════
Slack Webhook
────────────────────────────────────────
.venv/lib/python3.10/site-packages/litellm/proxy/_types.py:1314
────────────────────────────────────────
1312 alert_to_webhook_url: Optional[Dict] = Field(
1313 None,
1314 [ bhook_url: {'budget_alerts': '*****************************************************************************'}`",
1315 )
────────────────────────────────────────

packages/ragbits-core/src/ragbits/core/vector_stores/pgvector.py

mhordynski

lgtm, good job 🏅

kzamlynska added 3 commits December 30, 2024 11:58

add asyncpg to the requirements

1a081c9

pgVector in VectoreStore

514de37

pre commit checks

137c543

micpst linked an issue Jan 8, 2025 that may be closed by this pull request

feat: add support to pgvector #253

Closed

micpst changed the title ~~Kz/253 pgvector support~~ feat: add support to pgvector Jan 8, 2025

kzamlynska marked this pull request as draft January 8, 2025 15:25

ludwiktrammer reviewed Jan 8, 2025

View reviewed changes

kzamlynska added 9 commits January 8, 2025 19:10

remove connect methods and config class

03896ef

check if table exist

cd725c8

connection as argument

6a97150

commit checks

4aec79b

check table name

79bb2a9

commit checks

474bd5e

Merge branch 'main' into kz/253-pgvector-support

714a8a9

unit tests for pgvector

f2b5dc1

pre commit formatting

204b4cb

kzamlynska marked this pull request as ready for review January 10, 2025 08:49

kzamlynska added 7 commits January 14, 2025 14:39

Merge branch 'main' into kz/253-pgvector-support

323bd59

create table and index in one transaction

278b5ad

Merge branch 'main' into kz/253-pgvector-support

a206fc4

correct unit test

54d039f

correct unit test

5efd81d

formatting

a070a49

no table creation with list, remove abnd retrieve

2ac6d0f

ludwiktrammer reviewed Jan 20, 2025

View reviewed changes

packages/ragbits-core/src/ragbits/core/vector_stores/pgvector.py Outdated Show resolved Hide resolved

kzamlynska added 2 commits January 21, 2025 14:46

make SQL queries save again

7da67f7

precommit checks

3ac60b6

ludwiktrammer reviewed Jan 21, 2025

View reviewed changes

kzamlynska added 3 commits January 21, 2025 15:24

more sql queries corrections

598cc6c

adjusting pgvector tests

4827e26

Merge branch 'main' into kz/253-pgvector-support

88e3999

ludwiktrammer reviewed Jan 21, 2025

View reviewed changes

packages/ragbits-core/src/ragbits/core/vector_stores/pgvector.py Outdated Show resolved Hide resolved

kzamlynska added 13 commits January 22, 2025 15:13

Merge branch 'main' into kz/253-pgvector-support

523d9a8

correct sql queires once again

92f95f4

Merge branch 'main' into kz/253-pgvector-support

b1beb64

CREATE withour parameters

0d2ec05

reformat file

4f8d782

reformat file

d680ce0

reformat file

47fc68a

reformat file

e89f865

reformat file

69063fd

reformat file

5574aab

reformat file

40a845e

reformat file

0a09b26

reformat file

7c8a95f

mhordynski approved these changes Jan 24, 2025

View reviewed changes

kzamlynska merged commit e861024 into main Jan 24, 2025
6 checks passed

kzamlynska deleted the kz/253-pgvector-support branch January 24, 2025 10:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add support to pgvector #267

feat: add support to pgvector #267

kzamlynska commented Jan 8, 2025

github-actions bot commented Jan 8, 2025 •

edited

Loading

github-actions bot commented Jan 8, 2025 •

edited

Loading

mhordynski left a comment

feat: add support to pgvector #267

feat: add support to pgvector #267

Conversation

kzamlynska commented Jan 8, 2025

github-actions bot commented Jan 8, 2025 • edited Loading

Code Coverage Summary

Diff against main

github-actions bot commented Jan 8, 2025 • edited Loading

.venv/lib/python3.10/site-packages/PyJWT-2.9.0.dist-info/METADATA (secrets)

.venv/lib/python3.10/site-packages/litellm/llms/huggingface/huggingface_llms_metadata/hf_text_generation_models.txt (secrets)

.venv/lib/python3.10/site-packages/litellm/proxy/_types.py (secrets)

mhordynski left a comment

Choose a reason for hiding this comment

github-actions bot commented Jan 8, 2025 •

edited

Loading

github-actions bot commented Jan 8, 2025 •

edited

Loading