refactor pipelines to include contentSources #83
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
faberf
changed the title
…-engine into feature/contentpipelines
sauterl
reviewed
Jul 11, 2024
...engine-index/src/main/kotlin/org/vitrivr/engine/index/transform/ContentMergingTransformer.kt
Outdated
Show resolved
Hide resolved
…-engine into feature/contentpipelines
…files to be removed at a later stage.
…es' of github.com:vitrivr/vitrivr-engine into feature/contentpipelines
…her than descriptor to content.
…from template rather than given separately.
…-engine into feature/contentpipelines
…-engine into feature/contentpipelines
|
Hey, this PR is ready to review. We will add the documentation to the wiki once its merged. |
…-engine into feature/contentpipelines
…-engine into feature/contentpipelines
…-engine into feature/contentpipelines
|
It seems like the two concerns left unchanged in the latest commits are:
I have made the appropriate comments addressing these issues above. |
…-engine into feature/contentpipelines
Preventing naming conflicts e.g. "end"
|
What is the state of this PR? Have all issues been addressed? Currently, there are two merge-conflicts which ought to be resolved. Also, please make sure that all tests run successfully. |
Me and raphi will merge this today |
faberf
dismissed stale reviews from ppanopticon and lucaro
We discussed this in person.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR addresses the problem of multiple branches in a pipeline getting in each others way, by changing the use of retrievables to be append only. It is still the job of pipeline configurator to avoid race conditions by making sure that only parts of the retrievable are processed that are not being further updated. This is made easier via configurable contentSources which allow the configurator to specify which content should be used for an operator using a white list of other operators. This description is not very complete and the code needs to be cleaned up. Therefore this PR is a draft.