Testing performance improvements #504
Conversation
.../main/java/com/the_qa_company/qendpoint/core/dictionary/impl/MultipleLangBaseDictionary.java
Show resolved
Hide resolved
qendpoint-store/src/main/java/com/the_qa_company/qendpoint/compiler/SparqlRepository.java
Show resolved
Hide resolved
| @@ -184,7 +184,7 @@ public EndpointStore(EndpointFiles files, HDTOptions spec, boolean inMemDeletes, | |||
| throws IOException { | |||
| // load HDT file | |||
| this.spec = (spec = HDTOptions.ofNullable(spec)); | |||
| deleteDisabled = spec.getBoolean(OPTION_QENDPOINT_DELETE_DISABLE, false); | |||
| deleteDisabled = spec.getBoolean(OPTION_QENDPOINT_DELETE_DISABLE, true); | |||
There was a problem hiding this comment.
I must be doing something wrong somewhere. With the HDT and index files I got from Dennis I kept getting an exception related to the part of the code that handles deleted statements. And when I tried to change the config in the application.properties file it didn't seem to make a difference. So I explicitly disabled it in order to be able to test my queries.
There was a problem hiding this comment.
If you want to set a config, you can add it inside the repo_model.ttl file in the triple
mdlc:main mdlc:hdtSpec "qendpoint.delete.disable=true;" .By default, we need the deletes, so the default value should be false.
| @Override | ||
| public void meet(StatementPattern node) throws RuntimeException { | ||
| Var subjectVar = node.getSubjectVar(); | ||
| if (subjectVar != null && subjectVar.isAnonymous() && subjectVar.hasValue()) { | ||
| long id = converter.subjectToID(((Resource) subjectVar.getValue())); | ||
| if (id != -1) { | ||
| Var var1 = new Var(subjectVar.getName(), converter.idToSubjectHDTResource(id), true, | ||
| subjectVar.isConstant()); | ||
| node.replaceChildNode(subjectVar, var1); | ||
| } | ||
| } | ||
|
|
||
| Var predicateVar = node.getPredicateVar(); | ||
| if (predicateVar != null && predicateVar.isAnonymous() && predicateVar.hasValue()) { | ||
| long id = converter.predicateToID(((IRI) predicateVar.getValue())); | ||
| if (id != -1) { | ||
| Var var1 = new Var(predicateVar.getName(), converter.idToPredicateHDTResource(id), true, | ||
| predicateVar.isConstant()); | ||
| node.replaceChildNode(predicateVar, var1); | ||
| } | ||
| } | ||
|
|
||
| Var objectVar = node.getObjectVar(); | ||
| if (objectVar != null && objectVar.isAnonymous() && objectVar.hasValue()) { | ||
| long id = converter.objectToID((objectVar.getValue())); | ||
| if (id != -1) { | ||
| Var var1 = new Var(objectVar.getName(), converter.idToObjectHDTResource(id), true, | ||
| objectVar.isConstant()); | ||
| node.replaceChildNode(objectVar, var1); | ||
| } | ||
| } | ||
|
|
||
| Var contextVar = node.getContextVar(); | ||
| if (contextVar != null && contextVar.isAnonymous() && contextVar.hasValue()) { | ||
| long id = converter.contextToID((((Resource) contextVar.getValue()))); | ||
| if (id != -1) { | ||
| Var var1 = new Var(contextVar.getName(), converter.idToGraphHDTResource(id), true, | ||
| contextVar.isConstant()); | ||
| node.replaceChildNode(contextVar, var1); | ||
| } | ||
| } | ||
|
|
||
| } | ||
|
|
There was a problem hiding this comment.
Here is the addition where I optimize each var in the StatementPattern. For some reason the code in meet(Var var) didn't manage to make the predicate have the correct position. When working on the StatementPattern we already know which position every var has.
There was a problem hiding this comment.
Did you try to use a constant outside a triple to see if it works? For example in a filter.
There was a problem hiding this comment.
I haven't tried that. But I assume it would match with the existing meet method for vars. I haven't removed that one, since the meet on the statement pattern doesn't work for things like BIND or FILTER.
| const container = document.createElement("pre") | ||
|
|
||
| container.innerText = JSON.stringify(plan, null, 2) | ||
|
|
There was a problem hiding this comment.
Can you apply the formatter? npm run format
There was a problem hiding this comment.
Formatter didn't seem to do anything, but IntelliJ highlighted the issues and I fixed them by hand.
qendpoint-store/src/main/java/com/the_qa_company/qendpoint/store/EndpointStore.java
Outdated
Show resolved
Hide resolved
| } | ||
|
|
||
| } | ||
|
|
There was a problem hiding this comment.
I would be reassured with a small test if you can.
| String property = System.getProperty("prototypejoin"); | ||
| System.err.println("prototypejoin: " + property); | ||
| if (property == null || !property.equals("true")) { | ||
| System.out.println("prototypejoin is not enabled"); | ||
| return super.prepare(node, context); | ||
| } |
There was a problem hiding this comment.
To use the new prototype join:
java -Dprototypejoin=true -jar qendpoint-backend-2.1.2-exec.jar
|
Can I merge it or do you want to add something else? |
|
Don't merge this one. This is the branch with everything on it. I've created two different branches/PRs with the two sets of changes that we talked about in the meeting. |
|
You can merge this one here: #527 |
This is WIP for testing various performance improvements.
First commit contains various small changes to make the code build on my machine and to make it easier for me to test things.
The next commits contain different improvements that make a significant difference to the performance.