Issue #8 Add ScopeLink benchmarks #13
Conversation
ScopeLink benchmarks: - create ScopeLink using variable list declaration - create ScopeLink without variable list declaration - create ScopeLink using LambdaLink - add same ScopeLink into atomspace few times - add number of unique ScopeLinks into atomspace EvaluationLink benchmarks: - create EvaluationLink to compare with ScopeLink - add same EvaluationLink into atomspace few times - add number of unique EvaluationLinks into atomspace
| AtomSpace atomspace; | ||
| Handle X = atomspace.add_node(VARIABLE_NODE, "$X"); | ||
| Handle P = atomspace.add_node(PREDICATE_NODE, "P"); | ||
|
|
There was a problem hiding this comment.
This isn't really about EvaluationLinks, per se, but links-in-general. you could have used ListLink, instead.
There was a problem hiding this comment.
Oh, my comments is not right. EvaluationLink causes the FreeLink factory to run; but an ordinary ListLink would not. Some of the free-link stuff gets more expensive as the evaluationlink has more variables in it,. I dunno.
| for (auto _ : state) | ||
| { | ||
| atomspace.add_atom(links[i++ % number_of_links]); | ||
| } |
There was a problem hiding this comment.
What you are measuring here is a totally-disjoint graph. Pretend that node==vertex, and link==edge. You've created a large number of totally disconnected directed edges: * -->-- *. You've created 2N vertexes, and N edges connecting them together.
But real-world data is not like that. There are three or four more interesting cases:
- the clique -- N vertexes, and N^2 edges connecting them.
- the bipartite graph -- N left(start) vertexes, M right(finish) vertexes, and NM total edges,
- the scale-free network -- N vertexes, listed in order, with N edges from the first vertex to all of the others. N/2 edges from the second vertex to half the others. N/2 edges from the third vertex to half the others. N/4 edges from vertex 4,5,6,7 to random others. The N/8 edges fro 8 vertexes to random samplings of 1/8th of the others, etc.
I think I described the scale-free network correctly, There are several variations on scale-free. Natural language is scale-free. One of my natural-language datasets will not fit into 256GB RAM when it runs, it blows out all L1, L2, L3 caches, etc. For me, this is the most interesting benchmark.
For PLN, logic, I am not sure of what the typical graph/edge network they have.
There was a problem hiding this comment.
Oh, also - for the scale free network, the other interesting task is to walk it, randomly following edges, and then update the truth value for each vertex as you encounter it, while walking the network. Say, increment the count by one. This neither adds nor removes from the atomspace. But it does for the incoming sets to work hard, and the incoming sets will be all sorts of very different sizes.
There was a problem hiding this comment.
Which is another interesting point: in the totally-disconnected graph you created, the incoming sets are almost trivial. For the clique and the bipartite graph, they are always very large. Again, the scale-free network is interesting... and this affects the earlier benchmark: BM_CreateEvaluationLink - the create will run faster, when the vertexes are maximally independent. It will run slowly when they're no. I think in that benchmark, you create just two vertexes, and then create N edges between them. When you measure performance, I think you will see N logN behavior (or something like that..), because the incoming set is stored as a c++ std::set of weak pointers.
There was a problem hiding this comment.
Agree that in current state BM_AddEvaluationLink and BM_AddScopeLink are not very interesting. I wrote them before I looked to Atomspace code. I thought about removing it but found that after size of atomspace is increased significally performance decreases. I have commited tests with different atomspace sizes to demonstrate the difference:
$ ./atomspace/atomspace/benchmark --benchmark_filter='AddScope|AddEval'
2018-05-31 15:10:48
Running ./atomspace/atomspace/benchmark
Run on (12 X 4000 MHz CPU s)
CPU Caches:
L1 Data 32K (x6)
L1 Instruction 32K (x6)
L2 Unified 256K (x6)
L3 Unified 12288K (x1)
------------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------------
BM_AddScopeLink/4096 13869 ns 13869 ns 37463
BM_AddScopeLink/8192 15389 ns 15389 ns 41657
BM_AddScopeLink/16384 22546 ns 22545 ns 24945
BM_AddEvaluationLink/16384 5839 ns 5839 ns 108308
BM_AddEvaluationLink/32768 7933 ns 7933 ns 65748
BM_AddEvaluationLink/65536 11290 ns 11290 ns 62715
There was a problem hiding this comment.
Using scale-free network looks interesting, will add it to look what difference does it make.
|
The code looks reasonable to me. I'll let Nil look at it and merge. I made comments, but those don't need to be fixed before merging. Nil might want to review the complexity of the ScopeLink -- it seems to be over-simplified, and not very much like a "typical" scope link. |
|
re |
|
Anyway, I like it. Results on scale-free graphs might be the most useful, the most real-life-like |
|
I have found that numbers above are not correct as I used debug versions of libraries by accident. Correct numbers are below and they are ten times better: |
|
@vsbogd Just curious, what was compiled in debug mode? google benchmark? atomspace? cogutil? opencog benchmark? All? |
|
atomspace and cogutil - I have two versions built: debug and release and debug was loaded because of LD_LIBRARY_PRELOAD override. |
|
Nevertheless profile analysis gives the same results: EvaluationLink adding: |
|
It looks more or less as I would expect. |
Issue #8
I have added benchmarks for creating and adding ScopeLink into atomspace. I have added same EvaluationLink tests to compare performance.
(0) Benchmark results (atomspace and cogutil debug versions) see release results below in comment #13 (comment):
(1) I have used Google Benchmark to create this stuff but it can be discussed.
Pros:
Contras:
(2) I have found comparing BM_AddSameScopeLink and BM_AddSameEvaluationLink profiles that main additional complexity for ScopeLink adding is copying ScopeLink. It leads to second ScopeLink.init() method call which is heavy.
BM_AddSameScopeLink profile:
BM_AddSameEvaluationLink profile: