-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathbmcrock.temp.html
127 lines (111 loc) · 4.38 KB
/
bmcrock.temp.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
<html>
<title>
Benchmarks
</title>
<body>
<p>
<h3>
Benchmarks are a crock
</h3>
<p>
With modern superscalar architectures, 5-level
memory hierarchies, and wide data paths, changing the
alignment of instructions and data can easily change
the performance of a program by 20% or more, and Hans
Boehm has witnessed a spectacular 100% variation in
user CPU time while holding the executable file
constant.
Since much of this alignment is determined by the
linker, loader, and garbage collector, most
individual compiler optimizations are in the noise.
To evaluate a compiler properly, one must often look
at the code that it generates, not the timings.
<p>
<h4>Our benchmarks may not be representative</h4>
<p>
Many of our benchmarks test only a few aspects of performance.
Such benchmarks are good if your goal is to learn what an implementation
does well or not so well, which is our main concern.
Such benchmarks are not so good if your goal is to predict how well an
implementation will perform on "typical" Scheme programs.
<p>
Some of our benchmarks are derived from the computational kernels
of real programs, or contain modules that are derived from or known
to perform like the computational kernels of real programs:
<code><a href="benchmarksAboutR7.html#fft">fft</a></code>,
<code><a href="benchmarksAboutR7.html#nucleic">nucleic</a></code>,
<code><a href="benchmarksAboutR7.html#ray">ray</a></code>,
<code><a href="benchmarksAboutR7.html#simplex">simplex</a></code>,
<code><a href="benchmarksAboutR7.html#compiler">compiler</a></code>,
<code><a href="benchmarksAboutR7.html#conform">conform</a></code>,
<code><a href="benchmarksAboutR7.html#dynamic">dynamic</a></code>.
<code><a href="benchmarksAboutR7.html#earley">earley</a></code>,
<code><a href="benchmarksAboutR7.html#maze">maze</a></code>,
<code><a href="benchmarksAboutR7.html#parsing">parsing</a></code>,
<code><a href="benchmarksAboutR7.html#peval">peval</a></code>,
<code><a href="benchmarksAboutR7.html#scheme">scheme</a></code>,
<code><a href="benchmarksAboutR7.html#slatex">slatex</a></code>,
<code><a href="benchmarksAboutR7.html#nboyer">nboyer</a></code>,
<code><a href="benchmarksAboutR7.html#sboyer">sboyer</a></code>.
These benchmarks are not so good for determining what an implementation
does well or less well, because it may be hard to determine the reasons
for an unusually fast or slow timing.
If one of these benchmarks is similar to the programs that matter to you,
however, then it may be a good predictor of performance.
On the other hand, it may not be.
<p>
<h4>Real programs may not be representative either</h4>
<p>
The execution time of a program is often dominated by the time
spent in very small pieces of code. If an optimizing compiler happens
to do a particularly good job of optimizing these hot spots, then the
program will run quickly. If a compiler happens to do an unusually
poor job of optimizing one or more of these hot spots, then the program
will run slowly.
For example, compare
<code><a href="benchmarksAboutR7.html#takl">takl</a></code>
with
<code><a href="benchmarksAboutR7.html#ntakl">ntakl</a></code>,
or
<code><a href="benchmarksAboutR7.html#nboyer">nboyer</a></code>
with
<code><a href="benchmarksAboutR7.html#sboyer">sboyer</a></code>.
<p>
If the hot spots occur within library routines, then a compiler may
not affect the performance of the program very much. Its performance
may be determined by those library routines.
For example, consider the performance of <code>gcc</code> on the
<code><a href="benchmarksAboutR7.html#diviter">diviter</a></code>
or
<code><a href="benchmarksAboutR7.html#mperm">perm9</a></code>
benchmarks.
<p>
The performance of a benchmark, even if it is derived from a real program,
may not help to predict the performance of similar programs that have
different hot spots.
<p>
<h4>A note on C and C++</h4>
<p>
It is well known that C and C++ are faster than
any higher order or garbage collected language.
If some benchmark suggests otherwise, then this merely shows
that the author of that benchmark does not know how to write
efficient C code.
<p>
As an example of C code that is much faster than anything
that could be written in Scheme, I recommend
<blockquote>
<a href="http://www.cs.princeton.edu/~appel/">Andrew
W Appel</a>.
Intensional equality ;-) for continuations.
<cite>ACM SIGPLAN Notices</cite> 31(2),
February 1996, pages 55-57.
</blockquote>
<p>
<hr>
</p>
<p>
Last updated 7 March 2015.
</p>
</body>
</html>