123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108 |
- %M% %I% %E%
- The set of programs and documentation known as "lmbench" are distributed
- under the Free Software Foundation's General Public License with the
- following additional restrictions (which override any conflicting
- restrictions in the GPL):
- 1. You may not distribute results in any public forum, in any publication,
- or in any other way if you have modified the benchmarks.
- 2. You may not distribute the results for a fee of any kind. This includes
- web sites which generate revenue from advertising.
- If you have modifications or enhancements that you wish included in
- future versions, please mail those to me, Larry McVoy, at lm@bitmover.com.
- =========================================================================
- Rationale for the publication restrictions:
- In summary:
- a) LMbench is designed to measure enough of an OS that if you do well in
- all catagories, you've covered latency and bandwidth in networking,
- disks, file systems, VM systems, and memory systems.
- b) Multiple times in the past people have wanted to report partial results.
- Without exception, they were doing so to show a skewed view of whatever
- it was they were measuring (for example, one OS fit small processes into
- segments and used the segment register to switch them, getting good
- results, but did not want to report large process context switches
- because those didn't look as good).
- c) We insist that if you formally report LMbench results, you have to
- report all of them and make the raw results file easily available.
- Reporting all of them means in that same publication, a pointer
- does not count. Formally, in this context, means in a paper,
- on a web site, etc., but does not mean the exchange of results
- between OS developers who are tuning a particular subsystem.
- We have a lot of history with benchmarking and feel strongly that there
- is little to be gained and a lot to be lost if we allowed the results
- to be published in isolation, without the complete story being told.
- There has been a lot of discussion about this, with people not liking this
- restriction, more or less on the freedom principle as far as I can tell.
- We're not swayed by that, our position is that we are doing the right
- thing for the OS community and will stick to our guns on this one.
- It would be a different matter if there were 3 other competing
- benchmarking systems out there that did what LMbench does and didn't have
- the same reporting rules. There aren't and as long as that is the case,
- I see no reason to change my mind and lots of reasons not to do so. I'm
- sorry if I'm a pain in the ass on this topic, but I'm doing the right
- thing for you and the sooner people realize that the sooner we can get on
- to real work.
- Operating system design is a largely an art of balancing tradeoffs.
- In many cases improving one part of the system has negative effects
- on other parts of the system. The art is choosing which parts to
- optimize and which to not optimize. Just like in computer architecture,
- you can optimize the common instructions (RISC) or the uncommon
- instructions (CISC), but in either case there is usually a cost to
- pay (in RISC uncommon instructions are more expensive than common
- instructions, and in CISC common instructions are more expensive
- than required). The art lies in knowing which operations are
- important and optmizing those while minimizing the impact on the
- rest of the system.
- Since lmbench gives a good overview of many important system features,
- users may see the performance of the system as a whole, and can
- see where tradeoffs may have been made. This is the driving force
- behind the publication restriction: any idiot can optimize certain
- subsystems while completely destroying overall system performance.
- If said idiot publishes *only* the numbers relating to the optimized
- subsystem, then the costs of the optimization are hidden and readers
- will mistakenly believe that the optimization is a good idea. By
- including the publication restriction readers would be able to
- detect that the optimization improved the subsystem performance
- while damaging the rest of the system performance and would be able
- to make an informed decision as to the merits of the optimization.
- Note that these restrictions only apply to *publications*. We
- intend and encourage lmbench's use during design, development,
- and tweaking of systems and applications. If you are tuning the
- linux or BSD TCP stack, then by all means, use the networking
- benchmarks to evaluate the performance effects of various
- modifications; Swap results with other developers; use the
- networking numbers in isolation. The restrictions only kick
- in when you go to *publish* the results. If you sped up the
- TCP stack by a factor of 2 and want to publish a paper with the
- various tweaks or algorithms used to accomplish this goal, then
- you can publish the networking numbers to show the improvement.
- However, the paper *must* also include the rest of the standard
- lmbench numbers to show how your tweaks may (or may not) have
- impacted the rest of the system. The full set of numbers may
- be included in an appendix, but they *must* be included in the
- paper.
- This helps protect the community from adopting flawed technologies
- based on incomplete data. It also helps protect the community from
- misleading marketing which tries to sell systems based on partial
- (skewed) lmbench performance results.
- We have seen many cases in the past where partial or misleading
- benchmark results have caused great harm to the community, and
- we want to ensure that our benchmark is not used to perpetrate
- further harm and support false or misleading claims.
|