Remove logging

[viktor-morin]

Make it possible to not logg to HDD (drive).
I'm using benchmark inside Service Fabric (Azure) and it would be great to disable logging to file.
In the meanwhile have I changed logger.Writeline to logger?.Writeline which almost gives the solution.

[adamsitnik]

@AndreyAkinshin @mattwarren What do you guys think about adding this feature?

I think that I can add StreamLogger to DefaultConfig and get it done quite easily. The question is if we should allow this to happen?

[adamsitnik]

@viktor-morin why closed? by accident I assume?

[viktor-morin]

For me that works mostly in cloud (Azure) and dont want to save files locally on the VM, and also be able to run several benchmarks simultaneously, I think this is a good feature. Should be quite easy to implement. From my view I cant see any drawbacks :)

[viktor-morin]

Sorry, my misstake :)

[viktor-morin]

I would also like to introduce the possibility to have a random name on the executable file:
shortName = "BDN.Auto" + DateTime.UtcNow.ToString("HH-mm-ss-fff");
The reason for this is that we want to run several parallely benchmarktest and otherwise get the error "file is already is use"

[adamsitnik]

I would also like to introduce the possibility to have a random name on the executable file:
shortName = "BDN.Auto" + DateTime.UtcNow.ToString("HH-mm-ss-fff");

You can set KeepBenchmarkFiles = true in your config to have unique folder names. The problem then is that you might sometimes get PathTooLongException

[viktor-morin]

You can set KeepBenchmarkFiles = true in your config to have unique folder names. The problem then is that you might sometimes get PathTooLongException

This method throws an exception if I run two at the same time, so my comment is still valid:
using (var executable = File.Create(generateResult.ArtifactsPaths.ExecutablePath))

It refer to BDN.AUTO

[AndreyAkinshin]

@adamsitnik ,

I think that I can add StreamLogger to DefaultConfig and get it done quite easily.

It's a good idea to have the SteamLogger in the config and allow to don't include it in the config. The reason why we don't have in the current codebase is the following: our StreamLogger requires two additional steps: creating and disposing a file. Probably, we can add the corresponded methods (like Start/Finish or Initialize/Cleanup; the first one should take rootArtifactsFolderPath and IList<Benchmark>) in the ILogger interface (they will be empty for other loggers).

@viktor-morin,

and also be able to run several benchmarks simultaneously
Why do you want it? In general, it's a bad idea:

• The benchmarks will affect each other, so, you will get a bad accuracy
• BenchmarkDotNet adjusts amount of iteration and tries to minimize the confidence interval based on the current set of measurements. If you have two benchmark process at the same time, the standard deviation will be big, and total time will be bigger than a case with two consecutive runs.

So, why do you need this feature? What the your goal?

A few additional comments:

• It's nice to have an ability to customize generated folder names, but it's a part of a big refactoring which involved a benchmark generated process (we want to generate less projects, reuse them, and pass jobs via command line arguments). Will be implemented but not in the next version.
• In the future, we will have an InProcessToolchain which will not generate any additional files at all. But the project-based toolchains will be default anyway.
• Also we want to add concurrent benchmarks, but there are a lot of problems of here; we should do it really carefully to achieve a good level of accuracy. I'm working on it.

[viktor-morin]

The reason I want to run several benchmarks simultaneosuly was in a way to have concurrent benchmarks, as I see you are working on. I'm using Service Fabric and the Actor design pattern.

Azure Actors

At the moment I'm testing different WebAPI and looking at the response time (benchmarking this). I want to be able to call a WebAPI as I do right now severals times, but also simultaneously. I'm doing that today as a seperate Actor which is a singelton (one thread) to try to simulate each Actor as a "client" calling these WebAPI.

If I run this simultaneously are BenchmarkDotNet affecting eachother?

[AndreyAkinshin]

If I run this simultaneously are BenchmarkDotNet affecting eachother?

Yes, of course, you will not get any meaningful measurements. I know that concurrent benchmarks is a nice feature to have, a lot of people want it. But it's also a tricky feature, it's easy to make a lot of mistakes here.

[viktor-morin]

Yes, of course, you will not get any meaningful measurements. I know that concurrent benchmarks is a nice feature to have, a lot of people want it. But it's also a tricky feature, it's easy to make a lot of mistakes he

I mean, of course it will affect the measurements. But each code should be run with it own resources and in a singelton where it dosn't know about eachother. It that case shouldnt we be able to measure the concurrency of the backend and how fast the WebAPI is when several clients are asking for data simultaneously?

[AndreyAkinshin]

Ok, I get it. Anyway, you can't use BenchmarkDotNet for such experiments right now, sorry. We have a lot of different stages per each benchmarks (like pilot, warmup, etc.), we have several launches per each benchmark, and so on. This workflow will not work in you case because now there is no any way to synchronize these stages. Even stupid handwritten Stopwatch-based benchmark will be better for your case than several BenchmarkDotNet session at the same time.

[viktor-morin]

Okay, thanks for the input!

[adamsitnik]

Closing, #190 and #198 will cover this case