bb.util
Class Benchmark

java.lang.Object
  extended by bb.util.Benchmark

public class Benchmark
extends Object

Benchmarks the performance of a "task", defined here as arbitrary Java code that is contained inside either a Callable or Runnable. The task may be anything from a microbenchmark to a full blown application. This class implements all of the principles described in the two-part article Robust Java benchmarking, Part 1: Issues and Robust Java benchmarking, Part 2: Statistics And solutions.

Philosophy

The overall design goal was to make typical things easy to do, uncommon things possible to do, and bad things impossible to do. For example, as will be discussed in the usage section below, the choice was made to have the constructor do all the work, since that leads to the most simple and concise code. However, if something unusual needs to be done, such as changing one of the default measurement parameters, then that is possible too.

A related design goal was to automate as many aspects as possible, and minimize input from and interaction with the user. For example, this class tries to detect as many issues as possible (e.g. it checks if the measurements have outliers).

Usage

Most users of this class can perform reliable benchmarking simply by calling System.out.println("My task: " + new Benchmark(task)); where task is either a Callable or Runnable created by the user to encapsulate the code to be measured. This single argument Benchmark constructor will carry out multiple execution time measurements and perform statistics calculations on them, so that upon return the new instance is populated with results. This code also makes an implicit call to toString which will describe the essential statistics.

If custom tuning of the measurement parameters is required, then more complicated constructors may be used.

One simple tuning is if the task internally executes the same identical action many times, and you want statistics for that individual action (as opposed to the task as a whole). This is easily done with code like this: System.out.println("My task: " + new Benchmark(task, n)); where n is the number of actions that you know task performs. See Benchmark.Params.numberActions for more discussion.

Another simple tuning is if the task should only be executed once. This is easily done with code like this: System.out.println("My task: " + new Benchmark(task, false)); There will be no statistics reported in this case, just the single time measurement. See Benchmark.Params.manyExecutions for more discussion.

For uncommon situations, ultimate tunability over every measurement parameter may be achieved by constructing an instance of the Benchmark.Params inner class, modifying it as desired, and then directly supplying it along with task to the appropriate constructor. For example, here is how to configure Benchmark to measure CPU time:

Benchmark.Params params = new Benchmark.Params();
params.setMeasureCpuTime(true); // default is false (i.e. measure elapsed time)
System.out.println("Some task's CPU time: " + new Benchmark(task, params));

Multiple benchmarks

If you need to perform multiple benchmarks, be careful about doing them all in a single JVM session. A possible issue is that the JVM may make optimization decisions (branch predictions, memory layout, etc) that are appropriate for the first benchmark, but may be suboptimal for subsequent ones. Since there appears to be no way to programmatically tell the JVM to drop all of its optimizations and start profiling from scratch, the only alternative is to do every benchmark in a dedicated JVM session. The usual way to do this is to write a shell script which generates and runs all the benchmarking scenarios. The shell script files for the latest version of my benchmarking article code now contain examples of how to do this.

Result report

In the examples above, Benchmark's toString method was implicitly invoked to get a summary result report. However, if very detailed results are needed, then toStringFull may be used instead. Furthermore, various accessors are available if you need customized results.

Note that all the toStringXXX methods of this class, as well as its inner classes, have the property that regardless of whether or not the result contains multiple lines, the final char is never a newline. This means that the user should always call println on the results if a PrintWriter is being used for output.

Interpretation of results

The two main results reported by this class are point estimates for the arithmetic mean and standard deviation (sd) of a task's execution time.

Another important concern is the accuracy of those estimates, since inaccurate results, obviously, limit the conclusions that can be drawn (e.g. that one task executes faster than another). This class determines the accuracy of its estimates by producing confidence intervals for both the mean and the sd.

Blocks versus actions

There are many places in the documentation below where I use the terms block and action. These terms are fully defined in the Block statistics versus action statistics section of the article supplement.

Subclassability

This class was designed to maximize subclass opportunities. For instance, all members--including those of its inner classes--are public or protected, and are never final. Also, the methods are fine grained to enable selective overriding.

Multithread safety

This class is not multithread safe.

Alternatives

See also Execute.

Author:
Brent Boyer

Nested Class Summary
protected static class Benchmark.JvmState
          Records some characteristics of the JVM state.
static class Benchmark.LfsrAdvancer
          The run method of this class simply advances the internal state of an Lfsr instance.
protected static class Benchmark.Measurement
          Records information about an execution time measurement.
static class Benchmark.Params
          Holds parameters which specify how the benchmarking is carried out.
static class Benchmark.Stats
          Holds execution time statistics.
static class Benchmark.UnitTest
          See the Overview page of the project's javadocs for a general description of this unit test class.
 
Field Summary
protected  Object callResult
          If task is a Callable, then records the last result of executing task.call().
protected  String cleanIssues
          Records any issues that were found while doing a final JVM clean after all measurements were carried out.
protected  Benchmark.Measurement[] measurements
          If params.
protected static Map<Benchmark.Params,Benchmark.Stats> noiseMap
          Maps a Params instance to the block statistics that result from benchmarking noiseTask.
protected static Runnable noiseTask
          Task used to estimate the execution time noise floor.
protected  long numberExecutions
          Records how many executions of task were performed for each time measurement.
protected  String outlierIssues
          Records any outlier issues that were found in the measurements.
protected  Benchmark.Params params
          Controls how this instance's benchmarking is done.
protected  String serialCorrelationIssues
          Records any serial correlation issues that were found in the measurements.
protected  Benchmark.Stats statsAction
          If params.
protected  Benchmark.Stats statsBlock
          If params.
protected  Object task
          Packages the target code that will be benchmarked.
protected  double timeExecFirst
          First execution time for task.
 
Constructor Summary
protected Benchmark()
          No-arg constructor.
  Benchmark(Callable<T> task)
          Convenience constructor that simply calls this(task, new Benchmark.Params.Benchmark.Params()).
  Benchmark(Callable<T> task, Benchmark.Params params)
          Constructor that measures the execution time of a Callable task.
  Benchmark(Callable<T> task, boolean manyExecutions)
          Convenience constructor that simply calls this(task, new Params(manyExecutions)).
  Benchmark(Callable<T> task, boolean manyExecutions, long numberActions)
          Convenience constructor that simply calls this(task, new Params(manyExecutions, numberActions)).
  Benchmark(Callable<T> task, long numberActions)
          Convenience constructor that simply calls this(task, new Params(numberActions)).
  Benchmark(Runnable task)
          Convenience constructor that simply calls this(task, new Benchmark.Params.Benchmark.Params()).
  Benchmark(Runnable task, Benchmark.Params params)
          Identical to the Callable constructor except that task is a Runnable.
  Benchmark(Runnable task, boolean manyExecutions)
          Convenience constructor that simply calls this(task, new Params(manyExecutions)).
  Benchmark(Runnable task, boolean manyExecutions, long numberActions)
          Convenience constructor that simply calls this(task, new Params(manyExecutions, numberActions)).
  Benchmark(Runnable task, long numberActions)
          Convenience constructor that simply calls this(task, new Params(numberActions)).
 
Method Summary
protected  void appendIssues(String issues, StringBuilder sb)
          Helper method for toStringFull which handles issues fields like cleanIssues.
protected  void calculateStats()
          Calculates statsBlock from measurements.
protected  void checkState()
          Checks that all the state of this instance satisfies its contracts.
protected  void cleanJvm()
          Attempts to restore the JVM to a pristine state by aggressively performing object finalization and garbage collection.
protected  void cleanJvmFinal()
          Measures how long it takes to execute a call to cleanJvm.
protected  void clearUserMsgs()
          Clears any output that may have been written by calls to sendUserMsg.
protected  void determineNumberExecutions()
          Determines how many executions of task are required in order for the sum of their execution times to equal params.
protected  void diagnoseOutliers()
          Diagnoses measurements for the presence of outliers.
protected  String diagnoseSd(double mean, double sd)
          Diagnoses the sd parameter to see if it really represents the intrinsic variation in task's execution time or if it is just reflecting environmental noise.
protected  void diagnoseSerialCorrelation()
          Diagnoses measurements for the presence of serial correlation.
protected  void doMeasurementFirst()
          Performs the first measurement of task's execution time.
protected  void doMeasurements()
          Measures many many executions of task.
 Object getCallResult()
          Returns callResult.
 double getFirst()
          Returns the action execution time as determined from the first measurement, namely, timeExecFirst / params.
 double getMean()
          Returns the mean of the action execution times.
 long getNumberActionsPerMeasurement()
          Returns the number of action executions in each measurement.
 double getSd()
          Returns the standard deviation of the action execution times.
 Benchmark.Stats getStats()
          Alias for getStatsAction.
 Benchmark.Stats getStatsAction()
          Returns statsAction.
 Benchmark.Stats getStatsBlock()
          Returns statsBlock.
protected  double[] getTimes()
          Returns an array of just the execution times stored in measurements.
protected  String issueSummary()
          Returns a summary of any issues that occured.
protected  Benchmark.Measurement measure(long n)
          Measures the execution time of n calls of task.
protected  void osSpecificPreparation()
          Calls operating system specific commands to prepare for benchmarking.
protected  void perform(Object task, Benchmark.Params params)
          Carries out the benchmarking of task according to the specifications found in params.
protected  void preventDce()
          Attempts to prevent a sophisticated compiler from using dead-code elimination to skip computations that should be benchmarked.
protected  void sendUserMsg(String msg)
          Displays a non-critical message to the user.
protected  double timeDiffSeconds(long t1, long t2)
          Robustly calculates and returns the difference t2 - t1 in seconds.
protected  long timeNs()
          Returns the time since some fixed but arbitrary offset, in nanoseconds.
 String toString()
          Returns a String representation of this instance.
 String toStringFull()
          Returns a String representation of this instance with full details and explanations.
protected  void warmupJvm()
          In order to give hotspot optimization a chance to complete, task is executed many times (with no recording of the execution time).
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
 

Field Detail

noiseTask

protected static Runnable noiseTask
Task used to estimate the execution time noise floor.

Must have minimal "true" execution time variation. Instead, should only be subject to external noise (e.g. from operating system context switches and/or time measurement errors). So, in a perfect environment, this task should always have the same execution time (after some JVM warmup phase). Also, desire the task's steady state execution time to be in a reasonable range, say between 1 ms and 1 s.

Perhaps one way to write a low noise task is if it satisfies these properties:

  1. is entirely CPU bound
  2. corollary: has no synchronization, does no I/O
  3. uses no memory beyond the CPU's registers
  4. corollary: cannot cause any garbage collection

It is probably best if task repeatedly performs a very simple but irreducable computation. This computation should not give the JIT compiler many oportunities for optimization. It also must guarantee to always be performed (i.e. no dead-code elimination).

Contract: must satisfy the contract of task. Additionally, to meet the no garbage collection condition, is restricted to being a Runnable whose run method should create no new objects.


noiseMap

protected static Map<Benchmark.Params,Benchmark.Stats> noiseMap
Maps a Params instance to the block statistics that result from benchmarking noiseTask. In particular, if params is a key of this Map, then new Benchmark(noiseTask, params).statsBlock is the value.

Contract: is never null.


task

protected Object task
Packages the target code that will be benchmarked.

Contract: is never null and is always either a Callable or Runnable instance. To ensure that the computation that it carries out is never be subject to dead-code elimination (DCE), the following rules must be obeyed:

  1. if a Callable, then the computation done by call must be used to generate the result returned by call
  2. if a Runnable, then the computation done by run must be stored in some internal state. The toString method must be reimplemented, overriding Object's, and must use that state in its result.
If these rules are obeyed, then preventDce should guarantee that DCE never occurs.


params

protected Benchmark.Params params
Controls how this instance's benchmarking is done.

Contract: is never null.


timeExecFirst

protected double timeExecFirst
First execution time for task. In general, this value includes all initialization effects (e.g. class loading, hotspot warmup, etc).

Warning: the value is actually just the first measurement of task that was done by this Benchmark instance. So, if other code in this same JVM session has previously executed task, then this value will not, in fact, reflect initialization effects.

Units: seconds.

Contract: is >= 0, and is never NaN or infinite.


numberExecutions

protected long numberExecutions
Records how many executions of task were performed for each time measurement.

Contract: is > 0.


measurements

protected Benchmark.Measurement[] measurements
If params.getManyExecutions returns true, then this instance performed multiple execution time measurements and stored the results here. The values are only those from the steady state execution profile stage, and do not include the JVM warmup runs. Each measurement is over numberExecutions of task, and is stored in this array in the order that it occurred (i.e. this array is not sorted).

Contract: is null if params.getManyExecutions() returns false, else is non-null with length = params.Benchmark.Params.getNumberMeasurements().


callResult

protected Object callResult
If task is a Callable, then records the last result of executing task.call().

This is used to prevent dead-code elimination; see task and preventDce.

Contract: is always null if task is a Runnable; otherwise, may be anything.


cleanIssues

protected String cleanIssues
Records any issues that were found while doing a final JVM clean after all measurements were carried out.

Contract: is either null if there are no issues, else is non blank.


outlierIssues

protected String outlierIssues
Records any outlier issues that were found in the measurements.

Contract: is either null if there are no issues, else is non blank.


serialCorrelationIssues

protected String serialCorrelationIssues
Records any serial correlation issues that were found in the measurements.

Contract: is either null if there are no issues, else is non blank.


statsBlock

protected Benchmark.Stats statsBlock
If params.Benchmark.Params.getManyExecutions() returns true, holds the block statistics. In this case, what actually is measured is a block of numberExecutions calls of task. The statistics of these measurements are stored here.

Contract: is null if params.getManyExecutions() returns false, else is non-null.


statsAction

protected Benchmark.Stats statsAction
If params.Benchmark.Params.getManyExecutions() returns true, holds the action statistics. These are always calculated from statsBlock and are never directly measured (altho, if there is but 1 action per measurement, then this instance is the same as statsBlock).

Contract: is null if params.getManyExecutions() returns false, else is non-null.

Constructor Detail

Benchmark

public Benchmark(Callable<T> task)
          throws IllegalArgumentException,
                 IllegalStateException,
                 Exception
Convenience constructor that simply calls this(task, new Benchmark.Params.Benchmark.Params()).

Throws:
IllegalArgumentException
IllegalStateException
Exception

Benchmark

public Benchmark(Callable<T> task,
                 boolean manyExecutions)
          throws IllegalArgumentException,
                 IllegalStateException,
                 Exception
Convenience constructor that simply calls this(task, new Params(manyExecutions)).

Throws:
IllegalArgumentException
IllegalStateException
Exception

Benchmark

public Benchmark(Callable<T> task,
                 long numberActions)
          throws IllegalArgumentException,
                 IllegalStateException,
                 Exception
Convenience constructor that simply calls this(task, new Params(numberActions)).

Throws:
IllegalArgumentException
IllegalStateException
Exception

Benchmark

public Benchmark(Callable<T> task,
                 boolean manyExecutions,
                 long numberActions)
          throws IllegalArgumentException,
                 IllegalStateException,
                 Exception
Convenience constructor that simply calls this(task, new Params(manyExecutions, numberActions)).

Throws:
IllegalArgumentException
IllegalStateException
Exception

Benchmark

public Benchmark(Callable<T> task,
                 Benchmark.Params params)
          throws IllegalArgumentException,
                 IllegalStateException,
                 Exception
Constructor that measures the execution time of a Callable task. Simply calls perform(task, params). So, when this constructor finishes, it is fully populated with results.

Parameters:
task - the Callable whose call method's execution time will be measured; is assigned to the task field
params - assigned to the params field
Throws:
IllegalArgumentException - if task == null; params == null
IllegalStateException - if some problem is detected
Exception - (or some subclass) if task.call throws it

Benchmark

public Benchmark(Runnable task)
          throws IllegalArgumentException,
                 IllegalStateException
Convenience constructor that simply calls this(task, new Benchmark.Params.Benchmark.Params()).

Throws:
IllegalArgumentException
IllegalStateException

Benchmark

public Benchmark(Runnable task,
                 boolean manyExecutions)
          throws IllegalArgumentException,
                 IllegalStateException
Convenience constructor that simply calls this(task, new Params(manyExecutions)).

Throws:
IllegalArgumentException
IllegalStateException

Benchmark

public Benchmark(Runnable task,
                 long numberActions)
          throws IllegalArgumentException,
                 IllegalStateException
Convenience constructor that simply calls this(task, new Params(numberActions)).

Throws:
IllegalArgumentException
IllegalStateException

Benchmark

public Benchmark(Runnable task,
                 boolean manyExecutions,
                 long numberActions)
          throws IllegalArgumentException,
                 IllegalStateException
Convenience constructor that simply calls this(task, new Params(manyExecutions, numberActions)).

Throws:
IllegalArgumentException
IllegalStateException

Benchmark

public Benchmark(Runnable task,
                 Benchmark.Params params)
          throws IllegalArgumentException,
                 IllegalStateException,
                 RuntimeException
Identical to the Callable constructor except that task is a Runnable.

Parameters:
task - the Runnable whose run method's execution time will be measured; is assigned to the task field
params - assigned to the params field
Throws:
IllegalArgumentException - if task == null; params == null
IllegalStateException - if some problem is detected
RuntimeException - (or some subclass) if some other problem occurs

Benchmark

protected Benchmark()
No-arg constructor.

Allows instance creation without causing a benchmark. For example, subclasses may wish to not have the constructor automatically carry out benchmarking. Another use is to test instance methods of this class without doing a benchmark first (e.g. see UnitTest.test_diagnoseSerialCorrelation).

Method Detail

perform

protected void perform(Object task,
                       Benchmark.Params params)
                throws IllegalArgumentException,
                       IllegalStateException,
                       Exception
Carries out the benchmarking of task according to the specifications found in params.

Parameters:
task - either a Callable or a Runnable; packages the target code that will be benchmarked; is assigned to the task field
params - assigned to the params field
Throws:
IllegalArgumentException - if task == null; task is neither a Callable or Runnable; params == null
IllegalStateException - if some problem is detected
Exception - (or some subclass) if task is a Callable and task.call throws it

osSpecificPreparation

protected void osSpecificPreparation()
Calls operating system specific commands to prepare for benchmarking.

The current implementation only does something if the operating system is determined to be from Microsoft. In this case, it forces pending idle tasks to execute by running the command
        Rundll32.exe advapi32.dll,ProcessIdleTasks
Warning: this may take several minutes to complete, especially if this is the first time that it has been called in a while.

This method never throws an Exception. If one occurs, it is caught and its stack trace is printed to System.err.


doMeasurementFirst

protected void doMeasurementFirst()
                           throws Exception
Performs the first measurement of task's execution time. The result is recorded in timeExecFirst.

Throws:
Exception - (or some subclass) if task is a Callable and task.call throws it

warmupJvm

protected void warmupJvm()
                  throws Exception
In order to give hotspot optimization a chance to complete, task is executed many times (with no recording of the execution time). This phase lasts for the time specified by params.Benchmark.Params.getWarmupTime().

Throws:
Exception - (or some subclass) if task is a Callable and task.call throws it

determineNumberExecutions

protected void determineNumberExecutions()
                                  throws Exception
Determines how many executions of task are required in order for the sum of their execution times to equal params.getExecutionTimeGoal. The result is stored in numberExecutions.

Throws:
Exception - (or some subclass) if task is a Callable and task.call throws it

doMeasurements

protected void doMeasurements()
                       throws Exception
Measures many many executions of task. Each measurement is over a block of numberExecutions calls to task. The number of measurements is specified by params.getNumberMeasurements. The results are recorded in measurements.

Throws:
Exception - (or some subclass) if task is a Callable and task.call throws it

preventDce

protected void preventDce()
Attempts to prevent a sophisticated compiler from using dead-code elimination to skip computations that should be benchmarked.

The implementation here writes data to the console that always depends on both task.toString() and callResult.toString(). This ensures that as long as task obeys its contract, regardless of whether it is a Runnable or Callable, then bad DCE never occurs. Actually, to cut down on low level output like this, this method only prints out this information if some conditional logic is passed. The form of this logic should cause output to almost never appear, however, the JVM does not know this, and so will be forced to do all the computations.


cleanJvmFinal

protected void cleanJvmFinal()
Measures how long it takes to execute a call to cleanJvm. Compares this time with total task execution time held in measurements, and issues a warning if this cleanJvm time is excessive (since this means that the task execution times do not properly account for cleanup costs).


diagnoseOutliers

protected void diagnoseOutliers()
                         throws IllegalStateException
Diagnoses measurements for the presence of outliers.

Implementation here uses the interquartile range to define what constitutes an outlier (i.e. the boxplot technique) and assigns a non-null message to outlierIssues if any outliers are detected.

Throws:
IllegalStateException - if called when measurements == null

diagnoseSerialCorrelation

protected void diagnoseSerialCorrelation()
                                  throws IllegalStateException
Diagnoses measurements for the presence of serial correlation.

Implementation here computes the autocorrelation function as a function of lag, counts how many values fall outside their 95% confidence interval, and assigns a non-null message to serialCorrelationIssues if this count exceeds the expected count.

Throws:
IllegalStateException - if called when measurements == null

calculateStats

protected void calculateStats()
                       throws IllegalStateException
Calculates statsBlock from measurements. Then derives statsAction from statsBlock.

Throws:
IllegalStateException - if called when measurements == null

getTimes

protected double[] getTimes()
                     throws IllegalStateException
Returns an array of just the execution times stored in measurements.

Throws:
IllegalStateException - if called when measurements == null

diagnoseSd

protected String diagnoseSd(double mean,
                            double sd)
Diagnoses the sd parameter to see if it really represents the intrinsic variation in task's execution time or if it is just reflecting environmental noise. See the "Standard deviation measurement issues" section of the article supplement for more details.

Parameters:
mean - the mean execution time; must be from the block statistics
sd - the standard deviation of the execution time; must be from the block statistics

checkState

protected void checkState()
                   throws IllegalStateException
Checks that all the state of this instance satisfies its contracts.

Throws:
IllegalStateException - if any violation is found

cleanJvm

protected void cleanJvm()
Attempts to restore the JVM to a pristine state by aggressively performing object finalization and garbage collection.


sendUserMsg

protected void sendUserMsg(String msg)
Displays a non-critical message to the user.

Only use this method for information like program progress. Do not use it for results or error messages.

Implementation here only does something if params.getConsoleFeedback returns true. If that is the case, it writes msg to the current console line. In order to avoid flooding the console with data, it truncates msg if necessary in order to fit on a single line, and overwrites anything that was previously on the current console line.


clearUserMsgs

protected void clearUserMsgs()
Clears any output that may have been written by calls to sendUserMsg.


measure

protected Benchmark.Measurement measure(long n)
                                 throws IllegalArgumentException,
                                        Exception
Measures the execution time of n calls of task.

Units: seconds.

Contract: the result is always >= 0.

Throws:
IllegalArgumentException - if n <= 0
Exception - (or some subclass) if task is a Callable and task.call throws it

timeNs

protected long timeNs()
Returns the time since some fixed but arbitrary offset, in nanoseconds. Therefore, is only useful for differential time measurements. CPU time is used if params.getMeasureCpuTime returns true, else elapsed (aka wall clock) time is used.


timeDiffSeconds

protected double timeDiffSeconds(long t1,
                                 long t2)
                          throws IllegalArgumentException
Robustly calculates and returns the difference t2 - t1 in seconds.

Parameters:
t1 - first time, in nanoseconds
t2 - second time, in nanoseconds
Throws:
IllegalArgumentException - if an issue is discovered

getFirst

public double getFirst()
Returns the action execution time as determined from the first measurement, namely, timeExecFirst / params.Benchmark.Params.getNumberActions().


getNumberActionsPerMeasurement

public long getNumberActionsPerMeasurement()
Returns the number of action executions in each measurement. This is equal to params.getNumberActions() * numberExecutions.


getCallResult

public Object getCallResult()
Returns callResult.


getStatsBlock

public Benchmark.Stats getStatsBlock()
                              throws IllegalStateException
Returns statsBlock.

Throws:
IllegalStateException - if params.getManyExecutions() returns false (there are no Stats)

getStatsAction

public Benchmark.Stats getStatsAction()
                               throws IllegalStateException
Returns statsAction.

Throws:
IllegalStateException - if params.getManyExecutions() returns false (there are no Stats)

getStats

public Benchmark.Stats getStats()
                         throws IllegalStateException
Alias for getStatsAction. This is a convenience method, since most users will typically only be interested in the action statistics, therefore this provides an abbreviation.

Throws:
IllegalStateException - if params.getManyExecutions() returns false (there are no Stats)

getMean

public double getMean()
               throws IllegalStateException
Returns the mean of the action execution times.

This is a convenience method that simply returns getStats().getMean().

Throws:
IllegalStateException - if params.getManyExecutions() returns false (there are no Stats)

getSd

public double getSd()
             throws IllegalStateException
Returns the standard deviation of the action execution times.

This is a convenience method that simply returns getStats().getSd().

Warning: the result may be unreliable. The toString and toStringFull methods will warn if there are issues with it. See the "Standard deviation warnings" section of the article supplement for more details.

Throws:
IllegalStateException - if params.getManyExecutions() returns false (there are no Stats)

toString

public String toString()
Returns a String representation of this instance.

The implementation here merely summarizes the important information: the action's first execution time, the action's statistics (if available), and high level warnings.

Overrides:
toString in class Object

issueSummary

protected String issueSummary()
Returns a summary of any issues that occured.


toStringFull

public String toStringFull()
Returns a String representation of this instance with full details and explanations.


appendIssues

protected void appendIssues(String issues,
                            StringBuilder sb)
Helper method for toStringFull which handles issues fields like cleanIssues.