Golang High-Performance Programming EP5: Benchmark

When we attempt to optimize code performance, the first step is understanding the current performance to establish a baseline. The Go language standard library includes the testing framework, which provides benchmarking capabilities. This article will cover how to use benchmarking for performance testing, how to improve the accuracy of benchmarks and introduce two tools to help facilitate benchmarking.

This article was first published in the Medium MPP plan. If you are a Medium user, please follow me on Medium. Thank you very much.

The environment highly influences performance testing. To ensure the repeatability of tests, strive to keep the testing environment as stable as possible:

Idle Machine: Ensure the machine is idle and do not run other tasks or share hardware resources with others during testing.
Power-Saving Mode: Disable power-saving mode, which is often enabled by default on laptops.
Avoid Virtual Machines and Cloud Servers: Avoid using virtual machines and cloud servers for testing, as they typically oversell CPU and memory resources, leading to unstable performance results.
Multiple Runs: A single test run is meaningless; statistically significant results require multiple runs. For most benchmarks, I typically execute the test 20 times.

How Benchmarking Works

Benchmarking involves repeatedly calling a function and recording execution time and other metrics to measure its performance. Here is a typical benchmark function:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


func fib(n int) int {  
    if n < 2 {  
       return n  
    }  
    return fib(n-1) + fib(n-2)  
}  

func BenchmarkFib20(b *testing.B) {  
    for i := 0; i < b.N; i++ {  
       // Call the function we're benchmarking  
       fib(20)  
    }  
}

Benchmark functions must start with Benchmark. We can run it as follows:

1
2


go test -bench .
go test -bench="BenchmarkFib20"

Understanding `b.N`

The benchmark test parameter b *testing.B includes an attribute b.N, which represents the number of times the test case needs to run. The value of b.N differs for each test case.

How is this value determined? b.N starts at 1, and if the test case can complete within 1 second, the value of b.N will increase and execute again. The value of b.N increases approximately in the sequence of 1, 2, 3, 5, 10, 20, 30, 50, 100, etc., accelerating as it goes.

Other Parameters

count: Controls the number of runs, independent of b.N.
benchtime: Controls the run time, limited to 1 second.
benchmem: Measures the number of memory allocations.
cpu: Specifies the number of CPU cores to use; by default, it uses go.

Additionally, benchmark tests support other go test parameters like cpuprofile, memprofile, and trace.

Improving Accuracy

Reducing System Noise with `perflock`

perflock limits CPU clock frequency, thereby minimizing the system’s impact on performance tests and reducing noise, resulting in more reliable and consistent performance measurement results.

Using `ResetTimer`

If preparation work is required before the benchmark starts, and the preparation work is time-consuming, you should ignore the time consumed by this part of the code.

Using `StopTimer` and `StartTimer`

StopTimer and StartTimer follow the same principle. When preparation and cleanup work is needed before and after each function call, use StopTimer to pause timing and StartTimer to resume timing.

Measuring Benchmark Results

benchstat is an official comparison tool used to compare performance differences between two benchmark runs.

Benchstat computes statistical summaries and A/B comparisons of Go benchmarks.

We use benchstat to compare the performance of bubbleSort and quickSort.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20


func softNums() {  
    //bubbleSort(nums)  
    //quickSort(nums)
}  
  
func initNums(count int) {  
    rand.Seed(time.Now().UnixNano())  
    nums = make([]int, count)  
    for i := 0; i < count; i++ {  
       nums[i] = rand.Intn(count)  
    }  
}  

func BenchmarkSoftNums(b *testing.B) {  
    initNums(10000)  
    b.ResetTimer()  
    for i := 0; i < b.N; i++ {  
       softNums()  
    }  
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


go test -bench="BenchmarkSoftNums" -count=10 |tee quicksoft.txt
go test -bench="BenchmarkSoftNums" -count=10 |tee bubblesoft.txt

➜  benchmark git:(main) ✗ benchstat bubblesoft.txt quicksoft.txt                         
goos: darwin
goarch: arm64
pkg: blog-example/go/benchmark
            │ bubblesoft.txt │            quicksoft.txt            │
            │     sec/op     │   sec/op     vs base                │
SoftNums-10    31942.4µ ± 1%   775.8µ ± 2%  -97.57% (p=0.000 n=10)

Isn’t this result very clear? You can include it in reports or GitHub issues, making it look very professional.

Third-Party Tools

bench

bench is a benchmarking tool for Go programs that provides integrated performance measurement, automatic performance locking, statistical analysis, and color indication.

funcbench

funcbench is a Prometheus project tool used to automate Go code benchmarking and performance comparison. Here are its main features and functions:

Supports comparing local branches with GitHub branches.
Features performance locking and performance comparison to improve development efficiency.
If the project is large, consider using this approach.

How Benchmarking Works

Understanding b.N