How to do Benchmarking with Go

I will be talking about benchmarking in Go in this article.

But first, what is benchmarking in general?
Simply put, it means running a set of programs to assess the relative performance against a set of standards.

In our case, we want to judge whether Go codes that we wrote are performing well or not.

How to do benchmarking in Go

Go included a default benchmarking tool inside testing package.

To differentiate benchmarking from testing, you need to prefix the function with BenchmarkXxx.

func BenchmarkXxx(*testing.B)

If you happen to have other testing functions and want to run only all benchmarks, we can do so by:

go test -bench .

Or if you want to run individual benchmark:

go test -bench FUNCTION_NAME
go test -bench BenchmarkUseConcatOperator
go test -bench Use

-bench argument uses regex pattern so if you define -bench Use it will run all benchmarks that contain the string "Use" on their names.

Benchmark tests setup

Let's set up some simple benchmark tests. I will explain the thought process but if you just want to see the source code you can check my GitHub Gist here.

First, we want to decide on a theme for testing. String concatenation sounds like an interesting topic since in Go we could do it in various ways.

We'll create a main.go file and setup three functions for concatenating strings:

• First is using the simple concat operator "+"
• Second is utilizing bytes.Buffer package
• Third is utilizing strings.Builder package

package main

import (
    "bytes"
    "math/rand"
    "strings"
)

func main() {
    // This main function does nothing since we only want to test benchmarking
}

func UseConcatOperator(slice []string) string {
    var s string
    for _, val := range slice {
        s = s + val
    }
    return s
}

func UseBytesBuffer(slice []string) string {
    var b bytes.Buffer
    for _, val := range slice {
        b.WriteString(val)
    }
    return b.String()
}

func UseStringsBuilder(slice []string) string {
    var sb strings.Builder
    for _, val := range slice {
        sb.WriteString(val)
    }
    return sb.String()
}

Now that we have three functions to concatenate strings, we want to see which one actually performs the best when concatenating a slice of strings.

Obviously, we need to create that slice of strings for testing, adding another function below the three.

func GetSliceOfStrings() []string {
    size := 1000
    slice := make([]string, size)
    for i := 0; i < size; i++ {
        s := "abcdedfghijklmnopqrstuvwxyz"
        slice[i] = s
    }
    return slice
}

There you have it. We got our functions to test. Now on to the test file, we shall name it main_test.go.

package main

import (
    "testing"
)

var slice = GetSliceOfStrings()

func BenchmarkUseConcatOperator(b *testing.B) {
    for i := 0; i < b.N; i++ {
        UseConcatOperator(slice)
    }
}

func BenchmarkUseBytesBuffer(b *testing.B) {
    for i := 0; i < b.N; i++ {
        UseBytesBuffer(slice)
    }
}

func BenchmarkUseStringsBuilder(b *testing.B) {
    for i := 0; i < b.N; i++ {
        UseStringsBuilder(slice)
    }
}

It's clear at a glance that we have a benchmarking function for each of our string concatenation functions. And we provide the parameter for each testing with a slice of 1000 strings.

We can start benchmarking with a minimal command:

go test -bench .

Which will yield these results:

C:\Users\user\playground> go test -bench .
goos: windows
goarch: amd64
pkg: playground
BenchmarkUseConcatOperator-8         276           4009217 ns/op
BenchmarkUseBytesBuffer-8          25038             45427 ns/op
BenchmarkUseStringsBuilder-8       26460             50240 ns/op
PASS
ok      playground      5.024s

It's a little bit hard to read but each of our benchmark functions ran the actual function they are assigned to. For instance, BenchmarkUseConcatOperator ran UseConcatOperator 276 times at a speed of 4,009,217ns per operation.

Since each benchmark is run for a minimum of 1 second by default, compared to the other concatenate functions, UseConcatOperator is the slowest.

If we want to modify the parameters a bit we can do so.
Perhaps we want to set each benchmarking test to do x repetition...

go test -bench . -benchtime 10000x

Result:

C:\Users\user\playground> go test -bench . -benchtime 10000x
goos: windows
goarch: amd64
pkg: playground
BenchmarkUseConcatOperator-8       10000           4301341 ns/op
BenchmarkUseBytesBuffer-8          10000             45964 ns/op
BenchmarkUseStringsBuilder-8       10000             42709 ns/op
PASS
ok      playground      43.958s

Or set each benchmarking test to last x seconds instead of default 1s...

go test -bench . -benchtime 5s

Result:

C:\Users\user\playground> go test -bench . -benchtime 10000x
goos: windows
goarch: amd64
pkg: playground
BenchmarkUseConcatOperator-8        1365           4324163 ns/op
BenchmarkUseBytesBuffer-8         125238             48508 ns/op
BenchmarkUseStringsBuilder-8      134504             44652 ns/op
PASS
ok      playground      19.419s

From these benchmarks, we can safely conclude that using strings.Builder gives the best performance in string concatenation.

More test flags

What can we do to get more information from our benchmarks? We can use flags from Go's official documentation.

• -benchmem Gives out details on memory allocation.

BenchmarkUseConcatOperator-8         256           4292437 ns/op        14227566 B/op       1000 allocs/op

type BenchmarkResult struct {
    N         int           // The number of iterations.
    T         time.Duration // The total time taken.
    Bytes     int64         // Bytes processed in one iteration.
    MemAllocs uint64        // The total number of memory allocations; added in Go 1.1
    MemBytes  uint64        // The total number of bytes allocated; added in Go 1.1
}

• -cpuprofile FILE_NAME (ex: cpu.prof)

Write a CPU profile to the specified file (cpu.prof) before exiting.
Use go tool pprof to read the file.

C:\Users\user\playground> go tool pprof cpu.prof
Type: cpu
Time: Mar 26, 2021 at 2:46pm (JST)
Duration: 5.42s, Total samples = 10.15s (187.21%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top10

• -memprofile FILE_NAME (ex: mem.prof)

Write a memory allocation profile to the specified file (mem.prof) before exiting.
Use go tool pprof to read the file.

• -count

Set up the number of repetitions for each benchmarking test.

C:\Users\user\playground> go test -bench . -count 5
goos: windows
goarch: amd64
pkg: playground
BenchmarkUseConcatOperator-8         289           3667618 ns/op
BenchmarkUseConcatOperator-8         291           3874053 ns/op
BenchmarkUseConcatOperator-8         307           3939308 ns/op
BenchmarkUseConcatOperator-8         328           3883875 ns/op
BenchmarkUseConcatOperator-8         327           3869996 ns/op

• -cpu Define the number of CPUs (a list of GOMAXPROCS) used during benchmarking.

Closing remarks

Benchmarking is one of the important points when testing your program. Knowing how well your program performs helps a lot in fine-tuning small details.

However, benchmarking should not be the very focus of work in the early stage of development. It's better to make sure business requirements are met first, before mulling over program optimization.