Last time we shared the method of debugging Go code using assembly language. Assembly language allows us to easily trace low-level knowledge of the Go runtime and other underlying details. In this article, we will introduce the powerful debugging tool called “go tool”, which, when mastered, can elevate your development skills to the next level.

This article focuses on practical techniques for debugging in Golang and the effective usage of related tools, so you no longer need to worry about how to debug Golang code. Golang, as a modern language, comes with built-in debugging capabilities from the very beginning:

• Golang tools are directly integrated into the language tools, supporting memory analysis, CPU analysis, and blocking lock analysis, among others.
• Delve and GDB are the most commonly used debug tools, allowing you to dive deeper into program debugging.
• Delve is currently the most user-friendly Golang debugging program, and IDE debugging is essentially calling dlv, such as in Goland.
• Unit testing is deeply integrated into the language design, making it very convenient to execute unit tests and generate code coverage.

Golang tools

Golang integrates a variety of useful tools at the language level, which are the essence of the experience accumulated by Robert Griesemer, Rob Pike, Ken Thompson, and other experts. After installing Golang, you can see all the built-in tools by executing go tool.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

hxzhouh@hxzhouhdeMacBook-Pro ~> go tool
 addr2line
 asm
 buildid
 cgo
 compile
 covdata
 cover
 dist
 distpack
 doc
 fix
 link
 nm
 objdump
 pack
 pprof
 test2json
 trace
 vet

Here, I will focus on selecting several commonly used debug tools:

• nm: view the symbol table (equivalent to the system nm command).
• objdump: disassembly tool, used to analyze binary files (equivalent to the system objdump command).
• pprof: metric and performance analysis tool.
• cover: code coverage generation.
• trace: sampling over a period of time, metric tracking and analysis tool.
• compile: code assembly.

Now, I will demonstrate the usage of these tools with an example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39

package main
 ​
 import (
     "log"
     "net"
 )
 ​
 var helloCount int
 ​
 func main() {
     listener, err := net.Listen("tcp", ":50050")
     if err != nil {
         log.Fatal(err)
     }
     defer listener.Close()
 ​
     for {
         conn, err := listener.Accept()
         if err != nil {
             log.Fatal(err)
         }
         tcpConn := conn.(*net.TCPConn)
         err = tcpConn.SetNoDelay(true)
         if err != nil {
             log.Println(err)
         }
         go handleConnection(conn)
     }
 }
 ​
 func handleConnection(conn net.Conn) {
     defer conn.Close()
     helloCount++
     resp := []byte("Hello count: ")
     resp = append(resp, []byte(string(rune(helloCount)))...)
     resp = append(resp, '\n')
     conn.Write(resp)
 }
 // go build main.go

nm

The nm command is used to view the symbol table, which is equivalent to the system nm command and is very useful. When setting breakpoints, if you don’t know the function symbol of the breakpoint, you can use this command to find out (this command operates on binary program files).

compile

Assemble a specific file:

1

go tool compile -N -l -S main.go

You will be able to see the assembly code corresponding to your Go code (please note that this command operates on Go code text), which is cool.

objdump

The objdump tool is used to disassemble binaries, equivalent to the system objdump (please note that this command parses binary program files).

1
2

go tool objdump main.o
go tool objdump -s DoFunc main.o  // Disassembling specific functions

Assembly code may not be needed in 90% of scenarios, but if you have experience working with C programs, in certain special situations, inferring application behavior by disassembling a piece of logic may be your only way out. This is because the code running in production usually has optimization enabled, which can cause your code to not match. Furthermore, you cannot attach to processes at will in production environments. Many times, you only have a core file to troubleshoot.

pprof

pprof supports four types of analysis:

• CPU: CPU analysis, sampling calls that consume CPU resources, which is generally used to locate and troubleshoot areas of high computational resource consumption in programs.
• Memory: Memory analysis, which is generally used to troubleshoot memory usage, memory leaks, and other issues.
• Block: Blocking analysis, which samples the blocking calls in the program.
• Mutex: Mutex analysis, which samples the competition for mutex locks.

For more information about pprof, you can refer to this article.

trace

Program trace debugging:

1

go tool trace -http=":6060" ./ssd_336959_20190704_105540_trace

The trace command allows you to trace and collect information over a period of time, then dump it to a file, and finally analyze the dump file using go tool trace and open it in a web format.

Unit Testing

The importance of unit testing is self-evident. In Golang, files ending with _test.go are considered test files. As a modern language, Golang supports unit testing at the language tool level.

Running Unit Tests

There are two ways to execute unit tests:

• Run go test directly, which is the simplest method.
• Compile the test files first, then run them. This method provides more flexibility.

Running go test

1
2
3
4
5
6

// Run go test directly in your project directory.
 go test .
 // Specify the running function.
 go test -run=TestPutAndGetKeyValue
 // Print details.
 go test -v

Compilation and Execution

Essentially, running Golang unit tests involves compiling *_test.go files into binaries and then running these binaries. When you execute go test, the tool handles these actions for you, but you can also perform them separately.

Compile the test files to generate the test executable:

1
2
3
4

// Compile the .test file first.
go test -c -coverpkg=. -covermode=atomic -o 1TestSwapInt32_in_sync_atomic.test sync/atomic
 // Specify running a file.
./1TestSwapInt32_in_sync_atomic.test -test.timeout=10m0s -test.v=true -test.run=TestSwapInt32

This method is usually used in the following scenarios:

1. Compile on one machine and run tests on another.
2. Debugging test programs.

Code Coverage Analysis

Golang’s code coverage is based on unit tests, which serve as the starting point for measuring code coverage of your business code. The operation is simple:

1. Add the -coverprofile parameter when running tests to record code coverage.
2. Use the go tool cover command to analyze and generate coverage reports.

1
2

go test -coverprofile=coverage.out
go tool cover -func=coverage.out

The output will be similar to the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

hxzhouh@hxzhouhdeMacBook-Pro ~/w/g/s/sync> go tool cover -func=coverage.out                                                                                                                heads/go1.21.4?
sync/cond.go:47:        NewCond                         100.0%
sync/cond.go:66:        Wait                            100.0%
sync/cond.go:81:        Signal                          100.0%
sync/cond.go:90:        Broadcast                       100.0%
sync/cond.go:98:        check                           100.0%
sync/cond.go:116:       Lock                            0.0%
sync/cond.go:117:       Unlock                          0.0%
sync/map.go:104:        newEntry                        100.0%
sync/map.go:110:        loadReadOnly                    100.0%
sync/map.go:120:        Load                            100.0%
sync/map.go:145:        load                            100.0%
.......

This way, you can see the code coverage for each function.

Program Debugging

Program debugging mainly relies on two tools:

1
2

1. dlv
2. gdb

Here, I recommend dlv because GDB’s functionality is limited. GDB does not understand Golang’s specific types such as channels, maps, and slices. GDB’s native support for goroutines is limited since it only understands threads. However, GDB has one irreplaceable feature, which is the gcore command.

dlv Debugging

Debugging Binaries

1

dlv exec <path/to/binary> [flags]

For example:

1

dlv exec ./main

Debugging binaries with dlv and passing arguments:

1

dlv exec ./main -- --audit=./d

Debugging Processes

1

dlv attach ${pid} [executable] [flags]

The process ID is mandatory. For example:

1

dlv attach 12808 ./main

Debugging Core Files

Debugging core files with dlv and redirecting standard output to a file:

1

dlv core <executable> <core> [flags]

1

dlv core ./main core.277282

Common Debugging Syntax

System Summary

Program Execution

1. call: call a function (note that this will cause the entire program to run).
2. continue: resume execution.
3. next: step over.
4. restart: restart.
5. step: step into a function.
6. step-instruction: step into a specific assembly instruction.
7. stepout: step out of the current function.

Breakpoint-related

1. break (alias: b): set a breakpoint.
2. breakpoints (alias: bp): print all breakpoint information.
3. clear: clear a breakpoint.
4. clearall: clear all breakpoints.
5. condition (alias: cond): set a conditional breakpoint.
6. on: set a command to be executed when a breakpoint is hit.
7. trace (alias: t): set a tracepoint, which is also a breakpoint but does not stop the program when hit; it only prints a line of information. This command is useful in certain scenarios where stopping the program affects logic (e.g., business timeouts), and you only want to print a specific variable.

Information Printing

• args: print function arguments.
• examinemem (alias: x): a powerful command for examining memory, similar to gdb’s x command.
• locals: print local variables.
• print (alias: p): print an expression or variable.
• regs: print register information.
• set: set variable value.
• vars: print global variables (package variables).
• whatis: print type information.

Goroutine-related

• goroutine (alias: gr): print information of a specific goroutine.
• goroutines (alias: grs): list all goroutines.
• thread (alias: tr): switch to a specific thread.
• threads: print information of all threads.

Stack-related

• deferred: execute commands in the context of a defer function.
• down: move up the stack.
• frame: jump to a specific stack frame.
• stack (alias: bt): print stack information.
• up: move down the stack.

Other Commands

• config: modify configurations.
• disassemble (alias: disass): disassemble.
• edit (alias: ed): omitted.
• exit (alias: quit | q): omitted.
• funcs: print all function symbols.
• libraries: print all loaded dynamic libraries.
• list (alias: ls | l): display source code.
• source: load commands.
• sources: print source code.
• types: print all type information.

The above commands cover the complete set of commands supported by dlv, which meet our debugging needs (some commands are only applicable during development and debugging, as it is not possible to single-step debug on production code in most cases).

Application Examples

Print Global Variables

1

(dlv) vars

This is very useful for inspecting global variables.

Conditional Breakpoints

1
2
3
4
5
6

# Set a breakpoint first.
(dlv) b
# Check breakpoint information.
(dlv) bp
# Customize the condition.
(dlv) condition 2 i==2 && j==7 && z==32

Inspecting the Stack

1
2
3
4

# Show all stacks.
(dlv) goroutines
# Expand all stacks.
(dlv) goroutines -t

Examining Memory

1

(dlv) x -fmt hex -len 20 0xc00008af38

The x command is the same as gdb’s x command.

gdb Debugging

GDB’s support for Golang debugging is achieved through a Python script called src/runtime/runtime-gdb.py, so its functionality is limited. GDB can only perform basic variable printing and cannot understand some of Golang’s specific types such as channels, maps, and slices. GDB cannot directly debug goroutines because it only understands threads. However, GDB has one feature that cannot be replaced, which is the gcore command.

dlv Debugging Example

Debugging a Binary

1

dlv exec ./main

Debugging a Process

1

dlv attach 12808 ./main

Debugging a Core File

1

dlv core ./main core.277282

gdb Debugging Example

1

gdb ./main

Print Global Variables (note the single quotation marks)

1

(gdb) p 'runtime.firstmoduledata'

Due to GDB’s limited understanding of Golang’s type system, sometimes it may not be able to print variables, so please pay attention to this.

Print Array Length

1

(gdb) p $len(xxx)

Therefore, I usually only use GDB to generate core files.

Tips and Tricks

Don’t know how to set breakpoints in functions?

Sometimes you don’t know how to set breakpoints in a function. You can use nm to query the function and then set a breakpoint, which will ensure that you hit the breakpoint.

Don’t know the calling context?

Add the following line in your code:

1

debug.PrintStack()

This will print the stack trace at the current code position, allowing you to understand the calling path of the function.

Don’t know how to enable pprof?

There are two ways to enable pprof, corresponding to two packages:

• net/http/pprof: used in web server scenarios.
• runtime/pprof: used in non-server applications.

These two packages are essentially the same, with net/http/pprof being a web wrapper on top of runtime/pprof.

Using net/http/pprof

1

import _ "net/http/pprof"

Using runtime/pprof

This method is usually used for performance optimization. When running a program that is not a server application, you want to find bottlenecks, so you typically use this method.

1
2
3
4
5
6
7
8

// CPU pprof file path
    f, err := os.Create("cpufile.pprof")
    if err != nil {
        log.Fatal(err)
    }
    // Start CPU pprof
    pprof.StartCPUProfile(f)
    defer pprof.StopCPUProfile()

Why does the code sometimes execute unexpectedly during single-step debugging?

This situation is usually caused by compiler optimization, such as function inlining and removal of redundant logic and parameters from the compiled binary. This can cause unexpected execution during dlv single-step debugging or prevent the printing of certain variables. The solution to this problem is to disable compiler optimization.

1

go build -gcflags "-N -l"

Conclusion

This article provides a systematic overview of the techniques and usage of Golang program debugging:

1. The language tool package provides built-in tools that support assembly, disassembly, pprof analysis, symbol table queries, and other practical functions.
2. The language tool package integrates unit testing, and code coverage relies on triggering unit tests.
3. The powerful dlv/gdb tools serve as the main debugging tools, supporting the analysis of binaries, processes, and core files.