reflect provides Go with the ability to dynamically obtain the types and values of objects at runtime and create objects dynamically. Reflection can help abstract and simplify code, enhancing development efficiency. The Go standard library and many open-source projects utilize Go’s reflection capabilities, such as the json package for serialization and deserialization, and ORM frameworks like gorm/xorm. This article aims to explore reflect and how to improve its performance.
This article was first published in the Medium MPP plan. If you are a Medium user, please follow me on Medium. Thank you very much.
Example Code
How to Simplify Code Using Reflection
Let’s implement a simple functionality using reflection to see how it can help us simplify the code.
Suppose we have a configuration class Config, with each field representing a configuration item. We need to read environment variables prefixed with CONFIG_xxxx and initialize Config accordingly.
List 1: MySQL config
1
2
3
4
5
6
7
8
9
|
// https://github.com/go-sql-driver/mysql/blob/v1.8.1/dsn.go#L37
type Config struct {
User string `json:"user"` // Username
Passwd string `json:"passwd"` // Password (requires User)
Net string `json:"net"` // Network (e.g. "tcp", "tcp6", "unix". default: "tcp")
Addr string `json:"addr"` // Address (default: "127.0.0.1:3306" for "tcp" and "/tmp/mysql.sock" for "unix")
DBName string `json:"db_name"` // Database name
// γγγγγ
}
|
It’s easy to write code like this:
List 2: Initializing Config using a for loop
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
|
func InitConfig() *Config {
cfg := &Config{}
keys := []string{"CONFIG_MYSQL_USER", "CONFIG_MYSQL_PASSWD", "CONFIG_MYSQL_NET", "CONFIG_MYSQL_ADDR", "CONFIG_MYSQL_DB_NAME"}
for _, key := range keys {
if env, exist := os.LookupEnv(key); exist {
switch key {
case "CONFIG_MYSQL_USER":
cfg.User = env
case "CONFIG_MYSQL_PASSWORD":
cfg.Passwd = env
case "CONFIG_MYSQL_NET":
cfg.Net = env
case "CONFIG_MYSQL_ADDR":
cfg.Addr = env
case "CONFIG_MYSQL_DB_NAME":
cfg.DBName = env
}
}
}
return cfg
}
|
However, using hardcoding means that if the Config structure changes, such as modifying the corresponding json fields, deleting, or adding a configuration item, this logic also needs to change. A bigger problem is that it’s very error-prone and hard to test.
If we switch to using reflection, the implementation code would look like this:
List 3: Initializing Config using reflect
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
|
func InitConfig2() *Config {
config := Config{}
typ := reflect.TypeOf(config)
value := reflect.Indirect(reflect.ValueOf(&config))
for i := 0; i < typ.NumField(); i++ {
f := typ.Field(i)
if v, ok := f.Tag.Lookup("json"); ok {
key := fmt.Sprintf("CONFIG_MYSQL_%s", strings.ToUpper(v))
if env, exist := os.LookupEnv(key); exist {
value.FieldByName(f.Name).Set(reflect.ValueOf(env))
}
}
}
return &config
}
|
The implementation logic is quite simple:
- At runtime, use reflection to obtain the
Tag property of each field in Config and concatenate the corresponding environment variable names.
- Check if the environment variable exists. If it does, assign its value to the field.
This way, regardless of adding or deleting fields in Config, the InitConfig function doesn’t need modification. Isn’t it much simpler than using a for loop?
We’ve often heard that the performance of reflection is poor. When comparing JSON parsing libraries, we also verified that the official JSON Unmarshal library performs relatively poorly because it needs to execute more instructions. So, how poor is the performance of reflection exactly? Let’s perform a benchmark to find out.
List 4: Benchmark test for New and Reflect Performance
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
|
func BenchmarkNew(b *testing.B) {
var config *Config
for i := 0; i < b.N; i++ {
config = new(Config)
}
_ = config
}
func BenchmarkReflectNew(b *testing.B) {
var config *Config
typ := reflect.TypeOf(Config{})
b.ResetTimer()
for i := 0; i < b.N; i++ {
config, _ = reflect.New(typ).Interface().(*Config)
}
_ = config
}
----
β go_reflect git:(main) β go test --bench .
goos: darwin
goarch: arm64
pkg: blog-example/go/go_reflect
BenchmarkNew-10 47675076 25.40 ns/op
BenchmarkReflectNew-10 36163776 32.51 ns/op
PASS
ok blog-example/go/go_reflect 3.895s
|
If it’s just a creation scenario, the performance gap between the two isn’t very significant.
Let’s test the field modification scenario. There are two ways to modify fields using reflection:
FieldByName and Field index mode.
We’ll test the performance of both modes separately.
List 5: Testing the performance of modifying fields
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
|
func BenchmarkFieldSet(b *testing.B) {
config := new(Config)
b.ResetTimer()
for i := 0; i < b.N; i++ {
config.Net = "tcp4"
config.Addr = "127.0.0.1:3306"
config.Passwd = "123456"
config.User = "admin"
}
}
func BenchmarkFieldSetFieldByName(b *testing.B) {
config := new(Config)
value := reflect.ValueOf(config).Elem()
b.ResetTimer()
for i := 0; i < b.N; i++ {
value.FieldByName("Net").SetString("tcp4")
value.FieldByName("Addr").SetString("127.0.0.1:3306")
value.FieldByName("Passwd").SetString("123456")
value.FieldByName("User").SetString("admin")
}
}
func BenchmarkFieldSetField(b *testing.B) {
config := new(Config)
value := reflect.Indirect(reflect.ValueOf(config))
b.ResetTimer()
for i := 0; i < b.N; i++ {
value.Field(0).SetString("tcp4")
value.Field(1).SetString("127.0.0.1:3306")
value.Field(2).SetString("123456")
value.Field(3).SetString("admin")
}
}
----
β go_reflect git:(main) β go test --bench="BenchmarkFieldSet*"
goos: darwin
goarch: arm64
pkg: blog-example/go/go_reflect
BenchmarkFieldSet-10 1000000000 0.3282 ns/op
BenchmarkFieldSetFieldByName-10 6471114 185.3 ns/op
BenchmarkFieldSetField-10 100000000 11.88 ns/op
PASS
ok blog-example/go/go_reflect 3.910s
|
The performance difference is significant. Non-reflective methods compared to the reflect Field index mode have a gap of two orders of magnitude. Compared to the reflect FieldByName mode, the gap reaches three orders of magnitude. One puzzling point is that the FieldByName mode and the Field index mode also have an order of magnitude difference. However, we can find the answer in the source code.
List 6: reflect/value.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
|
func (v Value) FieldByName(name string) Value {
v.mustBe(Struct)
if f, ok := toRType(v.typ()).FieldByName(name); ok {
return v.FieldByIndex(f.Index)
}
return Value{}
}
// FieldByIndex returns the nested field corresponding to index.// It panics if evaluation requires stepping through a nil
// pointer or a field that is not a struct.
func (v Value) FieldByIndex(index []int) Value {
if len(index) == 1 {
return v.Field(index[0])
}
v.mustBe(Struct)
for i, x := range index {
if i > 0 {
if v.Kind() == Pointer && v.typ().Elem
().Kind() == abi.Struct {
if v.IsNil() {
panic("reflect: indirection through nil pointer to embedded struct")
}
v = v.Elem()
}
}
v = v.Field(x)
}
return v
}
|
List 7: reflect/type.go
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
|
func (t *rtype) FieldByName(name string) (StructField, bool) {
if t.Kind() != Struct {
panic("reflect: FieldByName of non-struct type " + t.String())
}
tt := (*structType)(unsafe.Pointer(t))
return tt.FieldByName(name)
}
// FieldByName returns the struct field with the given name// and a boolean to indicate if the field was found.
func (t *structType) FieldByName(name string) (f StructField, present bool) {
// Quick check for top-level name, or struct without embedded fields.
hasEmbeds := false
if name != "" {
for i := range t.Fields {
tf := &t.Fields[i]
if tf.Name.Name() == name {
return t.Field(i), true
}
if tf.Embedded() {
hasEmbeds = true
}
}
}
if !hasEmbeds {
return
}
return t.FieldByNameFunc(func(s string) bool { return s == name })
}
|
Internally, fields are stored sequentially, so accessing them by index has an O(1) efficiency, while accessing by Name requires traversing all fields, which has an O(N) efficiency. The more fields (including methods) a struct contains, the greater the efficiency difference between the two methods. However, remembering the order of fields can be error-prone.
Avoid Using Reflect Whenever Possible
Using reflection for assignments is highly inefficient. If there are alternative methods, avoid using reflection, especially in repeatedly called hotspots. For example, in an RPC protocol, where structures need to be serialized and deserialized, avoid using Go’s built-in json Marshal and Unmarshal methods because the standard library’s JSON serialization and deserialization are implemented using reflection. Using fastjson instead of the standard library can yield a performance improvement of about ten times.
Use Field Index Mode Whenever Possible
From the previous benchmark, the Field index mode is nearly an order of magnitude faster than the FieldByName method. This difference becomes more evident when there are more fields. However, using the Field index mode can be cumbersome because you need to remember the indexes, making it difficult to modify. In this case, you can use a map to cache the names and indexes.
For example:
List 8: Cache FieldByName and Field Index
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
|
func BenchmarkFieldSetFieldByNameCache(b *testing.B) {
config := new(Config)
typ := reflect.TypeOf(Config{})
value := reflect.ValueOf(config).Elem()
cache := make(map[string]int)
for i := 0; i < typ.NumField(); i++ {
cache[typ.Field(i).Name] = i
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
value.Field(cache["Net"]).SetString("tcp4")
value.Field(cache["Addr"]).SetString("127.0.0.1:3306")
value.Field(cache["Passwd"]).SetString("123456")
value.Field(cache["User"]).SetString("admin")
}
}
----
BenchmarkFieldSetFieldByNameCache-10 32121740 36.85 ns/op
|
This method shows a fourfold improvement over directly using FieldByName, which is quite significant.
Conclusion
- This article does not delve into the details of
reflect. If you want to learn more about reflect, you can refer to the following articles:
- Learning to Use Go Reflection
- Go101: Reflection
- Reflect is used extensively in various foundational libraries. Using reflect appropriately can simplify code, but it comes with performance costs, potentially reducing performance by up to three orders of magnitude in extreme cases.
- You can optimize performance using Field index + cache.
Do you have any other thoughts on reflection? Leave a comment and discuss it with me.
