Golang High-Performance Programming EP4: reflect
Practical Tips and Best Practices for Enhancing Reflection Performance
reflect allows Go to dynamically obtain the types and values of objects at runtime and create objects dynamically. Reflection can help abstract and simplify code, enhancing development efficiency. The Go standard library and many open-source projects utilize Go’s reflection capabilities, such as the json package for serialization and deserialization and ORM frameworks like gorm/xorm. This article aims to explore and reflect on how to improve its performance.
How to Simplify Code Using Reflection
Let’s implement a simple functionality using reflection to see how it can help us simplify the code. Suppose we have a configuration class Config, each field representing a configuration item. We need to read environment variables prefixed with CONFIG_xxxx and initialize Config them accordingly.
List 1: MySQL config
// https://github.com/go-sql-driver/mysql/blob/v1.8.1/dsn.go#L37
type Config struct {
User string `json:"user"` // Username
Passwd string `json:"passwd"` // Password (requires User)
Net string `json:"net"` // Network (e.g. "tcp", "tcp6", "unix". default: "tcp")
Addr string `json:"addr"` // Address (default: "127.0.0.1:3306" for "tcp" and "/tmp/mysql.sock" for "unix")
DBName string `json:"db_name"` // Database name
//.....
}Footguns
It’s easy to write code like this:
List 2: Initializing Config using a for-loop
func InitConfig() *Config {
cfg := &Config{}
keys := []string{"CONFIG_MYSQL_USER", "CONFIG_MYSQL_PASSWD", "CONFIG_MYSQL_NET", "CONFIG_MYSQL_ADDR", "CONFIG_MYSQL_DB_NAME"}
for _, key := range keys {
if env, exist := os.LookupEnv(key); exist {
switch key {
case "CONFIG_MYSQL_USER":
cfg.User = env
case "CONFIG_MYSQL_PASSWORD":
cfg.Passwd = env
case "CONFIG_MYSQL_NET":
cfg.Net = env
case "CONFIG_MYSQL_ADDR":
cfg.Addr = env
case "CONFIG_MYSQL_DB_NAME":
cfg.DBName = env
}
}
}
return cfg
}However, using hardcoding means that if the Config structure changes, such as modifying the corresponding json This logic must also change when fields change, or a configuration item is deleted or added. A bigger problem is that it's very error-prone and hard to test.
If we switch to using reflection, the implementation code would look like this:
List 3: Initializing Config using reflect
func InitConfig2() *Config {
config := Config{}
typ := reflect.TypeOf(config)
value := reflect.Indirect(reflect.ValueOf(&config))
for i := 0; i < typ.NumField(); i++ {
f := typ.Field(i)
if v, ok := f.Tag.Lookup("json"); ok {
key := fmt.Sprintf("CONFIG_MYSQL_%s", strings.ToUpper(v))
if env, exist := os.LookupEnv(key); exist {
value.FieldByName(f.Name).Set(reflect.ValueOf(env))
}
}
}
return &config
}The implementation logic is quite simple:
- At runtime, use reflection to obtain the
Tagproperty of each field inConfigand concatenate the corresponding environment variable names. - Check if the environment variable exists. If it does, assign its value to the field. This way, regardless of adding or deleting fields
Config, theInitConfigthe function doesn't need modification. Isn't it much more straightforward than using a for loop?
Reflection Performance
We’ve often heard that the performance of reflection is poor. When comparing JSON parsing libraries, we also verified that the official JSON Unmarshal library performs relatively poorly because it needs to execute more instructions. So, how poor is the performance of reflection exactly? Let's perform a benchmark to find out.
List 4: Benchmark test for New and Reflect Performance
func BenchmarkNew(b *testing.B) {
var config *Config
for i := 0; i < b.N; i++ {
config = new(Config)
}
_ = config
}
func BenchmarkReflectNew(b *testing.B) {
var config *Config
typ := reflect.TypeOf(Config{})
b.ResetTimer()
for i := 0; i < b.N; i++ {
config, _ = reflect.New(typ).Interface().(*Config)
}
_ = config
}
----
➜ go_reflect git:(main) ✗ go test --bench .
goos: darwin
goarch: arm64
pkg: blog-example/go/go_reflect
BenchmarkNew-10 47675076 25.40 ns/op
BenchmarkReflectNew-10 36163776 32.51 ns/op
PASS
ok blog-example/go/go_reflect 3.895sTheir performance gap isn't significant if it’s just a creation scenario. Let’s test the field modification scenario. Two ways to modify fields using reflection are FieldByName and Field index mode. We'll test the performance of both modes separately.
List 5: Testing the performance of modifying fields
func BenchmarkFieldSet(b *testing.B) {
config := new(Config)
b.ResetTimer()
for i := 0; i < b.N; i++ {
config.Net = "tcp4"
config.Addr = "127.0.0.1:3306"
config.Passwd = "123456"
config.User = "admin"
}
}
func BenchmarkFieldSetFieldByName(b *testing.B) {
config := new(Config)
value := reflect.ValueOf(config).Elem()
b.ResetTimer()
for i := 0; i < b.N; i++ {
value.FieldByName("Net").SetString("tcp4")
value.FieldByName("Addr").SetString("127.0.0.1:3306")
value.FieldByName("Passwd").SetString("123456")
value.FieldByName("User").SetString("admin")
}
}
func BenchmarkFieldSetField(b *testing.B) {
config := new(Config)
value := reflect.Indirect(reflect.ValueOf(config))
b.ResetTimer()
for i := 0; i < b.N; i++ {
value.Field(0).SetString("tcp4")
value.Field(1).SetString("127.0.0.1:3306")
value.Field(2).SetString("123456")
value.Field(3).SetString("admin")
}
}
----
➜ go_reflect git:(main) ✗ go test --bench="BenchmarkFieldSet*"
goos: darwin
goarch: arm64
pkg: blog-example/go/go_reflect
BenchmarkFieldSet-10 1000000000 0.3282 ns/op
BenchmarkFieldSetFieldByName-10 6471114 185.3 ns/op
BenchmarkFieldSetField-10 100000000 11.88 ns/op
PASS
ok blog-example/go/go_reflect 3.910sThe performance difference is significant. Non-reflective methods have a gap of two orders of magnitude compared to the reflect Field index mode. Compared to the reflect FieldByName mode, the gap reaches three orders of magnitude. One puzzling point is that the FieldByName mode and the Field index mode also have an order of magnitude difference. However, we can find the answer in the source code.
List 6: reflect/value.go
func (v Value) FieldByName(name string) Value {
v.mustBe(Struct)
if f, ok := toRType(v.typ()).FieldByName(name); ok {
return v.FieldByIndex(f.Index)
}
return Value{}
}
// FieldByIndex returns the nested field corresponding to index.// It panics if evaluation requires stepping through a nil
// pointer or a field that is not a struct.
func (v Value) FieldByIndex(index []int) Value {
if len(index) == 1 {
return v.Field(index[0])
}
v.mustBe(Struct)
for i, x := range index {
if i > 0 {
if v.Kind() == Pointer && v.typ().Elem
().Kind() == abi.Struct {
if v.IsNil() {
panic("reflect: indirection through nil pointer to embedded struct")
}
v = v.Elem()
}
}
v = v.Field(x)
}
return v
}List 7: reflect/type.go
func (t *rtype) FieldByName(name string) (StructField, bool) {
if t.Kind() != Struct {
panic("reflect: FieldByName of non-struct type " + t.String())
}
tt := (*structType)(unsafe.Pointer(t))
return tt.FieldByName(name)
}
// FieldByName returns the struct field with the given name// and a boolean to indicate if the field was found.
func (t *structType) FieldByName(name string) (f StructField, present bool) {
// Quick check for top-level name, or struct without embedded fields.
hasEmbeds := false
if name != "" {
for i := range t.Fields {
tf := &t.Fields[i]
if tf.Name.Name() == name {
return t.Field(i), true
}
if tf.Embedded() {
hasEmbeds = true
}
}
}
if !hasEmbeds {
return
}
return t.FieldByNameFunc(func(s string) bool { return s == name })
}Internally, fields are stored sequentially, so accessing them by index has an O(1) efficiency, while accessing by Name requires traversing all fields, which has an O(N) efficiency. The more fields (including methods) a struct contains, the more significant the efficiency difference between the two methods. However, remembering the order of fields can be error-prone.
How to Improve Performance
Avoid Using Reflect Whenever Possible
Using reflection for assignments is highly inefficient. If alternative methods exist, avoid using reflection, especially in repeatedly called hotspots. For example, in an RPC protocol, where structures need to be serialized and deserialized, avoid using Go’s built-in json Marshal and Unmarshal methods because the standard library's JSON serialization and deserialization are implemented using reflection. Using fastjson instead of the standard library can yield a performance improvement of about ten times.
Use Field Index Mode Whenever Possible
From the previous benchmark, the Field index mode is nearly an order of magnitude faster than the FieldByName method. This difference becomes more evident when there are more fields. However, using the Field index mode can be cumbersome because you need to remember the indexes, making it difficult to modify. In this case, you can use a map to cache the names and indexes.For example:
List 8: Cache FieldByName and Field Index
func BenchmarkFieldSetFieldByNameCache(b *testing.B) {
config := new(Config)
typ := reflect.TypeOf(Config{})
value := reflect.ValueOf(config).Elem()
cache := make(map[string]int)
for i := 0; i < typ.NumField(); i++ {
cache[typ.Field(i).Name] = i
}
b.ResetTimer()
for i := 0; i < b.N; i++ {
value.Field(cache["Net"]).SetString("tcp4")
value.Field(cache["Addr"]).SetString("127.0.0.1:3306")
value.Field(cache["Passwd"]).SetString("123456")
value.Field(cache["User"]).SetString("admin")
}
}
----
BenchmarkFieldSetFieldByNameCache-10 32121740 36.85 ns/opThis method shows a fourfold improvement over direct use FieldByName, which is quite significant.
Conclusion
- This article does not delve into the details of
reflect. If you want to learn more aboutreflect, you can refer to the following articles:
2. Reflect is used extensively in various foundational libraries. Using reflect appropriately can simplify code, but it comes with performance costs, potentially reducing performance by up to three orders of magnitude in extreme cases.
3. You can optimize performance using Field index + cache.
Do you have any other thoughts on reflection? Leave a comment and discuss it with me.
