RPC Action EP2: Using Protobuf and Creating a Custom Plugin

In the previous article, I implemented a simple RPC interface using the net/rpc package and tried out the Gob encoding that comes with net/rpc and JSON encoding to learn some basics of Golang RPC. In this post, I’ll combine net/rpc with protobuf and create my protobuf plugin to help us generate code, so let’s get started.

This article was first published in the Medium MPP plan. If you are a Medium user, please follow me on Medium. Thank you very much.

We must have used gRPC and protobuf during our work, but they are not bound. gRPC can be encoded using JSON, and protobuf can be implemented in other languages.

Protocol Buffers (Protobuf) is a free and open-source cross-platform data format used to serialize structured data. It is useful in developing programs that communicate with each other over a network or for storing data. The method involves an interface description language that describes the structure of some data and a program that generates source code from that description for generating or parsing a stream of bytes that represents the structured data.

An example of using protobuf

First we write a proto file hello-service.proto that defines a message “String”

1
2
3
4
5
6
7


syntax = "proto3";
package api;
option  go_package="api";

message String {
  string value = 1;
}

Then use the protoc utility to generate the Go code for the message String

1

protoc --go_out=. hello-service.proto

Then we modify the Hello function’s arguments to use the String generated by the protobuf file.

1
2
3


type HelloServiceInterface = interface {  
    Hello(request api.String, reply *api.String) error  
}  

Using it is no different from before, even, it is not as convenient as using string directly. So why should we use protobuf? As I said earlier, using Protobuf to define language-independent RPC service interfaces and messages, and then using the protoc tool to generate code in different languages, is where its real value lies. For example, use the official plugin protoc-gen-go to generate gRPC code.

1

protoc --go_out=plugins=grpc. hello-service.proto

Plugin system for protoc

To generate code from protobuf files, we must install the protoc , but the protoc doesn’t know what our target language is, so we need plugins to help us generate code. how does protoc’s plugin system work? Take the above grpc as an example.
There is a --go_out parameter here. Since the plugin we’re calling is protoc-gen-go, the parameter is called go_out; if the name was XXX, the parameter would be called XXX_out.
When protoc is running, it will first parse the protobuf file and generate a set of Protocol Buffers-encoded descriptive data. It will first determine whether or not the go plugin is included in protoc, and then it will try to look for protoc-gen-go in $PATH, and if it can’t find it, it will report an error, and then it will run protoc-gen-go. protoc-gen-go command and sends the description data to the plugin command via stdin. After the plugin generates the file contents, it then inputs Protocol Buffers encoded data to stdout to tell protoc to generate the specific file.
plugins=grpc is a plugin that comes with protoc-gen-go in order to invoke it. If you don’t use it, it will only generate a message in Go, but you can use this plugin to generate grpc-related code.

Customize a protoc plugin

If we add Hello interface timing to protobuf, can we customize a protoc plugin to generate code directly?

1
2
3
4
5
6
7
8
9


syntax = "proto3";  
package api;  
option  go_package="./api";  
service HelloService {  
  rpc Hello (String) returns (String) {}  
}  
message String {  
  string value = 1;
}

Objective

For this article, my goal was to create a plugin that would then be used to generate RPC server-side and client-side code that would look something like this.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35


// HelloService_rpc.pb.go
type HelloServiceInterface interface {  
    Hello(String, *String) error  
}  
  
func RegisterHelloService(  
    srv *rpc.Server, x HelloServiceInterface,  
) error {  
    if err := srv.RegisterName("HelloService", x); err != nil {  
       return err  
    }  
    return nil  
}  
  
type HelloServiceClient struct {  
    *rpc.Client  
}  
  
var _ HelloServiceInterface = (*HelloServiceClient)(nil)  
  
func DialHelloService(network, address string) (  
    *HelloServiceClient, error,  
) {  
    c, err := rpc.Dial(network, address)  
    if err != nil {  
       return nil, err  
    }  
    return &HelloServiceClient{Client: c}, nil  
}  
  
func (p *HelloServiceClient) Hello(  
    in String, out *String,  
) error {  
    return p.Client.Call("HelloService.Hello", in, out)  
}

This would change our business code to look like the following

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36


// service
func main() {  
    listener, err := net.Listen("tcp", ":1234")  
    if err != nil {  
       log.Fatal("ListenTCP error:", err)  
    }  
    _ = api.RegisterHelloService(rpc.DefaultServer, new(HelloService))  
    for {  
       conn, err := listener.Accept()  
       if err != nil {  
          log.Fatal("Accept error:", err)  
       }  
       go rpc.ServeConn(conn)  
    }  
}  
  
type HelloService struct{}  
  
func (p *HelloService) Hello(request api.String, reply *api.String) error {  
    log.Println("HelloService.proto Hello")  
    *reply = api.String{Value: "Hello:" + request.Value}  
    return nil  
}
// client.go
func main() {  
    client, err := api.DialHelloService("tcp", "localhost:1234")  
    if err != nil {  
       log.Fatal("net.Dial:", err)  
    }  
    reply := &api.String{}  
    err = client.Hello(api.String{Value: "Hello"}, reply)  
    if err != nil {  
       log.Fatal(err)  
    }  
    log.Println(reply)  
}

Based on the generated code, our workload is already much smaller and the chances of error are already very small. A good start.
Based on the api code above, we can pull out a template file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


const tmplService = `  
package {{.PackageName}}  
import (  
    "net/rpc")  
{{$root := .}}  
type {{.ServiceName}}Interface interface {  
    {{- range $_, $m := .MethodList}}    {{$m.MethodName}}({{$m.InputTypeName}}, *{{$m.OutputTypeName}}) error    {{- end}}}  
func Register{{.ServiceName}}(  
    srv *rpc.Server, x {{.ServiceName}}Interface,) error {  
    if err := srv.RegisterName("{{.ServiceName}}", x); err != nil {        return err    }    return nil}  
type {{.ServiceName}}Client struct {  
    *rpc.Client}  
var _ {{.ServiceName}}Interface = (*{{.ServiceName}}Client)(nil)  
func Dial{{.ServiceName}}(network, address string) (  
    *{{.ServiceName}}Client, error,) {  
    c, err := rpc.Dial(network, address)    if err != nil {        return nil, err    }    return &{{.ServiceName}}Client{Client: c}, nil}  
{{range $_, $m := .MethodList}}  
func (p *{{$root.ServiceName}}Client) {{$m.MethodName}}(  
    in {{$m.InputTypeName}}, out *{{$m.OutputTypeName}},) error {  
    return p.Client.Call("{{$root.ServiceName}}.{{$m.MethodName}}", in, out)}  
{{end}}  
`

The whole template is clear, and there are some placeholders in it, such as MethodName, ServiceName, etc., which we’ll cover later.

How to develop a plug-in?

Google released the Go language API 1, which introduces a new package google.golang.org/protobuf/compile R/protogen, which greatly reduces the difficulty of plugins development:

First of all, we create a go language project, such as protoc-gen-go-spprpc
Then we need to define a protogen.Options, then call its Run method, and pass in a func(*protogen.Plugin) error callback. This is the end of the main process code.
We can also set the ParamFunc parameter of protogen.Options, so that protogen will automatically parse the parameters passed by the command line for us. Operations such as reading and decoding protobuf information from standard input, encoding input information into protobuf and writing stdout are all handled by protogen. What we need to do is to interact with protogen.Plugin to implement code generation logic.

The most important thing for each service is the name of the service, and then each service has a set of methods. For the method defined by the service, the most important thing is the name of the method, as well as the name of the input parameter and the output parameter type. Let’s first define a ServiceData to describe the meta information of the service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12


// ServiceData 
type ServiceData struct {  
    PackageName string  
    ServiceName string  
    MethodList  []Method  
}
// Method 
type Method struct {  
    MethodName     string  
    InputTypeName  string  
    OutputTypeName string  
}

Then comes the main logic, and the code generation logic, and finally the call to tmpl to generate the code.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44


func main() {  
    protogen.Options{}.Run(func(gen *protogen.Plugin) error {  
       for _, file := range gen.Files {  
          if !file.Generate {  
             continue  
          }  
          generateFile(gen, file)  
       }  
       return nil  
    })  
}  
  
// generateFile function definition
func generateFile(gen *protogen.Plugin, file *protogen.File) {  
    filename := file.GeneratedFilenamePrefix + "_rpc.pb.go"  
    g := gen.NewGeneratedFile(filename, file.GoImportPath)  
    tmpl, err := template.New("service").Parse(tmplService)  
    if err != nil {  
       log.Fatalf("Error parsing template: %v", err)  
    }  
    packageName := string(file.GoPackageName)  
// Iterate over each service to generate code
    for _, service := range file.Services {  
       serviceData := ServiceData{  
          ServiceName: service.GoName,  
          PackageName: packageName,  
       }  
       for _, method := range service.Methods {  
          inputType := method.Input.GoIdent.GoName  
          outputType := method.Output.GoIdent.GoName  
  
          serviceData.MethodList = append(serviceData.MethodList, Method{  
             MethodName:     method.GoName,  
             InputTypeName:  inputType,  
             OutputTypeName: outputType,  
          })  
       }  
// Perform template rendering
       err = tmpl.Execute(g, serviceData)  
       if err != nil {  
          log.Fatalf("Error executing template: %v", err)  
       }  
    }  
}

Debug plugin

Finally, we put the compiled binary execution file protoc-gen-go-spprpc in $PATH, and then run protoc to generate the code we want.

1

protoc --go_out=.. --go-spprpc_out=.. HelloService.proto

Because protoc-gen-go-spprpc has to depend on protoc to run, it’s a bit tricky to debug. We can use

1

fmt.Fprintf(os.Stderr, "Fprintln: %v\n", err)

To print the error log to debug.

Summary

That’s all there is to this article. We first implemented an RPC call using protobuf and then created a protobuf plugin to help us generate the code. This opens the door for us to learn protobuf + RPC, and is our path to a thorough understanding of gRPC. I hope everyone can master this technology.