v 1.0.0 release

1. add middleware docs
2. optimize code structure

.github/workflows/release.yml

@@ -24,7 +24,7 @@ jobs:
       - name: Go Build
         run: |
           make build GOOS=linux ARCH=amd64 TARGET=superlcx_linux_amd64;  \
-          make build GOOS=darwin arch=amd64 TARRGET=superlcx_darwin_amd64; \
+          make build GOOS=darwin arch=amd64 TARGET=superlcx_darwin_amd64; \
           make build GOOS=windows arch=amd64 TARGET=superlcx_windows_amd64.exe;
 
       - name: Get tag

README.CN.md

@@ -5,14 +5,17 @@
 
 # 用法
 ```bash
-~/superlcx(master*) » ./superlcx -h                                                                                              dashjay@zhaowenjies-MacBook-Pro
+~/superlcx(master) » ./superlcx -h                                                                                               dashjay@zhaowenjies-MacBook-Pro
 Usage of ./superlcx:
+  -M string
+        middleware, comma separated if more than one, eg: --M stdout,dumps
   -host string
         target host:port (default "0.0.0.0:8081")
   -l int
         listen port (default 8080)
   -m string
         run mode (default "proxy")
+  -v    show version and about then exit.
 ```
 
 ### 工作模式
@@ -26,3 +29,26 @@ Usage of ./superlcx:
 
 - 融合模式（开发中）
     - 期待优点：占用非常小的内存，提供丰富的接口例如`proxy_pass`等
+
+### -M 中间件
+当工作在代理模式下，可以调用中间件来对过程中的流量进行分析。例如，系统内置stdout中间件（示例中间件），可以通过`-M stdout`来使用。
+（必须在代理模式下才能生效）
+
+如果想自己实现一个中间件请查看：
+[中间件的编写规范](./docs/middleware.CN.md)
+
+编写后可以放入middlewares文件夹下，建议文件结构保持如下。
+```
+middlewares
+└── stdout
+    └── handler.go
+```
+
+接口在`handler.go`下分别暴露如下，如需载入配置，请自行载入，后期打算引入通用map来协助配置。
+```
+func HandleRequest(req *http.Request)
+func HandleResponse(req *http.Response)
+```
+
+**警告：** 中间件组件通常会在req和resp上做一些事情，这必然会对bufio做很多io，内存抖动可能非常严重。
+我考虑过编写一个中心中间件来转储req和resp，然后流式调用所有中间件，比如pipeline，但是如果其中一些更改了req或resp，也会导致其他问题，所以现在中间件组件不会相互影响。(你可以把上下文设置到req体的header里，这样可以与之后的中间件进行一些交互)

README.md

@@ -5,14 +5,18 @@ port transfer tool with middleware kit
 
 # usage
 ```bash
-~/superlcx(master*) » ./superlcx -h                                                                                              dashjay@zhaowenjies-MacBook-Pro
+~/superlcx(master) » ./superlcx -h                                                                                               dashjay@zhaowenjies-MacBook-Pro
 Usage of ./superlcx:
+  -M string
+        middleware, comma separated if more than one, eg: --M stdout,dumps
   -host string
         target host:port (default "0.0.0.0:8081")
   -l int
         listen port (default 8080)
   -m string
         run mode (default "proxy")
+  -v    show version and about then exit.
+
 ```
 
 ### mode
@@ -25,4 +29,27 @@ Usage of ./superlcx:
     - disadvantages: the copy mode know nothing about application layer. all things it can do is dumping them all out.
     
 - hybrid(ing)
-    - (expect)advantages: allocate low memory and expose more API like proxy_pass...etc
+    - (expect)advantages: allocate low memory and expose more API like proxy_pass...etc
+
+### -M middleware
+When working in the proxy mode, middleware can be invoked to analyze the traffic in the process. For example, the built-in stdout middleware (sample middleware) can be used via '-M stdout'.
+(Must be in the proxy mode to work)
+
+If you want to implement your own middleware, check out:
+[Middleware standard](./docs/middleware.md)
+
+Once every thing ok, it can be placed under the middlewares folder. It is disambiguated tha that the file structure remain as follows.
+```
+middlewares
+└── stdout
+    └── handler.go
+```
+
+The interfaces are exposing as follows under 'handler.go'. If you need to load the configuration, load it yourself.(I will then use TOML to pass the configuration)
+```
+func HandleRequest(req *http.Request)
+func HandleResponse(req *http.Response)
+```
+
+**WARNING:** Middleware components may do something on the req and resp body, That's bound to do a lot of io with bufio, the memory jitters can be very serious.
+I thought about writing a center middleware to dump the req and resp out, and then call all middleware like pipeline, but if some of them change the req or resp, it can also cause other problems, so now middleware components do not affect each other.

core/copy.go

@@ -13,7 +13,6 @@ type SapCopy struct {
 }
 
 func (s *SapCopy) Serve() {
-
 	for {
 		conn, err := s.lis.Accept()
 		if err != nil {
@@ -33,7 +32,6 @@ func (s *SapCopy) Serve() {
 				io.Copy(conn2, conn)
 				wg.Done()
 			}()
-
 			go func() {
 				io.Copy(conn, conn2)
 				wg.Done()

core/proxy.go

@@ -20,6 +20,11 @@ type Proxy struct {
 	respHandlers []func(resp *http.Response)
 }
 
+func (p *Proxy) RegisterMiddleware(reqH func(req *http.Request), respH func(resp *http.Response)) {
+	p.reqHandlers = append(p.reqHandlers, reqH)
+	p.respHandlers = append(p.respHandlers, respH)
+}
+
 // NewSapProxy 构建一个SapProxy
 func NewSapProxy(lis net.Listener, defaultHost string, middleware string) *Proxy {
 	u, err := url.Parse(fmt.Sprintf("http://%s", defaultHost))
@@ -33,42 +38,12 @@ func NewSapProxy(lis net.Listener, defaultHost string, middleware string) *Proxy
 		reqHandlers:  make([]func(req *http.Request), 0),
 		respHandlers: make([]func(resp *http.Response), 0),
 	}
-
-	find := func(pluginName string) (func(req *http.Request), func(resp *http.Response)) {
-		p, err := plugin.Open(pluginName)
-		if err != nil {
-			panic(err)
-		}
-		req, err := p.Lookup("HandleRequest")
-		if err != nil {
-			panic(err)
-		}
-		resp, err := p.Lookup("HandleResponse")
-		if err != nil {
-			panic(err)
-		}
-		return req.(func(req *http.Request)), resp.(func(resp *http.Response))
-	}
-
-	if middleware != "" {
-		ms := strings.Split(middleware, ",")
-		for _, m := range ms {
-			switch m {
-			case "stdout":
-				p.reqHandlers = append(p.reqHandlers, stdout.HandleRequest)
-				p.respHandlers = append(p.respHandlers, stdout.HandleResponse)
-			default:
-				reqH, respH := find(m)
-				p.reqHandlers = append(p.reqHandlers, reqH)
-				p.respHandlers = append(p.respHandlers, respH)
-			}
-		}
-	}
+	p.Register(middleware)
 	return p
 }
 
-func (s *Proxy) director(req *http.Request) {
-	for _, fn := range s.reqHandlers {
+func (p *Proxy) director(req *http.Request) {
+	for _, fn := range p.reqHandlers {
 		fn(req)
 	}
 	if _, ok := req.Header["User-Agent"]; !ok {
@@ -86,7 +61,7 @@ func (s *Proxy) director(req *http.Request) {
 		}
 		return a + b
 	}
-	target := s.defaultUrl
+	target := p.defaultUrl
 	targetQuery := target.RawQuery
 	req.URL.Scheme = target.Scheme
 	req.URL.Host = target.Host
@@ -100,7 +75,7 @@ func (s *Proxy) director(req *http.Request) {
 
 type myTripper struct {
 	http.RoundTripper
-	s *Proxy
+	p *Proxy
 }
 
 func (t *myTripper) RoundTrip(req *http.Request) (resp *http.Response, err error) {
@@ -111,21 +86,52 @@ func (t *myTripper) RoundTrip(req *http.Request) (resp *http.Response, err error
 		return nil, err
 	}
 
-	for _, fn := range t.s.respHandlers {
+	for _, fn := range t.p.respHandlers {
 		fn(resp)
 	}
 
 	return resp, nil
 }
-func (s *Proxy) modifyResponse(r *http.Response) error {
+func (p *Proxy) modifyResponse(r *http.Response) error {
 	return nil
 }
 
-func (s *Proxy) Serve() {
-	p := &httputil.ReverseProxy{
-		Director:       s.director,
-		Transport:      &myTripper{RoundTripper: http.DefaultTransport, s: s},
-		ModifyResponse: s.modifyResponse,
+func (p *Proxy) Serve() {
+	proxy := &httputil.ReverseProxy{
+		Director:       p.director,
+		Transport:      &myTripper{RoundTripper: http.DefaultTransport, p: p},
+		ModifyResponse: p.modifyResponse,
+	}
+	panic(http.Serve(p.lis, proxy))
+}
+
+func (p *Proxy) Register(middleware string) {
+	if middleware != "" {
+		ms := strings.Split(middleware, ",")
+		for _, m := range ms {
+			switch m {
+			case "stdout":
+				p.RegisterMiddleware(stdout.HandleRequest, stdout.HandleResponse)
+			default:
+				reqH, respH := find(m)
+				p.RegisterMiddleware(reqH, respH)
+			}
+		}
+	}
+}
+
+func find(pluginName string) (func(req *http.Request), func(resp *http.Response)) {
+	p, err := plugin.Open(pluginName)
+	if err != nil {
+		panic(err)
+	}
+	req, err := p.Lookup("HandleRequest")
+	if err != nil {
+		panic(err)
+	}
+	resp, err := p.Lookup("HandleResponse")
+	if err != nil {
+		panic(err)
 	}
-	panic(http.Serve(s.lis, p))
+	return req.(func(req *http.Request)), resp.(func(resp *http.Response))
 }

docs/middleware.CN.md

@@ -0,0 +1,31 @@
+## 中间件的编写
+[EN](./middleware.md)[中文]
+
+中间件基于请求体(http.Request)和返回体(http.Response)来编写，由于两个结构都包含一些buf，例如请求和返回体的body.
+
+因此为了防止body被中间件读取后，客户端或者服务端无法收到真实的请求体和返回体，我们需要做一些操作。
+
+示例的中间件stdout中这样写到：
+```gotemplate
+func handleRequest(over chan bool, req *http.Request) {
+	content, err := httputil.DumpRequest(req, true)
+	over <- true
+	if err != nil {
+		panic(err)
+	}
+...
+```
+`httputil.DumpRequest`会将请求body读出（如果存在的话），并且将body还给req这个请求体一份。如果我们自己写中间件也应该这样来写。
+
+如果你不能很好的处理请求和返回中body，可能会造成严重的后果：
+- body为空，读时EOF error
+- body已经close，读时panic
+
+为了避免程序因为这些原因崩溃，建议使用如下方式来编写中间件。
+
+```
+content, err := httputil.DumpRequest(req, true)
+newReq, err := http.ReadRequest(bufio.NewReader(bytes.NewReader(content)))
+```
+
+这样可以完全避免req请求体收到影响，能够正常发送到服务端以收到回复。

docs/middleware.md

@@ -0,0 +1,33 @@
+[EN][中文](./middleware.CN.md)
+
+## write a middleware
+
+
+Middleware based on request body（ http.Request ）And return body（ http.Response ）Because both structures contain buf, such as the body of request and response body.
+
+Therefore, in order to prevent superlcx read body, so that the client or server can't receive the real request body or response body, we need to do some operations.
+
+The standard middleware `stdout` of the example write follows: 
+```gotemplate
+func handleRequest(over chan bool, req *http.Request) {
+	content, err := httputil.DumpRequest(req, true)
+	over <- true
+	if err != nil {
+		panic(err)
+	}
+...
+```
+
+The request body(if exists) will be read out by `httputil.DumpRequest`, and a copy of the request body will be assign to origin req. If we write a middleware by ourselves, we should do the same.
+
+If you can't handle the body in the request and response well, you may have serious consequences:
+- Body is empty, which can cause EOF error during reading.
+- The body has been closed, panic when you read it.
+
+In order to prevent the program from crashing for these reasons, it is recommended to write the middleware in the following way.
+```
+content, err := httputil.DumpRequest(req, true)
+newReq, err := http.ReadRequest(bufio.NewReader(bytes.NewReader(content)))
+```
+
+In this way, it can completely avoid the influence of req request body and send it to the server to receive the reply.