参考资料

演讲 - Slides

开源库 - Github

拓展Github - 1M Tcp连接

1. Go实现一个Web服务

Go Web服务分配Goroutine处理每个请求,这样当请求数太多时,就会开启很多当文件描述符。

2. 优化内存需求

需要足够的内存管理每个打开的文件描述符,瓶颈在于内存资源的限制,可以通过ulimits来限定最大文件打开数。

2.1 pprof

pprof包通过其可视化工具提供服务,分析HTTP服务器运行时数据

1
2
3
4
5
6
7
import _ "net/http/pprof"

go func() {
  if err := http.ListenAndServe("localhost:6060", nil); err != nil {
    log.Fatalf("Pprof failed: %v", err) 
  }
}()

2.2 内存消耗

计算公式:

$$a = buf_{net/http}$$

$$b = buf_{gorilla/ws}$$

$$Mem = conns\times(goroutine+a+b)$$

单个连接消耗内存约20K,根据公式,计算出百万连接内存消耗大约20G。

3 内存优化

优化Groutine数量

优化net/http对象资源分配

重用分配的buffer访问

3.1 优化Groutine

通道中何时存在数据,重用goroutine并减少内存占用

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// 优化前
func ws(w http.ResponseWriter, r *http.Request) { 
  // Upgrade connection ...
  // Read messages from socket
	for {
		_, msg, err := conn.ReadMessage()
		if err != nil {
			return
		}
		log.Printf("msg: %s", string(msg))
	}
}
// 优化后
func ws(w http.ResponseWriter, r *http.Request) { 
  // Upgrade connection ...
  for {
    _, msg, err := conn.ReadMessage() 
    if err != nil {
      log.Printf("Failed to read message %v", err) 
      conn.Close()
      return
    }
    log.Println(string(msg)) 
  }
}

select / poll

1
2
3
4
5
6
7
t := &syscall.Timeval{/* timeout for the call */}
if _, err := syscall.Select(maxFD+1, fds, nil, nil, t); err != nil {
return nil, err }
for _, fd := range fds {
  if fdIsSet(fdset, fd) {
  } 
}

epoll

1
2
3
4
5
6
7
8
9
fd, err := unix.EpollCreate1(0) 
if err != nil {
  return nil, err 
}
fd := websocketFD(conn)
err := unix.EpollCtl(e.fd, syscall.EPOLL_CTL_ADD, fd, &unix.EpollEvent{Events: unix.POLLIN | unix.POLLHUP, Fd: int32(fd)})
if err != nil {
  return err
}

优化后的内存消耗降低30% $$Mem = conns*buf_{gorilla/ws}$$

3.2 buffer分配

gorilla/websocket keeps a reference to the underlying buffers given by Hijack()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
var br *bufio.Reader
if u.ReadBufferSize == 0 && bufioReaderSize(netConn, brw.Reader) > 256 {
    // Reuse hijacked buffered reader as connection reader.
    br = brw.Reader
}
buf := bufioWriterBuffer(netConn, brw.Writer)
var writeBuf []byte
if u.WriteBufferPool == nil && u.WriteBufferSize == 0 && len(buf) >= maxFrameHeaderSize+256 {
    // Reuse hijacked write buffer as connection buffer.
    writeBuf = buf
}
c := newConn(netConn, true, u.ReadBufferSize, u.WriteBufferSize, u.WriteBufferPool, br, writeBuf)

可替代的websockets库:github.com/gobwas/ws

  • 在I/O操作过程,没有中间分配
  • 低级API,允许构建数据包处理和缓冲区的逻辑
  • 零拷贝
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
import "github.com/gobwas/ws"

func wsHandler(w http.ResponseWriter, r *http.Request) {
  conn, _, _, err := ws.UpgradeHTTP(r, w)
  if err != nil {
    return
  }
  // Add to epoll
}

for {
    // Fetch ready connections with epoll logic
    msg, _, err := wsutil.ReadClientData(conn)
    if err == nil {
        log.Printf("msg: %s", string(msg))
    } else {
    } 
}

内存分配降低97%,从20G降低到600M

$$Mem\approx conns$$