使用多个并发的TLS拨号器时，Golang中的常驻内存持续增长

知识点掌握了，还需要不断练习才能熟练运用。下面golang学习网给大家带来一个Golang开发实战，手把手教大家学习《使用多个并发的TLS拨号器时，Golang中的常驻内存持续增长》，在实现功能的过程中也带大家重新温习相关知识点，温故而知新，回头看看说不定又有不一样的感悟！

好吧，这个问题已经困扰我好几个星期了，我不知道我错过了什么，也不知道泄漏在哪里，甚至不知道它是否存在。我的工作量相当简单。获取 url 列表，启动一个 goroutine 池，从通道中提取 url，并使用 tls.dialer 创建到它们的 tls 连接。下面是内存图的快照，显示了我的代码的不断上升和 poc。

我的猜测是这与 tls 包完成的分配有关，因为它似乎只会爬到它连接到的更“成功”的 url。 ie。如果大多数都没有连接，我就看不到内存的稳定增长。

这是运行中途的 pprof 输出：

showing nodes accounting for 190.70mb, 95.58% of 199.53mb total
dropped 34 nodes (cum <= 1mb)
showing top 20 nodes out of 77
   flat  flat%   sum%        cum   cum%
   51.52mb 25.82% 25.82%    51.52mb 25.82%  runtime.malg
   24.10mb 12.08% 37.90%    24.10mb 12.08%  bytes.makeslice
   17.07mb  8.55% 46.45%    41.17mb 20.63%  crypto/tls.(*conn).readhandshake
   15mb  7.52% 53.97%    78.85mb 39.52%  crypto/tls.dial
   11mb  5.51% 59.48%    11.50mb  5.76%  net.(*netfd).connect
   10mb  5.01% 64.50%    15.42mb  7.73%  context.withdeadline
   9mb  4.51% 69.01%        9mb  4.51%  net.newfd (inline)
   8mb  4.01% 73.02%    10.84mb  5.43%  time.afterfunc
   7mb  3.51% 76.53%    52.93mb 26.53%  net.(*dialer).dialcontext
   5.50mb  2.76% 79.28%     5.50mb  2.76%  context.(*cancelctx).done
   5mb  2.51% 81.79%    84.35mb 42.28%  main.main.func3
   5mb  2.51% 84.30%        5mb  2.51%  net.(*netfd).connect.func2
   4.50mb  2.26% 86.55%     4.50mb  2.26%  time.gofunc
   4mb  2.01% 88.56%        4mb  2.01%  crypto/tls.client (inline)
   3.16mb  1.58% 90.14%     3.16mb  1.58%  main.main
   2.84mb  1.42% 91.56%     2.84mb  1.42%  time.starttimer
   2.50mb  1.25% 92.82%     2.50mb  1.25%  crypto/aes.(*aesciphergcm).newgcm
   2.50mb  1.25% 94.07%     2.50mb  1.25%  net.(*resolver).internetaddrlist.func1
   1.50mb  0.75% 94.82%     1.50mb  0.75%  crypto/tls.(*config).clone
   1.50mb  0.75% 95.58%     1.50mb  0.75%  crypto/aes.newcipher

package main

import (
    "crypto/tls"
    "net"
    "sync"
    "time"
)

func connecttotarget(targetstring string, dialer *net.dialer, config *tls.config) {
    tconn, err := tls.dialwithdialer(dialer,"tcp", targetstring, config)
    if err == nil {
        //do something with connection
        tconn.close()
    }
}

func main() {
    workers := 256 * 256 //65536
    tlsconfig := &tls.config{
        insecureskipverify: true,
    }
    dialer := &net.dialer{
        fallbackdelay: -1,
        keepalive:     -1,
        timeout:       time.duration(60) * time.second,
    }

    targetschan := make(chan string, workers)
    var workerdone sync.waitgroup
    workerdone.add(workers)

    for i := 0; i < workers; i++ {
        go func(functionwg *sync.waitgroup, dialer *net.dialer, tlsconfig *tls.config, targets chan string) {
            for targettoconnect := range targets {
                connecttotarget(targettoconnect, dialer, tlsconfig)
            }

            functionwg.done()
        }(&workerdone, dialer, tlsconfig,targetschan)
    }

    targets := []string{} //in the actual code this reads from a file containing the list since it is large
    for _,target := range targets {
        targetschan <- target
    }


    close(targetschan)
    workerdone.wait()
}

更新：

这是第一个 pprof（拍摄了 10 分钟）与我拍摄的最后一个 pprof 相比，这是在它稳定爬升了一段时间之后。

Showing nodes accounting for 329.76MB, 83.32% of 395.77MB total
Dropped 57 nodes (cum <= 1.98MB)
  flat  flat%   sum%        cum   cum%
  199.43MB 50.39% 50.39%   199.43MB 50.39%  bytes.makeSlice
  80.80MB 20.42% 70.81%   280.22MB 70.80%  crypto/tls.(*Conn).readHandshake
  28.02MB  7.08% 77.89%    28.02MB  7.08%  crypto/tls.Client (inline)
  18.01MB  4.55% 82.44%    18.01MB  4.55%  crypto/aes.(*aesCipherGCM).NewGCM
  11MB  2.78% 85.22%       11MB  2.78%  crypto/aes.newCipher
  9.50MB  2.40% 87.62%     9.50MB  2.40%  crypto/tls.(*Config).Clone
  -8MB  2.02% 85.60%    15.53MB  3.92%  crypto/tls.dial
  -5.50MB  1.39% 84.21%    -5.50MB  1.39%  net.(*netFD).connect
  -5MB  1.26% 82.94%    -5.50MB  1.39%  context.WithDeadline
  -4.50MB  1.14% 81.81%   -11.50MB  2.91%  net.(*Dialer).DialContext
  3.50MB  0.88% 82.69%     3.50MB  0.88%  net.sockaddrToTCP
  -3MB  0.76% 81.93%       -3MB  0.76%  time.AfterFunc
   2MB  0.51% 82.44%    17.53MB  4.43%  main.serverCert
   1.50MB  0.38% 82.82%        2MB  0.51%  crypto/tls.(*cipherSuiteTLS13).expandLabel
   1MB  0.25% 83.07%       18MB  4.55%  crypto/tls.aeadAESGCM
   1MB  0.25% 83.32%       10MB  2.53%  crypto/tls.aeadAESGCMTLS13
   0.50MB  0.13% 83.45%   201.93MB 51.02%  crypto/tls.(*Conn).readRecordOrCCS
   -0.50MB  0.13% 83.32%       -2MB  0.51%  net.(*sysDialer).dialSingle
   0     0% 83.32%   118.09MB 29.84%  bytes.(*Buffer).Grow (inline)
   0     0% 83.32%    81.33MB 20.55%  bytes.(*Buffer).Write
   0     0% 83.32%   199.43MB 50.39%  bytes.(*Buffer).grow
   0     0% 83.32%       11MB  2.78%  crypto/aes.NewCipher
   0     0% 83.32%    18.01MB  4.55%  crypto/cipher.NewGCM (inline)
   0     0% 83.32%    18.01MB  4.55%  crypto/cipher.newGCMWithNonceAndTagSize
   0     0% 83.32%   318.24MB 80.41%  crypto/tls.(*Conn).Handshake
   0     0% 83.32%   318.24MB 80.41%  crypto/tls.(*Conn).clientHandshake
   0     0% 83.32%     3.01MB  0.76%  crypto/tls.(*Conn).readChangeCipherSpec (inline)
   0     0% 83.32%   118.09MB 29.84%  crypto/tls.(*Conn).readFromUntil
   0     0% 83.32%   198.92MB 50.26%  crypto/tls.(*Conn).readRecord (inline)
   0     0% 83.32%    11.58MB  2.93%  crypto/tls.(*Conn).retryReadRecord
   0     0% 83.32%   154.61MB 39.06%  crypto/tls.(*clientHandshakeState).doFullHandshake
   0     0% 83.32%    22.51MB  5.69%  crypto/tls.(*clientHandshakeState).establishKeys
   0     0% 83.32%   180.12MB 45.51%  crypto/tls.(*clientHandshakeState).handshake
   0     0% 83.32%     3.01MB  0.76%  crypto/tls.(*clientHandshakeState).readFinished
   0     0% 83.32%       12MB  3.03%  crypto/tls.(*clientHandshakeStateTLS13).establishHandshakeKeys
   0     0% 83.32%   117.50MB 29.69%  crypto/tls.(*clientHandshakeStateTLS13).handshake
   0     0% 83.32%    92.92MB 23.48%  crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate
   0     0% 83.32%    11.58MB  2.93%  crypto/tls.(*clientHandshakeStateTLS13).readServerParameters
   0     0% 83.32%    10.50MB  2.65%  crypto/tls.(*halfConn).setTrafficSecret
   0     0% 83.32%    15.53MB  3.92%  crypto/tls.DialWithDialer (inline)
   0     0% 83.32%        3MB  0.76%  crypto/tls.cipherAES
   0     0% 83.32%   318.24MB 80.41%  crypto/tls.dial.func2
   0     0% 83.32%    17.53MB  4.43%  main.main.func3
   0     0% 83.32%       -2MB  0.51%  net.(*sysDialer).dialSerial
   0     0% 83.32%       -2MB  0.51%  net.internetSocket
   0     0% 83.32%       -2MB  0.51%  net.socket

这是相同数据的火焰图：

最大的罪犯是在握手读取期间调用的 bytes.makeslice。这可能意味着每次 goroutine 创建一个新的 tls.dialwithdialer 来连接到 url 时，缓冲区都会被保留。这会让我感到惊讶，因为我希望 clos​​e() 方法能够逐出这些缓冲区。

正确答案

事实证明，//do some with connection 中的代码比我想象的更重要。即使在 tls.Dial 级别，您也必须读取“正文”。我现在显然错误的假设是 tls.Dial 只是设置连接，并且由于 GET / HTTP 1.1 请求尚未发送，因此不需要从线路上读取数据。这导致所有充满服务器响应的缓冲区闲置。

_, _= ioutil.ReadAll(tConn) 将其全部修复为一行。我感觉自己聪明多了，同时也觉得自己很愚蠢。附带说明一下，在此级别，如果服务器响应缓慢，ReadAll() 可能会挂起很长时间。 tConn.SetReadDeadline(time.Now().Add(time.Second * timeout)) 也解决了这个问题。

文中关于的知识介绍，希望对你的学习有所帮助！若是受益匪浅，那就动动鼠标收藏这篇《使用多个并发的TLS拨号器时，Golang中的常驻内存持续增长》文章吧，也可关注golang学习网公众号了解相关技术文章。