local-search.xml

<?xml version="1.0" encoding="utf-8"?>
<search>
  
  
  
  <entry>
    <title></title>
    <link href="/2021/05/28/Go/debug/"/>
    <url>/2021/05/28/Go/debug/</url>
    
    <content type="html"><![CDATA[<p>一些常用的debug，性能测试工具记录</p><a id="more"></a><h2 id="deubg">deubg</h2><h3 id="build">build</h3><p>一般我们想生成汇编代码，无非就是:</p><div class="hljs"><pre><code class="hljs shell">go build -gcflags -S main.go</code></pre></div><p>或:</p><div class="hljs"><pre><code class="hljs shell">go tools compile -S main.go</code></pre></div><p>但是通过以下命令，可以生成更加详细的汇编优化过程:</p><div class="hljs"><pre><code class="hljs shell">GOSSAFUNC=main go build main.go</code></pre></div><h3 id="gdb">gdb</h3><p>列举用到的文件,其中entry_files的地址就是函数开始运行的地址</p><div class="hljs"><pre><code class="hljs shell">info files</code></pre></div><p>可以直接<code>b address</code>，然后运行<code>c</code></p><h3 id="dlv">dlv</h3><p>比较契合golang，</p><p><code>dlv debug</code>： 在当前目录寻找main函数，并开始调试</p><p>具体进入dlv后输入help查看命令:</p><p>常用:<code>print(p)</code>:打印local<code>b xxx.go:123</code> break某一行<code>c</code> :continue<code>n</code>:下一行<code>step</code>:下一步<code>restart</code>:重启</p><h2 id="performance">Performance</h2><h3 id="go-pprof">Go pprof</h3><p>有两个包</p><p><code>runtime/pprof</code><code>net/pprof</code>用于网络型，可以实时监控</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">go</span> tool pprof  xxx.pprof</code></pre></div><p>也可以展示http网页</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">go</span> tool pprof -http localhost:<span class="hljs-number">9190</span> xxx.pprof</code></pre></div><h3 id="go-test">Go test</h3><p>pprof还可以与go test结合</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">go</span> test -bench . -cpuprofile=cpu.prof<span class="hljs-keyword">go</span> test -bench . -memprofile=./mem.prof</code></pre></div>]]></content>
    
    
    
  </entry>
  
  
  
  <entry>
    <title>Something about Seckill</title>
    <link href="/2020/11/09/Comcon/seckillDesign/"/>
    <url>/2020/11/09/Comcon/seckillDesign/</url>
    
    <content type="html"><![CDATA[<h1>常说的秒杀系统</h1><a id="more"></a><p>无非在高并发下就是解决以下问题:</p><ol><li>超卖</li><li>少卖</li><li>人数过多，造成异常</li></ol><p>其实，第3点就是会导致1,2点的问题；</p><h2 id="整个流程">整个流程</h2><p>网关 - &gt; 多台redis集群(读写分离) -&gt; 消息队列 -&gt; 持久化</p><h2 id="超卖">超卖</h2>]]></content>
    
    
    
    <tags>
      
      <tag>Design</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Linux Cheat sheet</title>
    <link href="/2020/10/03/Comcon/linuxCommand/"/>
    <url>/2020/10/03/Comcon/linuxCommand/</url>
    
    <content type="html"><![CDATA[<ul><li>psFrom MANNUAL:</li></ul><p>EXAMPLES</p><div class="hljs"><pre><code class="hljs s">To see every process on the system using standard syntax:   ps -e   ps -ef   ps -eF   ps -elyTo see every process on the system using BSD syntax:   ps ax   ps axuTo print a process tree:   ps -ejH   ps axjfTo get info about threads:   ps -eLf   ps axmsTo get security info:   ps -eo euser,ruser,suser,fuser,f,comm,label   ps axZ   ps -eMTo see every process running as root (real &amp; effective ID) in user format:   ps -U root -u root uTo see every process with a user-defined format:   ps -eo pid,tid,class,rtprio,ni,pri,psr,pcpu,stat,wchan:14,comm   ps axo stat,euid,ruid,tty,tpgid,sess,pgrp,ppid,pid,pcpu,comm   ps -Ao pid,tt,user,fname,tmout,f,wchanPrint only the process IDs of syslogd:   ps -C syslogd -o pid=Print only the name of PID 42:   ps -q 42 -o comm=</code></pre></div><ul><li>转换文件编码格式:</li></ul><div class="hljs"><pre><code class="hljs shell">iconv -f [encoding] -t [encoding] sourcefile -o outputfile</code></pre></div><ul><li>strace</li></ul><p>一般用于<strong>系统调用</strong>的trace;</p><ul><li>tcpdump可以结合wireshark分析tcpdump</li></ul><ul><li>sysctl -a查看sys设置参数，包括ip设置，tcp设置等;相关设置在<code>/etc/sysctl.conf</code>中;</li></ul><p>在 <code>/proc/sys/net</code>也可以观察到;</p><ul><li><p>tcpdump实时抓取当前连接，同wireshark相似</p></li><li><p>tcp连接状态</p></li></ul><div class="hljs"><pre><code class="hljs s">netstat -na | awk &apos;/^tcp/ &#123;++S[$NF]&#125; END &#123;for(a in S) print a, S[a]&#125;&apos;</code></pre></div><ul><li>chmod改变权限权限组分为<code>拥有者</code>，<code>群组</code>，<code>其他用户</code>三种级别,分别为<code>RWS</code>，对应读、写、执行；</li></ul><p>组成的权限为二进制的bit array，如下所示:</p><div class="hljs"><pre><code class="hljs s">r-- = 100-w- = 010--x = 001--- = 000</code></pre></div><p>注意：对于文件夹来说，<code>s</code>执行,是其读取<code>r</code>(ls),<code>w</code>写入(在里面进行创建删除更改等操作)的前提，没有<code>s</code>权限，</p><ul><li><p>free相对与<code>top</code> 命令来说，free命令是准确展示内存和cache的，因为linux本身的机制，有些内存来不及回收，当做<code>cache</code>存留在内存中，所以top命令看到的内存大小是不一定正确的，可能有大部分cache占用了;</p></li><li><p>查找程序莫名被killed的原因</p></li></ul><ol><li><code>/var/log</code>下查找，每天的log中过滤killed process等字眼即可可以通过<code>egrep</code>(同grep -e，只不过匹配原则不一样):</li></ol><div class="hljs"><pre><code class="hljs s">egrep -i &apos;killed process&apos; /var/log/messages</code></pre></div><ol start="2"><li>或者<code>journalctl</code>:</li></ol><div class="hljs"><pre><code class="hljs s">journalctl -xb | egrep -i &apos;killed process&apos;</code></pre></div><div class="hljs"><pre><code class="hljs s">//journalctl -k//一段范围journalctl --since=&quot;2018-09-21 10:21:00&quot; --until=&quot;2018-09-21 10:22:00&quot;//日志占用空间journalctl --disk-usage</code></pre></div><ol start="3"><li>或者<code>dmesg</code>来查看开机信息:</li></ol><ul><li>time_wait过多</li></ul><p>即连接过多,且瞬时断开的较多，一般是短连接断开;那样就有对策：加大点连接总数（端口数）、增大连接时长、重用socket，及时回收或者上游确定是否要用短连接，可否改为长连接?</p><p><code>net.ipv4.tcp_syncookies = 1</code></p><p>表示开启SYN Cookies。当出现SYN等待队列溢出时，启用cookies来处理，可防范少量SYN攻击，默认为0，表示关闭；</p><p><code>net.ipv4.tcp_tw_reuse = 1</code></p><p>表示开启重用。允许将TIME-WAIT sockets重新用于新的TCP连接，默认为0，表示关闭；</p><p><code>net.ipv4.tcp_tw_recycle = 1</code></p><p>表示开启TCP连接中<code>TIME-WAIT</code> sockets的快速回收，默认为0，表示关闭。</p><p>系统tcp_timestamps缺省就是开启的，所以当<code>tcp_tw_recycle</code>被开启后，实际上这种行为就被激活了.如果服务器身处NAT环境，安全起见，通常要禁止<code>tcp_tw_recycle</code>，至于TIME_WAIT连接过多的问题，可以通过激活tcp_tw_reuse来缓解。</p><p><code>net.ipv4.tcp_max_tw_buckets = 5000</code></p><p>表示系统同时保持TIME_WAIT套接字的最大数量，如果超过这个数字，TIME_WAIT套接字将立刻被清除并打印警告信息。默认为180000，改为 5000。对于Apache、Nginx等服务器，上几行的参数可以很好地减少TIME_WAIT套接字数量，但是对于Squid，效果却不大。此项参数可以控制TIME_WAIT套接字的最大数量，避免Squid服务器被大量的TIME_WAIT套接字拖死。</p><p><code>net.ipv4.tcp_max_syn_backlog = 8192</code></p><p>表示SYN队列的长度，默认为1024，加大队列长度为8192，可以容纳更多等待连接的网络连接数。</p><p><code>net.ipv4.tcp_keepalive_time = 1200</code></p><p>表示当keepalive起用的时候，TCP发送keepalive消息的频度。缺省是2小时，改为20分钟。</p><p><code>net.ipv4.ip_local_port_range = 1024 65000</code></p><p>表示用于向外连接的端口范围。缺省情况下很小：32768到61000，改为1024到65000。</p>]]></content>
    
    
    
    <tags>
      
      <tag>Linux</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang Sync Package</title>
    <link href="/2020/10/02/Go/sync/"/>
    <url>/2020/10/02/Go/sync/</url>
    
    <content type="html"><![CDATA[<h1>Sync包的一些笔记</h1><a id="more"></a><h2 id="sync-once">Sync.Once</h2><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Once is an object that will perform exactly one action.</span><span class="hljs-keyword">type</span> Once <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// done indicates whether the action has been performed.</span><span class="hljs-comment">// It is first in the struct because it is used in the hot path.</span><span class="hljs-comment">// The hot path is inlined at every call site.</span><span class="hljs-comment">// Placing done first allows more compact instructions on some architectures (amd64/x86),</span><span class="hljs-comment">// and fewer instructions (to calculate offset) on other architectures.</span>done <span class="hljs-keyword">uint32</span>m    Mutex&#125;</code></pre></div><p><code>once.Do()</code>方法中注意下要进行一次原子操作和一次加锁</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Do calls the function f if and only if Do is being called for the</span><span class="hljs-comment">// first time for this instance of Once. In other words, given</span><span class="hljs-comment">// var once Once</span><span class="hljs-comment">// if once.Do(f) is called multiple times, only the first call will invoke f,</span><span class="hljs-comment">// even if f has a different value in each invocation. A new instance of</span><span class="hljs-comment">// Once is required for each function to execute.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Do is intended for initialization that must be run exactly once. Since f</span><span class="hljs-comment">// is niladic, it may be necessary to use a function literal to capture the</span><span class="hljs-comment">// arguments to a function to be invoked by Do:</span><span class="hljs-comment">// config.once.Do(func() &#123; config.init(filename) &#125;)</span><span class="hljs-comment">//</span><span class="hljs-comment">// Because no call to Do returns until the one call to f returns, if f causes</span><span class="hljs-comment">// Do to be called, it will deadlock.</span><span class="hljs-comment">//</span><span class="hljs-comment">// If f panics, Do considers it to have returned; future calls of Do return</span><span class="hljs-comment">// without calling f.</span><span class="hljs-comment">//</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(o *Once)</span> <span class="hljs-title">Do</span><span class="hljs-params">(f <span class="hljs-keyword">func</span>()</span>)</span> &#123;<span class="hljs-comment">// Note: Here is an incorrect implementation of Do:</span><span class="hljs-comment">//</span><span class="hljs-comment">//if atomic.CompareAndSwapUint32(&amp;o.done, 0, 1) &#123;</span><span class="hljs-comment">//f()</span><span class="hljs-comment">//&#125;</span>    <span class="hljs-comment">//</span>    <span class="hljs-comment">//注意，上面是不正确的，因为要保证，atomic.StoreUint32在f返回后才能调用;</span>    <span class="hljs-comment">//两个goroutine同时进来call f，快的一个会直接call f，这个时候慢的那一个不会等待快的f完成，而是直接返回</span><span class="hljs-comment">// Do guarantees that when it returns, f has finished.</span><span class="hljs-comment">// This implementation would not implement that guarantee:</span><span class="hljs-comment">// given two simultaneous calls, the winner of the cas would</span><span class="hljs-comment">// call f, and the second would return immediately, without</span><span class="hljs-comment">// waiting for the first's call to f to complete.</span><span class="hljs-comment">// This is why the slow path falls back to a mutex, and why</span><span class="hljs-comment">// the atomic.StoreUint32 must be delayed until after f returns.</span><span class="hljs-keyword">if</span> atomic.LoadUint32(&amp;o.done) == <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// Outlined slow-path to allow inlining of the fast-path.</span>o.doSlow(f)&#125;&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(o *Once)</span> <span class="hljs-title">doSlow</span><span class="hljs-params">(f <span class="hljs-keyword">func</span>()</span>)</span> &#123;o.m.Lock()    <span class="hljs-keyword">defer</span> o.m.Unlock()     <span class="hljs-comment">//这里需要再次重新判断下，因为atomic.LoadUint32取出状态值到o.m.Lock之间是有可能存在其它gotoutine改变status的状态值的</span><span class="hljs-keyword">if</span> o.done == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">defer</span> atomic.StoreUint32(&amp;o.done, <span class="hljs-number">1</span>)f()&#125;&#125;</code></pre></div><h2 id="sync-waitgroup">sync.WaitGroup</h2><p>首先看下其结构：旧的结构</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> WaitGroup <span class="hljs-keyword">struct</span> &#123;noCopy noCopy state1 [<span class="hljs-number">12</span>]<span class="hljs-keyword">byte</span><span class="hljs-comment">//对齐有4个byte要浪费空间</span>sema   <span class="hljs-keyword">uint32</span>&#125;</code></pre></div><p>现在的结构</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// A WaitGroup waits for a collection of goroutines to finish.</span><span class="hljs-comment">// The main goroutine calls Add to set the number of</span><span class="hljs-comment">// goroutines to wait for. Then each of the goroutines</span><span class="hljs-comment">// runs and calls Done when finished. At the same time,</span><span class="hljs-comment">// Wait can be used to block until all goroutines have finished.</span><span class="hljs-comment">//</span><span class="hljs-comment">// A WaitGroup must not be copied after first use.</span><span class="hljs-keyword">type</span> WaitGroup <span class="hljs-keyword">struct</span> &#123;noCopy noCopy<span class="hljs-comment">// 64-bit value: high 32 bits are counter, low 32 bits are waiter count.</span><span class="hljs-comment">// 64-bit atomic operations require 64-bit alignment, but 32-bit</span><span class="hljs-comment">// compilers do not ensure it. So we allocate 12 bytes and then use</span><span class="hljs-comment">// the aligned 8 bytes in them as state, and the other 4 as storage</span><span class="hljs-comment">// for the sema.</span><span class="hljs-comment">//64位: (goroutine计数器)(等待计数器)(信号量)</span><span class="hljs-comment">//不是64位的组成: (信号量)(goroutine计数器)(等待计数器)</span>state1 [<span class="hljs-number">3</span>]<span class="hljs-keyword">uint32</span>&#125;</code></pre></div><ul><li><p>带有nocopy，即使用时只可以传指针copy的隐藏含义是，这个结构体在传参的时候所有字段都会被复制，那么试想一下多线程下，同时对<code>state1</code>进行修改，明显不能保证线程安全;</p></li><li><p>state1是一个3个uint32元素的数组，高32位为计数器（当前仍未执行结束的goroutine数目），第二个32位为等待goroutine完成的数目（有多少个等待者）,第三个可以理解为信号量semaphore， 至于为什么是3个元素，这个跟内存对齐有关(以前的结构不一样会浪费4个byte)，如下:</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// state returns pointers to the state and sema fields stored within wg.state1.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(wg *WaitGroup)</span> <span class="hljs-title">state</span><span class="hljs-params">()</span> <span class="hljs-params">(statep *<span class="hljs-keyword">uint64</span>, semap *<span class="hljs-keyword">uint32</span>)</span></span> &#123;<span class="hljs-comment">//判断是否8位对齐</span><span class="hljs-keyword">if</span> <span class="hljs-keyword">uintptr</span>(unsafe.Pointer(&amp;wg.state1))%<span class="hljs-number">8</span> == <span class="hljs-number">0</span> &#123;<span class="hljs-comment">//前面8个bytes做uint64指针statep，后面4bytes做sema</span><span class="hljs-keyword">return</span> (*<span class="hljs-keyword">uint64</span>)(unsafe.Pointer(&amp;wg.state1)), &amp;wg.state1[<span class="hljs-number">2</span>]&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//不是64位的组成: (信号量)(goroutine计数器)(等待计数器)</span><span class="hljs-keyword">return</span> (*<span class="hljs-keyword">uint64</span>)(unsafe.Pointer(&amp;wg.state1[<span class="hljs-number">1</span>])), &amp;wg.state1[<span class="hljs-number">0</span>]&#125;&#125;</code></pre></div><p><code>Add(delta)</code>方法: 将delta加到<code>[0]state1</code> 中<code>Wait()</code>,[1]state1(waiter counter) 减一;<code>Done()</code>，[0]state1(counter)减一;</p><h2 id="sync-mutex">Sync.Mutex</h2><p>互斥锁, 分为正常模式和饥饿模式结构体，注意也是<code>nocopy</code>的:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// A Mutex is a mutual exclusion lock.</span><span class="hljs-comment">// The zero value for a Mutex is an unlocked mutex.</span><span class="hljs-comment">//</span><span class="hljs-comment">// A Mutex must not be copied after first use.</span><span class="hljs-keyword">type</span> Mutex <span class="hljs-keyword">struct</span> &#123;state <span class="hljs-keyword">int32</span>sema  <span class="hljs-keyword">uint32</span>&#125;</code></pre></div><p><code>sema</code>就是Semaphore(信号量，mutex实际就是一种特殊的信号量)，平常用的<code>runtime.Semacquire</code>、<code>runtime.SemaRelease</code>等函数使用的用作同步的原语；</p><p>其中<code>state</code>有几个状态:</p><ul><li><code>mutextLocked</code></li><li><code>mutexWoken</code></li><li><code>mutexStarving</code></li><li><code>mutexWaiterShift</code></li></ul><p><code>starvationThresholdNs</code>:饥饿状态下,为了保证公平性，占用的过期时间</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">const</span> (mutexLocked = <span class="hljs-number">1</span> &lt;&lt; <span class="hljs-literal">iota</span> <span class="hljs-comment">// mutex is locked</span>mutexWokenmutexStarvingmutexWaiterShift = <span class="hljs-literal">iota</span><span class="hljs-comment">// Mutex fairness.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Mutex can be in 2 modes of operations: normal and starvation.</span><span class="hljs-comment">// In normal mode waiters are queued in FIFO order, but a woken up waiter</span><span class="hljs-comment">// does not own the mutex and competes with new arriving goroutines over</span><span class="hljs-comment">// the ownership. New arriving goroutines have an advantage -- they are</span><span class="hljs-comment">// already running on CPU and there can be lots of them, so a woken up</span><span class="hljs-comment">// waiter has good chances of losing. In such case it is queued at front</span><span class="hljs-comment">// of the wait queue. If a waiter fails to acquire the mutex for more than 1ms,</span><span class="hljs-comment">// it switches mutex to the starvation mode.</span><span class="hljs-comment">//</span><span class="hljs-comment">// In starvation mode ownership of the mutex is directly handed off from</span><span class="hljs-comment">// the unlocking goroutine to the waiter at the front of the queue.</span><span class="hljs-comment">// New arriving goroutines don't try to acquire the mutex even if it appears</span><span class="hljs-comment">// to be unlocked, and don't try to spin. Instead they queue themselves at</span><span class="hljs-comment">// the tail of the wait queue.</span><span class="hljs-comment">//</span><span class="hljs-comment">// If a waiter receives ownership of the mutex and sees that either</span><span class="hljs-comment">// (1) it is the last waiter in the queue, or (2) it waited for less than 1 ms,</span><span class="hljs-comment">// it switches mutex back to normal operation mode.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Normal mode has considerably better performance as a goroutine can acquire</span><span class="hljs-comment">// a mutex several times in a row even if there are blocked waiters.</span><span class="hljs-comment">// Starvation mode is important to prevent pathological cases of tail latency.</span>starvationThresholdNs = <span class="hljs-number">1e6</span>)</code></pre></div><p>其实上面的注释写的比较明白:</p><ul><li><p><code>正常模式</code>: <code>mutex</code>的等待者都以FIFO顺序在排队，前一个协程执行结束解锁时，会唤醒等待队列中的一个协程，唤醒的协程和刚来的协程（<strong>还没进入队列</strong>）竞争锁的所有权,因为刚进来的协程当前正在持有着cpu，资源也不需要重新调度，对比之下刚唤醒的协程大概率竞争不过新来的协程，所以当唤醒的协程沉睡超过<code>1ms</code>，会将锁置为饥饿模式;</p></li><li><p><code>饥饿模式</code>: 解锁的协程将锁的所有权移交给等待队列中的<strong>队首</strong>协程，新到来的协程<strong>不会</strong>尝试去获取锁的控制权（即时当前是可获取状态），也<strong>不会</strong>尝试去自旋，会直接加入到队尾等待被唤醒;</p></li></ul><p><code>饥饿模式</code>还会解除:</p><ul><li>当前获取<code>mutex</code>所有权的协程是阻塞队列的最后一个协程;</li><li>或者该协程等待时间小于<code>1ms</code>，则将饥饿模式转换成正常模式;</li></ul>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Interview</title>
    <link href="/2020/09/30/Comcon/interviewBackend/"/>
    <url>/2020/09/30/Comcon/interviewBackend/</url>
    
    <content type="html"><![CDATA[<h1>面试准备(后端)</h1><h2 id="语言go">语言go</h2><h3 id="1-内存管理">1. 内存管理</h3><h4 id="总体结构">总体结构</h4><ul><li><p>tcmalloc</p></li><li><p>mspan</p></li><li><p>mcache</p></li><li><p>mheap</p></li></ul><h4 id="sync-pool">sync.Pool</h4><ul><li><p>环状列表</p></li><li><p>victimCache (1.13)</p></li></ul><h3 id="2-同步">2. 同步</h3><h4 id="关键字及其用法原理">关键字及其用法原理</h4><h5 id="channel">channel</h5><h5 id="select">select</h5><h5 id="sync-waitgroup">sync.WaitGroup</h5><h4 id="思想">思想</h4><ul><li>名句</li></ul><h3 id="3-垃圾回收">3. 垃圾回收</h3><h4 id="三色波面推进-标记整理">三色波面推进(标记整理)</h4><h5 id="具体相关字段">具体相关字段</h5><h5 id="流程">流程</h5><h4 id="比较其他垃圾回收器">比较其他垃圾回收器</h4><ul><li>引用计数</li></ul><ul><li><p>标记回收</p></li><li><p>标记回收整理</p></li><li><p>同步时回收</p></li></ul><h3 id="4-reflection">4. reflection</h3><h4 id="iface-eface">iface &amp; eface</h4><h3 id="5-继承-组合">5. 继承(组合)</h3><h4 id="一些常考的cases">一些常考的cases</h4><p>string(), error(), read(), write()</p><h3 id="6-net包">6. net包</h3><h4 id="epoll">epoll</h4><h4 id="优秀的连接池包">优秀的连接池包</h4><ul><li>fasthttp</li></ul><h2 id="计算机网络">计算机网络</h2><p>总体七层</p><h3 id="应用层常用">应用层常用</h3><p>HTTP 80端口</p><h5 id="头-幂等设计">头，幂等设计</h5><h5 id="http1-1-http2-https">HTTP1.1, HTTP2 &amp; HTTPS</h5><h5 id="安全">安全</h5><ul><li>JWT</li><li>HTTPS</li><li>session</li></ul><h3 id="传输层">传输层</h3><h5 id="总体">总体</h5><h5 id="tcp">tcp</h5><ul><li><p>结构40 Bytes五元组&lt;source ip, dest ip&gt;标志位(RST,ACK,SYN,FIN)</p></li><li><p>面向连接，可靠</p><ol><li>选择重传(快重传)</li><li>三次握手，四次挥手</li><li>linux下常用默认设置</li></ol></li><li><p>安全</p><ol><li>校验码,纠错</li></ol></li></ul><h5 id="udp">udp</h5><ul><li><p>结构</p><p>五元组</p></li><li><p>不可靠不用握手</p></li><li><p>安全</p><ol><li>crc只可以检测，不可以纠错</li></ol></li></ul><h3 id="网络层">网络层</h3><h5 id="ip协议">IP协议</h5><p>MMU(max ) 1500 Bytes</p><h5 id="dns">DNS</h5><ul><li>43端口</li><li>权威，顶级，根</li><li>递归，迭代</li></ul><h5 id="icmp">ICMP</h5><p>trace route, ping</p><h5 id="arp">ARP</h5><h5 id="dhcp">DHCP</h5><h3 id="链路层">链路层</h3><h5 id="mac地址">MAC地址</h5><h5 id="bgp">BGP &amp;</h5><ul><li>域间路由,域内路由</li></ul><ul><li>链路毒逆转</li></ul><h3 id="物理层">物理层</h3><p>略</p><ul><li>几种光纤</li></ul><h2 id="数据库">数据库</h2><h3 id="持久型">持久型</h3><p>MySQL, 5.7</p><h4 id="单机事务">单机事务</h4><ul><li><p>ACID</p></li><li><p>隔离性四个级别</p><ol><li>串行</li><li>repeatable Read, 保证写，可能有幻读(范围内的读取读取不到插入或者删除),一般是快照实现(MVVC)</li><li>unrepeatable Read，</li></ol></li></ul><h4 id="基本常用语法">基本常用语法</h4><ul><li>explain</li><li></li></ul><h4 id="indexing">indexing</h4><ul><li><p>聚簇索引</p></li><li><p>非聚簇索引</p></li></ul><h4 id="源码生成">源码生成</h4><h3 id="缓存内存型">缓存内存型</h3><p>Redis</p><h4 id="概括">概括</h4><ol><li><p>内存跑</p></li><li><p>单线程(网络部分), IO多路复用(epoll)</p></li><li><p>优秀的数据结构</p></li></ol><h4 id="缓存的几种应对">缓存的几种应对</h4><ul><li><p>读多写时，先删除，再写读时，直接读</p></li><li><p>写多写后再更新</p></li></ul><h4 id="基本结构-对外api">基本结构(对外API)</h4><p>string</p><ul><li>ziplist</li></ul><p>list</p><ul><li>lpop, rpop,lpush, rpushllen</li><li></li></ul><p>dict</p><ul><li><p>ziplist</p></li><li><p>hashmap</p></li></ul><p>set</p><ul><li>dict</li></ul><p>zset</p><ul><li>quicklist+ dict</li></ul><h4 id="aof-rdb">aof &amp; rdb</h4><p>暂时略</p><h4 id="replicated-sentinel-clusters">replicated &amp; sentinel &amp; clusters</h4><ul><li><p>offset复制</p></li><li><p>raft log</p></li></ul><h3 id="操作系统-linux">操作系统(Linux)</h3><h4 id="编码">编码</h4><p>大顶端，小顶端ASCII</p><h4 id="进程-线程">进程，线程</h4><ul><li><ol><li>分配资源的基本单位</li><li>task_struct{}</li></ol></li><li><p>调度的基本单位</p></li></ul><h4 id="调度">调度</h4><ul><li><p>时间片轮转</p></li><li><p>优先级(nice 值 -20 ~ 19)</p></li><li><p>抢断式</p></li></ul><h4 id="软中断和下半部">软中断和下半部</h4><h4 id="内存分配">内存分配</h4><h4 id="文件">文件</h4><ul><li>一切皆文件,都有一个fd</li></ul><h4 id="硬盘">硬盘</h4><ul><li>电梯法</li></ul><h3 id="计算机组成原理">计算机组成原理</h3><h4 id="补码-原码-反码">补码，原码，反码</h4><p>正数,补码=原码,反码符号位除外全部取反</p><h4 id="ieee754">IEEE754</h4><p>ps: 0.1+0.2!=0.3 问题</p><p>0.1无法用2进制准确表达</p><h4 id="总线bus">总线BUS</h4><h4 id="go中常用汇编-plan9">Go中常用汇编(plan9)</h4><ul><li>SP stack pointerPC</li></ul><p>常见命令:ADDSUB</p><p>MOVSYMBOL+OFFSET(SP),伪SPOFFSET(SP),真SP</p><p>JMP</p><h3 id="分布式">分布式</h3><p>CAP分区容错一致性可用性</p><h4 id="负载均衡">负载均衡</h4><ol><li>负载均衡算法</li><li>健康检查(从IP层到HTTP层)</li><li>session的保持方式</li></ol><h4 id="事务">事务</h4><p>2PC -&gt; 3PC</p><h4 id="共识算法">共识算法</h4><p>Paxos -&gt; MultiPaxos -&gt; ZAB -&gt; Raft</p><h2 id="project">Project</h2><ol><li><p>日志的输出kafka,处理; 包括失败下线，成功率，设计任务调度队列</p></li><li><p>缓存的任务调度队列，针对不同级别的任务队列,进程池的的大小来区分优先级;</p></li></ol><h2 id="项目">项目</h2><ol><li>job调度，使用工作池来限制其权重速率大小</li><li>限流，redis hmget，存储</li></ol><ol start="3"><li>kafka，排除消息堆积:提交到broker的偏移量处开始消费。我在网上查阅了消息堆积和消息重复的一些原因，发现问题可能出现在kafka的poll()设置上。查阅kafka官网发现我用的那个版本的kafka主要有以下几个比较关键的指标:a. max.poll.records一次poll返回的最大记录数默认是500b. max.poll.interval.ms两次poll方法最大时间间隔这个参数，默认是300s这次问题出现的原因为由于业务上下方的消息增量变多，导致堆积的消息过多，每一批poll()的处理都能达到500条消息，导致poll之后消费的时间过长。<a href="http://xn--max-0h9ds1qh8cfvr5b940gtml0j7cfa174j.poll.interval.ms" target="_blank" rel="noopener">服务端约定了和客户端max.poll.interval.ms</a>，两次poll最大间隔。如果客户端处理一批消息花费的时间超过了这个限制时间，broker可能就会把消费者客户端移除掉，提交偏移量又会报错。所以拉取偏移量没有提交到broker，分区又rebalance，下一次重新分配分区时，消费者会从最新的已提交偏移量处开始消费，这里就出现了重复消费的问题。而服务注册中心zookeeper以为客户端失效进行rebalance，因此连接到另外一台消费服务器，然而另外一台服务器也出现poll()超时，又进行rebalance…如此循环，才出现了一直重发消息，导致消息数量被消费后下降很慢。</li></ol><p>rebalance:</p><p>consumer订阅topic中的一个或者多个partition中的消息，一个consumer group下可以有多个consumer，一条消息只能被group中的一个consumer消费。consumer和consumer group的关系是动态维护的，并不固定，当某个consumer卡住或者挂掉时，该consumer订阅的partition会被重新分配给该group下其它consumer，用于保证服务的可用性。为维护consumer和group之间的关系，consumer会定期向服务端的coordinator(一个负责维持客户端与服务端关系的协调者)发送心跳heartbeat，当consumer因为某种原因如死机无法在session.timeout.ms配置的时间间隔内发送heartbeat时，coordinator会认为该consumer已死，它所订阅的partition会被重新分配给同一group的其它consumer，该过程叫：rebalanced。</p><p>方案:</p><p>使用Kafka时，<a href="http://xn--pollkafkamax-dy4svf980ep8x3o4blth4qcz94bnidgzmr22blrmna1534d0qfe99gq2bp53ahf9b.poll.interval.ms" target="_blank" rel="noopener">消费者每次poll的数据业务处理时间不能超过kafka的max.poll.interval.ms</a>，可以考虑调大超时时间或者调小每次poll的数据量。增加max.poll.interval.ms处理时长(默认间隔300s)</p><ul><li>max.poll.interval.ms=300</li></ul><p>修改分区拉取阈值(默认50s,建议压测评估调小)</p><p>max.poll.records = 50</p><ul><li>可以考虑增强消费者的消费能力，使用线程池消费或者将消费者中耗时业务改成异步，并保证对消息是幂等处理</li><li>不但要有消息积压的监控，还可以考虑做消息消费速度的监控（前后两次offset比较）</li></ul><ol start="4"><li>redis, 缓存的一致性，write aside，write through</li></ol><ol start="5"><li>failbackup,用资源换延迟，发送多个请求，有一个返回就用那个</li></ol><h1>项目</h1><p>库存管理数据库升级,MySQL -&gt; Mongo 聚合型，业务要求，库存和价格混合覆盖,省钱;</p><p>线程池实现job的更新队列速度，改进，top类别使用第三方队列，可以考虑全部用第三方队列，宕机之后方便恢复;</p><p>促销：秒杀，方案从读缓存，不存在入Mongo(write through,read through)到直接在Redis上进行增加减少(一段时间或大促过后更新到持久层)促销定时器,(redis zset-&gt;时间轮);</p><p>过滤平台请求：kafka入队信息，定时器计数;</p><p>运营系统，对账(减内存占用，保证准确性);</p>]]></content>
    
    
    
    <tags>
      
      <tag>Interview</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Defer</title>
    <link href="/2020/09/02/Go/defer/"/>
    <url>/2020/09/02/Go/defer/</url>
    
    <content type="html"><![CDATA[<p>defer的一些东西</p><a id="more"></a><h2 id="看看个坑">看看个坑</h2><p>我们先来看一个例子:</p><div class="hljs"><pre><code class="hljs go">func() &#123;  var run func() = nil  defer run()  fmt.Println("runs")&#125;</code></pre></div><p>结果是:</p><div class="hljs"><pre><code class="hljs undefined">panic: runtime error: invalid memory<span class="hljs-built_in"> address </span><span class="hljs-keyword">or</span> <span class="hljs-literal">nil</span> pointer dereference</code></pre></div><h2 id="详情">详情</h2><p>在<code>g</code>和<code>p</code>上都有相关结构,其中<code>p</code>上的为一个<code>defer pool</code>//todo</p><h3 id="一些相关结构">一些相关结构</h3><p><code>_defer</code>的结构：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// A _defer holds an entry on the list of deferred calls.</span><span class="hljs-comment">// If you add a field here, add code to clear it in freedefer and deferProcStack</span><span class="hljs-comment">// This struct must match the code in cmd/compile/internal/gc/reflect.go:deferstruct</span><span class="hljs-comment">// and cmd/compile/internal/gc/ssa.go:(*state).call.</span><span class="hljs-comment">// Some defers will be allocated on the stack and some on the heap.</span><span class="hljs-comment">// All defers are logically part of the stack, so write barriers to</span><span class="hljs-comment">// initialize them are not required. All defers must be manually scanned,</span><span class="hljs-comment">// and for heap defers, marked.</span><span class="hljs-keyword">type</span> _defer <span class="hljs-keyword">struct</span> &#123;    <span class="hljs-comment">//参数和return的大小</span>siz     <span class="hljs-keyword">int32</span> <span class="hljs-comment">// includes both arguments and results</span>started <span class="hljs-keyword">bool</span><span class="hljs-comment">//标记开始</span>heap    <span class="hljs-keyword">bool</span><span class="hljs-comment">// openDefer indicates that this _defer is for a frame with open-coded</span><span class="hljs-comment">// defers. We have only one defer record for the entire frame (which may</span><span class="hljs-comment">// currently have 0, 1, or more defers active).</span><span class="hljs-comment">//这里的openCoded是在编译时的一些优化，在编译时会直接将defer的方法插入到运行中函数尾部,避免deferproc和deferprocStack操作</span>    openDefer <span class="hljs-keyword">bool</span>    <span class="hljs-comment">//栈的指针</span>sp        <span class="hljs-keyword">uintptr</span>  <span class="hljs-comment">// sp at time of defer</span>    <span class="hljs-comment">//调用方的程序计数器</span>    pc        <span class="hljs-keyword">uintptr</span>  <span class="hljs-comment">// pc at time of defer</span>    <span class="hljs-comment">//defer传入的函数</span>    fn        *funcval <span class="hljs-comment">// can be nil for open-coded defers</span>    <span class="hljs-comment">//触发延迟调用的结构体，可能为空</span>    _panic    *_panic  <span class="hljs-comment">// panic that is running defer，每次添加到链表头部</span>link      *_defer <span class="hljs-comment">//每次添加到链表头部</span><span class="hljs-comment">// If openDefer is true, the fields below record values about the stack</span><span class="hljs-comment">// frame and associated function that has the open-coded defer(s). sp</span><span class="hljs-comment">// above will be the sp for the frame, and pc will be address of the</span><span class="hljs-comment">// deferreturn call in the function.</span>fd   unsafe.Pointer <span class="hljs-comment">// funcdata for the function associated with the frame</span>varp <span class="hljs-keyword">uintptr</span>        <span class="hljs-comment">// value of varp for the stack frame</span><span class="hljs-comment">// framepc is the current pc associated with the stack frame. Together,</span><span class="hljs-comment">// with sp above (which is the sp associated with the stack frame),</span><span class="hljs-comment">// framepc/sp can be used as pc/sp pair to continue a stack trace via</span><span class="hljs-comment">// gentraceback().</span>framepc <span class="hljs-keyword">uintptr</span>&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//只允许在stack上</span><span class="hljs-comment">//go:notinheap</span><span class="hljs-keyword">type</span> _panic <span class="hljs-keyword">struct</span> &#123;argp      unsafe.Pointer <span class="hljs-comment">// pointer to arguments of deferred call run during panic; cannot move - known to liblink</span>arg       <span class="hljs-keyword">interface</span>&#123;&#125;    <span class="hljs-comment">// argument to panic</span>link      *_panic        <span class="hljs-comment">// link to earlier panic</span><span class="hljs-comment">//标记return的位置</span>pc        <span class="hljs-keyword">uintptr</span>        <span class="hljs-comment">// where to return to in runtime if this panic is bypassed</span>sp        unsafe.Pointer <span class="hljs-comment">// where to return to in runtime if this panic is bypassed</span>recovered <span class="hljs-keyword">bool</span>           <span class="hljs-comment">// whether this panic is over，标记是否已经调用recover</span>aborted   <span class="hljs-keyword">bool</span>           <span class="hljs-comment">// the panic was aborted</span>goexit    <span class="hljs-keyword">bool</span>&#125;</code></pre></div><p><code>defer</code>在整个程序时实现主要分成编译期和运行时有不同的动作：</p><h3 id="编译期-有三种不同的编译方式-1-14新增一种">编译期(有三种不同的编译方式,1.14新增一种)</h3><ol><li><code>defer</code> 关键字<strong>转换成</strong>了<code>deferproc</code>,堆上分配 ;</li><li>还会在所有调用defer的函数末尾插入<code>deferreturn</code>，栈上分配(1.13新增), ssa会预留defer空间(1.13，在函数体内最多执行一次就会调用<code>cmd/compile/internal/gc.state.call</code>将结构体分配到栈并调用<code>runtime.deferprocStack</code>);</li><li><code>open-coded</code>(1.14新增),只会在以下情况:<ul><li>函数的 <code>defer</code> 数量少于或者等于 8 个；</li><li>函数的 <code>defer</code> 关键字不能在循环中执行(包括goto)；</li><li>函数的 <code>return</code> 语句与 <code>defer</code> 语句的乘积小于或者等于 15 个；编译时会根据以上条件判断是否开启;</li></ul></li></ol><h4 id="转换defer关键字">转换defer关键字</h4><p><code>cmd/compile/internal/gc.state.stmt</code>会处理<code>defer</code>关键字</p><p><code>cmd/compile/internal/gc.state.call</code> 该函数负责所有函数和方法调用生成中间代码:</p><ol><li>获取需要执行的函数名、闭包指针、代码指针和函数调用的接收方；</li><li>获取栈地址并将函数或者方法的参数写入栈中；</li><li>使用 <code>cmd/compile/internal/gc.state.newValue1A</code> 以及相关函数生成函数调用的中间代码；</li><li>如果当前调用的函数是 defer，那么就会单独生成相关的结束代码块；</li><li>获取函数的返回值地址并结束当前调用；</li></ol><p>针对<code>open-coded</code>，主要就是在某些情况下使用,如下，默认超过 <code>maxOpenDefers = 8</code>就不能使用Open-coded</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">walkstmt</span><span class="hljs-params">(n *Node)</span> *<span class="hljs-title">Node</span></span> &#123;...<span class="hljs-keyword">case</span> ODEFER:Curfn.Func.SetHasDefer(<span class="hljs-literal">true</span>)Curfn.Func.numDefers++<span class="hljs-keyword">if</span> Curfn.Func.numDefers &gt; maxOpenDefers &#123;<span class="hljs-comment">// Don't allow open-coded defers if there are more than</span><span class="hljs-comment">// 8 defers in the function, since we use a single</span><span class="hljs-comment">// byte to record active defers.</span>Curfn.Func.SetOpenCodedDeferDisallowed(<span class="hljs-literal">true</span>)&#125;<span class="hljs-keyword">if</span> n.Esc != EscNever &#123;<span class="hljs-comment">// If n.Esc is not EscNever, then this defer occurs in a loop,</span><span class="hljs-comment">// so open-coded defers cannot be used in this function.</span>Curfn.Func.SetOpenCodedDeferDisallowed(<span class="hljs-literal">true</span>)&#125;<span class="hljs-keyword">fallthrough</span>...&#125;</code></pre></div><h4 id="栈上分配-消耗更少">栈上分配(消耗更少)</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// deferprocStack queues a new deferred function with a defer record on the stack.</span><span class="hljs-comment">// The defer record must have its siz and fn fields initialized.</span><span class="hljs-comment">// All other fields can contain junk.</span><span class="hljs-comment">// The defer record must be immediately followed in memory by</span><span class="hljs-comment">// the arguments of the defer.</span><span class="hljs-comment">// Nosplit because the arguments on the stack won't be scanned</span><span class="hljs-comment">// until the defer record is spliced into the gp._defer list.</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">deferprocStack</span><span class="hljs-params">(d *_defer)</span></span> &#123;gp := getg()<span class="hljs-keyword">if</span> gp.m.curg != gp &#123;<span class="hljs-comment">// go code on the system stack can't defer</span>throw(<span class="hljs-string">"defer on system stack"</span>)&#125;<span class="hljs-comment">// siz and fn are already set.</span><span class="hljs-comment">// The other fields are junk on entry to deferprocStack and</span><span class="hljs-comment">// are initialized here.</span>d.started = <span class="hljs-literal">false</span>d.heap = <span class="hljs-literal">false</span>d.openDefer = <span class="hljs-literal">false</span>d.sp = getcallersp()d.pc = getcallerpc()d.framepc = <span class="hljs-number">0</span>d.varp = <span class="hljs-number">0</span><span class="hljs-comment">// The lines below implement:</span><span class="hljs-comment">//   d.panic = nil</span><span class="hljs-comment">//   d.fd = nil</span><span class="hljs-comment">//   d.link = gp._defer</span><span class="hljs-comment">//   gp._defer = d</span><span class="hljs-comment">// But without write barriers. The first three are writes to</span><span class="hljs-comment">// the stack so they don't need a write barrier, and furthermore</span><span class="hljs-comment">// are to uninitialized memory, so they must not use a write barrier.</span><span class="hljs-comment">// The fourth write does not require a write barrier because we</span><span class="hljs-comment">// explicitly mark all the defer structures, so we don't need to</span><span class="hljs-comment">// keep track of pointers to them with a write barrier.</span>*(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(&amp;d._panic)) = <span class="hljs-number">0</span>*(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(&amp;d.fd)) = <span class="hljs-number">0</span>*(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(&amp;d.link)) = <span class="hljs-keyword">uintptr</span>(unsafe.Pointer(gp._defer))*(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(&amp;gp._defer)) = <span class="hljs-keyword">uintptr</span>(unsafe.Pointer(d))return0()<span class="hljs-comment">// No code can go here - the C return register has</span><span class="hljs-comment">// been set and must not be clobbered.</span>&#125;</code></pre></div><h3 id="运行时">运行时</h3><ol><li><code>deferproc</code>会将一个新的_defer结构体追加到当前goroutine的链表头(分配一个<code>_defer</code>对象并加入延迟参数)</li><li><code>deferreturn</code>会从goroutine的链表(已经在函数调用栈中)取出<code>_defer</code>并执行</li></ol><h4 id="创建延迟调用-堆上">创建延迟调用(堆上)</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Create a new deferred function fn with siz bytes of arguments.</span><span class="hljs-comment">// The compiler turns a defer statement into a call to this.</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">deferproc</span><span class="hljs-params">(siz <span class="hljs-keyword">int32</span>, fn *funcval)</span></span> &#123; <span class="hljs-comment">// arguments of fn follow fn</span>gp := getg()<span class="hljs-keyword">if</span> gp.m.curg != gp &#123;<span class="hljs-comment">// go code on the system stack can't defer</span>throw(<span class="hljs-string">"defer on system stack"</span>)    &#125;    <span class="hljs-comment">//-------------------1. 创建一个_defer延迟调用</span>    <span class="hljs-comment">// the arguments of fn are in a perilous state. The stack map</span><span class="hljs-comment">// for deferproc does not describe them. So we can't let garbage</span><span class="hljs-comment">// collection or stack copying trigger until we've copied them out</span><span class="hljs-comment">// to somewhere safe. The memmove below does that.</span><span class="hljs-comment">// Until the copy completes, we can only call nosplit routines.</span>sp := getcallersp()argp := <span class="hljs-keyword">uintptr</span>(unsafe.Pointer(&amp;fn)) + unsafe.Sizeof(fn)callerpc := getcallerpc()    <span class="hljs-comment">//新的_defer结构，重点！</span>d := newdefer(siz)<span class="hljs-keyword">if</span> d._panic != <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"deferproc: d.panic != nil after newdefer"</span>)&#125;    d.link = gp._defer    <span class="hljs-comment">//赋值fn,pc,sp等</span>gp._defer = d    d.fn = fnd.pc = callerpcd.sp = sp<span class="hljs-keyword">switch</span> siz &#123;<span class="hljs-keyword">case</span> <span class="hljs-number">0</span>:<span class="hljs-comment">// Do nothing.</span><span class="hljs-keyword">case</span> sys.PtrSize:*(*<span class="hljs-keyword">uintptr</span>)(deferArgs(d)) = *(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(argp))<span class="hljs-keyword">default</span>:memmove(deferArgs(d), unsafe.Pointer(argp), <span class="hljs-keyword">uintptr</span>(siz))    &#125;    <span class="hljs-comment">//避免无限递归调用deferreturn，其是唯一一个不会触发由延迟调用的函数</span>    <span class="hljs-comment">// deferproc returns 0 normally.</span><span class="hljs-comment">// a deferred func that stops a panic</span><span class="hljs-comment">// makes the deferproc return 1.</span><span class="hljs-comment">// the code the compiler generates always</span><span class="hljs-comment">// checks the return value and jumps to the</span>    <span class="hljs-comment">// end of the function if deferproc returns != 0.</span>    <span class="hljs-comment">//其会在deferproc的最后去发信号给不会跳转到deferreturn的goroutine</span>    <span class="hljs-comment">// return0 is a stub used to return 0 from deferproc.</span>    <span class="hljs-comment">// It is called at the very end of deferproc to signal</span>    <span class="hljs-comment">// the calling Go function that it should not jump</span>    <span class="hljs-comment">// to deferreturn.</span>    <span class="hljs-comment">// in asm_*.s</span>return0()&#125;</code></pre></div><p>其中<code>newdefer()</code>的作用就是要获取一个<code>_defer</code>结构,大概分为几种方式:</p><ul><li><p>从 全局调度器 的延迟调用缓存池<code>sched.deferpool</code>中取出结构体并将该结构体追加到当前goroutine的缓存池</p></li><li><p>从goroutine绑定的<code>p</code>的延迟调用缓存池<code>pp.deferpool</code>中取出</p></li><li><p><code>mallocgc()</code>创建一个新的结构体</p></li></ul><p>无论哪种，最后都会被放入到<code>link</code>链表的最前面</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Allocate a Defer, usually using per-P pool.</span><span class="hljs-comment">// Each defer must be released with freedefer.  The defer is not</span><span class="hljs-comment">// added to any defer chain yet.</span><span class="hljs-comment">//</span><span class="hljs-comment">// This must not grow the stack because there may be a frame without</span><span class="hljs-comment">// stack map information when this is called.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newdefer</span><span class="hljs-params">(siz <span class="hljs-keyword">int32</span>)</span> *_<span class="hljs-title">defer</span></span> &#123;<span class="hljs-keyword">var</span> d *_defersc := deferclass(<span class="hljs-keyword">uintptr</span>(siz))gp := getg()<span class="hljs-keyword">if</span> sc &lt; <span class="hljs-keyword">uintptr</span>(<span class="hljs-built_in">len</span>(p&#123;&#125;.deferpool)) &#123;        pp := gp.m.p.ptr()        <span class="hljs-comment">//--------------1. 从sched.deferpool里面拿--------------</span><span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(pp.deferpool[sc]) == <span class="hljs-number">0</span> &amp;&amp; sched.deferpool[sc] != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// Take the slow path on the system stack so</span><span class="hljs-comment">// we don't grow newdefer's stack.</span>systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;lock(&amp;sched.deferlock)<span class="hljs-keyword">for</span> <span class="hljs-built_in">len</span>(pp.deferpool[sc]) &lt; <span class="hljs-built_in">cap</span>(pp.deferpool[sc])/<span class="hljs-number">2</span> &amp;&amp; sched.deferpool[sc] != <span class="hljs-literal">nil</span> &#123;                    <span class="hljs-comment">//拿出_defer</span>d := sched.deferpool[sc]sched.deferpool[sc] = d.link                    d.link = <span class="hljs-literal">nil</span>                    <span class="hljs-comment">//append到当前goroutine的缓存池中</span>pp.deferpool[sc] = <span class="hljs-built_in">append</span>(pp.deferpool[sc], d)&#125;unlock(&amp;sched.deferlock)&#125;)        &#125;        <span class="hljs-comment">//-----------------2. 从gourtine的pp.deferpool中拿</span><span class="hljs-keyword">if</span> n := <span class="hljs-built_in">len</span>(pp.deferpool[sc]); n &gt; <span class="hljs-number">0</span> &#123;d = pp.deferpool[sc][n<span class="hljs-number">-1</span>]pp.deferpool[sc][n<span class="hljs-number">-1</span>] = <span class="hljs-literal">nil</span>pp.deferpool[sc] = pp.deferpool[sc][:n<span class="hljs-number">-1</span>]&#125;    &#125;    <span class="hljs-keyword">if</span> d == <span class="hljs-literal">nil</span> &#123;        <span class="hljs-comment">//-----------------3. mallogc一个新的结构体--------</span><span class="hljs-comment">// Allocate new defer+args.</span>systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;total := roundupsize(totaldefersize(<span class="hljs-keyword">uintptr</span>(siz)))d = (*_defer)(mallocgc(total, deferType, <span class="hljs-literal">true</span>))&#125;)<span class="hljs-keyword">if</span> debugCachedWork &#123;<span class="hljs-comment">// Duplicate the tail below so if there's a</span><span class="hljs-comment">// crash in checkPut we can tell if d was just</span><span class="hljs-comment">// allocated or came from the pool.</span>            d.siz = siz            <span class="hljs-comment">//追加到link上面</span>d.link = gp._defergp._defer = d<span class="hljs-keyword">return</span> d&#125;&#125;d.siz = sizd.heap = <span class="hljs-literal">true</span><span class="hljs-keyword">return</span> d&#125;</code></pre></div><p>上面<code>defer</code>关键字插入是从后到前，但是其执行是从前到后，即为什么运行好像一个栈一样</p><h4 id="执行延迟调用-堆上">执行延迟调用(堆上)</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Run a deferred function if there is one.</span><span class="hljs-comment">// The compiler inserts a call to this at the end of any</span><span class="hljs-comment">// function which calls defer.</span><span class="hljs-comment">// If there is a deferred function, this will call runtime·jmpdefer,</span><span class="hljs-comment">// which will jump to the deferred function such that it appears</span><span class="hljs-comment">// to have been called by the caller of deferreturn at the point</span><span class="hljs-comment">// just before deferreturn was called. The effect is that deferreturn</span><span class="hljs-comment">// is called again and again until there are no more deferred functions.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Declared as nosplit, because the function should not be preempted once we start</span><span class="hljs-comment">// modifying the caller's frame in order to reuse the frame to call the deferred</span><span class="hljs-comment">// function.</span><span class="hljs-comment">//</span><span class="hljs-comment">// The single argument isn't actually used - it just has its address</span><span class="hljs-comment">// taken so it can be matched against pending defers.</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">deferreturn</span><span class="hljs-params">(arg0 <span class="hljs-keyword">uintptr</span>)</span></span> &#123;    gp := getg()    <span class="hljs-comment">//取出_defer</span>d := gp._defer<span class="hljs-keyword">if</span> d == <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span>&#125;sp := getcallersp()<span class="hljs-keyword">if</span> d.sp != sp &#123;<span class="hljs-keyword">return</span>&#125;<span class="hljs-keyword">if</span> d.openDefer &#123;<span class="hljs-comment">//1.14 opencoded</span>done := runOpenDeferFrame(gp, d)<span class="hljs-keyword">if</span> !done &#123;throw(<span class="hljs-string">"unfinished open-coded defers in deferreturn"</span>)&#125;gp._defer = d.linkfreedefer(d)<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">// Moving arguments around.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Everything called after this point must be recursively</span><span class="hljs-comment">// nosplit because the garbage collector won't know the form</span><span class="hljs-comment">// of the arguments until the jmpdefer can flip the PC over to</span><span class="hljs-comment">// fn.</span><span class="hljs-keyword">switch</span> d.siz &#123;<span class="hljs-keyword">case</span> <span class="hljs-number">0</span>:<span class="hljs-comment">// Do nothing.</span><span class="hljs-keyword">case</span> sys.PtrSize:*(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(&amp;arg0)) = *(*<span class="hljs-keyword">uintptr</span>)(deferArgs(d))<span class="hljs-keyword">default</span>:memmove(unsafe.Pointer(&amp;arg0), deferArgs(d), <span class="hljs-keyword">uintptr</span>(d.siz))&#125;fn := d.fnd.fn = <span class="hljs-literal">nil</span>gp._defer = d.linkfreedefer(d)<span class="hljs-comment">// If the defer function pointer is nil, force the seg fault to happen</span><span class="hljs-comment">// here rather than in jmpdefer. gentraceback() throws an error if it is</span><span class="hljs-comment">// called with a callback on an LR architecture and jmpdefer is on the</span><span class="hljs-comment">// stack, because the stack trace can be incorrect in that case - see</span><span class="hljs-comment">// issue #8153).</span>    _ = fn.fn    <span class="hljs-comment">//传入_defer的fn和其参数</span>jmpdefer(fn, <span class="hljs-keyword">uintptr</span>(unsafe.Pointer(&amp;arg0)))&#125;</code></pre></div><p><code>jmpdefer</code>是汇编实现的runtime函数，目的就是跳转<code>defer</code>所在代码，并在执行结束之后跳转回<code>deferreturn</code></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// func jmpdefer(fv *funcval, argp uintptr)</span><span class="hljs-comment">// argp is a caller SP.</span><span class="hljs-comment">// called from deferreturn.</span><span class="hljs-comment">// 1. pop the caller</span><span class="hljs-comment">// 2. sub 5 bytes from the callers return</span><span class="hljs-comment">// 3. jmp to the argument</span>TEXT runtime·jmpdefer(SB), NOSPLIT, $<span class="hljs-number">0</span><span class="hljs-number">-16</span>MOVQfv+<span class="hljs-number">0</span>(FP), DX<span class="hljs-comment">// fn</span>MOVQargp+<span class="hljs-number">8</span>(FP), BX<span class="hljs-comment">// caller sp</span>LEAQ<span class="hljs-number">-8</span>(BX), SP<span class="hljs-comment">// caller sp after CALL</span>MOVQ<span class="hljs-number">-8</span>(SP), BP<span class="hljs-comment">// restore BP as if deferreturn returned (harmless if framepointers not in use)</span>SUBQ$<span class="hljs-number">5</span>, (SP)<span class="hljs-comment">// return to CALL again</span>MOVQ<span class="hljs-number">0</span>(DX), BXJMPBX<span class="hljs-comment">// but first run the deferred function</span></code></pre></div><p>//todo ???<code>deferreturn</code> 函数会多次判断当前 Goroutine 的 _defer 链表中是否有未执行的剩余结构，在所有的延迟函数调用都执行完成之后，该函数才会返回;</p><ul><li>调用<code>deferproc</code>创建新的延迟调用时就会立刻copy函数的参数，所以参数在defer声明时就已经开始执行计算</li></ul><h4 id="open-coded">open-coded</h4><p>对于<code>运行中</code>才确定的defer，可以看到其中1.14新增的<code>open-coded</code>，采用位操作(<code>defer bits</code>，即变量<code>df</code>)来确认分支:</p><ul><li><p>首先直接在<code>defer func()</code>这种会直接插入;</p></li><li><p>其次，在<code>条件判断分支</code>中的 defer,则要在<code>运行时</code>记录每个defer是否被执行，从而便于判断最后的延迟调用该执行哪些函数;</p></li></ul><p>原理：</p><p>同一个函数内每出现一个 defer 都会为其分配 <code>var df byte</code>，如果被执行到则设为 <code>1</code>，否则设为 <code>0</code>(比如<code>df|=1</code>)，当到达函数返回之前需要判断延迟调用时，则用掩码(比如<code>df&amp;1&gt;0</code>判断第一个defer是否存在)判断每个位置的比特，若为 1 则调用延迟函数，否则跳过。</p><p>为了轻量，官方将延迟比特限制为 1 个字节，即 8 个比特，这就是为什么不能超过 8 个 <code>defer</code> 的原因，若超过依然会选择堆栈分配，但显然大部分情况不会超过 8 个;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// runOpenDeferFrame runs the active open-coded defers in the frame specified by</span><span class="hljs-comment">// d. It normally processes all active defers in the frame, but stops immediately</span><span class="hljs-comment">// if a defer does a successful recover. It returns true if there are no</span><span class="hljs-comment">// remaining defers to run in the frame.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">runOpenDeferFrame</span><span class="hljs-params">(gp *g, d *_defer)</span> <span class="hljs-title">bool</span></span> &#123;done := <span class="hljs-literal">true</span>fd := d.fd<span class="hljs-comment">// Skip the maxargsize</span>_, fd = readvarintUnsafe(fd)deferBitsOffset, fd := readvarintUnsafe(fd)<span class="hljs-comment">//defer的数量</span>nDefers, fd := readvarintUnsafe(fd)<span class="hljs-comment">//获得deferBits，因为open-coded最多支持8个，所以默认bits数量就是8,等于一个字节</span><span class="hljs-comment">//而且这些bits是运行时设置的</span>deferBits := *(*<span class="hljs-keyword">uint8</span>)(unsafe.Pointer(d.varp - <span class="hljs-keyword">uintptr</span>(deferBitsOffset)))<span class="hljs-keyword">for</span> i := <span class="hljs-keyword">int</span>(nDefers) - <span class="hljs-number">1</span>; i &gt;= <span class="hljs-number">0</span>; i-- &#123;<span class="hljs-comment">// read the funcdata info for this defer</span><span class="hljs-keyword">var</span> argWidth, closureOffset, nArgs <span class="hljs-keyword">uint32</span>argWidth, fd = readvarintUnsafe(fd)closureOffset, fd = readvarintUnsafe(fd)nArgs, fd = readvarintUnsafe(fd)<span class="hljs-comment">//移位来判定,掩码</span><span class="hljs-keyword">if</span> deferBits&amp;(<span class="hljs-number">1</span>&lt;&lt;i) == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">for</span> j := <span class="hljs-keyword">uint32</span>(<span class="hljs-number">0</span>); j &lt; nArgs; j++ &#123;_, fd = readvarintUnsafe(fd)_, fd = readvarintUnsafe(fd)_, fd = readvarintUnsafe(fd)&#125;<span class="hljs-keyword">continue</span>&#125;closure := *(**funcval)(unsafe.Pointer(d.varp - <span class="hljs-keyword">uintptr</span>(closureOffset)))d.fn = closuredeferArgs := deferArgs(d)<span class="hljs-comment">// If there is an interface receiver or method receiver, it is</span><span class="hljs-comment">// described/included as the first arg.</span><span class="hljs-keyword">for</span> j := <span class="hljs-keyword">uint32</span>(<span class="hljs-number">0</span>); j &lt; nArgs; j++ &#123;<span class="hljs-keyword">var</span> argOffset, argLen, argCallOffset <span class="hljs-keyword">uint32</span>argOffset, fd = readvarintUnsafe(fd)argLen, fd = readvarintUnsafe(fd)argCallOffset, fd = readvarintUnsafe(fd)memmove(unsafe.Pointer(<span class="hljs-keyword">uintptr</span>(deferArgs)+<span class="hljs-keyword">uintptr</span>(argCallOffset)),unsafe.Pointer(d.varp-<span class="hljs-keyword">uintptr</span>(argOffset)),<span class="hljs-keyword">uintptr</span>(argLen))&#125;<span class="hljs-comment">//移位到下一位</span>deferBits = deferBits &amp;^ (<span class="hljs-number">1</span> &lt;&lt; i)*(*<span class="hljs-keyword">uint8</span>)(unsafe.Pointer(d.varp - <span class="hljs-keyword">uintptr</span>(deferBitsOffset))) = deferBitsp := d._panicreflectcallSave(p, unsafe.Pointer(closure), deferArgs, argWidth)<span class="hljs-keyword">if</span> p != <span class="hljs-literal">nil</span> &amp;&amp; p.aborted &#123;<span class="hljs-keyword">break</span>&#125;d.fn = <span class="hljs-literal">nil</span><span class="hljs-comment">// These args are just a copy, so can be cleared immediately</span>memclrNoHeapPointers(deferArgs, <span class="hljs-keyword">uintptr</span>(argWidth))<span class="hljs-keyword">if</span> d._panic != <span class="hljs-literal">nil</span> &amp;&amp; d._panic.recovered &#123;done = deferBits == <span class="hljs-number">0</span><span class="hljs-keyword">break</span>&#125;&#125;<span class="hljs-keyword">return</span> done&#125;</code></pre></div><h5 id="缺点">缺点</h5><p>官方给出的测试中提高了几乎有一个0，但是要注意到一种问题，如果在插入的多个oepn-coded代码之间<code>panic</code>或者<code>goExit()</code>，下面的代码就不会进入，而是去找注册的<code>defer</code>;针对这种情况，程序还会去进行一次栈扫描;</p><p>当然针对这些情况，<code>_defer</code>结构体就新增了以下几个字段来帮忙找到未注册到链表的defer函数:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> _defer <span class="hljs-keyword">struct</span> &#123;...    openDefer <span class="hljs-keyword">bool</span><span class="hljs-comment">// If openDefer is true, the fields below record values about the stack</span><span class="hljs-comment">// frame and associated function that has the open-coded defer(s). sp</span><span class="hljs-comment">// above will be the sp for the frame, and pc will be address of the</span><span class="hljs-comment">// deferreturn call in the function.</span>fd   unsafe.Pointer <span class="hljs-comment">// funcdata for the function associated with the frame</span>varp <span class="hljs-keyword">uintptr</span>        <span class="hljs-comment">// value of varp for the stack frame</span><span class="hljs-comment">// framepc is the current pc associated with the stack frame. Together,</span><span class="hljs-comment">// with sp above (which is the sp associated with the stack frame),</span><span class="hljs-comment">// framepc/sp can be used as pc/sp pair to continue a stack trace via</span><span class="hljs-comment">// gentraceback().</span>framepc <span class="hljs-keyword">uintptr</span>&#125;</code></pre></div><p>其实就导致在1.14中，如果使用opencoded，defer的确会变快了，但是在有panic的情况，却更慢了;</p>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Kafka notes</title>
    <link href="/2020/08/30/Comcon/kafka/"/>
    <url>/2020/08/30/Comcon/kafka/</url>
    
    <content type="html"><![CDATA[<h2 id="kafka一些名词">kafka一些名词</h2><p><code>Broker</code>：消息中间件处理结点，一个Kafka节点就是一个broker，多个broker可以组成一个Kafka集群。</p><p><code>Topic</code>：一类消息，例如page view日志、click日志等都可以以topic的形式存在，Kafka集群能够同时负责多个topic的分发。</p><p><code>Partition</code>：topic物理上的分组，一个topic可以分为多个partition，每个partition是一个有序的队列。</p><p><code>Segment</code>：partition物理上由多个segment组成，下面2.2和2.3有详细说明。</p><p><code>offset</code>：每个partition都由一系列有序的、不可变的消息组成，这些消息被连续的追加到partition中;partition中的每个消息都有一个连续的序列号叫做offset,用于partition唯一标识一条消息</p><p><code>producer</code>: 生产者，负责发布信息到broker</p><p><code>consumer</code>: 消费者，每个consumer都属于一个consumer group，可以指定其名字；同一个topic只能被同一个consumer group下的一个consumer消费，但是多个consumer group可以同时消费该topic</p><h2 id="思考">思考</h2><p>作为一个消息队列，要求有哪些</p><ol><li>队列（数据结构）</li><li>持久性（可以恢复数据），怎么存储，怎么恢复,过期的数据处理（因为不可能全部保存）</li><li>分布式（高可用），基于zookeeper带来的ZAB选举协议，后面会慢慢去掉该依赖</li><li>速度（性能） 持久化速度，网络IO速度（编码）；消费端exactly-once消费</li><li>如何设置相关参数?</li><li>其他功能？ 延时队列（场景：并发量大，订单超时取消）</li></ol><h2 id="持久性">持久性</h2><h3 id="怎么存">怎么存</h3><p>通过日志存储：</p><p>Topic 包含 ParitionParition包含log</p><h4 id="存到哪个partition呢">存到哪个partition呢?</h4><p>可以在server.properties设置里面<code>num.Partitions</code>设置多个partition,存到哪个要由客户端决定</p><h4 id="log又是怎么存的">log又是怎么存的?</h4><p>log实际是分为00000000000000000000.index  00000000000000000000.log  00000000000000000000.timeindex  00000000000000000001.snapshot  leader-epoch-checkpoint</p><h4 id="存储的时机">存储的时机</h4><p>同样可以从<code>server.properties</code>中看到</p><div class="hljs"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> Messages are immediately written to the filesystem but by default we only fsync() to sync</span><span class="hljs-meta">#</span><span class="bash"> the OS cache lazily. The following configurations control the flush of data to disk.</span><span class="hljs-meta">#</span><span class="bash"> There are a few important trade-offs here:</span><span class="hljs-meta">#</span><span class="bash">    1. Durability: Unflushed data may be lost <span class="hljs-keyword">if</span> you are not using replication.</span><span class="hljs-meta">#</span><span class="bash">    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.</span><span class="hljs-meta">#</span><span class="bash">    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.</span><span class="hljs-meta">#</span><span class="bash"> The settings below allow one to configure the flush policy to flush data after a period of time or</span><span class="hljs-meta">#</span><span class="bash"> every N messages (or both). This can be <span class="hljs-keyword">done</span> globally and overridden on a per-topic basis.</span><span class="hljs-meta">#</span><span class="bash"> The number of messages to accept before forcing a flush of data to disk</span><span class="hljs-meta">#</span><span class="bash">log.flush.interval.messages=10000</span><span class="hljs-meta">#</span><span class="bash"> The maximum amount of time a message can sit <span class="hljs-keyword">in</span> a <span class="hljs-built_in">log</span> before we force a flush</span><span class="hljs-meta">#</span><span class="bash">log.flush.interval.ms=1000</span></code></pre></div><p>其中<code>log.flush.interval.ms=1000</code> 和<code>log.flush.interval.messages=10000</code> 就是刷盘的时机；</p><h4 id="压缩">压缩</h4><p>从文档中可以获得其设计内容:</p><blockquote><blockquote><p>Log compaction is handled by the log cleaner, a pool of background threads that recopy log segment files, removing records whose key   appears in the head of the log. Each compactor thread works as follows:It chooses the log that has the highest ratio of log head to log tailIt creates a succinct summary of the last offset for each key in the head of the logIt recopies the log from beginning to end removing keys which have a later occurrence in the log. New, clean segments are swapped into the log immediately so the additional disk space required is just one additional log segment (not a fully copy of the log).The summary of the log head is essentially just a space-compact hash table. It uses exactly 24 bytes per entry. As a result with 8GB of cleaner buffer one cleaner iteration can clean around 366GB of log head (assuming 1k messages).</p></blockquote></blockquote><ol><li>压缩由<code>log cleaner</code>处理，<code>log cleaner</code>是一堆由后台的复制logsegment files的线程池，负责移除那些key在log头部的记录;</li><li>每个压缩线程会选择最经常出现的记录(整个log)</li><li>其会 在 log的头部创建每个key的最近一次offset的summary</li><li>其会从头recopy相关的log到尾部，然后移除在靠后出现的keys；新的，干净的segments会被立即替换到log（因此只需<strong>一次</strong>额外的log segment IO即可）</li><li></li></ol><h3 id="怎么恢复">怎么恢复</h3><h3 id="过期删除">过期删除</h3><p>同样，log不可能无限增长，都设置了删除log的时机，也是由<code>log cleaner</code>处理，条件</p><div class="hljs"><pre><code class="hljs shell"><span class="hljs-meta">#</span><span class="bash"> The following configurations control the disposal of <span class="hljs-built_in">log</span> segments. The policy can</span><span class="hljs-meta">#</span><span class="bash"> be <span class="hljs-built_in">set</span> to delete segments after a period of time, or after a given size has accumulated.</span><span class="hljs-meta">#</span><span class="bash"> A segment will be deleted whenever *either* of these criteria are met. Deletion always happens</span><span class="hljs-meta">#</span><span class="bash"> from the end of the <span class="hljs-built_in">log</span>.</span><span class="hljs-meta">#</span><span class="bash"> The minimum age of a <span class="hljs-built_in">log</span> file to be eligible <span class="hljs-keyword">for</span> deletion due to age</span>log.retention.hours=168<span class="hljs-meta">#</span><span class="bash"> A size-based retention policy <span class="hljs-keyword">for</span> logs. Segments are pruned from the <span class="hljs-built_in">log</span> unless the remaining</span><span class="hljs-meta">#</span><span class="bash"> segments drop below log.retention.bytes. Functions independently of log.retention.hours.</span><span class="hljs-meta">#</span><span class="bash">log.retention.bytes=1073741824</span><span class="hljs-meta">#</span><span class="bash"> The maximum size of a <span class="hljs-built_in">log</span> segment file. When this size is reached a new <span class="hljs-built_in">log</span> segment will be created.</span>log.segment.bytes=1073741824<span class="hljs-meta">#</span><span class="bash"> The interval at <span class="hljs-built_in">which</span> <span class="hljs-built_in">log</span> segments are checked to see <span class="hljs-keyword">if</span> they can be deleted according</span><span class="hljs-meta">#</span><span class="bash"> to the retention policies</span>log.retention.check.interval.ms=300000</code></pre></div><h2 id="速度">速度</h2><h3 id="网络层">网络层</h3><p>使用NIO server;</p><h4 id="zerocopy">zerocopy</h4><p>接口是<code>MessageSet</code>,写入是其<code>writeTo</code>方法，</p><p>其不使用buffer write的原因是这个方法允许file-backed messages set去使用更加有效率的java中的<code>FileChannel.transferTo()</code>方法,其实际就是操作系统的<code>sendfile()</code>系统调用！</p><p>在kafka中用于<code>FileRecords</code>的<code>writeTo()</code>方法</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-meta">@Override</span><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">long</span> wri <span class="hljs-title">teTo</span><span class="hljs-params">(GatheringByteChannel destChannel, <span class="hljs-keyword">long</span> offset, <span class="hljs-keyword">int</span> length)</span> <span class="hljs-keyword">throws</span> IOException </span>&#123;    <span class="hljs-keyword">long</span> newSize = Math.min(channel.size(), end) - start;    <span class="hljs-keyword">int</span> oldSize = sizeInBytes();    <span class="hljs-keyword">if</span> (newSize &lt; oldSize)        <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> KafkaException(String.format(                <span class="hljs-string">"Size of FileRecords %s has been truncated during write: old size %d, new size %d"</span>,                file.getAbsolutePath(), oldSize, newSize));    <span class="hljs-keyword">long</span> position = start + offset;    <span class="hljs-keyword">int</span> count = Math.min(length, oldSize);    <span class="hljs-keyword">final</span> <span class="hljs-keyword">long</span> bytesTransferred;    <span class="hljs-keyword">if</span> (destChannel <span class="hljs-keyword">instanceof</span> TransportLayer) &#123;        TransportLayer tl = (TransportLayer) destChannel;        bytesTransferred = tl.transferFrom(channel, position, count);    &#125; <span class="hljs-keyword">else</span> &#123;        <span class="hljs-comment">//这里就是直接调用java的transferto</span>        bytesTransferred = channel.transferTo(position, count, destChannel);    &#125;    <span class="hljs-keyword">return</span> bytesTransferred;&#125;</code></pre></div><p>这种设计还有一种名字叫做<code>zeroCopy</code>:</p><h5 id="传统read-write">传统read write</h5><p>传统的<code>read()</code>大概经历了如图 <img src="/img/zeroCopy.png" srcset="/img/loading.gif" alt="所示"></p><p>整个操作两次 cpu copy，两次 DMA copy，四次上下文切换(读两次，写两次)，两次系统调用;</p><h5 id="sendfile">sendfile</h5><p>而<code>zeroCopy</code>可以直接从内核缓冲区写入socket缓冲区，避免了普通的<code>read()</code>方法的各种<strong>上下文交换</strong>以及<strong>内核空间缓冲区与用户空间缓冲区</strong>的交换</p><p>改写后就会变成<img src="/img/zeroCopyreal.png" srcset="/img/loading.gif" alt="如图"></p><p>整个过程两次 DMA copy，两次上下文切换，1 次 cpu copy，一个系统调用;</p><h5 id="scatter-gather-dma-sendfile">Scatter/Gather DMA + sendfile</h5><p>ps：看到上图，既然read buffer与socekt buffer之间还有一次CPU copy，如果我们将该copy去掉呢？</p><p>首先讲下原因： 这是因为在一般的Block DMA方式中，<code>源物理地址</code>和<code>目标物理地址</code>都得是<strong>连续</strong>的，所以一次只能传输物理上连续的一块数据，每传输一个块发起一次中断，直到传输完成，所以必须要在两个缓冲区之间拷贝数据;</p><p>而<code>Scatter/Gather DMA</code>方式则不同，会预先维护一个物理上<code>不连续的块描述符</code>的链表，描述符中包含有数据的<code>起始地址</code>和<code>长度</code>;传输时((DMA控制器负责)只需要遍历链表，按序传输数据，全部完成后发起一次中断即可，效率比<code>Block DMA</code></p><blockquote><blockquote><p>Block DMA: 如果在传输完一块物理上连续的数据后引起一次中断，然后再由主机进行下一块物理上连续的数据传输</p></blockquote></blockquote><p>要高。也就是说，硬件可以通过<code>Scatter/Gather DMA</code>直接从内核缓冲区中取得全部数据，不需要再从内核缓冲区向Socket缓冲区拷贝数据(再次省去了一次CPU copy)。因此上面的时序图还可以进一步简化;</p><p>如<img src="/img/zeroCopyScatterGather.png" srcset="/img/loading.gif" alt="下图"></p><p>总的只需要2次上下文切换，0次CPU copy,3 次DMAcopy；</p><h5 id="mmap-unmap内存映射支持">mmap/Unmap内存映射支持</h5><p>上面的<code>Scatter/gather</code>很明显DMA从内核取得数据是不可以更改数据的，如果要修改数据，则需要<code>内存映射</code>的支持,可以将文件数据映射到内核地址空间，修改完后刷回去;</p><p>当然了，<code>mmap</code>也是系统调用，会引起上下文切换,不需要上下切换的只不过是改的过程,然后进行一次<code>write()</code>的调用，对应的的最后也要调用一个<code>unmap()</code>系统调用,所以这里总共会有六次上下文切换,</p><p>另外，还需要在TLB中维护所有数据对应的地址空间;</p><p><img src="/img/zeroCopymap.png" srcset="/img/loading.gif" alt="如图"></p><p>总的需要6次上下文切换，1次CPU copy，2次DMA copy</p><h5 id="splice">Splice</h5><p>还有一种方法<code>Splice</code>是基于管道的，可以用于在内核区进行,同样避免了用户态切换以及CPU的copy:<img src="/img/zeroCopySplice.png" srcset="/img/loading.gif" alt="如图"></p><p>要特别注意的就是这两个buffer其中一个的<code>fd</code>必须是管道设备</p><h4 id="线程设计">线程设计</h4><p>进程方面也是一个acceptor的线程和N个processor线程，其中processor线程都会处理<strong>固定数目</strong>的connection;</p><h3 id="存储">存储</h3><h4 id="顺序读写">顺序读写</h4><p>都是append到文件中;</p><h4 id="page-cache">page cache</h4><p>kafka借用了系统的page cache的来处理缓存;原因有几点：</p><ul><li>如果自己管理cache，因为其为自定义object，jvm会将其设为object overhead，浪费空间；</li><li>JVM会有自己的GC，过大的堆也会影响GC，降低吞吐量</li><li>如果Kafka崩溃，在内存中的数据也会不见；</li></ul><h4 id="分区">分区</h4><p>topic可以分为多个partition，每个partition又可以分为多个segment，每次操作都是对小部分操作，大量减少消耗和增加并行性;</p><h3 id="数据本身处理">数据本身处理</h3><h4 id="批量发送">批量发送</h4><h4 id="压缩-v2">压缩</h4><p>Gzip,snappy压缩数据</p><h2 id="分布式">分布式</h2><h3 id="同步问题">同步问题</h3><p>主要由几点保证：</p><h4 id="1-replicated">1. Replicated</h4><p>ISR：In-Sync Replicated每个Partition都有一个ISR</p><p>AR: assigned Replicated</p><p>ISR由leader维护,follower</p><h4 id="2-hw-leo">2. HW，LEO</h4><p>看几幅图<img src="/img/kafkaDef.png" srcset="/img/loading.gif" alt="盗图"></p><ul><li><p>Base Offset: 第一条日志的offset，</p></li><li><p>HW: replicated的leader和follower都会有这个值，leader中的HW决定consumer可以消费到哪个offset；如图[0,8]的消息都可以被消费;</p></li><li><p>LEO: log end offset,即日志末端位移，代表日志文件中下一条待写入消息的offset，同样leader和follower都有该值；leader收到消息后这个值+1,但follower要从leader副本fetch到消息后才可以增加，最后leader取两者之间的LEO的最小值作为HW，更新自己的HW；</p></li></ul><p>几个副本中，最旧版本offset即为High watermark，恢复数据时也以这个为准</p><h5 id="epoch">epoch</h5><p>在0.11以后的版本，新增了(Leader epoch);解决了 宕机问题造成的offset数据不一致以及数据丢失两种问题:</p><ul><li>数据丢失:</li></ul><ul><li>数据不一致:</li></ul><p>增加了epoch,存储在leader broker上，与offset组成<code>(epoch, offset)</code>key-value pair,其会被定期写入一个checkpoint;</p><ul><li><p>Leader每<strong>发生一次变化</strong>epoch就会+1，offset就代表该epoch版本的Leader写入的第一条日志的位移;</p></li><li><p>Leader首次写入就会增加一个记录；</p></li></ul><h4 id="3-ack">3. ACK</h4><h2 id="如何设置相关参数">如何设置相关参数</h2><ul><li><p>Partition数量设置最好是broker的整数倍，这样topic会平均分到每个broker相同的partitions</p></li><li><p>另一个需要特别注意的问题是<code>lagging consumer</code>，即那些消费速率慢、明显落后的consumer;它们要读取的数据有较大概率不在broker page cache中，因此会增加很多不必要的读盘操作;比这更坏的是，<code>lagging consumer</code>读取的“冷”数据仍然会进入<code>page cache</code>，污染了多数正常consumer要读取的“热”数据，连带着正常consumer的性能变差;</p></li><li><p>page_cache的设置：<code>/proc/sys/vm/dirty_writeback_centisecs</code>：flush检查的周期。单位为0.01秒，默认值500，即5秒。每次检查都会按照以下三个参数控制的逻辑来处理。<code>/proc/sys/vm/dirty_expire_centisecs</code>：如果page cache中的页被标记为dirty的时间超过了这个值，就会被直接刷到磁盘。单位为0.01秒。默认值3000，即半分钟。<code>/proc/sys/vm/dirty_background_ratio</code>：如果dirty page的总大小占空闲内存量的比例超过了该值，就会在后台调度flusher线程异步写磁盘，不会阻塞当前的write()操作。默认值为10%。<code>/proc/sys/vm/dirty_ratio</code>：如果dirty page的总大小占总内存量的比例超过了该值，就会阻塞所有进程的write()操作，并且强制每个进程将自己的文件写入磁盘。默认值为20%。一般可以考虑调整dirty_expire_centisecs 和 dirty_background_ratio</p></li></ul><h2 id="一些版本更替问题">一些版本更替问题</h2><h3 id="主要是对zookeeper依赖问题">主要是对zookeeper依赖问题</h3><h4 id="客户端">客户端</h4><ul><li>Kafka 0.8时代，Kafka有3个客户端，分别是Producer、Consumer和Admin Tool;</li></ul><p>其中Producer负责向Kafka写消息，Consumer负责从Kafka读消息，而Admin Tool执行各种运维任务，比如创建或删除主题等;其中Consumer的位移数据保存在ZooKeeper上，因此Consumer端的位移提交和位移获取操作都需要访问ZooKeeper，另外Admin Tool执行运维操作也要访问ZooKeeper，比如在对应的ZooKeeper znode上创建一个临时节点，然后由预定义的Watch触发相应的处理逻辑。</p><h4 id="服务端-broker">服务端(broker)</h4><p>依赖较为严重:</p><p>主要是</p><p>-Broker注册管理(靠zk的quorum)-ACL安全层配置-动态参数管理-ISR管理-Controller选举</p><h1>参考</h1><p><a href="https://www.jianshu.com/p/92f33aa0ff52" target="_blank" rel="noopener">https://www.jianshu.com/p/92f33aa0ff52</a></p>]]></content>
    
    
    
    <tags>
      
      <tag>kafka</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Transaction Isolation in Innodb</title>
    <link href="/2020/08/03/MySQL/InnoDBLock/"/>
    <url>/2020/08/03/MySQL/InnoDBLock/</url>
    
    <content type="html"><![CDATA[<p>事务的一些隔离问题</p><a id="more"></a><h2 id="事务的加锁级别">事务的加锁级别</h2><p>说到烂了</p><ol><li>Read Uncommitted读未提交，只<strong>防止脏写（其实脏写只是写并发的一个特例而已，还可能有更新丢失这种情况）</strong>，<strong>不防止脏读</strong>，这个级别他喵 的就等于没有隔离;但是注意这种对于一些sum，count操作其实是有用的，只不过我们日常少用;</li></ol><ol start="2"><li>Read Committed</li></ol><p>很多数据库默认就是这个级别，<strong>防止单条记录(读取到)的脏读</strong>，但是对于批量范围操作就力不从心，即不能防止幻读e.g:</p><ol start="3"><li>Repeatable Read</li></ol><p>在Read Commited的基础上使用快照来隔离不同版本，保证了单条记录加锁以及范围读取的加锁，新的满足查询条件的记录不能够插入，可以解决幻读 (<strong>这里指得是只读查询时的，对于读-写事务来讲，还是可能会有写倾斜</strong>),幻读指的就是同一个事务中，连续两次当前读得到了两个完全相同(一般有返回多条)的记录;如果索引(非唯一)的情况下，有Repeatable Read在这两次之间就又加了一个细化的锁(Gap Lock)保证这两次读之间不会插入新的提交值详情可以看下面的<a href>Gap锁</a></p><p>同Read Commited一样的是都有写锁，但是读不加锁，采用保存了多个版本(MVCC)来实现；</p><p>有些数据库会直接使用MVCC保存<strong>一个已提交的旧版本</strong>和一个<strong>未提交的新版本</strong>来实现<strong>Read Commited</strong>，虽然未提供完整的快照隔离，这种操作(Read Commited)是对于每一个不同的查询单独创建一个快照，而完整的快照级别隔离则需要对<strong>整个事务</strong>用<strong>一个快照</strong>进行</p><ol start="4"><li>Serilizable全部操作串行化，并发度下降，不推荐</li></ol><h3 id="写事务">写事务</h3><ul><li><p>脏写</p></li><li><p>更新丢失</p></li></ul><ul><li>写倾斜</li></ul><h3 id="锁与索引的关系">锁与索引的关系</h3><p>这里会有人问到，加锁的话怎么用索引呢？</p><ul><li>有一种方案是：索引直接指向数据对象的所有版本，然后使用一些策略过滤掉对当前事务不可见的版本，当后台gc进程决定删除某些旧对象版本时，对应的索引也要删除;一些数据库会把同一对象的不同版本放在同一个内存页中;一些有b-tree结构的数据库则使用append/copy-on-write方法，需要更新的时候会创建一个修改副本，copy一些必要的内容，然后让父节点或者递归一直到root的节点都指向新创建的节点，</li></ul>]]></content>
    
    
    
    <tags>
      
      <tag>MySQL</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang Some bugs notes</title>
    <link href="/2020/07/23/Go/somebugs/"/>
    <url>/2020/07/23/Go/somebugs/</url>
    
    <content type="html"><![CDATA[<p>记录一些自己发现或者是他人发现的一些bug</p><a id="more"></a><h3 id="一个无限循环">一个无限循环</h3><p><a href="https://github.com/golang/go/issues/40367" target="_blank" rel="noopener">issues</a></p><p>一个奇怪的程序会陷入无限循环</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    rates := []<span class="hljs-keyword">int32</span>&#123;<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>&#125;<span class="hljs-keyword">for</span> s, r := <span class="hljs-keyword">range</span> rates &#123;<span class="hljs-keyword">if</span> s+<span class="hljs-number">1</span> &lt; <span class="hljs-number">1</span> &#123; <span class="hljs-comment">//但是将 (s+1) 换成 (s+其他数字) 则可以通过,又或者将panic去掉，改成其他，都可以通过编译</span><span class="hljs-built_in">panic</span>(<span class="hljs-string">"abc"</span>)&#125;fmt.Println(s, r)&#125;&#125;</code></pre></div><p>Expected:</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-symbol">0 </span><span class="hljs-number">1</span><span class="hljs-symbol">1 </span><span class="hljs-number">2</span><span class="hljs-symbol">2 </span><span class="hljs-number">3</span><span class="hljs-symbol">3 </span><span class="hljs-number">4</span><span class="hljs-symbol">4 </span><span class="hljs-number">5</span></code></pre></div><p>Got:</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-number">0</span> <span class="hljs-number">1</span><span class="hljs-number">1</span> <span class="hljs-number">2</span><span class="hljs-number">2</span> <span class="hljs-number">3</span><span class="hljs-number">3</span> <span class="hljs-number">4</span><span class="hljs-number">4</span> <span class="hljs-number">5</span><span class="hljs-number">5</span> <span class="hljs-number">115616</span><span class="hljs-number">6</span> <span class="hljs-number">192</span><span class="hljs-number">7</span> <span class="hljs-number">6717248</span><span class="hljs-number">8</span> <span class="hljs-number">0</span><span class="hljs-number">9</span> <span class="hljs-number">115696</span><span class="hljs-number">10</span> <span class="hljs-number">192</span><span class="hljs-number">11</span> <span class="hljs-number">6717376</span><span class="hljs-number">12</span> <span class="hljs-number">0</span><span class="hljs-number">13</span> <span class="hljs-number">9447072</span><span class="hljs-number">14</span> <span class="hljs-number">192</span><span class="hljs-number">15</span> <span class="hljs-number">393048</span><span class="hljs-number">16</span> <span class="hljs-number">192</span><span class="hljs-number">17</span> <span class="hljs-number">4391790</span><span class="hljs-number">18</span> <span class="hljs-number">0</span><span class="hljs-number">19</span> <span class="hljs-number">139360</span><span class="hljs-number">20</span> <span class="hljs-number">192</span>......</code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Notes about Zookeeper</title>
    <link href="/2020/07/10/Comcon/zookeeper/"/>
    <url>/2020/07/10/Comcon/zookeeper/</url>
    
    <content type="html"><![CDATA[<p>占坑</p><a id="more"></a><p>其名字的来源就是Yahoo的项目组，因为其内部很多以动物为名，所以一种可以方便管理多个服务的程序zookeeper就出现了…</p><p>首先，Zookeeper是一个分布式的程序，所以很自然就会采用分布式地部署(集群);</p><p>提供的功能说大不大:</p><ul><li>管理用户程序提交的数据</li><li>节点监听服务(服务治理，调度等)</li></ul><h2 id="常见的概念">常见的概念</h2><h3 id="session">Session</h3><p>顾名思义，指server和client端之间的一个连接，这个是一个TCP的长连接;</p><p>开始时，每个client都会获得一个sessionId，这个sessionID一定要保持 <strong>全局唯一</strong>，因为下面很多选举以及其他功能都要靠该字段保证</p><h3 id="znode">ZNode</h3><p>zookeeper中的节点分为两类：</p><ol><li>即是机器，机器节点</li><li>数据节点ZNode</li></ol><p>其中数据节点ZNode的数据结构其实是一种树状结构，由斜杠 <strong>/</strong> 进行分割的路径就是一个ZNode，比如/spls/hols,每个节点都有保存自己的信息，还有一系列metadata</p><p>而ZNode又可以分为<strong>持久节点(PERSISTENT)<strong>和</strong>临时节点(EPHEMERAL)</strong>;</p><p>顾名思义，持久指的是除非被进行显式移除操作，否则会一直存在zookeeper中，而临时节点一般是与session绑定，一旦session过期或者失效就会立即被移除；</p><p>这里还有另外两种ZNode:</p><ul><li>持久顺序节点(PERSISTENT_SEQUENTIAL), 同持久节点差不多,额外允许一个SEQUENTIAL参数(int)，由父节点维护的自增数字，会加在当前节点后面;</li><li>临时顺序节点(EPHEMERAL_SEQUENTIAL), 同临时节点差不多只不过就是增加了一个顺序参数;</li></ul><h2 id="zab协议">ZAB协议</h2><p>这里就是重头戏了！！！</p><p>ZAB协议是专门给zookeeper设计的支持崩溃回复的原子广播协议，其包括两种模式：</p><ul><li>崩溃模式</li><li>消息广播</li></ul><h3 id="消息广播">消息广播</h3><p>这个是最基本的一个模式，实际上同2PC十分接近:步骤大概如下:</p><ol><li>Leader将client的request转化成一个Proposal</li><li>Leader为每一个Follower准备了一个FIFO队列，并把proposal发送到队列上；</li><li>Follower会从队列头部拿出proposal，然后将ACK放入回队列中,</li><li>Leader如果从队列收到follower一半以上ACK反馈，就会将commit又发回到队列中</li><li>Follower继续从队列头部拿数据，拿到commit命令就完成;</li></ol><p>但是这里每一步中leader都可能崩溃(毕竟Leader压力也大)接下来就要进入到崩溃模式</p><h3 id="崩溃模式">崩溃模式</h3><p>大家都要恰饭，不可能等leader自己好起来，所以就要选举一个新的leader，这里很自然就涉及两个操作</p><ul><li>选举新的leader</li><li>新的leader要继续之前的工作</li></ul><h4 id="选举新的leader">选举新的leader</h4><p>这里首先可以描述几个状态:</p><ul><li>Looking状态</li><li>following状态</li><li>Leading状态</li></ul><p>比较容易理解，至于选举的时机,无疑就是在 初始化程序的时候 和 leader崩溃后</p><p>之前的文章有写过raft等分布式协议，ZAB也是一个有选举的协议，所以一定有：</p><h5 id="任命周期">任命周期</h5><p><strong>election epoch</strong>，对应于raft的term，一般分布式协议都有以消息标记一个事件，这种事件会同其他事件区分开，而区分的方法很多就是使用了时间戳；ZAB中，每个消息都赋予了一个zxid，zxid全局唯一；</p><p>zxid由两部分组成:</p><ul><li>高32位为epoch</li><li>低32位为epoch内自增id，每次epoch更新，低32位自增id就会归0 （万一一直不更新？？？）</li></ul><h5 id="log">log</h5><p>ZAB和raft一样都是基于复制状态机(replicated state machine)来记录不同机器的行为，其都使用log来记录一般有两种方式:</p><ul><li>所有日志取最新的一条 (raft,ZAB)</li><li>所有已提交的最新的一条</li></ul><p>Zookeeper: peerEpoch大的优先，然后到zxid大的优先Raft: Term大的优先，再到entry的index大的优先</p><p>但是这里Zookeeper与raft有一些不同的是：</p><p>Zookeeper会将两个比较分为两轮，而Raft会结合到一起进行判断 （？？？）</p><p>但是关于log的最新值问题之前已经讨论过，raft中可能会有<img src="/img/logSequenceProblem.png" srcset="/img/loading.gif" alt="这种情况"></p><p>(a)中S1是leader，且部分复制到log entry到其他机子，暂时只复制到S2;</p><p>(b)S1崩溃，S5在接收到 S3，S4以及自己的vote 选举成功，进入新的term，并接受了一个<strong>不同的entry(index 2)</strong></p><p>©接着S5崩溃，S1重启，被选举为leader，然后继续复制状态，在这个时候term 2 的log已经被大多数的机子复制，但是未commit，这个时候接下来就可能有两种情况:</p><ul><li><p>(d1) 如果S1再次崩溃，S5会再次被选举（接到S2，S3，S4）的vote，然后用自己term3的entry覆盖所有的entry（即使在©的情况下，S1的log entry已经被大部分接收，但未commit，也可能会被覆盖）</p></li><li><p>(d2) 然而如果S1在崩溃前复制自己term的entry到一半以上的机子，这时候根据规则就会commit(因为S5是不会赢得选举);这个就是理想的状态,为了解决(d1)覆盖问题，raft <strong>不允许</strong> commit 前一个term的log entry;</p></li></ul><p>选举出来的leader可能不包含已经提交过的log，raft针对这个做出了一个限制：</p><ul><li>log的提交不能直接提交之前term的已经过半的entry，意思是如果遇到这些term过半的entry就视为未提交的其原因就是，raft选举永远是单向的</li></ul><blockquote><blockquote><p>This means that log entries only flow in one direction, from leaders to followers, and leaders never overwrite existing entries in their logs</p></blockquote></blockquote><p>解决这个问题，都很明显，双向确认即可，Zookeeper即是采用该手段，每次选举完leader后，都会更新follower的log，所以每一轮大家的数据都会保持一致(???具体代码)</p><h5 id="投票次数">投票次数</h5><p>Raft每个server每一轮只投一次，一般来讲哪些candidate先RequestVote就会先获得投票，这样的结果是可能会造成所有candidate都没有收到过半的票然后重试；raft解决这个方法就是随机 + - 时间来发起投票;</p><p>Zookeeper每个server在某个electionEpoch内，可以投多次票，只要遇到更大的票就更新，然后再分发新的票给所有server，这样不会造成raft投票不成功的现象，同时也可以选出含有log更多的server(???)，但是明显时间就会花费更多;</p><h5 id="重启server加入集群中">重启server加入集群中</h5><p>Raft启动后，会收到leader的appendEntries；Zookeeper的server启动后，会向所有server发送投票通知，这个时候收到处在LOOKING，FOLLOWING的server的投票，则该server放弃自己的投票（？？？？）</p><h2 id="缺点">缺点</h2><ul><li><p>zk 在选举的过程中的为了保证最终一致性是不能写入和读取的，也就是说 zk 在选举过程中是不可用的，所以从 cap 理论来说 zk 属于 cp,而注册中心往往的要求的是 ap;</p></li><li><p>同时，zk 写入只能在 leader 节点操作，当有大量服务提供者同时启动时，有可能造成 leader 节点负载过大，从而死机，然后重新选举，死机，形成恶性循环，所以 zk 并不太适合作为注册中心;</p></li></ul><p>.nethttp://jiurl.nease</p>]]></content>
    
    
    
    <tags>
      
      <tag>Distributed</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Notes about MySQL Log</title>
    <link href="/2020/07/03/MySQL/log/"/>
    <url>/2020/07/03/MySQL/log/</url>
    
    <content type="html"><![CDATA[<h2 id="几种常用的log">几种常用的log</h2><h3 id="binlog">binlog</h3><p>归档日志（二进制日志）在MySql Server上进行(因为主从复制mysql支持多种引擎，所以和redo log分开)</p><ul><li><p>作用：用于复制，在主从复制中，从库利用主库上的binlog进行重播，实现主从同步，用于数据库的基于时间点的还原。</p></li><li><p>内容：逻辑格式的日志，可以简单认为就是执行过的事务中的sql语句；但又不完全是sql语句这么简单，而是包括了执行的sql语句（增删改）反向的信息，也就意味着delete对应着delete本身和其反向的insert；update对应着update执行前后的版本的信息；insert对应着delete和insert本身的信息。</p></li></ul><p>binlog 有三种模式：Statement（基于 SQL 语句的复制）、Row（基于行的复制） 以及 Mixed（混合模式）</p><h3 id="redo-log">redo log</h3><p>重做日志， 在MySQL引擎上，即我们常说的<code>WAL</code>设计</p><ul><li><p>作用：确保事务的持久性。防止在发生故障的时间点，尚有脏页未写入磁盘，在重启mysql服务的时候，根据redo log进行重做，从而达到事务的持久性这一特性；</p></li><li><p>内容：物理格式的日志，记录的是物理数据页面的修改的信息，其redo log是顺序写入redo log file的物理文件中去的；</p></li></ul><h4 id="设计">设计</h4><p>是一个环状链表</p><h3 id="undo-log">undo log</h3><p>回滚日志</p><ul><li><p>作用：保存了事务发生之前的数据的一个版本，可以用于回滚，同时可以提供多版本并发控制下的读（MVCC），也即非锁定读</p></li><li><p>内容：逻辑格式的日志，在执行undo的时候，仅仅是将数据从逻辑上恢复至事务之前的状态，而不是从物理页面上操作实现的，这一点是不同于redo log的；</p></li></ul>]]></content>
    
    
    
    <tags>
      
      <tag>MySQL</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang type</title>
    <link href="/2020/06/24/Go/type/"/>
    <url>/2020/06/24/Go/type/</url>
    
    <content type="html"><![CDATA[<p>//todo</p><a id="more"></a><p>类型</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// a copy of runtime.typeAlg</span><span class="hljs-keyword">type</span> typeAlg <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// function for hashing objects of this type</span>    <span class="hljs-comment">// (ptr to object, seed) -&gt; hash</span>    <span class="hljs-comment">//用作识别</span>hash <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(unsafe.Pointer, <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-title">uintptr</span>// <span class="hljs-title">function</span> <span class="hljs-title">for</span> <span class="hljs-title">comparing</span> <span class="hljs-title">objects</span> <span class="hljs-title">of</span> <span class="hljs-title">this</span> <span class="hljs-title">type</span>    // <span class="hljs-params">(ptr to object A, ptr to object B)</span> -&gt; ==?    //用作表明该类型是不是可比较的<span class="hljs-params">(<span class="hljs-keyword">string</span>就是不可比较的,    //context.WithValue传入的key就是要求要是可比较的)</span><span class="hljs-title">equal</span> <span class="hljs-title">func</span><span class="hljs-params">(unsafe.Pointer, unsafe.Pointer)</span> <span class="hljs-title">bool</span>&#125;</span></code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang指针</title>
    <link href="/2020/05/03/Go/Pointer/"/>
    <url>/2020/05/03/Go/Pointer/</url>
    
    <content type="html"><![CDATA[<p>比较容易混淆三种值:</p><a id="more"></a><h1>几个奇怪的值</h1><p>指针值(Pointer value):比如stirng,int这种其实是一种指针值</p><p>指针(Pointer): *T,跟c一样，长这个样子的就是指针</p><p>uintptr:一个足够大的整数；</p><div class="hljs"><pre><code class="hljs go"> <span class="hljs-comment">// Pointer represents a pointer to an arbitrary type. There are four special operations</span><span class="hljs-comment">// available for type Pointer that are not available for other types:</span><span class="hljs-comment">//- A pointer value of any type can be converted to a Pointer.</span><span class="hljs-comment">//gc的时候会有write barrier</span>unsafe.Pointer(&amp;a)<span class="hljs-comment">//- A Pointer can be converted to a pointer value of any type.</span>(*<span class="hljs-keyword">string</span>)(unsafe.Pointer(somePointer))<span class="hljs-comment">//- A uintptr can be converted to a Pointer.</span>unsafe.Pointer(<span class="hljs-keyword">uintptr</span>(somePointer)+unsafe.sizeof(&amp;a))<span class="hljs-comment">//- A Pointer can be converted to a uintptr.</span><span class="hljs-keyword">uintptr</span>(unsafe.Pointer(&amp;a))</code></pre></div><blockquote><blockquote><p>任何类型的指针可以和unsafe.Pointer相互转换;uintptr类型和unsafe.Pointer可以相互转换;</p></blockquote></blockquote><h2 id="一些cases">一些cases</h2><p>因为指针可以随意读取内存而不经过go自己的type系统，要谨慎使用，注释里面有写到不可用作临时变量，如下：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//错误！</span>tmp := <span class="hljs-keyword">uintptr</span>(unsafe.Pointer(&amp;x)) + unsafe.Offsetof(x.b)pb := (*<span class="hljs-keyword">int16</span>)(unsafe.Pointer(tmp))*pb = <span class="hljs-number">42</span></code></pre></div><p>这里主要跟gc的实现有关系，有时候垃圾回收器会移动（整理）一些变量来降低碎片度；变量被移动后当然其地址也会被改变，这个指针就要指向新地址。</p><p>而上面这段代码中<code>unsafe.Pointer</code>就是一个<strong>指向变量的指针</strong>，被移动后就要改变，但是tmp实际在这段代码中却是作为一个<strong>普通的整数</strong>，值不应该被改变!</p><p>而针对<code>uintptr</code>，因为垃圾回收器<strong>识别不出</strong>，tmp其实是一个<strong>指向x的指针</strong>，当到第二行时，可能x地址已经被转移，tmp此时就不会是这个地址了，第三行可能向一个不知道什么地址赋值，肯定会出错。</p><p>还有一些:</p><div class="hljs"><pre><code class="hljs go">pT := <span class="hljs-keyword">uintptr</span>(unsafe.Pointer(<span class="hljs-built_in">new</span>(T))) <span class="hljs-comment">// 提示: 错误!</span></code></pre></div><p>这里<code>new(T)</code>创建后并没有<strong>指针引用</strong>，这行跑完后获得的<code>uintptr</code>无指针的语义，垃圾回收器可能会立即回收掉内存空间，所以pT得到的就是无效的地址</p><p>比较常见的</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">Bytes2String</span><span class="hljs-params">(b []<span class="hljs-keyword">byte</span>)</span> <span class="hljs-title">string</span></span>&#123;    <span class="hljs-keyword">return</span> *(*<span class="hljs-keyword">string</span>)(unsafe.Pointer(&amp;b))&#125;</code></pre></div><p>PS: 这里<code>string</code>和<code>bytes</code>的转换有个小坑,如下其转换的原理是将一个<code>byteHeader</code>的头部的指针指到<code>stringHeader</code>的指针,所以这个指针指向的内容仍然是有<code>string</code>的不可改变特性;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">String2Bytes</span><span class="hljs-params">(s <span class="hljs-keyword">string</span>)</span> []<span class="hljs-title">byte</span></span> &#123;x := (*[<span class="hljs-number">2</span>]<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(&amp;s))h := [<span class="hljs-number">3</span>]<span class="hljs-keyword">uintptr</span>&#123;x[<span class="hljs-number">0</span>], x[<span class="hljs-number">1</span>], x[<span class="hljs-number">1</span>]&#125;<span class="hljs-keyword">return</span> *(*[]<span class="hljs-keyword">byte</span>)(unsafe.Pointer(&amp;h))&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    a:=<span class="hljs-string">"abc"</span>    b:=String2Bytes(a)    b[<span class="hljs-number">0</span>]=<span class="hljs-string">'z'</span>&#125;<span class="hljs-comment">//得到结果</span><span class="hljs-comment">//unexpected fault address 0x90820d</span><span class="hljs-comment">//fatal error: fault</span><span class="hljs-comment">//[signal SIGSEGV: segmentation violation code=0x2 addr=0x90820d pc=0x81ced4]</span></code></pre></div><h2 id="uinputr">uinputr</h2><p><code>uintptr</code>是一个整数，但是它没有指针的语义，即使它有保存某个对象的地址，但是垃圾回收器在这个对象转移的时候也不会更新uintptr的值，除非重新声明这个对象（很自然就不会有写屏障)，这里上面几个cases都有体现</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// A uintptr is an integer, not a reference.</span><span class="hljs-comment">// Converting a Pointer to a uintptr creates an integer value</span><span class="hljs-comment">// with no pointer semantics.</span><span class="hljs-comment">// Even if a uintptr holds the address of some object,</span><span class="hljs-comment">// the garbage collector will not update that uintptr's value</span><span class="hljs-comment">// if the object moves, nor will that uintptr keep the object</span><span class="hljs-comment">// from being reclaimed.</span><span class="hljs-comment">//</span></code></pre></div><h2 id="pointer">Pointer</h2><ul><li><p>不能做运算</p></li><li><p>不同类型的指针不能进行比较</p></li><li><p>不同类型的指针不能互相转换</p></li></ul>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>MySQL InnoDB Ｂ+tree</title>
    <link href="/2020/04/03/MySQL/btree/"/>
    <url>/2020/04/03/MySQL/btree/</url>
    
    <content type="html"><![CDATA[<p>我们都知道计算机在存储数据的时候，有最小存储单元;</p><p>在计算机中磁盘(机械硬盘)存储数据最小单元是<strong>扇区</strong>，一个扇区的大小是512字节，而文件系统（例如XFS/EXT4）他的最小单元是块，一个块的大小是4k，而对于我们的InnoDB存储引擎也有自己的最小储存单元——页（Page），一个页的大小是16K。</p>]]></content>
    
    
    
    <tags>
      
      <tag>MySQL</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang Reflection</title>
    <link href="/2020/03/26/Go/Reflect/"/>
    <url>/2020/03/26/Go/Reflect/</url>
    
    <content type="html"><![CDATA[<h1>Reflect Package</h1><p>很多类型在<code>runtime</code>里是未导出的，比如<code>_type</code>,<code>typeAlg</code>,<code>interfaceType</code>,<code>eface</code>,<code>iface</code>等等，reflect包就是用于对应导出这些类型;所以他们的结果在两个包中其实是一样的;</p><a id="more"></a><p>空接口:reflect包下:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// emptyInterface is the header for an interface&#123;&#125; value.</span><span class="hljs-keyword">type</span> emptyInterface <span class="hljs-keyword">struct</span> &#123;typ  *rtype<span class="hljs-comment">//指的是动态类型</span>word unsafe.Pointer&#125;<span class="hljs-comment">// rtype is the common implementation of most values.</span><span class="hljs-comment">// It is embedded in other struct types.</span><span class="hljs-comment">//</span><span class="hljs-comment">// rtype must be kept in sync with ../runtime/type.go:/^type._type.</span><span class="hljs-keyword">type</span> rtype <span class="hljs-keyword">struct</span> &#123;size       <span class="hljs-keyword">uintptr</span>ptrdata    <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// number of bytes in the type that can contain pointers</span>hash       <span class="hljs-keyword">uint32</span>  <span class="hljs-comment">// hash of type; avoids computation in hash tables</span>tflag      tflag   <span class="hljs-comment">// extra type information flags</span>align      <span class="hljs-keyword">uint8</span>   <span class="hljs-comment">// alignment of variable with this type</span>fieldAlign <span class="hljs-keyword">uint8</span>   <span class="hljs-comment">// alignment of struct field with this type</span>kind       <span class="hljs-keyword">uint8</span>   <span class="hljs-comment">// enumeration for C</span><span class="hljs-comment">// function for comparing objects of this type</span><span class="hljs-comment">// (ptr to object A, ptr to object B) -&gt; ==?</span>equal     <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(unsafe.Pointer, unsafe.Pointer)</span> <span class="hljs-title">bool</span><span class="hljs-title">gcdata</span>    *<span class="hljs-title">byte</span>   // <span class="hljs-title">garbage</span> <span class="hljs-title">collection</span> <span class="hljs-title">data</span><span class="hljs-title">str</span>       <span class="hljs-title">nameOff</span> // <span class="hljs-title">string</span> <span class="hljs-title">form</span><span class="hljs-title">ptrToThis</span> <span class="hljs-title">typeOff</span> // <span class="hljs-title">type</span> <span class="hljs-title">for</span> <span class="hljs-title">pointer</span> <span class="hljs-title">to</span> <span class="hljs-title">this</span> <span class="hljs-title">type</span>, <span class="hljs-title">may</span> <span class="hljs-title">be</span> <span class="hljs-title">zero</span>&#125;</span></code></pre></div><p>runtime包下:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> eface <span class="hljs-keyword">struct</span> &#123;_type *_typedata  unsafe.Pointer&#125;</code></pre></div><p>非空接口:reflect包下:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// nonEmptyInterface is the header for an interface value with methods.</span><span class="hljs-keyword">type</span> nonEmptyInterface <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// see ../runtime/iface.go:/Itab</span>itab *<span class="hljs-keyword">struct</span> &#123;ityp *rtype <span class="hljs-comment">// static interface type</span>typ  *rtype <span class="hljs-comment">// dynamic concrete type</span>hash <span class="hljs-keyword">uint32</span> <span class="hljs-comment">// copy of typ.hash</span>_    [<span class="hljs-number">4</span>]<span class="hljs-keyword">byte</span>fun  [<span class="hljs-number">100000</span>]unsafe.Pointer <span class="hljs-comment">// method table</span>&#125;word unsafe.Pointer&#125;</code></pre></div><p>runtime包下:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> iface <span class="hljs-keyword">struct</span> &#123;tab  *itabdata unsafe.Pointer&#125;<span class="hljs-keyword">type</span> itab <span class="hljs-keyword">struct</span> &#123;inter *interfacetype_type *_typehash  <span class="hljs-keyword">uint32</span> <span class="hljs-comment">// copy of _type.hash. Used for type switches.</span>_     [<span class="hljs-number">4</span>]<span class="hljs-keyword">byte</span>fun   [<span class="hljs-number">1</span>]<span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// variable sized. fun[0]==0 means _type does not implement inter.</span>&#125;</code></pre></div><h3 id="1-reflect-type">1. reflect.Type</h3><p><code>TypeOf</code>方法:</p><ul><li>将控接口i变成<code>emptyInterface</code>类型,然后将其赋值给eface</li></ul><p>注意<code>eface</code>在runtime包中</p><div class="hljs"><pre><code class="hljs go"></code></pre></div><p><code>reflect</code>包中<code>emptyInterface</code>和<code>runtime.eface</code>是一样的</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// TypeOf returns the reflection Type that represents the dynamic type of i.</span><span class="hljs-comment">// If i is a nil interface value, TypeOf returns nil.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">TypeOf</span><span class="hljs-params">(i <span class="hljs-keyword">interface</span>&#123;&#125;)</span> <span class="hljs-title">Type</span></span> &#123;eface := *(*emptyInterface)(unsafe.Pointer(&amp;i))<span class="hljs-keyword">return</span> toType(eface.typ)&#125;</code></pre></div><p><code>reflect.TypeOf</code>方法传入的值要是体现地址特性，我们一般直接使用传入变量（变量一般指的是copy值，但是编译器在使用到该方法时会有一个<strong>隐式的复制值</strong>，传入到<code>TypeOf</code>上）</p><div class="hljs"><pre><code class="hljs go">Package tt<span class="hljs-keyword">type</span> T1 <span class="hljs-keyword">struct</span>&#123;    Name <span class="hljs-keyword">string</span>&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(t T1)</span> <span class="hljs-title">A</span><span class="hljs-params">()</span></span>&#123;    <span class="hljs-built_in">println</span>(<span class="hljs-string">"A"</span>)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(t T1)</span> <span class="hljs-title">B</span><span class="hljs-params">()</span></span>&#123;    <span class="hljs-built_in">println</span>(<span class="hljs-string">"B"</span>)&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    a:=tt.T1&#123;Name:<span class="hljs-string">"mhh"</span>&#125;    <span class="hljs-comment">//这里使用</span>    t:=reflect.TypeOf(a)    <span class="hljs-built_in">println</span>(t.Name(),t.NumMethod())&#125;</code></pre></div><h3 id="2-reflect-value">2. reflect.Value</h3><p>上面主要是用反射读取信息，</p><p>使用反射修改信息用到<code>reflect.Value</code>:</p><ul><li><code>typ</code>用作存储 反射变量的类型元数据 指针</li><li><code>ptr</code>存储数据的地址</li><li><code>flag</code>存储反射值的一些描述信息(是不是指针，是不是method，只读等等)</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> Value <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// typ holds the type of the value represented by a Value.</span>typ *rtype<span class="hljs-comment">// Pointer-valued data or, if flagIndir is set, pointer to data.</span><span class="hljs-comment">// Valid when either flagIndir is set or typ.pointers() is true.</span>ptr unsafe.Pointer<span class="hljs-comment">// flag holds metadata about the value.</span><span class="hljs-comment">// The lowest bits are flag bits:</span><span class="hljs-comment">//- flagStickyRO: obtained via unexported not embedded field, so read-only</span><span class="hljs-comment">//- flagEmbedRO: obtained via unexported embedded field, so read-only</span><span class="hljs-comment">//- flagIndir: val holds a pointer to the data</span><span class="hljs-comment">//- flagAddr: v.CanAddr is true (implies flagIndir)</span><span class="hljs-comment">//- flagMethod: v is a method value.</span><span class="hljs-comment">// The next five bits give the Kind of the value.</span><span class="hljs-comment">// This repeats typ.Kind() except for method values.</span><span class="hljs-comment">// The remaining 23+ bits give a method number for method values.</span><span class="hljs-comment">// If flag.kind() != Func, code can assume that flagMethod is unset.</span><span class="hljs-comment">// If ifaceIndir(typ), code can assume that flagIndir is set.</span>flag<span class="hljs-comment">// A method value represents a curried method invocation</span><span class="hljs-comment">// like r.Read for some receiver r. The typ+val+flag bits describe</span><span class="hljs-comment">// the receiver r, but the flag's Kind bits say Func (methods are</span><span class="hljs-comment">// functions), and the top bits of the flag give the method number</span><span class="hljs-comment">// in r's type's method table.</span>&#125;</code></pre></div><ul><li>通常通过<code>reflect.ValueOf</code>拿到这个变量的value,注意传入的是空接口(同<code>reflect.TypeOf</code>处理方法一样)</li><li>另外还会显式的<code>escapes(i)</code>将指向的变量逃逸到堆上</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// ValueOf returns a new Value initialized to the concrete value</span><span class="hljs-comment">// stored in the interface i. ValueOf(nil) returns the zero Value.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">ValueOf</span><span class="hljs-params">(i <span class="hljs-keyword">interface</span>&#123;&#125;)</span> <span class="hljs-title">Value</span></span> &#123;<span class="hljs-keyword">if</span> i == <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> Value&#123;&#125;&#125;<span class="hljs-comment">// <span class="hljs-doctag">TODO:</span> Maybe allow contents of a Value to live on the stack.</span><span class="hljs-comment">// For now we make the contents always escape to the heap. It</span><span class="hljs-comment">// makes life easier in a few places (see chanrecv/mapassign</span><span class="hljs-comment">// comment below).</span>escapes(i)<span class="hljs-keyword">return</span> unpackEface(i)&#125;</code></pre></div><p>关于逃逸:同TypeOf函数一样，编译器会隐式创建一个copy值，但是因为逃逸，这个真正的数据会在堆上创建，而保留在栈上的是这个数据的地址，但是因为这个是一个临时变量，毫无意义的修改不会影响到原来的<code>s</code>变量,所以会发生panic(use unaddressable value)</p><div class="hljs"><pre><code class="hljs go">s:=<span class="hljs-string">"test"</span>t:=reflect.ValueOf(s)t.SetString(<span class="hljs-string">"test1"</span>) <span class="hljs-comment">//panic</span><span class="hljs-built_in">println</span>(s)</code></pre></div><p>更改后：</p><ul><li>将s的地址传入</li><li>调用<code>Elem</code>方法，会将<code>t.ptr</code>指向的变量s包装成<code>reflect.Value</code>的返回值</li></ul><div class="hljs"><pre><code class="hljs go">s:=<span class="hljs-string">"test"</span>t:=reflect.ValueOf(&amp;s)t=t.Elem()t.SetString(<span class="hljs-string">"test1"</span>) <span class="hljs-comment">//panic</span><span class="hljs-built_in">println</span>(s)</code></pre></div><h3 id="3-reflect-kind">3. reflect.Kind</h3><h2 id="interface">Interface{}</h2><p>接下来说一说经常用到的 interface{} 结构体，传参的时候如果接收参数是interface{}，其实就用到隐式的反射，但是这个interface{}本身的属性也比较特殊；</p><p>我们可以先看一个问题:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> Animal <span class="hljs-keyword">interface</span>&#123;    Walk()&#125;<span class="hljs-keyword">type</span> Cat <span class="hljs-keyword">struct</span>&#123;&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *Cat)</span> <span class="hljs-title">Walk</span><span class="hljs-params">()</span></span>&#123;    log.Println(<span class="hljs-string">"walking! cat"</span>)&#125;<span class="hljs-keyword">type</span> i2 <span class="hljs-keyword">interface</span>&#123;&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    <span class="hljs-keyword">var</span> t1 i1    <span class="hljs-keyword">var</span> t2 i2    <span class="hljs-keyword">if</span> t1==t2&#123;        log.Println(<span class="hljs-string">"equal!"</span>)    &#125;<span class="hljs-keyword">else</span>&#123;        log.Println(<span class="hljs-string">"nope!"</span>)    &#125;&#125;</code></pre></div><p>除了反射其还可以当做原本的用途： 多态 （OOP）的特征之一，只是golang是ducktype类型，实现多态的几个要求：</p><ol><li>有interface接口和方法，有子类把那个接口的方法都实现了，编译器就会自动认为这个结构体就是使用了这个接口</li><li>父类指针指向子类的具体对象</li></ol><p>满足以上就可以实现<strong>多态</strong></p><p>一般有两种写法：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//方法内嵌</span><span class="hljs-keyword">type</span> doInterfaceWithMethod <span class="hljs-keyword">interface</span>&#123;    Do1(<span class="hljs-keyword">string</span>) <span class="hljs-keyword">string</span>&#125;<span class="hljs-comment">//方法放在外面</span><span class="hljs-keyword">type</span> doInterfaceWithoutMethod <span class="hljs-keyword">interface</span>&#123;&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(m *doInterfaceWithoutMethod)</span> <span class="hljs-title">Do1</span><span class="hljs-params">(s <span class="hljs-keyword">string</span>)</span> <span class="hljs-title">string</span></span>&#123;&#125;</code></pre></div><h2 id="额外">额外</h2><h3 id="深复制">深复制</h3><ol><li><p>val:=reflect.ValueOf(x)</p></li><li><p>k:=val.Kind():判断类型</p></li><li><p>k.NumField():获得fields的数目</p></li><li><p>逐一field去判断类型:</p></li></ol><h3 id="缓存itab类型">缓存itab类型</h3><p>go会缓存itab的类型为一个k-v的表(底层有一个数组排列)，需要一个itab会首先从<code>itabTable</code>里面找</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">const</span> itabInitSize = <span class="hljs-number">512</span><span class="hljs-comment">// Note: change the formula in the mallocgc call in itabAdd if you change these fields.</span><span class="hljs-keyword">type</span> itabTableType <span class="hljs-keyword">struct</span> &#123;size    <span class="hljs-keyword">uintptr</span>             <span class="hljs-comment">// length of entries array. Always a power of 2.</span>count   <span class="hljs-keyword">uintptr</span>             <span class="hljs-comment">// current number of filled entries.</span>entries [itabInitSize]*itab <span class="hljs-comment">// really [size] large</span>&#125;</code></pre></div><p>如果能够找得到，就直接使用里面的itab值，否则会生成一个新的itab,存入<code>itabTableType.entries</code>字段中</p><p>key的hash 计算:</p><ul><li>用接口类型的hash值XOR 动态类型的类型hash值</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">itabHashFunc</span><span class="hljs-params">(inter *interfacetype, typ *_type)</span> <span class="hljs-title">uintptr</span></span> &#123;<span class="hljs-comment">// compiler has provided some good hash codes for us.</span><span class="hljs-keyword">return</span> <span class="hljs-keyword">uintptr</span>(inter.typ.hash ^ typ.hash)&#125;</code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang sync.Pool</title>
    <link href="/2020/03/24/Go/syncPool/"/>
    <url>/2020/03/24/Go/syncPool/</url>
    
    <content type="html"><![CDATA[<p>golang进程池</p><a id="more"></a><h1>Sync.Pool</h1><p>这个是1.13后的大改进，大幅度削减开销;首先明确一个<strong>目标</strong>就是：</p><p>池类技术都是为了减少资源的多次分配，在这里就是<strong>减少GC</strong>的压力，以及提高缓存命中率</p><h2 id="结构">结构</h2><p>相关内容主要在 sync/pool.go 和 sync/poolqueue.go上</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> Pool <span class="hljs-keyword">struct</span> &#123;noCopy noCopy <span class="hljs-comment">//这个是一个保证了第一次使用不会被copy的结构,防止被复制，很多结构也有用到这个，比如syncgroup等</span><span class="hljs-comment">//本地per-P的固定大小的pool，具体结构其实是[P]poolLocal</span>local     unsafe.Pointer <span class="hljs-comment">// local fixed-size per-P pool, actual type is [P]poolLocal</span>localSize <span class="hljs-keyword">uintptr</span>        <span class="hljs-comment">// size of the local array</span>  <span class="hljs-comment">//1.13的重要更新！！！这个东西实际上在gc用到（会将其保存下来避免gc），后面再说</span>    victim     unsafe.Pointer <span class="hljs-comment">// local from previous cycle</span>victimSize <span class="hljs-keyword">uintptr</span>        <span class="hljs-comment">// size of victims array</span><span class="hljs-comment">// New optionally specifies a function to generate</span><span class="hljs-comment">// a value when Get would otherwise return nil.</span><span class="hljs-comment">// It may not be changed concurrently with calls to Get.</span>New <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span> <span class="hljs-title">interface</span></span>&#123;&#125;&#125;<span class="hljs-keyword">type</span> poolLocal <span class="hljs-keyword">struct</span> &#123;poolLocalInternal<span class="hljs-comment">// Prevents false sharing on widespread platforms with</span><span class="hljs-comment">// 128 mod (cache line size) = 0 .</span>pad [<span class="hljs-number">128</span> - unsafe.Sizeof(poolLocalInternal&#123;&#125;)%<span class="hljs-number">128</span>]<span class="hljs-keyword">byte</span>&#125;<span class="hljs-comment">// Local per-P Pool appendix.</span><span class="hljs-keyword">type</span> poolLocalInternal <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">//不为空可以复用，如果为空则要从shared队列拿对象</span>private <span class="hljs-keyword">interface</span>&#123;&#125; <span class="hljs-comment">// Can be used only by the respective P.</span>shared  poolChain   <span class="hljs-comment">// Local P can pushHead/popHead; any P can popTail.</span>&#125;</code></pre></div><p>容易想到池类技术多用队列来实现;</p><p>但是这里使用了让人感叹“喵啊喵啊”的结构：<strong>环式队列</strong></p><h3 id="pooldequeue">poolDequeue</h3><h4 id="基本结构">基本结构</h4><p>poolDequeue是一个无锁，固定大小，单生产者，多消费者的一个环形队列，生产者可以在head或者tail加上元素，但是消费者只能在tail消费</p><p>其特点就是会将<strong>不使用的slots设为nil</strong>，这一点咋一看不是觉得应该这样做嘛，但细节上还是有点复杂,下面先看一下基本结构</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> poolDequeue <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// headTail packs together a 32-bit head index and a 32-bit</span><span class="hljs-comment">// tail index. Both are indexes into vals modulo len(vals)-1.</span><span class="hljs-comment">//</span><span class="hljs-comment">// tail = index of oldest data in queue</span><span class="hljs-comment">// head = index of next slot to fill</span><span class="hljs-comment">//头部，生产者放入data</span><span class="hljs-comment">// Slots in the range [tail, head) are owned by consumers.</span><span class="hljs-comment">//消费者只消费tail</span><span class="hljs-comment">// A consumer continues to own a slot outside this range until</span><span class="hljs-comment">// it nils the slot, at which point ownership passes to the</span><span class="hljs-comment">// producer.</span><span class="hljs-comment">//</span><span class="hljs-comment">// The head index is stored in the most-significant bits so</span><span class="hljs-comment">// that we can atomically add to it and the overflow is</span><span class="hljs-comment">// harmless.</span>headTail <span class="hljs-keyword">uint64</span><span class="hljs-comment">// vals is a ring buffer of interface&#123;&#125; values stored in this</span><span class="hljs-comment">// dequeue. The size of this must be a power of 2.</span><span class="hljs-comment">//</span><span class="hljs-comment">//vals就是这个环形buffer，长度是2的幂次</span><span class="hljs-comment">// vals[i].typ is nil if the slot is empty and non-nil</span><span class="hljs-comment">// otherwise. A slot is still in use until *both* the tail</span><span class="hljs-comment">// index has moved beyond it and typ has been set to nil. This</span><span class="hljs-comment">// is set to nil atomically by the consumer and read</span><span class="hljs-comment">// atomically by the producer.</span>vals []eface&#125; <span class="hljs-comment">// 类似于没有方法 interface&#123;&#125;</span><span class="hljs-keyword">type</span> eface <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">//这个slot是空的话，typ将会=nil；</span><span class="hljs-comment">//而且每次读写改变slot状态都会是原子性操作</span>typ, val unsafe.Pointer&#125;<span class="hljs-comment">//还有一些定义</span><span class="hljs-comment">// dequeueLimit is the maximum size of a poolDequeue.</span><span class="hljs-comment">//</span><span class="hljs-comment">// This must be at most (1&lt;&lt;dequeueBits)/2 because detecting fullness</span><span class="hljs-comment">// depends on wrapping around the ring buffer without wrapping around</span><span class="hljs-comment">// the index. We divide by 4 so this fits in an int on 32-bit.</span><span class="hljs-keyword">const</span> dequeueLimit = (<span class="hljs-number">1</span> &lt;&lt; dequeueBits) / <span class="hljs-number">4</span></code></pre></div><p>先回答一下之前的问题：</p><ul><li>vals[i].typ如果是nil = 该slot为空否则一定为空(必要条件)</li><li>判断slot是否还在被使用要结合index(已经移到前面即为空)和vals[i].typ是否为空来决定</li><li>消费者设置其为nil以及生产者读取都是<strong>原子操作</strong></li></ul><p>还有可以看到上面规定了dequeue的最大limit，为什么呢？上面的解释是 检测是否队列满取决于该ring buffer而不是其index，理解&gt;???</p><h4 id="pushhead">pushHead</h4><p>队列，由 <strong>单个</strong> 生产者推入head，如果满了就返回false;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">const</span> dequeueBits = <span class="hljs-number">32</span><span class="hljs-comment">// pushHead adds val at the head of the queue. It returns false if the</span><span class="hljs-comment">// queue is full. It must only be called by a single producer.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(d *poolDequeue)</span> <span class="hljs-title">pushHead</span><span class="hljs-params">(val <span class="hljs-keyword">interface</span>&#123;&#125;)</span> <span class="hljs-title">bool</span></span> &#123;ptrs := atomic.LoadUint64(&amp;d.headTail)<span class="hljs-comment">//根据headTail计算出真正的head和tail</span>head, tail := d.unpack(ptrs) <span class="hljs-comment">//这里的dequeueBits</span><span class="hljs-comment">//const dequeueBits = 32</span><span class="hljs-comment">//这里判断???  </span><span class="hljs-keyword">if</span> (tail+<span class="hljs-keyword">uint32</span>(<span class="hljs-built_in">len</span>(d.vals)))&amp;(<span class="hljs-number">1</span>&lt;&lt;dequeueBits<span class="hljs-number">-1</span>) == head &#123;<span class="hljs-comment">// Queue is full.</span><span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;slot := &amp;d.vals[head&amp;<span class="hljs-keyword">uint32</span>(<span class="hljs-built_in">len</span>(d.vals)<span class="hljs-number">-1</span>)]<span class="hljs-comment">// Check if the head slot has been released by popTail.</span>typ := atomic.LoadPointer(&amp;slot.typ)<span class="hljs-keyword">if</span> typ != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// Another goroutine is still cleaning up the tail, so</span><span class="hljs-comment">// the queue is actually still full.</span><span class="hljs-comment">//其他goroutine正在清除(consume)tail</span><span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;<span class="hljs-comment">// The head slot is free, so we own it.</span><span class="hljs-keyword">if</span> val == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">//这里实际上是*struct&#123;&#125;类型，代表interface&#123;&#125;(nil)，因为我们使用nil来代表空的slot，所以要一种sentinel value (可以理解为标记值) 来代表nil</span>val = dequeueNil(<span class="hljs-literal">nil</span>)&#125;<span class="hljs-comment">//slot是eface类型，slot转为interface&#123;&#125;，val就可以直接赋值给slot，又因为eface是interface&#123;&#125;其中一种实现，slot.typ和slot.val则不为空, 这里其实也是之前判断是否满队列的原因</span>*(*<span class="hljs-keyword">interface</span>&#123;&#125;)(unsafe.Pointer(slot)) = val<span class="hljs-comment">// Increment head. This passes ownership of slot to popTail</span><span class="hljs-comment">// and acts as a store barrier for writing the slot.</span><span class="hljs-comment">//插入后head +1 </span>atomic.AddUint64(&amp;d.headTail, <span class="hljs-number">1</span>&lt;&lt;dequeueBits)<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>&#125;<span class="hljs-comment">//计算head和tail的index</span><span class="hljs-comment">//实际前32位是head，后32位是tail</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(d *poolDequeue)</span> <span class="hljs-title">unpack</span><span class="hljs-params">(ptrs <span class="hljs-keyword">uint64</span>)</span> <span class="hljs-params">(head, tail <span class="hljs-keyword">uint32</span>)</span></span> &#123;<span class="hljs-comment">//dequeueBits = 32</span><span class="hljs-keyword">const</span> mask = <span class="hljs-number">1</span>&lt;&lt;dequeueBits - <span class="hljs-number">1</span>head = <span class="hljs-keyword">uint32</span>((ptrs &gt;&gt; dequeueBits) &amp; mask)tail = <span class="hljs-keyword">uint32</span>(ptrs &amp; mask)<span class="hljs-keyword">return</span>&#125;</code></pre></div><h4 id="poptail">PopTail</h4><p>这个就是消费者(<strong>多个</strong>)所用的，pop出队列尾的元素</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(d *poolDequeue)</span> <span class="hljs-title">popTail</span><span class="hljs-params">()</span> <span class="hljs-params">(<span class="hljs-keyword">interface</span>&#123;&#125;, <span class="hljs-keyword">bool</span>)</span></span> &#123;<span class="hljs-keyword">var</span> slot *eface<span class="hljs-keyword">for</span> &#123;ptrs := atomic.LoadUint64(&amp;d.headTail)head, tail := d.unpack(ptrs)<span class="hljs-comment">//同样，先判断是否为空</span><span class="hljs-keyword">if</span> tail == head &#123;<span class="hljs-comment">// Queue is empty.</span><span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, <span class="hljs-literal">false</span>&#125;<span class="hljs-comment">// Confirm head and tail (for our speculative check</span><span class="hljs-comment">// above) and increment tail. If this succeeds, then</span><span class="hljs-comment">// we own the slot at tail.</span>ptrs2 := d.pack(head, tail+<span class="hljs-number">1</span>)<span class="hljs-keyword">if</span> atomic.CompareAndSwapUint64(&amp;d.headTail, ptrs, ptrs2) &#123;<span class="hljs-comment">// Success.</span>slot = &amp;d.vals[tail&amp;<span class="hljs-keyword">uint32</span>(<span class="hljs-built_in">len</span>(d.vals)<span class="hljs-number">-1</span>)]<span class="hljs-keyword">break</span>&#125;&#125;<span class="hljs-comment">// We now own slot.</span>val := *(*<span class="hljs-keyword">interface</span>&#123;&#125;)(unsafe.Pointer(slot))<span class="hljs-keyword">if</span> val == dequeueNil(<span class="hljs-literal">nil</span>) &#123;val = <span class="hljs-literal">nil</span>&#125;<span class="hljs-comment">// Tell pushHead that we're done with this slot. Zeroing the</span><span class="hljs-comment">// slot is also important so we don't leave behind references</span><span class="hljs-comment">// that could keep this object live longer than necessary.</span><span class="hljs-comment">//</span><span class="hljs-comment">// We write to val first and then publish that we're done with</span><span class="hljs-comment">// this slot by atomically writing to typ.</span><span class="hljs-comment">//将当前的slot设为空</span>slot.val = <span class="hljs-literal">nil</span>atomic.StorePointer(&amp;slot.typ, <span class="hljs-literal">nil</span>)<span class="hljs-comment">// At this point pushHead owns the slot.</span><span class="hljs-keyword">return</span> val, <span class="hljs-literal">true</span>&#125;</code></pre></div><h4 id="pophead">PopHead</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// popHead removes and returns the element at the head of the queue.</span><span class="hljs-comment">// It returns false if the queue is empty. It must only be called by a</span><span class="hljs-comment">// single producer.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(d *poolDequeue)</span> <span class="hljs-title">popHead</span><span class="hljs-params">()</span> <span class="hljs-params">(<span class="hljs-keyword">interface</span>&#123;&#125;, <span class="hljs-keyword">bool</span>)</span></span> &#123;<span class="hljs-keyword">var</span> slot *eface<span class="hljs-keyword">for</span> &#123;ptrs := atomic.LoadUint64(&amp;d.headTail)<span class="hljs-comment">//解析出head，tail</span>head, tail := d.unpack(ptrs)<span class="hljs-keyword">if</span> tail == head &#123;<span class="hljs-comment">// Queue is empty.</span><span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, <span class="hljs-literal">false</span>&#125;<span class="hljs-comment">// Confirm tail and decrement head. We do this before</span><span class="hljs-comment">// reading the value to take back ownership of this</span><span class="hljs-comment">// slot.</span><span class="hljs-comment">//pophead即是head--，然后再用pack计算出pophead之后的ptr2，然后用原子方法设置ptr为ptr2,放回d.headTail,并取出其slot</span>head--ptrs2 := d.pack(head, tail)<span class="hljs-keyword">if</span> atomic.CompareAndSwapUint64(&amp;d.headTail, ptrs, ptrs2) &#123;<span class="hljs-comment">//成功更新该slot，跳出循环</span><span class="hljs-comment">// We successfully took back slot.</span>slot = &amp;d.vals[head&amp;<span class="hljs-keyword">uint32</span>(<span class="hljs-built_in">len</span>(d.vals)<span class="hljs-number">-1</span>)]<span class="hljs-keyword">break</span>&#125;<span class="hljs-comment">//如果失败了，重新进行</span><span class="hljs-comment">//失败的情况可能是更新失败，</span>&#125;val := *(*<span class="hljs-keyword">interface</span>&#123;&#125;)(unsafe.Pointer(slot))<span class="hljs-keyword">if</span> val == dequeueNil(<span class="hljs-literal">nil</span>) &#123;val = <span class="hljs-literal">nil</span>&#125;<span class="hljs-comment">// Zero the slot. Unlike popTail, this isn't racing with</span><span class="hljs-comment">// pushHead, so we don't need to be careful here.</span>*slot = eface&#123;&#125;<span class="hljs-keyword">return</span> val, <span class="hljs-literal">true</span>&#125;</code></pre></div><ul><li>注意： 至于为什么要先设置headTail，再取slot，目的是可能其他P会在当前P steal对象，多个P调用本地P的popTail的时候，race现象就变严重，这样做让某个P如果拿到了，其他P就无法再拿到对应的对象（因为headtail改变了，位置不一样）</li></ul><h3 id="poolchain">PoolChain</h3><p>其中还有一个结构是配合pooldequeue实现了双线链表的poolChain可以看做poolchain实际上就是一个<strong>动态大小</strong>版本的poolDeque</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// poolChain is a dynamically-sized version of poolDequeue.</span><span class="hljs-comment">//poolchain实际上就是一个动态大小版本的poolDeque</span><span class="hljs-comment">// This is implemented as a doubly-linked list queue of poolDequeues</span><span class="hljs-comment">// where each dequeue is double the size of the previous one. Once a</span><span class="hljs-comment">// dequeue fills up, this allocates a new one and only ever pushes to</span><span class="hljs-comment">// the latest dequeue. Pops happen from the other end of the list and</span><span class="hljs-comment">// once a dequeue is exhausted, it gets removed from the list.</span><span class="hljs-comment">//这个poolchain实际就是有双倍长度的poolDequeue，当其中一个dequeue被填充数据，其会分配一个新的dequeue，且把这个填充数据放入最新的一个dequeue上；</span><span class="hljs-keyword">type</span> poolChain <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">//因为只被生产者所用(推入)，不保证顺序，所以不必要保证串行性</span><span class="hljs-comment">// head is the poolDequeue to push to. This is only accessed</span><span class="hljs-comment">// by the producer, so doesn't need to be synchronized.</span>head *poolChainElt<span class="hljs-comment">// tail is the poolDequeue to popTail from. This is accessed</span><span class="hljs-comment">// by consumers, so reads and writes must be atomic.</span>tail *poolChainElt&#125;<span class="hljs-keyword">type</span> poolChainElt <span class="hljs-keyword">struct</span> &#123;poolDequeue<span class="hljs-comment">// next and prev link to the adjacent poolChainElts in this</span><span class="hljs-comment">// poolChain.</span><span class="hljs-comment">//这里的next和prev指向相邻的poolChain元素,其中next是被生产者所写，消费者读取，只能从nil变为non-nil,prev则刚好相反</span><span class="hljs-comment">// next is written atomically by the producer and read</span><span class="hljs-comment">// atomically by the consumer. It only transitions from nil to</span><span class="hljs-comment">// non-nil.</span><span class="hljs-comment">//</span><span class="hljs-comment">// prev is written atomically by the consumer and read</span><span class="hljs-comment">// atomically by the producer. It only transitions from</span><span class="hljs-comment">// non-nil to nil.</span>next, prev *poolChainElt&#125;</code></pre></div><p>同理，同poolDequeue一样,包裹着它的poolChaint也有一样的方法：</p><h4 id="pushhead-v2">pushHead</h4><p>生产者增加元素,注意在当前ring buffer满了之后会初始化一个新的poolChainElt，其中poolDequque大小为原来的2倍</p><ul><li>该链表poolChain的初始化大小为8</li><li>每次增多一个poolDequeue是前一个的2倍，且一定是2的幂次</li><li>poolDequeue的最大的长度是2^30，再多的poolDequeue也不会变</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *poolChain)</span> <span class="hljs-title">pushHead</span><span class="hljs-params">(val <span class="hljs-keyword">interface</span>&#123;&#125;)</span></span> &#123;d := c.head<span class="hljs-keyword">if</span> d == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// Initialize the chain.</span><span class="hljs-keyword">const</span> initSize = <span class="hljs-number">8</span> <span class="hljs-comment">// Must be a power of 2</span>d = <span class="hljs-built_in">new</span>(poolChainElt)d.vals = <span class="hljs-built_in">make</span>([]eface, initSize)c.head = dstorePoolChainElt(&amp;c.tail, d)&#125;<span class="hljs-comment">//先把该val插入到pooldeqeue中</span><span class="hljs-keyword">if</span> d.pushHead(val) &#123;<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">// The current dequeue is full. Allocate a new one of twice</span><span class="hljs-comment">// the size.</span><span class="hljs-comment">//当前pooldequeue满了,则设置一个新的poolDeque,!!!</span><span class="hljs-comment">//且新大小为前一次pooldequeue的两倍</span>newSize := <span class="hljs-built_in">len</span>(d.vals) * <span class="hljs-number">2</span><span class="hljs-comment">//dequeueLimit为最大的size，</span><span class="hljs-comment">//const dequeueLimit = (1&lt;&lt;dequeueBits)/4 = 2^30</span><span class="hljs-keyword">if</span> newSize &gt;= dequeueLimit &#123;<span class="hljs-comment">// Can't make it any bigger.</span>newSize = dequeueLimit&#125;d2 := &amp;poolChainElt&#123;prev: d&#125;d2.vals = <span class="hljs-built_in">make</span>([]eface, newSize)c.head = d2<span class="hljs-comment">//实际就是将d.next指向新的一个ring buffer(poolChainElt),其结构体下面有</span>storePoolChainElt(&amp;d.next, d2)d2.pushHead(val)&#125;</code></pre></div><h4 id="poptail-v2">popTail</h4><p>消费者消费队列</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *poolChain)</span> <span class="hljs-title">popTail</span><span class="hljs-params">()</span> <span class="hljs-params">(<span class="hljs-keyword">interface</span>&#123;&#125;, <span class="hljs-keyword">bool</span>)</span></span> &#123;d := loadPoolChainElt(&amp;c.tail)<span class="hljs-keyword">if</span> d == <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, <span class="hljs-literal">false</span>&#125;<span class="hljs-keyword">for</span> &#123;<span class="hljs-comment">// It's important that we load the next pointer</span><span class="hljs-comment">// *before* popping the tail. In general, d may be</span><span class="hljs-comment">// transiently empty, but if next is non-nil before</span><span class="hljs-comment">// the pop and the pop fails, then d is permanently</span><span class="hljs-comment">// empty, which is the only condition under which it's</span><span class="hljs-comment">// safe to drop d from the chain.</span>d2 := loadPoolChainElt(&amp;d.next)<span class="hljs-keyword">if</span> val, ok := d.popTail(); ok &#123;<span class="hljs-keyword">return</span> val, ok&#125;<span class="hljs-keyword">if</span> d2 == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// This is the only dequeue. It's empty right</span><span class="hljs-comment">// now, but could be pushed to in the future.</span><span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, <span class="hljs-literal">false</span>&#125;<span class="hljs-comment">// The tail of the chain has been drained, so move on</span><span class="hljs-comment">// to the next dequeue. Try to drop it from the chain</span><span class="hljs-comment">// so the next pop doesn't have to look at the empty</span><span class="hljs-comment">// dequeue again.</span><span class="hljs-comment">//这里注意</span><span class="hljs-keyword">if</span> atomic.CompareAndSwapPointer((*unsafe.Pointer)(unsafe.Pointer(&amp;c.tail)), unsafe.Pointer(d), unsafe.Pointer(d2)) &#123;<span class="hljs-comment">// We won the race. Clear the prev pointer so</span><span class="hljs-comment">// the garbage collector can collect the empty</span><span class="hljs-comment">// dequeue and so popHead doesn't back up</span><span class="hljs-comment">// further than necessary.</span>storePoolChainElt(&amp;d2.prev, <span class="hljs-literal">nil</span>)&#125;d = d2&#125;&#125;</code></pre></div><ul><li>！！！！注意到上面有一段原子操作，主要可能有<strong>消费者是其他P</strong>的情况下， <strong>popTail</strong> 明显就与popHead以及pushHead有race</li></ul><h4 id="pophead-v2">popHead</h4><p>逻辑比较简单，一个个pooldequeue去找，找完就往前一个元素继续</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *poolChain)</span> <span class="hljs-title">popHead</span><span class="hljs-params">()</span> <span class="hljs-params">(<span class="hljs-keyword">interface</span>&#123;&#125;, <span class="hljs-keyword">bool</span>)</span></span> &#123;d := c.head<span class="hljs-keyword">for</span> d != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">//首先从pooldequeue中pophead</span><span class="hljs-keyword">if</span> val, ok := d.popHead(); ok &#123;<span class="hljs-keyword">return</span> val, ok&#125;<span class="hljs-comment">// There may still be unconsumed elements in the</span><span class="hljs-comment">// previous dequeue, so try backing up.</span><span class="hljs-comment">//pop完当前的pooldequeue则load前面的poolChainElt</span>d = loadPoolChainElt(&amp;d.prev)&#125;<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, <span class="hljs-literal">false</span>&#125;</code></pre></div><p>综合上面的各个结构，大概画了<img src="/img/syncpool.png" srcset="/img/loading.gif" alt="一个图"></p><p>看到这里可能就有点疑问了，<strong>为啥有popTail，又要有popHead呢？？？</strong>这也是其设计的 “喵啊喵啊” 之处，具体可以继续看下面的**Get()**方法</p><h3 id="由功能出发-猜结构">由功能出发，猜结构</h3><p>上面谈到的两种结构都有点印象了，下面就是真正如何使用:池类技术不用问，get，set(put)各一个，还有超过了size之后的清空</p><h4 id="put">Put</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Put adds x to the pool.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(p *Pool)</span> <span class="hljs-title">Put</span><span class="hljs-params">(x <span class="hljs-keyword">interface</span>&#123;&#125;)</span></span> &#123;<span class="hljs-keyword">if</span> x == <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span>&#125;<span class="hljs-keyword">if</span> race.Enabled &#123;<span class="hljs-keyword">if</span> fastrand()%<span class="hljs-number">4</span> == <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// Randomly drop x on floor.</span><span class="hljs-keyword">return</span>&#125;race.ReleaseMerge(poolRaceAddr(x))race.Disable()    &#125;    <span class="hljs-comment">//这里pin</span>l, _ := p.pin()<span class="hljs-keyword">if</span> l.private == <span class="hljs-literal">nil</span> &#123;l.private = xx = <span class="hljs-literal">nil</span>&#125;<span class="hljs-keyword">if</span> x != <span class="hljs-literal">nil</span> &#123;l.shared.pushHead(x)&#125;runtime_procUnpin()<span class="hljs-keyword">if</span> race.Enabled &#123;race.Enable()&#125;&#125;</code></pre></div><p>这里的 l, _ := p.pin() <strong>pin()</strong> 函数就值得深入看一下:</p><ol><li>大概意思就是，这个pin函数会pin住当前goroutine，<strong>防止抢占</strong>(可以看一下goroutines一节)</li><li>原子性操作atomic.LoadUintPtr()保证不会同步问题</li><li>返回值是本地poolLocal pool(poolChain和private)的指针，和这个p的id</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// pin pins the current goroutine to P, disables preemption and</span><span class="hljs-comment">// returns poolLocal pool for the P and the P's id.</span><span class="hljs-comment">// Caller must call runtime_procUnpin() when done with the pool.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(p *Pool)</span> <span class="hljs-title">pin</span><span class="hljs-params">()</span> <span class="hljs-params">(*poolLocal, <span class="hljs-keyword">int</span>)</span></span> &#123;pid := runtime_procPin()<span class="hljs-comment">// In pinSlow we store to local and then to localSize, here we load in opposite order.</span><span class="hljs-comment">// Since we've disabled preemption, GC cannot happen in between.</span><span class="hljs-comment">// Thus here we must observe local at least as large localSize.</span><span class="hljs-comment">// We can observe a newer/larger local, it is fine (we must observe its zero-initialized-ness).</span><span class="hljs-comment">//获取localsize，锁住</span>s := atomic.LoadUintptr(&amp;p.localSize) <span class="hljs-comment">// load-acquire</span><span class="hljs-comment">//</span>l := p.local                          <span class="hljs-comment">// load-consume</span><span class="hljs-keyword">if</span> <span class="hljs-keyword">uintptr</span>(pid) &lt; s &#123;<span class="hljs-keyword">return</span> indexLocal(l, pid), pid&#125;<span class="hljs-keyword">return</span> p.pinSlow()&#125;</code></pre></div><p>注意: 可以看一下这个<strong>runtime_procPin()</strong>，是runtime的汇编代码用来锁住调度过程(禁止抢占),这里主要是要获得当前P的id，如果被抢占可能P的id会变化;一定要配合runtime_procUnpin()解锁;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(p *Pool)</span> <span class="hljs-title">pinSlow</span><span class="hljs-params">()</span> <span class="hljs-params">(*poolLocal, <span class="hljs-keyword">int</span>)</span></span> &#123;<span class="hljs-comment">// Retry under the mutex.</span><span class="hljs-comment">// Can not lock the mutex while pinned.</span>runtime_procUnpin()allPoolsMu.Lock()<span class="hljs-keyword">defer</span> allPoolsMu.Unlock()pid := runtime_procPin()<span class="hljs-comment">// poolCleanup won't be called while we are pinned.</span>s := p.localSizel := p.local<span class="hljs-comment">//uintptr(pid)小于[]localpool的size，则一定在[]localpool里面，直接进去拿</span><span class="hljs-keyword">if</span> <span class="hljs-keyword">uintptr</span>(pid) &lt; s &#123;<span class="hljs-keyword">return</span> indexLocal(l, pid), pid&#125;<span class="hljs-keyword">if</span> p.local == <span class="hljs-literal">nil</span> &#123;allPools = <span class="hljs-built_in">append</span>(allPools, p)&#125;<span class="hljs-comment">// If GOMAXPROCS changes between GCs, we re-allocate the array and lose the old one.</span>size := runtime.GOMAXPROCS(<span class="hljs-number">0</span>)<span class="hljs-comment">//创建新的local</span>local := <span class="hljs-built_in">make</span>([]poolLocal, size)atomic.StorePointer(&amp;p.local, unsafe.Pointer(&amp;local[<span class="hljs-number">0</span>])) <span class="hljs-comment">// store-release</span>atomic.StoreUintptr(&amp;p.localSize, <span class="hljs-keyword">uintptr</span>(size))         <span class="hljs-comment">// store-release</span><span class="hljs-keyword">return</span> &amp;local[pid], pid&#125;</code></pre></div><ul><li>接下来，在p.pinSlow()还会进行一些判断,首先，在解锁了抢占然后再次调用runtime_ProcPin()为的就是获取最新的P的id;</li></ul><p><strong>目的</strong>: 个人理解是尽量减少 <strong>p.local([]poolLocal)</strong> 的创建，因为在解绑runtime_unProcPin()与下一次绑定之间可能P的id会变化，可以先检查切换的新的这个P的id里面是不是已经有 <strong>p.local([]poolLocal)</strong></p><ul><li>如果没有p.local没有对象就会创建新的一个 <strong>[]poollocal</strong> ，旧的poolocal就会进入GC</li></ul><h4 id="get">Get</h4><p>与set有一定的相似</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(p *Pool)</span> <span class="hljs-title">Get</span><span class="hljs-params">()</span> <span class="hljs-title">interface</span></span>&#123;&#125; &#123;<span class="hljs-keyword">if</span> race.Enabled &#123;race.Disable()    &#125;    <span class="hljs-comment">//与set一样，都要先"锁住"当前goroutine</span>l, pid := p.pin()<span class="hljs-comment">//获得可复用的private对象</span>x := l.privatel.private = <span class="hljs-literal">nil</span><span class="hljs-keyword">if</span> x == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">//private无复用的对象，只能从shared []poollocal拿</span><span class="hljs-comment">// Try to pop the head of the local shard. We prefer</span><span class="hljs-comment">// the head over the tail for temporal locality of</span><span class="hljs-comment">// reuse.</span><span class="hljs-comment">//首先会从本地的sharedpopHead</span>x, _ = l.shared.popHead()<span class="hljs-comment">//如果没有，就会进行getSlow()</span><span class="hljs-keyword">if</span> x == <span class="hljs-literal">nil</span> &#123;x = p.getSlow(pid)&#125;&#125;runtime_procUnpin()<span class="hljs-keyword">if</span> race.Enabled &#123;race.Enable()<span class="hljs-keyword">if</span> x != <span class="hljs-literal">nil</span> &#123;race.Acquire(poolRaceAddr(x))&#125;&#125;<span class="hljs-keyword">if</span> x == <span class="hljs-literal">nil</span> &amp;&amp; p.New != <span class="hljs-literal">nil</span> &#123;x = p.New()&#125;<span class="hljs-keyword">return</span> x&#125;</code></pre></div><p>这里就可以回答上面的问题了（<strong>为啥有popTail，又要有popHead呢？？？</strong>)：</p><ul><li>首先会从本地的sharedpopHead</li><li>如果在poollocal中找不到对象，则要调用**getSlow()**获得对象,getslow()代码如下</li><li>下面代码中 <strong>l.shared.popTail()</strong> 就发现是从其他P steal <strong>尾部</strong>获得poolChain</li></ul><p>这里都可以解释为什么有些地方不用锁:</p><blockquote><blockquote><blockquote><p>本地的就从head取对象，steal其他的P的对象就从其他P的tail取对象</p></blockquote></blockquote></blockquote><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(p *Pool)</span> <span class="hljs-title">getSlow</span><span class="hljs-params">(pid <span class="hljs-keyword">int</span>)</span> <span class="hljs-title">interface</span></span>&#123;&#125; &#123;<span class="hljs-comment">// See the comment in pin regarding ordering of the loads.</span>size := atomic.LoadUintptr(&amp;p.localSize) <span class="hljs-comment">// load-acquire</span>locals := p.local                        <span class="hljs-comment">// load-consume</span><span class="hljs-comment">// Try to steal one element from other procs.</span><span class="hljs-keyword">for</span> i := <span class="hljs-number">0</span>; i &lt; <span class="hljs-keyword">int</span>(size); i++ &#123;l := indexLocal(locals, (pid+i+<span class="hljs-number">1</span>)%<span class="hljs-keyword">int</span>(size))<span class="hljs-comment">//从其他P的tail获得对象</span><span class="hljs-keyword">if</span> x, _ := l.shared.popTail(); x != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> x&#125;&#125;<span class="hljs-comment">// Try the victim cache. We do this after attempting to steal</span><span class="hljs-comment">// from all primary caches because we want objects in the</span><span class="hljs-comment">// victim cache to age out if at all possible.</span>size = atomic.LoadUintptr(&amp;p.victimSize)<span class="hljs-keyword">if</span> <span class="hljs-keyword">uintptr</span>(pid) &gt;= size &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>&#125;locals = p.victiml := indexLocal(locals, pid)<span class="hljs-keyword">if</span> x := l.private; x != <span class="hljs-literal">nil</span> &#123;l.private = <span class="hljs-literal">nil</span><span class="hljs-keyword">return</span> x&#125;<span class="hljs-keyword">for</span> i := <span class="hljs-number">0</span>; i &lt; <span class="hljs-keyword">int</span>(size); i++ &#123;l := indexLocal(locals, (pid+i)%<span class="hljs-keyword">int</span>(size))<span class="hljs-keyword">if</span> x, _ := l.shared.popTail(); x != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> x&#125;&#125;<span class="hljs-comment">// Mark the victim cache as empty for future gets don't bother</span><span class="hljs-comment">// with it.</span>atomic.StoreUintptr(&amp;p.victimSize, <span class="hljs-number">0</span>)<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>&#125;</code></pre></div><ul><li>因为本地P无对象，会尝试从其他p中steal对象</li><li><strong>victime cache</strong> (1.13新增！！！)(其实属于计算机架构设计里面的词)代码中拿到其他p的时候，会先从victim cache中获取对象 (locals []poolChain)，然后定位slot，如果该slot的private为空则又从shared里面poptail拿到对象</li><li>最后还要注意，如果找不到对象，会将victim cache设置为空 (设置victimSize=0) ，防止下一次再次从victim里面查找</li></ul><h3 id="victimcache">VictimCache</h3><p>涉及了gc</p><p>在pool包初始化时即注册了poolCleanUp()函数,该函数用于初始化victimcache字段</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">init</span><span class="hljs-params">()</span></span> &#123;runtime_registerPoolCleanup(poolCleanup)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">poolCleanup</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-comment">// This function is called with the world stopped, at the beginning of a garbage collection.</span><span class="hljs-comment">// It must not allocate and probably should not call any runtime functions.</span><span class="hljs-comment">// Because the world is stopped, no pool user can be in a</span><span class="hljs-comment">// pinned section (in effect, this has all Ps pinned).</span><span class="hljs-comment">// Drop victim caches from all pools.</span><span class="hljs-keyword">for</span> _, p := <span class="hljs-keyword">range</span> oldPools &#123;p.victim = <span class="hljs-literal">nil</span>p.victimSize = <span class="hljs-number">0</span>&#125;<span class="hljs-comment">// Move primary cache to victim cache.</span><span class="hljs-keyword">for</span> _, p := <span class="hljs-keyword">range</span> allPools &#123;p.victim = p.localp.victimSize = p.localSizep.local = <span class="hljs-literal">nil</span>p.localSize = <span class="hljs-number">0</span>&#125;<span class="hljs-comment">// The pools with non-empty primary caches now have non-empty</span><span class="hljs-comment">// victim caches and no pools have primary caches.</span>oldPools, allPools = allPools, <span class="hljs-literal">nil</span>&#125;</code></pre></div><ul><li>该函数在stw的时候会被调用(在gc开始的时候),其不能分配也不应该调用任何runtime的函数，原因是防止？？？如果gc发生在goorutine与 shared.poolChain 进行 put/get时，会保留整个pool，下一次gc就会浪费多一倍内存</li><li>因为stw，所有pool的user不能在pinned的部分</li><li>首先会将所有当前的pools(oldPools) victim cache置为0</li><li>然后将主要的cache(allPools里面的locals([]poolLocal)字段)移到当前的victim字段</li><li>更新oldPools和allPools</li></ul><p>可以对比一下1.12的poolCleanUp</p><p><strong>1.12的poolCleanup</strong></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">poolCleanup</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-comment">// 该函数会注册到运行时 GC 阶段(前)，此时为 STW 状态，不需要加锁</span><span class="hljs-comment">// 它必须不处理分配且不调用任何运行时函数，防御性的将一切归零，有以下两点原因:</span><span class="hljs-comment">// 1. 防止整个 Pool 的 false retention???</span><span class="hljs-comment">// 2. 如果 GC 发生在当有 goroutine 与 l.shared 进行 Put/Get 时，它会保留整个 Pool.</span><span class="hljs-comment">//   那么下个 GC 周期的内存消耗将会翻倍。</span><span class="hljs-comment">// 遍历所有 Pool 实例，接触相关引用，交由 GC 进行回收</span><span class="hljs-keyword">for</span> i, p := <span class="hljs-keyword">range</span> allPools &#123;allPools[i] = <span class="hljs-literal">nil</span><span class="hljs-keyword">for</span> i := <span class="hljs-number">0</span>; i &lt; <span class="hljs-keyword">int</span>(p.localSize); i++ &#123;l := indexLocal(p.local, i)l.private = <span class="hljs-literal">nil</span><span class="hljs-keyword">for</span> j := <span class="hljs-keyword">range</span> l.shared &#123;l.shared[j] = <span class="hljs-literal">nil</span>&#125;l.shared = <span class="hljs-literal">nil</span>&#125;p.local = <span class="hljs-literal">nil</span>p.localSize = <span class="hljs-number">0</span>&#125;allPools = []*Pool&#123;&#125;&#125;</code></pre></div><ul><li>其每次gc stw都遍历allPools并清空local，private,shared，导致的结果就是时间gc消耗的时间变长，以及下一次进行分配的时候时间变长，以及下一次内存的消耗也会增多(虽然总的来讲是不变)</li></ul>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Redis basis</title>
    <link href="/2020/03/01/redis/basic/"/>
    <url>/2020/03/01/redis/basic/</url>
    
    <content type="html"><![CDATA[<h1>直接从几道常见的面试题出发</h1><a id="more"></a><p>以下相关代码都是redis5.0版本</p><h2 id="1-支持的数据类型">1. 支持的数据类型</h2><h3 id="基本数据结构">基本数据结构</h3><h4 id="1-string">1. string</h4><p>sds大概的结构如下:</p><div class="hljs"><pre><code class="hljs c"><span class="hljs-keyword">typedef</span> <span class="hljs-class"><span class="hljs-keyword">struct</span> <span class="hljs-title">sds</span>&#123;</span>    <span class="hljs-keyword">int</span> len <span class="hljs-comment">//含有数据的长度</span>    <span class="hljs-keyword">int</span> <span class="hljs-built_in">free</span><span class="hljs-comment">//空的长度</span>    byte[] arr<span class="hljs-comment">//底层数组</span>&#125;</code></pre></div><h4 id="2-dict">2. dict</h4><h4 id="3-list">3. list</h4><h4 id="4-set">4. set</h4><h4 id="5-sortedset">5. sortedSet</h4><h3 id="redisobject">redisObject</h3><h2 id="2-持久化">2. 持久化</h2><h2 id="3-redis常用命令">3. redis常用命令</h2><h2 id="4-redis内存淘汰机制">4. redis内存淘汰机制</h2><p>lru-volatilelru-allkeyslru-randomkeys</p><h2 id="5-redis持久化">5. redis持久化</h2><ul><li><p>RDB</p></li><li><p>AOF</p></li></ul><h2 id="6-redis作为队列？？？">6. redis作为队列？？？</h2><p>优点：</p><ol><li>天生数据结构,接口支持</li><li></li></ol><p>缺点：</p><ol><li>持久化</li></ol><p>注意点：</p><h2 id="7-redis架构模式">7. redis架构模式</h2><ul><li><p>Reactor</p></li><li></li></ul><h2 id="8-缓存相关">8. 缓存相关</h2><ul><li><p>缓存穿透热点数据击穿解决：锁住资源</p></li><li><p>缓存雪崩同一时间大量失效解决：random keys</p></li></ul><h2 id="9-分布式锁">9. 分布式锁</h2><h3 id="基本版本">基本版本</h3><ul><li><p>set key -n -x [time]</p></li><li></li></ul><h2 id="10-单线程支撑高并发原理">10. 单线程支撑高并发原理</h2><h2 id="11-并发竞争问题的解决">11. 并发竞争问题的解决</h2>]]></content>
    
    
    
    <tags>
      
      <tag>redis</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang Garbage Collection</title>
    <link href="/2020/02/24/Go/gc/"/>
    <url>/2020/02/24/Go/gc/</url>
    
    <content type="html"><![CDATA[<p>看了下runtime的<s>代码</s>（注释），总结一哈</p><a id="more"></a><h2 id="目的">目的</h2><p>对于所有的垃圾回收器的目的（指标）无非就是以下几个：</p><ol><li>所占用程序的时间，停顿时间</li><li>频率</li><li>CPU占比</li><li>内存占比（堆开销）</li><li>内存的分配方式（碎片化程度）</li><li>内存释放方式</li><li>并发效果</li><li>是否智能化（根据某些系统条件进行调节）</li><li>是否可以自定义化参数</li></ol><h2 id="静态语言类一般会在程序的三个阶段涉及gc的操作">静态语言类一般会在程序的三个阶段涉及gc的操作</h2><ul><li>编译期</li><li>运行时内存分配</li><li>运行时扫描</li></ul><h2 id="现在有的比较流行的gc">现在有的比较流行的GC</h2><ol><li><p>简单的refcount，比如redis，即 引用多一个，refcount就+1</p></li><li><p>mark &amp; sweep，标记然后用监视内存的程序或者lazy清理（这个一般不会），golang现在用的就是这种</p></li><li><p>比较牛批的分代收集，如JAVA，什么新生代，老年代，eden等；不过这些都是比较老的比如JAVA一类的</p></li></ol><h2 id="go的gc">Go的GC</h2><h3 id="总流程">总流程</h3><ul><li>清理终止阶段；暂停程序，所有的处理器在这时会进入安全点（Safe point）；如果当前垃圾收集循环是强制触发的，我们还需要处理还未被清理的内存管理单元；</li><li>标记阶段；将状态切换至 <code>_GCmark</code>、开启写屏障、用户程序协助（Mutator Assiste）并将根对象入队；恢复执行程序，标记进程和用于协助的用户程序会开始并发标记内存中的对象，写屏障会将被覆盖的指针和新指针都标记成灰色，而所有新创建的对象都会被直接标记成黑色；开始扫描根对象，包括所有 Goroutine 的栈、全局对象以及不在堆中的运行时数据结构，扫描 Goroutine 栈期间会暂停当前处理器；依次处理灰色队列中的对象，将对象标记成黑色并将它们指向的对象标记成灰色；使用分布式的终止算法检查剩余的工作，发现标记阶段完成后进入标记终止阶段；</li><li>标记终止阶段；暂停程序、将状态切换至 _GCmarktermination 并关闭辅助标记的用户程序；清理处理器上的线程缓存；</li><li>清理阶段；将状态切换至 _GCoff 开始清理阶段，初始化清理状态并关闭写屏障；恢复用户程序，所有新创建的对象会标记成白色；后台并发清理所有的内存管理单元，当 Goroutine 申请新的内存管理单元时就会触发清理；</li></ul><h3 id="1-编译阶段">1. 编译阶段</h3><ul><li><p>内存对齐略，这个可以参考自己的memManage</p></li><li><p>初始化一些字段我们直接</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> _type <span class="hljs-keyword">struct</span> &#123;...ptrdata    <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// size of memory prefix holding all pointers</span>...<span class="hljs-comment">// gcdata stores the GC type data for the garbage collector.</span><span class="hljs-comment">// If the KindGCProg bit is set in kind, gcdata is a GC program.</span><span class="hljs-comment">// Otherwise it is a ptrmask bitmap. See mbitmap.go for details.</span>gcdata    *<span class="hljs-keyword">byte</span>...&#125;</code></pre></div><p>举个例子</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> testStruct <span class="hljs-keyword">struct</span>&#123;ptr <span class="hljs-keyword">uintptr</span><span class="hljs-comment">//8</span>A <span class="hljs-keyword">uint8</span> <span class="hljs-comment">//1</span>B *<span class="hljs-keyword">uint8</span><span class="hljs-comment">//8</span>C <span class="hljs-keyword">uint32</span><span class="hljs-comment">//4</span>D *<span class="hljs-keyword">uint64</span><span class="hljs-comment">//8</span>E <span class="hljs-keyword">uint64</span><span class="hljs-comment">//8</span>&#125;</code></pre></div><p>我直接打点在<code>mbitmap.go:947</code>即<code>heapBitSetType</code>上面</p><div class="hljs"><pre><code class="hljs yml"><span class="hljs-string">*runtime._type</span> <span class="hljs-string">&#123;</span><span class="hljs-attr">size:</span> <span class="hljs-number">376</span><span class="hljs-string">,</span> <span class="hljs-string">//该对象有多少个字(64位一个字=64bits=8Bytes)</span><span class="hljs-attr">ptrdata:</span> <span class="hljs-number">360</span><span class="hljs-string">,</span> <span class="hljs-string">//其中的指针有多少个字</span><span class="hljs-attr">hash:</span> <span class="hljs-number">3901217204</span><span class="hljs-string">,</span> <span class="hljs-attr">tflag:</span> <span class="hljs-string">tflagUncommon|tflagExtraStar|tflagNamed</span> <span class="hljs-string">(7),</span><span class="hljs-attr">align:</span> <span class="hljs-number">8</span><span class="hljs-string">,</span> <span class="hljs-attr">fieldAlign:</span> <span class="hljs-number">8</span><span class="hljs-string">,</span> <span class="hljs-attr">kind:</span> <span class="hljs-number">25</span><span class="hljs-string">,</span><span class="hljs-attr">equal:</span> <span class="hljs-string">nil,</span> <span class="hljs-attr">gcdata:</span> <span class="hljs-string">*112,</span> <span class="hljs-string">//0111</span> <span class="hljs-number">0000</span><span class="hljs-attr">str:</span> <span class="hljs-number">11827</span><span class="hljs-string">,</span> <span class="hljs-attr">ptrToThis:</span> <span class="hljs-number">41536</span><span class="hljs-string">&#125;</span></code></pre></div><ul><li><p>ptrdata</p><p>指针截止的长度</p></li></ul><ul><li><p>重点看这个字段 <code>gcdata</code> :</p><p><code>112</code>的二进制就是 <code>0111 0000</code>, 而<code>0111 0000</code> reverse一下就变成<code>(0000 1110)</code><sub>2</sub> ，第2,3,4个bit为1，分别对应</p></li></ul><h3 id="2-运行阶段">2. 运行阶段</h3><p>主要是<code>heapBitsSetType</code>这个函数</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">heapBitsSetType</span><span class="hljs-params">(x, size, dataSize <span class="hljs-keyword">uintptr</span>, typ *_type)</span></span> &#123;&#125;</code></pre></div><p>其传入参数可以看到:</p><ul><li>x :</li></ul><p>结构(对象)的<strong>开始地址</strong>，是uintptr</p><ul><li>size:</li></ul><ul><li><p>dataSize:</p><p>其值永远都是每次roundup（可能根据<strong>sizetoclass(可以看下memManage那篇文章)</strong>)得到的大小,但是，在分配<strong>defer块</strong>的时候<strong>不会</strong>,sizeof(defer{})可以看见至少有6个字(6*64bits=48 Bytes,64位下),可能会偏大;</p></li><li><p>typ:</p></li></ul><p>传入的类型，记录了结构的gc的map(gcdata)，大小，类型，hash等一系列值</p><p>为什么不用原子性保证，并发问题？注释里面给出答案：因为每次都</p><ul><li>只会从一个span里面分配空间</li><li>span的bitmap每次都只会在规定的byte边界内（内存对齐的结果）</li></ul><p>所以写的冲突是不会出现的；</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// There can only be one allocation from a given span active at a time,</span><span class="hljs-comment">// and the bitmap for a span always falls on byte boundaries,</span><span class="hljs-comment">// so there are no write-write races for access to the heap bitmap.</span><span class="hljs-comment">// Hence, heapBitsSetType can access the bitmap without atomics.</span><span class="hljs-comment">//</span></code></pre></div><p>主要逻辑:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">heapBitsSetType</span><span class="hljs-params">(x, size, dataSize <span class="hljs-keyword">uintptr</span>, typ *_type)</span></span> &#123;...<span class="hljs-comment">//1. 通过分配地址反查到heap的heapBits结构</span>h := heapBitsForAddr(x)<span class="hljs-comment">//获取到类型的指针bitmap</span>ptrmask := typ.gcdata <span class="hljs-comment">// start of 1-bit pointer mask (or GC program, handled below)</span>...<span class="hljs-keyword">var</span> ( ...)<span class="hljs-comment">//将h.bitp堆上的bitmap取出</span>hbitp = h.bitp...<span class="hljs-comment">//该类型的bitmap</span>p = ptrmask....<span class="hljs-keyword">if</span> p != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">//保存bitmap第一个Byte</span>b = <span class="hljs-keyword">uintptr</span>(*p)<span class="hljs-comment">//p指向下一个Byte</span>p = add1(p)nb = <span class="hljs-number">8</span>&#125;<span class="hljs-comment">//我们的结构是48==48，最简单的struct</span><span class="hljs-keyword">if</span> typ.size == dataSize &#123;<span class="hljs-comment">// Single entry: can stop once we reach the non-pointer data.</span><span class="hljs-comment">//nw = 5 = 40 / 8  ，说明扫描到第5个字段即可,因为ptrdata就是已经划定了范围[0,40]</span>nw = typ.ptrdata / sys.PtrSize&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//针对array的</span><span class="hljs-comment">// Repeated instances of typ in an array.</span><span class="hljs-comment">// Have to process first N-1 entries in full, but can stop</span><span class="hljs-comment">// once we reach the non-pointer data in the final entry.</span>nw = ((dataSize/typ.size<span class="hljs-number">-1</span>)*typ.size + typ.ptrdata) / sys.PtrSize&#125;<span class="hljs-comment">//如果nw=0，该struct无指针</span><span class="hljs-keyword">if</span> nw == <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// No pointers! Caller was supposed to check.</span><span class="hljs-built_in">println</span>(<span class="hljs-string">"runtime: invalid type "</span>, typ.<span class="hljs-keyword">string</span>())throw(<span class="hljs-string">"heapBitsSetType: called with non-pointer type"</span>)<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">//至少要写入两个字，因为noscan 的编码要求todo???</span><span class="hljs-keyword">if</span> nw &lt; <span class="hljs-number">2</span> &#123;<span class="hljs-comment">// Must write at least 2 words, because the "no scan"</span><span class="hljs-comment">// encoding doesn't take effect until the third word.</span>nw = <span class="hljs-number">2</span>&#125;<span class="hljs-comment">//接下来较为重要:</span><span class="hljs-comment">// Phase 1: Special case for leading byte (shift==0) or half-byte (shift==2).</span><span class="hljs-comment">// The leading byte is special because it contains the bits for word 1,</span><span class="hljs-comment">// which does not have the scan bit set.</span><span class="hljs-comment">// The leading half-byte is special because it's a half a byte,</span><span class="hljs-comment">// so we have to be careful with the bits already there.</span><span class="hljs-keyword">switch</span> &#123;<span class="hljs-keyword">default</span>:throw(<span class="hljs-string">"heapBitsSetType: unexpected shift"</span>)<span class="hljs-keyword">case</span> h.shift == <span class="hljs-number">0</span>:<span class="hljs-comment">// Ptrmask and heap bitmap are aligned.</span><span class="hljs-comment">// Handle first byte of bitmap specially.</span><span class="hljs-comment">//</span><span class="hljs-comment">// The first byte we write out covers the first four</span><span class="hljs-comment">// words of the object. The scan/dead bit on the first</span><span class="hljs-comment">// word must be set to scan since there are pointers</span><span class="hljs-comment">// somewhere in the object. The scan/dead bit on the</span><span class="hljs-comment">// second word is the checkmark, so we don't set it.</span><span class="hljs-comment">// In all following words, we set the scan/dead</span><span class="hljs-comment">// appropriately to indicate that the object contains</span><span class="hljs-comment">// to the next 2-bit entry in the bitmap.</span><span class="hljs-comment">//</span><span class="hljs-comment">// <span class="hljs-doctag">TODO:</span> It doesn't matter if we set the checkmark, so</span><span class="hljs-comment">// maybe this case isn't needed any more.</span><span class="hljs-comment">//b是类型的,b = 0001 0100</span><span class="hljs-comment">//bitPointerAll = 0000 1111</span><span class="hljs-comment">//hb = 0000 0100</span>hb = b &amp; bitPointerAllhb |= bitScan | bitScan&lt;&lt;(<span class="hljs-number">2</span>*heapBitsShift) | bitScan&lt;&lt;(<span class="hljs-number">3</span>*heapBitsShift)<span class="hljs-keyword">if</span> w += <span class="hljs-number">4</span>; w &gt;= nw &#123;<span class="hljs-keyword">goto</span> Phase3&#125;*hbitp = <span class="hljs-keyword">uint8</span>(hb)<span class="hljs-comment">//指针往后一个字节(递进一个)</span>hbitp = add1(hbitp)b &gt;&gt;= <span class="hljs-number">4</span>nb -= <span class="hljs-number">4</span><span class="hljs-keyword">case</span> sys.PtrSize == <span class="hljs-number">8</span> &amp;&amp; h.shift == <span class="hljs-number">2</span>:...&#125;<span class="hljs-comment">// Phase 2: Full bytes in bitmap, up to but not including write to last byte (full or partial) in bitmap.</span><span class="hljs-comment">// The loop computes the bits for that last write but does not execute the write;</span><span class="hljs-comment">// it leaves the bits in hb for processing by phase 3.</span><span class="hljs-comment">// To avoid repeated adjustment of nb, we subtract out the 4 bits we're going to</span><span class="hljs-comment">// use in the first half of the loop right now, and then we only adjust nb explicitly</span><span class="hljs-comment">// if the 8 bits used by each iteration isn't balanced by 8 bits loaded mid-loop.</span><span class="hljs-comment">//继续处理后4个bit!!!</span>nb -= <span class="hljs-number">4</span><span class="hljs-keyword">for</span> &#123;<span class="hljs-comment">// Emit bitmap byte.</span><span class="hljs-comment">// b has at least nb+4 bits, with one exception:</span><span class="hljs-comment">// if w+4 &gt;= nw, then b has only nw-w bits,</span><span class="hljs-comment">// but we'll stop at the break and then truncate</span><span class="hljs-comment">// appropriately in Phase 3.</span>hb = b &amp; bitPointerAllhb |= bitScanAll<span class="hljs-keyword">if</span> w += <span class="hljs-number">4</span>; w &gt;= nw &#123;<span class="hljs-comment">//已经处理完成，有指针的字段都包含在已经处理的ptrmask范围内</span><span class="hljs-keyword">break</span>&#125;...&#125;<span class="hljs-comment">//第三阶段,写入最后的byte或者是部分byte然后将剩下的bitmap置0</span>Phase3:<span class="hljs-comment">// Phase 3: Write last byte or partial byte and zero the rest of the bitmap entries.</span><span class="hljs-keyword">if</span> w &gt; nw &#123;<span class="hljs-comment">// Counting the 4 entries in hb not yet written to memory,</span><span class="hljs-comment">// there are more entries than possible pointer slots.</span><span class="hljs-comment">// Discard the excess entries (can't be more than 3).</span>mask := <span class="hljs-keyword">uintptr</span>(<span class="hljs-number">1</span>)&lt;&lt;(<span class="hljs-number">4</span>-(w-nw)) - <span class="hljs-number">1</span>hb &amp;= mask | mask&lt;&lt;<span class="hljs-number">4</span> <span class="hljs-comment">// apply mask to both pointer bits and scan bits</span>&#125;<span class="hljs-comment">// Change nw from counting possibly-pointer words to total words in allocation.</span>nw = size / sys.PtrSize<span class="hljs-comment">// Write whole bitmap bytes.</span><span class="hljs-comment">// The first is hb, the rest are zero.</span><span class="hljs-keyword">if</span> w &lt;= nw &#123;*hbitp = <span class="hljs-keyword">uint8</span>(hb)hbitp = add1(hbitp)hb = <span class="hljs-number">0</span> <span class="hljs-comment">// for possible final half-byte below</span><span class="hljs-keyword">for</span> w += <span class="hljs-number">4</span>; w &lt;= nw; w += <span class="hljs-number">4</span> &#123;*hbitp = <span class="hljs-number">0</span>hbitp = add1(hbitp)&#125;&#125;<span class="hljs-comment">// Write final partial bitmap byte if any.</span><span class="hljs-comment">// We know w &gt; nw, or else we'd still be in the loop above.</span><span class="hljs-comment">// It can be bigger only due to the 4 entries in hb that it counts.</span><span class="hljs-comment">// If w == nw+4 then there's nothing left to do: we wrote all nw entries</span><span class="hljs-comment">// and can discard the 4 sitting in hb.</span><span class="hljs-comment">// But if w == nw+2, we need to write first two in hb.</span><span class="hljs-comment">// The byte is shared with the next object, so be careful with</span><span class="hljs-comment">// existing bits.</span><span class="hljs-keyword">if</span> w == nw+<span class="hljs-number">2</span> &#123;*hbitp = *hbitp&amp;^(bitPointer|bitScan|(bitPointer|bitScan)&lt;&lt;heapBitsShift) | <span class="hljs-keyword">uint8</span>(hb)&#125;Phase4:<span class="hljs-comment">// Phase 4: Copy unrolled bitmap to per-arena bitmaps, if necessary.</span>...&#125;</code></pre></div><p><code>heapSetBitsType</code>函数实际上主要做的就是 设置 <code>h.bitp</code>这个值,每分配一块内存，都会有一个bitmap对应这个内存块，指明指针的位置</p><h3 id="运行扫描阶段">运行扫描阶段</h3><p>主要由两个行为：</p><h4 id="1-scanstack">1. scanstack</h4><p>从markroot开始，栈 、全局变量、寄存器等根对象开始扫描，创建一个DAG，将root对象放入一个队列中；</p><div class="hljs"><pre><code class="hljs go"></code></pre></div><h4 id="2-scanobject">2. scanobject</h4><p>异步的goroutine运行<code>gcDrain</code>函数，从队列里消费对象，</p><h3 id="一些gc相关的设置">一些gc相关的设置</h3><h4 id="触发">触发</h4><p>触发gc可以由runtime.GC()(不一定,会判定是否执行)</p><p>提供了两个参数来控制GC</p><h4 id="1-gcpercent">1.GCPercent</h4><blockquote><blockquote><p>The first one is GCPercent. Basically this is a knob that adjusts how much CPU you want to use and how much memory you want to use. The default is 100 which means that half the heap is dedicated to live memory and half the heap is dedicated to allocation. You can modify this in either direction.</p></blockquote></blockquote><p>代码在runtime/mgc.go中:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Initialized from $GOGC.  GOGC=off means no GC.</span><span class="hljs-keyword">var</span> gcpercent <span class="hljs-keyword">int32</span>......<span class="hljs-comment">//go:linkname setGCPercent runtime/debug.setGCPercent</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">setGCPercent</span><span class="hljs-params">(in <span class="hljs-keyword">int32</span>)</span> <span class="hljs-params">(out <span class="hljs-keyword">int32</span>)</span></span> &#123;<span class="hljs-comment">// Run on the system stack since we grab the heap lock.</span>systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;lock(&amp;mheap_.lock)out = gcpercent<span class="hljs-keyword">if</span> in &lt; <span class="hljs-number">0</span> &#123;in = <span class="hljs-number">-1</span>&#125;gcpercent = inheapminimum = defaultHeapMinimum * <span class="hljs-keyword">uint64</span>(gcpercent) / <span class="hljs-number">100</span><span class="hljs-comment">// Update pacing in response to gcpercent change.</span>gcSetTriggerRatio(memstats.triggerRatio)unlock(&amp;mheap_.lock)&#125;)<span class="hljs-comment">// Pacing changed, so the scavenger should be awoken.</span>wakeScavenger()<span class="hljs-comment">// If we just disabled GC, wait for any concurrent GC mark to</span><span class="hljs-comment">// finish so we always return with no GC running.</span><span class="hljs-keyword">if</span> in &lt; <span class="hljs-number">0</span> &#123;gcWaitOnMark(atomic.Load(&amp;work.cycles))&#125;<span class="hljs-keyword">return</span> out&#125;</code></pre></div><ul><li>默认值是100，意味着一半的堆会用于实时内存，另一半的堆会用来分配</li><li></li></ul><h4 id="2-maxheap">2.Maxheap</h4><blockquote><blockquote><p>MaxHeap, which is not yet released but is being used and evaluated internally, lets the programmer set what the maximum heap size should be. Out of memory, OOMs, are tough on Go; temporary spikes in memory usage should be handled by increasing CPU costs, not by aborting. Basically if the GC sees memory pressure it informs the application that it should shed load. Once things are back to normal the GC informs the application that it can go back to its regular load. MaxHeap also provides a lot more flexibility in scheduling. Instead of always being paranoid about how much memory is available the runtime can size the heap up to the MaxHeap.</p></blockquote></blockquote><ul><li>还在实验中… 最主要提供一些监控，会提示程序内存不足；还会使调度更加灵活；</li></ul><h4 id="方法-三色回收">方法-三色回收</h4><p>golang现在使用的是叫 <strong>三色回收</strong>的东西：比较旧的版本:大概步骤：</p><ol><li>所有对象初始都设为 <strong>白色</strong></li><li>从RootSet出发（即堆里面的对象，比如全局变量，所有的栈对象等），标记第一次所有可达到的对象为<strong>灰色</strong> ,这个过程,这里会stop the world;</li><li>然后紧接着在第一个发现的各个对象上继续寻找引用这些对象的对象们，找到后（或者在这些基础上已经找不到了）就把第一次所有可达的对象转为<strong>黑色</strong>,这时会start the world;而这些找到的对象们就标为<strong>灰色</strong> ;</li><li>重复第二,三步,直到所有</li></ol><div class="hljs"><pre><code class="hljs golang"></code></pre></div><h4 id="读写屏障">读写屏障</h4><p>现在采用Dijkstra插入写屏障和Yuusa删除写屏障构成了混合写屏障:</p><p>主要用处是将 被覆盖的对象标记成<strong>灰色</strong> 并 在<strong>当前栈没有扫描时</strong>将新对象也标记成灰色:</p><p>伪代码:</p><div class="hljs"><pre><code class="hljs undefined">writePointer(slot, <span class="hljs-keyword">ptr</span>):    shade(*slot)    <span class="hljs-keyword">if</span> current stack <span class="hljs-keyword">is</span> grey:        shade(<span class="hljs-keyword">ptr</span>)    *slot = <span class="hljs-keyword">ptr</span></code></pre></div><p><strong>为了移除栈的重扫描过程</strong>，除了引入混合写屏障之外,在垃圾收集的<code>标记阶段</code>，我们还需要将创建的所有新对象都标记成<strong>黑色</strong>，防止新分配的栈内存和堆内存中的对象被错误地回收，因为栈内存在标记阶段最终都会变为黑色，所以不再需要重新扫描栈空间;</p><h3 id="做过的一些优化">做过的一些优化</h3><h4 id="size-segregated-span-分片隔离">size segregated span （分片隔离）</h4><ol><li>garbage collector要快速地找到object的开始位置，如果能知道在某个span中的object的大小，就可以直接往下舍入查找到位置</li><li>更小的碎片化</li><li>内部结构化？</li><li></li></ol><h3 id="调查一下其他部分">调查一下其他部分</h3><p>有这个一个变量： <strong>gcphase</strong></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Garbage collector phase.</span><span class="hljs-comment">// Indicates to write barrier and synchronization task to perform.</span><span class="hljs-keyword">var</span> gcphase <span class="hljs-keyword">uint32</span></code></pre></div><p>write barrier写屏障</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// The compiler knows about this variable.</span><span class="hljs-comment">// If you change it, you must change builtin/runtime.go, too.</span><span class="hljs-comment">// If you change the first four bytes, you must also change the write</span><span class="hljs-comment">// barrier insertion code.</span><span class="hljs-keyword">var</span> writeBarrier <span class="hljs-keyword">struct</span> &#123;enabled <span class="hljs-keyword">bool</span>    <span class="hljs-comment">// compiler emits a check of this before calling write barrier</span>pad     [<span class="hljs-number">3</span>]<span class="hljs-keyword">byte</span> <span class="hljs-comment">// compiler uses 32-bit load for "enabled" field</span>needed  <span class="hljs-keyword">bool</span>    <span class="hljs-comment">// whether we need a write barrier for current GC phase</span>cgo     <span class="hljs-keyword">bool</span>    <span class="hljs-comment">// whether we need a write barrier for a cgo check</span>alignme <span class="hljs-keyword">uint64</span>  <span class="hljs-comment">// guarantee alignment so that compiler can use a 32 or 64-bit load</span>&#125;</code></pre></div><h3 id="具体实现">具体实现</h3><ol><li>bitmap/runtime/mbitmap.go使用bitmap</li></ol>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang Memory Allocator</title>
    <link href="/2020/02/24/Go/memManage/"/>
    <url>/2020/02/24/Go/memManage/</url>
    
    <content type="html"><![CDATA[<h1>golang内存管理</h1><a id="more"></a><p>go的内存管理是基于tcmalloc，<a href="http://goog-perftools.sourceforge.net/doc/tcmalloc.html" target="_blank" rel="noopener">这个连接</a>看详情盗来<img src="/img/tcmalloc.jpg" srcset="/img/loading.gif" alt="一张图"></p><p>任何大小的内存页可以被分割成<strong>一系列同样大小的object</strong>,这些规定的大小size则被定义在<a href="#sizetoclass">sizetoclass</a>,然后被一个<strong>bitmap</strong>管理</p><h2 id="内存总体结构设计">内存总体结构设计</h2><p>暂时将linux amd64作为例子</p><h3 id="虚拟内存布局">虚拟内存布局</h3><ul><li><p>1.10以前，内存不是初始化就分配虚拟内存arena大小为512G，为了方便将其分为一个个page，所以总共也有512G/8KB = 65536个page</p><p>span区域存放指向span的指针，表示arena区域page所属的span，所以其大小即为 512GB/8KB* 8B(指针大小) = 512M</p><p>bitmap主要用于GC，两个bit表示arena中一个字的可用状态，所以表示为 (512GB/ 8(8个byte一个字，即指令长度)) * 2 /8 (8个bit一个byte) = 16G 长度</p></li><li><p>1.11以后</p><p>改成两阶段稀疏索引方式，内存允许超过512G，也可以允许不连续内存mheap中的arenas字段实际是一个指针数组，每个<code>heapArena</code>管理一个<strong>64MB</strong>的内存其结构如下:</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//// heapArenaBitmapBytes is the size of each heap arena's bitmap.</span>heapArenaBitmapBytes = heapArenaBytes / (sys.PtrSize * <span class="hljs-number">8</span> / <span class="hljs-number">2</span>)pagesPerArena = heapArenaBytes / pageSize<span class="hljs-comment">// A heapArena stores metadata for a heap arena. heapArenas are stored</span><span class="hljs-comment">// outside of the Go heap and accessed via the mheap_.arenas index.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:notinheap</span><span class="hljs-keyword">type</span> heapArena <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// bitmap stores the pointer/scalar bitmap for the words in</span><span class="hljs-comment">// this arena. See mbitmap.go for a description. Use the</span><span class="hljs-comment">// heapBits type to access this.</span><span class="hljs-comment">//即是内存中bitmap对应</span>bitmap [heapArenaBitmapBytes]<span class="hljs-keyword">byte</span><span class="hljs-comment">// spans maps from virtual address page ID within this arena to *mspan.</span><span class="hljs-comment">// For allocated spans, their pages map to the span itself.</span><span class="hljs-comment">// For free spans, only the lowest and highest pages map to the span itself.</span><span class="hljs-comment">// Internal pages map to an arbitrary span.</span><span class="hljs-comment">// For pages that have never been allocated, spans entries are nil.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Modifications are protected by mheap.lock. Reads can be</span><span class="hljs-comment">// performed without locking, but ONLY from indexes that are</span><span class="hljs-comment">// known to contain in-use or stack spans. This means there</span><span class="hljs-comment">// must not be a safe-point between establishing that an</span><span class="hljs-comment">// address is live and looking it up in the spans array.</span><span class="hljs-comment">//这里的safe-point一般就指STW和栈扫描时期</span><span class="hljs-comment">//与内存中spans对应</span>spans [pagesPerArena]*mspan<span class="hljs-comment">// pageInUse is a bitmap that indicates which spans are in</span><span class="hljs-comment">// state mSpanInUse. This bitmap is indexed by page number,</span><span class="hljs-comment">// but only the bit corresponding to the first page in each</span><span class="hljs-comment">// span is used.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Reads and writes are atomic.</span>pageInUse [pagesPerArena / <span class="hljs-number">8</span>]<span class="hljs-keyword">uint8</span><span class="hljs-comment">// pageMarks is a bitmap that indicates which spans have any</span><span class="hljs-comment">// marked objects on them. Like pageInUse, only the bit</span><span class="hljs-comment">// corresponding to the first page in each span is used.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Writes are done atomically during marking. Reads are</span><span class="hljs-comment">// non-atomic and lock-free since they only occur during</span><span class="hljs-comment">// sweeping (and hence never race with writes).</span><span class="hljs-comment">//</span><span class="hljs-comment">// This is used to quickly find whole spans that can be freed.</span><span class="hljs-comment">//</span><span class="hljs-comment">// TODO(austin): It would be nice if this was uint64 for</span><span class="hljs-comment">// faster scanning, but we don't have 64-bit atomic bit</span><span class="hljs-comment">// operations.</span>pageMarks [pagesPerArena / <span class="hljs-number">8</span>]<span class="hljs-keyword">uint8</span><span class="hljs-comment">// zeroedBase marks the first byte of the first page in this</span><span class="hljs-comment">// arena which hasn't been used yet and is therefore already</span><span class="hljs-comment">// zero. zeroedBase is relative to the arena base.</span><span class="hljs-comment">// Increases monotonically until it hits heapArenaBytes.</span><span class="hljs-comment">//</span><span class="hljs-comment">// This field is sufficient to determine if an allocation</span><span class="hljs-comment">// needs to be zeroed because the page allocator follows an</span><span class="hljs-comment">// address-ordered first-fit policy.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Read atomically and written with an atomic CAS.</span><span class="hljs-comment">//字段指向了该结构体管理的内存的基地址</span><span class="hljs-comment">//很多在堆上的零值都是指向了这里</span>zeroedBase <span class="hljs-keyword">uintptr</span>&#125;</code></pre></div><p>bitmap和spans功能不变</p><h2 id="基本内存管理级别">基本内存管理级别</h2><p>类似于<a href="http://goog-perftools.sourceforge.net/doc/tcmalloc.html" target="_blank" rel="noopener">TCMalloc</a></p><p>大概概括:其目的是 减少多线程对内存请求时候的锁竞争，在对小内存的申请时甚至可以无锁操作，获取大内存时用spinlocks；但是其在TLS会预分配一部分空间，所以启动时相比dlmalloc等其他内存分配器空间较大，但是最终会接近;</p><p><strong>class_to_allocnpages</strong>总共有<code>67</code>???个范围(应该是程序经过测试得来的数值)</p><p>栈的分配也是多层次和多class的</p><h3 id="mspan-主要使用该机制减少碎片">mspan (主要使用该机制减少碎片):</h3><p><strong>基本单元内存单元</strong>被内存堆管理的的页面，至少一个页(8KB)，用于范围分配内存，比如16-32B则分配32B,112~128则分配<strong>128B</strong>的span</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> mspan <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">//实际是一个doubly-linked lilst</span>next *mspan     <span class="hljs-comment">// next span in list, or nil if none</span>prev *mspan     <span class="hljs-comment">// previous span in list, or nil if none</span>list *mSpanList <span class="hljs-comment">// For debugging. <span class="hljs-doctag">TODO:</span> Remove.</span>startAddr <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// address of first byte of span aka s.base()</span>npages    <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// number of pages in span</span>manualFreeList gclinkptr <span class="hljs-comment">// list of free objects in mSpanManual spans</span><span class="hljs-comment">// freeindex is the slot index between 0 and nelems at which to begin scanning</span><span class="hljs-comment">// for the next free object in this span.</span><span class="hljs-comment">// Each allocation scans allocBits starting at freeindex until it encounters a 0</span><span class="hljs-comment">// indicating a free object. freeindex is then adjusted so that subsequent scans begin</span><span class="hljs-comment">// just past the newly discovered free object.</span><span class="hljs-comment">//</span><span class="hljs-comment">// If freeindex == nelem, this span has no free objects.</span><span class="hljs-comment">//</span><span class="hljs-comment">// allocBits is a bitmap of objects in this span.</span><span class="hljs-comment">// If n &gt;= freeindex and allocBits[n/8] &amp; (1&lt;&lt;(n%8)) is 0</span><span class="hljs-comment">//???为什么这样计算</span><span class="hljs-comment">// then object n is free;</span><span class="hljs-comment">// otherwise, object n is allocated. Bits starting at nelem are</span><span class="hljs-comment">// undefined and should never be referenced.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Object n starts at address n*elemsize + (start &lt;&lt; pageShift).</span>freeindex <span class="hljs-keyword">uintptr</span><span class="hljs-comment">// <span class="hljs-doctag">TODO:</span> Look up nelems from sizeclass and remove this field if it</span><span class="hljs-comment">// helps performance.</span>nelems <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// number of object in the span.</span><span class="hljs-comment">// Cache of the allocBits at freeindex. allocCache is shifted</span><span class="hljs-comment">// such that the lowest bit corresponds to the bit freeindex.</span><span class="hljs-comment">// allocCache holds the complement of allocBits, thus allowing</span><span class="hljs-comment">// ctz (count trailing zero) to use it directly.</span><span class="hljs-comment">// allocCache may contain bits beyond s.nelems; the caller must ignore</span><span class="hljs-comment">// these.</span><span class="hljs-comment">//- allocBits在freeIndex上的缓存;</span><span class="hljs-comment">//- allocBits的补码,ctz可以直接使用来计算查找空闲的内存;</span><span class="hljs-comment">//- 会包含一些s.nelems前面的位，调用者要忽略这些位;</span><span class="hljs-comment">//在mspan.refillAllocCache()时改变值;</span>allocCache <span class="hljs-keyword">uint64</span><span class="hljs-comment">// allocBits and gcmarkBits hold pointers to a span's mark and</span><span class="hljs-comment">// allocation bits. The pointers are 8 byte aligned.</span><span class="hljs-comment">// There are three arenas where this data is held.</span><span class="hljs-comment">// free: Dirty arenas that are no longer accessed</span><span class="hljs-comment">//       and can be reused.</span><span class="hljs-comment">// next: Holds information to be used in the next GC cycle.</span><span class="hljs-comment">// current: Information being used during this GC cycle.</span><span class="hljs-comment">// previous: Information being used during the last GC cycle.</span><span class="hljs-comment">// A new GC cycle starts with the call to finishsweep_m.</span><span class="hljs-comment">// finishsweep_m moves the previous arena to the free arena,</span><span class="hljs-comment">// the current arena to the previous arena, and</span><span class="hljs-comment">// the next arena to the current arena.</span><span class="hljs-comment">// The next arena is populated as the spans request</span><span class="hljs-comment">// memory to hold gcmarkBits for the next GC cycle as well</span><span class="hljs-comment">// as allocBits for newly allocated spans.</span><span class="hljs-comment">//</span><span class="hljs-comment">// The pointer arithmetic is done "by hand" instead of using</span><span class="hljs-comment">// arrays to avoid bounds checks along critical performance</span><span class="hljs-comment">// paths.</span><span class="hljs-comment">// The sweep will free the old allocBits and set allocBits to the</span><span class="hljs-comment">// gcmarkBits. The gcmarkBits are replaced with a fresh zeroed</span><span class="hljs-comment">// out memory.</span>allocBits  *gcBitsgcmarkBits *gcBits<span class="hljs-comment">// sweep generation:</span><span class="hljs-comment">// if sweepgen == h-&gt;sweepgen - 2, the span needs sweeping</span><span class="hljs-comment">// if sweepgen == h-&gt;sweepgen - 1, the span is currently being swept</span><span class="hljs-comment">// if sweepgen == h-&gt;sweepgen, the span is swept and ready to use</span><span class="hljs-comment">// if sweepgen == h-&gt;sweepgen + 1, the span was cached before sweep began and is still cached, and needs sweeping</span><span class="hljs-comment">// if sweepgen == h-&gt;sweepgen + 3, the span was swept and then cached and is still cached</span><span class="hljs-comment">// h-&gt;sweepgen is incremented by 2 after every GC</span><span class="hljs-comment">//sweepgen 的分代，用于垃圾回收，与</span>sweepgen    <span class="hljs-keyword">uint32</span>divMul      <span class="hljs-keyword">uint16</span>     <span class="hljs-comment">// for divide by elemsize - divMagic.mul</span>baseMask    <span class="hljs-keyword">uint16</span>     <span class="hljs-comment">// if non-0, elemsize is a power of 2, &amp; this will get object allocation base</span>allocCount  <span class="hljs-keyword">uint16</span>     <span class="hljs-comment">// number of allocated objects</span>spanclass   spanClass  <span class="hljs-comment">// size class and noscan (uint8)</span><span class="hljs-comment">//管理状态</span>state       mSpanState <span class="hljs-comment">// mspaninuse etc</span>needzero    <span class="hljs-keyword">uint8</span>      <span class="hljs-comment">// needs to be zeroed before allocation</span>divShift    <span class="hljs-keyword">uint8</span>      <span class="hljs-comment">// for divide by elemsize - divMagic.shift</span>divShift2   <span class="hljs-keyword">uint8</span>      <span class="hljs-comment">// for divide by elemsize - divMagic.shift2</span>scavenged   <span class="hljs-keyword">bool</span>       <span class="hljs-comment">// whether this span has had its pages released to the OS</span>elemsize    <span class="hljs-keyword">uintptr</span>    <span class="hljs-comment">// computed from sizeclass or from npages</span>limit       <span class="hljs-keyword">uintptr</span>    <span class="hljs-comment">// end of data in span</span>speciallock mutex      <span class="hljs-comment">// guards specials list</span>specials    *special   <span class="hljs-comment">// linked list of special records sorted by offset.</span>&#125;</code></pre></div><p>主要的字段:</p><ul><li><p><code>startAddr</code>和<code>npages</code>确定结构体管理的多个页所在的内存，每个页都是8KB</p></li><li><p><code>elementsize</code>: slot大小，Byte为单位</p></li><li><p><code>freeindex</code>，&lt;该值的已经被分配，&gt;=该位置的可能未被分配，需要配合allocCache查找,每次分配后，freeindex设置为分配的slot+1,用作扫描野种空闲对象的初始索引;</p></li><li><p><code>allocBits</code> 标记内存占用状态，表示上一次<strong>GC</strong>之后哪一些slot被使用，0未使用或释放，1已分配</p></li><li><p><code>gcmarkBits</code> 标记内存回收状态</p></li><li><p>每次gc完的sweep阶段，将<code>allocBits</code>设置为<code>gcmarkbits</code></p></li><li><p><code>allocCache</code> 是<code>allocBits</code>的表示从freeindex开始的64个slot的分配情况，1为未分配，0为已分配，使用ctz(Count trailing zeros指令)找到第一个非0位，使用完了就从allocBits加载，取反；</p></li><li><p><code>sweepgen</code>,垃圾回收的分代状态，主要拥有同<code>mheap</code>中的当前<code>mSpan</code>的<code>sweepgen</code>进行比较:</p><p>每次GC，<code>h-&gt;sweepgen</code>都会 +2 ;</p><ol><li>如果 sweepgen == h-&gt;sweepgen - 2, 这个span需要清除</li><li>如果 sweepgen == h-&gt;sweepgen - 1, 这个span正在被清除</li><li>if sweepgen == h-&gt;sweepgen, 这个span已经被清除过并且就绪可用</li><li>if sweepgen == h-&gt;sweepgen + 1, 这个span在清除之前就被缓存且仍在缓存中，需要被清除（很明显这里mSpan的sweepgen&gt;h.sweepgen，证明已经活过了上一次清除,即被缓存下来);</li><li>if sweepgen == h-&gt;sweepgen + 3, 这个span被清除后缓存，且在缓存中(一般来讲是不需要再缓存了???)</li></ol></li></ul><h4 id="管理结构">管理结构</h4><ul><li><p>当结构体管理内存不足时</p><p>其会以<code>页</code>为单位向堆申请内存, 涉及<code>npages</code>字段</p></li><li><p>当用户程序或者线程向<code>mspan</code>申请内存时</p><p>该结构会使用<code>allocCache</code>字段以<strong>对象为单位</strong>在管理的内存中快速查找待分配的空间,</p></li></ul><p>如果找得到就返回，如果找不到，更上一级的<code>mcache</code>会调用<code>mcache.refill</code>更新内存管理单元<code>mspan</code>满足需求</p><h4 id="状态">状态</h4><p>结构体下面的</p><p><code>mSpanStateBox</code> 会用来存储<code>mSpan</code>的状态</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> mSpan <span class="hljs-keyword">struct</span>&#123;...state mSpanStateBox...&#125;<span class="hljs-comment">// An mspan representing actual memory has state mSpanInUse,</span><span class="hljs-comment">// mSpanManual, or mSpanFree. Transitions between these states are</span><span class="hljs-comment">// constrained as follows:</span><span class="hljs-comment">//</span><span class="hljs-comment">// * A span may transition from free to in-use or manual during any GC</span><span class="hljs-comment">//   phase.</span><span class="hljs-comment">//</span><span class="hljs-comment">// * During sweeping (gcphase == _GCoff), a span may transition from</span><span class="hljs-comment">//   in-use to free (as a result of sweeping) or manual to free (as a</span><span class="hljs-comment">//   result of stacks being freed).</span><span class="hljs-comment">//</span><span class="hljs-comment">// * During GC (gcphase != _GCoff), a span *must not* transition from</span><span class="hljs-comment">//   manual or in-use to free. Because concurrent GC may read a pointer</span><span class="hljs-comment">//   and then look up its span, the span state must be monotonic.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Setting mspan.state to mSpanInUse or mSpanManual must be done</span><span class="hljs-comment">// atomically and only after all other span fields are valid.</span><span class="hljs-comment">// Likewise, if inspecting a span is contingent on it being</span><span class="hljs-comment">// mSpanInUse, the state should be loaded atomically and checked</span><span class="hljs-comment">// before depending on other fields. This allows the garbage collector</span><span class="hljs-comment">// to safely deal with potentially invalid pointers, since resolving</span><span class="hljs-comment">// such pointers may race with a span being allocated.</span><span class="hljs-keyword">type</span> mSpanState <span class="hljs-keyword">uint8</span><span class="hljs-keyword">const</span> (mSpanDead   mSpanState = <span class="hljs-literal">iota</span>mSpanInUse             <span class="hljs-comment">// allocated for garbage collected heap</span>mSpanManual            <span class="hljs-comment">// allocated for manual management (e.g., stack allocator)</span>)</code></pre></div><p>里面一大段注释其实差不多讲明了状态转换:</p><ol><li>当<code>mspan</code>在空闲的堆中，就会处于<code>mspanFree</code>状态</li><li>当<code>mspan</code>已经被分配，就会处于<code>mspanInUse</code>,<code>mSpanMannual</code>状态,其转换有:<ul><li>gc的任何阶段，可能从<code>mSpanFree</code>转到<code>mSpanInUse</code>,<code>mSpanMannual</code></li><li>gc的<strong>清除</strong>阶段，可能从<code>mSpanInUse</code>转到<code>mSpanMannual</code>，<code>mSpanFree</code></li><li>gc的标记阶段,可能从<code>mSpanInUse</code>转到<code>mSpanMannual</code>,<code>mSpanFree</code></li></ul></li></ol><p>其状态转换还一定要是原子操作，避免竞态条件</p><h4 id="spanclass">spanClass</h4><p><code>spanClass</code></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> mSpan <span class="hljs-keyword">struct</span>&#123;...spanClass spanClass...&#125;<span class="hljs-comment">// A spanClass represents the size class and noscan-ness of a span.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Each size class has a noscan spanClass and a scan spanClass. The</span><span class="hljs-comment">// noscan spanClass contains only noscan objects, which do not contain</span><span class="hljs-comment">// pointers and thus do not need to be scanned by the garbage</span><span class="hljs-comment">// collector.</span><span class="hljs-comment">//前7位存其ID，最后一位是noscan位</span><span class="hljs-keyword">type</span> spanClass <span class="hljs-keyword">uint8</span></code></pre></div><p>这个是<code>mSpan</code>的跨度类,决定了内存管理单元的存储对象<strong>大小</strong>和<strong>数量</strong>,</p><ul><li><p>有67种，在<code>class_to_size</code> 和 <code>class_to_allocnpages</code>上,参照下面<a href="#%E8%A1%A5%E5%85%A8%E7%9A%84spanClass">补全的表格</a></p></li><li><p>其除了保存类别的ID(前7位)，还会保存<code>noscan</code>标记位(最后一位)，这个标记了对象<strong>是否包含指针</strong>，gc会对无该标记位的<code>mspan</code>扫描</p></li></ul><p>以下的函数为ID和标记位的保存方式:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">makeSpanClass</span><span class="hljs-params">(sizeclass <span class="hljs-keyword">uint8</span>, noscan <span class="hljs-keyword">bool</span>)</span> <span class="hljs-title">spanClass</span></span>&#123;<span class="hljs-keyword">return</span> spanClass(sizeclass&lt;&lt;<span class="hljs-number">1</span>) | spanClass(bool2int(noscan))&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(sc spanClass)</span> <span class="hljs-title">sizeclass</span><span class="hljs-params">()</span> <span class="hljs-title">int8</span></span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-keyword">int8</span>(sc &gt;&gt; <span class="hljs-number">1</span>)&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(sc spanClass)</span> <span class="hljs-title">noscan</span><span class="hljs-params">()</span> <span class="hljs-title">bool</span></span> &#123;<span class="hljs-keyword">return</span> sc&amp;<span class="hljs-number">1</span> != <span class="hljs-number">0</span>&#125;</code></pre></div><h3 id="mcache">mcache</h3><p>主要用来缓存<code>小对象</code>(0~32KB),里面就有基本单元<code>mspan</code>, 与线程上的处理器一一绑定;</p><ul><li>多层次的cache用来减少分配冲突，<code>mcache</code>是<code>per-P</code>的，所以无锁；小于16B直接使用P中的<code>macache</code>???</li></ul><p>每一个<code>mcache</code>都有<em><em>67</em>2</em>*个<code>mspan</code>结构，都在<code>alloc</code>字段中;</p><ul><li>其初始化的时候不包含<code>mspan</code>,只有当用户申请内存才会从上一级<code>mspan</code>来满足要求</li></ul><h4 id="结构">结构</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Per-thread (in Go, per-P) cache for small objects.</span><span class="hljs-comment">// No locking needed because it is per-thread (per-P).</span><span class="hljs-comment">//</span><span class="hljs-comment">// mcaches are allocated from non-GC'd memory, so any heap pointers</span><span class="hljs-comment">// must be specially handled.</span><span class="hljs-comment">//</span><span class="hljs-comment">//因为只用作local P，所以自然无锁</span><span class="hljs-comment">//go:notinheap</span><span class="hljs-comment">//这个标志说明了不在heap中，也可以理解，per - P好明显不在公共heap中</span><span class="hljs-keyword">type</span> mcache <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// The following members are accessed on every malloc,</span><span class="hljs-comment">// so they are grouped here for better caching.</span>next_sample <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// trigger heap sample after allocating this many bytes</span>local_scan  <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// bytes of scannable heap allocated</span><span class="hljs-comment">// Allocator cache for tiny objects w/o pointers.</span><span class="hljs-comment">// See "Tiny allocator" comment in malloc.go.</span><span class="hljs-comment">//具体可以见下面的tiny allocator分配器</span><span class="hljs-comment">// tiny points to the beginning of the current tiny block, or</span><span class="hljs-comment">// nil if there is no current tiny block.</span><span class="hljs-comment">//</span><span class="hljs-comment">// tiny is a heap pointer. Since mcache is in non-GC'd memory,</span><span class="hljs-comment">// we handle it by clearing it in releaseAll during mark</span><span class="hljs-comment">// termination.</span>tiny             <span class="hljs-keyword">uintptr</span>tinyoffset       <span class="hljs-keyword">uintptr</span>local_tinyallocs <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// number of tiny allocs not counted in other stats</span><span class="hljs-comment">// The rest is not accessed on every malloc.</span><span class="hljs-comment">//都保存在这里,67*2个</span>alloc [numSpanClasses]*mspan <span class="hljs-comment">// spans to allocate from, indexed by spanClass</span>stackcache [_NumStackOrders]stackfreelist<span class="hljs-comment">// Local allocator stats, flushed during GC.</span>local_largefree  <span class="hljs-keyword">uintptr</span>                  <span class="hljs-comment">// bytes freed for large objects (&gt;maxsmallsize)</span>local_nlargefree <span class="hljs-keyword">uintptr</span>                  <span class="hljs-comment">// number of frees for large objects (&gt;maxsmallsize)</span>local_nsmallfree [_NumSizeClasses]<span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// number of frees for small objects (&lt;=maxsmallsize)</span><span class="hljs-comment">// flushGen indicates the sweepgen during which this mcache</span><span class="hljs-comment">// was last flushed. If flushGen != mheap_.sweepgen, the spans</span><span class="hljs-comment">// in this mcache are stale and need to the flushed so they</span><span class="hljs-comment">// can be swept. This is done in acquirep.</span><span class="hljs-comment">//用作标记该mcache在最后一次写入磁盘的sweep状态</span><span class="hljs-comment">//如果其不等于mheap中的sweepgen,证明该mcahce内的spans都是稳定的且需要被写入磁盘，所以可以被清除；</span><span class="hljs-comment">//整个过程在acquirep中完成;</span>flushGen <span class="hljs-keyword">uint32</span>&#125;</code></pre></div><h4 id="初始化">初始化</h4><p>如下代码所示，实际会在systemstack中使用<code>mheap</code>的mcache分配器分配新的<code>mcache</code></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">allocmcache</span><span class="hljs-params">()</span> *<span class="hljs-title">mcache</span></span> &#123;<span class="hljs-keyword">var</span> c *mcachesystemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;lock(&amp;mheap_.lock)c = (*mcache)(mheap_.cachealloc.alloc())<span class="hljs-comment">//让flushGen设为mheap的sweepgen，未稳定，所以不可以被清除;</span>c.flushGen = mheap_.sweepgenunlock(&amp;mheap_.lock)&#125;)<span class="hljs-keyword">for</span> i := <span class="hljs-keyword">range</span> c.alloc &#123;<span class="hljs-comment">//每个元素只是一个emptySpan占位符;</span>c.alloc[i] = &amp;emptymspan&#125;c.next_sample = nextSample()<span class="hljs-keyword">return</span> c&#125;</code></pre></div><h4 id="替换已有的mspan">替换已有的mspan</h4><p><code>mcache.refill</code>方法会使<code>mcache</code>获取一个指定的spanClass(从下一级<code>mcentral</code>中)，被替换的mSpan<strong>不可以</strong>有空闲的内存空间(否则可能会频繁替换),而获取的mspan中需要至少包含一个空闲对象用于分配内存;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// refill acquires a new span of span class spc for c. This span will</span><span class="hljs-comment">// have at least one free object. The current span in c must be full.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Must run in a non-preemptible context since otherwise the owner of</span><span class="hljs-comment">// c could change.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *mcache)</span> <span class="hljs-title">refill</span><span class="hljs-params">(spc spanClass)</span></span> &#123;<span class="hljs-comment">// Return the current cached span to the central lists.</span>s := c.alloc[spc]<span class="hljs-comment">//allocCount必须要=nelems判断span的空间已经被用完</span><span class="hljs-keyword">if</span> <span class="hljs-keyword">uintptr</span>(s.allocCount) != s.nelems &#123;throw(<span class="hljs-string">"refill of span with free space remaining"</span>)&#125;<span class="hljs-keyword">if</span> s != &amp;emptymspan &#123;<span class="hljs-comment">// Mark this span as no longer cached.</span><span class="hljs-keyword">if</span> s.sweepgen != mheap_.sweepgen+<span class="hljs-number">3</span> &#123;throw(<span class="hljs-string">"bad sweepgen in refill"</span>)&#125;<span class="hljs-comment">//标记该mSpan=mheap.sweepgen+3， 即不需要再缓存???</span>atomic.Store(&amp;s.sweepgen, mheap_.sweepgen)&#125;<span class="hljs-comment">// Get a new cached span from the central lists.</span><span class="hljs-comment">//从mcentral取的mSpan</span>s = mheap_.central[spc].mcentral.cacheSpan()<span class="hljs-keyword">if</span> s == <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"out of memory"</span>)&#125;<span class="hljs-comment">//判断取来的mSpan是不是已经无空余空间了</span><span class="hljs-keyword">if</span> <span class="hljs-keyword">uintptr</span>(s.allocCount) == s.nelems &#123;throw(<span class="hljs-string">"span has no free space"</span>)&#125;<span class="hljs-comment">// Indicate that this span is cached and prevent asynchronous</span><span class="hljs-comment">// sweeping in the next sweep phase.</span>s.sweepgen = mheap_.sweepgen + <span class="hljs-number">3</span>&#125;</code></pre></div><p>注意这个是向<code>mcache</code>插入<code>mSpan</code>的唯一方法(上一级是<code>nextFree</code>函数，在<code>mallocgc</code>上被使用);</p><h4 id="分配超小-tiny-对象-16kb-的字段">分配超小(tiny)对象(&lt;16KB)的字段</h4><p>mcache还对微小对象进行了分配处理:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> mcache <span class="hljs-keyword">struct</span> &#123;....tiny             <span class="hljs-keyword">uintptr</span>tinyoffset       <span class="hljs-keyword">uintptr</span>local_tinyallocs <span class="hljs-keyword">uintptr</span>....&#125;</code></pre></div><p>其中要注意：</p><ul><li>tinyAllocator只分配非指针类型的内存</li><li><code>tiny</code>指向堆中的一块内存，<code>tinyOffset</code>是下一个空闲内存的offset，<code>localtinyallocs</code>会记录内存分配器中分配的对象个数</li></ul><p>可以见下面的<a href="#tinyAllocator">分配器</a></p><h3 id="mcentral">mcentral</h3><p>可以看做是一种缓存(<code>mcache</code>的下一级)，但内存管理单元<code>mspan</code>不存在空闲对象时，程序就会从这个缓存中拿新的内存单元;</p><p>与<code>mcache</code>不同，该结构的<code>mSpan</code>需要加锁访问(非per-P);</p><p>全局有 <code>67 × 2</code> (指针的有67，非指针的有67) 个对应不同size的span <strong>后备</strong>mcentral</p><p>收集所有特定size的span，如果也被用完，则再次转向<code>mheap</code>申请(这个<code>mcentral</code>其实就是属于<code>mheap</code>的，其都会<strong>直接</strong>从系统中申请内存)</p><h4 id="结构体">结构体</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> mcentral <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">//加锁</span>lock      mutexspanclass spanClass<span class="hljs-comment">//管理包含空闲对象的mSpan list</span>nonempty  mSpanList <span class="hljs-comment">// list of spans with a free object, ie a nonempty free list</span><span class="hljs-comment">//不包含空闲对象对象或者是在mcache中缓存的mSpan list</span>empty     mSpanList <span class="hljs-comment">// list of spans with no free objects (or cached in an mcache)</span><span class="hljs-comment">// nmalloc is the cumulative count of objects allocated from</span><span class="hljs-comment">// this mcentral, assuming all spans in mcaches are</span><span class="hljs-comment">// fully-allocated. Written atomically, read under STW.</span>nmalloc <span class="hljs-keyword">uint64</span>&#125;</code></pre></div><p>可以看到mcentral有两个<code>mSpanList</code>，分别是包含空闲对象的链表和不包含空闲对象的链表;</p><p>初始化时两个链表都不包含任何内存，扩容时<code>nmalloc</code>会记录分配的对象个数,<strong>写入</strong>该值时是用<code>atomic</code>写入，<strong>读取</strong>因为是在STW中，所以无锁(这个很像mSpan中???);</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> mheap <span class="hljs-keyword">struct</span>&#123;...<span class="hljs-comment">// central free lists for small size classes.</span><span class="hljs-comment">// the padding makes sure that the mcentrals are</span><span class="hljs-comment">// spaced CacheLinePadSize bytes apart, so that each mcentral.lock</span><span class="hljs-comment">// gets its own cache line.</span><span class="hljs-comment">// central is indexed by spanClass.</span><span class="hljs-comment">//numSpanClasses = _NumSizeClasses &lt;&lt; 1 = 134</span>central [numSpanClasses]<span class="hljs-keyword">struct</span> &#123;mcentral mcentral pad      [cpu.CacheLinePadSize - unsafe.Sizeof(mcentral&#123;&#125;)%cpu.CacheLinePadSize]<span class="hljs-keyword">byte</span>&#125;....&#125;</code></pre></div><p>代码中的pad是用作分割多个<code>mcentral</code>，以<code>CacheLinePadSize</code>个Bytes分割开，所以每一个<code>mcentral</code>的lock可以得到自己的cache line我认为可以看做是内存对齐的一种方法(???是不是捏，是的，为了<strong>不同列表间互斥锁不会伪共享</strong>)</p><h4 id="获得mspan">获得mSpan</h4><p><code>mcache</code>会通过<code>mcentral.cacheSpan</code>方法获取新的内存管理单元<code>mSpan</code>:</p><p>可以大致分为几个步骤:</p><ol><li><p>从有空闲对象的<code>mSpan</code>链表获取可以使用的内存管理单元;</p></li><li><p>从没有空闲对象的 <code>mSpan</code> 链表中查找可以使用的内存管理单元；</p></li><li><p>调用 <code>mcentral.grow</code> 从堆中申请新的内存管理单元；</p></li><li><p>更新内存管理单元的 <code>allocCache</code> 等字段帮助快速分配内存；</p></li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *mcentral)</span> <span class="hljs-title">cacheSpan</span><span class="hljs-params">()</span> *<span class="hljs-title">mspan</span></span> &#123;<span class="hljs-comment">// Deduct credit for this span allocation and sweep if necessary.</span>spanBytes := <span class="hljs-keyword">uintptr</span>(class_to_allocnpages[c.spanclass.sizeclass()]) * _PageSizedeductSweepCredit(spanBytes, <span class="hljs-number">0</span>)lock(&amp;c.lock)traceDone := <span class="hljs-literal">false</span><span class="hljs-keyword">if</span> trace.enabled &#123;traceGCSweepStart()&#125;<span class="hljs-comment">//mheap中的sweepgen</span>sg := mheap_.sweepgenretry:<span class="hljs-keyword">var</span> s *mspan<span class="hljs-comment">//从nonempty的list一一取出mSpan，判断其sweepgen</span><span class="hljs-keyword">for</span> s = c.nonempty.first; s != <span class="hljs-literal">nil</span>; s = s.next &#123;<span class="hljs-comment">//1. mSpan等待回收状态(mspan.sweepgen = h.sweepgen-2)</span><span class="hljs-keyword">if</span> s.sweepgen == sg<span class="hljs-number">-2</span> &amp;&amp; atomic.Cas(&amp;s.sweepgen, sg<span class="hljs-number">-2</span>, sg<span class="hljs-number">-1</span>) &#123;<span class="hljs-comment">//从nonempty中移除，因为是等待回收了，就肯定是属于空闲链表;</span>c.nonempty.remove(s)<span class="hljs-comment">//插入empty链表</span>c.empty.insertBack(s)unlock(&amp;c.lock)<span class="hljs-comment">//并且清理该mSpan</span>s.sweep(<span class="hljs-literal">true</span>)<span class="hljs-keyword">goto</span> havespan&#125;<span class="hljs-comment">//2. mspan.sweepgen=h.sweepgen-1, 正在被清除</span><span class="hljs-keyword">if</span> s.sweepgen == sg<span class="hljs-number">-1</span> &#123;<span class="hljs-comment">// the span is being swept by background sweeper, skip</span><span class="hljs-keyword">continue</span>&#125;<span class="hljs-comment">//3. mspan已经被清除</span><span class="hljs-comment">// we have a nonempty span that does not require sweeping, allocate from it</span>c.nonempty.remove(s)c.empty.insertBack(s)unlock(&amp;c.lock)<span class="hljs-keyword">goto</span> havespan&#125;....&#125;</code></pre></div><p>接下来如果<code>mcentral</code>没在<code>nonemtpy</code>中找到可用的mSpan，会继续遍历其<code>empty</code>链表,处理几乎一样(可以注意一下用到了<code>freeIndex</code>来快速判断有无可用<code>mSpan</code>);</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *mcentral)</span> <span class="hljs-title">cacheSpan</span><span class="hljs-params">()</span> *<span class="hljs-title">mspan</span></span> &#123;...retry:...<span class="hljs-keyword">for</span> s = c.empty.first; s != <span class="hljs-literal">nil</span>; s = s.next &#123;<span class="hljs-keyword">if</span> s.sweepgen == sg<span class="hljs-number">-2</span> &amp;&amp; atomic.Cas(&amp;s.sweepgen, sg<span class="hljs-number">-2</span>, sg<span class="hljs-number">-1</span>) &#123;<span class="hljs-comment">// we have an empty span that requires sweeping,</span><span class="hljs-comment">// sweep it and see if we can free some space in it</span>c.empty.remove(s)<span class="hljs-comment">// swept spans are at the end of the list</span>c.empty.insertBack(s)unlock(&amp;c.lock)s.sweep(<span class="hljs-literal">true</span>)freeIndex := s.nextFreeIndex()<span class="hljs-keyword">if</span> freeIndex != s.nelems &#123;s.freeindex = freeIndex<span class="hljs-keyword">goto</span> havespan&#125;lock(&amp;c.lock)<span class="hljs-comment">// the span is still empty after sweep</span><span class="hljs-comment">// it is already in the empty list, so just retry</span><span class="hljs-keyword">goto</span> retry&#125;<span class="hljs-keyword">if</span> s.sweepgen == sg<span class="hljs-number">-1</span> &#123;<span class="hljs-comment">// the span is being swept by background sweeper, skip</span><span class="hljs-keyword">continue</span>&#125;<span class="hljs-comment">// already swept empty span,</span><span class="hljs-comment">// all subsequent ones must also be either swept or in process of sweeping</span><span class="hljs-keyword">break</span>&#125;...&#125;</code></pre></div><p>如果在<code>nonempty</code>和<code>empty</code>都找不到可用的<code>mSpan</code>，就会触发申请内存<code>c.grow</code>;</p><p>最后只要有申请到可用的<code>mSpan</code>进入HaveSpan中，都会对<code>allocCache</code>,<code>allocBits</code>等字段进行更新;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *mcentral)</span> <span class="hljs-title">cacheSpan</span><span class="hljs-params">()</span> *<span class="hljs-title">mspan</span></span> &#123;...retry:...<span class="hljs-keyword">if</span> trace.enabled &#123;traceGCSweepDone()traceDone = <span class="hljs-literal">true</span>&#125;unlock(&amp;c.lock)<span class="hljs-comment">// Replenish central list if empty.</span><span class="hljs-comment">//申请内存</span>s = c.grow()<span class="hljs-keyword">if</span> s == <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>&#125;lock(&amp;c.lock)<span class="hljs-comment">//将其放入到empty链表中</span>c.empty.insertBack(s)unlock(&amp;c.lock)<span class="hljs-comment">// At this point s is a non-empty span, queued at the end of the empty list,</span><span class="hljs-comment">// c is unlocked.</span>havespan:<span class="hljs-keyword">if</span> trace.enabled &amp;&amp; !traceDone &#123;traceGCSweepDone()&#125;n := <span class="hljs-keyword">int</span>(s.nelems) - <span class="hljs-keyword">int</span>(s.allocCount)<span class="hljs-keyword">if</span> n == <span class="hljs-number">0</span> || s.freeindex == s.nelems || <span class="hljs-keyword">uintptr</span>(s.allocCount) == s.nelems &#123;throw(<span class="hljs-string">"span has no free objects"</span>)&#125;<span class="hljs-comment">// Assume all objects from this span will be allocated in the</span><span class="hljs-comment">// mcache. If it gets uncached, we'll adjust this.</span>atomic.Xadd64(&amp;c.nmalloc, <span class="hljs-keyword">int64</span>(n))usedBytes := <span class="hljs-keyword">uintptr</span>(s.allocCount) * s.elemsizeatomic.Xadd64(&amp;memstats.heap_live, <span class="hljs-keyword">int64</span>(spanBytes)-<span class="hljs-keyword">int64</span>(usedBytes))<span class="hljs-keyword">if</span> trace.enabled &#123;<span class="hljs-comment">// heap_live changed.</span>traceHeapAlloc()&#125;<span class="hljs-keyword">if</span> gcBlackenEnabled != <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// heap_live changed.</span>gcController.revise()&#125;freeByteBase := s.freeindex &amp;^ (<span class="hljs-number">64</span> - <span class="hljs-number">1</span>)whichByte := freeByteBase / <span class="hljs-number">8</span><span class="hljs-comment">// Init alloc bits cache.</span>s.refillAllocCache(whichByte)<span class="hljs-comment">// Adjust the allocCache so that s.freeindex corresponds to the low bit in</span><span class="hljs-comment">// s.allocCache.</span>s.allocCache &gt;&gt;= s.freeindex % <span class="hljs-number">64</span><span class="hljs-keyword">return</span> s&#125;</code></pre></div><p><code>mcentral.grow()</code>方法如下：</p><ol><li>会根据预先分配的<code>class_to_allocnpages</code>和<code>class_to_size</code>进行分配page或者spanClass对应的size;</li><li>并且调用<code>mheap.alloc</code>获取新的<code>mSpan</code></li><li>获取<code>mSpan</code>后，会初始化<code>mSpan.liimt</code>字段，并清除在<code>mheap</code>上的<code>bitmap</code></li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// grow allocates a new empty span from the heap and initializes it for c's size class.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *mcentral)</span> <span class="hljs-title">grow</span><span class="hljs-params">()</span> *<span class="hljs-title">mspan</span></span> &#123;npages := <span class="hljs-keyword">uintptr</span>(class_to_allocnpages[c.spanclass.sizeclass()])size := <span class="hljs-keyword">uintptr</span>(class_to_size[c.spanclass.sizeclass()])s := mheap_.alloc(npages, c.spanclass, <span class="hljs-literal">true</span>)<span class="hljs-keyword">if</span> s == <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>&#125;<span class="hljs-comment">// Use division by multiplication and shifts to quickly compute:</span><span class="hljs-comment">// n := (npages &lt;&lt; _PageShift) / size</span><span class="hljs-comment">//size =binary( s.divShift * uintptr(s.divMul) &gt;&gt; s.divShift2 )???todo</span>n := (npages &lt;&lt; _PageShift) &gt;&gt; s.divShift * <span class="hljs-keyword">uintptr</span>(s.divMul) &gt;&gt; s.divShift2s.limit = s.base() + size*nheapBitsForAddr(s.base()).initSpan(s)<span class="hljs-keyword">return</span> s&#125;</code></pre></div><h3 id="mheap">mheap:</h3><p>全局只有<strong>一个</strong>内存堆，主要可以关注<code>mcentral</code>和<code>arenas</code>相关字段;</p><p>以页为粒度(8KB，在64位上每个<code>heapArena</code>都会管理64MB内存)进行管理,</p><p>上面有列举出<code>mcentral</code>的结构体;其中<code>numSpanClasses</code>=134, 其中67个为scan的<code>mcentral</code>,另外67个为noscan;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//</span><span class="hljs-keyword">type</span> mheap <span class="hljs-keyword">struct</span>&#123;central [numSpanClasses]<span class="hljs-keyword">struct</span> &#123;mcentral mcentralpad      [cpu.CacheLinePadSize - unsafe.Sizeof(mcentral&#123;&#125;)%cpu.CacheLinePadSize]<span class="hljs-keyword">byte</span>&#125;&#125;</code></pre></div><p>其中<code>arenas</code>字段在不同OS上面有不同的长度;其长度采用的就是多级缓存思想来计算(todo???)</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> mheap <span class="hljs-keyword">struct</span>&#123;<span class="hljs-comment">// arenas is the heap arena map. It points to the metadata for</span><span class="hljs-comment">// the heap for every arena frame of the entire usable virtual</span><span class="hljs-comment">// address space.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Use arenaIndex to compute indexes into this array.</span><span class="hljs-comment">//</span><span class="hljs-comment">// For regions of the address space that are not backed by the</span><span class="hljs-comment">// Go heap, the arena map contains nil.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Modifications are protected by mheap_.lock. Reads can be</span><span class="hljs-comment">// performed without locking; however, a given entry can</span><span class="hljs-comment">// transition from nil to non-nil at any time when the lock</span><span class="hljs-comment">// isn't held. (Entries never transitions back to nil.)</span><span class="hljs-comment">//</span><span class="hljs-comment">// In general, this is a two-level mapping consisting of an L1</span><span class="hljs-comment">// map and possibly many L2 maps. This saves space when there</span><span class="hljs-comment">// are a huge number of arena frames. However, on many</span><span class="hljs-comment">// platforms (even 64-bit), arenaL1Bits is 0, making this</span><span class="hljs-comment">// effectively a single-level map. In this case, arenas[0]</span><span class="hljs-comment">// will never be nil.</span>arenas [<span class="hljs-number">1</span> &lt;&lt; arenaL1Bits]*[<span class="hljs-number">1</span> &lt;&lt; arenaL2Bits]*heapArena&#125;</code></pre></div><p>结构体为<code>treap</code>，维护空闲连续page，归还内存到heap中时，连续地址会合并；大于32KB内存申请直接从<code>mheap</code>中拿，剩下的则先使用当前P的<code>mcache</code>中对应的<code>size class</code>分配，如果其对应的span已经无可用的块，则向<code>mcentral</code>请求，如果没有则在mheap申请，如果还不够则要向操作系统申请;</p><h4 id="初始化-v2">初始化</h4><p>回到我们的初始化，终于可以从头开始进行：</p><ol><li>首先是一系列分配器,全部属于<code>fixAlloc</code>，都有<code>init</code>方法;可以参考<a href="#go%E7%9A%84%E5%88%86%E9%85%8D%E5%86%85%E5%AD%98%E7%AD%96%E7%95%A5">下一节</a>,对每一种都详细说明</li><li>还会在该方法中初始化所有的<code>mcentral</code>，这些<code>mcentral</code>会维护全局的内存管理单元，各个线程会通过<code>mcentral</code>获取新的内存单元。</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(h *mheap)</span> <span class="hljs-title">init</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-comment">//一系列alloc</span>h.spanalloc.init(unsafe.Sizeof(mspan&#123;&#125;), recordspan, unsafe.Pointer(h), &amp;memstats.mspan_sys)h.cachealloc.init(unsafe.Sizeof(mcache&#123;&#125;), <span class="hljs-literal">nil</span>, <span class="hljs-literal">nil</span>, &amp;memstats.mcache_sys)h.specialfinalizeralloc.init(unsafe.Sizeof(specialfinalizer&#123;&#125;), <span class="hljs-literal">nil</span>, <span class="hljs-literal">nil</span>, &amp;memstats.other_sys)h.specialprofilealloc.init(unsafe.Sizeof(specialprofile&#123;&#125;), <span class="hljs-literal">nil</span>, <span class="hljs-literal">nil</span>, &amp;memstats.other_sys)h.arenaHintAlloc.init(unsafe.Sizeof(arenaHint&#123;&#125;), <span class="hljs-literal">nil</span>, <span class="hljs-literal">nil</span>, &amp;memstats.other_sys)<span class="hljs-comment">// Don't zero mspan allocations. Background sweeping can</span><span class="hljs-comment">// inspect a span concurrently with allocating it, so it's</span><span class="hljs-comment">// important that the span's sweepgen survive across freeing</span><span class="hljs-comment">// and re-allocating a span to prevent background sweeping</span><span class="hljs-comment">// from improperly cas'ing it from 0.</span><span class="hljs-comment">//</span><span class="hljs-comment">// This is safe because mspan contains no heap pointers.</span>h.spanalloc.zero = <span class="hljs-literal">false</span><span class="hljs-comment">// h-&gt;mapcache needs no init</span><span class="hljs-keyword">for</span> i := <span class="hljs-keyword">range</span> h.central &#123;h.central[i].mcentral.init(spanClass(i))&#125;h.pages.init(&amp;h.lock, &amp;memstats.gc_sys)&#125;</code></pre></div><p>其中<code>mheap</code>自带一个<code>alloc</code>方法，在系统栈中获得新的<code>mSpan</code>:在分配n个页面时，为了防止内存大量占用和堆的增加，会检查<code>sweepdone</code>字段，然后<code>reclaim</code>至少n个pages;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// alloc allocates a new span of npage pages from the GC'd heap.</span><span class="hljs-comment">//</span><span class="hljs-comment">// spanclass indicates the span's size class and scannability.</span><span class="hljs-comment">//</span><span class="hljs-comment">// If needzero is true, the memory for the returned span will be zeroed.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(h *mheap)</span> <span class="hljs-title">alloc</span><span class="hljs-params">(npages <span class="hljs-keyword">uintptr</span>, spanclass spanClass, needzero <span class="hljs-keyword">bool</span>)</span> *<span class="hljs-title">mspan</span></span> &#123;<span class="hljs-comment">// Don't do any operations that lock the heap on the G stack.</span><span class="hljs-comment">// It might trigger stack growth, and the stack growth code needs</span><span class="hljs-comment">// to be able to allocate heap.</span><span class="hljs-keyword">var</span> s *mspansystemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-comment">// To prevent excessive heap growth, before allocating n pages</span><span class="hljs-comment">// we need to sweep and reclaim at least n pages.</span><span class="hljs-comment">//会检查回收n个pages</span><span class="hljs-keyword">if</span> h.sweepdone == <span class="hljs-number">0</span> &#123;h.reclaim(npages)&#125;s = h.allocSpan(npages, <span class="hljs-literal">false</span>, spanclass, &amp;memstats.heap_inuse)&#125;)<span class="hljs-keyword">if</span> s != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">if</span> needzero &amp;&amp; s.needzero != <span class="hljs-number">0</span> &#123;memclrNoHeapPointers(unsafe.Pointer(s.base()), s.npages&lt;&lt;_PageShift)&#125;s.needzero = <span class="hljs-number">0</span>&#125;<span class="hljs-keyword">return</span> s&#125;</code></pre></div><p>接下来在<code>mheap.allocSpan</code>方法中，大概步骤为:</p><ol><li><p>选择从堆上分配新的内存页和<code>mSpan</code>;</p></li><li><p>初始化<code>mSpan</code>并将其加入<code>mheap</code>的list</p></li></ol><ul><li>注意有小优化，当要求的npage较小的时候 <code>npages&lt;pageCahcePages/4</code>，会尝试<code>pages.AllocToCache()</code>拿pageCache，然后从拿到的page Cache申请内存;</li><li>较大的时候,就会直接用<code>pages.Alloc()</code>在页堆上拿内存;</li><li>内存不足时，会调用<code>mheap.grow</code>扩容并重新调用<code>pages.Alloc()</code>,不成功则说明OOM了，整个机器都没有内存，直接中止;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// allocSpan allocates an mspan which owns npages worth of memory.</span><span class="hljs-comment">//</span><span class="hljs-comment">// If manual == false, allocSpan allocates a heap span of class spanclass</span><span class="hljs-comment">// and updates heap accounting. If manual == true, allocSpan allocates a</span><span class="hljs-comment">// manually-managed span (spanclass is ignored), and the caller is</span><span class="hljs-comment">// responsible for any accounting related to its use of the span. Either</span><span class="hljs-comment">// way, allocSpan will atomically add the bytes in the newly allocated</span><span class="hljs-comment">// span to *sysStat.</span><span class="hljs-comment">//</span><span class="hljs-comment">// The returned span is fully initialized.</span><span class="hljs-comment">//</span><span class="hljs-comment">// h must not be locked.</span><span class="hljs-comment">//mheap不可以被锁？？？（初始化为什么不能被锁住???）</span><span class="hljs-comment">// allocSpan must be called on the system stack both because it acquires</span><span class="hljs-comment">// the heap lock and because it must block GC transitions.</span><span class="hljs-comment">// 但是这个allocSpan方法必须在systemstack内，因为其要用heap的锁以及要阻塞GC；</span><span class="hljs-comment">//go:systemstack</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(h *mheap)</span> <span class="hljs-title">allocSpan</span><span class="hljs-params">(npages <span class="hljs-keyword">uintptr</span>, manual <span class="hljs-keyword">bool</span>, spanclass spanClass, sysStat *<span class="hljs-keyword">uint64</span>)</span> <span class="hljs-params">(s *mspan)</span></span> &#123;<span class="hljs-comment">// Function-global state.</span>gp := getg()base, scav := <span class="hljs-keyword">uintptr</span>(<span class="hljs-number">0</span>), <span class="hljs-keyword">uintptr</span>(<span class="hljs-number">0</span>)<span class="hljs-comment">// If the allocation is small enough, try the page cache!</span>pp := gp.m.p.ptr()<span class="hljs-keyword">if</span> pp != <span class="hljs-literal">nil</span> &amp;&amp; npages &lt; pageCachePages/<span class="hljs-number">4</span> &#123;c := &amp;pp.pcache<span class="hljs-comment">// If the cache is empty, refill it.</span><span class="hljs-keyword">if</span> c.empty() &#123;lock(&amp;h.lock)*c = h.pages.allocToCache()unlock(&amp;h.lock)&#125;<span class="hljs-comment">// Try to allocate from the cache.</span>base, scav = c.alloc(npages)<span class="hljs-keyword">if</span> base != <span class="hljs-number">0</span> &#123;s = h.tryAllocMSpan()<span class="hljs-keyword">if</span> s != <span class="hljs-literal">nil</span> &amp;&amp; gcBlackenEnabled == <span class="hljs-number">0</span> &amp;&amp; (manual || spanclass.sizeclass() != <span class="hljs-number">0</span>) &#123;<span class="hljs-keyword">goto</span> HaveSpan&#125;<span class="hljs-comment">// We're either running duing GC, failed to acquire a mspan,</span><span class="hljs-comment">// or the allocation is for a large object. This means we</span><span class="hljs-comment">// have to lock the heap and do a bunch of extra work,</span><span class="hljs-comment">// so go down the HaveBaseLocked path.</span><span class="hljs-comment">//</span><span class="hljs-comment">// We must do this during GC to avoid skew with heap_scan</span><span class="hljs-comment">// since we flush mcache stats whenever we lock.</span><span class="hljs-comment">//</span><span class="hljs-comment">// TODO(mknyszek): It would be nice to not have to</span><span class="hljs-comment">// lock the heap if it's a large allocation, but</span><span class="hljs-comment">// it's fine for now. The critical section here is</span><span class="hljs-comment">// short and large object allocations are relatively</span><span class="hljs-comment">// infrequent.</span>&#125;&#125;<span class="hljs-comment">// For one reason or another, we couldn't get the</span><span class="hljs-comment">// whole job done without the heap lock.</span>lock(&amp;h.lock)<span class="hljs-keyword">if</span> base == <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// Try to acquire a base address.</span>base, scav = h.pages.alloc(npages)<span class="hljs-keyword">if</span> base == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> !h.grow(npages) &#123;unlock(&amp;h.lock)<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>&#125;base, scav = h.pages.alloc(npages)<span class="hljs-keyword">if</span> base == <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"grew heap, but no adequate free space found"</span>)&#125;&#125;&#125;<span class="hljs-keyword">if</span> s == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// We failed to get an mspan earlier, so grab</span><span class="hljs-comment">// one now that we have the heap lock.</span>s = h.allocMSpanLocked()&#125;...&#125;</code></pre></div><p>分配到<code>mSpan</code>后,都要对所有字段进行初始化,一些<code>freeindex</code> , <code>allocCache</code>, <code>gcmarkBits</code>等</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(h *mheap)</span> <span class="hljs-title">allocSpan</span><span class="hljs-params">(npages <span class="hljs-keyword">uintptr</span>, manual <span class="hljs-keyword">bool</span>, spanclass spanClass, sysStat *<span class="hljs-keyword">uint64</span>)</span> <span class="hljs-params">(s *mspan)</span></span> &#123;...HaveSpan:<span class="hljs-comment">// At this point, both s != nil and base != 0, and the heap</span><span class="hljs-comment">// lock is no longer held. Initialize the span.</span>s.init(base, npages)<span class="hljs-keyword">if</span> h.allocNeedsZero(base, npages) &#123;s.needzero = <span class="hljs-number">1</span>&#125;nbytes := npages * pageSize<span class="hljs-keyword">if</span> manual &#123;s.manualFreeList = <span class="hljs-number">0</span>s.nelems = <span class="hljs-number">0</span>s.limit = s.base() + s.npages*pageSize<span class="hljs-comment">// Manually managed memory doesn't count toward heap_sys.</span>mSysStatDec(&amp;memstats.heap_sys, s.npages*pageSize)s.state.set(mSpanManual)&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">// We must set span properties before the span is published anywhere</span><span class="hljs-comment">// since we're not holding the heap lock.</span>s.spanclass = spanclass<span class="hljs-keyword">if</span> sizeclass := spanclass.sizeclass(); sizeclass == <span class="hljs-number">0</span> &#123;s.elemsize = nbytess.nelems = <span class="hljs-number">1</span>s.divShift = <span class="hljs-number">0</span>s.divMul = <span class="hljs-number">0</span>s.divShift2 = <span class="hljs-number">0</span>s.baseMask = <span class="hljs-number">0</span>&#125; <span class="hljs-keyword">else</span> &#123;s.elemsize = <span class="hljs-keyword">uintptr</span>(class_to_size[sizeclass])s.nelems = nbytes / s.elemsizem := &amp;class_to_divmagic[sizeclass]s.divShift = m.shifts.divMul = m.muls.divShift2 = m.shift2s.baseMask = m.baseMask&#125;<span class="hljs-comment">// Initialize mark and allocation structures.</span>s.freeindex = <span class="hljs-number">0</span>s.allocCache = ^<span class="hljs-keyword">uint64</span>(<span class="hljs-number">0</span>) <span class="hljs-comment">// all 1s indicating all free.</span>s.gcmarkBits = newMarkBits(s.nelems)s.allocBits = newAllocBits(s.nelems)<span class="hljs-comment">// It's safe to access h.sweepgen without the heap lock because it's</span><span class="hljs-comment">// only ever updated with the world stopped and we run on the</span><span class="hljs-comment">// systemstack which blocks a STW transition.</span>atomic.Store(&amp;s.sweepgen, h.sweepgen)<span class="hljs-comment">// Now that the span is filled in, set its state. This</span><span class="hljs-comment">// is a publication barrier for the other fields in</span><span class="hljs-comment">// the span. While valid pointers into this span</span><span class="hljs-comment">// should never be visible until the span is returned,</span><span class="hljs-comment">// if the garbage collector finds an invalid pointer,</span><span class="hljs-comment">// access to the span may race with initialization of</span><span class="hljs-comment">// the span. We resolve this race by atomically</span><span class="hljs-comment">// setting the state after the span is fully</span><span class="hljs-comment">// initialized, and atomically checking the state in</span><span class="hljs-comment">// any situation where a pointer is suspect.</span>s.state.set(mSpanInUse)&#125;...&#125;</code></pre></div><ul><li>fixalloc</li></ul><p>一个不定长度的列表，用来管理<strong>不在堆上</strong>的固定的对象，这些对象都是runtime上大小固定的结构，比如mspan，mcache</p><ul><li><p>mstatisitc</p><p>提供管理信息</p></li></ul><h4 id="扩容">扩容</h4><p><code>mheap</code>的<code>grow()</code>方法扩容，会从堆中拿回最少的npage(期望)需要的内存，并返回成功与否，并且整个过程一定要被<code>锁住</code>;</p><ul><li>如果在当前的<code>arena</code>无足够的空间，会向OS要求分配更多的<code>arena</code>空间（但不保证连续，所以我们需要请求整个chunks）</li><li>接着会扩容<code>arena</code>区域，并更新页分配器的metadata</li><li>heap增长，要考虑回收一部分内存，在某些情况（？？？哪些）会调用<code>scavenge</code>回收一些空闲的内存页;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(h *mheap)</span> <span class="hljs-title">grow</span><span class="hljs-params">(npage <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-title">bool</span></span> &#123;<span class="hljs-comment">// We must grow the heap in whole palloc chunks.</span>ask := alignUp(npage, pallocChunkPages) * pageSizetotalGrowth := <span class="hljs-keyword">uintptr</span>(<span class="hljs-number">0</span>)nBase := alignUp(h.curArena.base+ask, physPageSize)<span class="hljs-keyword">if</span> nBase &gt; h.curArena.end &#123;<span class="hljs-comment">// Not enough room in the current arena. Allocate more</span><span class="hljs-comment">// arena space. This may not be contiguous with the</span><span class="hljs-comment">// current arena, so we have to request the full ask.</span>av, asize := h.sysAlloc(ask)<span class="hljs-keyword">if</span> av == <span class="hljs-literal">nil</span> &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"runtime: out of memory: cannot allocate "</span>, ask, <span class="hljs-string">"-byte block ("</span>, memstats.heap_sys, <span class="hljs-string">" in use)\n"</span>)<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;<span class="hljs-keyword">if</span> <span class="hljs-keyword">uintptr</span>(av) == h.curArena.end &#123;<span class="hljs-comment">// The new space is contiguous with the old</span><span class="hljs-comment">// space, so just extend the current space.</span>h.curArena.end = <span class="hljs-keyword">uintptr</span>(av) + asize&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">// The new space is discontiguous. Track what</span><span class="hljs-comment">// remains of the current space and switch to</span><span class="hljs-comment">// the new space. This should be rare.</span><span class="hljs-keyword">if</span> size := h.curArena.end - h.curArena.base; size != <span class="hljs-number">0</span> &#123;h.pages.grow(h.curArena.base, size)totalGrowth += size&#125;<span class="hljs-comment">// Switch to the new space.</span>h.curArena.base = <span class="hljs-keyword">uintptr</span>(av)h.curArena.end = <span class="hljs-keyword">uintptr</span>(av) + asize&#125;<span class="hljs-comment">// The memory just allocated counts as both released</span><span class="hljs-comment">// and idle, even though it's not yet backed by spans.</span><span class="hljs-comment">//</span><span class="hljs-comment">// The allocation is always aligned to the heap arena</span><span class="hljs-comment">// size which is always &gt; physPageSize, so its safe to</span><span class="hljs-comment">// just add directly to heap_released.</span>mSysStatInc(&amp;memstats.heap_released, asize)mSysStatInc(&amp;memstats.heap_idle, asize)<span class="hljs-comment">// Recalculate nBase</span>nBase = alignUp(h.curArena.base+ask, physPageSize)&#125;<span class="hljs-comment">// Grow into the current arena.</span>v := h.curArena.baseh.curArena.base = nBaseh.pages.grow(v, nBase-v)totalGrowth += nBase - v<span class="hljs-comment">// We just caused a heap growth, so scavenge down what will soon be used.</span><span class="hljs-comment">// By scavenging inline we deal with the failure to allocate out of</span><span class="hljs-comment">// memory fragments by scavenging the memory fragments that are least</span><span class="hljs-comment">// likely to be re-used.</span><span class="hljs-keyword">if</span> retained := heapRetained(); retained+<span class="hljs-keyword">uint64</span>(totalGrowth) &gt; h.scavengeGoal &#123;todo := totalGrowth<span class="hljs-keyword">if</span> overage := <span class="hljs-keyword">uintptr</span>(retained + <span class="hljs-keyword">uint64</span>(totalGrowth) - h.scavengeGoal); todo &gt; overage &#123;todo = overage&#125;h.pages.scavenge(todo, <span class="hljs-literal">true</span>)&#125;<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>&#125;</code></pre></div><p>其中<code>sysAlloc</code>是尝试申请虚拟内存，大概分为以下步骤：</p><ol><li>先在预留的区域申请内存,这里会使用<code>linearAlloc.alloc</code>方法;</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(h *mheap)</span> <span class="hljs-title">sysAlloc</span><span class="hljs-params">(n <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-params">(v unsafe.Pointer, size <span class="hljs-keyword">uintptr</span>)</span></span> &#123;n = alignUp(n, heapArenaBytes)<span class="hljs-comment">// 1. First, try the arena pre-reservation.</span>v = h.arena.alloc(n, heapArenaBytes, &amp;memstats.heap_sys)<span class="hljs-keyword">if</span> v != <span class="hljs-literal">nil</span> &#123;size = n<span class="hljs-keyword">goto</span> mapped&#125;...</code></pre></div><ol start="2"><li>然后尝试用拿回的值预留的内存中申请一块可以使用的空间,如果无可用空间，就会根据<code>arenaHints</code>在目标地址上尝试扩容;</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(h *mheap)</span> <span class="hljs-title">sysAlloc</span><span class="hljs-params">(n <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-params">(v unsafe.Pointer, size <span class="hljs-keyword">uintptr</span>)</span></span> &#123;...<span class="hljs-comment">// Try to grow the heap at a hint address.</span><span class="hljs-keyword">for</span> h.arenaHints != <span class="hljs-literal">nil</span> &#123;hint := h.arenaHintsp := hint.addr<span class="hljs-keyword">if</span> hint.down &#123;p -= n&#125;<span class="hljs-keyword">if</span> p+n &lt; p &#123;<span class="hljs-comment">// We can't use this, so don't ask.</span>v = <span class="hljs-literal">nil</span>&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> arenaIndex(p+n<span class="hljs-number">-1</span>) &gt;= <span class="hljs-number">1</span>&lt;&lt;arenaBits &#123;<span class="hljs-comment">// Outside addressable heap. Can't use.</span>v = <span class="hljs-literal">nil</span>&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//</span>v = sysReserve(unsafe.Pointer(p), n)&#125;<span class="hljs-keyword">if</span> p == <span class="hljs-keyword">uintptr</span>(v) &#123;<span class="hljs-comment">// Success. Update the hint.</span><span class="hljs-keyword">if</span> !hint.down &#123;p += n&#125;hint.addr = psize = n<span class="hljs-keyword">break</span>&#125;<span class="hljs-comment">// Failed. Discard this hint and try the next.</span><span class="hljs-comment">//</span><span class="hljs-comment">// <span class="hljs-doctag">TODO:</span> This would be cleaner if sysReserve could be</span><span class="hljs-comment">// told to only return the requested address. In</span><span class="hljs-comment">// particular, this is already how Windows behaves, so</span><span class="hljs-comment">// it would simplify things there.</span><span class="hljs-keyword">if</span> v != <span class="hljs-literal">nil</span> &#123;sysFree(v, n, <span class="hljs-literal">nil</span>)&#125;h.arenaHints = hint.nexth.arenaHintAlloc.free(unsafe.Pointer(hint))&#125;<span class="hljs-keyword">if</span> size == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> raceenabled &#123;<span class="hljs-comment">// The race detector assumes the heap lives in</span><span class="hljs-comment">// [0x00c000000000, 0x00e000000000), but we</span><span class="hljs-comment">// just ran out of hints in this region. Give</span><span class="hljs-comment">// a nice failure.</span>throw(<span class="hljs-string">"too many address space collisions for -race mode"</span>)&#125;<span class="hljs-comment">// All of the hints failed, so we'll take any</span><span class="hljs-comment">// (sufficiently aligned) address the kernel will give</span><span class="hljs-comment">// us.</span>v, size = sysReserveAligned(<span class="hljs-literal">nil</span>, n, heapArenaBytes)<span class="hljs-keyword">if</span> v == <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, <span class="hljs-number">0</span>&#125;<span class="hljs-comment">// Create new hints for extending this region.</span>hint := (*arenaHint)(h.arenaHintAlloc.alloc())hint.addr, hint.down = <span class="hljs-keyword">uintptr</span>(v), <span class="hljs-literal">true</span>hint.next, mheap_.arenaHints = mheap_.arenaHints, hinthint = (*arenaHint)(h.arenaHintAlloc.alloc())hint.addr = <span class="hljs-keyword">uintptr</span>(v) + sizehint.next, mheap_.arenaHints = mheap_.arenaHints, hint&#125;...<span class="hljs-comment">//从Reserved状态到Prepared状态</span>sysMap(v, size, &amp;memstats.heap_sys)...&#125;</code></pre></div><ol start="3"><li>到达mapped状态后，会初始化一个新的<code>heapArena</code>来管理刚刚申请的内存空间,该结构体会被加入页堆的二维数组中</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(h *mheap)</span> <span class="hljs-title">sysAlloc</span><span class="hljs-params">(n <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-params">(v unsafe.Pointer, size <span class="hljs-keyword">uintptr</span>)</span></span> &#123;...mapped:<span class="hljs-comment">// Create arena metadata.</span><span class="hljs-keyword">for</span> ri := arenaIndex(<span class="hljs-keyword">uintptr</span>(v)); ri &lt;= arenaIndex(<span class="hljs-keyword">uintptr</span>(v)+size<span class="hljs-number">-1</span>); ri++ &#123;l2 := h.arenas[ri.l1()]<span class="hljs-keyword">if</span> l2 == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// Allocate an L2 arena map.</span>l2 = (*[<span class="hljs-number">1</span> &lt;&lt; arenaL2Bits]*heapArena)(persistentalloc(unsafe.Sizeof(*l2), sys.PtrSize, <span class="hljs-literal">nil</span>))<span class="hljs-keyword">if</span> l2 == <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"out of memory allocating heap arena map"</span>)&#125;atomic.StorepNoWB(unsafe.Pointer(&amp;h.arenas[ri.l1()]), unsafe.Pointer(l2))&#125;<span class="hljs-keyword">if</span> l2[ri.l2()] != <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"arena already initialized"</span>)&#125;<span class="hljs-keyword">var</span> r *heapArenar = (*heapArena)(h.heapArenaAlloc.alloc(unsafe.Sizeof(*r), sys.PtrSize, &amp;memstats.gc_sys))<span class="hljs-keyword">if</span> r == <span class="hljs-literal">nil</span> &#123;r = (*heapArena)(persistentalloc(unsafe.Sizeof(*r), sys.PtrSize, &amp;memstats.gc_sys))<span class="hljs-keyword">if</span> r == <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"out of memory allocating heap arena metadata"</span>)&#125;&#125;<span class="hljs-comment">// Add the arena to the arenas list.</span><span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(h.allArenas) == <span class="hljs-built_in">cap</span>(h.allArenas) &#123;size := <span class="hljs-number">2</span> * <span class="hljs-keyword">uintptr</span>(<span class="hljs-built_in">cap</span>(h.allArenas)) * sys.PtrSize<span class="hljs-keyword">if</span> size == <span class="hljs-number">0</span> &#123;size = physPageSize&#125;newArray := (*notInHeap)(persistentalloc(size, sys.PtrSize, &amp;memstats.gc_sys))<span class="hljs-keyword">if</span> newArray == <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"out of memory allocating allArenas"</span>)&#125;oldSlice := h.allArenas*(*notInHeapSlice)(unsafe.Pointer(&amp;h.allArenas)) = notInHeapSlice&#123;newArray, <span class="hljs-built_in">len</span>(h.allArenas), <span class="hljs-keyword">int</span>(size / sys.PtrSize)&#125;<span class="hljs-built_in">copy</span>(h.allArenas, oldSlice)<span class="hljs-comment">// Do not free the old backing array because</span><span class="hljs-comment">// there may be concurrent readers. Since we</span><span class="hljs-comment">// double the array each time, this can lead</span><span class="hljs-comment">// to at most 2x waste.</span>&#125;<span class="hljs-comment">//保存每个mapped的arena的index;</span>h.allArenas = h.allArenas[:<span class="hljs-built_in">len</span>(h.allArenas)+<span class="hljs-number">1</span>]h.allArenas[<span class="hljs-built_in">len</span>(h.allArenas)<span class="hljs-number">-1</span>] = ri<span class="hljs-comment">// Store atomically just in case an object from the</span><span class="hljs-comment">// new heap arena becomes visible before the heap lock</span><span class="hljs-comment">// is released (which shouldn't happen, but there's</span><span class="hljs-comment">// little downside to this).</span>atomic.StorepNoWB(unsafe.Pointer(&amp;l2[ri.l2()]), unsafe.Pointer(r))&#125;&#125;</code></pre></div><h2 id="内存策略：">内存策略：</h2><p>堆上的所有对象都会通过<code>runtime.newobject</code>方法分配内存，该方法会调用<code>mallocgc</code>分配指定内存大小的空间:其中申请的大小类型不同会有不同行为:</p><h3 id="小对象-小于32k-和大对象的不同">小对象（小于32K）和大对象的不同</h3><ol><li>小于32KB的小对象时，会直接去 P (process)的cache的list里面拿，大于32KB的才去堆拿;拿的过程会roundup一下大小，然后在当前P的mcachexmspan</li></ol><div class="hljs"><pre><code class="hljs golang"><span class="hljs-comment">// Allocate an object of size bytes.</span><span class="hljs-comment">// Small objects are allocated from the per-P cache's free lists.</span><span class="hljs-comment">// Large objects (&gt; 32 kB) are allocated straight from the heap.</span><span class="hljs-comment">//typ有两种，一种noscan，一种是scan，表示分配对象是否包含指针</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">mallocgc</span><span class="hljs-params">(size <span class="hljs-keyword">uintptr</span>, typ *_type, needzero <span class="hljs-keyword">bool</span>)</span> <span class="hljs-title">unsafe</span>.<span class="hljs-title">Pointer</span></span> &#123;<span class="hljs-keyword">if</span> gcphase == _GCmarktermination &#123;throw(<span class="hljs-string">"mallocgc called with gcphase == _GCmarktermination"</span>)&#125;<span class="hljs-keyword">if</span> size == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span> unsafe.Pointer(&amp;zerobase)&#125;......<span class="hljs-comment">// Set mp.mallocing to keep from being preempted by GC.</span><span class="hljs-comment">//设置状态位，为正在分配mallocing,防止被GC抢占</span>mp := acquirem()<span class="hljs-keyword">if</span> mp.mallocing != <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"malloc deadlock"</span>)&#125;<span class="hljs-keyword">if</span> mp.gsignal == getg() &#123;throw(<span class="hljs-string">"malloc during signal"</span>)&#125;mp.mallocing = <span class="hljs-number">1</span>shouldhelpgc := <span class="hljs-literal">false</span>dataSize := sizec := gomcache()<span class="hljs-keyword">var</span> x unsafe.Pointernoscan := typ == <span class="hljs-literal">nil</span> || typ.ptrdata == <span class="hljs-number">0</span><span class="hljs-keyword">if</span> size &lt;= maxSmallSize &#123;...<span class="hljs-comment">//tiny对象与小对象的分配，见下一节</span>&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//大对象</span><span class="hljs-keyword">var</span> s *mspanshouldhelpgc = <span class="hljs-literal">true</span>systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;s = largeAlloc(size, needzero, noscan)&#125;)s.freeindex = <span class="hljs-number">1</span>s.allocCount = <span class="hljs-number">1</span>x = unsafe.Pointer(s.base())size = s.elemsize&#125;<span class="hljs-keyword">var</span> scanSize <span class="hljs-keyword">uintptr</span><span class="hljs-keyword">if</span> !noscan &#123;<span class="hljs-comment">// If allocating a defer+arg block, now that we've picked a malloc size</span><span class="hljs-comment">// large enough to hold everything, cut the "asked for" size down to</span><span class="hljs-comment">// just the defer header, so that the GC bitmap will record the arg block</span><span class="hljs-comment">// as containing nothing at all (as if it were unused space at the end of</span><span class="hljs-comment">// a malloc block caused by size rounding).</span><span class="hljs-comment">// The defer arg areas are scanned as part of scanstack.</span><span class="hljs-keyword">if</span> typ == deferType &#123;dataSize = unsafe.Sizeof(_defer&#123;&#125;)&#125;heapBitsSetType(<span class="hljs-keyword">uintptr</span>(x), size, dataSize, typ)<span class="hljs-keyword">if</span> dataSize &gt; typ.size &#123;<span class="hljs-comment">// Array allocation. If there are any</span><span class="hljs-comment">// pointers, GC has to scan to the last</span><span class="hljs-comment">// element.</span><span class="hljs-keyword">if</span> typ.ptrdata != <span class="hljs-number">0</span> &#123;scanSize = dataSize - typ.size + typ.ptrdata&#125;&#125; <span class="hljs-keyword">else</span> &#123;scanSize = typ.ptrdata&#125;c.local_scan += scanSize&#125;<span class="hljs-comment">// Ensure that the stores above that initialize x to</span><span class="hljs-comment">// type-safe memory and set the heap bits occur before</span><span class="hljs-comment">// the caller can make x observable to the garbage</span><span class="hljs-comment">// collector. Otherwise, on weakly ordered machines,</span><span class="hljs-comment">// the garbage collector could follow a pointer to x,</span><span class="hljs-comment">// but see uninitialized memory or stale heap bits.</span>publicationBarrier()<span class="hljs-comment">// Allocate black during GC.</span><span class="hljs-comment">// All slots hold nil so no scanning is needed.</span><span class="hljs-comment">// This may be racing with GC so do it atomically if there can be</span><span class="hljs-comment">// a race marking the bit.</span><span class="hljs-keyword">if</span> gcphase != _GCoff &#123;gcmarknewobject(<span class="hljs-keyword">uintptr</span>(x), size, scanSize)&#125;<span class="hljs-keyword">if</span> raceenabled &#123;racemalloc(x, size)&#125;<span class="hljs-keyword">if</span> msanenabled &#123;msanmalloc(x, size)&#125;mp.mallocing = <span class="hljs-number">0</span>releasem(mp)<span class="hljs-keyword">if</span> debug.allocfreetrace != <span class="hljs-number">0</span> &#123;tracealloc(x, size, typ)&#125;<span class="hljs-keyword">if</span> rate := MemProfileRate; rate &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> rate != <span class="hljs-number">1</span> &amp;&amp; size &lt; c.next_sample &#123;c.next_sample -= size&#125; <span class="hljs-keyword">else</span> &#123;mp := acquirem()profilealloc(mp, x, size)releasem(mp)&#125;&#125;...<span class="hljs-keyword">return</span> x&#125;</code></pre></div><h3 id="各种分配器">各种分配器</h3><h4 id="tinyallocator">TinyAllocator</h4><p>上面的源代码中，当size&lt;=maxSmallSize时，实际上调用了一种tinyAllocator的分配器</p><p>该分配是 <strong>会结合多个申请内存请求 成为 申请一个内存块的请求(即合并小对象存储下来)</strong>；所以当所有的子对象都不可达时，这个内存块才会被释放（同时这些对象一定不含指针，这个很好理解，有指针又要从指针处进行可达性查询）作用是<strong>避免可能内存浪费</strong></p><p>其中这个maxTinySize是可调整的，调整成8bytes就一定不会浪费任何内存（一个页就8B），但是这样就没什么意义（不用combine）照这个道理，16bytes就可能会造成2× 最坏情况的内存浪费，32B就会造成4×最坏情况的内存浪费</p><ul><li><p>其中从tinyallocator分配的对象不能被显式释放(?)，所以每次我们显式释放对象的时候，自然的要保证这个对象是大于maxTinySize</p></li><li><p>主要的对象是一些小字符串和一些独立的变量，json的benchmark使用了这个分配器，减少了20%的堆size</p></li></ul><p>其结构<img src="/img/tinyObjects.png" srcset="/img/loading.gif" alt="如图"></p><ul><li>每个P在本地维护了专门的memory block来存储tinyObject，分配时根据tinyoffset和需要的size及对齐来判断该block是否容纳该object，如果可以就返回地址</li></ul><p>大概流程为:</p><ul><li><p>offset要进行roundup,<code>mcache</code>的<code>tiny</code>字段实际指向了<code>maxTinySize</code>大小的块，块中如果还有合适的内存，会通过基地址和位移来获取这块内存;</p></li><li><p>当内存块不含有合适的内存时，就会从<code>spanClass</code>获得对应的<code>mspan</code>，然后调用<code>runtime.nextFreeIndex</code>方法获得空闲内存；如果空闲内存无法获得，会接着调用<code>mcache.nextFree</code>方法从<code>mcentral</code>或者<code>mheap</code>中获取;</p></li><li><p>获得空闲内存块后，会用<code>runtime.memclrNoHeapPointers</code>清空空闲内存中的数据，更新<code>tiny</code>和<code>tinyoffset</code>并返回新内存块</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">mallocgc</span><span class="hljs-params">(size <span class="hljs-keyword">uintptr</span>, typ *_type, needzero <span class="hljs-keyword">bool</span>)</span> <span class="hljs-title">unsafe</span>.<span class="hljs-title">Pointer</span></span> &#123;....<span class="hljs-comment">//maxSmallsize = 32786</span><span class="hljs-keyword">if</span> size&lt;=maxSmallSize &#123;<span class="hljs-comment">//申请对象不含指针(noscan)且小于16B则会调用tiny allocator</span><span class="hljs-keyword">if</span> noscan &amp;&amp; size &lt; maxTinySize &#123;<span class="hljs-comment">// Tiny allocator.</span><span class="hljs-comment">//</span><span class="hljs-comment">//一种分配器</span><span class="hljs-comment">// Tiny allocator combines several tiny allocation requests</span><span class="hljs-comment">// into a single memory block. The resulting memory block</span><span class="hljs-comment">// is freed when all subobjects are unreachable. The subobjects</span><span class="hljs-comment">// must be noscan (don't have pointers), this ensures that</span><span class="hljs-comment">// the amount of potentially wasted memory is bounded.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Size of the memory block used for combining (maxTinySize) is tunable.</span><span class="hljs-comment">// Current setting is 16 bytes, which relates to 2x worst case memory</span><span class="hljs-comment">// wastage (when all but one subobjects are unreachable).</span><span class="hljs-comment">// 8 bytes would result in no wastage at all, but provides less</span><span class="hljs-comment">// opportunities for combining.</span><span class="hljs-comment">// 32 bytes provides more opportunities for combining,</span><span class="hljs-comment">// but can lead to 4x worst case wastage.</span><span class="hljs-comment">// The best case winning is 8x regardless of block size.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Objects obtained from tiny allocator must not be freed explicitly.</span><span class="hljs-comment">// So when an object will be freed explicitly, we ensure that</span><span class="hljs-comment">// its size &gt;= maxTinySize.</span><span class="hljs-comment">//</span><span class="hljs-comment">// SetFinalizer has a special case for objects potentially coming</span><span class="hljs-comment">// from tiny allocator, it such case it allows to set finalizers</span><span class="hljs-comment">// for an inner byte of a memory block.</span><span class="hljs-comment">//</span><span class="hljs-comment">// The main targets of tiny allocator are small strings and</span><span class="hljs-comment">// standalone escaping variables. On a json benchmark</span><span class="hljs-comment">// the allocator reduces number of allocations by ~12% and</span><span class="hljs-comment">// reduces heap size by ~20%.</span>off := c.tinyoffset<span class="hljs-comment">// Align tiny pointer for required (conservative) alignment.</span><span class="hljs-keyword">if</span> size&amp;<span class="hljs-number">7</span> == <span class="hljs-number">0</span> &#123;off = round(off, <span class="hljs-number">8</span>)&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> size&amp;<span class="hljs-number">3</span> == <span class="hljs-number">0</span> &#123;off = round(off, <span class="hljs-number">4</span>)&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> size&amp;<span class="hljs-number">1</span> == <span class="hljs-number">0</span> &#123;off = round(off, <span class="hljs-number">2</span>)&#125;<span class="hljs-comment">//这里发现还有剩余空间</span><span class="hljs-keyword">if</span> off+size &lt;= maxTinySize &amp;&amp; c.tiny != <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// The object fits into existing tiny block.</span><span class="hljs-comment">//该对象适合于已有的tinyblock</span><span class="hljs-comment">//移位获得剩余空间地址</span>x = unsafe.Pointer(c.tiny + off)c.tinyoffset = off + sizec.local_tinyallocs++mp.mallocing = <span class="hljs-number">0</span>releasem(mp)<span class="hljs-keyword">return</span> x&#125;<span class="hljs-comment">// Allocate a new maxTinySize block.</span>span := c.alloc[tinySpanClass]<span class="hljs-comment">//从mspan的alloccache，快速获取</span>v := nextFreeFast(span)<span class="hljs-keyword">if</span> v == <span class="hljs-number">0</span> &#123;<span class="hljs-comment">//从mcentral或mheap获取;</span>v, _, shouldhelpgc = c.nextFree(tinySpanClass)&#125;<span class="hljs-comment">//获得后清空内存块</span>x = unsafe.Pointer(v)(*[<span class="hljs-number">2</span>]<span class="hljs-keyword">uint64</span>)(x)[<span class="hljs-number">0</span>] = <span class="hljs-number">0</span>(*[<span class="hljs-number">2</span>]<span class="hljs-keyword">uint64</span>)(x)[<span class="hljs-number">1</span>] = <span class="hljs-number">0</span><span class="hljs-comment">// See if we need to replace the existing tiny block with the new one</span><span class="hljs-comment">// based on amount of remaining free space.</span><span class="hljs-comment">//将tiny和tinyoffset放回内存块</span><span class="hljs-keyword">if</span> size &lt; c.tinyoffset || c.tiny == <span class="hljs-number">0</span> &#123;c.tiny = <span class="hljs-keyword">uintptr</span>(x)c.tinyoffset = size&#125;size = maxTinySize&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//在32KB以内，16B以上的小对象</span><span class="hljs-keyword">var</span> sizeclass <span class="hljs-keyword">uint8</span><span class="hljs-comment">//smallSizeMax = 1024</span><span class="hljs-keyword">if</span> size &lt;= smallSizeMax<span class="hljs-number">-8</span> &#123;sizeclass = size_to_class8[(size+smallSizeDiv<span class="hljs-number">-1</span>)/smallSizeDiv]&#125; <span class="hljs-keyword">else</span> &#123;sizeclass = size_to_class128[(size-smallSizeMax+largeSizeDiv<span class="hljs-number">-1</span>)/largeSizeDiv]&#125;size = <span class="hljs-keyword">uintptr</span>(class_to_size[sizeclass])spc := makeSpanClass(sizeclass, noscan)span := c.alloc[spc]<span class="hljs-comment">//大同小异，先从mspan</span>v := nextFreeFast(span)<span class="hljs-keyword">if</span> v == <span class="hljs-number">0</span> &#123;<span class="hljs-comment">//再从mcentral和mheap</span>v, span, shouldhelpgc = c.nextFree(spc)&#125;x = unsafe.Pointer(v)<span class="hljs-keyword">if</span> needzero &amp;&amp; span.needzero != <span class="hljs-number">0</span> &#123;memclrNoHeapPointers(unsafe.Pointer(v), size)&#125;&#125;&#125;<span class="hljs-keyword">else</span>&#123;....&#125;....&#125;</code></pre></div><p>以上<code>nextFreeFast</code>从<code>mSpan</code>取空闲空间</p><ul><li>使用<code>mspan.allocCache</code>字段快速计算是否有空闲对象</li><li>如果有，取出并更新<code>mspan.freeIndex</code>和<code>mspan.allocCache</code></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// nextFreeFast returns the next free object if one is quickly available.</span><span class="hljs-comment">// Otherwise it returns 0.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">nextFreeFast</span><span class="hljs-params">(s *mspan)</span> <span class="hljs-title">gclinkptr</span></span> &#123;<span class="hljs-comment">//从右边(low-order)开始数0的个数,都是0就返回64</span>theBit := sys.Ctz64(s.allocCache) <span class="hljs-comment">// Is there a free object in the allocCache?</span><span class="hljs-keyword">if</span> theBit &lt; <span class="hljs-number">64</span> &#123;result := s.freeindex + <span class="hljs-keyword">uintptr</span>(theBit)<span class="hljs-keyword">if</span> result &lt; s.nelems &#123;freeidx := result + <span class="hljs-number">1</span><span class="hljs-keyword">if</span> freeidx%<span class="hljs-number">64</span> == <span class="hljs-number">0</span> &amp;&amp; freeidx != s.nelems &#123;<span class="hljs-keyword">return</span> <span class="hljs-number">0</span>&#125;s.allocCache &gt;&gt;= <span class="hljs-keyword">uint</span>(theBit + <span class="hljs-number">1</span>)s.freeindex = freeidxs.allocCount++<span class="hljs-keyword">return</span> gclinkptr(result*s.elemsize + s.base())&#125;&#125;<span class="hljs-keyword">return</span> <span class="hljs-number">0</span>&#125;</code></pre></div><p><code>nextFree()</code>方法,从<code>mcache</code>,<code>mcentral</code>,<code>mheap</code>取空间:</p><ul><li>要在不被抢占的上下文环境运行，因为<code>mcache</code>可能被其他P抢占;</li><li>其实还是要先从<code>mSpan</code>通过<code>nextFreeIndex</code>检查有无可用对象</li><li>无的话，会调用<code>mcache.refill</code>从<code>mcentral</code>拿缓存(详细可见前几节),然后更新<code>freeIndex</code>为接下来判断是否真的拿到了准确值;</li><li></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// nextFree returns the next free object from the cached span if one is available.</span><span class="hljs-comment">// Otherwise it refills the cache with a span with an available object and</span><span class="hljs-comment">// returns that object along with a flag indicating that this was a heavy</span><span class="hljs-comment">// weight allocation. If it is a heavy weight allocation the caller must</span><span class="hljs-comment">// determine whether a new GC cycle needs to be started or if the GC is active</span><span class="hljs-comment">// whether this goroutine needs to assist the GC.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Must run in a non-preemptible context since otherwise the owner of</span><span class="hljs-comment">// c could change.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *mcache)</span> <span class="hljs-title">nextFree</span><span class="hljs-params">(spc spanClass)</span> <span class="hljs-params">(v gclinkptr, s *mspan, shouldhelpgc <span class="hljs-keyword">bool</span>)</span></span> &#123;s = c.alloc[spc]shouldhelpgc = <span class="hljs-literal">false</span>freeIndex := s.nextFreeIndex()<span class="hljs-keyword">if</span> freeIndex == s.nelems &#123;<span class="hljs-comment">// The span is full.</span><span class="hljs-keyword">if</span> <span class="hljs-keyword">uintptr</span>(s.allocCount) != s.nelems &#123;<span class="hljs-built_in">println</span>(<span class="hljs-string">"runtime: s.allocCount="</span>, s.allocCount, <span class="hljs-string">"s.nelems="</span>, s.nelems)throw(<span class="hljs-string">"s.allocCount != s.nelems &amp;&amp; freeIndex == s.nelems"</span>)&#125;c.refill(spc)shouldhelpgc = <span class="hljs-literal">true</span>s = c.alloc[spc]freeIndex = s.nextFreeIndex()&#125;<span class="hljs-keyword">if</span> freeIndex &gt;= s.nelems &#123;throw(<span class="hljs-string">"freeIndex is not valid"</span>)&#125;v = gclinkptr(freeIndex*s.elemsize + s.base())s.allocCount++<span class="hljs-keyword">if</span> <span class="hljs-keyword">uintptr</span>(s.allocCount) &gt; s.nelems &#123;<span class="hljs-built_in">println</span>(<span class="hljs-string">"s.allocCount="</span>, s.allocCount, <span class="hljs-string">"s.nelems="</span>, s.nelems)throw(<span class="hljs-string">"s.allocCount &gt; s.nelems"</span>)&#125;<span class="hljs-keyword">return</span>&#125;</code></pre></div><h4 id="大对象分配">大对象分配</h4><p>接上一节，大对象的分配会调用<code>largeAlloc</code>方法</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">mallocgc</span><span class="hljs-params">(size <span class="hljs-keyword">uintptr</span>, typ *_type, needzero <span class="hljs-keyword">bool</span>)</span> <span class="hljs-title">unsafe</span>.<span class="hljs-title">Pointer</span></span> &#123;...<span class="hljs-keyword">if</span> size &lt;= maxSmallSize &#123;....<span class="hljs-comment">//小对象分配</span>&#125;<span class="hljs-keyword">else</span>&#123;<span class="hljs-keyword">var</span> s *mspanshouldhelpgc = <span class="hljs-literal">true</span>systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;s = largeAlloc(size, needzero, noscan)&#125;)s.freeindex = <span class="hljs-number">1</span>s.allocCount = <span class="hljs-number">1</span>x = unsafe.Pointer(s.base())size = s.elemsize&#125;...<span class="hljs-comment">// Ensure that the stores above that initialize x to</span><span class="hljs-comment">// type-safe memory and set the heap bits occur before</span><span class="hljs-comment">// the caller can make x observable to the garbage</span><span class="hljs-comment">// collector. Otherwise, on weakly ordered machines,</span><span class="hljs-comment">// the garbage collector could follow a pointer to x,</span><span class="hljs-comment">// but see uninitialized memory or stale heap bits.</span>publicationBarrier()<span class="hljs-comment">// Allocate black during GC.</span><span class="hljs-comment">// All slots hold nil so no scanning is needed.</span><span class="hljs-comment">// This may be racing with GC so do it atomically if there can be</span><span class="hljs-comment">// a race marking the bit.</span><span class="hljs-keyword">if</span> gcphase != _GCoff &#123;gcmarknewobject(<span class="hljs-keyword">uintptr</span>(x), size, scanSize)&#125;<span class="hljs-keyword">if</span> raceenabled &#123;racemalloc(x, size)&#125;<span class="hljs-keyword">if</span> msanenabled &#123;msanmalloc(x, size)&#125;mp.mallocing = <span class="hljs-number">0</span>releasem(mp)<span class="hljs-keyword">if</span> debug.allocfreetrace != <span class="hljs-number">0</span> &#123;tracealloc(x, size, typ)&#125;<span class="hljs-keyword">if</span> rate := MemProfileRate; rate &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> rate != <span class="hljs-number">1</span> &amp;&amp; size &lt; c.next_sample &#123;c.next_sample -= size&#125; <span class="hljs-keyword">else</span> &#123;mp := acquirem()profilealloc(mp, x, size)releasem(mp)&#125;&#125;<span class="hljs-keyword">if</span> assistG != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// Account for internal fragmentation in the assist</span><span class="hljs-comment">// debt now that we know it.</span>assistG.gcAssistBytes -= <span class="hljs-keyword">int64</span>(size - dataSize)&#125;<span class="hljs-keyword">if</span> shouldhelpgc &#123;<span class="hljs-keyword">if</span> t := (gcTrigger&#123;kind: gcTriggerHeap&#125;); t.test() &#123;gcStart(t)&#125;&#125;&#125;</code></pre></div><p>其中<code>largeAlloc</code>函数会计算所需要的页数，计算成8KB的倍数为对象在堆上申请内存:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">largeAlloc</span><span class="hljs-params">(size <span class="hljs-keyword">uintptr</span>, needzero <span class="hljs-keyword">bool</span>, noscan <span class="hljs-keyword">bool</span>)</span> *<span class="hljs-title">mspan</span></span> &#123;<span class="hljs-comment">// print("largeAlloc size=", size, "\n")</span><span class="hljs-keyword">if</span> size+_PageSize &lt; size &#123;throw(<span class="hljs-string">"out of memory"</span>)&#125;npages := size &gt;&gt; _PageShift<span class="hljs-comment">//计算页数,PageMask= 8191,一个页8KB，不够就往上取整</span><span class="hljs-keyword">if</span> size&amp;_PageMask != <span class="hljs-number">0</span> &#123;npages++&#125;<span class="hljs-comment">// Deduct credit for this span allocation and sweep if</span><span class="hljs-comment">// necessary. mHeap_Alloc will also sweep npages, so this only</span><span class="hljs-comment">// pays the debt down to npage pages.</span>deductSweepCredit(npages*_PageSize, npages)<span class="hljs-comment">//份额内存空间</span>s := mheap_.alloc(npages, makeSpanClass(<span class="hljs-number">0</span>, noscan), needzero)<span class="hljs-keyword">if</span> s == <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"out of memory"</span>)&#125;s.limit = s.base() + sizeheapBitsForAddr(s.base()).initSpan(s)<span class="hljs-keyword">return</span> s&#125;</code></pre></div><h4 id="分配defer-arg-块">分配defer+arg 块</h4><p>最后，如果该typ类型不是指针，还要判断其是否是<code>defer</code>类型，进行处理，更新<code>local_scan</code></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">mallocgc</span><span class="hljs-params">(size <span class="hljs-keyword">uintptr</span>, typ *_type, needzero <span class="hljs-keyword">bool</span>)</span> <span class="hljs-title">unsafe</span>.<span class="hljs-title">Pointer</span></span> &#123;...<span class="hljs-keyword">var</span> scanSize <span class="hljs-keyword">uintptr</span><span class="hljs-keyword">if</span> !noscan &#123;<span class="hljs-comment">// If allocating a defer+arg block, now that we've picked a malloc size</span><span class="hljs-comment">// large enough to hold everything, cut the "asked for" size down to</span><span class="hljs-comment">// just the defer header, so that the GC bitmap will record the arg block</span><span class="hljs-comment">// as containing nothing at all (as if it were unused space at the end of</span><span class="hljs-comment">// a malloc block caused by size rounding).</span><span class="hljs-comment">// The defer arg areas are scanned as part of scanstack.</span><span class="hljs-keyword">if</span> typ == deferType &#123;dataSize = unsafe.Sizeof(_defer&#123;&#125;)&#125;heapBitsSetType(<span class="hljs-keyword">uintptr</span>(x), size, dataSize, typ)<span class="hljs-keyword">if</span> dataSize &gt; typ.size &#123;<span class="hljs-comment">// Array allocation. If there are any</span><span class="hljs-comment">// pointers, GC has to scan to the last</span><span class="hljs-comment">// element.</span><span class="hljs-keyword">if</span> typ.ptrdata != <span class="hljs-number">0</span> &#123;scanSize = dataSize - typ.size + typ.ptrdata&#125;&#125; <span class="hljs-keyword">else</span> &#123;scanSize = typ.ptrdata&#125;c.local_scan += scanSize&#125;</code></pre></div><h4 id="fixalloc">fixAlloc</h4><p>因为我们的都知道go分配对象是在go gc heap中，并且由mspan，mcache，mcentral这些结构管理，但是这些结构的对象又是在哪里管理和分配呢？</p><p><code>fixalloc</code>就是做这个的：前面讲到<code>fixalloc</code>都是mheap中固定的结构</p><ul><li>主要目的就是一次性分配一大块内存(注意persistentalloc方法，使用是mmap，不指定地址，分配内存不再arena范围内，从进程空间获得可能百来KB)，每次请求对应的结构体大小，释放时就放在list链表中</li></ul><p>大概的分配有以下集中</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> mheap <span class="hljs-keyword">struct</span>&#123;...spanalloc             fixalloc <span class="hljs-comment">// allocator for span*</span>cachealloc            fixalloc <span class="hljs-comment">// allocator for mcache*</span>treapalloc            fixalloc <span class="hljs-comment">// allocator for treapNodes*</span>specialfinalizeralloc fixalloc <span class="hljs-comment">// allocator for specialfinalizer*</span>specialprofilealloc   fixalloc <span class="hljs-comment">// allocator for specialprofile*</span>speciallock           mutex    <span class="hljs-comment">// lock for special record allocators.</span>arenaHintAlloc        fixalloc <span class="hljs-comment">// allocator for arenaHints</span>...&#125;</code></pre></div><p>有前面提到的<code>init</code>方法，还可以分别通过</p><ul><li><code>fixalloc.alloc</code></li><li><code>fixalloc.free</code>来分配以及释放内存</li></ul><h3 id="一些重要参数">一些重要参数</h3><ul><li><p>go_memstats_sys_bytes: 进程从操作系统获得内存的总字节数，包含了go运行的stack，heap还有其他数据结构相关的虚拟地址空间</p></li><li><p>go_memstats_heap_inuse_bytes: 在span中真正被使用的字节数；其中不包括可能已经返回到操作系统，或者可以重用进行对分配、可以将作为堆栈内存重用的字节 (?)</p></li><li><p>go_memstats_heap_idle_bytes: 在span中空闲的字节数;</p></li><li><p>go_memstats_stack_sys_bytes: 栈内存字节数；主要用于goroutine栈内存的分配;</p></li></ul><p>由以上参数结合代码其实可以知道大概span在内存中有几种状态:</p><ol><li><p>idle不包含对象或者其他数据，空闲的物理内存可以释放回OS（虚拟地址不会释放！！！），或者将其转换成inuse状态或者stack span</p></li><li><p>inuse,至少包含一个mheap，并且可能有空闲空间分配更多堆对象</p></li><li><p>stack span，只会在堆或者是栈内存其中之一</p></li></ol><h2 id="栈内存">栈内存</h2><p>前面提到的都是属于堆上的内存，由allocator和gc负责管理;</p><h3 id="设计">设计</h3><ul><li>寄存器栈内存一般靠编译器来分配和释放,都是随着函数的生命周期变化而变化；而传入到函数中，栈寄存器用于存储了<code>基址地址</code>和<code>栈顶地址</code>, Go中则主要涉及<code>BP</code>和<code>SP</code>两个栈寄存器;</li></ul><p>对于大顶端(目前大多数使用),栈区内存就是从高地址到低地址扩展，释放内存的时候只需要更改<code>SP</code>的值，操作速度极快，占用极少；</p><ul><li>线程栈</li></ul><p>对于大部分OS，分配线程栈大小默认一般是2M~4MB，但是当调用栈过深，也会对栈进行扩容</p><ul><li>逃逸分析</li></ul><p>Go会动态管理内存位置，会对对象进行<code>逃逸分析</code>，一般遵循：</p><ol><li>指向堆上的对象的指针不能在栈中;</li><li>指向栈对象的指针不能在栈对象回收后存活;</li></ol><h3 id="栈历史以及设计">栈历史以及设计</h3><p>过去曾经使用过分段的栈(segmented stack)，扩容的时候会创建新的栈空间，然后用链表连接起来，但是会导致两个问题:</p><ul><li>如果当前goroutine栈容量接近上线，任意函数调用都会触发栈扩展，那么函数调用返回后，栈又会缩容，就有热分裂(hot split)问题;</li><li>还会在goroutine的栈的分配中体现，如果其超过了分段栈的扩缩容阈值，都会出现从而导致更多的工作；</li></ul><p>现在则转向了连续栈(contiguous stack),<a href="https://docs.google.com/document/d/1wAaf1rYoM4S4gtnPh0zOlGzWtrZFQ5suE8qr2sD8uWQ/pub" target="_blank" rel="noopener">原设计文档</a>核心的原理就是当原栈空间不足，就会创建一个更大的空间并把原栈中所有值都移过去；所以当栈空间不足的时候，我们要考虑:</p><ol><li>内存空间分配更大的栈</li><li><strong>将旧栈的内容copy到新栈</strong>;</li><li>销毁并回收旧栈;</li></ol><p>最主要就在第二步，如果有<strong>指针类型</strong>呢？因为栈改变，内存肯定也会改变（<s>所以这里可以考虑来个映射？</s>)不考虑多余的映射的原因就是，之前提到的<code>逃逸分析</code>遵循的两个原则之一：<strong>指向堆上的对象的指针不能在栈中</strong>，意思就是这里发现的指针类型一定是指向栈内的，所以不需要考虑额外映射，就将所有**相对的变量（实际就是所有的）**一起进行调整即可。</p><h3 id="结构-v2">结构</h3><p>栈的结构体:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Stack describes a Go execution stack.</span><span class="hljs-comment">// The bounds of the stack are exactly [lo, hi),</span><span class="hljs-comment">// with no implicit data structures on either side.</span><span class="hljs-keyword">type</span> stack <span class="hljs-keyword">struct</span> &#123;lo <span class="hljs-keyword">uintptr</span>hi <span class="hljs-keyword">uintptr</span>&#125;</code></pre></div><p>看起来很简单，但是其生成过程要从<code>编译期</code>到<code>运行期</code>代码结合：</p><ul><li>编译期，<code>cmd/internal/obj/x86.stacksplit</code> 在调用函数前插入<code>runtime.morestack</code>或<code>runtime.morestack_noctxt</code></li><li>运行时，goroutine会在<code>runtime.mlag</code>中调用<code>runtime.stackalloc</code>申请新栈空间，并在编译器插入的<code>runtime.morestack</code>中检查栈空间是否充足;</li></ul><h3 id="栈初始化">栈初始化</h3><p><code>runtime/stack.go</code>中:主要两个结构体<code>runtime.stackpool</code>和<code>runtime.stacklarge</code>，分别表示全局栈缓存和大栈缓存，区分就是栈大小是否大于32KB;</p><p>而这两个结构体都与<code>mSpan</code>有关（<code>mSpanList</code>,<code>mcentral</code>中有nonempty和empty)，可以认为go的栈就是分配在堆上的</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Global pool of spans that have free stacks.</span><span class="hljs-comment">// Stacks are assigned an order according to size.</span><span class="hljs-comment">//     order = log_2(size/FixedStack)</span><span class="hljs-comment">// There is a free list for each order.</span><span class="hljs-keyword">var</span> stackpool [_NumStackOrders]<span class="hljs-keyword">struct</span> &#123;item stackpoolItem<span class="hljs-comment">//同mcentral很像，就是为了内存对齐而已;</span>_    [cpu.CacheLinePadSize - unsafe.Sizeof(stackpoolItem&#123;&#125;)%cpu.CacheLinePadSize]<span class="hljs-keyword">byte</span>&#125;<span class="hljs-comment">//go:notinheap</span><span class="hljs-keyword">type</span> stackpoolItem <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">//因为是全局栈，要加锁</span>mu   mutexspan mSpanList&#125;<span class="hljs-comment">// Global pool of large stack spans.</span><span class="hljs-keyword">var</span> stackLarge <span class="hljs-keyword">struct</span> &#123;lock mutexfree [heapAddrBits - pageShift]mSpanList <span class="hljs-comment">// free lists by log_2(s.npages)</span>&#125;</code></pre></div><p>再看一下初始化</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">stackinit</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-keyword">if</span> _StackCacheSize&amp;_PageMask != <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"cache size must be a multiple of page size"</span>)&#125;<span class="hljs-keyword">for</span> i := <span class="hljs-keyword">range</span> stackpool &#123;stackpool[i].item.span.init()&#125;<span class="hljs-keyword">for</span> i := <span class="hljs-keyword">range</span> stackLarge.free &#123;stackLarge.free[i].init()&#125;&#125;</code></pre></div><p>上面的结构<code>stackpoolItem</code>中有个锁，但是如果大家都用这个结构，就会发生锁竞争，所以在<code>mcache</code>中都增加了一个stackcache在<code>mcache</code>结构上</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Number of orders that get caching. Order 0 is FixedStack</span><span class="hljs-comment">// and each successive order is twice as large.</span><span class="hljs-comment">// We want to cache 2KB, 4KB, 8KB, and 16KB stacks. Larger stacks</span><span class="hljs-comment">// will be allocated directly.</span><span class="hljs-comment">// Since FixedStack is different on different systems, we</span><span class="hljs-comment">// must vary NumStackOrders to keep the same maximum cached size.</span><span class="hljs-comment">//   OS               | FixedStack | NumStackOrders</span><span class="hljs-comment">//   -----------------+------------+---------------</span><span class="hljs-comment">//   linux/darwin/bsd | 2KB        | 4</span><span class="hljs-comment">//   windows/32       | 4KB        | 3</span><span class="hljs-comment">//   windows/64       | 8KB        | 2</span><span class="hljs-comment">//   plan9            | 4KB        | 3</span>_NumStackOrders = <span class="hljs-number">4</span> - sys.PtrSize/<span class="hljs-number">4</span>*sys.GoosWindows - <span class="hljs-number">1</span>*sys.GoosPlan9<span class="hljs-keyword">type</span> mcache <span class="hljs-keyword">struct</span>&#123;...stackcache [_NumStackOrders]stackfreelist ...&#125;<span class="hljs-keyword">type</span> stackfreelist <span class="hljs-keyword">struct</span> &#123;list gclinkptr <span class="hljs-comment">// linked list of free stacks</span>size <span class="hljs-keyword">uintptr</span>   <span class="hljs-comment">// total size of stacks in list</span>&#125;</code></pre></div><p>Linux上 ,计算出的栈常数:<code>_StackCacheSize = 32768</code>,<code>_FixedStack = 2048</code>,  <code>_NumStackOrders = 4</code></p><p>大概结构<img src="/img/stackCache.png" srcset="/img/loading.gif" alt="如图"></p><p>stackCache是per-P的，在另外一篇文章<a href="../goroutine.html">goroutine</a>上讲过，主要用于分配goroutine的stack，同普通内存一样其分为多个segment，class, linux就分为2KB,4KB,8KB,16KB等级</p><p>其中 &gt; 16K的直接从全局stacklarge分配否则按照先从P的stackcache分配=&gt; 如果无法分配 =&gt; 从全局stackpool分配一批stack(stackpoolalloc)，赋给该p的stackcache，再从local stackcache分配</p><h3 id="栈的分配">栈的分配</h3><p>分配栈空间主要是由<code>runtime.stackalloc</code>方法进行分配，其一定要在系统栈上运行因为其使用的是Per-P的资源且不可以切分整个栈;</p><ul><li><p>其必须要在调度栈中运行？？？，所以在这个方法运行中，我们不会进行栈的扩容(否则会有死锁,可以试想栈扩容代码？？？)</p></li><li><p>小一点的的栈空间，会用全局栈<code>stackpool</code>或者<code>mcache.stackcache</code>(固定大小空闲链表)来分配</p></li><li><p>栈空间较大就从全局栈<code>stackpool</code>或者<code>stackLarge</code>分配;</p></li><li><p>如果栈空间大的<code>stackLarge</code>都拿不到，就从堆上申请;</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// stackalloc allocates an n byte stack.</span><span class="hljs-comment">//</span><span class="hljs-comment">// stackalloc must run on the system stack because it uses per-P</span><span class="hljs-comment">// resources and must not split the stack.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:systemstack</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">stackalloc</span><span class="hljs-params">(n <span class="hljs-keyword">uint32</span>)</span> <span class="hljs-title">stack</span></span> &#123;<span class="hljs-comment">// Stackalloc must be called on scheduler stack, so that we</span><span class="hljs-comment">// never try to grow the stack during the code that stackalloc runs.</span><span class="hljs-comment">// Doing so would cause a deadlock (issue 1547).</span><span class="hljs-comment">//判断是否在调度栈中</span>thisg := getg()<span class="hljs-keyword">if</span> thisg != thisg.m.g0 &#123;throw(<span class="hljs-string">"stackalloc not on scheduler stack"</span>)&#125;<span class="hljs-keyword">if</span> n&amp;(n<span class="hljs-number">-1</span>) != <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"stack size not a power of 2"</span>)&#125;<span class="hljs-keyword">if</span> stackDebug &gt;= <span class="hljs-number">1</span> &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"stackalloc "</span>, n, <span class="hljs-string">"\n"</span>)&#125;<span class="hljs-keyword">if</span> debug.efence != <span class="hljs-number">0</span> || stackFromSystem != <span class="hljs-number">0</span> &#123;n = <span class="hljs-keyword">uint32</span>(alignUp(<span class="hljs-keyword">uintptr</span>(n), physPageSize))v := sysAlloc(<span class="hljs-keyword">uintptr</span>(n), &amp;memstats.stacks_sys)<span class="hljs-keyword">if</span> v == <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"out of memory (stackalloc)"</span>)&#125;<span class="hljs-keyword">return</span> stack&#123;<span class="hljs-keyword">uintptr</span>(v), <span class="hljs-keyword">uintptr</span>(v) + <span class="hljs-keyword">uintptr</span>(n)&#125;&#125;<span class="hljs-comment">// Small stacks are allocated with a fixed-size free-list allocator.</span><span class="hljs-comment">// If we need a stack of a bigger size, we fall back on allocating</span><span class="hljs-comment">// a dedicated span.</span><span class="hljs-comment">//Linux amd64: </span><span class="hljs-comment">//_StackCacheSize = 32768, _FixedStack = 2048, _NumStackOrders = 4</span><span class="hljs-comment">//1. 较小的栈</span><span class="hljs-keyword">var</span> v unsafe.Pointer<span class="hljs-keyword">if</span> n &lt; _FixedStack&lt;&lt;_NumStackOrders &amp;&amp; n &lt; _StackCacheSize &#123;order := <span class="hljs-keyword">uint8</span>(<span class="hljs-number">0</span>)n2 := n<span class="hljs-keyword">for</span> n2 &gt; _FixedStack &#123;order++n2 &gt;&gt;= <span class="hljs-number">1</span>&#125;<span class="hljs-keyword">var</span> x gclinkptrc := thisg.m.mcache<span class="hljs-keyword">if</span> stackNoCache != <span class="hljs-number">0</span> || c == <span class="hljs-literal">nil</span> || thisg.m.preemptoff != <span class="hljs-string">""</span> &#123;<span class="hljs-comment">//从全局pool找</span><span class="hljs-comment">// c == nil can happen in the guts of exitsyscall or</span><span class="hljs-comment">// procresize. Just get a stack from the global pool.</span><span class="hljs-comment">// Also don't touch stackcache during gc</span><span class="hljs-comment">// as it's flushed concurrently.</span>lock(&amp;stackpool[order].item.mu)x = stackpoolalloc(order)unlock(&amp;stackpool[order].item.mu)&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//从mcache.stackCache找</span>x = c.stackcache[order].list<span class="hljs-keyword">if</span> x.ptr() == <span class="hljs-literal">nil</span> &#123;stackcacherefill(c, order)x = c.stackcache[order].list&#125;c.stackcache[order].list = x.ptr().nextc.stackcache[order].size -= <span class="hljs-keyword">uintptr</span>(n)&#125;v = unsafe.Pointer(x)&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//2. 较大的栈</span><span class="hljs-keyword">var</span> s *mspannpage := <span class="hljs-keyword">uintptr</span>(n) &gt;&gt; _PageShiftlog2npage := stacklog2(npage)<span class="hljs-comment">// Try to get a stack from the large stack cache.</span><span class="hljs-comment">//从stackLarge找</span>lock(&amp;stackLarge.lock)<span class="hljs-keyword">if</span> !stackLarge.free[log2npage].isEmpty() &#123;s = stackLarge.free[log2npage].firststackLarge.free[log2npage].remove(s)&#125;unlock(&amp;stackLarge.lock)<span class="hljs-keyword">if</span> s == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">//stackLarge找不到，只能从mheap拿</span><span class="hljs-comment">// Allocate a new stack from the heap.</span>s = mheap_.allocManual(npage, &amp;memstats.stacks_inuse)<span class="hljs-keyword">if</span> s == <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"out of memory"</span>)&#125;<span class="hljs-comment">//这里针对某些操作系统进行的处理</span>osStackAlloc(s)s.elemsize = <span class="hljs-keyword">uintptr</span>(n)&#125;v = unsafe.Pointer(s.base())&#125;...</code></pre></div><h3 id="栈扩容">栈扩容</h3><p>上面说到编译期中的<code>stackSplit</code>会插入<code>runtime.morestack</code>来检查栈空间是否充足，如果不充足，就会先保存当前栈信息，然后调用<code>runtime.newstack</code>来分配栈空间</p><ul><li><p>该方法是无writebarrier(error on write barrier in this or recursive callees)的，writebarrier会引发错误，因为其可以被栈扩容的其他nowritebarrier 方法(比如???)called,</p></li><li><p>首先会检查是当前goroutine否可以被抢占,如果可以就触发<code>runtime.gogo</code>调度器调度; 这里有个小问题，是否可以抢占其中要判断一种状态就是goroutine是否在<code>_Grunning</code>，但是gc会将其状态从<code>Gwaiting</code>改成<code>Gscanwaiting</code>,所以如果遇到gc要等其完成才能继续，但是如果gc在某种情况下依赖goroutine的锁才能完成，这里就会形成死锁;而代码中，将这个检查提前了，就可以避免；（即使状态从<code>Grunning</code>到<code>Gwaiting</code>或者反复???)</p></li><li><p>如果被gc<code>runtime.scanstack</code>方法标记了<code>preemptShrink</code>要收缩栈，则调用<code>runtime.shrinkstack</code></p></li><li><p>如果当前goroutine被<code>runtime.suspendG</code>挂起(<code>preemptStop</code>)，要Park当前goroutine让其他抢占，然后修改状态为<code>_Gpreempted</code></p></li><li><p>然后调用<code>runtime.gopreempt_m</code>让出goroutine，实际就是调用了<code>runtime.GoSched</code></p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Called from runtime·morestack when more stack is needed.</span><span class="hljs-comment">// Allocate larger stack and relocate to new stack.</span><span class="hljs-comment">// Stack growth is multiplicative, for constant amortized cost.</span><span class="hljs-comment">//</span><span class="hljs-comment">// g-&gt;atomicstatus will be Grunning or Gscanrunning upon entry.</span><span class="hljs-comment">// If the scheduler is trying to stop this g, then it will set preemptStop.</span><span class="hljs-comment">//</span><span class="hljs-comment">// This must be nowritebarrierrec because it can be called as part of</span><span class="hljs-comment">// stack growth from other nowritebarrierrec functions, but the</span><span class="hljs-comment">// compiler doesn't check this.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:nowritebarrierrec</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newstack</span><span class="hljs-params">()</span></span> &#123;thisg := getg()...gp := thisg.m.curg<span class="hljs-comment">//需要扩容的栈的goroutine</span>...<span class="hljs-comment">// <span class="hljs-doctag">NOTE:</span> stackguard0 may change underfoot, if another thread</span><span class="hljs-comment">// is about to try to preempt gp. Read it just once and use that same</span><span class="hljs-comment">// value now and below.</span><span class="hljs-comment">//这里stackguard0随时会变化，所以要用原子读;</span>preempt := atomic.Loaduintptr(&amp;gp.stackguard0) == stackPreempt<span class="hljs-comment">// Be conservative about where we preempt.</span><span class="hljs-comment">// We are interested in preempting user Go code, not runtime code.</span><span class="hljs-comment">// If we're holding locks, mallocing, or preemption is disabled, don't</span><span class="hljs-comment">// preempt.</span><span class="hljs-comment">//在这种情况下才抢占:</span><span class="hljs-comment">//mp.locks == 0 &amp;&amp; mp.mallocing == 0 &amp;&amp; mp.preemptoff == "" &amp;&amp; mp.p.ptr().status == _Prunning</span><span class="hljs-comment">// This check is very early in newstack so that even the status change</span><span class="hljs-comment">// from Grunning to Gwaiting and back doesn't happen in this case.</span><span class="hljs-comment">// That status change by itself can be viewed as a small preemption,</span><span class="hljs-comment">// because the GC might change Gwaiting to Gscanwaiting, and then</span><span class="hljs-comment">// this goroutine has to wait for the GC to finish before continuing.</span><span class="hljs-comment">// If the GC is in some way dependent on this goroutine (for example,</span><span class="hljs-comment">// it needs a lock held by the goroutine), that small preemption turns</span><span class="hljs-comment">// into a real deadlock.</span><span class="hljs-keyword">if</span> preempt &#123;<span class="hljs-keyword">if</span> !canPreemptM(thisg.m) &#123;<span class="hljs-comment">// Let the goroutine keep running for now.</span><span class="hljs-comment">// gp-&gt;preempt is set, so it will be preempted next time.</span>gp.stackguard0 = gp.stack.lo + _StackGuardgogo(&amp;gp.sched) <span class="hljs-comment">// never return</span>&#125;&#125;<span class="hljs-keyword">if</span> gp.stack.lo == <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"missing stack in newstack"</span>)&#125;sp := gp.sched.sp<span class="hljs-keyword">if</span> sys.ArchFamily == sys.AMD64 || sys.ArchFamily == sys.I386 || sys.ArchFamily == sys.WASM &#123;<span class="hljs-comment">// The call to morestack cost a word.</span>sp -= sys.PtrSize&#125;<span class="hljs-keyword">if</span> stackDebug &gt;= <span class="hljs-number">1</span> || sp &lt; gp.stack.lo &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"runtime: newstack sp="</span>, hex(sp), <span class="hljs-string">" stack=["</span>, hex(gp.stack.lo), <span class="hljs-string">", "</span>, hex(gp.stack.hi), <span class="hljs-string">"]\n"</span>,<span class="hljs-string">"\tmorebuf=&#123;pc:"</span>, hex(morebuf.pc), <span class="hljs-string">" sp:"</span>, hex(morebuf.sp), <span class="hljs-string">" lr:"</span>, hex(morebuf.lr), <span class="hljs-string">"&#125;\n"</span>,<span class="hljs-string">"\tsched=&#123;pc:"</span>, hex(gp.sched.pc), <span class="hljs-string">" sp:"</span>, hex(gp.sched.sp), <span class="hljs-string">" lr:"</span>, hex(gp.sched.lr), <span class="hljs-string">" ctxt:"</span>, gp.sched.ctxt, <span class="hljs-string">"&#125;\n"</span>)&#125;<span class="hljs-keyword">if</span> sp &lt; gp.stack.lo &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"runtime: gp="</span>, gp, <span class="hljs-string">", goid="</span>, gp.goid, <span class="hljs-string">", gp-&gt;status="</span>, hex(readgstatus(gp)), <span class="hljs-string">"\n "</span>)<span class="hljs-built_in">print</span>(<span class="hljs-string">"runtime: split stack overflow: "</span>, hex(sp), <span class="hljs-string">" &lt; "</span>, hex(gp.stack.lo), <span class="hljs-string">"\n"</span>)throw(<span class="hljs-string">"runtime: split stack overflow"</span>)&#125;<span class="hljs-keyword">if</span> preempt &#123;<span class="hljs-keyword">if</span> gp == thisg.m.g0 &#123;throw(<span class="hljs-string">"runtime: preempt g0"</span>)&#125;<span class="hljs-keyword">if</span> thisg.m.p == <span class="hljs-number">0</span> &amp;&amp; thisg.m.locks == <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"runtime: g is running but p is not"</span>)&#125;<span class="hljs-keyword">if</span> gp.preemptShrink &#123;<span class="hljs-comment">// We're at a synchronous safe point now, so</span><span class="hljs-comment">// do the pending stack shrink.</span>gp.preemptShrink = <span class="hljs-literal">false</span>shrinkstack(gp)&#125;<span class="hljs-keyword">if</span> gp.preemptStop &#123;preemptPark(gp) <span class="hljs-comment">// never returns</span>&#125;<span class="hljs-comment">// Act like goroutine called runtime.Gosched.</span>gopreempt_m(gp) <span class="hljs-comment">// never return</span>&#125;...&#125;</code></pre></div><p>如果不需要抢占:</p><ul><li>分配更大的空间然后移动栈,新空间是旧空间两倍，但是都会判断是否大于最大的栈（<code>var maxstacksize uintptr = 1 &lt;&lt; 20</code>)</li><li>将goroutine的状态从<code>Grunning</code>改为<code>Gcopystack</code></li><li>然后复制旧栈到新栈（期间gc是不会扫描这个goroutine，因为这个goroutine在<code>Gcopystack</code>状态)</li><li>然后将<code>Gcopystack</code>状态改为<code>Grunning</code>,再次调用<code>gogo</code></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newstack</span><span class="hljs-params">()</span></span> &#123;...<span class="hljs-comment">// Allocate a bigger segment and move the stack.</span>oldsize := gp.stack.hi - gp.stack.lonewsize := oldsize * <span class="hljs-number">2</span><span class="hljs-keyword">if</span> newsize &gt; maxstacksize &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"runtime: goroutine stack exceeds "</span>, maxstacksize, <span class="hljs-string">"-byte limit\n"</span>)<span class="hljs-built_in">print</span>(<span class="hljs-string">"runtime: sp="</span>, hex(sp), <span class="hljs-string">" stack=["</span>, hex(gp.stack.lo), <span class="hljs-string">", "</span>, hex(gp.stack.hi), <span class="hljs-string">"]\n"</span>)throw(<span class="hljs-string">"stack overflow"</span>)&#125;<span class="hljs-comment">// The goroutine must be executing in order to call newstack,</span><span class="hljs-comment">// so it must be Grunning (or Gscanrunning).</span>casgstatus(gp, _Grunning, _Gcopystack)<span class="hljs-comment">// The concurrent GC will not scan the stack while we are doing the copy since</span><span class="hljs-comment">// the gp is in a Gcopystack status.</span>copystack(gp, newsize)<span class="hljs-keyword">if</span> stackDebug &gt;= <span class="hljs-number">1</span> &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"stack grow done\n"</span>)&#125;casgstatus(gp, _Gcopystack, _Grunning)gogo(&amp;gp.sched)&#125;</code></pre></div><p>我们看到<code>copystack</code>函数中:会调用<code>stackalloc</code>分配空间，这个之前已经讲过(从全局或者stackache或者largestack获取)</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Copies gp's stack to a new stack of a different size.</span><span class="hljs-comment">// Caller must have changed gp status to Gcopystack.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">copystack</span><span class="hljs-params">(gp *g, newsize <span class="hljs-keyword">uintptr</span>)</span></span> &#123;<span class="hljs-keyword">if</span> gp.syscallsp != <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"stack growth not allowed in system call"</span>)&#125;old := gp.stack<span class="hljs-keyword">if</span> old.lo == <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"nil stackbase"</span>)&#125;used := old.hi - gp.sched.sp<span class="hljs-comment">// allocate new stack</span><span class="hljs-built_in">new</span> := stackalloc(<span class="hljs-keyword">uint32</span>(newsize))<span class="hljs-keyword">if</span> stackPoisonCopy != <span class="hljs-number">0</span> &#123;fillstack(<span class="hljs-built_in">new</span>, <span class="hljs-number">0xfd</span>)&#125;...&#125;</code></pre></div><p>较复杂的是，其中的指针如何复制:</p><ul><li><p>检查stack内是否有未锁的channel，调用<code>runtime.adjustsudogs</code>或<code>runtime.syncadjustsudogs</code>方法对<code>runtime.sudog</code>结构体进行调整（调整的实际就是sudog结构体,sudog represents a g in a wait list, such as for sending/receiving on a channel.);</p></li><li><p>用<code>runtime.memove</code>将旧栈（全部或剩下的数据）移到新栈；</p></li><li><p>调整剩余的一些指针，比如<code>ctxt</code>,<code>defers</code>,<code>panics</code>等等</p></li><li><p>其所有调整指针都会调用<code>runtime.adjustPointer</code>里面利用了新栈和旧栈内存地址的差来调整指针；</p></li><li><p>到最后调用<code>runtime.stackfree</code>释放旧栈空间;</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Copies gp's stack to a new stack of a different size.</span><span class="hljs-comment">// Caller must have changed gp status to Gcopystack.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">copystack</span><span class="hljs-params">(gp *g, newsize <span class="hljs-keyword">uintptr</span>)</span></span> &#123;...<span class="hljs-comment">// Compute adjustment.</span><span class="hljs-keyword">var</span> adjinfo adjustinfoadjinfo.old = oldadjinfo.delta = <span class="hljs-built_in">new</span>.hi - old.hi<span class="hljs-comment">// Adjust sudogs, synchronizing with channel ops if necessary.</span>ncopy := used<span class="hljs-keyword">if</span> !gp.activeStackChans &#123;adjustsudogs(gp, &amp;adjinfo)&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">// sudogs may be pointing in to the stack and gp has</span><span class="hljs-comment">// released channel locks, so other goroutines could</span><span class="hljs-comment">// be writing to gp's stack. Find the highest such</span><span class="hljs-comment">// pointer so we can handle everything there and below</span><span class="hljs-comment">// carefully. (This shouldn't be far from the bottom</span><span class="hljs-comment">// of the stack, so there's little cost in handling</span><span class="hljs-comment">// everything below it carefully.)</span>adjinfo.sghi = findsghi(gp, old)<span class="hljs-comment">// Synchronize with channel ops and copy the part of</span><span class="hljs-comment">// the stack they may interact with.</span>ncopy -= syncadjustsudogs(gp, used, &amp;adjinfo)&#125;<span class="hljs-comment">// Copy the stack (or the rest of it) to the new location</span>memmove(unsafe.Pointer(<span class="hljs-built_in">new</span>.hi-ncopy), unsafe.Pointer(old.hi-ncopy), ncopy)<span class="hljs-comment">// Adjust remaining structures that have pointers into stacks.</span><span class="hljs-comment">// We have to do most of these before we traceback the new</span><span class="hljs-comment">// stack because gentraceback uses them.</span>adjustctxt(gp, &amp;adjinfo)adjustdefers(gp, &amp;adjinfo)adjustpanics(gp, &amp;adjinfo)<span class="hljs-keyword">if</span> adjinfo.sghi != <span class="hljs-number">0</span> &#123;adjinfo.sghi += adjinfo.delta&#125;<span class="hljs-comment">// Swap out old stack for new one</span><span class="hljs-comment">//切换指向的栈</span>gp.stack = <span class="hljs-built_in">new</span>gp.stackguard0 = <span class="hljs-built_in">new</span>.lo + _StackGuard <span class="hljs-comment">// <span class="hljs-doctag">NOTE:</span> might clobber a preempt request</span>gp.sched.sp = <span class="hljs-built_in">new</span>.hi - usedgp.stktopsp += adjinfo.delta<span class="hljs-comment">// Adjust pointers in the new stack.</span>gentraceback(^<span class="hljs-keyword">uintptr</span>(<span class="hljs-number">0</span>), ^<span class="hljs-keyword">uintptr</span>(<span class="hljs-number">0</span>), <span class="hljs-number">0</span>, gp, <span class="hljs-number">0</span>, <span class="hljs-literal">nil</span>, <span class="hljs-number">0x7fffffff</span>, adjustframe, noescape(unsafe.Pointer(&amp;adjinfo)), <span class="hljs-number">0</span>)<span class="hljs-comment">// free old stack</span><span class="hljs-keyword">if</span> stackPoisonCopy != <span class="hljs-number">0</span> &#123;fillstack(old, <span class="hljs-number">0xfc</span>)&#125;stackfree(old)&#125;</code></pre></div><h3 id="栈缩容">栈缩容</h3><p>前面出现过的<code>runtime.shrinkstack</code></p><ul><li>要先检查我们当前是不是在goroutine上(own its stack)，是否安全(要有所有帧的pointers map才可以视为安全，两种情况会有不确定情况:1. syscall中 2.当前goroutine停止于 asynchronous safe point中)，不允许debug设置<code>shrinkoff</code>缩容，不允许对<code>gcBgMarkWorker</code>缩容等等（还有，但是</li><li></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Maybe shrink the stack being used by gp.</span><span class="hljs-comment">//</span><span class="hljs-comment">// gp must be stopped and we must own its stack. It may be in</span><span class="hljs-comment">// _Grunning, but only if this is our own user G.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">shrinkstack</span><span class="hljs-params">(gp *g)</span></span> &#123;<span class="hljs-keyword">if</span> gp.stack.lo == <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"missing stack in shrinkstack"</span>)&#125;<span class="hljs-comment">//如果不是_Gscan状态，我们无法通过这个状态获得栈，但是如果这个是我们当前使用的G，而且我们在系统栈中，就可以继续;</span><span class="hljs-keyword">if</span> s := readgstatus(gp); s&amp;_Gscan == <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// We don't own the stack via _Gscan. We could still</span><span class="hljs-comment">// own it if this is our own user G and we're on the</span><span class="hljs-comment">// system stack.</span><span class="hljs-keyword">if</span> !(gp == getg().m.curg &amp;&amp; getg() != getg().m.curg &amp;&amp; s == _Grunning) &#123;<span class="hljs-comment">// We don't own the stack.</span>throw(<span class="hljs-string">"bad status in shrinkstack"</span>)&#125;&#125;<span class="hljs-keyword">if</span> !isShrinkStackSafe(gp) &#123;throw(<span class="hljs-string">"shrinkstack at bad time"</span>)&#125;<span class="hljs-comment">// Check for self-shrinks while in a libcall. These may have</span><span class="hljs-comment">// pointers into the stack disguised as uintptrs, but these</span><span class="hljs-comment">// code paths should all be nosplit.</span><span class="hljs-comment">//???不懂是啥东西</span><span class="hljs-keyword">if</span> gp == getg().m.curg &amp;&amp; gp.m.libcallsp != <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"shrinking stack in libcall"</span>)&#125;<span class="hljs-keyword">if</span> debug.gcshrinkstackoff &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span>&#125;f := findfunc(gp.startpc)<span class="hljs-keyword">if</span> f.valid() &amp;&amp; f.funcID == funcID_gcBgMarkWorker &#123;<span class="hljs-comment">// We're not allowed to shrink the gcBgMarkWorker</span><span class="hljs-comment">// stack (see gcBgMarkWorker for explanation).</span><span class="hljs-keyword">return</span>&#125;...&#125;</code></pre></div><p>在一堆判断之后，确认可以缩容：</p><ul><li>有个最小值，小于最小值就不缩；</li><li>使用容量小于原来栈的1/4就缩容，缩容为原来size的1/2</li><li>原来的stack包括了SP指针以下的所有内容同stackguard空间(为了给nosplit function使用,nosplit function最大可以用的),这里用<code>stack.hi-sched.sp</code> + <code>_stackLimit</code></li><li></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">shrinkstack</span><span class="hljs-params">(gp *g)</span></span> &#123;....oldsize := gp.stack.hi - gp.stack.lonewsize := oldsize / <span class="hljs-number">2</span><span class="hljs-comment">// Don't shrink the allocation below the minimum-sized stack</span><span class="hljs-comment">// allocation.</span><span class="hljs-comment">//linux下&lt;2048 不缩了;</span><span class="hljs-keyword">if</span> newsize &lt; _FixedStack &#123;<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">// Compute how much of the stack is currently in use and only</span><span class="hljs-comment">// shrink the stack if gp is using less than a quarter of its</span><span class="hljs-comment">// current stack. The currently used stack includes everything</span><span class="hljs-comment">// down to the SP plus the stack guard space that ensures</span><span class="hljs-comment">// there's room for nosplit functions.</span>avail := gp.stack.hi - gp.stack.lo<span class="hljs-comment">//使用容量小于原来栈的1/4就缩容</span><span class="hljs-comment">// The maximum number of bytes that a chain of NOSPLIT</span><span class="hljs-comment">// functions can use.</span><span class="hljs-comment">//_StackLimit = _StackGuard - _StackSystem - _StackSmall</span><span class="hljs-keyword">if</span> used := gp.stack.hi - gp.sched.sp + _StackLimit; used &gt;= avail/<span class="hljs-number">4</span> &#123;<span class="hljs-keyword">return</span>&#125;<span class="hljs-keyword">if</span> stackDebug &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"shrinking stack "</span>, oldsize, <span class="hljs-string">"-&gt;"</span>, newsize, <span class="hljs-string">"\n"</span>)&#125;copystack(gp, newsize)&#125;</code></pre></div><h3 id="gogo继续运行goroutine">gogo继续运行goroutine</h3><p>//todo ???<code>gogo</code>方法是一串汇编，不会return，会直接跳出函数</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// func gogo(buf *gobuf)</span><span class="hljs-comment">// restore state from Gobuf; longjmp</span>TEXT runtime·gogo(SB), NOSPLIT, $<span class="hljs-number">16</span><span class="hljs-number">-8</span>MOVQbuf+<span class="hljs-number">0</span>(FP), BX<span class="hljs-comment">// gobuf</span>MOVQgobuf_g(BX), DXMOVQ<span class="hljs-number">0</span>(DX), CX<span class="hljs-comment">// make sure g != nil</span>get_tls(CX)MOVQDX, g(CX)MOVQgobuf_sp(BX), SP<span class="hljs-comment">// restore SP</span>MOVQgobuf_ret(BX), AXMOVQgobuf_ctxt(BX), DXMOVQgobuf_bp(BX), BPMOVQ$<span class="hljs-number">0</span>, gobuf_sp(BX)<span class="hljs-comment">// clear to help garbage collector</span>MOVQ$<span class="hljs-number">0</span>, gobuf_ret(BX)MOVQ$<span class="hljs-number">0</span>, gobuf_ctxt(BX)MOVQ$<span class="hljs-number">0</span>, gobuf_bp(BX)MOVQgobuf_pc(BX), BXJMPBX</code></pre></div><h2 id="内存对齐以及一些分配规则-补充前面的tcmalloc">内存对齐以及一些分配规则(补充前面的tcmalloc)</h2><p>runtime/msize.go</p><div class="hljs"><pre><code class="hljs golang"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">roundupsize</span><span class="hljs-params">(size <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-title">uintptr</span></span> &#123;<span class="hljs-comment">//size&lt;32768</span><span class="hljs-keyword">if</span> size &lt; _MaxSmallSize &#123;<span class="hljs-keyword">if</span> size &lt;= smallSizeMax<span class="hljs-number">-8</span> &#123;<span class="hljs-comment">//这里面的字段是go对特定class设定的对应大小</span><span class="hljs-keyword">return</span> <span class="hljs-keyword">uintptr</span>(class_to_size[size_to_class8[(size+smallSizeDiv<span class="hljs-number">-1</span>)/smallSizeDiv]])&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-keyword">uintptr</span>(class_to_size[size_to_class128[(size-smallSizeMax+largeSizeDiv<span class="hljs-number">-1</span>)/largeSizeDiv]])&#125;&#125;<span class="hljs-comment">//size为负数,_PageSize=1&lt;&lt;13 </span><span class="hljs-keyword">if</span> size+_PageSize &lt; size &#123;<span class="hljs-keyword">return</span> size&#125;<span class="hljs-keyword">return</span> round(size, _PageSize)&#125;<span class="hljs-comment">//该运算在下面会提到</span><span class="hljs-comment">// round n up to a multiple of a.  a must be a power of 2.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">round</span><span class="hljs-params">(n, a <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-title">uintptr</span></span> &#123;<span class="hljs-keyword">return</span> (n + a - <span class="hljs-number">1</span>) &amp;^ (a - <span class="hljs-number">1</span>)&#125;</code></pre></div><h3 id="补全的spanclass">补全的spanClass!!!</h3><p>注意到<strong>class_to_size</strong>和<strong>size_to_class</strong>等等字段</p><p><a id="sizetoclass">[sizetoclass]</a>实际上在runtime/sizeclasses.go里面可以体现出go对不同大小的class设置的size：每个span都带有一个sizeclass，即表明该span的page应该被怎么用；PS: <strong>可以参照tcmalloc 实现思想基本一直</strong></p><blockquote><blockquote><p>class0表示单独分配一个&gt;32KB对象的span，有67个size，每个size有两种，分配用于有指针和无指针对象，所以有个67*2= 134个class (即上面提到的numSpanClasses)</p></blockquote></blockquote><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// class  bytes/obj  bytes/span  objects  tail waste  max waste</span><span class="hljs-comment">//     1          8        8192     1024           0     87.50%</span><span class="hljs-comment">//     2         16        8192      512           0     43.75%</span><span class="hljs-comment">//     3         32        8192      256           0     46.88%</span><span class="hljs-comment">//     4         48        8192      170          32     31.52%</span><span class="hljs-comment">//     5         64        8192      128           0     23.44%</span><span class="hljs-comment">//     6         80        8192      102          32     19.07%</span><span class="hljs-comment">//     7         96        8192       85          32     15.95%</span><span class="hljs-comment">//     8        112        8192       73          16     13.56%</span><span class="hljs-comment">//     9        128        8192       64           0     11.72%</span><span class="hljs-comment">//    10        144        8192       56         128     11.82%</span><span class="hljs-comment">//    11        160        8192       51          32      9.73%</span>......<span class="hljs-comment">//    60      19072       57344        3         128      3.57%</span><span class="hljs-comment">//    61      20480       40960        2           0      6.87%</span><span class="hljs-comment">//    62      21760       65536        3         256      6.25%</span><span class="hljs-comment">//    63      24576       24576        1           0     11.45%</span><span class="hljs-comment">//    64      27264       81920        3         128     10.00%</span><span class="hljs-comment">//    65      28672       57344        2           0      4.91%</span><span class="hljs-comment">//    66      32768       32768        1           0     12.50%</span></code></pre></div><p>上面的代码是8B~32KB的不同class的span的大小，对象的数目，浪费的空间</p><p>如:class=3 时， 对象上限为32B，管理一个页(span=8KB)，最多可以有256个对象，刚刚好</p><p>$$tailWaste = (bytes/span)mod(objects)$$</p><p>当对象为 17B的时候:</p><p>$$\frac{((32-17)*256+0)}{8192} = 0.4687$$</p><p>除了上面的66个跨度，还会存储一个<code>ID=0</code>的跨度，其就是用作管理<strong>大于</strong>32KB的大对象;</p><p>可以看到bytes/obj一栏，就是go预定义objects大小，最小是8B，最大是32KB（注意这里只是在32KB以内，还有大于32KB以外的），所以都可以解释到<strong>slice</strong>在扩容的时候可能会不遵守*2和1.25倍扩容的规则；</p><p>相关的main方法可以在classToSize的转换runtime/mksizeclass.go中找到</p><h3 id="golang的位运算">golang的位运算</h3><p>有时候会经常看见会用一些全局常量：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//指针大小,一般64位就是8</span><span class="hljs-keyword">const</span> PtrSize = <span class="hljs-number">4</span> &lt;&lt; (^<span class="hljs-keyword">uintptr</span>(<span class="hljs-number">0</span>) &gt;&gt; <span class="hljs-number">63</span>) <span class="hljs-comment">//8</span></code></pre></div><p>sys.PtrSize, sys.RegSize等等</p><h4 id="1-运算">1. ^运算</h4><ul><li>用作单目运算时， ^ 指的就是取反,等于一些语言的 ~ 符号（这里注意都一样取补码）</li></ul><p>ps: 这里复习一下，</p><p>正数取反：化为二进制，得到补码(正数补码和原码一样)，再对补码每位取反</p><p>负数取反：化为二进制，得到补码(所有除符号位的每位取反，+1)，然后再对补码全部每位取反</p><div class="hljs"><pre><code class="hljs go">x:=^<span class="hljs-number">3</span><span class="hljs-comment">//3=》 0011=》 1100=-4 </span>log.Printf(<span class="hljs-string">"%d"</span>,x)<span class="hljs-comment">//-4</span>x:=^(<span class="hljs-number">-3</span>)<span class="hljs-comment">//-3=》 1011 =》 1100 =》 1101 =》 0010=2</span>log.Printf(<span class="hljs-string">"%d"</span>,x)<span class="hljs-comment">//2</span></code></pre></div><p>也可以用比较直接的方法：^a= -(a+1)</p><ul><li>用作双目运算符时则为异或（XOR）相同为0，相异为1</li></ul><h4 id="2-运算">2. &amp;^运算</h4><p>将运算符号左边数据相异保留，相同置为0;</p><p>符合：</p><ul><li>右侧为0，左侧数不变，</li><li>右侧是1，左侧清零</li><li>符合结合法即 a&amp;^b=a&amp;(^b)</li></ul><p>经常用该符号作内存对齐如runtime/stubs.go里面</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// round n up to a multiple of a.  a must be a power of 2.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">round</span><span class="hljs-params">(n, a <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-title">uintptr</span></span> &#123;<span class="hljs-keyword">return</span> (n + a - <span class="hljs-number">1</span>) &amp;^ (a - <span class="hljs-number">1</span>)&#125;<span class="hljs-comment">//可以有这种说法：</span><span class="hljs-comment">//找到最大位为１的位数，然后用１左移该位数即是roundup后的结果,</span><span class="hljs-comment">//比如6 : 110,最大为为1的是在第三位，1&lt;&lt;3 = 1000 = 8,即十进制的8</span><span class="hljs-comment">// n=6,a=2 : 110 =&gt; (6+2-1) = 111 &amp;^ 001 = 110</span></code></pre></div><p>runtime/malloc.go</p>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Consensus</title>
    <link href="/2020/02/10/Comcon/consensus/"/>
    <url>/2020/02/10/Comcon/consensus/</url>
    
    <content type="html"><![CDATA[<p>基本的共识思想</p><a id="more"></a><p>想象一下在单机事务一般需要满足的是ACID特性，但是在多机系统中呢？</p><h2 id="在raft和paxos等共识算法之前">在Raft和paxos等共识算法之前</h2><h3 id="2pc">2PC</h3><p>假设 A1, A2, A3三台机子，</p><p>A：atomic原子性，明显不满足（线性）</p><p>C：一致性，共识算法I: 隔离性D：持久性，可以回滚</p><h3 id="线性全序广播">线性全序广播</h3><p>这个话题可以分为两个部分：1.线性 2.全序广播</p><h4 id="1-线性">1. 线性</h4><p>上面说过，线性化指的就是对于一个系统，操作呈现原子性，</p>]]></content>
    
    
    
    <tags>
      
      <tag>Distributed</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>OS basis</title>
    <link href="/2020/02/03/Comcon/os/"/>
    <url>/2020/02/03/Comcon/os/</url>
    
    <content type="html"><![CDATA[<p>一些乱糟糟的笔记</p><a id="more"></a><h2 id="虚拟内存">虚拟内存</h2><p>段页式以页为单位替换，以段为单位使用。</p><h2 id="进程">进程</h2><p>资源调度的基本单位</p><p>结合golang调度器的一系列思考：</p><h3 id="1-为什么要多个进程">1. 为什么要多个进程</h3><p>多任务处理</p><h3 id="2-为什么不可以在进程之间切换-还要加上线程？">2. 为什么不可以在进程之间切换，还要加上线程？</h3><p>回顾一下进程的定义，创建，销毁</p><ul><li><p>定义</p><p>就是运行期间的程序以及相关资源的集合体，多个进程可以共享一类资源，或者多个进程可以运行一个程序</p></li><li><p>创建</p><p>linux上是由系统fork()(其实通过clone()来实现) init进程（或者现有进程）来创建一个新进程，注意这里会返回两次值，一次回到父进程，一次回到新的子进程，其<strong>开销</strong>其实就是复制父进程页表以及给子进程创建唯一task_struct，说多点这里，也不会一下子复制所有信息，采用copy on write，写时复制;</p><ul><li><p>详情同样是在linux下，fork()后 通过slab分配到task_struct()，其会有用到对象着色以及缓存着色（有点像进程池？？？)</p><ol><li><p>fork()内会调用copy_process()方法，首先会调用dup_task_struct()创建一个<strong>内核栈</strong>,<strong>thread_info结构</strong>和<strong>task_struct</strong>，这些值都与<strong>当前进程(父进程)的值相同</strong></p></li><li><p>检查并确保新创建该子进程后，再检查当前进程数(默认short int 32768，当然你可以自己改/proc/sys/kernel/pid_max，忘了是不是这个了)</p></li><li><p>子进程就开始将一些值清零或者设为默认初始值来区分开，但都是一些统计信息(非继承的task_struct成员),task_struct的值大多数都没有变</p></li><li><p>子进程状态设置为<strong>TASK_UNINTERRUPTABLE</strong>，保证不被运行</p></li><li><p>copy_flags()更新flags成员。清零PF_SUPERPRIV(是否是超级root)。设置PF_FORKNOEXEC()(表明进程还未被调用exec())</p></li><li><p>调用alloc_pid()为新进程分配一个有效pid</p></li><li><p>根据传给clone()的参数，copy_process()copy或者共享打开的文件，文件系统信息，信号处理函数，进程地址空间，命名空间等;</p></li><li><p>最后copy_process()返回一个指向子进程的指针;</p></li></ol></li><li><p>相关命令clone(CLONE_VM | CLONE_FS| CLONE_FILES| CLONE_SIGHAND, 0)</p></li><li><p>额外还有一个vfork，不copy父进程的页面，其余同fork一样</p></li></ul></li><li><p>执行调用exec()函数分配到相应的地址空间，然后将程序放入，</p></li><li><p>销毁调用exit()，这时候会将所有资源释放，父进程可以通过wait4()检查子进程是否被终结，还需要调用wait()或waitip(),否则子进程进入zombie状态</p></li></ul><ol start="3"><li>进程之间如何调度</li></ol><ul><li>分清楚目的：I/O bound 还是 CPU bound ?</li><li>手段<ol><li>优先级？nice值？</li><li>时间片为基本单位</li></ol></li></ul>]]></content>
    
    
    
    <tags>
      
      <tag>OS</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang timer</title>
    <link href="/2020/02/03/Go/timer/"/>
    <url>/2020/02/03/Go/timer/</url>
    
    <content type="html"><![CDATA[<p>//TODO</p><a id="more"></a><p>我们比较熟悉的时间包：</p><h2 id="定义">定义</h2><ol><li>timer.C 是 一个channel，在timer过期后，这个只读chan会有一个值</li><li>除了AfterFunc方法外，一个timer一定要由NewTimer创建</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// The Timer type represents a single event.</span><span class="hljs-comment">// When the Timer expires, the current time will be sent on C,</span><span class="hljs-comment">// unless the Timer was created by AfterFunc.</span><span class="hljs-comment">// A Timer must be created with NewTimer or AfterFunc.</span><span class="hljs-keyword">type</span> Timer <span class="hljs-keyword">struct</span> &#123;C &lt;-<span class="hljs-keyword">chan</span> Time <span class="hljs-comment">//一个channel，在timer过期后，这个只读chan会有一个值</span>r runtimeTimer&#125;</code></pre></div><p>在1.13以下(实际还要加上1.10以后),timer的实现有些不一样，首先timer结构复杂了一些:</p><p><code>timerproc</code>和小顶堆分成最多64个<code>timerproc</code>协程和四叉堆，用来休眠就近时间的方法还是依赖<code>futex timeout</code>机制。默认timerproc数量会跟<code>GOMAXPROCS</code>一致的，但最大也就64个，因为会被64取摸;</p><p>但在1.14下，其性能优化了了几个数量级:</p><ul><li>其将存放事件的四叉堆放到了P中</li><li>取消了<code>timerproc</code>，使用netpoll的epollwait来做就近时间的休眠等待(这样每次<code>runtime.schedule</code>都可以检查到定时器上有无运行到时的timer)；</li></ul><h3 id="1-13版本下">1.13版本下</h3><p>回顾一下:</p><ul><li>new一个<code>timer</code>使用的是<code>NewTimer()</code>函数，实际调用了<code>startTimer(t *timer)</code>,</li><li>而<code>startTimer</code>函数实际就是调用了<code>runtime.addTimer</code>函数，实际增加一个timer t到当前<code>P</code>上，避免了修改其他P的堆上的timer的<code>when</code>字段，可能会导致堆无法排序:</li></ul><p>timer的结构是一个64长度的数组:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">const</span> timersLen = <span class="hljs-number">64</span><span class="hljs-keyword">var</span> timers [timersLen]<span class="hljs-keyword">struct</span> &#123;    timersBucket    <span class="hljs-comment">// The padding should eliminate false sharing</span><span class="hljs-comment">// between timersBucket values.</span>pad [cpu.CacheLinePadSize - unsafe.Sizeof(timersBucket&#123;&#125;)%cpu.CacheLinePadSize]<span class="hljs-keyword">byte</span>&#125;<span class="hljs-comment">//runtime下</span><span class="hljs-comment">//go:notinheap</span><span class="hljs-keyword">type</span> timersBucket <span class="hljs-keyword">struct</span> &#123;lock         mutexgp           *gcreated      <span class="hljs-keyword">bool</span>sleeping     <span class="hljs-keyword">bool</span>rescheduling <span class="hljs-keyword">bool</span>sleepUntil   <span class="hljs-keyword">int64</span>waitnote     notet            []*timer&#125;<span class="hljs-comment">// Package time knows the layout of this structure.</span><span class="hljs-comment">// If this struct changes, adjust ../time/sleep.go:/runtimeTimer.</span><span class="hljs-comment">// For GOOS=nacl, package syscall knows the layout of this structure.</span><span class="hljs-comment">// If this struct changes, adjust ../syscall/net_nacl.go:/runtimeTimer.</span><span class="hljs-keyword">type</span> timer <span class="hljs-keyword">struct</span> &#123;tb *timersBucket <span class="hljs-comment">// the bucket the timer lives in</span>i  <span class="hljs-keyword">int</span>           <span class="hljs-comment">// heap index</span><span class="hljs-comment">// Timer wakes up at when, and then at when+period, ... (period &gt; 0 only)</span><span class="hljs-comment">// each time calling f(arg, now) in the timer goroutine, so f must be</span><span class="hljs-comment">// a well-behaved function and not block.</span>when   <span class="hljs-keyword">int64</span>period <span class="hljs-keyword">int64</span>f      <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(<span class="hljs-keyword">interface</span>&#123;&#125;, <span class="hljs-keyword">uintptr</span>)</span><span class="hljs-title">arg</span>    <span class="hljs-title">interface</span></span>&#123;&#125;seq    <span class="hljs-keyword">uintptr</span>&#125;</code></pre></div><p>runtimeTimer结构:是一个接口</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Interface to timers implemented in package runtime.</span><span class="hljs-comment">// Must be in sync with ../runtime/time.go:/^type timer</span><span class="hljs-comment">//同runtime.timer结构一样</span><span class="hljs-keyword">type</span> runtimeTimer <span class="hljs-keyword">struct</span> &#123;tb <span class="hljs-keyword">uintptr</span>i  <span class="hljs-keyword">int</span>when   <span class="hljs-keyword">int64</span>period <span class="hljs-keyword">int64</span> <span class="hljs-comment">//是否是周期运行</span>f      <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(<span class="hljs-keyword">interface</span>&#123;&#125;, <span class="hljs-keyword">uintptr</span>)</span> // <span class="hljs-title">NOTE</span>: <span class="hljs-title">must</span> <span class="hljs-title">not</span> <span class="hljs-title">be</span> <span class="hljs-title">closure</span>,不能是闭包？？？，初始化计时器<span class="hljs-title">arg</span>    <span class="hljs-title">interface</span></span>&#123;&#125;seq    <span class="hljs-keyword">uintptr</span>&#125;</code></pre></div><p>调用的函数:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//1.13</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">addtimer</span><span class="hljs-params">(t *timer)</span></span> &#123;    <span class="hljs-comment">// 得到要被插入的 bucket</span>    tb := t.assignBucket()    <span class="hljs-comment">// 加锁，将timer插入到bucket中</span>    lock(&amp;tb.lock)    ok := tb.addtimerLocked(t)    unlock(&amp;tb.lock)    <span class="hljs-keyword">if</span> !ok &#123;    badTimer()    &#125;&#125;<span class="hljs-comment">//分配timer buckets</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(t *timer)</span> <span class="hljs-title">assignBucket</span><span class="hljs-params">()</span> *<span class="hljs-title">timersBucket</span></span> &#123;    id := <span class="hljs-keyword">uint8</span>(getg().m.p.ptr().id) % timersLen    t.tb = &amp;timers[id].timersBucket    <span class="hljs-keyword">return</span> t.tb&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(tb *timersBucket)</span> <span class="hljs-title">addtimerLocked</span><span class="hljs-params">(t *timer)</span> <span class="hljs-title">bool</span></span> &#123;    t.i = <span class="hljs-built_in">len</span>(tb.t)    tb.t = <span class="hljs-built_in">append</span>(tb.t, t)    <span class="hljs-keyword">if</span> !siftupTimer(tb.t, t.i) &#123;        <span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>    &#125;    <span class="hljs-keyword">if</span> t.i == <span class="hljs-number">0</span> &#123;        <span class="hljs-keyword">if</span> tb.sleeping &amp;&amp; tb.sleepUntil &gt; t.when &#123;            tb.sleeping = <span class="hljs-literal">false</span>            notewakeup(&amp;tb.waitnote)        &#125;        ...        <span class="hljs-keyword">if</span> !tb.created &#123;            tb.created = <span class="hljs-literal">true</span>            <span class="hljs-keyword">go</span> timerproc(tb)        &#125;    &#125;    <span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>&#125;</code></pre></div><p>我们发现有一个bucket的概念:</p><p>在1.13之前，有64个全局的timerBucket，timer整个生命周期全部由timerBucket管理和调度;</p><ul><li><p>然后再调用<code>timerproc</code>方法从堆顶拿timer，判断是否过期，到期就执行</p></li><li><p>bucket中无任务时，会调用<code>goparkunlock</code>来休眠该goroutine</p></li><li><p>至少有一个timer任务时，<code>notetsleepg</code>传入下次到期时间来休眠（<code>notetsleepg</code>其实有调用<code>entrysyscallback</code>触发<code>handoffp</code>，即一定触发到调度）</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">timerproc</span><span class="hljs-params">(tb *timersBucket)</span></span> &#123;    tb.gp = getg()<span class="hljs-keyword">for</span> &#123;lock(&amp;tb.lock)tb.sleeping = <span class="hljs-literal">false</span>now := nanotime()delta := <span class="hljs-keyword">int64</span>(<span class="hljs-number">-1</span>)<span class="hljs-keyword">for</span> &#123;<span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(tb.t) == <span class="hljs-number">0</span> &#123;delta = <span class="hljs-number">-1</span><span class="hljs-keyword">break</span>&#125;t := tb.t[<span class="hljs-number">0</span>]            delta = t.when - now            <span class="hljs-comment">//timer未到期</span><span class="hljs-keyword">if</span> delta &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">break</span>&#125;ok := <span class="hljs-literal">true</span><span class="hljs-keyword">if</span> t.period &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// leave in heap but adjust next time to fire</span>t.when += t.period * (<span class="hljs-number">1</span> + -delta/t.period)<span class="hljs-keyword">if</span> !siftdownTimer(tb.t, <span class="hljs-number">0</span>) &#123;ok = <span class="hljs-literal">false</span>&#125;&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">// remove from heap</span>last := <span class="hljs-built_in">len</span>(tb.t) - <span class="hljs-number">1</span><span class="hljs-keyword">if</span> last &gt; <span class="hljs-number">0</span> &#123;tb.t[<span class="hljs-number">0</span>] = tb.t[last]tb.t[<span class="hljs-number">0</span>].i = <span class="hljs-number">0</span>&#125;tb.t[last] = <span class="hljs-literal">nil</span>tb.t = tb.t[:last]<span class="hljs-keyword">if</span> last &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> !siftdownTimer(tb.t, <span class="hljs-number">0</span>) &#123;ok = <span class="hljs-literal">false</span>&#125;&#125;t.i = <span class="hljs-number">-1</span> <span class="hljs-comment">// mark as removed</span>&#125;f := t.farg := t.argseq := t.sequnlock(&amp;tb.lock)<span class="hljs-keyword">if</span> !ok &#123;badTimer()&#125;...f(arg, seq)lock(&amp;tb.lock)        &#125;        <span class="hljs-comment">//无任务剩下</span><span class="hljs-keyword">if</span> delta &lt; <span class="hljs-number">0</span> || faketime &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// No timers left - put goroutine to sleep.</span>tb.rescheduling = <span class="hljs-literal">true</span>goparkunlock(&amp;tb.lock, waitReasonTimerGoroutineIdle, traceEvGoBlock, <span class="hljs-number">1</span>)<span class="hljs-keyword">continue</span>&#125;<span class="hljs-comment">// At least one timer pending. Sleep until then.</span>tb.sleeping = <span class="hljs-literal">true</span>tb.sleepUntil = now + deltanoteclear(&amp;tb.waitnote)unlock(&amp;tb.lock)notetsleepg(&amp;tb.waitnote, delta)&#125;&#125;</code></pre></div><ul><li>其中<code>notetsleepg</code>会调用<code>notetsleepg_internal</code>,该函数最后实际会调用<code>futexsleep</code>来休眠</li><li>相应的要使用<code>futexwakeup</code>来唤醒</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Atomically,</span><span class="hljs-comment">//if(*addr == val) sleep</span><span class="hljs-comment">// Might be woken up spuriously; that's allowed.</span><span class="hljs-comment">// Don't sleep longer than ns; ns &lt; 0 means forever.</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">futexsleep</span><span class="hljs-params">(addr *<span class="hljs-keyword">uint32</span>, val <span class="hljs-keyword">uint32</span>, ns <span class="hljs-keyword">int64</span>)</span></span> &#123;<span class="hljs-comment">// Some Linux kernels have a bug where futex of</span><span class="hljs-comment">// FUTEX_WAIT returns an internal error code</span><span class="hljs-comment">// as an errno. Libpthread ignores the return value</span><span class="hljs-comment">// here, and so can we: as it says a few lines up,</span><span class="hljs-comment">// spurious wakeups are allowed.</span><span class="hljs-keyword">if</span> ns &lt; <span class="hljs-number">0</span> &#123;futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, <span class="hljs-literal">nil</span>, <span class="hljs-literal">nil</span>, <span class="hljs-number">0</span>)<span class="hljs-keyword">return</span>&#125;<span class="hljs-keyword">var</span> ts timespects.setNsec(ns)futex(unsafe.Pointer(addr), _FUTEX_WAIT_PRIVATE, val, unsafe.Pointer(&amp;ts), <span class="hljs-literal">nil</span>, <span class="hljs-number">0</span>)&#125;<span class="hljs-comment">// If any procs are sleeping on addr, wake up at most cnt.</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">futexwakeup</span><span class="hljs-params">(addr *<span class="hljs-keyword">uint32</span>, cnt <span class="hljs-keyword">uint32</span>)</span></span> &#123;ret := futex(unsafe.Pointer(addr), _FUTEX_WAKE_PRIVATE, cnt, <span class="hljs-literal">nil</span>, <span class="hljs-literal">nil</span>, <span class="hljs-number">0</span>)<span class="hljs-keyword">if</span> ret &gt;= <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">// I don't know that futex wakeup can return</span><span class="hljs-comment">// EAGAIN or EINTR, but if it does, it would be</span><span class="hljs-comment">// safe to loop and call futex again.</span>systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"futexwakeup addr="</span>, addr, <span class="hljs-string">" returned "</span>, ret, <span class="hljs-string">"\n"</span>)&#125;)*(*<span class="hljs-keyword">int32</span>)(unsafe.Pointer(<span class="hljs-keyword">uintptr</span>(<span class="hljs-number">0x1006</span>))) = <span class="hljs-number">0x1006</span>&#125;</code></pre></div><h3 id="1-14版本下">1.14版本下:</h3><p>处理器P中有:<code>timers</code>数组，用作四叉堆:timer的结构:</p><ul><li><code>pp</code>:</li><li><code>when</code>和<code>period</code>: 会在<code>when</code>的时候唤醒，然后下一次<code>when</code>+<code>period</code></li><li><code>f</code>:timer调用的function应该well-behave(???不是很明白)且不能阻塞的</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> p <span class="hljs-keyword">struct</span> &#123;        <span class="hljs-comment">// 保护timers堆读写安全</span>        timersLock mutex        <span class="hljs-comment">// 存放定时器任务</span>        timers []*timer            ,,,&#125;<span class="hljs-comment">//可以对比一下1.13的timer</span><span class="hljs-comment">//runtime.timer</span><span class="hljs-keyword">type</span> timer <span class="hljs-keyword">struct</span> &#123;    <span class="hljs-comment">// If this timer is on a heap, which P's heap it is on.</span><span class="hljs-comment">// puintptr rather than *p to match uintptr in the versions</span><span class="hljs-comment">// of this struct defined in other packages.</span>    pp puintptr  <span class="hljs-comment">// p的位置</span>    <span class="hljs-comment">// Timer wakes up at when, and then at when+period, ... (period &gt; 0 only)</span><span class="hljs-comment">// each time calling f(arg, now) in the timer goroutine, so f must be</span><span class="hljs-comment">// a well-behaved function and not block.</span>    when   <span class="hljs-keyword">int64</span> <span class="hljs-comment">// 到期时间</span>    period <span class="hljs-keyword">int64</span> <span class="hljs-comment">// 周期时间，适合ticker</span>    f      <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(<span class="hljs-keyword">interface</span>&#123;&#125;, <span class="hljs-keyword">uintptr</span>)</span> // 回调方法    <span class="hljs-title">arg</span>    <span class="hljs-title">interface</span></span>&#123;&#125;  <span class="hljs-comment">// 参数</span>    seq    <span class="hljs-keyword">uintptr</span>  <span class="hljs-comment">// 序号</span>    <span class="hljs-comment">//// What to set the when field to in timerModifiedXX status.</span>    nextwhen <span class="hljs-keyword">int64</span> <span class="hljs-comment">// 下次的到期时间</span>    status <span class="hljs-keyword">uint32</span> <span class="hljs-comment">// 状态</span>&#125;</code></pre></div><ul><li><code>status</code>:取值如下所示:</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Values for the timer status field.</span><span class="hljs-keyword">const</span> (<span class="hljs-comment">// Timer has no status set yet.</span>timerNoStatus = <span class="hljs-literal">iota</span><span class="hljs-comment">// Waiting for timer to fire.</span><span class="hljs-comment">// The timer is in some P's heap.</span>timerWaiting<span class="hljs-comment">// Running the timer function.</span><span class="hljs-comment">// A timer will only have this status briefly.</span>timerRunning<span class="hljs-comment">// The timer is deleted and should be removed.</span><span class="hljs-comment">// It should not be run, but it is still in some P's heap.</span>timerDeleted<span class="hljs-comment">// The timer is being removed.</span><span class="hljs-comment">// The timer will only have this status briefly.</span>timerRemoving<span class="hljs-comment">// The timer has been stopped.</span><span class="hljs-comment">// It is not in any P's heap.</span>timerRemoved<span class="hljs-comment">// The timer is being modified.</span><span class="hljs-comment">// The timer will only have this status briefly.</span>timerModifying<span class="hljs-comment">// The timer has been modified to an earlier time.</span><span class="hljs-comment">// The new when value is in the nextwhen field.</span><span class="hljs-comment">// The timer is in some P's heap, possibly in the wrong place.</span>timerModifiedEarlier<span class="hljs-comment">// The timer has been modified to the same or a later time.</span><span class="hljs-comment">// The new when value is in the nextwhen field.</span><span class="hljs-comment">// The timer is in some P's heap, possibly in the wrong place.</span>timerModifiedLater<span class="hljs-comment">// The timer has been modified and is being moved.</span><span class="hljs-comment">// The timer will only have this status briefly.</span>timerMoving)</code></pre></div><h4 id="大概流程">大概流程</h4><ul><li><code>startTimer</code>：<code>time/sleep.go</code>里面的startTimer实际就是runtime里面的startTimer</li><li><code>addTimer</code>:将定时任务放到当前P中</li><li><code>wakeNetPoller(when)</code>: when之前不会wakeup任何一个，全局的<code>sched.lastpoll</code>记录到上一次是否有wakeup，如果无就进行判断,</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// startTimer adds t to the timer heap.</span><span class="hljs-comment">//go:linkname startTimer time.startTimer</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">startTimer</span><span class="hljs-params">(t *timer)</span></span> &#123;<span class="hljs-keyword">if</span> raceenabled &#123;racerelease(unsafe.Pointer(t))&#125;addtimer(t)&#125;<span class="hljs-comment">// addtimer adds a timer to the current P.</span><span class="hljs-comment">// This should only be called with a newly created timer.</span><span class="hljs-comment">// That avoids the risk of changing the when field of a timer in some P's heap,</span><span class="hljs-comment">// which could cause the heap to become unsorted.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">addtimer</span><span class="hljs-params">(t *timer)</span></span> &#123;<span class="hljs-comment">// when must never be negative; otherwise runtimer will overflow</span><span class="hljs-comment">// during its delta calculation and never expire other runtime timers.</span><span class="hljs-keyword">if</span> t.when &lt; <span class="hljs-number">0</span> &#123;t.when = maxWhen&#125;<span class="hljs-keyword">if</span> t.status != timerNoStatus &#123;throw(<span class="hljs-string">"addtimer called with initialized timer"</span>)&#125;t.status = timerWaitingwhen := t.whenpp := getg().m.p.ptr()lock(&amp;pp.timersLock)cleantimers(pp)doaddtimer(pp, t)unlock(&amp;pp.timersLock)wakeNetPoller(when)&#125;<span class="hljs-comment">// wakeNetPoller wakes up the thread sleeping in the network poller,</span><span class="hljs-comment">// if there is one, and if it isn't going to wake up anyhow before</span><span class="hljs-comment">// the when argument.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">wakeNetPoller</span><span class="hljs-params">(when <span class="hljs-keyword">int64</span>)</span></span> &#123;<span class="hljs-keyword">if</span> atomic.Load64(&amp;sched.lastpoll) == <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// In findrunnable we ensure that when polling the pollUntil</span><span class="hljs-comment">// field is either zero or the time to which the current</span><span class="hljs-comment">// poll is expected to run. This can have a spurious wakeup</span><span class="hljs-comment">// but should never miss a wakeup.</span>pollerPollUntil := <span class="hljs-keyword">int64</span>(atomic.Load64(&amp;sched.pollUntil))<span class="hljs-keyword">if</span> pollerPollUntil == <span class="hljs-number">0</span> || pollerPollUntil &gt; when &#123;netpollBreak()&#125;&#125;&#125;</code></pre></div><p>这里注意<code>wakeNetPoller</code>激活netpoll的等待，是在<code>findrunnable</code>方法里面会超时阻塞(再次复习一次):</p><ul><li>初始化<code>netpollinit</code>中， 会创建一个读(netpollBreakRd)，写(netpollBreakWr)的管道(通过<code>nonblockingPipe</code>)</li><li>而激活的<code>wakeNetPoller</code>实际就是往<code>netpollBreakWr</code>管道里面写入东西，自然就会唤醒netPoll</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">var</span> (epfd <span class="hljs-keyword">int32</span> = <span class="hljs-number">-1</span> <span class="hljs-comment">// epoll descriptor</span>netpollBreakRd, netpollBreakWr <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// for netpollBreak</span>)<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">netpollinit</span><span class="hljs-params">()</span></span> &#123;epfd = epollcreate1(_EPOLL_CLOEXEC)<span class="hljs-keyword">if</span> epfd &lt; <span class="hljs-number">0</span> &#123;epfd = epollcreate(<span class="hljs-number">1024</span>)<span class="hljs-keyword">if</span> epfd &lt; <span class="hljs-number">0</span> &#123;<span class="hljs-built_in">println</span>(<span class="hljs-string">"runtime: epollcreate failed with"</span>, -epfd)throw(<span class="hljs-string">"runtime: netpollinit failed"</span>)&#125;closeonexec(epfd)&#125;r, w, errno := nonblockingPipe()<span class="hljs-keyword">if</span> errno != <span class="hljs-number">0</span> &#123;<span class="hljs-built_in">println</span>(<span class="hljs-string">"runtime: pipe failed with"</span>, -errno)throw(<span class="hljs-string">"runtime: pipe failed"</span>)&#125;ev := epollevent&#123;events: _EPOLLIN,&#125;*(**<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(&amp;ev.data)) = &amp;netpollBreakRderrno = epollctl(epfd, _EPOLL_CTL_ADD, r, &amp;ev)<span class="hljs-keyword">if</span> errno != <span class="hljs-number">0</span> &#123;<span class="hljs-built_in">println</span>(<span class="hljs-string">"runtime: epollctl failed with"</span>, -errno)throw(<span class="hljs-string">"runtime: epollctl failed"</span>)&#125;netpollBreakRd = <span class="hljs-keyword">uintptr</span>(r)netpollBreakWr = <span class="hljs-keyword">uintptr</span>(w)&#125;<span class="hljs-comment">// netpollBreak interrupts an epollwait.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">netpollBreak</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-keyword">for</span> &#123;<span class="hljs-keyword">var</span> b <span class="hljs-keyword">byte</span>n := write(netpollBreakWr, unsafe.Pointer(&amp;b), <span class="hljs-number">1</span>)<span class="hljs-keyword">if</span> n == <span class="hljs-number">1</span> &#123;<span class="hljs-keyword">break</span>&#125;<span class="hljs-keyword">if</span> n == -_EINTR &#123;<span class="hljs-keyword">continue</span>&#125;<span class="hljs-keyword">if</span> n == -_EAGAIN &#123;<span class="hljs-keyword">return</span>&#125;<span class="hljs-built_in">println</span>(<span class="hljs-string">"runtime: netpollBreak write failed with"</span>, -n)throw(<span class="hljs-string">"runtime: netpollBreak write failed"</span>)&#125;&#125;</code></pre></div><ul><li>而这一切都是在<code>findrunnable</code>中尝试进行netpoll检查时<code>netpoll(0)</code>方法用到epoll_wait,这里不仅监控到红黑树上的fd,又可以监控到定时任务的等待;</li><li>注意一下<code>checkTimers</code>方法获取<code>pollUntil</code>的方法，下面会继续讲解</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Finds a runnable goroutine to execute.</span><span class="hljs-comment">// Tries to steal from other P's, get g from local or global queue, poll network.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">findrunnable</span><span class="hljs-params">()</span> <span class="hljs-params">(gp *g, inheritTime <span class="hljs-keyword">bool</span>)</span></span> &#123;    ...    <span class="hljs-comment">// Poll network.</span><span class="hljs-comment">// This netpoll is only an optimization before we resort to stealing.</span><span class="hljs-comment">// We can safely skip it if there are no waiters or a thread is blocked</span><span class="hljs-comment">// in netpoll already. If there is any kind of logical race with that</span><span class="hljs-comment">// blocked thread (e.g. it has already returned from netpoll, but does</span><span class="hljs-comment">// not set lastpoll yet), this thread will do blocking netpoll below</span>    <span class="hljs-comment">// anyway.</span>    <span class="hljs-comment">//这里是非阻塞，还没进去呢</span><span class="hljs-keyword">if</span> netpollinited() &amp;&amp; atomic.Load(&amp;netpollWaiters) &gt; <span class="hljs-number">0</span> &amp;&amp; atomic.Load64(&amp;sched.lastpoll) != <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> list := netpoll(<span class="hljs-number">0</span>); !list.empty() &#123; <span class="hljs-comment">// non-blocking</span>gp := list.pop()injectglist(&amp;list)casgstatus(gp, _Gwaiting, _Grunnable)<span class="hljs-keyword">if</span> trace.enabled &#123;traceGoUnpark(gp, <span class="hljs-number">0</span>)&#125;<span class="hljs-keyword">return</span> gp, <span class="hljs-literal">false</span>&#125;    &#125;    ...    <span class="hljs-comment">//尝试4次，从其他p的runq偷，再从其他p的timer偷</span>    <span class="hljs-keyword">for</span> i := <span class="hljs-number">0</span>; i &lt; <span class="hljs-number">4</span>; i++ &#123;<span class="hljs-keyword">for</span> enum := stealOrder.start(fastrand()); !enum.done(); enum.next() &#123;<span class="hljs-keyword">if</span> sched.gcwaiting != <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">goto</span> top&#125;stealRunNextG := i &gt; <span class="hljs-number">2</span> <span class="hljs-comment">// first look for ready queues with more than 1 g</span>p2 := allp[enum.position()]<span class="hljs-keyword">if</span> _p_ == p2 &#123;<span class="hljs-keyword">continue</span>&#125;<span class="hljs-keyword">if</span> gp := runqsteal(_p_, p2, stealRunNextG); gp != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> gp, <span class="hljs-literal">false</span>&#125;<span class="hljs-comment">// Consider stealing timers from p2.</span><span class="hljs-comment">// This call to checkTimers is the only place where</span><span class="hljs-comment">// we hold a lock on a different P's timers.</span><span class="hljs-comment">// Lock contention can be a problem here, so avoid</span><span class="hljs-comment">// grabbing the lock if p2 is running and not marked</span><span class="hljs-comment">// for preemption. If p2 is running and not being</span>            <span class="hljs-comment">// preempted we assume it will handle its own timers.</span>            <span class="hljs-comment">//p2在运行且不在被抢占中，才可以认为其可以处理自己的timers</span><span class="hljs-keyword">if</span> i &gt; <span class="hljs-number">2</span> &amp;&amp; shouldStealTimers(p2) &#123;                <span class="hljs-comment">//执行已经到期的定时任务</span>                <span class="hljs-comment">//注意这里返回的w，下面详细讲解...</span>tnow, w, ran := checkTimers(p2, now)                now = tnow                <span class="hljs-keyword">if</span> w != <span class="hljs-number">0</span> &amp;&amp; (pollUntil == <span class="hljs-number">0</span> || w &lt; pollUntil) &#123;pollUntil = w&#125;<span class="hljs-keyword">if</span> ran &#123;<span class="hljs-comment">// Running the timers may have</span><span class="hljs-comment">// made an arbitrary number of G's</span><span class="hljs-comment">// ready and added them to this P's</span><span class="hljs-comment">// local run queue. That invalidates</span><span class="hljs-comment">// the assumption of runqsteal</span><span class="hljs-comment">// that is always has room to add</span><span class="hljs-comment">// stolen G's. So check now if there</span><span class="hljs-comment">// is a local G to run.</span><span class="hljs-keyword">if</span> gp, inheritTime := runqget(_p_); gp != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> gp, inheritTime&#125;ranTimer = <span class="hljs-literal">true</span>&#125;&#125;&#125;&#125;    <span class="hljs-comment">//计算delta值，距离当前时间最近的时间点的时间差</span>    delta := <span class="hljs-keyword">int64</span>(<span class="hljs-number">-1</span>)<span class="hljs-keyword">if</span> pollUntil != <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// checkTimers ensures that polluntil &gt; now.</span>delta = pollUntil - now&#125;    ...    <span class="hljs-comment">//最后还要进行一次阻塞的pollnetwork</span>    <span class="hljs-comment">// poll network</span><span class="hljs-keyword">if</span> netpollinited() &amp;&amp; (atomic.Load(&amp;netpollWaiters) &gt; <span class="hljs-number">0</span> || pollUntil != <span class="hljs-number">0</span>) &amp;&amp; atomic.Xchg64(&amp;sched.lastpoll, <span class="hljs-number">0</span>) != <span class="hljs-number">0</span> &#123;atomic.Store64(&amp;sched.pollUntil, <span class="hljs-keyword">uint64</span>(pollUntil))<span class="hljs-keyword">if</span> _g_.m.p != <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"findrunnable: netpoll with p"</span>)&#125;<span class="hljs-keyword">if</span> _g_.m.spinning &#123;throw(<span class="hljs-string">"findrunnable: netpoll with spinning"</span>)&#125;<span class="hljs-keyword">if</span> faketime != <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// When using fake time, just poll.</span>delta = <span class="hljs-number">0</span>&#125;list := netpoll(delta) <span class="hljs-comment">// block until new work is available</span>atomic.Store64(&amp;sched.pollUntil, <span class="hljs-number">0</span>)atomic.Store64(&amp;sched.lastpoll, <span class="hljs-keyword">uint64</span>(nanotime()))<span class="hljs-keyword">if</span> faketime != <span class="hljs-number">0</span> &amp;&amp; list.empty() &#123;<span class="hljs-comment">// Using fake time and nothing is ready; stop M.</span><span class="hljs-comment">// When all M's stop, checkdead will call timejump.</span>stopm()<span class="hljs-keyword">goto</span> top&#125;&#125;<span class="hljs-comment">// netpoll checks for ready network connections.</span><span class="hljs-comment">// Returns list of goroutines that become runnable.</span><span class="hljs-comment">// delay &lt; 0: blocks indefinitely</span><span class="hljs-comment">// delay == 0: does not block, just polls</span><span class="hljs-comment">// delay &gt; 0: block for up to that many nanoseconds</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">netpoll</span><span class="hljs-params">(delay <span class="hljs-keyword">int64</span>)</span> <span class="hljs-title">gList</span></span> &#123;    ...retry:    n := epollwait(epfd, &amp;events[<span class="hljs-number">0</span>], <span class="hljs-keyword">int32</span>(<span class="hljs-built_in">len</span>(events)), waitms)    ...&#125;</code></pre></div><h4 id="checktimers方法">CheckTimers方法</h4><p>继续上面的<code>checkTimers</code>方法:该方法主要是检查传入P上任何准备就绪的timers，通过<code>runtimers</code>来运行到期的定时任务，返回<code>下一次到期时间</code>以及<code>是否有定时任务到期</code></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// checkTimers runs any timers for the P that are ready.</span><span class="hljs-comment">// If now is not 0 it is the current time.</span><span class="hljs-comment">// It returns the current time or 0 if it is not known,</span><span class="hljs-comment">// and the time when the next timer should run or 0 if there is no next timer,</span><span class="hljs-comment">// and reports whether it ran any timers.</span><span class="hljs-comment">// If the time when the next timer should run is not 0,</span><span class="hljs-comment">// it is always larger than the returned time.</span><span class="hljs-comment">// We pass now in and out to avoid extra calls of nanotime.</span><span class="hljs-comment">//go:yeswritebarrierrec</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">checkTimers</span><span class="hljs-params">(pp *p, now <span class="hljs-keyword">int64</span>)</span> <span class="hljs-params">(rnow, pollUntil <span class="hljs-keyword">int64</span>, ran <span class="hljs-keyword">bool</span>)</span></span> &#123;<span class="hljs-comment">// If there are no timers to adjust, and the first timer on</span><span class="hljs-comment">// the heap is not yet ready to run, then there is nothing to do.</span><span class="hljs-keyword">if</span> atomic.Load(&amp;pp.adjustTimers) == <span class="hljs-number">0</span> &#123;next := <span class="hljs-keyword">int64</span>(atomic.Load64(&amp;pp.timer0When))<span class="hljs-keyword">if</span> next == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span> now, <span class="hljs-number">0</span>, <span class="hljs-literal">false</span>&#125;<span class="hljs-keyword">if</span> now == <span class="hljs-number">0</span> &#123;now = nanotime()&#125;<span class="hljs-keyword">if</span> now &lt; next &#123;<span class="hljs-comment">// Next timer is not ready to run.</span><span class="hljs-comment">// But keep going if we would clear deleted timers.</span><span class="hljs-comment">// This corresponds to the condition below where</span><span class="hljs-comment">// we decide whether to call clearDeletedTimers.</span><span class="hljs-keyword">if</span> pp != getg().m.p.ptr() || <span class="hljs-keyword">int</span>(atomic.Load(&amp;pp.deletedTimers)) &lt;= <span class="hljs-keyword">int</span>(atomic.Load(&amp;pp.numTimers)/<span class="hljs-number">4</span>) &#123;<span class="hljs-keyword">return</span> now, next, <span class="hljs-literal">false</span>&#125;&#125;&#125;lock(&amp;pp.timersLock)adjusttimers(pp)rnow = now<span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(pp.timers) &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> rnow == <span class="hljs-number">0</span> &#123;rnow = nanotime()&#125;<span class="hljs-keyword">for</span> <span class="hljs-built_in">len</span>(pp.timers) &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// Note that runtimer may temporarily unlock</span>            <span class="hljs-comment">// pp.timersLock.</span>            <span class="hljs-comment">//运行定时任务</span><span class="hljs-keyword">if</span> tw := runtimer(pp, rnow); tw != <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> tw &gt; <span class="hljs-number">0</span> &#123;pollUntil = tw&#125;<span class="hljs-keyword">break</span>&#125;ran = <span class="hljs-literal">true</span>&#125;&#125;<span class="hljs-comment">// If this is the local P, and there are a lot of deleted timers,</span><span class="hljs-comment">// clear them out. We only do this for the local P to reduce</span><span class="hljs-comment">// lock contention on timersLock.</span><span class="hljs-keyword">if</span> pp == getg().m.p.ptr() &amp;&amp; <span class="hljs-keyword">int</span>(atomic.Load(&amp;pp.deletedTimers)) &gt; <span class="hljs-built_in">len</span>(pp.timers)/<span class="hljs-number">4</span> &#123;clearDeletedTimers(pp)&#125;unlock(&amp;pp.timersLock)<span class="hljs-keyword">return</span> rnow, pollUntil, ran&#125;</code></pre></div><p>其中<code>runtimers</code>方法:</p><ul><li>遍历p的timer堆顶任务是否到期，如果是<code>timerWaiting</code>状态，进入尝试执行</li><li><code>runOneTimer</code>尝试执行，如果是周期任务，还会重新入队;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// runtimer examines the first timer in timers. If it is ready based on now,</span><span class="hljs-comment">// it runs the timer and removes or updates it.</span><span class="hljs-comment">// Returns 0 if it ran a timer, -1 if there are no more timers, or the time</span><span class="hljs-comment">// when the first timer should run.</span><span class="hljs-comment">// The caller must have locked the timers for pp.</span><span class="hljs-comment">// If a timer is run, this will temporarily unlock the timers.</span><span class="hljs-comment">//go:systemstack</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">runtimer</span><span class="hljs-params">(pp *p, now <span class="hljs-keyword">int64</span>)</span> <span class="hljs-title">int64</span></span> &#123;<span class="hljs-keyword">for</span> &#123;t := pp.timers[<span class="hljs-number">0</span>]<span class="hljs-keyword">if</span> t.pp.ptr() != pp &#123;throw(<span class="hljs-string">"runtimer: bad p"</span>)&#125;<span class="hljs-keyword">switch</span> s := atomic.Load(&amp;t.status); s &#123;<span class="hljs-keyword">case</span> timerWaiting:<span class="hljs-keyword">if</span> t.when &gt; now &#123;<span class="hljs-comment">// Not ready to run.</span><span class="hljs-keyword">return</span> t.when&#125;<span class="hljs-keyword">if</span> !atomic.Cas(&amp;t.status, s, timerRunning) &#123;<span class="hljs-keyword">continue</span>&#125;<span class="hljs-comment">// Note that runOneTimer may temporarily unlock</span><span class="hljs-comment">// pp.timersLock.</span>runOneTimer(pp, t, now)<span class="hljs-keyword">return</span> <span class="hljs-number">0</span><span class="hljs-keyword">case</span> timerDeleted:<span class="hljs-keyword">if</span> !atomic.Cas(&amp;t.status, s, timerRemoving) &#123;<span class="hljs-keyword">continue</span>&#125;dodeltimer0(pp)<span class="hljs-keyword">if</span> !atomic.Cas(&amp;t.status, timerRemoving, timerRemoved) &#123;badTimer()&#125;atomic.Xadd(&amp;pp.deletedTimers, <span class="hljs-number">-1</span>)<span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(pp.timers) == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-number">-1</span>&#125;<span class="hljs-keyword">case</span> timerModifiedEarlier, timerModifiedLater:<span class="hljs-keyword">if</span> !atomic.Cas(&amp;t.status, s, timerMoving) &#123;<span class="hljs-keyword">continue</span>&#125;t.when = t.nextwhendodeltimer0(pp)doaddtimer(pp, t)<span class="hljs-keyword">if</span> s == timerModifiedEarlier &#123;atomic.Xadd(&amp;pp.adjustTimers, <span class="hljs-number">-1</span>)&#125;<span class="hljs-keyword">if</span> !atomic.Cas(&amp;t.status, timerMoving, timerWaiting) &#123;badTimer()&#125;<span class="hljs-keyword">case</span> timerModifying:<span class="hljs-comment">// Wait for modification to complete.</span>osyield()<span class="hljs-keyword">case</span> timerNoStatus, timerRemoved:<span class="hljs-comment">// Should not see a new or inactive timer on the heap.</span>badTimer()<span class="hljs-keyword">case</span> timerRunning, timerRemoving, timerMoving:<span class="hljs-comment">// These should only be set when timers are locked,</span><span class="hljs-comment">// and we didn't do it.</span>badTimer()<span class="hljs-keyword">default</span>:badTimer()&#125;&#125;&#125;<span class="hljs-comment">// runOneTimer runs a single timer.</span><span class="hljs-comment">// The caller must have locked the timers for pp.</span><span class="hljs-comment">// This will temporarily unlock the timers while running the timer function.</span><span class="hljs-comment">//go:systemstack</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">runOneTimer</span><span class="hljs-params">(pp *p, t *timer, now <span class="hljs-keyword">int64</span>)</span></span> &#123;...f := t.farg := t.argseq := t.seq<span class="hljs-keyword">if</span> t.period &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// Leave in heap but adjust next time to fire.</span>delta := t.when - nowt.when += t.period * (<span class="hljs-number">1</span> + -delta/t.period)siftdownTimer(pp.timers, <span class="hljs-number">0</span>)<span class="hljs-keyword">if</span> !atomic.Cas(&amp;t.status, timerRunning, timerWaiting) &#123;badTimer()&#125;updateTimer0When(pp)&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">// Remove from heap.</span>dodeltimer0(pp)<span class="hljs-keyword">if</span> !atomic.Cas(&amp;t.status, timerRunning, timerNoStatus) &#123;badTimer()&#125;&#125;...unlock(&amp;pp.timersLock)f(arg, seq)lock(&amp;pp.timersLock)    ...&#125;</code></pre></div><h4 id="reset-d-duration-方法">Reset(d Duration)方法</h4><ol><li>返回true，如果这个timer被激活；返回false如果这个timer被停止或者过期(返回值最主要用来保持兼容性)</li><li>Reset只能被 停止或者过期的带有空(队列已经为空)channels的timers 调用</li><li></li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Reset changes the timer to expire after duration d.</span><span class="hljs-comment">// It returns true if the timer had been active, false if the timer had</span><span class="hljs-comment">// expired or been stopped.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Reset should be invoked only on stopped or expired timers with drained channels.</span><span class="hljs-comment">// If a program has already received a value from t.C, the timer is known</span><span class="hljs-comment">// to have expired and the channel drained, so t.Reset can be used directly.</span><span class="hljs-comment">// If a program has not yet received a value from t.C, however,</span><span class="hljs-comment">// the timer must be stopped and—if Stop reports that the timer expired</span><span class="hljs-comment">// before being stopped—the channel explicitly drained:</span><span class="hljs-comment">//</span><span class="hljs-comment">// if !t.Stop() &#123;</span><span class="hljs-comment">// &lt;-t.C</span><span class="hljs-comment">// &#125;</span><span class="hljs-comment">// t.Reset(d)</span><span class="hljs-comment">//</span><span class="hljs-comment">// This should not be done concurrent to other receives from the Timer's</span><span class="hljs-comment">// channel.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Note that it is not possible to use Reset's return value correctly, as there</span><span class="hljs-comment">// is a race condition between draining the channel and the new timer expiring.</span><span class="hljs-comment">// Reset should always be invoked on stopped or expired channels, as described above.</span><span class="hljs-comment">// The return value exists to preserve compatibility with existing programs.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(t *Timer)</span> <span class="hljs-title">Reset</span><span class="hljs-params">(d Duration)</span> <span class="hljs-title">bool</span></span> &#123;<span class="hljs-keyword">if</span> t.r.f == <span class="hljs-literal">nil</span> &#123;<span class="hljs-built_in">panic</span>(<span class="hljs-string">"time: Reset called on uninitialized Timer"</span>)&#125;w := when(d)active := stopTimer(&amp;t.r)t.r.when = wstartTimer(&amp;t.r)<span class="hljs-keyword">return</span> active&#125;</code></pre></div><h2 id="性能分析">性能分析</h2><ol><li><p>锁竞争?</p><p>go1.14虽然将timer放在timer内，但是p操作heap也是需要锁的(<code>findrunnable</code>可能要偷其他p的timers)；go1.13<code>timerProcs</code>有timers和对应的锁,最大64个,但是最大的问题是<code>timerproc</code>操作<code>notetsleepg</code>会有系统调用，接着<code>handoffp</code>，这里要涉及全局<code>sched.lock</code>锁;</p></li><li><p>runtime调度次数</p><p>go1.13的<code>timerproc</code>本身就是属于goroutine，受到runtime的调度影响；go1.14则将timer的工作交给了<code>runtime.schedule</code>，不需要额外调度;</p></li><li><p>上下文切换新增任务时，futex(在竞争导致操作结果不一致的时候还是会进入kernel)和epoll_wait都是syscall，无区别；但是go1.14无<code>timerproc</code>,新任务可以直接插入或者多次插入后再考虑是否休眠;</p></li></ol><h2 id="引出">引出</h2><h3 id="race-detector-竞态条件检测">Race Detector(竞态条件检测)</h3><p>很简单:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">go</span> run -race main.<span class="hljs-keyword">go</span> <span class="hljs-keyword">go</span> build -race mycmd   <span class="hljs-comment">// build the command</span><span class="hljs-keyword">go</span> install -race mypkg <span class="hljs-comment">// install the package</span></code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//官网的例子：</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span> &#123;     start := time.Now()    <span class="hljs-keyword">var</span> t *time.Timer     t = time.AfterFunc(randomDuration(), <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-comment">//这里的t是写入</span>         fmt.Println(time.Now().Sub(start))         t.Reset(randomDuration()) <span class="hljs-comment">//这里的t是读取，可能在一定情况下，randomDuration使得在这里读取前将t置为nil，所以读取了空值，会报nil pointer错误</span>     &#125;)     time.Sleep(<span class="hljs-number">5</span> * time.Second) &#125;  <span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">randomDuration</span><span class="hljs-params">()</span> <span class="hljs-title">time</span>.<span class="hljs-title">Duration</span></span> &#123;     <span class="hljs-keyword">return</span> time.Duration(rand.Int63n(<span class="hljs-number">1e9</span>)) &#125;</code></pre></div><p>改成以下版本:主要是让t变量只能从main的goroutine中读取和写入</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span> &#123;    start := time.Now()    reset := <span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">bool</span>)    <span class="hljs-keyword">var</span> t *time.Timer    t = time.AfterFunc(randomDuration(), <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;        fmt.Println(time.Now().Sub(start))        reset &lt;- <span class="hljs-literal">true</span>    &#125;)    <span class="hljs-keyword">for</span> time.Since(start) &lt; <span class="hljs-number">5</span>*time.Second &#123;        &lt;-reset        t.Reset(randomDuration())    &#125;&#125;</code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Reviews on I/O design</title>
    <link href="/2020/01/30/Comcon/DesignIO/"/>
    <url>/2020/01/30/Comcon/DesignIO/</url>
    
    <content type="html"><![CDATA[<a id="more"></a><h2 id="i-o设计模式">I/O设计模式</h2>]]></content>
    
    
    
    <tags>
      
      <tag>Design Pattern</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>IO Model</title>
    <link href="/2020/01/15/Comcon/IOModel/"/>
    <url>/2020/01/15/Comcon/IOModel/</url>
    
    <content type="html"><![CDATA[<a id="more"></a><h2 id="io模型归纳">IO模型归纳</h2><p>我们暂时只以网络IO为例子;我们先从硬件层面进行流程分析:</p><h3 id="1-硬件层面">1. 硬件层面</h3><h4 id="网卡">网卡</h4><p>网卡等于一种可以接收外面数据流的一种设备；</p><ol><li>网卡会先从网线处（或者无线）接收里面的数据</li><li>以某种方式<strong>写入到内存</strong>中</li></ol><p>其中涉及DMA，IO通路选择等硬件问题，我们暂时忽略</p><h4 id="cpu">CPU</h4><p>接着，我们都知道一般操作系统针对一些比较紧急的操作会进行<strong>中断(软，硬)</strong>，当网卡来数据的时候，明显就是比较紧急的事情（如果忽略不管，就会造成内存爆炸或者数据丢失）；</p><p>所以，数据从网卡接收到，会立即发送一个信号到CPU，网卡的中断程序就会被调用去处理数据；</p><h4 id="操作系统调度">操作系统调度</h4><h3 id="2-模型选择">2. 模型选择</h3><h4 id="同步">同步</h4><ol><li>阻塞:</li></ol><p>只能有一个I/O操作；</p><ol start="2"><li>非阻塞:</li></ol><p>指的是 可以同时阻塞多个I/O操作，所以看做是非阻塞的</p><h5 id="select">select:</h5><ul><li><p>优点：实现简单，只用单线程执行，占用资源少，不消耗过多CPU，也可以为多客户端提供服务;</p></li><li><p>缺点：每次调用都都要将fd_set从用户态copy到内核态，消耗过大;每次调用select都要进行多次轮询;可以检测到fd数量有限，默认是1024;因为是<code>水平触发</code>，如果没有完成对一个就绪fd进行IO，下次select也会将这些fd通知到进程(只能算的是特点吧);</p></li><li><p>可以看到select就是将事件的探测和响应放在一起，一旦响应过大，就会造成问题;</p></li></ul><h5 id="poll">poll</h5><p>与select区别不大，相同的是</p><ol><li>创建fd_set,设置关注事件</li><li>调用poll()，等待事件发生;</li></ol><p>区别:</p><ol><li>select要为读、写、异常都分别设置fd_set, poll只需要为fd设置一个即可</li><li>poll由链表实现，所以无最大值</li></ol><p>缺点:同select一样，用户态到内核态的copy;水平触发;</p><h5 id="epoll">epoll</h5><p>将fd交给了内核，一旦事件发生，内核负责通知;其支持<code>水平触发</code>和<code>边缘触发</code>，还有多个函数<code>epoll_create</code>,<code>epoll_wait</code>,<code>epoll_ctl</code>分别为创建epoll句柄，等待事件发生，注册要监听的事件;</p><p>优点:</p><ul><li>无最大并发连接数目限制</li><li>效率提升，只有活跃的fd才会callback</li><li>内存是由一块用户态和内核态通过<code>mmap()</code>共享内存，避免频繁copy，保证了整个过程只会copy一次;</li></ul>]]></content>
    
    
    
    <tags>
      
      <tag>networking</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Notes about Strings</title>
    <link href="/2019/08/18/Comcon/StringCompare/"/>
    <url>/2019/08/18/Comcon/StringCompare/</url>
    
    <content type="html"><![CDATA[<p>今天改同事的代码，在匹配字符串方面发现了点东西：levenshtein算法</p><a id="more"></a><h2 id="levenshtein">Levenshtein</h2><p>levenshtein距离指的就是从一个字符串到另外一个字符串中编辑单个字符所需要的次数（删除，更改，插入）举个 :chestnut: :cat --&gt; cite 的levenshtein是2</p><ol><li>cat–&gt; cit (a–&gt;i)</li><li>cit–&gt; cite (_ --&gt;e)</li></ol><h3 id="why">why?</h3><p>为啥子是这样呢，我们可以从矩阵的可视化来讲解:</p><h3 id="normalized-edit-distance">Normalized Edit distance</h3><p>Given two strings X and Y over a finite alphabet, the normalized edit distance between X and Y, d( X , Y ) is defined as the minimum of W( P ) / L ( P )w, here P is an editing path between X and Y , W ( P ) is the sum of the weights of the elementary edit operations of P, and L§ is the number of these operations (length of P).</p><h2 id="jaro-wrinkler">Jaro Wrinkler</h2><p>详细解释可见![wiki]<a href="https://en.wikipedia.org/wiki/Jaro%E2%80%93Winkler_distance" target="_blank" rel="noopener">https://en.wikipedia.org/wiki/Jaro–Winkler_distance</a></p><p>同样也是计算两个字符串的相似程度,但偏向于单词的前缀匹配</p>]]></content>
    
    
    
    <tags>
      
      <tag>String</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Go Scheduler</title>
    <link href="/2019/07/09/Go/Goscheduler/"/>
    <url>/2019/07/09/Go/Goscheduler/</url>
    
    <content type="html"><![CDATA[<p>东抄西拼，Goroutine 的模型，调度等，它与普通thread有何区别？先留个坑</p><a id="more"></a><h2 id="为什么有这个东西？">为什么有这个东西？</h2><ol><li><p>传统OS自带的线程一个占栈1MB，明显大的过分，所以编程语言自身得另外实现一些小的线程, 而goroutines一般就4KB左右，当然这个数值是可以调整的；</p></li><li><p>切换上下文的时候一般一个线程就消耗1μs,但是goroutine的切换则仅仅有0.2μs左右，大约快了80%;（里面避免了内核和用户态上的切换）</p></li></ol><h3 id="引用">引用</h3><p>《 Scalable Go Scheduler Design Doc》中有描述</p><blockquote><p>Goroutines are part of making concurrency easy to use. The idea, which has been around for a while, is to multiplex independently    executing functions—coroutines—onto a set of threads. When a coroutine blocks, such as by calling a blocking system call, the run-time automatically moves other coroutines on the same operating system thread to a different, runnable thread so they won’t be blocked. The programmer sees none of this, which is the point. The result, which we call goroutines, can be very cheap: unless they spend a lot of time in long-running system calls, they cost little more than the memory for the stack, which is just a few kilobytes.</p></blockquote><p>大概意思就是 当系统调用阻塞，runtime环境会自动把被阻塞在当前线程内的coroutines移到另一个线程，这种在go里面就叫goroutines;</p><p>而针对goroutines的大小，也做了如下设计:</p><blockquote><p>To make the stacks small, Go’s run-time uses segmented stacks. A newly minted goroutine is given a few kilobytes, which is almost always enough. When it isn’t, the run-time allocates (and frees) extension segments automatically. The overhead averages about three cheap instructions per function call. It is practical to create hundreds of thousands of goroutines in the same address space. If goroutines were just threads, system resources would run out at a much smaller number.</p></blockquote><p>然后对于goroutine的栈设计，使用了<strong>分段</strong>的栈， 而且对于分段的栈增加了灵活性，当空间不足的话就会自动分配更多的空间，而且因为这个不涉及内核层面，不用保存过多信息，所以你可以在同一个地址空间里创建上千个goroutines</p><ul><li><p>这些分段栈的基本功能</p><ol><li>保护回复上下文的函数</li><li>运行队列processQueue</li></ol></li></ul><p>要时刻明白对于线程来讲，其<strong>阻塞指的是切换了调度队列</strong>，不再进行当前的<strong>数据控制流</strong>，如果其他流满足条件，则会移出当前队列，调度会之前的数据流。同理goroutine也只是一个结构，记录了运行的函数，运行的位置等</p><h2 id="大致工作原理">大致工作原理</h2><p>首先go现在版本(1.13)已经是基于协作的抢占式调度;</p><p>根据历史提交，有多个部分与其相关:</p><ul><li><p><code>goroutine.stackgurad0 = stackPreempt</code>证明在抢占中</p></li><li><p><code>runtime.preemptone</code>和<code>runtime.preemptall</code>会改变<code>stackguard0</code>字段</p></li><li><p><code>runtime.stoptheworld</code>调用<code>runtime.preemptall</code>设置所有cpu上运行的goroutine<code>stackguard0</code>=stackPreempt</p></li><li><p><code>runtime.newstack</code>增加了抢占的代码，<code>canPreempt()</code>方法会让出当前goroutine</p></li><li><p>在sysmon下,运行超过10ms的goroutine,<code>runtime.retake</code>和<code>runtime.preemptone</code>会被执行</p></li></ul><p>综上，实现其调度大概步骤:</p><ol><li>编译器在其函数前插入<code>runtime.morestack</code></li><li>在垃圾回收stw，sysmon发现goroutine运行超过10ms，就发出抢占<code>stackPreempt</code></li><li>有函数被call时，可能会触发编译器插入的<code>morestack</code>,其调用了<code>newstack()</code>会检查goroutine的<code>stackguard0</code>字段,如果是<code>stackPreempt</code>,就可以被抢占</li></ol><p>以上造成的一些后果，loop下死循环等等，大部分在1.14得以解决</p><p>在1.14版本中，实现了非协作式抢占调度(增加了新的状态和字段);</p><ul><li><p>挂起的goroutine是在gc的栈扫描(<code>markroot</code>)时完成的，由<code>runtime.suspendG</code>和<code>runtime.resumeG</code>两个函数重构栈扫描这一过程；<code>runtime.suspendG</code>会将处于<code>_Grunning</code>状态的goroutine的<code>preemptStop</code>设为true;<code>runtime.preemptPark</code>(<code>newstack</code>也用到,当<code>preemptStop</code>为true时进行挂起)可以挂起当前goroutine，将其状态更新为<code>_Gpreempted</code>，触发重新调度,并让出当前线程控制权；</p></li><li><p>增加<code>runtime.asyncPreempt</code> 和 <code>runtime.asyncPreempt2</code>异步抢占 (汇编实现???)todo，并在<code>runtime.preemptone</code>增加异步逻辑;</p></li><li><p>支持向goroutine发送信号来暂停，<code>runtime.sighandler</code>函数注册<code>SIGURG</code>(<s>为什么是这个信号?为啥不是<code>SIGALRM</code>或者其他???</s>见下面)处理函数<code>runtime.doSigPreempt</code>实现<code>runtime.preemptM</code>,通过<code>SIGURG</code>向线程发送抢占;</p></li></ul><p>为什么使用<code>SIGURG</code>,注释中写了:</p><blockquote><blockquote><blockquote><p>// 1. It should be a signal that’s passed-through by debuggers by// default. On Linux, this is SIGALRM, SIGURG, SIGCHLD, SIGIO,// SIGVTALRM, SIGPROF, and SIGWINCH, plus some glibc-internal signals.//// 2. It shouldn’t be used internally by libc in mixed Go/C binaries// because libc may assume it’s the only thing that can handle these// signals. For example SIGCANCEL or SIGSETXID.//// 3. It should be a signal that can happen spuriously without// consequences. For example, SIGALRM is a bad choice because the// signal handler can’t tell if it was caused by the real process// alarm or not (arguably this means the signal is broken, but I// digress). SIGUSR1 and SIGUSR2 are also bad because those are often// used in meaningful ways by applications.//// 4. We need to deal with platforms without real-time signals (like// macOS), so those are out.</p></blockquote></blockquote></blockquote><p>大概意思就是</p><ol><li>要可以被debugger传输</li><li>不应该被libc使用</li><li>要有基本时序性，可以随意出现(SIGALARM就不行，不知道是不是由进程引起还是其他原因)</li><li>一些平台无实时信号(macOS)</li></ol><h2 id="调度模型">调度模型</h2><p>一般来说多线程调度模型有 work-sharing 和 work-stealing模型</p><p>go采用了后者，可以看看有关<a href="http://supertech.csail.mit.edu/papers/steal.pdf" target="_blank" rel="noopener">work-stealing的论文</a></p><p>架构：GPM</p><ul><li>G（goroutine）指的是go语言的goroutine（有些人叫它为协程，但其实跟coroutine有一点区别，因为coroutine单纯在用户态使用）</li></ul><h3 id="scheduler调度过程">Scheduler调度过程</h3><h4 id="大概流程">大概流程</h4><p>调度前的检查:</p><ol><li>是否分配有gc mark，如果有则要做gc mark</li><li>检查有无localq，有就运行</li><li>没有则看globalq</li><li>看一下net中有无poll的出来</li><li>从其他的p 偷一部分</li></ol><h4 id="禁止抢占">禁止抢占:</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">runtime_procPin</span><span class="hljs-params">()</span> <span class="hljs-title">int</span> //标记当前<span class="hljs-title">G</span>在<span class="hljs-title">M</span>上不会被抢占，并返回当前<span class="hljs-title">P</span>的<span class="hljs-title">ID</span></span></code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">runtime_procUnpin</span><span class="hljs-params">()</span> //解除抢占标志</span></code></pre></div><p>这里我们有两种方法：我们可以立即block或unblock多个M，或使其自旋；但这里会有性能损耗和花费不必要的cpu周期，方法是使用自旋而且burn CPU cycles(???)然而，这不应该影响在GOMAXPROCS=1的程序（command line，appengine这些）</p><p>自旋有两个等级：</p><ol><li>一个已经附着了一个P的idle M 会不断自旋寻找新的Goroutines</li><li>一个已经附着了一个P的M 自旋等待其他可用的P以上中，最多有GOMAXPROCS个自旋的goroutines， 等级（1）的idle M 不会阻塞 即使有等级（2）的idle M；当新的goroutine被创建或者M进入syscall或者M从idle变成busy，它会保证至少存在一个自旋M （或者所有P是busy），</li><li>这就保证了不会有当前运行着的Goroutines被其他 M 运行</li><li>也避免了过多的 M 同时 阻塞和释放阻塞</li></ol><h3 id="sysmon">Sysmon</h3><h4 id="用途">用途</h4><p>如果一个syscall或者是G本身的任务太长，当前G一直阻塞（因为本地队列的G是顺序执行），中止长任务就要由另外一个监控程序，即Sysmon</p><h4 id="详情">详情</h4><p>sysmon是在<strong>runtime初始化之后，执行代码之前</strong>，由runtime启动且<strong>不与任何P绑定</strong>（所以才可以不被阻塞）直接由一个M执行的协程，类似于linux的一些系统任务内核线程</p><p>具体设置如<img src="/img/sysmon.png" srcset="/img/loading.gif" alt="sysmon状态转换图"></p><p>具体流程为:</p><ol><li>main函数启动时:</li></ol><ul><li>调用<code>newm</code>创建一个存储了待执行函数（这里是<code>sysmon</code>的结构体<code>runtime.m</code>),注意其不需要P的绑定，所以goroutine会直接在这个m上面建立;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// The main goroutine.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span> &#123;...<span class="hljs-keyword">if</span> GOARCH != <span class="hljs-string">"wasm"</span> &#123; <span class="hljs-comment">// no threads on wasm yet, so no sysmon</span>systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;newm(sysmon, <span class="hljs-literal">nil</span>)&#125;)&#125;&#125;<span class="hljs-comment">// Create a new m. It will start off with a call to fn, or else the scheduler.</span><span class="hljs-comment">// fn needs to be static and not a heap allocated closure.</span><span class="hljs-comment">// May run with m.p==nil, so write barriers are not allowed.</span><span class="hljs-comment">//go:nowritebarrierrec</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newm</span><span class="hljs-params">(fn <span class="hljs-keyword">func</span>()</span>, _<span class="hljs-title">p_</span> *<span class="hljs-title">p</span>)</span> &#123;mp := allocm(_p_, fn)mp.nextp.set(_p_)mp.sigmask = initSigmask...newm1(mp)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newm1</span><span class="hljs-params">(mp *m)</span></span> &#123;...execLock.rlock() <span class="hljs-comment">// Prevent process clone.</span>newosproc(mp)execLock.runlock()&#125;</code></pre></div><p>一直调用到<code>newosproc</code>函数中，会<code>clone</code>一个新线程并在新线程中执行<code>runtime.mstart</code></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// May run with m.p==nil, so write barriers are not allowed.</span><span class="hljs-comment">//go:nowritebarrier</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newosproc</span><span class="hljs-params">(mp *m)</span></span> &#123;stk := unsafe.Pointer(mp.g0.stack.hi)<span class="hljs-comment">// Disable signals during clone, so that the new thread starts</span><span class="hljs-comment">// with signals disabled. It will enable them in minit.</span><span class="hljs-keyword">var</span> oset sigsetsigprocmask(_SIG_SETMASK, &amp;sigset_all, &amp;oset)ret := clone(cloneFlags, stk, unsafe.Pointer(mp), unsafe.Pointer(mp.g0), unsafe.Pointer(funcPC(mstart)))sigprocmask(_SIG_SETMASK, &amp;oset, <span class="hljs-literal">nil</span>)&#125;</code></pre></div><p>在新线程上，会运行传入的<code>sysmon</code>函数:</p><ul><li>最初的休眠时间为20μs,最长的休眠时间是10ms，当sysmon在50个循环都无法唤醒goroutine时，休眠时间在每个循环都加倍</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//go:nowritebarrierrec</span><span class="hljs-comment">//显然不同p绑定则不需要写屏障</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">sysmon</span><span class="hljs-params">()</span></span>&#123;<span class="hljs-comment">//为了将绑定的M不作deadlock检查</span>lock(&amp;sched.lock)sched.nmsys++checkdead()unlock(&amp;sched.lock)<span class="hljs-keyword">for</span>&#123;...&#125;...&#125;</code></pre></div><p><code>sysmon</code>中有一些值得探究的函数:</p><ol start="2"><li>检测死锁<code>checkdead()</code></li></ol><p>大致分为:</p><ul><li>检查是否存在正在运行的线程；</li><li>检查是否存在正在运行的goroutine；</li><li>检查P上是否有计时器；</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Check for deadlock situation.</span><span class="hljs-comment">// The check is based on number of running M's, if 0 -&gt; deadlock.</span><span class="hljs-comment">// sched.lock must be held.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">checkdead</span><span class="hljs-params">()</span></span> &#123;...<span class="hljs-comment">//上面都是一些panic，buildmode问题</span><span class="hljs-comment">//mcount：根据下一个待创建的线程id和释放的线程数得到系统中存在的线程数</span><span class="hljs-comment">//即(sched.mnext - sched.nmfreed)</span><span class="hljs-comment">//sched.nmidle 处于空闲状态的M数量</span><span class="hljs-comment">//sched.nmidlelocked处于锁定状态的M数量</span><span class="hljs-comment">//sched.nmsys处于系统调用的M数量</span>run := mcount() - sched.nmidle - sched.nmidlelocked - sched.nmsys<span class="hljs-keyword">if</span> run &gt; run0 &#123;<span class="hljs-comment">//无死锁</span><span class="hljs-keyword">return</span>&#125;<span class="hljs-keyword">if</span> run &lt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">//当前程序状态不一致</span><span class="hljs-built_in">print</span>(<span class="hljs-string">"runtime: checkdead: nmidle="</span>, sched.nmidle, <span class="hljs-string">" nmidlelocked="</span>, sched.nmidlelocked, <span class="hljs-string">" mcount="</span>, mcount(), <span class="hljs-string">" nmsys="</span>, sched.nmsys, <span class="hljs-string">"\n"</span>)throw(<span class="hljs-string">"checkdead: inconsistent counts"</span>)&#125;<span class="hljs-comment">//run==0</span>grunning := <span class="hljs-number">0</span>lock(&amp;allglock)<span class="hljs-comment">//2.是否有运行的goroutine</span><span class="hljs-keyword">for</span> i := <span class="hljs-number">0</span>; i &lt; <span class="hljs-built_in">len</span>(allgs); i++ &#123;gp := allgs[i]<span class="hljs-keyword">if</span> isSystemGoroutine(gp, <span class="hljs-literal">false</span>) &#123;<span class="hljs-keyword">continue</span>&#125;s := readgstatus(gp)<span class="hljs-keyword">switch</span> s &amp;^ _Gscan &#123;<span class="hljs-keyword">case</span> _Gwaiting,_Gpreempted:grunning++<span class="hljs-keyword">case</span> _Grunnable,_Grunning,_Gsyscall:<span class="hljs-comment">//有死锁</span>unlock(&amp;allglock)<span class="hljs-built_in">print</span>(<span class="hljs-string">"runtime: checkdead: find g "</span>, gp.goid, <span class="hljs-string">" in status "</span>, s, <span class="hljs-string">"\n"</span>)throw(<span class="hljs-string">"checkdead: runnable g"</span>)&#125;&#125;unlock(&amp;allglock)<span class="hljs-comment">//循环下来，所有M都在Gidle，Gdead，Gcopystack状态下</span><span class="hljs-keyword">if</span> grunning == <span class="hljs-number">0</span> &#123; <span class="hljs-comment">// possible if main goroutine calls runtime·Goexit()</span><span class="hljs-comment">//调用了goexit()</span>unlock(&amp;sched.lock) <span class="hljs-comment">// unlock so that GODEBUG=scheddetail=1 doesn't hang</span>throw(<span class="hljs-string">"no goroutines (main called runtime.Goexit) - deadlock!"</span>)&#125;<span class="hljs-comment">// Maybe jump time forward for playground.</span><span class="hljs-keyword">if</span> faketime != <span class="hljs-number">0</span> &#123;when, _p_ := timeSleepUntil()<span class="hljs-keyword">if</span> _p_ != <span class="hljs-literal">nil</span> &#123;faketime = when<span class="hljs-keyword">for</span> pp := &amp;sched.pidle; *pp != <span class="hljs-number">0</span>; pp = &amp;(*pp).ptr().link &#123;<span class="hljs-keyword">if</span> (*pp).ptr() == _p_ &#123;*pp = _p_.link<span class="hljs-keyword">break</span>&#125;&#125;mp := mget()<span class="hljs-keyword">if</span> mp == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// There should always be a free M since</span><span class="hljs-comment">// nothing is running.</span>throw(<span class="hljs-string">"checkdead: no m for timer"</span>)&#125;mp.nextp.set(_p_)notewakeup(&amp;mp.park)<span class="hljs-keyword">return</span>&#125;&#125;<span class="hljs-comment">// There are no goroutines running, so we can look at the P's.</span><span class="hljs-comment">//3. 存在等待的goroutine切不存在running的goroutine，检查P中的计时器</span><span class="hljs-keyword">for</span> _, _p_ := <span class="hljs-keyword">range</span> allp &#123;<span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(_p_.timers) &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">//如果有等待的计时器，则goroutine陷入Gidle是合理的，如果没有，那就永远不会唤醒，就是死锁</span><span class="hljs-keyword">return</span>&#125;&#125;getg().m.throwing = <span class="hljs-number">-1</span> <span class="hljs-comment">// do not dump full stacks</span>unlock(&amp;sched.lock)    <span class="hljs-comment">// unlock so that GODEBUG=scheddetail=1 doesn't hang</span>throw(<span class="hljs-string">"all goroutines are asleep - deadlock!"</span>)&#125;</code></pre></div><ol start="3"><li>一个forloop 会不断进行以下工作:</li></ol><ul><li>计时器，获得下一个需要被触发的timer</li></ul><p>PS: timer中管理了一个<code>最小堆</code>,堆顶timer就是最小时间，<code>checkTimers</code>方法会触发一次（但是其还不够，因为可能M都在忙，所以这里还要sysmon进行处理）</p><ul><li>netpoll，轮询获得需要处理的到期的fd</li><li>抢占(retake函数)运行时间较长的或者syscall的goroutine</li><li>gc，符合条件时强制回收</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">sysmon</span><span class="hljs-params">()</span></span>&#123;...<span class="hljs-keyword">for</span>&#123;<span class="hljs-comment">//一些延迟设定，每隔一定时间再去扫描</span><span class="hljs-keyword">if</span> idle == <span class="hljs-number">0</span> &#123; <span class="hljs-comment">// start with 20us sleep...</span>delay = <span class="hljs-number">20</span><span class="hljs-comment">//50个循环，加倍</span>&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> idle &gt; <span class="hljs-number">50</span> &#123; <span class="hljs-comment">// start doubling the sleep after 1ms...</span>delay *= <span class="hljs-number">2</span>&#125;<span class="hljs-comment">//超出最大，设为最大</span><span class="hljs-keyword">if</span> delay &gt; <span class="hljs-number">10</span>*<span class="hljs-number">1000</span> &#123; <span class="hljs-comment">// up to 10ms</span>delay = <span class="hljs-number">10</span> * <span class="hljs-number">1000</span>&#125;usleep(delay)<span class="hljs-comment">//----------------------1.timer 计时-----------------</span><span class="hljs-comment">//next为下次tick应该发生的时间，只会由sysmon()和checkdead()调用</span>now := nanotime()next, _ := timeSleepUntil()<span class="hljs-keyword">if</span> debug.schedtrace &lt;= <span class="hljs-number">0</span> &amp;&amp; (sched.gcwaiting != <span class="hljs-number">0</span> || atomic.Load(&amp;sched.npidle) == <span class="hljs-keyword">uint32</span>(gomaxprocs)) &#123;lock(&amp;sched.lock)<span class="hljs-keyword">if</span> atomic.Load(&amp;sched.gcwaiting) != <span class="hljs-number">0</span> || atomic.Load(&amp;sched.npidle) == <span class="hljs-keyword">uint32</span>(gomaxprocs) &#123;<span class="hljs-comment">//当前垃圾回收和所有处理器都处于闲置状态(npidle=gomaxprocs)，且没有要触发的计时器，sysmon陷入休眠</span><span class="hljs-keyword">if</span> next &gt; now &#123;atomic.Store(&amp;sched.sysmonwait, <span class="hljs-number">1</span>)unlock(&amp;sched.lock)<span class="hljs-comment">// Make wake-up period small enough</span><span class="hljs-comment">// for the sampling to be correct.</span><span class="hljs-comment">//计算休眠的时间</span>sleep := forcegcperiod / <span class="hljs-number">2</span><span class="hljs-keyword">if</span> next-now &lt; sleep &#123;sleep = next - now&#125;shouldRelax := sleep &gt;= osRelaxMinNS<span class="hljs-keyword">if</span> shouldRelax &#123;osRelax(<span class="hljs-literal">true</span>)&#125;<span class="hljs-comment">//信号量同步系统监控即将进入休眠的状态</span>notetsleep(&amp;sched.sysmonnote, sleep)<span class="hljs-keyword">if</span> shouldRelax &#123;osRelax(<span class="hljs-literal">false</span>)&#125;<span class="hljs-comment">//唤醒</span>now = nanotime()next, _ = timeSleepUntil()lock(&amp;sched.lock)atomic.Store(&amp;sched.sysmonwait, <span class="hljs-number">0</span>)<span class="hljs-comment">//唤醒之后通知 系统监控被唤醒</span>noteclear(&amp;sched.sysmonnote)&#125;<span class="hljs-comment">//唤醒后重置休眠时间</span>idle = <span class="hljs-number">0</span>delay = <span class="hljs-number">20</span><span class="hljs-comment">//如果在这之后，我们发现下一个计时器需要触发的时间小于当前时间，这也就说明所有的线程可能正在忙于运行 Goroutine，系统监控会启动新的线程来触发计时器，避免计时器的到期时间有较大的偏差。???</span>&#125;unlock(&amp;sched.lock)&#125;<span class="hljs-comment">//----------------2. netpolll-------------------</span><span class="hljs-comment">// poll network if not polled for more than 10ms</span><span class="hljs-comment">//先检查网络上的调用，超过10ms没有poll过则会poll一次</span>lastpoll := <span class="hljs-keyword">int64</span>(atomic.Load64(&amp;sched.lastpoll))<span class="hljs-keyword">if</span> netpollinited() &amp;&amp; lastpoll != <span class="hljs-number">0</span> &amp;&amp; lastpoll+<span class="hljs-number">10</span>*<span class="hljs-number">1000</span>*<span class="hljs-number">1000</span> &lt; now &#123;<span class="hljs-comment">//更新sched.lastpoll=now</span>atomic.Cas64(&amp;sched.lastpoll, <span class="hljs-keyword">uint64</span>(lastpoll), <span class="hljs-keyword">uint64</span>(now))<span class="hljs-comment">// netpoll checks for ready network connections.</span><span class="hljs-comment">// Returns list of goroutines that become runnable.</span><span class="hljs-comment">//返回runnable的goroutine list</span>list := netpoll(<span class="hljs-number">0</span>) <span class="hljs-comment">// non-blocking - returns list of goroutines</span><span class="hljs-keyword">if</span> !list.empty() &#123;<span class="hljs-comment">// Need to decrement number of idle locked M's</span><span class="hljs-comment">// (pretending that one more is running) before injectglist.</span><span class="hljs-comment">// Otherwise it can lead to the following situation:</span><span class="hljs-comment">// injectglist grabs all P's but before it starts M's to run the P's,</span><span class="hljs-comment">// another M returns from syscall, finishes running its G,</span><span class="hljs-comment">// observes that there is no work to do and no other running M's</span><span class="hljs-comment">// and reports deadlock.</span><span class="hljs-comment">//需要先减少lockedM的理由：</span><span class="hljs-comment">//injectglist会获取所有P 但是 在sched会启动M去运行一个P的G之前，另外一个M从syscall中返回，完成了这个G，这个时候sched启动的这个M就发现没东西去做，而且没有其他运行的M，会报deadlock todo???</span>incidlelocked(<span class="hljs-number">-1</span>)<span class="hljs-comment">// Injects the list of runnable G's into the scheduler and clears glist.</span><span class="hljs-comment">//实际上会将所有goroutine状态从_Gwaiting切换到_Grunnable并加入到全局队列等待，</span><span class="hljs-comment">//如果有空闲的P，就会通过startm()来启动线程执行这些任务</span>injectglist(&amp;list)incidlelocked(<span class="hljs-number">1</span>)&#125;&#125;<span class="hljs-comment">//todo ???为什么next&lt;now可以说明</span><span class="hljs-comment">//有timers本来应该被运行但未运行，一种可能就是有不可抢占的P，尝试开启一个M来运行</span><span class="hljs-keyword">if</span> next &lt; now &#123;<span class="hljs-comment">// There are timers that should have already run,</span><span class="hljs-comment">// perhaps because there is an unpreemptible P.</span><span class="hljs-comment">// Try to start an M to run them.</span>startm(<span class="hljs-literal">nil</span>, <span class="hljs-literal">false</span>)&#125;<span class="hljs-comment">//---------------------3. 抢占retake()---------------</span><span class="hljs-comment">//重点，重新拿取在syscall的P，并且抢占长久运行的G</span><span class="hljs-comment">// retake P's blocked in syscalls</span><span class="hljs-comment">// and preempt long running G's</span><span class="hljs-keyword">if</span> retake(now) != <span class="hljs-number">0</span> &#123;<span class="hljs-comment">//抢占了，恢复20μs的间隔</span>idle = <span class="hljs-number">0</span>&#125; <span class="hljs-keyword">else</span> &#123;idle++&#125;<span class="hljs-comment">//-----------------------4. gc force--------------</span><span class="hljs-comment">//还有检查是否要强制GC</span><span class="hljs-comment">// check if we need to force a GC</span><span class="hljs-keyword">if</span> t := (gcTrigger&#123;kind: gcTriggerTime, now: now&#125;); t.test() &amp;&amp; atomic.Load(&amp;forcegc.idle) != <span class="hljs-number">0</span> &#123;lock(&amp;forcegc.lock)forcegc.idle = <span class="hljs-number">0</span><span class="hljs-keyword">var</span> list gList<span class="hljs-comment">//会将用于gc的goroutine加入全局队列，让scheduler去选择p处理</span>list.push(forcegc.g)injectglist(&amp;list)unlock(&amp;forcegc.lock)&#125;...<span class="hljs-comment">//trace相关</span>&#125;&#125;</code></pre></div><p>主要讲一下抢占<code>retake()</code>,其sysmontick结构如下：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> sysmontick <span class="hljs-keyword">struct</span> &#123;schedtick   <span class="hljs-keyword">uint32</span> <span class="hljs-comment">//处理器调度次数</span>schedwhen   <span class="hljs-keyword">int64</span><span class="hljs-comment">//处理器上次调度时间</span>syscalltick <span class="hljs-keyword">uint32</span><span class="hljs-comment">//系统调用次数</span>syscallwhen <span class="hljs-keyword">int64</span><span class="hljs-comment">//系统调用时间</span>&#125;</code></pre></div><p><code>retake()</code>函数:</p><ul><li>当P处于 <code>_Prunning</code> 或者 <code>_Psyscall</code> 状态时，如果上一次触发调度的时间已经过去了 10ms，我们就会通过 runtime.preemptone 抢占当前处理器；</li><li>当P处于 <code>_Psyscall</code> 状态时，当<strong>处理器的运行队列不为空或者不存在空闲处理器时</strong> 或者 <strong>当系统调用时间超过了 10ms</strong>，都会调用 <code>runtime.handoffp</code> 让出处理器的使用权：</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">retake</span><span class="hljs-params">(now <span class="hljs-keyword">int64</span>)</span> <span class="hljs-title">uint32</span></span> &#123;n := <span class="hljs-number">0</span><span class="hljs-comment">// Prevent allp slice changes. This lock will be completely</span><span class="hljs-comment">// uncontended unless we're already stopping the world.</span><span class="hljs-comment">//allp       []*p  // len(allp) == gomaxprocs; may change at safe points, otherwise immutable</span><span class="hljs-comment">//防止p的数目变化</span>lock(&amp;allpLock)<span class="hljs-comment">// We can't use a range loop over allp because we may</span><span class="hljs-comment">// temporarily drop the allpLock. Hence, we need to re-fetch</span><span class="hljs-comment">// allp each time around the loop.</span><span class="hljs-keyword">for</span> i := <span class="hljs-number">0</span>; i &lt; <span class="hljs-built_in">len</span>(allp); i++ &#123;_p_ := allp[i]<span class="hljs-keyword">if</span> _p_ == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// This can happen if procresize has grown</span><span class="hljs-comment">// allp but not yet created new Ps.</span><span class="hljs-keyword">continue</span>&#125;pd := &amp;_p_.sysmonticks := _p_.statussysretake := <span class="hljs-literal">false</span><span class="hljs-comment">//1. 处于Prunning或者Psyscall，</span><span class="hljs-keyword">if</span> s == _Prunning || s == _Psyscall &#123;<span class="hljs-comment">//p在running或者syscall太久，进行抢占!</span><span class="hljs-comment">// Preempt G if it's running for too long.</span>t := <span class="hljs-keyword">int64</span>(_p_.schedtick)<span class="hljs-keyword">if</span> <span class="hljs-keyword">int64</span>(pd.schedtick) != t &#123;pd.schedtick = <span class="hljs-keyword">uint32</span>(t)pd.schedwhen = now&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> pd.schedwhen+forcePreemptNS &lt;= now &#123;<span class="hljs-comment">//超过10ms</span><span class="hljs-comment">//抢占</span>preemptone(_p_)<span class="hljs-comment">// In case of syscall, preemptone() doesn't</span><span class="hljs-comment">// work, because there is no M wired to P.</span><span class="hljs-comment">//syscall下preemptone不会作用，因为没有M同P联系住，这里设为true只是让下面的代码少跑一轮</span>sysretake = <span class="hljs-literal">true</span>&#125;&#125;<span class="hljs-comment">//2. Psyscall下</span><span class="hljs-keyword">if</span> s == _Psyscall &#123;<span class="hljs-comment">// Retake P from syscall if it's there for more than 1 sysmon tick (at least 20us).</span>t := <span class="hljs-keyword">int64</span>(_p_.syscalltick)<span class="hljs-keyword">if</span> !sysretake &amp;&amp; <span class="hljs-keyword">int64</span>(pd.syscalltick) != t &#123;pd.syscalltick = <span class="hljs-keyword">uint32</span>(t)pd.syscallwhen = now<span class="hljs-keyword">continue</span>&#125;<span class="hljs-comment">// On the one hand we don't want to retake Ps if there is no other work to do,</span><span class="hljs-comment">// but on the other hand we want to retake them eventually</span><span class="hljs-comment">// because they can prevent the sysmon thread from deep sleep.</span><span class="hljs-keyword">if</span> runqempty(_p_) &amp;&amp; atomic.Load(&amp;sched.nmspinning)+atomic.Load(&amp;sched.npidle) &gt; <span class="hljs-number">0</span> &amp;&amp; pd.syscallwhen+<span class="hljs-number">10</span>*<span class="hljs-number">1000</span>*<span class="hljs-number">1000</span> &gt; now &#123;<span class="hljs-keyword">continue</span>&#125;<span class="hljs-comment">// Drop allpLock so we can take sched.lock.</span>unlock(&amp;allpLock)<span class="hljs-comment">// Need to decrement number of idle locked M's</span><span class="hljs-comment">// (pretending that one more is running) before the CAS.</span><span class="hljs-comment">// Otherwise the M from which we retake can exit the syscall,</span><span class="hljs-comment">// increment nmidle and report deadlock.</span><span class="hljs-comment">//同上面一样，防止 当前retake的M退出syscall，导致idle的M增加(nmide++)，死锁发生</span>incidlelocked(<span class="hljs-number">-1</span>)<span class="hljs-comment">//1.当前P运行队列不为空或者不存在空闲的P时</span><span class="hljs-comment">//2. 系统调用时间超时10ms</span><span class="hljs-keyword">if</span> atomic.Cas(&amp;_p_.status, s, _Pidle) &#123;<span class="hljs-keyword">if</span> trace.enabled &#123;traceGoSysBlock(_p_)traceProcStop(_p_)&#125;n++_p_.syscalltick++<span class="hljs-comment">//执行syscall，handoff当前P，让出P，因为syscall的时候G是直接同M绑定的！！！</span>handoffp(_p_)&#125;incidlelocked(<span class="hljs-number">1</span>)lock(&amp;allpLock)&#125;&#125;unlock(&amp;allpLock)<span class="hljs-keyword">return</span> <span class="hljs-keyword">uint32</span>(n)&#125;</code></pre></div><p><code>retake</code>中的<code>handoffp</code>：</p><ul><li>先检查本地runq有无g，如果有，直接调用<code>startm</code>运行</li><li>检查有无gc <code>work</code>(todo???是一个全局的垃圾回收结构体)，如果有，调用<code>startm</code>开始</li><li></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Hands off P from syscall or locked M.</span><span class="hljs-comment">// Always runs without a P, so write barriers are not allowed.</span><span class="hljs-comment">//go:nowritebarrierrec</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">handoffp</span><span class="hljs-params">(_p_ *p)</span></span> &#123;<span class="hljs-comment">// handoffp must start an M in any situation where</span><span class="hljs-comment">// findrunnable would return a G to run on _p_.</span><span class="hljs-comment">// if it has local work, start it straight away</span><span class="hljs-keyword">if</span> !runqempty(_p_) || sched.runqsize != <span class="hljs-number">0</span> &#123;startm(_p_, <span class="hljs-literal">false</span>)<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">// if it has GC work, start it straight away</span><span class="hljs-keyword">if</span> gcBlackenEnabled != <span class="hljs-number">0</span> &amp;&amp; gcMarkWorkAvailable(_p_) &#123;startm(_p_, <span class="hljs-literal">false</span>)<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">// no local work, check that there are no spinning/idle M's,</span><span class="hljs-comment">// otherwise our help is not required</span><span class="hljs-keyword">if</span> atomic.Load(&amp;sched.nmspinning)+atomic.Load(&amp;sched.npidle) == <span class="hljs-number">0</span> &amp;&amp; atomic.Cas(&amp;sched.nmspinning, <span class="hljs-number">0</span>, <span class="hljs-number">1</span>) &#123; <span class="hljs-comment">// <span class="hljs-doctag">TODO:</span> fast atomic</span>startm(_p_, <span class="hljs-literal">true</span>)<span class="hljs-keyword">return</span>&#125;lock(&amp;sched.lock)<span class="hljs-keyword">if</span> sched.gcwaiting != <span class="hljs-number">0</span> &#123;_p_.status = _Pgcstopsched.stopwait--<span class="hljs-keyword">if</span> sched.stopwait == <span class="hljs-number">0</span> &#123;notewakeup(&amp;sched.stopnote)&#125;unlock(&amp;sched.lock)<span class="hljs-keyword">return</span>&#125;<span class="hljs-keyword">if</span> _p_.runSafePointFn != <span class="hljs-number">0</span> &amp;&amp; atomic.Cas(&amp;_p_.runSafePointFn, <span class="hljs-number">1</span>, <span class="hljs-number">0</span>) &#123;sched.safePointFn(_p_)sched.safePointWait--<span class="hljs-keyword">if</span> sched.safePointWait == <span class="hljs-number">0</span> &#123;notewakeup(&amp;sched.safePointNote)&#125;&#125;<span class="hljs-keyword">if</span> sched.runqsize != <span class="hljs-number">0</span> &#123;unlock(&amp;sched.lock)startm(_p_, <span class="hljs-literal">false</span>)<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">// If this is the last running P and nobody is polling network,</span><span class="hljs-comment">// need to wakeup another M to poll network.</span><span class="hljs-keyword">if</span> sched.npidle == <span class="hljs-keyword">uint32</span>(gomaxprocs<span class="hljs-number">-1</span>) &amp;&amp; atomic.Load64(&amp;sched.lastpoll) != <span class="hljs-number">0</span> &#123;unlock(&amp;sched.lock)startm(_p_, <span class="hljs-literal">false</span>)<span class="hljs-keyword">return</span>&#125;<span class="hljs-keyword">if</span> when := nobarrierWakeTime(_p_); when != <span class="hljs-number">0</span> &#123;wakeNetPoller(when)&#125;pidleput(_p_)unlock(&amp;sched.lock)&#125;</code></pre></div><ol start="4"><li>最后还要判断一下是否要gc</li></ol><ul><li>将传入的<code>forcegc.g</code>的list，每个g状态改为<code>Grunnable</code>然后放入全局<code>sched.list</code>;</li><li>接着还会检查一次<code>sched.npidle</code>看下有无空闲的p，调用<code>startm(nil, false)</code>立即开始</li><li>最后清空传入的<code>forcegc.g</code> 的list</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">sysmon</span><span class="hljs-params">()</span></span> &#123;...<span class="hljs-keyword">for</span>&#123;....<span class="hljs-comment">// 4. check if we need to force a GC</span><span class="hljs-keyword">if</span> t := (gcTrigger&#123;kind: gcTriggerTime, now: now&#125;); t.test() &amp;&amp; atomic.Load(&amp;forcegc.idle) != <span class="hljs-number">0</span> &#123;lock(&amp;forcegc.lock)forcegc.idle = <span class="hljs-number">0</span><span class="hljs-keyword">var</span> list gListlist.push(forcegc.g)injectglist(&amp;list)unlock(&amp;forcegc.lock)&#125;..<span class="hljs-comment">//trace</span>&#125;&#125;</code></pre></div><h3 id="gpm各个结构体">GPM各个结构体</h3><h4 id="g-goroutine">g (goroutine)</h4><p>要注意几个fields:</p><ol><li>与栈相关的, 另一篇文章提到过栈(<code>stackguard0</code>用作调度器抢占式调度);</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> g <span class="hljs-keyword">struct</span>&#123;stack       stack   <span class="hljs-comment">// offset known to runtime/cgo</span><span class="hljs-comment">//栈空间[lo,hi)</span><span class="hljs-comment">//type stack struct &#123;</span><span class="hljs-comment">//lo uintptr</span><span class="hljs-comment">//hi uintptr</span><span class="hljs-comment">//&#125;</span>stackguard0 <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// offset known to liblink</span>stackguard1 <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// offset known to liblink</span>&#125;</code></pre></div><ol start="2"><li>与抢占相关在之前文章也有提到过，<code>g.preemptStop</code>在抢占时会变成<code>_Gpreempted</code><code>g.preemptShrink</code>标记是否当前在shrink中</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> g <span class="hljs-keyword">struct</span>&#123;preempt       <span class="hljs-keyword">bool</span> <span class="hljs-comment">// preemption signal, duplicates stackguard0 = stackpreempt</span>preemptStop   <span class="hljs-keyword">bool</span> <span class="hljs-comment">// transition to _Gpreempted on preemption; otherwise, just deschedule</span>preemptShrink <span class="hljs-keyword">bool</span> <span class="hljs-comment">// shrink stack at synchronous safe point</span>&#125;</code></pre></div><ol start="3"><li>调度字段</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> g <span class="hljs-keyword">struct</span>&#123;m            *m      <span class="hljs-comment">// current m; offset known to arm liblink</span>sched        gobufatomicstatus <span class="hljs-keyword">uint32</span> <span class="hljs-comment">//上面提到的几种状态</span>&#125;</code></pre></div><p>其中<code>g.sched</code>字段就是调度时候保存的各种指针等信息，用来恢复上下文的时候使用:</p><p>依次是</p><ul><li><p><code>sp</code>,<code>pc</code>分别为栈指针，程序计数器</p></li><li><p><code>g</code>当前该gobuf的goroutine</p></li><li><p><code>ctxt</code>在之前的文章讲过，复制栈的时候要把这部分指针复制,用作</p></li><li><p><code>ret</code>系统用的return</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> gobuf <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// The offsets of sp, pc, and g are known to (hard-coded in) libmach.</span><span class="hljs-comment">//</span><span class="hljs-comment">// ctxt is unusual with respect to GC: it may be a</span><span class="hljs-comment">// heap-allocated funcval, so GC needs to track it, but it</span><span class="hljs-comment">// needs to be set and cleared from assembly, where it's</span><span class="hljs-comment">// difficult to have write barriers. However, ctxt is really a</span><span class="hljs-comment">// saved, live register, and we only ever exchange it between</span><span class="hljs-comment">// the real register and the gobuf. Hence, we treat it as a</span><span class="hljs-comment">// root during stack scanning, which means assembly that saves</span><span class="hljs-comment">// and restores it doesn't need write barriers. It's still</span><span class="hljs-comment">// typed as a pointer so that any other writes from Go get</span><span class="hljs-comment">// write barriers.</span>sp   <span class="hljs-keyword">uintptr</span>pc   <span class="hljs-keyword">uintptr</span>g    guintptrctxt unsafe.Pointerret  sys.Uintreglr   <span class="hljs-keyword">uintptr</span>bp   <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// for GOEXPERIMENT=framepointer</span>&#125;</code></pre></div><p>较全的结构</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> g <span class="hljs-keyword">struct</span> &#123;goid           <span class="hljs-keyword">int64</span><span class="hljs-comment">// Stack parameters.</span><span class="hljs-comment">// stack describes the actual stack memory: [stack.lo, stack.hi).</span><span class="hljs-comment">// stackguard0 is the stack pointer compared in the Go stack growth prologue.</span><span class="hljs-comment">//stackguard0用作栈的指针</span><span class="hljs-comment">// It is stack.lo+StackGuard normally, but can be StackPreempt to trigger a preemption.</span><span class="hljs-comment">// stackguard1 is the stack pointer compared in the C stack growth prologue.</span><span class="hljs-comment">// It is stack.lo+StackGuard on g0 and gsignal stacks.</span><span class="hljs-comment">// It is ~0 on other goroutine stacks, to trigger a call to morestackc (and crash).</span>stack       stack   <span class="hljs-comment">// offset known to runtime/cgo</span><span class="hljs-comment">//栈空间[lo,hi)</span><span class="hljs-comment">//type stack struct &#123;</span><span class="hljs-comment">//lo uintptr</span><span class="hljs-comment">//hi uintptr</span><span class="hljs-comment">//&#125;</span>stackguard0 <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// offset known to liblink</span>stackguard1 <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// offset known to liblink</span>    ...m              *m      <span class="hljs-comment">// current m; offset known to arm liblink</span><span class="hljs-comment">//调度器,上下文保存的信息所在地</span>sched          gobuf    ...param          unsafe.Pointer <span class="hljs-comment">// passed parameter on wakeup</span>...schedlink      guintptrwaitsince      <span class="hljs-keyword">int64</span>      <span class="hljs-comment">// approx time when the g become blocked</span>waitreason     waitReason <span class="hljs-comment">// if status==Gwaiting</span>...preemptscan    <span class="hljs-keyword">bool</span>       <span class="hljs-comment">// preempted g does scan for gc</span>gcscandone     <span class="hljs-keyword">bool</span>       <span class="hljs-comment">// g has scanned stack; protected by _Gscan bit in status</span>gcscanvalid    <span class="hljs-keyword">bool</span>       <span class="hljs-comment">// false at start of gc cycle, true if G has not run since last scan; <span class="hljs-doctag">TODO:</span> remove?</span>...raceignore     <span class="hljs-keyword">int8</span>       <span class="hljs-comment">// ignore race detection events</span>sysblocktraced <span class="hljs-keyword">bool</span>       <span class="hljs-comment">// StartTrace has emitted EvGoInSyscall about this goroutine</span>sysexitticks   <span class="hljs-keyword">int64</span>      <span class="hljs-comment">// cputicks when syscall has returned (for tracing)</span>traceseq       <span class="hljs-keyword">uint64</span>     <span class="hljs-comment">// trace event sequencer</span>tracelastp     puintptr   <span class="hljs-comment">// last P emitted an event for this goroutine</span>    lockedm        muintptr        <span class="hljs-comment">//本身的寄存器状态</span>sig            <span class="hljs-keyword">uint32</span>writebuf       []<span class="hljs-keyword">byte</span>sigcode0       <span class="hljs-keyword">uintptr</span>sigcode1       <span class="hljs-keyword">uintptr</span>sigpc          <span class="hljs-keyword">uintptr</span>    gopc           <span class="hljs-keyword">uintptr</span>         <span class="hljs-comment">// pc of go statement that created this goroutine</span>    ....startpc        <span class="hljs-keyword">uintptr</span>         <span class="hljs-comment">// pc of goroutine function</span>racectx        <span class="hljs-keyword">uintptr</span>waiting        *sudog         <span class="hljs-comment">// sudog structures this g is waiting on (that have a valid elem ptr); in lock order</span>.....<span class="hljs-comment">// Per-G GC state</span><span class="hljs-comment">// gcAssistBytes is this G's GC assist credit in terms of</span><span class="hljs-comment">// bytes allocated. If this is positive, then the G has credit</span><span class="hljs-comment">// to allocate gcAssistBytes bytes without assisting. If this</span><span class="hljs-comment">// is negative, then the G must correct this by performing</span><span class="hljs-comment">// scan work. We track this in bytes to make it fast to update</span><span class="hljs-comment">// and check for debt in the malloc hot path. The assist ratio</span><span class="hljs-comment">// determines how this corresponds to scan work debt.</span>gcAssistBytes <span class="hljs-keyword">int64</span>&#125;</code></pre></div><h4 id="m-machine">M（machine）</h4><p>指的就是OS原生线程,是真正调度资源的单位，M是idle或者syscall中，需要P的调度</p><ol><li>其中<code>m.curg</code>为当前线程上运行的用户goroutine(注意，<code>getg()</code>拿到的是m的当前所有类型的goroutine)</li></ol><p><code>m.g0</code>为持有调度栈的goroutine</p><ol start="2"><li>还有几个处理器相关的字段:<code>m.p</code>: 正在运行的处理器<code>m.nextp</code>:暂存的处理器<code>m.oldp</code>:执行系统调用之前使用的线程处理器</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> m <span class="hljs-keyword">struct</span> &#123;id            <span class="hljs-keyword">int64</span>    <span class="hljs-comment">//g0是一个调用栈</span>g0      *g     <span class="hljs-comment">// goroutine with scheduling stack</span>morebuf gobuf  <span class="hljs-comment">// gobuf arg to morestack</span>procid        <span class="hljs-keyword">uint64</span>       <span class="hljs-comment">// for debuggers, but offset not hard-coded</span><span class="hljs-comment">//底层的线程id</span>    ...    <span class="hljs-comment">//这个信号处理的goroutines</span>gsignal       *g           <span class="hljs-comment">// signal-handling g</span>goSigStack    gsignalStack <span class="hljs-comment">// Go-allocated signal handling stack</span>    sigmask       sigset       <span class="hljs-comment">// storage for saved signal mask</span><span class="hljs-comment">//TLS启动时候要使用</span><span class="hljs-comment">//传给FS寄存器的局部变量</span>tls           [<span class="hljs-number">6</span>]<span class="hljs-keyword">uintptr</span>   <span class="hljs-comment">// thread-local storage (for x86 extern register)</span><span class="hljs-comment">//m启动时的函数，会传给clone</span>mstartfn      <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span>&#123;&#125;  <span class="hljs-comment">//当前运行的goroutine,&#123;&#125;在语法中是错误的，这里为了使markdown解析而加上</span><span class="hljs-comment">//当前运行代码的g</span>    curg          *g       <span class="hljs-comment">// current running goroutine</span>    caughtsig     guintptr <span class="hljs-comment">// goroutine running during fatal signal</span><span class="hljs-comment">//处理器相关,与P绑定</span>    p             puintptr <span class="hljs-comment">// attached p for executing go code (nil if not executing go code)</span>nextp         puintptr    oldp          puintptr <span class="hljs-comment">// the p that was attached before executing a syscall</span>mallocing     <span class="hljs-keyword">int32</span>    throwing      <span class="hljs-keyword">int32</span>    <span class="hljs-comment">//如果不等于"",没有发生抢占</span>    preemptoff    <span class="hljs-keyword">string</span> <span class="hljs-comment">// if != "", keep curg running on this m</span>    locks         <span class="hljs-keyword">int32</span>....        <span class="hljs-comment">//m正在自旋，寻找可以attach的工作对象(P), m找不到可运行的g</span>spinning      <span class="hljs-keyword">bool</span> <span class="hljs-comment">// m is out of work and is actively looking for work</span>blocked       <span class="hljs-keyword">bool</span> <span class="hljs-comment">// m is blocked on a note</span>    .....    <span class="hljs-comment">//如果=0，则可以清空g0以及清除该m，是原子性的操作</span>freeWait      <span class="hljs-keyword">uint32</span> <span class="hljs-comment">// if == 0, safe to free g0 and delete m (atomic)</span>fastrand      [<span class="hljs-number">2</span>]<span class="hljs-keyword">uint32</span>needextram    <span class="hljs-keyword">bool</span>    traceback     <span class="hljs-keyword">uint8</span>...<span class="hljs-comment">//里面一些cgo的代码</span>park          note    alllink       *m <span class="hljs-comment">// on allm</span>    <span class="hljs-comment">//调度链，是一个m的指针</span>    schedlink     muintptr<span class="hljs-comment">//每一个P(Per-thread)的用于存储小对象的cache,没有锁，因为都在一个P内，运行代码时绑定的p中的mcache</span>    mcache        *mcache    <span class="hljs-comment">//goroutine的指针,uintptr可以避过写屏障, 主要用于Gobuf goroutine状态或者是那些不经过P的调度列表</span><span class="hljs-comment">//是否与某个g一直绑定</span>    lockedg       guintptr    <span class="hljs-comment">//创建当前thread的栈</span>createstack   [<span class="hljs-number">32</span>]<span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// stack that created this thread.</span>        ...<span class="hljs-comment">//track用</span>    <span class="hljs-comment">//下一个等待锁的M</span>nextwaitm     muintptr    <span class="hljs-comment">// next m waiting for lock</span>    <span class="hljs-comment">//一些锁的操作</span>    waitunlockf   <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(*g, unsafe.Pointer)</span> <span class="hljs-title">bool</span><span class="hljs-title">waitlock</span>      <span class="hljs-title">unsafe</span>.<span class="hljs-title">Pointer</span><span class="hljs-title">waittraceev</span>   <span class="hljs-title">byte</span><span class="hljs-title">waittraceskip</span> <span class="hljs-title">int</span><span class="hljs-title">startingtrace</span> <span class="hljs-title">bool</span><span class="hljs-title">syscalltick</span>   <span class="hljs-title">uint32</span><span class="hljs-title">thread</span>        <span class="hljs-title">uintptr</span> // <span class="hljs-title">thread</span> <span class="hljs-title">handle</span><span class="hljs-title">freelink</span>      *<span class="hljs-title">m</span>      // <span class="hljs-title">on</span> <span class="hljs-title">sched</span>.<span class="hljs-title">freem</span>    ...    //<span class="hljs-title">debug</span><span class="hljs-title">dlogPerM</span>    //表明操作系统相关<span class="hljs-title">mOS</span>&#125;</span></code></pre></div><h4 id="p-process">P（Process）</h4><p>指的是go语言中的调度器，M就是用P才能调度G;</p><p>可以看到P是内嵌于 M 和 G 之间的，其提供线程需要的上下文，都会负责调度线程上的waitq，使每个M可以执行多个G，并在IO时候切换G，提高效率;</p><p><code>p.status</code>字段也有几种状态:</p><ol><li><code>_Pidle</code>:处理器没有运行用户代码或者调度器，被空闲队列或者改变其状态的结构持有，运行队列为空</li><li><code>_Prunning</code>:被线程 M 持有，并且正在执行用户代码或者调度器</li><li><code>_Psyscall</code>:没有执行用户代码，当前线程陷入系统调用</li><li><code>_Pgcstop</code>:被线程 M 持有，当前处理器由于垃圾回收被停止</li><li><code>_Pdead</code>:当前处理器已经不被使用</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> p <span class="hljs-keyword">struct</span> &#123;    <span class="hljs-comment">//每一个p都有自己的id</span>    id          <span class="hljs-keyword">int32</span>    <span class="hljs-comment">//状态，有_Pidle ,_Prunning,_Psyscall, _Pgcstop, _Pdead</span>    status      <span class="hljs-keyword">uint32</span> <span class="hljs-comment">// one of pidle/prunning/...</span>        link        puintptr        <span class="hljs-comment">//每次调度都会自增</span>schedtick   <span class="hljs-keyword">uint32</span>     <span class="hljs-comment">// incremented on every scheduler call</span>    syscalltick <span class="hljs-keyword">uint32</span>     <span class="hljs-comment">// incremented on every system call</span>    <span class="hljs-comment">//go程序启动时候的sysmon用</span>    sysmontick  sysmontick <span class="hljs-comment">// last tick observed by sysmon</span>    <span class="hljs-comment">//指的是后面指针连接的一个m,同时该m也有一个指针连向自己 ???</span>    m           muintptr   <span class="hljs-comment">// back-link to associated m (nil if idle)</span>    <span class="hljs-comment">//</span>mcache      *mcacheraceprocctx <span class="hljs-keyword">uintptr</span>    <span class="hljs-comment">//defer的池,defer函数，结构在此</span>deferpool    [<span class="hljs-number">5</span>][]*_defer <span class="hljs-comment">// pool of available defer structs of different sizes (see panic.go)</span>deferpoolbuf [<span class="hljs-number">5</span>][<span class="hljs-number">32</span>]*_defer    <span class="hljs-comment">//goroutine的id生成，能平均分到每一个idgen中</span><span class="hljs-comment">// Cache of goroutine ids, amortizes accesses to runtime·sched.goidgen.</span>goidcache    <span class="hljs-keyword">uint64</span>goidcacheend <span class="hljs-keyword">uint64</span>    <span class="hljs-comment">//这个就是连接的可运行的goroutines队列,可以不加锁访问（都在一个P里面，没必要加锁）</span><span class="hljs-comment">// Queue of runnable goroutines. Accessed without lock.</span>runqhead <span class="hljs-keyword">uint32</span>runqtail <span class="hljs-keyword">uint32</span>runq     [<span class="hljs-number">256</span>]guintptr<span class="hljs-comment">// runnext, if non-nil, is a runnable G that was ready'd by</span><span class="hljs-comment">// the current G and should be run next instead of what's in</span><span class="hljs-comment">// runq if there's time remaining in the running G's time</span><span class="hljs-comment">// slice. It will inherit the time left in the current time</span><span class="hljs-comment">// slice. If a set of goroutines is locked in a</span><span class="hljs-comment">// communicate-and-wait pattern, this schedules that set as a</span><span class="hljs-comment">// unit and eliminates the (potentially large) scheduling</span><span class="hljs-comment">// latency that otherwise arises from adding the ready'd</span><span class="hljs-comment">// goroutines to the end of the run queue.</span><span class="hljs-comment">//如果runnext非空，则是一个runnable状态的g，如果在当前时间片中还有剩余，则runnext指向的就是下一个应该运行的g而不使用runq里面的g，其会继承剩下的时间；</span>runnext guintptr<span class="hljs-comment">// Available G's (status == Gdead)</span>gFree <span class="hljs-keyword">struct</span> &#123;gListn <span class="hljs-keyword">int32</span>&#125;    <span class="hljs-comment">//sudog相关</span>sudogcache []*sudogsudogbuf   [<span class="hljs-number">128</span>]*sudog...<span class="hljs-comment">//trace的一些东西</span>palloc persistentAlloc <span class="hljs-comment">// per-P to avoid mutex</span>    <span class="hljs-comment">//用作优化内存对齐</span>_ <span class="hljs-keyword">uint32</span> <span class="hljs-comment">// Alignment for atomic fields below</span><span class="hljs-comment">// Per-P GC state</span>gcAssistTime         <span class="hljs-keyword">int64</span>    <span class="hljs-comment">// Nanoseconds in assistAlloc</span>gcFractionalMarkTime <span class="hljs-keyword">int64</span>    <span class="hljs-comment">// Nanoseconds in fractional mark worker (atomic)</span>gcBgMarkWorker       guintptr <span class="hljs-comment">// (atomic)</span>gcMarkWorkerMode     gcMarkWorkerMode<span class="hljs-comment">// gcMarkWorkerStartTime is the nanotime() at which this mark</span><span class="hljs-comment">// worker started.</span>gcMarkWorkerStartTime <span class="hljs-keyword">int64</span><span class="hljs-comment">// gcw is this P's GC work buffer cache. The work buffer is</span><span class="hljs-comment">// filled by write barriers, drained by mutator assists, and</span><span class="hljs-comment">// disposed on certain GC state transitions.</span>gcw gcWork<span class="hljs-comment">// wbBuf is this P's GC write barrier buffer.</span><span class="hljs-comment">//</span><span class="hljs-comment">// <span class="hljs-doctag">TODO:</span> Consider caching this in the running G.</span>wbBuf wbBufrunSafePointFn <span class="hljs-keyword">uint32</span> <span class="hljs-comment">// if 1, run sched.safePointFn at next safe point</span>pad cpu.CacheLinePad&#125;</code></pre></div><p>ps：其实可以注意下<code>p.runq</code>是一个256位(？？？这里为什么是256呢，有一个讲法就是一般一个cache line是128，在多核情况下，256的设置会避免拿cache的失败)的循环列表</p><h3 id="scheddt-调度器">SchedDt 调度器</h3><p>主要在<code>runtime.schedinit()</code>上:</p><p>涉及了GPM几个部分的初始化，其中其实还有其他cpu信息等初始化:<img src="/img/schedInit.png" srcset="/img/loading.gif" alt="如图"></p><h4 id="初始化m">初始化m</h4><p>通过<code>mcommoninit()</code>方法初始化<code>allm</code></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">mcommoninit</span><span class="hljs-params">(mp *m)</span></span> &#123;_g_ := getg()<span class="hljs-comment">// g0 stack won't make sense for user (and is not necessary unwindable).</span><span class="hljs-keyword">if</span> _g_ != _g_.m.g0 &#123;callers(<span class="hljs-number">1</span>, mp.createstack[:])&#125;lock(&amp;sched.lock)<span class="hljs-keyword">if</span> sched.mnext+<span class="hljs-number">1</span> &lt; sched.mnext &#123;throw(<span class="hljs-string">"runtime: thread ID overflow"</span>)&#125;mp.id = sched.mnextsched.mnext++checkmcount()mp.fastrand[<span class="hljs-number">0</span>] = <span class="hljs-keyword">uint32</span>(int64Hash(<span class="hljs-keyword">uint64</span>(mp.id), fastrandseed))mp.fastrand[<span class="hljs-number">1</span>] = <span class="hljs-keyword">uint32</span>(int64Hash(<span class="hljs-keyword">uint64</span>(cputicks()), ^fastrandseed))<span class="hljs-keyword">if</span> mp.fastrand[<span class="hljs-number">0</span>]|mp.fastrand[<span class="hljs-number">1</span>] == <span class="hljs-number">0</span> &#123;mp.fastrand[<span class="hljs-number">1</span>] = <span class="hljs-number">1</span>&#125;mpreinit(mp)<span class="hljs-keyword">if</span> mp.gsignal != <span class="hljs-literal">nil</span> &#123;mp.gsignal.stackguard1 = mp.gsignal.stack.lo + _StackGuard&#125;<span class="hljs-comment">// Add to allm so garbage collector doesn't free g-&gt;m</span><span class="hljs-comment">// when it is just in a register or thread-local storage.</span>mp.alllink = allm<span class="hljs-comment">// NumCgoCall() iterates over allm w/o schedlock,</span><span class="hljs-comment">// so we need to publish it safely.</span>atomicstorep(unsafe.Pointer(&amp;allm), unsafe.Pointer(mp))unlock(&amp;sched.lock)<span class="hljs-comment">// Allocate memory to hold a cgo traceback if the cgo call crashes.</span><span class="hljs-keyword">if</span> iscgo || GOOS == <span class="hljs-string">"solaris"</span> || GOOS == <span class="hljs-string">"illumos"</span> || GOOS == <span class="hljs-string">"windows"</span> &#123;mp.cgoCallers = <span class="hljs-built_in">new</span>(cgoCallers)&#125;&#125;</code></pre></div><h4 id="初始化p-procresize">初始化p(procresize)</h4><ul><li>设置了maxcount，可以有10000个线程，但是同时运行的线程仍然受<code>GOMAXPROCS</code>设置影响</li><li>获取最大运行procs后会调用<code>procresize</code>来更新程序中处理器数量，调度器进入锁定状态，不会执行任何goroutine</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// The bootstrap sequence is:</span><span class="hljs-comment">//</span><span class="hljs-comment">//call osinit</span><span class="hljs-comment">//call schedinit</span><span class="hljs-comment">//make &amp; queue new G</span><span class="hljs-comment">//call runtime·mstart</span><span class="hljs-comment">//</span><span class="hljs-comment">// The new G calls runtime·main.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">schedinit</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-comment">// raceinit must be the first call to race detector.</span><span class="hljs-comment">// In particular, it must be done before mallocinit below calls racemapshadow.</span>_g_ := getg()...sched.maxmcount = <span class="hljs-number">10000</span>....sched.lastpoll = <span class="hljs-keyword">uint64</span>(nanotime())procs := ncpu<span class="hljs-keyword">if</span> n, ok := atoi32(gogetenv(<span class="hljs-string">"GOMAXPROCS"</span>)); ok &amp;&amp; n &gt; <span class="hljs-number">0</span> &#123;procs = n&#125;<span class="hljs-keyword">if</span> procresize(procs) != <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"unknown runnable goroutine during bootstrap"</span>)&#125;&#125;</code></pre></div><p><img src="/img/pstatus.png" srcset="/img/loading.gif" alt="大概流程"></p><p>P在GOMAXPROCS中，所有的P被组织成一个数组，当GOMAXPROCS改变时会触发 stop the world来重新调整P 数组的长度一些变量会从sched中分离出到P中;<code>procresize</code>大概流程:</p><ol><li>记录一下调度时间<code>sched.procresizetime</code>；</li><li>如果全局<code>allp</code> slice 小于期望值，会对其扩容；</li><li>new一个新的处理器结构体，并调用<code>runtime.p.init</code>方法初始化<code>allp</code>里面的p(会将其状态设为<code>_Pgcstop</code>,这个函数可用于创建新的p或复用之前销毁的p)；</li><li>如果当前p还可以使用，将p设为<code>_Prunning</code>;</li><li>否则将当前<code>m</code>和<code>allp[0]</code>绑定</li><li><code>destroy</code>方法释放不使用的旧P；</li><li>Trim一下<code>allp</code>使其跟传入的<code>nproc</code>长度相等</li><li><code>allp</code>slice 中除去当前p之外，将其中的任务通过<code>pidleput()</code>方法将无任务的p放入全局<code>sched.pidle</code>队列</li><li>除去当前p之外，将有任务的p放入p的<code>link</code>结构，连成一个链表</li></ol><p>因为运行初始化p的时候，是刚刚初始化M结束，因此第 6 步中的绑定 M 会将当前的 P 绑定到初始 M 上;而后由于程序刚刚开始，P 队列是空的，所以他们都会被链接到可运行的 P <code>link</code>上处于 <code>_Pidle</code> 状态</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Change number of processors. The world is stopped, sched is locked.</span><span class="hljs-comment">// gcworkbufs are not being modified by either the GC or</span><span class="hljs-comment">// the write barrier code.</span><span class="hljs-comment">// Returns list of Ps with local work, they need to be scheduled by the caller.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">procresize</span><span class="hljs-params">(nprocs <span class="hljs-keyword">int32</span>)</span> *<span class="hljs-title">p</span></span> &#123;old := gomaxprocs<span class="hljs-keyword">if</span> old &lt; <span class="hljs-number">0</span> || nprocs &lt;= <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"procresize: invalid arg"</span>)&#125;<span class="hljs-keyword">if</span> trace.enabled &#123;traceGomaxprocs(nprocs)&#125;<span class="hljs-comment">// update statistics</span>now := nanotime()<span class="hljs-keyword">if</span> sched.procresizetime != <span class="hljs-number">0</span> &#123;sched.totaltime += <span class="hljs-keyword">int64</span>(old) * (now - sched.procresizetime)&#125;sched.procresizetime = now<span class="hljs-comment">// Grow allp if necessary.</span><span class="hljs-comment">//1. 扩容allp</span><span class="hljs-keyword">if</span> nprocs &gt; <span class="hljs-keyword">int32</span>(<span class="hljs-built_in">len</span>(allp)) &#123;<span class="hljs-comment">// Synchronize with retake, which could be running</span><span class="hljs-comment">// concurrently since it doesn't run on a P.</span>lock(&amp;allpLock)<span class="hljs-keyword">if</span> nprocs &lt;= <span class="hljs-keyword">int32</span>(<span class="hljs-built_in">cap</span>(allp)) &#123;allp = allp[:nprocs]&#125; <span class="hljs-keyword">else</span> &#123;nallp := <span class="hljs-built_in">make</span>([]*p, nprocs)<span class="hljs-comment">// Copy everything up to allp's cap so we</span><span class="hljs-comment">// never lose old allocated Ps.</span><span class="hljs-built_in">copy</span>(nallp, allp[:<span class="hljs-built_in">cap</span>(allp)])allp = nallp&#125;unlock(&amp;allpLock)&#125;<span class="hljs-comment">// initialize new P's</span><span class="hljs-comment">//2. 初始化新P</span><span class="hljs-keyword">for</span> i := old; i &lt; nprocs; i++ &#123;pp := allp[i]<span class="hljs-keyword">if</span> pp == <span class="hljs-literal">nil</span> &#123;pp = <span class="hljs-built_in">new</span>(p)&#125;pp.init(i)atomicstorep(unsafe.Pointer(&amp;allp[i]), unsafe.Pointer(pp))&#125;_g_ := getg()<span class="hljs-comment">//当前p存在，且不是在缩小CPU后(id在nprocs范围内)</span><span class="hljs-keyword">if</span> _g_.m.p != <span class="hljs-number">0</span> &amp;&amp; _g_.m.p.ptr().id &lt; nprocs &#123;<span class="hljs-comment">// continue to use the current P</span>_g_.m.p.ptr().status = _Prunning_g_.m.p.ptr().mcache.prepareForSweep()&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//3. 将当前m和`allp[0]`绑定，并设置p为_Pidle状态</span><span class="hljs-comment">// release the current P and acquire allp[0].</span><span class="hljs-comment">//</span><span class="hljs-comment">// We must do this before destroying our current P</span><span class="hljs-comment">// because p.destroy itself has write barriers, so we</span><span class="hljs-comment">// need to do that from a valid P.</span><span class="hljs-keyword">if</span> _g_.m.p != <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> trace.enabled &#123;<span class="hljs-comment">// Pretend that we were descheduled</span><span class="hljs-comment">// and then scheduled again to keep</span><span class="hljs-comment">// the trace sane.</span>traceGoSched()traceProcStop(_g_.m.p.ptr())&#125;_g_.m.p.ptr().m = <span class="hljs-number">0</span>&#125;_g_.m.p = <span class="hljs-number">0</span>_g_.m.mcache = <span class="hljs-literal">nil</span>p := allp[<span class="hljs-number">0</span>]p.m = <span class="hljs-number">0</span>p.status = _Pidleacquirep(p)<span class="hljs-keyword">if</span> trace.enabled &#123;traceGoStart()&#125;&#125;<span class="hljs-comment">//4.</span><span class="hljs-comment">// release resources from unused P's</span><span class="hljs-keyword">for</span> i := nprocs; i &lt; old; i++ &#123;p := allp[i]p.destroy()<span class="hljs-comment">// can't free P itself because it can be referenced by an M in syscall</span>&#125;<span class="hljs-comment">// Trim allp.</span><span class="hljs-keyword">if</span> <span class="hljs-keyword">int32</span>(<span class="hljs-built_in">len</span>(allp)) != nprocs &#123;lock(&amp;allpLock)allp = allp[:nprocs]unlock(&amp;allpLock)&#125;<span class="hljs-keyword">var</span> runnablePs *p<span class="hljs-keyword">for</span> i := nprocs - <span class="hljs-number">1</span>; i &gt;= <span class="hljs-number">0</span>; i-- &#123;p := allp[i]<span class="hljs-keyword">if</span> _g_.m.p.ptr() == p &#123;<span class="hljs-keyword">continue</span>&#125;p.status = _Pidle<span class="hljs-comment">//无任务的p，放入全局队列sched.pidle</span><span class="hljs-keyword">if</span> runqempty(p) &#123;pidleput(p)&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//有任务的p，放入链表</span>p.m.set(mget())p.link.set(runnablePs)runnablePs = p&#125;&#125;stealOrder.reset(<span class="hljs-keyword">uint32</span>(nprocs))<span class="hljs-keyword">var</span> int32p *<span class="hljs-keyword">int32</span> = &amp;gomaxprocs <span class="hljs-comment">// make compiler check that gomaxprocs is an int32</span>atomic.Store((*<span class="hljs-keyword">uint32</span>)(unsafe.Pointer(int32p)), <span class="hljs-keyword">uint32</span>(nprocs))<span class="hljs-keyword">return</span> runnablePs&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> schedt <span class="hljs-keyword">struct</span>&#123;lock mutex<span class="hljs-comment">//锁用于调用globalq的时候使用</span><span class="hljs-comment">// When increasing nmidle, nmidlelocked, nmsys, or nmfreed, be</span><span class="hljs-comment">// sure to call checkdead().</span><span class="hljs-comment">//空闲的m</span>midle        muintptr <span class="hljs-comment">// idle m's waiting for work</span>nmidle       <span class="hljs-keyword">int32</span>    <span class="hljs-comment">// number of idle m's waiting for work</span>nmidlelocked <span class="hljs-keyword">int32</span>    <span class="hljs-comment">// number of locked m's waiting for work</span>mnext        <span class="hljs-keyword">int64</span>    <span class="hljs-comment">// number of m's that have been created and next M ID</span>maxmcount    <span class="hljs-keyword">int32</span>    <span class="hljs-comment">// maximum number of m's allowed (or die)</span>nmsys        <span class="hljs-keyword">int32</span>    <span class="hljs-comment">// number of system m's not counted for deadlock</span>nmfreed      <span class="hljs-keyword">int64</span>    <span class="hljs-comment">// cumulative number of freed m's</span>ngsys <span class="hljs-keyword">uint32</span> <span class="hljs-comment">// number of system goroutines; updated atomically</span><span class="hljs-comment">//空闲的p</span>pidle      puintptr <span class="hljs-comment">// idle p's</span>npidle     <span class="hljs-keyword">uint32</span><span class="hljs-comment">//在spinning状态的m的数量</span>nmspinning <span class="hljs-keyword">uint32</span> <span class="hljs-comment">// See "Worker thread parking/unparking" comment in proc.go.</span><span class="hljs-comment">// Global runnable queue.</span><span class="hljs-comment">//可运行 的globalq</span>runq     gQueuerunqsize <span class="hljs-keyword">int32</span><span class="hljs-comment">// disable controls selective disabling of the scheduler.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Use schedEnableUser to control this.</span><span class="hljs-comment">//</span><span class="hljs-comment">// disable is protected by sched.lock.</span>disable <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// user disables scheduling of user goroutines.</span>user     <span class="hljs-keyword">bool</span>runnable gQueue <span class="hljs-comment">// pending runnable Gs</span>n        <span class="hljs-keyword">int32</span>  <span class="hljs-comment">// length of runnable</span>&#125;<span class="hljs-comment">// Global cache of dead G's.</span><span class="hljs-comment">//全局空余剩下的g</span>gFree <span class="hljs-keyword">struct</span> &#123;lock    mutexstack   gList <span class="hljs-comment">// Gs with stacks</span>noStack gList <span class="hljs-comment">// Gs without stacks</span>n       <span class="hljs-keyword">int32</span>&#125;&#125;</code></pre></div><p>相关结构可以在runtime/runtime2.go 中找到</p><h3 id="初始化生成新的goroutine">初始化生成新的goroutine</h3><p>g的状态转换图:</p><p><img src="/img/gstatus.png" srcset="/img/loading.gif" alt="一个状态图"></p><p>状态的详细描述:</p><ul><li><code>_Gidle</code>:刚刚被分配还没有初始化</li><li><code>_Grunnable</code>:没有执行代码，没有栈的所有权，存储在runq（local or global???）中；</li><li><code>_Grunning</code>:可以执行代码，拥有栈的所有权，绑定了M，P；</li><li><code>_Gsyscall</code>: 正在执行系统调用，拥有栈的所有权，没有执行用户代码，被赋予了内核线程 M 但是不在运行队列上;</li><li><code>_Gwaiting</code>:由于运行时而被阻塞，没有执行用户代码并且不在runq上，但是可能存在于<code>Channel</code>的等待队列上或者<code>lock</code>内等等;</li><li><code>_Gdead</code>:没有被使用，没有执行代码，<strong>可能有分配的栈???</strong>,或者在<code>gFree</code>(g的一个字段,全部状态都是<code>Gdead</code>);</li><li><code>_Gcopystack</code>:栈正在被拷贝，没有执行代码，不在运行队列上;</li><li><code>_Gpreempted</code>:由于抢占而被阻塞，没有执行用户代码并且不在运行队列上，等待唤醒;</li><li><code>_Gscan</code>: GC 正在扫描栈空间，没有执行代码，可以与其他状态同时存（其值为0x1000,其他有些状态比如<code>_GscanRunning</code>=2，直接加上去）</li></ul><p>当有新的Goroutine被创建或者是现存的goroutine更新为runnable状态，它会被push到当前P的runnable goroutine list里面，当P完成了执行goroutine，它会</p><ul><li>首先从自己的runnable g list里面pop一个goroutine，如果list是空的，它会随机选取其他P，并且偷取其list的一半runnable goroutine</li></ul><p>当M 创建了新的goroutine，它要保证有其他M执行这个goroutine同样的，如果M进入了syscall阶段，它也要保证有其他M可以执行这个goroutine</p><p>语言层面上，当然是编译器先检查有无<code>go</code>关键字，在编译期:<code>cmd/compile/internal/gc.state.stmt</code>和<code>cmd/compile/internal/gc.state.call</code>会将其转换成<code>runtime.newproc</code>函数</p><p>??? 不懂，ssa要学一下才行了</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(s *state)</span> <span class="hljs-title">call</span><span class="hljs-params">(n *Node, k callKind)</span> *<span class="hljs-title">ssa</span>.<span class="hljs-title">Value</span></span> &#123;<span class="hljs-keyword">if</span> k == callDeferStack &#123;...&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-keyword">switch</span> &#123;<span class="hljs-keyword">case</span> k == callGo:call = s.newValue1A(ssa.OpStaticCall, types.TypeMem, newproc, s.mem())<span class="hljs-keyword">default</span>:&#125;&#125;...&#125;</code></pre></div><ul><li><code>runtime.newproc</code>函数，使用了<code>g0</code>系统栈创建goroutine，还有go后面接着的function,传入参数有:<code>fn</code> 函数入口地址, <code>argp</code> 为参数起始地址, <code>siz</code>参数长度, <code>gp</code>（g0），调用方 <code>pc（goroutine）</code>（因为其假定了<strong>函数的传入参数一定跟在fn的地址后面</strong>，如果split了栈，则无法寻找到对应的传入参数，所以加上nosplit）</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Create a new g running fn with siz bytes of arguments.</span><span class="hljs-comment">// Put it on the queue of g's waiting to run.</span><span class="hljs-comment">// The compiler turns a go statement into a call to this.</span><span class="hljs-comment">// Cannot split the stack because it assumes that the arguments</span><span class="hljs-comment">// are available sequentially after &amp;fn; they would not be</span><span class="hljs-comment">// copied if a stack split occurred.</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newproc</span><span class="hljs-params">(siz <span class="hljs-keyword">int32</span>, fn *funcval)</span></span> &#123;<span class="hljs-comment">// 从 fn 的地址增加一个指针的长度，从而获取第一参数地址</span>argp := add(unsafe.Pointer(&amp;fn), sys.PtrSize)gp := getg()pc := getcallerpc()<span class="hljs-comment">// 获取调用方 PC/IP 寄存器值</span><span class="hljs-comment">//用 g0 系统栈创建goroutine</span>systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-comment">// fn 函数入口地址, argp 为参数起始地址, siz 参数长度, gp（g0），调用方 pc（goroutine）</span>newproc1(fn, argp, siz, gp, pc)&#125;)&#125;<span class="hljs-keyword">type</span> funcval <span class="hljs-keyword">struct</span> &#123;fn <span class="hljs-keyword">uintptr</span><span class="hljs-comment">//变长的变量，fn数据的头部指针</span><span class="hljs-comment">// variable-size, fn-specific data here</span>&#125;<span class="hljs-comment">//caller的pc值</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">getcallerpc</span><span class="hljs-params">()</span> <span class="hljs-title">uintptr</span></span></code></pre></div><p>看一个例子:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">package</span> main<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">sayhi</span><span class="hljs-params">(s <span class="hljs-keyword">string</span>)</span></span>&#123;<span class="hljs-built_in">println</span>(s)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-keyword">go</span> sayhi(<span class="hljs-string">"hi"</span>)&#125;</code></pre></div><p>汇编解析:</p><div class="hljs"><pre><code class="hljs plan9">0x0000 00000 (/home/main.go:7)TEXT&quot;&quot;.main(SB), ABIInternal, $40-00x0000 00000 (/home/main.go:7)MOVQ(TLS), CX0x0009 00009 (/home/main.go:7)CMPQSP, 16(CX)0x000d 00013 (/home/main.go:7)PCDATA$0, $-20x000d 00013 (/home/main.go:7)JLS840x000f 00015 (/home/main.go:7)PCDATA$0, $-10x000f 00015 (/home/main.go:7)SUBQ$40, SP0x0013 00019 (/home/main.go:7)MOVQBP, 32(SP)0x0018 00024 (/home/main.go:7)LEAQ32(SP), BP0x001d 00029 (/home/main.go:7)PCDATA$0, $-20x001d 00029 (/home/main.go:7)PCDATA$1, $-20x001d 00029 (/home/main.go:7)FUNCDATA$0, gclocals·33cdeccccebe80329f1fdbee7f5874cb(SB) //gc用,局部函数调用参数，需要回收0x001d 00029 (/home/main.go:7)FUNCDATA$1, gclocals·33cdeccccebe80329f1fdbee7f5874cb(SB)0x001d 00029 (/home/main.go:7)FUNCDATA$2, gclocals·9fb7f0986f647f17cb53dda1484e0f7a(SB)0x001d 00029 (/home/main.go:8)PCDATA$0, $00x001d 00029 (/home/main.go:8)PCDATA$1, $00x001d 00029 (/home/main.go:8)MOVL$16, (SP)0x0024 00036 (/home/main.go:8)PCDATA$0, $10x0024 00036 (/home/main.go:8)LEAQ&quot;&quot;.sayhi·f(SB), AX0x002b 00043 (/home/main.go:8)PCDATA$0, $00x002b 00043 (/home/main.go:8)MOVQAX, 8(SP)0x0030 00048 (/home/main.go:8)PCDATA$0, $10x0030 00048 (/home/main.go:8)LEAQgo.string.&quot;hi&quot;(SB), AX // 将 &quot;hi&quot; 的地址给 AX0x0037 00055 (/home/main.go:8)PCDATA$0, $00x0037 00055 (/home/main.go:8)MOVQAX, 16(SP)  // 将 AX 的值放到 16(SP)0x003c 00060 (/home/main.go:8)MOVQ$2, 24(SP)0x0045 00069 (/home/main.go:8)CALLruntime.newproc(SB)0x004a 00074 (/home/main.go:9)MOVQ32(SP), BP0x004f 00079 (/home/main.go:9)ADDQ$40, SP0x0053 00083 (/home/main.go:9)RET0x0054 00084 (/home/main.go:9)NOP0x0054 00084 (/home/main.go:7)PCDATA$1, $-10x0054 00084 (/home/main.go:7)PCDATA$0, $-20x0054 00084 (/home/main.go:7)CALLruntime.morestack_noctxt(SB)0x0059 00089 (/home/main.go:7)PCDATA$0, $-10x0059 00089 (/home/main.go:7)JMP0LEAQ go.string.*+1874(SB), AX // 将 &quot;hello world&quot; 的地址给 AXMOVQ AX, 0x10(SP)             // 将 AX 的值放到 0x10MOVL $0x10, 0(SP)             // 将最后一个参数的位置存到栈顶 0x00LEAQ go.func.*+67(SB), AX     // 将 go 语句调用的函数入口地址给 AXMOVQ AX, 0x8(SP)              // 将 AX 存入 0x08CALL runtime.newproc(SB)      // 调用 newproc</code></pre></div><p>//???todo</p><div class="hljs"><pre><code class="hljs s">             栈布局      |                 |       高地址      |                 |      +-----------------+       |     &amp;&quot;hi&quot;  |0x16  +-----------------+ &lt;--- fn + sys.PtrSize      |      sayhi      |0x08  +-----------------+ &lt;--- fn      |       siz       |0x00  +-----------------+ SP      |    newproc PC   |        +-----------------+ callerpc: 要运行的 Goroutine 的 PC      |                 |      |                 |       低地址</code></pre></div><p>注意到会在系统栈下调用<code>newproc1</code>，传入的是go关键字后函数的地址，这个函数caller的pc值、goroutine，传入参数地址大小等等信息</p><ol><li>首先就是创建<code>newg</code>,会调用<code>gfget</code>从<code>gfree</code>链表或者全局<code>sched.gFree</code>(已经执行过的g)上拿到空闲的goroutine或者创建一个新的goroutine，都没有就创建一个goroutine</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Create a new g running fn with narg bytes of arguments starting</span><span class="hljs-comment">// at argp. callerpc is the address of the go statement that created</span><span class="hljs-comment">// this. The new g is put on the queue of g's waiting to run.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newproc1</span><span class="hljs-params">(fn *funcval, argp unsafe.Pointer, narg <span class="hljs-keyword">int32</span>, callergp *g, callerpc <span class="hljs-keyword">uintptr</span>)</span></span> &#123;_g_ := getg()<span class="hljs-keyword">if</span> fn == <span class="hljs-literal">nil</span> &#123;_g_.m.throwing = <span class="hljs-number">-1</span> <span class="hljs-comment">// do not dump full stacks</span>throw(<span class="hljs-string">"go of nil func value"</span>)&#125;acquirem() <span class="hljs-comment">// disable preemption because it can be holding p in a local var</span>siz := nargsiz = (siz + <span class="hljs-number">7</span>) &amp;^ <span class="hljs-number">7</span><span class="hljs-comment">// We could allocate a larger initial stack if necessary.</span><span class="hljs-comment">// Not worth it: this is almost always an error.</span><span class="hljs-comment">// 4*sizeof(uintreg): extra space added below</span><span class="hljs-comment">// sizeof(uintreg): caller's LR (arm) or return address (x86, in gostartcall).</span><span class="hljs-keyword">if</span> siz &gt;= _StackMin<span class="hljs-number">-4</span>*sys.RegSize-sys.RegSize &#123;throw(<span class="hljs-string">"newproc: function arguments too large for new goroutine"</span>)&#125;_p_ := _g_.m.p.ptr()newg := gfget(_p_)<span class="hljs-keyword">if</span> newg == <span class="hljs-literal">nil</span> &#123;newg = malg(_StackMin)casgstatus(newg, _Gidle, _Gdead)allgadd(newg) <span class="hljs-comment">// publishes with a g-&gt;status of Gdead so GC scanner doesn't look at uninitialized stack.</span>&#125;...&#125;</code></pre></div><p>注意到获得<code>newg</code>的方法中，有<code>gfget</code>和<code>malg</code>两种</p><ul><li><p><code>gfget</code>:如果是当前p的<code>gfree</code> list为空，就从全局调度器<code>sched.gFree</code>转移到当前p上，上限为32;</p></li><li><p>如果充足的话就会从<code>gfree</code>列表<strong>头部</strong>返回一个新的goroutine；</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Get from gfree list.</span><span class="hljs-comment">// If local list is empty, grab a batch from global list.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">gfget</span><span class="hljs-params">(_p_ *p)</span> *<span class="hljs-title">g</span></span> &#123;retry:<span class="hljs-keyword">if</span> _p_.gFree.empty() &amp;&amp; (!sched.gFree.stack.empty() || !sched.gFree.noStack.empty()) &#123;lock(&amp;sched.gFree.lock)<span class="hljs-comment">// Move a batch of free Gs to the P.</span><span class="hljs-keyword">for</span> _p_.gFree.n &lt; <span class="hljs-number">32</span> &#123;<span class="hljs-comment">// Prefer Gs with stacks.</span>gp := sched.gFree.stack.pop()<span class="hljs-keyword">if</span> gp == <span class="hljs-literal">nil</span> &#123;gp = sched.gFree.noStack.pop()<span class="hljs-keyword">if</span> gp == <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">break</span>&#125;&#125;sched.gFree.n--_p_.gFree.push(gp)_p_.gFree.n++&#125;unlock(&amp;sched.gFree.lock)<span class="hljs-keyword">goto</span> retry&#125;</code></pre></div><p>因为这里我们讨论的是初始化，所以上述两种情况都不会发生，会直接使用下面:</p><ul><li><code>malg</code>来初始化一个新的goroutine:</li></ul><p>call <code>newg</code>方法，然后分配2KB栈空间,并设为<code>_Gidle</code>状态(值为0)</p><p>创建完成后，其返回的值，会放入到全局<code>allg</code> slice上面，从<code>_Gidle</code>设为<code>Gdead</code>状态，所以gc也不会扫描这个未初始化的栈</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Allocate a new g, with a stack big enough for stacksize bytes.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">malg</span><span class="hljs-params">(stacksize <span class="hljs-keyword">int32</span>)</span> *<span class="hljs-title">g</span></span> &#123;newg := <span class="hljs-built_in">new</span>(g)<span class="hljs-keyword">if</span> stacksize &gt;= <span class="hljs-number">0</span> &#123;<span class="hljs-comment">//stacksize = 2KB</span>stacksize = round2(_StackSystem + stacksize)systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;newg.stack = stackalloc(<span class="hljs-keyword">uint32</span>(stacksize))&#125;)newg.stackguard0 = newg.stack.lo + _StackGuardnewg.stackguard1 = ^<span class="hljs-keyword">uintptr</span>(<span class="hljs-number">0</span>)<span class="hljs-comment">// Clear the bottom word of the stack. We record g</span><span class="hljs-comment">// there on gsignal stack during VDSO on ARM and ARM64.</span>*(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(newg.stack.lo)) = <span class="hljs-number">0</span>&#125;<span class="hljs-keyword">return</span> newg&#125;</code></pre></div><ol start="2"><li>接下来就用<code>memove</code>copy fn的所有参数到栈中,<code>argp</code>和<code>narg</code>分别为参数内存地址和大小，根据要执行函数的入口地址和参数，初始化执行栈的 <code>SP</code> 和参数的入栈位置，并将需要的参数拷贝一份存入执行栈中;</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newproc1</span><span class="hljs-params">(fn *funcval, argp unsafe.Pointer, narg <span class="hljs-keyword">int32</span>, callergp *g, callerpc <span class="hljs-keyword">uintptr</span>)</span></span> &#123;....<span class="hljs-comment">//内存对齐</span>totalSize := <span class="hljs-number">4</span>*sys.RegSize + <span class="hljs-keyword">uintptr</span>(siz) + sys.MinFrameSize <span class="hljs-comment">// extra space in case of reads slightly beyond frame</span>totalSize += -totalSize &amp; (sys.SpAlign - <span class="hljs-number">1</span>)                  <span class="hljs-comment">// align to spAlign</span>sp := newg.stack.hi - totalSize<span class="hljs-comment">//栈的地址</span>spArg := sp<span class="hljs-keyword">if</span> usesLR &#123;<span class="hljs-comment">// caller's LR</span>*(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(sp)) = <span class="hljs-number">0</span>prepGoExitFrame(sp)spArg += sys.MinFrameSize&#125;<span class="hljs-comment">//处理传入的参数，有参数时</span><span class="hljs-keyword">if</span> narg &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">//从argp的位置开始，复制narg个bytes到spArg</span>memmove(unsafe.Pointer(spArg), argp, <span class="hljs-keyword">uintptr</span>(narg))<span class="hljs-comment">// This is a stack-to-stack copy. If write barriers</span><span class="hljs-comment">// are enabled and the source stack is grey (the</span><span class="hljs-comment">// destination is always black), then perform a</span><span class="hljs-comment">// barrier copy. We do this *after* the memmove</span><span class="hljs-comment">// because the destination stack may have garbage on</span><span class="hljs-comment">// it.</span><span class="hljs-comment">//栈到栈的copy，如果用了写屏障，且源栈为灰色（目标始终为黑色），则执行barrier copy，因为目标栈上可能有垃圾，</span><span class="hljs-keyword">if</span> writeBarrier.needed &amp;&amp; !_g_.m.curg.gcscandone &#123;f := findfunc(fn.fn)stkmap := (*stackmap)(funcdata(f, _FUNCDATA_ArgsPointerMaps))<span class="hljs-keyword">if</span> stkmap.nbit &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// We're in the prologue, so it's always stack map index 0.</span>bv := stackmapdata(stkmap, <span class="hljs-number">0</span>)bulkBarrierBitmap(spArg, spArg, <span class="hljs-keyword">uintptr</span>(bv.n)*sys.PtrSize, <span class="hljs-number">0</span>, bv.bytedata)&#125;&#125;&#125;...</code></pre></div><ol start="3"><li>然后复制之后，根据SP以及相关参数清理创建并初始化g的运行现场，然后将调用方、要执行的函数的入口 PC 进行保存，并将其状态改为<code>Grunnable</code>;</li></ol><p>其中<code>gostartcallfn</code>方法在(后面)[#调度循环]会详细聊到</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newproc1</span><span class="hljs-params">(fn *funcval, argp unsafe.Pointer, narg <span class="hljs-keyword">int32</span>, callergp *g, callerpc <span class="hljs-keyword">uintptr</span>)</span></span> &#123;...<span class="hljs-comment">//// memclrNoHeapPointers clears n bytes starting at ptr.</span><span class="hljs-comment">//清除所有sched</span>memclrNoHeapPointers(unsafe.Pointer(&amp;newg.sched), unsafe.Sizeof(newg.sched))newg.sched.sp = spnewg.stktopsp = sp<span class="hljs-comment">//注意这里， 将pc改为,runtime.goexit 函数和</span><span class="hljs-comment">//将g改为新的goroutine</span>newg.sched.pc = funcPC(goexit) + sys.PCQuantum <span class="hljs-comment">// +PCQuantum so that previous instruction is in same function</span>newg.sched.g = guintptr(unsafe.Pointer(newg))gostartcallfn(&amp;newg.sched, fn)newg.gopc = callerpcnewg.ancestors = saveAncestors(callergp)newg.startpc = fn.fn<span class="hljs-keyword">if</span> _g_.m.curg != <span class="hljs-literal">nil</span> &#123;newg.labels = _g_.m.curg.labels&#125;<span class="hljs-keyword">if</span> isSystemGoroutine(newg, <span class="hljs-literal">false</span>) &#123;atomic.Xadd(&amp;sched.ngsys, +<span class="hljs-number">1</span>)&#125;<span class="hljs-comment">//更改状态</span>casgstatus(newg, _Gdead, _Grunnable)<span class="hljs-keyword">if</span> _p_.goidcache == _p_.goidcacheend &#123;<span class="hljs-comment">// Sched.goidgen is the last allocated id,</span><span class="hljs-comment">// this batch must be [sched.goidgen+1, sched.goidgen+GoidCacheBatch].</span><span class="hljs-comment">// At startup sched.goidgen=0, so main goroutine receives goid=1.</span>_p_.goidcache = atomic.Xadd64(&amp;sched.goidgen, _GoidCacheBatch)_p_.goidcache -= _GoidCacheBatch - <span class="hljs-number">1</span>_p_.goidcacheend = _p_.goidcache + _GoidCacheBatch&#125;newg.goid = <span class="hljs-keyword">int64</span>(_p_.goidcache)_p_.goidcache++...</code></pre></div><ol start="4"><li>给 Goroutine 分配 id，并将其放入 P 本地队列的队头或全局队列（初始化阶段队列肯定不是满的，因此不可能放入全局队列）,最后将初始化好的goroutine放入runq</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//接上面</span>...<span class="hljs-comment">//放入runq,传入是true，放入runnext，如果是false就会放入runq的尾部</span>runqput(_p_, newg, <span class="hljs-literal">true</span>)<span class="hljs-keyword">if</span> atomic.Load(&amp;sched.npidle) != <span class="hljs-number">0</span> &amp;&amp; atomic.Load(&amp;sched.nmspinning) == <span class="hljs-number">0</span> &amp;&amp; mainStarted &#123;wakep()&#125;releasem(_g_.m)</code></pre></div><p>注意这个<code>runqput</code>方法中</p><ul><li>首先会将其放入local runnable q，并将其放入<code>_p_.runnext</code></li><li>如果传入next为<code>false</code>且本地runq未满，就会放入本地runq尾部</li><li>如果传入next为<code>true</code>,并将其放入<code>_p_.runnext</code></li><li>如果本地runq满了，runnext会将其放入全局<code>sched.runq</code>(这里创建goroutine传入的是true)</li><li>这个只会被其拥有者P运行;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// runqput tries to put g on the local runnable queue.</span><span class="hljs-comment">// If next is false, runqput adds g to the tail of the runnable queue.</span><span class="hljs-comment">// If next is true, runqput puts g in the _p_.runnext slot.</span><span class="hljs-comment">// If the run queue is full, runnext puts g on the global queue.</span><span class="hljs-comment">// Executed only by the owner P.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">runqput</span><span class="hljs-params">(_p_ *p, gp *g, next <span class="hljs-keyword">bool</span>)</span></span> &#123;<span class="hljs-keyword">if</span> randomizeScheduler &amp;&amp; next &amp;&amp; fastrand()%<span class="hljs-number">2</span> == <span class="hljs-number">0</span> &#123;next = <span class="hljs-literal">false</span>&#125;<span class="hljs-keyword">if</span> next &#123;retryNext:oldnext := _p_.runnext<span class="hljs-keyword">if</span> !_p_.runnext.cas(oldnext, guintptr(unsafe.Pointer(gp))) &#123;<span class="hljs-keyword">goto</span> retryNext&#125;<span class="hljs-keyword">if</span> oldnext == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">// Kick the old runnext out to the regular run queue.</span>gp = oldnext.ptr()&#125;retry:h := atomic.LoadAcq(&amp;_p_.runqhead) <span class="hljs-comment">// load-acquire, synchronize with consumers</span>t := _p_.runqtail<span class="hljs-keyword">if</span> t-h &lt; <span class="hljs-keyword">uint32</span>(<span class="hljs-built_in">len</span>(_p_.runq)) &#123;_p_.runq[t%<span class="hljs-keyword">uint32</span>(<span class="hljs-built_in">len</span>(_p_.runq))].set(gp)atomic.StoreRel(&amp;_p_.runqtail, t+<span class="hljs-number">1</span>) <span class="hljs-comment">// store-release, makes the item available for consumption</span><span class="hljs-keyword">return</span>&#125;<span class="hljs-keyword">if</span> runqputslow(_p_, gp, h, t) &#123;<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">// the queue is not full, now the put above must succeed</span><span class="hljs-keyword">goto</span> retry&#125;</code></pre></div><ol start="5"><li>检查空闲的 P，将其唤醒，准备执行 G，但我们目前处于初始化阶段，主goroutine尚未开始执行，<strong>因此这里不会唤醒 P</strong>;</li></ol><p>提多一句，整个<code>newproc</code>在nosplit环境下，所以执行过程中是不会发生扩容和抢占；</p><h3 id="调度循环">调度循环</h3><p>上面提到<code>gostartcallfn</code>方法，其还会对<code>sched.pc</code>和<code>sched.sp</code>进行一些处理:其注释说的就是还有一些对<code>g.sched</code>还有一些</p><ul><li><p>上面讲到pc实际上保存的就是程序接下来的运行地址,<code>buf.pc = uintptr(fn)</code>就是从fn开始运行</p></li><li><p>但是这个<code>sp</code>老实说我不看这文章我tm让我找都找不到原因;</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// adjust Gobuf as if it executed a call to fn</span><span class="hljs-comment">// and then did an immediate gosave.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">gostartcallfn</span><span class="hljs-params">(gobuf *gobuf, fv *funcval)</span></span> &#123;<span class="hljs-keyword">var</span> fn unsafe.Pointer<span class="hljs-keyword">if</span> fv != <span class="hljs-literal">nil</span> &#123;fn = unsafe.Pointer(fv.fn)&#125; <span class="hljs-keyword">else</span> &#123;fn = unsafe.Pointer(funcPC(nilfunc))&#125;gostartcall(gobuf, fn, unsafe.Pointer(fv))&#125;<span class="hljs-comment">// adjust Gobuf as if it executed a call to fn with context ctxt</span><span class="hljs-comment">// and then did an immediate gosave.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">gostartcall</span><span class="hljs-params">(buf *gobuf, fn, ctxt unsafe.Pointer)</span></span> &#123;sp := buf.sp<span class="hljs-comment">//为什么???</span><span class="hljs-keyword">if</span> sys.RegSize &gt; sys.PtrSize &#123;sp -= sys.PtrSize*(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(sp)) = <span class="hljs-number">0</span>&#125;sp -= sys.PtrSize*(*<span class="hljs-keyword">uintptr</span>)(unsafe.Pointer(sp)) = buf.pcbuf.sp = spbuf.pc = <span class="hljs-keyword">uintptr</span>(fn)buf.ctxt = ctxt&#125;</code></pre></div><h4 id="mstart开始">mstart开始</h4><p>调度从<code>runtime.mstart</code>开始，无栈要求，所以no split;其因为无P，所以暂时不需要写屏障;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// mstart is the entry-point for new Ms.</span><span class="hljs-comment">//</span><span class="hljs-comment">// This must not split the stack because we may not even have stack</span><span class="hljs-comment">// bounds set up yet.</span><span class="hljs-comment">//</span><span class="hljs-comment">// May run during STW (because it doesn't have a P yet), so write</span><span class="hljs-comment">// barriers are not allowed.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-comment">//go:nowritebarrierrec</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">mstart</span><span class="hljs-params">()</span></span> &#123;_g_ := getg()osStack := _g_.stack.lo == <span class="hljs-number">0</span><span class="hljs-keyword">if</span> osStack &#123;<span class="hljs-comment">// Initialize stack bounds from system stack.</span><span class="hljs-comment">// Cgo may have left stack size in stack.hi.</span><span class="hljs-comment">// minit may update the stack bounds.</span>size := _g_.stack.hi<span class="hljs-keyword">if</span> size == <span class="hljs-number">0</span> &#123;size = <span class="hljs-number">8192</span> * sys.StackGuardMultiplier&#125;_g_.stack.hi = <span class="hljs-keyword">uintptr</span>(noescape(unsafe.Pointer(&amp;size)))_g_.stack.lo = _g_.stack.hi - size + <span class="hljs-number">1024</span>&#125;<span class="hljs-comment">// Initialize stack guard so that we can start calling regular</span><span class="hljs-comment">// Go code.</span>_g_.stackguard0 = _g_.stack.lo + _StackGuard<span class="hljs-comment">// This is the g0, so we can also call go:systemstack</span><span class="hljs-comment">// functions, which check stackguard1.</span>_g_.stackguard1 = _g_.stackguard0mstart1()<span class="hljs-comment">// Exit this thread.</span><span class="hljs-keyword">switch</span> GOOS &#123;<span class="hljs-keyword">case</span> <span class="hljs-string">"windows"</span>, <span class="hljs-string">"solaris"</span>, <span class="hljs-string">"illumos"</span>, <span class="hljs-string">"plan9"</span>, <span class="hljs-string">"darwin"</span>, <span class="hljs-string">"aix"</span>:<span class="hljs-comment">// Windows, Solaris, illumos, Darwin, AIX and Plan 9 always system-allocate</span><span class="hljs-comment">// the stack, but put it in _g_.stack before mstart,</span><span class="hljs-comment">// so the logic above hasn't set osStack yet.</span>osStack = <span class="hljs-literal">true</span>&#125;mexit(osStack)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">mstart1</span><span class="hljs-params">()</span></span> &#123;_g_ := getg()<span class="hljs-keyword">if</span> _g_ != _g_.m.g0 &#123;throw(<span class="hljs-string">"bad runtime·mstart"</span>)&#125;<span class="hljs-comment">// Record the caller for use as the top of stack in mcall and</span><span class="hljs-comment">// for terminating the thread.</span><span class="hljs-comment">// We're never coming back to mstart1 after we call schedule,</span><span class="hljs-comment">// so other calls can reuse the current frame.</span>save(getcallerpc(), getcallersp())asminit()minit()<span class="hljs-comment">// Install signal handlers; after minit so that minit can</span><span class="hljs-comment">// prepare the thread to be able to handle the signals.</span><span class="hljs-keyword">if</span> _g_.m == &amp;m0 &#123;mstartm0()&#125;<span class="hljs-keyword">if</span> fn := _g_.m.mstartfn; fn != <span class="hljs-literal">nil</span> &#123;fn()&#125;<span class="hljs-keyword">if</span> _g_.m != &amp;m0 &#123;acquirep(_g_.m.nextp.ptr())_g_.m.nextp = <span class="hljs-number">0</span>&#125;schedule()&#125;</code></pre></div><p><code>mstart</code>初始化了<code>stackguard0</code> 和 <code>stackguard1</code>，<code>mstart1()</code>调用<code>runtime.schedule()</code>大概步骤:</p><ul><li><p>为了保证公平(可能两个goroutine互相切换)，一段时间（判断<code>g.schedtick</code>)会先检查全局队列，如果有，会从全局队列中拿一些goroutine来运行;</p></li><li><p>拿不到就接着从本地runq拿</p></li><li><p>再拿不到就要用<code>findrunnable</code>方法来拿,注意这个方法是阻塞的</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// One round of scheduler: find a runnable goroutine and execute it.</span><span class="hljs-comment">// Never returns.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">schedule</span><span class="hljs-params">()</span></span> &#123;_g_ := getg()...top:pp := _g_.m.p.ptr()pp.preempt = <span class="hljs-literal">false</span><span class="hljs-keyword">if</span> sched.gcwaiting != <span class="hljs-number">0</span> &#123;gcstopm()<span class="hljs-keyword">goto</span> top&#125;<span class="hljs-keyword">if</span> pp.runSafePointFn != <span class="hljs-number">0</span> &#123;runSafePointFn()&#125;<span class="hljs-comment">// Sanity check: if we are spinning, the run queue should be empty.</span><span class="hljs-comment">// Check this before calling checkTimers, as that might call</span><span class="hljs-comment">// goready to put a ready goroutine on the local run queue.</span><span class="hljs-keyword">if</span> _g_.m.spinning &amp;&amp; (pp.runnext != <span class="hljs-number">0</span> || pp.runqhead != pp.runqtail) &#123;throw(<span class="hljs-string">"schedule: spinning with local work"</span>)&#125;checkTimers(pp, <span class="hljs-number">0</span>)<span class="hljs-keyword">var</span> gp *g<span class="hljs-keyword">var</span> inheritTime <span class="hljs-keyword">bool</span>...<span class="hljs-comment">//先检查全局队列</span><span class="hljs-keyword">if</span> gp == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// Check the global runnable queue once in a while to ensure fairness.</span><span class="hljs-comment">// Otherwise two goroutines can completely occupy the local runqueue</span><span class="hljs-comment">// by constantly respawning each other.</span><span class="hljs-keyword">if</span> _g_.m.p.ptr().schedtick%<span class="hljs-number">61</span> == <span class="hljs-number">0</span> &amp;&amp; sched.runqsize &gt; <span class="hljs-number">0</span> &#123;lock(&amp;sched.lock)gp = globrunqget(_g_.m.p.ptr(), <span class="hljs-number">1</span>)unlock(&amp;sched.lock)&#125;&#125;<span class="hljs-keyword">if</span> gp == <span class="hljs-literal">nil</span> &#123;gp, inheritTime = runqget(_g_.m.p.ptr())<span class="hljs-comment">// We can see gp != nil here even if the M is spinning,</span><span class="hljs-comment">// if checkTimers added a local goroutine via goready.</span>&#125;<span class="hljs-keyword">if</span> gp == <span class="hljs-literal">nil</span> &#123;gp, inheritTime = findrunnable() <span class="hljs-comment">// blocks until work is available</span>&#125;&#125;</code></pre></div><p>下面是<code>findrunnable</code>的大概流程:</p><ul><li>从本地，全局队列拿</li><li>通过<code>runtime.runqsteal</code>尝试从其他处理器§拿goroutine，这个是随机获取(可以看一下<code>RandomOrder</code>这个结构,其方法保证了公平性)这个过程还可能将其计时器都拿过来(有什么用？？？针对该goroutine，可以保持时间继续？？？)</li><li>还找不到，就从netpoll里面拿;</li><li>最后跑到<code>runtime.execute</code>方法执行获取的goroutine，做好准备工作后，就通过<code>runtime.gogo</code>(汇编，之前的文章提到过)将goroutine调度到当前线程上;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Schedules gp to run on the current M.</span><span class="hljs-comment">// If inheritTime is true, gp inherits the remaining time in the</span><span class="hljs-comment">// current time slice. Otherwise, it starts a new time slice.</span><span class="hljs-comment">// Never returns.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Write barriers are allowed because this is called immediately after</span><span class="hljs-comment">// acquiring a P in several places.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:yeswritebarrierrec</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">execute</span><span class="hljs-params">(gp *g, inheritTime <span class="hljs-keyword">bool</span>)</span></span> &#123;_g_ := getg()<span class="hljs-comment">// Assign gp.m before entering _Grunning so running Gs have an</span><span class="hljs-comment">// M.</span>_g_.m.curg = gpgp.m = _g_.mcasgstatus(gp, _Grunnable, _Grunning)gp.waitsince = <span class="hljs-number">0</span>gp.preempt = <span class="hljs-literal">false</span>gp.stackguard0 = gp.stack.lo + _StackGuard<span class="hljs-keyword">if</span> !inheritTime &#123;_g_.m.p.ptr().schedtick++&#125;<span class="hljs-comment">// Check whether the profiler needs to be turned on or off.</span>hz := sched.profilehz<span class="hljs-keyword">if</span> _g_.m.profilehz != hz &#123;setThreadCPUProfiler(hz)&#125;<span class="hljs-keyword">if</span> trace.enabled &#123;<span class="hljs-comment">// GoSysExit has to happen when we have a P, but before GoStart.</span><span class="hljs-comment">// So we emit it here.</span><span class="hljs-keyword">if</span> gp.syscallsp != <span class="hljs-number">0</span> &amp;&amp; gp.sysblocktraced &#123;traceGoSysExit(gp.sysexitticks)&#125;traceGoStart()&#125;gogo(&amp;gp.sched)&#125;</code></pre></div><p>有关于<code>gogo</code>函数，这里再次post一次(<code>linux amd64</code>上):</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// func gogo(buf *gobuf)</span><span class="hljs-comment">// restore state from Gobuf; longjmp</span>TEXT runtime·gogo(SB), NOSPLIT, $<span class="hljs-number">16</span><span class="hljs-number">-8</span>MOVQbuf+<span class="hljs-number">0</span>(FP), BX<span class="hljs-comment">// gobuf获取调度信息</span>MOVQgobuf_g(BX), DXMOVQ<span class="hljs-number">0</span>(DX), CX<span class="hljs-comment">// make sure g != nil</span>get_tls(CX)<span class="hljs-comment">//获得当前线程</span>MOVQDX, g(CX)MOVQgobuf_sp(BX), SP<span class="hljs-comment">//  1.restore SP 将 runtime.goexit 函数的 PC 恢复到 SP 中</span>MOVQgobuf_ret(BX), AXMOVQgobuf_ctxt(BX), DXMOVQgobuf_bp(BX), BPMOVQ$<span class="hljs-number">0</span>, gobuf_sp(BX)<span class="hljs-comment">// clear to help garbage collector</span>MOVQ$<span class="hljs-number">0</span>, gobuf_ret(BX)MOVQ$<span class="hljs-number">0</span>, gobuf_ctxt(BX)MOVQ$<span class="hljs-number">0</span>, gobuf_bp(BX)MOVQgobuf_pc(BX), BX<span class="hljs-comment">// 2. 获取待执行函数的程序计数器</span>JMPBX <span class="hljs-comment">// 3. 开始执行</span></code></pre></div><p><code>runtime.gobuf</code>中取出了<code>runtime.goexit</code>的pc和待执行函数的pc，其中:</p><ul><li><code>runtime.goexit</code> 的程序计数器被放到了栈 SP 上；</li><li>待执行函数的程序计数器被放到了寄存器 BX 上；</li></ul><p>一般来讲，go的函数调用都会使用<code>CALL</code>指令，会先将返回地址加入到栈寄存器<code>SP</code>中，然后跳转到目标函数,当目标函数返回后，会从栈中查找调用的地址，并跳转回调用方继续执行剩下的代码，上面注释的1.2.3就是该过程</p><p>接着，在<code>JMP BX</code>命令后，当goroutine运行的函数返回时，就会跳转到<code>runtime.goexit</code>所在位置执行函数:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//????</span>TEXT runtime·goexit(SB),NOSPLIT,$<span class="hljs-number">0</span><span class="hljs-number">-0</span>CALLruntime·goexit1(SB)<span class="hljs-comment">// Finishes execution of the current goroutine.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">goexit1</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-keyword">if</span> raceenabled &#123;racegoend()&#125;<span class="hljs-keyword">if</span> trace.enabled &#123;traceGoEnd()&#125;mcall(goexit0)&#125;</code></pre></div><p>我们最终在当前线程的<code>m.g0</code>栈上调用了<code>goexit0</code>函数：</p><ul><li>该函数会将goroutine设为<code>_Gdead</code>状态，清除其中字段，移除goroutine和M关联;</li><li>调用<code>runtime.gfput</code>重新加入处理器goroutine的空闲列表<code>gFree</code></li><li>但是最后也会再次触发<code>runtime.schedule</code>，一切再次从头再来成为一个循环;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// goexit continuation on g0.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">goexit0</span><span class="hljs-params">(gp *g)</span></span> &#123;_g_ := getg()casgstatus(gp, _Grunning, _Gdead)<span class="hljs-keyword">if</span> isSystemGoroutine(gp, <span class="hljs-literal">false</span>) &#123;atomic.Xadd(&amp;sched.ngsys, <span class="hljs-number">-1</span>)&#125;gp.m = <span class="hljs-literal">nil</span>locked := gp.lockedm != <span class="hljs-number">0</span>gp.lockedm = <span class="hljs-number">0</span>_g_.m.lockedg = <span class="hljs-number">0</span>gp.preemptStop = <span class="hljs-literal">false</span>gp.paniconfault = <span class="hljs-literal">false</span>gp._defer = <span class="hljs-literal">nil</span> <span class="hljs-comment">// should be true already but just in case.</span>gp._panic = <span class="hljs-literal">nil</span> <span class="hljs-comment">// non-nil for Goexit during panic. points at stack-allocated data.</span>gp.writebuf = <span class="hljs-literal">nil</span>gp.waitreason = <span class="hljs-number">0</span>gp.param = <span class="hljs-literal">nil</span>gp.labels = <span class="hljs-literal">nil</span>gp.timer = <span class="hljs-literal">nil</span><span class="hljs-keyword">if</span> gcBlackenEnabled != <span class="hljs-number">0</span> &amp;&amp; gp.gcAssistBytes &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// Flush assist credit to the global pool. This gives</span><span class="hljs-comment">// better information to pacing if the application is</span><span class="hljs-comment">// rapidly creating an exiting goroutines.</span>scanCredit := <span class="hljs-keyword">int64</span>(gcController.assistWorkPerByte * <span class="hljs-keyword">float64</span>(gp.gcAssistBytes))atomic.Xaddint64(&amp;gcController.bgScanCredit, scanCredit)gp.gcAssistBytes = <span class="hljs-number">0</span>&#125;dropg()<span class="hljs-keyword">if</span> GOARCH == <span class="hljs-string">"wasm"</span> &#123; <span class="hljs-comment">// no threads yet on wasm</span>gfput(_g_.m.p.ptr(), gp)schedule() <span class="hljs-comment">// never returns</span>&#125;<span class="hljs-keyword">if</span> _g_.m.lockedInt != <span class="hljs-number">0</span> &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"invalid m-&gt;lockedInt = "</span>, _g_.m.lockedInt, <span class="hljs-string">"\n"</span>)throw(<span class="hljs-string">"internal lockOSThread error"</span>)&#125;<span class="hljs-comment">//加入gfree，重用</span>gfput(_g_.m.p.ptr(), gp)<span class="hljs-keyword">if</span> locked &#123;<span class="hljs-comment">// The goroutine may have locked this thread because</span><span class="hljs-comment">// it put it in an unusual kernel state. Kill it</span><span class="hljs-comment">// rather than returning it to the thread pool.</span><span class="hljs-comment">// Return to mstart, which will release the P and exit</span><span class="hljs-comment">// the thread.</span><span class="hljs-keyword">if</span> GOOS != <span class="hljs-string">"plan9"</span> &#123; <span class="hljs-comment">// See golang.org/issue/22227.</span>gogo(&amp;_g_.m.g0.sched)&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">// Clear lockedExt on plan9 since we may end up re-using</span><span class="hljs-comment">// this thread.</span>_g_.m.lockedExt = <span class="hljs-number">0</span>&#125;&#125;schedule()&#125;</code></pre></div><ul><li>上面调用<code>goexit0</code>是通过<code>mcall</code>,mcall定义如下:？？？？为什么被重新调度g，fn不可以返回?因为这里可能进行重新一次调度，选了一个新的goroutine来占用m，详细可以见下面的<a href="#%E8%B0%83%E5%BA%A6%E6%97%B6%E6%9C%BA">调度时机</a>一般来将<code>mcall</code>作用是在goroutine变化时候调用的，在g0栈上执行新的函数</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// mcall switches from the g to the g0 stack and invokes fn(g),</span><span class="hljs-comment">// where g is the goroutine that made the call.</span><span class="hljs-comment">// mcall saves g's current PC/SP in g-&gt;sched so that it can be restored later.</span><span class="hljs-comment">// It is up to fn to arrange for that later execution, typically by recording</span><span class="hljs-comment">// g in a data structure, causing something to call ready(g) later.</span><span class="hljs-comment">// mcall returns to the original goroutine g later, when g has been rescheduled.</span><span class="hljs-comment">//????</span><span class="hljs-comment">// fn must not return at all; typically it ends by calling schedule, to let the m</span><span class="hljs-comment">// run other goroutines.</span><span class="hljs-comment">//</span><span class="hljs-comment">// mcall can only be called from g stacks (not g0, not gsignal).</span><span class="hljs-comment">// mcall只能在g的栈被调用;</span><span class="hljs-comment">// This must NOT be go:noescape: if fn is a stack-allocated closure,</span><span class="hljs-comment">// fn puts g on a run queue, and g executes before fn returns, the</span><span class="hljs-comment">// closure will be invalidated while it is still executing.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">mcall</span><span class="hljs-params">(fn <span class="hljs-keyword">func</span>(*g)</span>)</span></code></pre></div><p>综上,整个goroutine如果无抢占情况下的:调度循环如图<img src="/img/scheduleLoop.png" srcset="/img/loading.gif" alt="所示"></p><h3 id="调度时机">调度时机</h3><p>可以看这幅图<img src="/img/scheduleTrigger.png" srcset="/img/loading.gif" alt="一幅图">其实就是看一下哪里调用了<code>runtime.schedule</code>方法:</p><h4 id="主动挂起">主动挂起</h4><p>gopark方法,由channel操作、sleep、netpoll_block、gc、select等待</p><p><code>runtime.gopark</code>-&gt;<code>runtime.park_m</code></p><p><code>runtime.gopark</code>这个方法会将当前goroutine暂停，被暂停的任务不会放回runq:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Puts the current goroutine into a waiting state and calls unlockf.</span><span class="hljs-comment">// If unlockf returns false, the goroutine is resumed.</span><span class="hljs-comment">// unlockf must not access this G's stack, as it may be moved between</span><span class="hljs-comment">// the call to gopark and the call to unlockf.</span><span class="hljs-comment">// Reason explains why the goroutine has been parked.</span><span class="hljs-comment">// It is displayed in stack traces and heap dumps.</span><span class="hljs-comment">// Reasons should be unique and descriptive.</span><span class="hljs-comment">// Do not re-use reasons, add new ones.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">gopark</span><span class="hljs-params">(unlockf <span class="hljs-keyword">func</span>(*g, unsafe.Pointer)</span> <span class="hljs-title">bool</span>, <span class="hljs-title">lock</span> <span class="hljs-title">unsafe</span>.<span class="hljs-title">Pointer</span>, <span class="hljs-title">reason</span> <span class="hljs-title">waitReason</span>, <span class="hljs-title">traceEv</span> <span class="hljs-title">byte</span>, <span class="hljs-title">traceskip</span> <span class="hljs-title">int</span>)</span> &#123;<span class="hljs-keyword">if</span> reason != waitReasonSleep &#123;checkTimeouts() <span class="hljs-comment">// timeouts may expire while two goroutines keep the scheduler busy</span>&#125;mp := acquirem()gp := mp.curgstatus := readgstatus(gp)<span class="hljs-keyword">if</span> status != _Grunning &amp;&amp; status != _Gscanrunning &#123;throw(<span class="hljs-string">"gopark: bad g status"</span>)&#125;mp.waitlock = lockmp.waitunlockf = unlockfgp.waitreason = reasonmp.waittraceev = traceEvmp.waittraceskip = traceskipreleasem(mp)<span class="hljs-comment">// can't do anything that might move the G between Ms here.</span>mcall(park_m)&#125;</code></pre></div><p><code>runtime.park_m</code>:</p><ul><li>将状态变成<code>_Gwaiting</code></li><li>然后<code>dropg</code>将当前用户goroutine<code>m.curg</code>与m断开</li><li>此时就可以调用<code>schedule</code>触发新一轮调度</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// park continuation on g0.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">park_m</span><span class="hljs-params">(gp *g)</span></span> &#123;_g_ := getg()<span class="hljs-keyword">if</span> trace.enabled &#123;traceGoPark(_g_.m.waittraceev, _g_.m.waittraceskip)&#125;casgstatus(gp, _Grunning, _Gwaiting)dropg()<span class="hljs-keyword">if</span> fn := _g_.m.waitunlockf; fn != <span class="hljs-literal">nil</span> &#123;ok := fn(gp, _g_.m.waitlock)_g_.m.waitunlockf = <span class="hljs-literal">nil</span>_g_.m.waitlock = <span class="hljs-literal">nil</span><span class="hljs-keyword">if</span> !ok &#123;<span class="hljs-keyword">if</span> trace.enabled &#123;traceGoUnpark(gp, <span class="hljs-number">2</span>)&#125;casgstatus(gp, _Gwaiting, _Grunnable)execute(gp, <span class="hljs-literal">true</span>) <span class="hljs-comment">// Schedule it back, never returns.</span>&#125;&#125;schedule()&#125;</code></pre></div><ul><li>在goroutine等待的条件满足后(???哪里满足，位置???)，会调用<code>runtime.goready</code>将之前gopark进入<code>_Gwaiting</code>状态的goroutine唤醒；</li><li><code>runtime.ready</code>将goroutine状态从<code>_Gwaiting</code>或者<code>_Gscanwaiting</code>变为<code>_Grunnable</code>，并进入runq</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">goready</span><span class="hljs-params">(gp *g, traceskip <span class="hljs-keyword">int</span>)</span></span> &#123;systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;ready(gp, traceskip, <span class="hljs-literal">true</span>)&#125;)&#125;<span class="hljs-comment">// Mark gp ready to run.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">ready</span><span class="hljs-params">(gp *g, traceskip <span class="hljs-keyword">int</span>, next <span class="hljs-keyword">bool</span>)</span></span> &#123;<span class="hljs-keyword">if</span> trace.enabled &#123;traceGoUnpark(gp, traceskip)&#125;status := readgstatus(gp)<span class="hljs-comment">// Mark runnable.</span>_g_ := getg()mp := acquirem() <span class="hljs-comment">// disable preemption because it can be holding p in a local var</span><span class="hljs-keyword">if</span> status&amp;^_Gscan != _Gwaiting &#123;dumpgstatus(gp)throw(<span class="hljs-string">"bad g-&gt;status in ready"</span>)&#125;<span class="hljs-comment">// status is Gwaiting or Gscanwaiting, make Grunnable and put on runq</span>casgstatus(gp, _Gwaiting, _Grunnable)runqput(_g_.m.p.ptr(), gp, next)<span class="hljs-keyword">if</span> atomic.Load(&amp;sched.npidle) != <span class="hljs-number">0</span> &amp;&amp; atomic.Load(&amp;sched.nmspinning) == <span class="hljs-number">0</span> &#123;wakep()&#125;releasem(mp)&#125;</code></pre></div><h4 id="系统调用">系统调用</h4><p>这部分代码主要是汇编组成:</p><p><code>syscall.Syscall</code>同样是<code>linux amd64</code>:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//</span><span class="hljs-comment">// System call support for AMD64, Darwin</span><span class="hljs-comment">//</span><span class="hljs-comment">// Trap # in AX, args in DI SI DX, return in AX DX</span><span class="hljs-comment">// func Syscall(trap, a1, a2, a3 uintptr) (r1, r2 uintptr, err Errno);</span>TEXT·Syscall(SB),NOSPLIT,$<span class="hljs-number">0</span><span class="hljs-number">-56</span>CALLruntime·entersyscall(SB)MOVQa1+<span class="hljs-number">8</span>(FP), DIMOVQa2+<span class="hljs-number">16</span>(FP), SIMOVQa3+<span class="hljs-number">24</span>(FP), DXMOVQtrap+<span class="hljs-number">0</span>(FP), AX<span class="hljs-comment">// syscall entry</span>ADDQ$<span class="hljs-number">0x2000000</span>, AXSYSCALLJCCokMOVQ$<span class="hljs-number">-1</span>, r1+<span class="hljs-number">32</span>(FP)MOVQ$<span class="hljs-number">0</span>, r2+<span class="hljs-number">40</span>(FP)MOVQAX, err+<span class="hljs-number">48</span>(FP)CALLruntime·exitsyscall(SB)RETok:MOVQAX, r1+<span class="hljs-number">32</span>(FP)MOVQDX, r2+<span class="hljs-number">40</span>(FP)MOVQ$<span class="hljs-number">0</span>, err+<span class="hljs-number">48</span>(FP)CALLruntime·exitsyscall(SB)RET</code></pre></div><p>注意到<code>runtime.entersyscall</code>,实际就是提供syscall前保存pc、sp，以便恢复;然后调用<code>reentersyscall</code>:</p><ul><li>首先会先<code>m.lock++</code>，不让抢占（但是这里可能会造成goroutine状态<code>atomicstatus=Gsyscall</code>与<code>g.sched</code>调度器中的状态不一致，不可以让gc发现???如何???)，会造成内存不一致？？？</li><li>不可以调用任何会造成split stack的函数,因为其调用了<code>gosave</code>:其会使<code>g.sched</code>指向调用者的栈(如果split，就不知道指去哪里了)以便立即返回</li><li><code>save(pc, sp)</code>保存当前pc和sp</li><li>更新goroutine状态为<code>_Gsyscall</code></li><li>分离goroutine和P，并将P状态更新为<code>_Psyscall</code>，这时候会陷入syscall，要等待返回;</li><li>释放锁<code>m.lock--</code>,可能就有其他goroutine来抢P资源了</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Standard syscall entry used by the go syscall library and normal cgo calls.</span><span class="hljs-comment">//</span><span class="hljs-comment">// This is exported via linkname to assembly in the syscall package.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-comment">//go:linkname entersyscall</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">entersyscall</span><span class="hljs-params">()</span></span> &#123;reentersyscall(getcallerpc(), getcallersp())&#125;<span class="hljs-comment">// The goroutine g is about to enter a system call.</span><span class="hljs-comment">// Record that it's not using the cpu anymore.</span><span class="hljs-comment">// This is called only from the go syscall library and cgocall,</span><span class="hljs-comment">// not from the low-level system calls used by the runtime.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Entersyscall cannot split the stack: the gosave must</span><span class="hljs-comment">// make g-&gt;sched refer to the caller's stack segment, because</span><span class="hljs-comment">// entersyscall is going to return immediately after.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Nothing entersyscall calls can split the stack either.</span><span class="hljs-comment">// We cannot safely move the stack during an active call to syscall,</span><span class="hljs-comment">// because we do not know which of the uintptr arguments are</span><span class="hljs-comment">// really pointers (back into the stack).</span><span class="hljs-comment">// In practice, this means that we make the fast path run through</span><span class="hljs-comment">// entersyscall doing no-split things, and the slow path has to use systemstack</span><span class="hljs-comment">// to run bigger things on the system stack.</span><span class="hljs-comment">//</span><span class="hljs-comment">// reentersyscall is the entry point used by cgo callbacks, where explicitly</span><span class="hljs-comment">// saved SP and PC are restored. This is needed when exitsyscall will be called</span><span class="hljs-comment">// from a function further up in the call stack than the parent, as g-&gt;syscallsp</span><span class="hljs-comment">// must always point to a valid stack frame. entersyscall below is the normal</span><span class="hljs-comment">// entry point for syscalls, which obtains the SP and PC from the caller.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Syscall tracing:</span><span class="hljs-comment">// At the start of a syscall we emit traceGoSysCall to capture the stack trace.</span><span class="hljs-comment">// If the syscall does not block, that is it, we do not emit any other events.</span><span class="hljs-comment">// If the syscall blocks (that is, P is retaken), retaker emits traceGoSysBlock;</span><span class="hljs-comment">// when syscall returns we emit traceGoSysExit and when the goroutine starts running</span><span class="hljs-comment">// (potentially instantly, if exitsyscallfast returns true) we emit traceGoStart.</span><span class="hljs-comment">// To ensure that traceGoSysExit is emitted strictly after traceGoSysBlock,</span><span class="hljs-comment">// we remember current value of syscalltick in m (_g_.m.syscalltick = _g_.m.p.ptr().syscalltick),</span><span class="hljs-comment">// whoever emits traceGoSysBlock increments p.syscalltick afterwards;</span><span class="hljs-comment">// and we wait for the increment before emitting traceGoSysExit.</span><span class="hljs-comment">// Note that the increment is done even if tracing is not enabled,</span><span class="hljs-comment">// because tracing can be enabled in the middle of syscall. We don't want the wait to hang.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">reentersyscall</span><span class="hljs-params">(pc, sp <span class="hljs-keyword">uintptr</span>)</span></span> &#123;_g_ := getg()<span class="hljs-comment">// Disable preemption because during this function g is in Gsyscall status,</span><span class="hljs-comment">// but can have inconsistent g-&gt;sched, do not let GC observe it.</span>_g_.m.locks++<span class="hljs-comment">// Entersyscall must not call any function that might split/grow the stack.</span><span class="hljs-comment">// (See details in comment above.)</span><span class="hljs-comment">// Catch calls that might, by replacing the stack guard with something that</span><span class="hljs-comment">// will trip any stack check and leaving a flag to tell newstack to die.</span>_g_.stackguard0 = stackPreempt_g_.throwsplit = <span class="hljs-literal">true</span><span class="hljs-comment">// Leave SP around for GC and traceback.</span>save(pc, sp)_g_.syscallsp = sp_g_.syscallpc = pccasgstatus(_g_, _Grunning, _Gsyscall)<span class="hljs-keyword">if</span> _g_.syscallsp &lt; _g_.stack.lo || _g_.stack.hi &lt; _g_.syscallsp &#123;systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"entersyscall inconsistent "</span>, hex(_g_.syscallsp), <span class="hljs-string">" ["</span>, hex(_g_.stack.lo), <span class="hljs-string">","</span>, hex(_g_.stack.hi), <span class="hljs-string">"]\n"</span>)throw(<span class="hljs-string">"entersyscall"</span>)&#125;)&#125;<span class="hljs-keyword">if</span> trace.enabled &#123;systemstack(traceGoSysCall)<span class="hljs-comment">// systemstack itself clobbers g.sched.&#123;pc,sp&#125; and we might</span><span class="hljs-comment">// need them later when the G is genuinely blocked in a</span><span class="hljs-comment">// syscall</span>save(pc, sp)&#125;<span class="hljs-keyword">if</span> atomic.Load(&amp;sched.sysmonwait) != <span class="hljs-number">0</span> &#123;systemstack(entersyscall_sysmon)save(pc, sp)&#125;<span class="hljs-keyword">if</span> _g_.m.p.ptr().runSafePointFn != <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// runSafePointFn may stack split if run on this stack</span>systemstack(runSafePointFn)save(pc, sp)&#125;_g_.m.syscalltick = _g_.m.p.ptr().syscalltick_g_.sysblocktraced = <span class="hljs-literal">true</span>_g_.m.mcache = <span class="hljs-literal">nil</span>pp := _g_.m.p.ptr()pp.m = <span class="hljs-number">0</span>_g_.m.oldp.set(pp)_g_.m.p = <span class="hljs-number">0</span>atomic.Store(&amp;pp.status, _Psyscall)<span class="hljs-keyword">if</span> sched.gcwaiting != <span class="hljs-number">0</span> &#123;systemstack(entersyscall_gcwait)save(pc, sp)&#125;_g_.m.locks--&#125;</code></pre></div><p><code>runtime.exitsyscall</code>则从syscall中恢复,比较复杂:</p><ul><li>同样要锁住<code>m.locks++</code>，不给抢占先,</li><li>写屏障不能用，p可能在syscall过程中被窃取(即当前p不是syscall前的p)</li><li>会走到两个路径<code>exitsyscallfast</code>或者通过<code>mcall</code>(切换到调度器的goroutine)调用<code>exitsyscall0</code>进行退出</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// The goroutine g exited its system call.</span><span class="hljs-comment">// Arrange for it to run on a cpu again.</span><span class="hljs-comment">// This is called only from the go syscall library, not</span><span class="hljs-comment">// from the low-level system calls used by the runtime.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Write barriers are not allowed because our P may have been stolen.</span><span class="hljs-comment">//</span><span class="hljs-comment">// This is exported via linkname to assembly in the syscall package.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:nosplit</span><span class="hljs-comment">//go:nowritebarrierrec</span><span class="hljs-comment">//go:linkname exitsyscall</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">exitsyscall</span><span class="hljs-params">()</span></span> &#123;_g_ := getg()_g_.m.locks++ <span class="hljs-comment">// see comment in entersyscall</span><span class="hljs-keyword">if</span> getcallersp() &gt; _g_.syscallsp &#123;throw(<span class="hljs-string">"exitsyscall: syscall frame is no longer valid"</span>)&#125;_g_.waitsince = <span class="hljs-number">0</span>oldp := _g_.m.oldp.ptr()_g_.m.oldp = <span class="hljs-number">0</span><span class="hljs-keyword">if</span> exitsyscallfast(oldp) &#123;<span class="hljs-keyword">if</span> _g_.m.mcache == <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"lost mcache"</span>)&#125;<span class="hljs-keyword">if</span> trace.enabled &#123;<span class="hljs-keyword">if</span> oldp != _g_.m.p.ptr() || _g_.m.syscalltick != _g_.m.p.ptr().syscalltick &#123;systemstack(traceGoStart)&#125;&#125;<span class="hljs-comment">// There's a cpu for us, so we can run.</span>_g_.m.p.ptr().syscalltick++<span class="hljs-comment">// We need to cas the status and scan before resuming...</span>casgstatus(_g_, _Gsyscall, _Grunning)<span class="hljs-comment">// Garbage collector isn't running (since we are),</span><span class="hljs-comment">// so okay to clear syscallsp.</span>_g_.syscallsp = <span class="hljs-number">0</span>_g_.m.locks--<span class="hljs-keyword">if</span> _g_.preempt &#123;<span class="hljs-comment">// restore the preemption request in case we've cleared it in newstack</span>_g_.stackguard0 = stackPreempt&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">// otherwise restore the real _StackGuard, we've spoiled it in entersyscall/entersyscallblock</span>_g_.stackguard0 = _g_.stack.lo + _StackGuard&#125;_g_.throwsplit = <span class="hljs-literal">false</span><span class="hljs-keyword">if</span> sched.disable.user &amp;&amp; !schedEnabled(_g_) &#123;<span class="hljs-comment">// Scheduling of this goroutine is disabled.</span>Gosched()&#125;<span class="hljs-keyword">return</span>&#125;_g_.sysexitticks = <span class="hljs-number">0</span><span class="hljs-keyword">if</span> trace.enabled &#123;<span class="hljs-comment">// Wait till traceGoSysBlock event is emitted.</span><span class="hljs-comment">// This ensures consistency of the trace (the goroutine is started after it is blocked).</span><span class="hljs-keyword">for</span> oldp != <span class="hljs-literal">nil</span> &amp;&amp; oldp.syscalltick == _g_.m.syscalltick &#123;osyield()&#125;<span class="hljs-comment">// We can't trace syscall exit right now because we don't have a P.</span><span class="hljs-comment">// Tracing code can invoke write barriers that cannot run without a P.</span><span class="hljs-comment">// So instead we remember the syscall exit time and emit the event</span><span class="hljs-comment">// in execute when we have a P.</span>_g_.sysexitticks = cputicks()&#125;_g_.m.locks--<span class="hljs-comment">// Call the scheduler.</span>mcall(exitsyscall0)<span class="hljs-keyword">if</span> _g_.m.mcache == <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"lost mcache"</span>)&#125;<span class="hljs-comment">// Scheduler returned, so we're allowed to run now.</span><span class="hljs-comment">// Delete the syscallsp information that we left for</span><span class="hljs-comment">// the garbage collector during the system call.</span><span class="hljs-comment">// Must wait until now because until gosched returns</span><span class="hljs-comment">// we don't know for sure that the garbage collector</span><span class="hljs-comment">// is not running.</span>_g_.syscallsp = <span class="hljs-number">0</span>_g_.m.p.ptr().syscalltick++_g_.throwsplit = <span class="hljs-literal">false</span>&#125;</code></pre></div><p>针对<code>exitsyscallfast</code>:</p><ul><li>如果goroutine处于<code>_Psyscall</code>状态，尝试用<code>wirep</code>将其与goroutine与之前旧的<code>g.m.oldP</code>(就是syscall之前会保存进来的)相连接;</li><li>如果全局调度器中有其他的空闲p，就会在<code>systemstack</code>下调用<code>exitsyscallfast_pidle</code>来获取p，其中<code>exitsyscallfast_pidle</code>方法会使用<code>acquirep</code>调用空闲的p来接管当前goroutine</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//go:nosplit</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">exitsyscallfast</span><span class="hljs-params">(oldp *p)</span> <span class="hljs-title">bool</span></span> &#123;_g_ := getg()<span class="hljs-comment">// Freezetheworld sets stopwait but does not retake P's.</span><span class="hljs-keyword">if</span> sched.stopwait == freezeStopWait &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;<span class="hljs-comment">// Try to re-acquire the last P.</span><span class="hljs-keyword">if</span> oldp != <span class="hljs-literal">nil</span> &amp;&amp; oldp.status == _Psyscall &amp;&amp; atomic.Cas(&amp;oldp.status, _Psyscall, _Pidle) &#123;<span class="hljs-comment">// There's a cpu for us, so we can run.</span>wirep(oldp)exitsyscallfast_reacquired()<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>&#125;<span class="hljs-comment">// Try to get any other idle P.</span><span class="hljs-keyword">if</span> sched.pidle != <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">var</span> ok <span class="hljs-keyword">bool</span>systemstack(<span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;ok = exitsyscallfast_pidle()<span class="hljs-keyword">if</span> ok &amp;&amp; trace.enabled &#123;<span class="hljs-keyword">if</span> oldp != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// Wait till traceGoSysBlock event is emitted.</span><span class="hljs-comment">// This ensures consistency of the trace (the goroutine is started after it is blocked).</span><span class="hljs-keyword">for</span> oldp.syscalltick == _g_.m.syscalltick &#123;osyield()&#125;&#125;traceGoSysExit(<span class="hljs-number">0</span>)&#125;&#125;)<span class="hljs-keyword">if</span> ok &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>&#125;&#125;<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;</code></pre></div><p>另一个<code>exitsyscall0</code>（较慢）:</p><ul><li>首先会将<code>Gsyscall</code>状态设为<code>Grunnable</code></li><li>然后<code>dropg</code>断开g与P的联系;</li><li>然后锁住全局调度器,<code>pidleget()</code>获得空闲p，然这个p接管goroutine</li><li>如果<code>pidleget()</code>无法获得p，则将当前g放入全局的<code>sched.runq</code>,等待调度器;</li><li>解锁，下面就是一系列调度;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// exitsyscall slow path on g0.</span><span class="hljs-comment">// Failed to acquire P, enqueue gp as runnable.</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:nowritebarrierrec</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">exitsyscall0</span><span class="hljs-params">(gp *g)</span></span> &#123;_g_ := getg()casgstatus(gp, _Gsyscall, _Grunnable)dropg()lock(&amp;sched.lock)<span class="hljs-keyword">var</span> _p_ *p<span class="hljs-keyword">if</span> schedEnabled(_g_) &#123;_p_ = pidleget()&#125;<span class="hljs-keyword">if</span> _p_ == <span class="hljs-literal">nil</span> &#123;globrunqput(gp)&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> atomic.Load(&amp;sched.sysmonwait) != <span class="hljs-number">0</span> &#123;atomic.Store(&amp;sched.sysmonwait, <span class="hljs-number">0</span>)notewakeup(&amp;sched.sysmonnote)&#125;unlock(&amp;sched.lock)<span class="hljs-keyword">if</span> _p_ != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">//将p与当前m联系起来</span>acquirep(_p_)<span class="hljs-comment">//开始在当前M开始运行传入的gp(goroutine)</span>execute(gp, <span class="hljs-literal">false</span>) <span class="hljs-comment">// Never returns.</span>&#125;<span class="hljs-keyword">if</span> _g_.m.lockedg != <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// Wait until another thread schedules gp and so m again.</span>stoplockedm()execute(gp, <span class="hljs-literal">false</span>) <span class="hljs-comment">// Never returns.</span>&#125;stopm()schedule() <span class="hljs-comment">// Never returns.</span>&#125;</code></pre></div><h4 id="协作式调度">协作式调度</h4><p>协作式调度主要是依靠<code>runtime.GoSched()</code>主动让出P，但该函数无法挂起goroutine,调度器会自动调度当前的goroutine???</p><ul><li>会更新当前g状态<code>Grunning</code>到<code>Grunnable</code></li><li>断开g与当前P的状态</li><li>将当前g放入全局<code>sched.runq</code></li><li>开始调度</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//go:nosplit</span><span class="hljs-comment">// Gosched yields the processor, allowing other goroutines to run. It does not</span><span class="hljs-comment">// suspend the current goroutine, so execution resumes automatically.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">Gosched</span><span class="hljs-params">()</span></span> &#123;checkTimeouts()mcall(gosched_m)&#125;<span class="hljs-comment">// Gosched continuation on g0.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">gosched_m</span><span class="hljs-params">(gp *g)</span></span> &#123;<span class="hljs-keyword">if</span> trace.enabled &#123;traceGoSched()&#125;goschedImpl(gp)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">goschedImpl</span><span class="hljs-params">(gp *g)</span></span> &#123;status := readgstatus(gp)<span class="hljs-keyword">if</span> status&amp;^_Gscan != _Grunning &#123;dumpgstatus(gp)throw(<span class="hljs-string">"bad g status"</span>)&#125;casgstatus(gp, _Grunning, _Grunnable)dropg()lock(&amp;sched.lock)globrunqput(gp)unlock(&amp;sched.lock)schedule()&#125;</code></pre></div><h4 id="sysmon-v2">sysmon</h4><p>主要对运行时间过长的，强行让出<code>p</code>中的<code>schedtick</code> 和 <code>schedwhen</code>与当前时间计算，运算出是否超时，要让出<code>runtime.retake</code>方法中的<code>runtime.preemptone</code>进行异步抢占:</p><ul><li><code>retake()</code>方法会先锁住全局<code>allp</code>，注意这里一定要在stw条件下，否则不成功</li></ul><ul><li>注意在<code>syscall</code>中,<code>preemptone()</code>无法工作，因为此时，无P与M关联(syscall会让出p)</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">sysmon</span><span class="hljs-params">()</span></span> &#123;    ...        <span class="hljs-comment">// retake P's blocked in syscalls</span>        <span class="hljs-comment">// and preempt long running G's</span>        <span class="hljs-keyword">if</span> retake(now) != <span class="hljs-number">0</span> &#123;            idle = <span class="hljs-number">0</span>        &#125; <span class="hljs-keyword">else</span> &#123;            idle++        &#125;    ...&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">retake</span><span class="hljs-params">(now <span class="hljs-keyword">int64</span>)</span> <span class="hljs-title">uint32</span></span> &#123;<span class="hljs-comment">//锁住全局allp，不让改变</span>lock(&amp;allpLock)    <span class="hljs-keyword">for</span> i := <span class="hljs-number">0</span>; i &lt; <span class="hljs-built_in">len</span>(allp); i++ &#123;    ...        <span class="hljs-keyword">if</span> s == _Prunning || s == _Psyscall &#123;            <span class="hljs-comment">// Preempt G if it's running for too long.</span>            t := <span class="hljs-keyword">int64</span>(_p_.schedtick)            <span class="hljs-comment">//G对应的schedtick跟监控的不一致，则需要重新更新一些sched的数值</span>            <span class="hljs-keyword">if</span> <span class="hljs-keyword">int64</span>(pd.schedtick) != t &#123;                pd.schedtick = <span class="hljs-keyword">uint32</span>(t)                pd.schedwhen = now            &#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> pd.schedwhen+forcePreemptNS &lt;= now &#123;<span class="hljs-comment">//forcePreemptNS=10ms</span>                <span class="hljs-comment">// 如果超过了10ms就需要进行抢占了</span>                preemptone(_p_)                <span class="hljs-comment">// In case of syscall, preemptone() doesn't</span>                <span class="hljs-comment">// work, because there is no M wired to P.</span>                sysretake = <span class="hljs-literal">true</span>            &#125;        &#125;    ...    &#125;&#125;</code></pre></div><p>在符合条件下会调用<code>preemptone()</code>:</p><ul><li>该方法是尽力模型，有可能错误地通知goroutine；</li><li>即使它正确通知了goroutine，goroutine也可能会忽视请求如果该goroutine同时在执行<code>newstack</code>函数(<code>runtime·morestack</code>会call这个函数，用于栈扩容)</li><li><code>不需要锁</code>（上面allp已经锁住）</li><li>真正的抢占会发生在未来的某个时间点，当<code>gp.status!=Grunnning</code>时会被标记出</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Tell the goroutine running on processor P to stop.</span><span class="hljs-comment">// This function is purely best-effort. It can incorrectly fail to inform the</span><span class="hljs-comment">// goroutine. It can send inform the wrong goroutine. Even if it informs the</span><span class="hljs-comment">// correct goroutine, that goroutine might ignore the request if it is</span><span class="hljs-comment">// simultaneously executing newstack.</span><span class="hljs-comment">// No lock needs to be held.</span><span class="hljs-comment">// Returns true if preemption request was issued.</span><span class="hljs-comment">// The actual preemption will happen at some point in the future</span><span class="hljs-comment">// and will be indicated by the gp-&gt;status no longer being</span><span class="hljs-comment">// Grunning</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">preemptone</span><span class="hljs-params">(_p_ *p)</span> <span class="hljs-title">bool</span></span> &#123;mp := _p_.m.ptr()<span class="hljs-keyword">if</span> mp == <span class="hljs-literal">nil</span> || mp == getg().m &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;gp := mp.curg<span class="hljs-keyword">if</span> gp == <span class="hljs-literal">nil</span> || gp == mp.g0 &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;<span class="hljs-comment">//这个同stackguard0 = stackPreempt属性其实一样</span>gp.preempt = <span class="hljs-literal">true</span><span class="hljs-comment">// Every call in a go routine checks for stack overflow by</span><span class="hljs-comment">// comparing the current stack pointer to gp-&gt;stackguard0.</span><span class="hljs-comment">// Setting gp-&gt;stackguard0 to StackPreempt folds</span><span class="hljs-comment">// preemption into the normal stack overflow check.</span><span class="hljs-comment">//在一个goroutine中检查栈溢出都是靠对比 现在栈指针 和 gp.stackguard0, 设置gp.stackguard0=stackPreempt 就是等于 将抢占放入一般的栈溢出检查中？？？？;</span>gp.stackguard0 = stackPreempt<span class="hljs-comment">// Request an async preemption of this P.</span><span class="hljs-keyword">if</span> preemptMSupported &amp;&amp; debug.asyncpreemptoff == <span class="hljs-number">0</span> &#123;_p_.preempt = <span class="hljs-literal">true</span>preemptM(mp)&#125;<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>&#125;</code></pre></div><ol><li>Channel,mutex之类同步操作发生阻塞</li><li>time.sleep</li><li>主动调用runtime.GoSched()</li><li>网络IO阻塞</li><li>gc</li><li>运行过久或者系统调用过久</li></ol><h3 id="os线程锁">OS线程锁</h3><p><code>runtime.LockOSThread</code> //todo</p><h3 id="死锁检测和终止">死锁检测和终止</h3><p>//todo???当所有P是idle的时候进行检测（全局idle P的原子计数）</p><p>旋转态-&gt;不旋态的转换中，可能和创建一个新的goroutine和创建一部分或其他需要unpark的工作线程 的时候发生竞态条件如果转换和创建都失败，我们就可以以半静态cpu未充分利用结束；goroutine 准备步骤是：提交一个goroutine去local queue，store-style memory 屏障，检查sched.nmspinning</p><p>不旋态-&gt;旋转态是： 减少nmspinning，store-style memory 屏障，检查新的work的所有per-P work queue而且以上都不适用于global run queue</p><h3 id="协作式抢占">协作式抢占</h3><p><code>retake()</code> 调用<code>runtime.preemptone()</code>将被抢占的G的<code>stackguard0</code> 设为<code>stackPreempt</code>，被设置标志的G下一次进行函数调用的时候，检查栈空间失败。然后会触发morestack() (汇编代码,asm_xxx.s)然后进行一连串的函数调用大概流程</p><p>morestack()–&gt; newstack()–&gt; gopreempt_m() --&gt; goschedImpl() --&gt; schedule()</p><h2 id="补充：">补充：</h2><p>网上的经验（为什么呢？？？)：</p><p>这个goroutine类似于线程池管理(c++线程池原理相似)，</p><ol><li>遇到阻塞的情况，怎么扩展进程池，使其不会因为任务阻塞或者同步独占线程</li></ol><ol start="2"><li><p>goroutine类似green threads（Green threads），是application自己维护的执行过程；很多goroutines实际上被有限个操作系统管理的threads执行;</p></li><li><p>goroutine的调度往往发生在I/O和系统调用的时候。如果创建的goroutines都是跑for循环做纯计算（没有I/O），那就需要我们自己时不常的调用 runtime.Gosched()，否则那几个在thread上跑的goroutines会霸占着threads，不让其他goroutines有机会跑起来;</p></li><li><p>用户代码造成的协程同步造成的阻塞，只是切换(gopark)协程，而不是阻塞线程，<strong>m和p仍结合</strong>，去寻找新的可执行的g;</p></li><li><p>上层封装了epoll，网络fd会设置成NonBlocking模式，返回EAGAIN则gopark当前goroutine，在m调度，sysmon中，gc start the world等阶段均会poll出ready的goroutine进行运行或者添加到全局runq中</p></li></ol><p><strong>一些小细节</strong>代码经常发现一些编辑器生成的//go:nosplit字样</p><blockquote><blockquote><p>The //go:nosplit directive specifies that the next function declared in the file must not include a stack overflow check. This is most commonly used by low-level runtime sources invoked at times when it is unsafe for the calling goroutine to be preempted.</p></blockquote></blockquote><p>大意即为这个生成函数不能含有检查栈溢出的代码，即会跳过栈溢出检查（why???个人认为是设计问题，就不允许有检查栈移除代码），有时goroutine要被抢占陷入不安全情况时，被底层runtime调用</p><h2 id="systemstack">SystemStack</h2><p>SystemStack(fn func())系统栈 被不同地方调用会有不同的表现方式：</p><ul><li><p>直接调用fn并返回 需要满足：</p><ul><li><p>被 单个线程的g0 stack调用    或</p></li><li><p>被信号处理的栈(gsignal)调用,m中有个gsinal字段 ???</p></li></ul></li><li><p>否则，都从一个普通的goroutine的有限的stack中调用</p><p>表现： 会先切去线程的栈，调用fn，然后切回来该goroutine的栈</p></li></ul><div class="hljs"><pre><code class="hljs golang"><span class="hljs-comment">// systemstack runs fn on a system stack.</span><span class="hljs-comment">// If systemstack is called from the per-OS-thread (g0) stack, or</span><span class="hljs-comment">// if systemstack is called from the signal handling (gsignal) stack,</span><span class="hljs-comment">// systemstack calls fn directly and returns.</span><span class="hljs-comment">// Otherwise, systemstack is being called from the limited stack</span><span class="hljs-comment">// of an ordinary goroutine. In this case, systemstack switches</span><span class="hljs-comment">// to the per-OS-thread stack, calls fn, and switches back.</span><span class="hljs-comment">// It is common to use a func literal as the argument, in order</span><span class="hljs-comment">// to share inputs and outputs with the code around the call</span><span class="hljs-comment">// to system stack:</span><span class="hljs-comment">//</span><span class="hljs-comment">//... set up y ...</span><span class="hljs-comment">//systemstack(func() &#123;</span><span class="hljs-comment">//x = bigcall(y)</span><span class="hljs-comment">//&#125;)</span><span class="hljs-comment">//... use x ...</span><span class="hljs-comment">//</span><span class="hljs-comment">//go:noescape</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">systemstack</span><span class="hljs-params">(fn <span class="hljs-keyword">func</span>()</span>)</span></code></pre></div><h3 id="一个大概的go程序启动流程">一个大概的go程序启动流程</h3><p>golang注释中有大概写明:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// The bootstrap sequence is:</span><span class="hljs-comment">//</span><span class="hljs-comment">//call osinit</span><span class="hljs-comment">//call schedinit</span><span class="hljs-comment">//make &amp; queue new G</span><span class="hljs-comment">//call runtime·mstart</span><span class="hljs-comment">// The new G calls runtime·main.</span></code></pre></div><p><img src="/img/" srcset="/img/loading.gif" alt="大概的流程图"></p><p>go程序的入口点是runtime.rt0_go, 流程是:</p><ol><li>分配栈空间, 需要2个本地变量+2个函数参数, 然后向8对齐</li></ol><p>把传入的argc和argv保存到栈上(rdx寄存器通常用作上下文存储)</p><p>更新g0中的stackguard的值, stackguard用于检测栈空间是否不足, 需要分配新的栈空间(栈扩展会申请多一块栈空间并把现在的复制过去)</p><p>获取当前cpu的信息并保存到各个全局变量</p><p>调用_cgo_init如果函数存在</p><ol start="2"><li>初始化当前线程的TLS(thread-local-storage), 设置FS寄存器为m0.tls+8(获取时会-8)这里跟SP寄存器有关(伪的SP寄存器的地址 = 硬件SP寄存器+8，64位机)</li></ol><p>测试TLS是否工作</p><p>设置g0到TLS中, 表示当前的g是g0</p><p>设置m0.g0 = g0</p><p>设置g0.m = m0</p><h4 id="特殊的m0和g0">特殊的m0和g0</h4><ul><li><p><code>M0</code> 是启动程序后的编号为 0 的<code>主线程</code>，这个 M 对应的实例会在全局变量<code>runtime.m0</code>中，不需要在heap上分配，<code>M0</code>负责执行初始化操作和启动第一个 G， 在之后 M0 就和其他的 M 一样了。</p></li><li><p><code>G0</code> 是每次启动一个 M 都会第一个创建的 gourtine，<code>G0</code> 仅用于<strong>负责调度</strong>的 G(作用);<code>G0</code> 不指向任何可执行的函数，每个 M 都会有一个自己的 <code>G0</code>。在调度或系统调用时会使用 G0 的栈空间(栈空间是一定的,Unix一般是8MB)，全局变量的 <code>G0</code> 是 <code>M0</code> 的 <code>G0</code> (这种一般指sysmon,垃圾回收器等，注意调度器本身属于第三种goroutine，不是g0，也不是普通的goroutine)</p></li></ul><ol start="3"><li>调用runtime.check做一些检查</li></ol><p>调用runtime.args保存传入的argc和argv到全局变量</p><p>调用runtime.osinit根据系统执行不同的初始化</p><p>这里(linux x64)设置了全局变量ncpu等于cpu核心数量</p><ol start="4"><li>调用**runtime.schedinit()**执行共同的初始化</li></ol><p>这里的处理比较多:</p><ul><li><p>首先会调用raceinit()检查race condition</p></li><li><p>然后进接这tracebackinit()和moduledateverify()，分别为一些变量提前初始化和包的验证</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">tracebackinit</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-comment">// Go variable initialization happens late during runtime startup.</span><span class="hljs-comment">// Instead of initializing the variables above in the declarations,</span><span class="hljs-comment">// schedinit calls this function so that the variables are</span><span class="hljs-comment">// initialized and available earlier in the startup sequence.</span>skipPC = funcPC(skipPleaseUseCallersFrames)&#125;</code></pre></div><ul><li>会初始化栈空间分配器(stackinit)</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">stackinit</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-comment">//// Per-P, per order stack segment cache size.</span><span class="hljs-comment">//_StackCacheSize = 32 * 1024</span><span class="hljs-comment">// stack的分段大小一定要是pagesize的倍数（容易理解，方便对齐）</span><span class="hljs-comment">//_PageShift = 13</span><span class="hljs-comment">//_PageSize = 1 &lt;&lt; _PageShift = 8192</span><span class="hljs-comment">//_PageMask = _PageSize - 1</span><span class="hljs-keyword">if</span> _StackCacheSize&amp;_PageMask != <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"cache size must be a multiple of page size"</span>)&#125;<span class="hljs-comment">//stackpool就是一个span的双向链表</span><span class="hljs-keyword">for</span> i := <span class="hljs-keyword">range</span> stackpool &#123;stackpool[i].init()&#125;<span class="hljs-comment">// stackLarge的free是一个list ， 大小为 log_2(s.npages)</span><span class="hljs-keyword">for</span> i := <span class="hljs-keyword">range</span> stackLarge.free &#123;stackLarge.free[i].init()&#125;&#125;</code></pre></div><p>这里插入一副<s>盗</s>借来的<img src="/img/mheap.png" srcset="/img/loading.gif" alt="图">更加明确发现一些奇怪的特点</p><ul><li><p>mallocinit()</p><ol><li>这个最主要是检查page，huge page大小是不是2的倍数以及是不是大于最小页大小(4KB)</li><li>然后就初始化 heap，在memManage 那篇文章有讲到,会初始化多个fixalloc，包括treap，span，cache，specialfinalizer，specialprofile，arenaHint还有getg()获得当前g的指针，以及初始化当前mcache(allocmcache())</li><li>创建初始化的arena区域(即是heap)的增长规则,注意在64bit机器中，其做了一些优化:从中间空间开始分配,如上面的图一样，<ul><li>可以更加容易地增长连续空间</li><li>使其更加容易debug</li><li>为了gccgo区别于其他数据</li><li>UTF8编码</li></ul></li></ol></li></ul><ul><li><p>mcommoninit(<em>g</em>.m),这里是一些公共初始化主要对_g_.m即自己的m进行一些初始化</p></li><li><p>按cpu核心数量或GOMAXPROCS的值生成P(cpuinit)</p></li></ul><div class="hljs"><pre><code class="hljs go">cpuinit()       <span class="hljs-comment">// must run before alginit</span></code></pre></div><ul><li>alginit</li></ul><div class="hljs"><pre><code class="hljs go">alginit()       <span class="hljs-comment">// maps must not be used before this call</span></code></pre></div><ul><li>生成P的处理在procresize中</li></ul><p>更改了P的数目，期间stop the world并锁住sched，返回本地的所有p</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">schedinit</span><span class="hljs-params">()</span></span>&#123;...sched.lastpoll = <span class="hljs-keyword">uint64</span>(nanotime())procs := ncpu<span class="hljs-keyword">if</span> n, ok := atoi32(gogetenv(<span class="hljs-string">"GOMAXPROCS"</span>)); ok &amp;&amp; n &gt; <span class="hljs-number">0</span> &#123;procs = n&#125;<span class="hljs-keyword">if</span> procresize(procs) != <span class="hljs-literal">nil</span> &#123;throw(<span class="hljs-string">"unknown runnable goroutine during bootstrap"</span>)&#125;...&#125;<span class="hljs-comment">// Change number of processors. The world is stopped, sched is locked.</span><span class="hljs-comment">// gcworkbufs are not being modified by either the GC or</span><span class="hljs-comment">// the write barrier code.</span><span class="hljs-comment">// Returns list of Ps with local work, they need to be scheduled by the caller.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">procresize</span><span class="hljs-params">(nprocs <span class="hljs-keyword">int32</span>)</span> *<span class="hljs-title">p</span></span> &#123; ... &#125;</code></pre></div><ol start="5"><li><p>调用runtime.newproc创建一个新的goroutine, 指向的是runtime.mainruntime.newproc这个函数在创建普通的goroutine时也会使用;</p></li><li><p>调用runtime·mstart启动m0</p><ul><li>启动后m0会不断从运行队列获取G并运行, runtime.mstart调用后不会返回</li><li>runtime.mstart这个函数是m的入口点(不仅仅是m0), 在下面的&quot;调度器的实现&quot;中会详细讲解</li></ul></li></ol><h3 id="runtime-main之后">runtime.main之后</h3><p>第一个被调度的G会运行runtime.main, 流程是:</p><p>标记主函数已调用, 设置mainStarted = true</p><p>启动一个新的M执行sysmon函数, 这个函数会监控全局的状态并对运行时间过长的G进行抢占</p><p>要求G必须在当前M(系统主线程)上执行</p><p>调用runtime_init函数</p><p>调用gcenable函数</p><p>调用main.init函数, 如果函数存在</p><p>不再要求G必须在当前M上运行</p><p>如果程序是作为c的类库编译的, 在这里返回</p><p>调用main.main函数</p><p>如果当前发生了panic, 则等待panic处理</p><p>调用exit(0)退出程序</p><h2 id="defer函数">Defer函数</h2><p>平常用的</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">do</span><span class="hljs-params">()</span></span>&#123;<span class="hljs-keyword">defer</span> done()&#125;</code></pre></div><p>其结构在 runtime2.go 结构体g中</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> g <span class="hljs-keyword">struct</span>&#123;goid <span class="hljs-keyword">int64</span>...<span class="hljs-comment">//其结构有些在stack中有些在heap中，但是逻辑上都属于stack，所以写屏障是没有必要的;</span>_defer *<span class="hljs-keyword">defer</span>&#123;siz     <span class="hljs-keyword">int32</span> <span class="hljs-comment">// includes both arguments and results</span>started <span class="hljs-keyword">bool</span>heap    <span class="hljs-keyword">bool</span>sp      <span class="hljs-keyword">uintptr</span> <span class="hljs-comment">// sp at time of defer</span>pc      <span class="hljs-keyword">uintptr</span>fn      *funcval <span class="hljs-comment">//调用的函数</span>_panic  *_panic <span class="hljs-comment">// panic that is running defer</span>link    *_defer&#125; ...&#125;</code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Notes about MySQL indexing</title>
    <link href="/2019/07/03/MySQL/MySQLIndexing/"/>
    <url>/2019/07/03/MySQL/MySQLIndexing/</url>
    
    <content type="html"><![CDATA[<p>一些注意事项</p><a id="more"></a><h2 id="主键的问题">主键的问题</h2><h3 id="复合主键和单个主键">复合主键和单个主键</h3><h4 id="复合主键">复合主键</h4><p>好处显而易见，可以依据多个列进行排序，但明显地，字符类的列相比int会比较复杂</p><h3 id="自增主键和自定义主键">自增主键和自定义主键</h3><h4 id="自增主键">自增主键</h4><p>好处：</p><ol><li>由系统生成，顺序递增，速度肯定快</li><li>int占用空间小，易排序</li></ol><p>缺点：</p><ol><li>自增主键可能不连续</li><li>水平分片架构会出问题，全局不能保证唯一</li></ol><h4 id="自定义主键">自定义主键</h4><p>好处：</p><ol><li>水平分片</li></ol><p>缺点：</p><ol><li>随机I/O，影响查找等操作</li></ol><h2 id="聚簇索引">聚簇索引</h2><ul><li><p>当表有聚簇索引的时候，它的数据行实际上存放在索引的叶子页（leaf page）上聚簇代表了数据行和相邻的键值紧凑地存储在一起</p></li><li><p>InnoDB使用聚集索引，数据记录本身被存于主索引（一颗B+Tree）的叶子节点上。这就要求同一个叶子节点内（大小为一个内存页或磁盘页）的各条数据记录按主键顺序存放，因此每当有一条新的记录插入时，MySQL会根据其主键将其插入适当的节点和位置，如果页面达到装载因子（InnoDB默认为15/16），则开辟一个新的页（节点）。</p></li></ul><p>如果表使用自增主键，那么每次插入新的记录，记录就会顺序添加到当前索引节点的后续位置，当一页写满，就会自动开辟一个新的页。这样就会形成一个紧凑的索引结构，近似顺序填满。由于每次插入时也不需要移动已有数据，因此效率很高，也不会增加很多开销在维护索引上。</p><h2 id="大数据问题">大数据问题</h2><h3 id="1-一次性插入大量数据">1. 一次性插入大量数据</h3><p>MySQL 5.7 Refman官方文件给出的提示：8.2.4.1 Optimizing INSERT operation8.5.5 Bulk data loading in INNODB</p><blockquote><blockquote><p>To optimize insert speed, combine many small operations into a single large operation. Ideally, you make a single connection, send the data for many new rows at once, and delay all index updates and consistency checking until the very end.</p></blockquote></blockquote><blockquote><blockquote><p>If you are inserting many rows from the same client at the same time, use INSERT statements with multiple VALUES lists to insert several rows at a time. This is considerably faster (many times faster in some cases) than using separate single-row INSERT statements. If you are adding data to a nonempty table, you can tune the bulk_insert_buffer_size variable to make data insertion even faster. See Section 5.1.7, “Server System Variables”.</p></blockquote></blockquote><blockquote><blockquote><p>When loading a table from a text file, use LOAD DATA. This is usually 20 times faster than using INSERT statements. See Section 13.2.6, “LOAD DATA Syntax”.</p></blockquote></blockquote><blockquote><blockquote><p>Take advantage of the fact that columns have default values. Insert values explicitly only when the value to be inserted differs from the default. This reduces the parsing that MySQL must do and improves the insert speed.</p></blockquote></blockquote><h4 id="如果是从一个客户端同时插入多行">如果是从一个客户端同时插入多行</h4><ol><li>就是要合并多个插入操作成一个，如：</li></ol><div class="hljs"><pre><code class="hljs sql"><span class="hljs-keyword">insert</span> <span class="hljs-keyword">into</span> table1(<span class="hljs-string">`id`</span>,<span class="hljs-string">`name`</span>,<span class="hljs-string">`sex`</span>) <span class="hljs-keyword">values</span>(<span class="hljs-number">1</span>,<span class="hljs-string">"a"</span>,<span class="hljs-number">1</span>)<span class="hljs-keyword">insert</span> <span class="hljs-keyword">into</span> table1(<span class="hljs-string">`id`</span>,<span class="hljs-string">`name`</span>,<span class="hljs-string">`sex`</span>) <span class="hljs-keyword">values</span>(<span class="hljs-number">2</span>,<span class="hljs-string">"b"</span>,<span class="hljs-number">1</span>)<span class="hljs-keyword">insert</span> <span class="hljs-keyword">into</span> table1(<span class="hljs-string">`id`</span>,<span class="hljs-string">`name`</span>,<span class="hljs-string">`sex`</span>) <span class="hljs-keyword">values</span>(<span class="hljs-number">3</span>,<span class="hljs-string">"c"</span>,<span class="hljs-number">0</span>)</code></pre></div><p>改为</p><div class="hljs"><pre><code class="hljs sql"><span class="hljs-keyword">insert</span> <span class="hljs-keyword">into</span> table1(<span class="hljs-string">`id`</span>,<span class="hljs-string">`name`</span>,<span class="hljs-string">`sex`</span>) <span class="hljs-keyword">values</span>(<span class="hljs-number">1</span>,<span class="hljs-string">"a"</span>,<span class="hljs-number">1</span>),                                            (<span class="hljs-number">2</span>,<span class="hljs-string">"b"</span>,<span class="hljs-number">1</span>),                                            (<span class="hljs-number">3</span>,<span class="hljs-string">"c"</span>,<span class="hljs-number">0</span>);</code></pre></div><ol start="2"><li>开启 <strong>bulk_insert_buffer_size</strong> 更加加速插入</li></ol><h4 id="从文件中导入数据">从文件中导入数据</h4><p>使用 <strong>LOAD DATA</strong></p><h4 id="利用default-value">利用default value</h4><p>每当插入数据不同于default值的时候再插入，使用ignore</p><div class="hljs"><pre><code class="hljs sql"><span class="hljs-keyword">insert</span> <span class="hljs-keyword">ignore</span> <span class="hljs-keyword">into</span> table1 <span class="hljs-keyword">values</span> (<span class="hljs-number">1</span>,<span class="hljs-string">'a'</span>，<span class="hljs-number">1</span>);</code></pre></div><p>甚至还区分了 InnoDB引擎和MyISAM引擎的做法：</p><h4 id="innodb：">InnoDB：</h4><ol><li>关闭autocommit，因为每次插入，InnoDB都会写log;</li><li>如果有unqiue 限制插入的列，可以暂时关闭 unique_checks;</li><li>如果有外键在列，可以暂时关闭外键约束foreign_key_checks;</li><li>插入数据的时候，如果数据能按照primary key的顺序插入，会大大加快速度（因为主键是聚簇索引）</li><li>数据有自增主键的时候，把innodb_autoinc_lock_mode从1改为2（14.6.1.4)</li></ol><h4 id="myisam">MyISAM</h4><p>LSM(log structured)模型，主要是顺序读写，写性能&gt;读性能;//todo</p><h2 id="log">Log</h2><p>Undo日志记录某数据被修改前的值，可以用来在事务失败时进行回滚；Redo日志记录某数据块被修改后的值，可以用来恢复未写入data file的已成功事务更新的数据。</p><h2 id="查询">查询</h2><h3 id="in-查询">in 查询</h3><div class="hljs"><pre><code class="hljs sql"><span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> table_a <span class="hljs-keyword">where</span> A <span class="hljs-keyword">in</span> () <span class="hljs-keyword">AND</span> B <span class="hljs-keyword">in</span> ()</code></pre></div><p>其实际会先从A筛选in出来再回一次表筛选B</p><p>可以改写为</p><div class="hljs"><pre><code class="hljs sql"><span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> table_a <span class="hljs-keyword">where</span> (A,B) <span class="hljs-keyword">in</span> ((<span class="hljs-number">1.</span>..n),(<span class="hljs-number">2.</span>..m))</code></pre></div><h3 id="order-by">order by</h3><div class="hljs"><pre><code class="hljs sql"><span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> tb1 <span class="hljs-keyword">order</span> <span class="hljs-keyword">by</span> idx1 <span class="hljs-keyword">limit</span> <span class="hljs-number">4</span>,<span class="hljs-number">1</span>;<span class="hljs-keyword">select</span> * <span class="hljs-keyword">from</span> tb1 <span class="hljs-keyword">force</span> <span class="hljs-keyword">index</span>(idx1) <span class="hljs-keyword">limit</span> <span class="hljs-number">4</span>,<span class="hljs-number">1</span>;</code></pre></div><p>某种情况下得到不同结果;</p><p>具体使用哪一种排序方式是优化器决定的，总的说来如下：</p><p>直接利用索引避免排序：用于有索引且回表效率高的情况下</p><p>快速排序算法：如果没有索引大量排序的情况下</p><p>堆排序算法：如果没有索引排序量不大的情况下</p><p>快排和堆排不稳定，</p><h3 id="count-count-1-count-column">count(*),count(1),count(column)</h3><p>明确在MYISAM和Innodb的速率是不一样的:</p><ul><li>MYISAM下count(*)可以直接得出数值，复杂度O(1),因为保存了一个变量</li><li>INNODB因为支持了事务，有repeatable的隔离级别（用了MVCC），所以不同事务有不同的数据版本，不能采用保存一个变量这种做法</li></ul><p>然后count(*)和count(1)在INNODB中底层的性能其实是一致的：</p><p><code>count(*)</code>会计算所有值<code>count(1)</code>会计算non-nil的值</p><p>INNODB会有一个小优化，会使用最小的二级索引</p><p><code>count(column)</code>在拿到值时先判断是否为空，然后再累加;如果遇到的是二级索引，则要再回表一次根据主键得到数据，多了一次IO；</p><h3 id="比较">比较</h3><p>(a,b)&gt;(x,y) 等价于：</p><div class="hljs"><pre><code class="hljs undefined">(<span class="hljs-name">a</span> &gt; x) OR ((<span class="hljs-name">a</span> = x) AND (<span class="hljs-name">b</span> &gt; y))</code></pre></div><h3 id="index">Index</h3><h4 id="coveringindex-覆盖索引">coveringIndex(覆盖索引)</h4><p>指的是:一个查询语句的执行只用从索引中就能够取得，不必从数据表中读取。也可以称之为实现了索引覆盖;</p><ol><li>索引项通常比记录要小，所以MySQL访问更少的数据</li><li>索引都按值的大小顺序存储，相对于随机访问记录，需要更少的I/O</li><li>大多数据引擎能更好的缓存索引，比如MyISAM只缓存索引</li><li>覆盖索引对于InnoDB表尤其有用，因为InnoDB使用聚集索引组织数据，如果二级索引中包含查询所需的数据，就不再需要在聚集索引中查找了</li></ol><p><strong>重要!!!</strong></p><ul><li><p>select只能select在索引上的值（由定义可知道，索引保存的值即是你需要的值，否则又tm要回表查一次）</p></li><li><p>一个覆盖索必须满足查询中给定表用到的所有的列仅仅是个前提条件，这个索引还必须包含指定表上包括WHERE子句, ORDER BY, GROUP BY子句等等</p></li><li><p>覆盖索引不能有<code>like</code></p></li></ul><p>重复的index还会把覆盖索引给覆盖掉:</p><div class="hljs"><pre><code class="hljs sql"><span class="hljs-comment">#创建了联合的覆盖索引后</span>index idx_n_id(name,id)<span class="hljs-comment">## 再单独创建一个索引在name上</span>index idx_n(name)<span class="hljs-comment"># 会将之前的索引覆盖掉</span></code></pre></div><h4 id="唯一索引和普通索引">唯一索引和普通索引</h4><ol><li><p>唯一索引<strong>搜索</strong>满足的第一条记录会立马返回，通知检索（因为唯一性的保证）。但是这个区别并没有很大的性能区别，因为Innodb是按照页（默认16KB）读写的，读数据的时候是从B+树的根节点开始搜索，搜索的时候将整个页从硬盘加载到内存。</p></li><li><p>唯一索引在<strong>插入</strong>的时候会多做些判断，想要做这个判断就必须先把数据页读入内存。但是普通索引不需要做这个判断，就可以把需要更新的数据做判断：</p><ul><li>如果数据在内存则直接更新；</li><li>如果不在也不加载内存，而是先写入change buffer，等下次查询的时候再执行change buffer。</li></ul></li></ol><p>这样看来普通索引会相对性能好一些。</p><p>但是注意：如果业务场景是写入后立马有查询，其实还是会立马需要把数据页加载到内存，这样的情况下其实并不能带来优化IO的操作。</p><h4 id="最左匹配原则">最左匹配原则</h4><div class="hljs"><pre><code class="hljs sql">index id_key1_key2_key3()</code></pre></div><p>针对上面的index，根据该原则，业务上各个列的 <strong>使用频率和重要性</strong> 应该是key1&gt;key2&gt;key3</p><ul><li>而且只有key1在条件内才会用到该索引</li><li><code>id_key1_key2_key3</code> 等于创建了 <code>id_key1</code>, <code>id_key1_key2</code>, <code>id_key1_key2_key3</code>三个索引</li><li>在该索引内使用范围查找会使其失效（in 属于精确查找）</li></ul><h2 id="explain">Explain</h2><p><code>id</code>:（select 查询的序列号，包含一组数字，表示查询中执行select子句或操作表的顺序）</p><ul><li>id相同，执行顺序从上往下</li><li>id全不同，如果是子查询，id的序号会递增，id值越大优先级越高，越先被执行</li><li>id部分相同，执行顺序是先按照数字大的先执行，然后数字相同的按照从上往下的顺序执行</li></ul><p><code>select_type</code>:（查询类型，用于区别普通查询、联合查询、子查询等复杂查询）</p><ul><li>SIMPLE ：简单的select查询，查询中不包含子查询或UNION</li><li>PRIMARY：查询中若包含任何复杂的子部分，最外层查询被标记为PRIMARY</li><li>SUBQUERY：在select或where列表中包含了子查询</li><li>DERIVED：在from列表中包含的子查询被标记为DERIVED，MySQL会递归执行这些子查询，把结果放在临时表里</li><li>UNION：若第二个select出现在UNION之后，则被标记为UNION，若UNION包含在from子句的子查询中，外层select将被标记为DERIVED</li><li>UNION RESULT：从UNION表获取结果的select<code>table</code>:显示这一行的数据是关于哪张表的</li></ul><p><code>type</code>:（显示查询使用了那种类型，从最好到最差依次排列 system &gt; const &gt; eq_ref &gt; ref &gt; fulltext &gt; ref_or_null &gt; index_merge &gt; unique_subquery &gt; index_subquery &gt; range &gt; index &gt; ALL ）</p><p>( 一般来说，得保证查询至少达到range级别，最好到达ref)</p><div class="hljs"><pre><code>system：表只有一行记录（等于系统表），是 const 类型的特例，平时不会出现const：表示通过索引一次就找到了，const 用于比较 primary key 或 unique 索引，因为只要匹配一行数据，所以很快，如将主键置于 where 列表中，mysql 就能将该查询转换为一个常量eq_ref：唯一性索引扫描，对于每个索引键，表中只有一条记录与之匹配，常见于主键或唯一索引扫描ref：非唯一性索引扫描，范围匹配某个单独值得所有行。本质上也是一种索引访问，他返回所有匹配某个单独值的行，然而，它可能也会找到多个符合条件的行，多以他应该属于查找和扫描的混合体range：只检索给定范围的行，使用一个索引来选择行。key列显示使用了哪个索引，一般就是在你的where语句中出现了between、&lt;、&gt;、in等的查询，这种范围扫描索引比全表扫描要好，因为它只需开始于索引的某一点，而结束于另一点，不用扫描全部索引index：Full Index Scan，index于ALL区别为index类型只遍历索引树。通常比ALL快，因为索引文件通常比数据文件小。（也就是说虽然all和index都是读全表，但index是从索引中读取的，而all是从硬盘中读的）ALL：Full Table Scan，将遍历全表找到匹配的行possible_keys（显示可能应用在这张表中的索引，一个或多个，查询涉及到的字段若存在索引，则该索引将被列出，但不一定被查询实际使用）</code></pre></div><p><code>key</code>:</p><p>实际使用的索引，如果为NULL，则没有使用索引</p><p>查询中若使用了覆盖索引，则该索引和查询的 select 字段重叠，仅出现在key列表中</p><p>explain-key<code>key_len</code></p><p>表示索引中使用的字节数，可通过该列计算查询中使用的索引的长度。在不损失精确性的情况下，长度越短越好key_len显示的值为索引字段的最大可能长度，并非实际使用长度，即key_len是根据表定义计算而得，不是通过表内检索出的ref（显示索引的哪一列被使用了，如果可能的话，是一个常数。哪些列或常量被用于查找索引列上的值）</p><p><code>rows</code>（根据表统计信息及索引选用情况，大致估算找到所需的记录所需要读取的行数）</p><p><code>Extra</code>（包含不适合在其他列中显示但十分重要的额外信息）</p><ul><li><p>using filesort: 说明mysql会对数据使用一个外部的索引排序，不是按照表内的索引顺序进行读取。mysql中无法利用索引完成的排序操作称为“文件排序”。常见于order by和group by语句中</p></li><li><p>Using temporary：使用了临时表保存中间结果，mysql在对查询结果排序时使用临时表。常见于排序order by和分组查询group by。</p><p>using index：表示相应的select操作中使用了覆盖索引，避免访问了表的数据行，效率不错，如果同时出现using where，表明索引被用来执行索引键值的查找；否则索引被用来读取数据而非执行查找操作</p><p>using where：使用了where过滤</p><p>using join buffer：使用了连接缓存</p><p>impossible where：where子句的值总是false，不能用来获取任何元祖</p><p>select tables optimized away：在没有group by子句的情况下，基于索引优化操作或对于MyISAM存储引擎优化COUNT(*)操作，不必等到执行阶段再进行计算，查询执行计划生成的阶段即完成优化</p><p>distinct：优化distinct操作，在找到第一匹配的元祖后即停止找同样值的动作</p></li></ul>]]></content>
    
    
    
    <tags>
      
      <tag>MySQL</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Something about Software Enginnering</title>
    <link href="/2019/07/02/Comcon/SoftwareEngineer/"/>
    <url>/2019/07/02/Comcon/SoftwareEngineer/</url>
    
    <content type="html"><![CDATA[<p>软件工程的通用概念！//todo留坑</p><a id="more"></a><h2 id="fan-in-fan-out">fan-in , fan-out</h2><p>扇入，扇出fan-in: 指一个模块调用多个模块fan-out:指一个模块被多个模块调用</p><p>fan-out影响I/O</p>]]></content>
    
    
    
    <tags>
      
      <tag>se</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Something about Networking</title>
    <link href="/2019/07/02/Comcon/somethingAboutTcp/"/>
    <url>/2019/07/02/Comcon/somethingAboutTcp/</url>
    
    <content type="html"><![CDATA[<p>记录了一些很基础的但经常被问到的问题</p><a id="more"></a><h3 id="1-为什么多个tcp连接会比单个tcp连接快？-tmd面试傻了-居然没答出来这个">1. 为什么多个tcp连接会比单个tcp连接快？（tmd面试傻了，居然没答出来这个）</h3><p>一开始看见，这不是显而易见的吗？？？</p><p>后来发现，其实他想听到的答案是：</p><h4 id="tcp的流量窗口rwnd-接收方-拥塞控制cwnd-发送方">tcp的流量窗口rwnd(接收方),拥塞控制cwnd(发送方)</h4><p>如下图(盗图)：<img src="/img/tcpWindow.jpg" srcset="/img/loading.gif" alt="tcpwindow">绿色为 发送者发送，且接收者acked黄色为 发送者发送，接收者未确认（in-flight）蓝色为 可用但为发送</p><p>cwnd= width（in-flight）+width（not sent）</p><p>发送速率：<strong>rate = cwnd / RTT byte/sec</strong>即发送速率在RTT（往返时延）一定的情况下，只受cwnd影响</p><h4 id="慢启动-拥塞避免">慢启动，拥塞避免</h4><p>注意: 建立连接后，client和server都有一个窗口，每次进行调整窗口都是调整自己的窗口控制发送速率(当然可以传送报文，让对方调整速率);</p><p><code>慢启动</code>(其实一点也不慢): tcp会进行直到丢包,每接到一个ack就会把窗口<code>（cwnd）×2</code>，超过<code>ssThreshold</code>就进行拥塞避免每一轮<code>RTT</code>只增加 <code>1/cwnd</code>（但是以上这个已经是废弃了!!! ）<code>拥塞避免</code>: 到达<code>sshThreshold</code>后会以线性速度上升，但是如果超时（出现拥塞），就将<code>ssThreshold</code>设置为出现拥塞的窗口的一半，然后将窗口设为1，重新开始慢启动</p><p><code>快恢复</code>: 现在慢启动已经废弃了，在收到3个重复确认的ACK时就采用快恢复重传算法，<code>ssThreshold</code>设置为出现拥塞的窗口的一半，将cwnd设为<code>ssThreshold</code>的一半但不采用慢开始;</p><p>当出现丢包的，有以下两种状况：</p><ul><li>接收者发送给发送者的ACK丢失，会导致 timeout</li><li>发送者发送给接收者的数据丢失，发送者会收到接受者的重复ACK，如果收到三个重复的ACK，可以确认为丢包</li></ul><p>这里盗一幅图:</p><p><img src="/img/tcpBlock.png" srcset="/img/loading.gif" alt="如图"></p><ol start="3"><li>路由器（多个TCP有拥塞控制）给出带宽为R，有K个连接经过最后每个连接平均分的都会是 R/K</li></ol><p>终上：</p><p>一个tcp连接很可能不能把当前路由的带宽都用完，而且连接也有重传的情况，所以多个tcp连接可以最大保证速率</p><p><strong><em>但是这个问题在HTTP/2则不是一回事了，因为其靠帧来实现有序性，而且是连接复用(同域名下)，所以多个连接反而会浪费资源</em></strong></p><h3 id="总结">总结</h3><p>回答问题，要从特么原理开始一步步推导来說，不能想当然</p><h3 id="2-各种握手挥手-我求求我自己把这些gdx记得滚瓜烂熟-每次都漏一点点">2. 各种握手挥手（我求求我自己把这些gdx记得滚瓜烂熟，每次都漏一点点）</h3><h4 id="tcp三次握手连接-three-way-handshake">TCP三次握手连接(three-way handshake)</h4><p>直接上他🐎图：<img src="/img/TCPshakeFhand.jpg" srcset="/img/loading.gif" alt="tcpconn"></p><ol><li>client发送server<br><code>SYN=1</code>（同步位，这种报文不能携带数据，但要<strong>消耗一个序号</strong>）自己的序号 <strong>Seq=client_w ( isn )</strong> 给 server （期间client从CLOSED到SYN-SENT状态，server从CLOSED到LISTEN状态）</li></ol><blockquote><blockquote><p>ps: <strong>isn</strong>泛指一种计算序号的算法，有一种是每4μs+1(动态随机,增加安全性，为了避免被第三方猜测到，从而被第三方伪造的RST报文Reset)，直到2^32归零,因为2MSL的限制，所以几乎不可能重复；意义: 发送方的字节数据编号的原点，让对方生成一个合法的接收窗口;</p></blockquote></blockquote><ol start="2"><li><p>server收到报文，把确认报文段中的SYN和ACK都设为1<strong>SYN=1</strong>(同理<strong>要消耗一个序号</strong>) 和 <strong>ACK=1</strong>自己的序号 Seq=server_w, ack= client_w + 1 到client （期间client仍然是SYN-SENT状态，server从LISTEN状态到SYN-RCVD状态）</p></li><li><p>client收到确认报文后，还要给server发送确认收到。把<strong>确认ACK=1</strong>,ack=server_w+1(之前SYN消耗了序号)自己的序号Seq=client_w+1 (因为没有SYN，所以可以携带数据,<strong>但如果不携带数据则不消耗序号</strong>，这里<code>(可以)</code>携带了数据，所以Seq是上一次的序号+1);（client端进入established状态）</p></li><li><p>server收到确认报文后，进入<code>established</code>状态，全部连接完成</p></li></ol><h5 id="半连接状态-即服务器syn-recv状态">半连接状态(即服务器SYN-RECV状态)</h5><p>服务器维护一个半连接队列(<code>BackLogs</code>表示半连接队列的最大容纳数目)，存放半连接。该队列为每个客户端的SYN包（syn=j）开设一个条目，该条目表明服务器已收到SYN包，并向客户发出确认，正在等待客户的ACK确认包。这些条目所标识的连接在服务器处于<code>SYN_RECV</code>状态，当服务器收到客户的确认包时，删除该条目，服务器进入<code>ESTABLISHED</code>状态。</p><h5 id="syn-ack重传次数">SYN-ACK重传次数</h5><p>服务器发送完<code>SYN－ACK</code>包，如果未收到客户确认包，服务器进行首次重传，等待一段时间仍未收到客户确认包，进行第二次重传，如果重传次数超过系统规定的<strong>最大重传次数</strong>，系统将该连接信息从半连接队列中删除。注意，每次重传等待的时间不一定相同;</p><h5 id="三次握手能避免啥🐔儿东西呢-为啥两次不行？">三次握手能避免啥🐔儿东西呢（为啥两次不行？）</h5><p>我们假设client给server发了个连接请求，但是请求丢失，然后client再发一个，server收到这个然后建立连接，这里面client发送了两个请求会有以下情况出现：</p><p>client发出的第一个请求没有丢，只是网络阻塞;但接下来，server收到后以为是新的一个连接，server返回一个确认报文，但client收到server确认报文会发现自己并没有建立连接请求，所以会忽视server的确认报文，也不会向server发送数据但server会<strong>一直开着连接等client的数据，白白浪费资源!</strong>（但采用三次握手的话就没得事，server端没有接到client的第三次确认，就知道client没有要建立连接）</p><h4 id="tcp四次挥手">TCP四次挥手</h4><p><img src="/img/TCPgoodbye.jpg" srcset="/img/loading.gif" alt="tcpfour"></p><ol><li><p>client 发送<code>FIN=1</code>（终止位，跟同步位一样都在header里面，<del>自己特喵去看</del>下面给你画一个算了,但这里注意，<strong>无论带不带数据，这厮都要消耗一个序号！</strong>）<code>seq=client_u</code> （这里序号是<strong>前面一个传送过的数据最后一个字节的序号+1</strong>）;(client进入<code>FIN-WAIT-1</code>状态)</p></li><li><p>server收到释放报文返回<code>ACK=1</code><code>Seq=server_u</code>(同理，也是前面一个传送过的数据的最后一个字节序号+1);(server进入CLOSE-WAIT状态，但实际上是个半关闭状态，server-&gt;client方向的连接保持，但client-&gt;server已经没有<strong>数据</strong>要传输了)</p></li><li><p>client收到server的确认后，进入<code>FIN-WAIT2</code>状态，等待server的连接释放报文</p></li><li><p>如果server没有数据要发给client了（server持续发送数据到client，也可以不发），携带报文 <code>FIN=1</code>, <code>ACK=1</code>, <code>seq=server_u2</code>(新，可能发送了一些数据)<code>ack=client_u+1</code>（重复上次已发送的确认号）(server进入LAST-ACK状态)</p></li><li><p>(紧接3)client收到释放报文后，返回确认报文：ACK=1确认号<strong>ack=server_u2+1</strong>序号<strong>seq=client_u+1</strong> （前面1的FIN报文消耗了一个序号client_u）然后client进入TIME_WAIT状态，然后经过2MSL（RFC 793设为2mins，但其实应该要用更小的数值: 60s 在linux内<code>/proc/sys/net/ipv4/tcp_fin_timeout</code>）再进入CLOSED状态client撤销相应的TCB（传输控制块）后，结束连接</p></li></ol><h5 id="为啥要等2msl">为啥要等2MSL</h5><ol><li><p>保证client发送的最后一个ACK报文能够到达server;因为这个报文可能会丢失，然后处在LAST-ACK的server收不到已发送的FIN+ACK报文的确认。正常情况下，server会重传FIN+ACK报文，client接着重传最后一个ACK报文，重新计时;但如果client不等待，发送完ACK报文直接释放连接进入CLOSED，client就没法收到server重传的FIN+ACK报文，也不会再发送一次确认报文（因为关闭了啊！），这样server重传的东西client就收不到，server也就不会正常进入CLOSED状态</p></li><li><p>防止出现 ‘已失效的连接请求报文段’(如同三次握手时间的client第一次发出的报文);client发送完最后一个ACK报文段后，经过2MSL，可以确定本连接持续时间内的所有报文段都从网络消失，这样就不会出现旧的连接报文段了</p></li></ol><h5 id="为啥连接用三次-断开要四次呢？">为啥连接用三次，断开要四次呢？</h5><ul><li>主要区别还是在连接时，server收到连接请求后可以直接返回SYN+ACK报文；</li><li>但是断开时，server收到了client的FIN报文后，可能还有数据没有传完，只能先发回一个ACK，等到最后数据都传完了才会发FIN，所以为了避免没传完数据就关闭了的情况，只能加多一次连接;如果是三次(最后一次server返回的ACK+FIN还有一堆数据合并为最后一次response)，会造成长时间阻塞，导致客户端以为上一次的FIN没有到server，然后重传;</li></ul><p>PS：server的cclosed状态要比client的要早一点MSL&gt;=TTL</p><p>连接当中如果用HTTPS，参考之前写的</p><h5 id="keepalive-timer">keepalive timer</h5><p>存在header里面的keep-alive是http协议的，可以维持长连接,HTTP/1.1开启;</p><p>2MSL是被client的一个TIME-WAIT timer设置的，实际上TCP<strong>还有</strong>另外一个keepalive timer，主要用来探测端到端的连接有没有失效linux命令</p><div class="hljs"><pre><code class="hljs s">sysctl -a | grep keepalive</code></pre></div><p>默认7200s检测一次，一次最多重传9个包,每个包间隔75s</p><p>但是因为该设置不太合理，比如检测间隔太长等,很多应用没有开启</p><h5 id="tcp-fast-open">TCP fast open</h5><p>顾名思义是一个减少连接时延的方法，实现在第二次握手就可以传输响应数据,主要就是使用SYN cookie实现具体做法：</p><ol><li><p>首轮的三次握手中，服务端接到SYN不会立即回复SYN+ACK，而是通过计算得到一个SYN Cookie,然后将这个Cookie放到TCP报文的fast open选项，然后返回，接下来就是正常的三次握手余下的流程;</p></li><li><p>接下来的握手中，客户端会将之前缓存的<strong>cookie</strong>，<strong>SYN</strong>和<strong>HTTP request</strong> 一起发给服务端，如果合法(不合法可能是过期的原因)直接返回SYN+ACK;</p></li><li><p>(重要！！！)接下来服务端就可以直接发送数据而不用等客户端的ACK了！！！</p></li><li><p>但是确保三次握手协议不变，client端最后还是会返回ACK</p></li></ol><p>linux可以通过</p><div class="hljs"><pre><code class="hljs bash">cat /proc/sys/net/ipv4/tcp_fastopen</code></pre></div><p>检查是否开启，1为client端，2为server，3为both</p><p><strong>注意一下这个功能可能被一些防火墙隔离</strong></p><h4 id="tcp-粘包问题">TCP 粘包问题</h4><p>这个其实没有特定的术语指明粘包这种东西，只是一种现象，主要是应用层对缓冲区中的数据解析出了问题（因为tcp是<code>流式传输</code>，字节流可能没有传完，所以一般在接收端要重组分组）,没有正确处理消息边界的原因（发送方可能用了<code>nagle</code>或者接收方接收到没有立即处理）;</p><p>处理方法:</p><ol><li>发送方设置<code>TCP_NODELAY</code>;</li><li>接收方在应用层，要和发送方商量，让<code>发送方带上消息边界（这时也要保证消息内容不含消息边界）</code>或者<code>消息长度(http自带的header就可以content-length解析)</code></li></ol><p>这里又有了，UDP会不会有这个问题呢？</p><ul><li>不会；因为UDP是基于<code>数据报(datagram)</code>来发送的，不会有分组，在接收端是有一个<code>skbuff</code>是一个链表来接收一个个数据报；</li></ul><h3 id="3-有哪些字段-一些字段的意义">3. 有哪些字段，一些字段的意义</h3><p>比较关键的字段</p><ul><li>明确方向源端口(2B)，目的端口(2B)</li><li>明确报文本身信息头部长度(4bit)</li><li>可靠性ack(8B)，seq号码(8B) , 窗口大小(16B),校验和(16B)</li><li>一些标志的bitACK,SYN,FIN ,RST,PSH(不放入缓存直接被程序用),…???</li><li>额外的字段kind(1B)+length(1B)+info(8B)比如有<strong>timestamp</strong>字段:kind = 8 , length = 10, info由timestamp(4B)和timestamp echo(4B)组成</li></ul><h3 id="4-一些拥塞控制优化吞吐量的算法">4. 一些拥塞控制优化吞吐量的算法</h3><h4 id="nagle">Nagle</h4><p>有一种情景是client端不断发给server端很小的包，一次1B，这样发1KB就要1K次；Nagle就是解决这种问题，具体做法</p><ul><li>第一次发1B，立即发送</li><li>后面的发送要满足<ol><li>数据要达到MSS</li><li>之前所有包ACK都收到</li></ol></li></ul><h4 id="延迟确认">延迟确认</h4><p>指的是把一段时间内ACK合并然后延迟回复,tcp要求这个时延必须小于500ms，unix一般不超过200ms但是有一些<strong>不能</strong>延迟确认</p><ul><li>接收到一个大于一个帧的报文，且需要调整窗口大小</li><li>TCP处于quick ack模式(tcp_in_quick_mode)</li><li>有乱序的包</li></ul><h4 id="byte-queue-limits-bql">Byte Queue Limits (BQL)</h4><h4 id="tcp-small-queues-tsq">TCP Small Queues (TSQ)</h4><h4 id="early-departure-time-edt">Early Departure Time (EDT)</h4><h3 id="5-其他">5. 其他</h3><h4 id="udp有什么是tcp不可以代替的">UDP有什么是TCP不可以代替的?</h4><ol><li><p>授时协议，tcp的重传会加大时间计算的误差</p></li><li><p>广播</p></li></ol><p>如果要将UDP包装成TCP:</p><ol><li>增加ACK/Seq机制</li><li>发送和接收缓冲区</li><li>超时重传机制</li></ol><h4 id="tcp-backlogs">TCP Backlogs</h4><p>针对SYN floods攻击，linux下使用了**两个队列(模型)**来缓解:</p><ul><li><p>一个是有最小metadata的SYN Backlog(未完成的连接队列);</p><p>每个这样的syn分节对应其中一项：已由某个客户发出到达服务器，而服务器正在等待完成相应的TCP三鹿握手的过程，这些套接字处于SYN_RCVD状态。</p></li><li><p>一个是监听功能的Listen SYN Backlog(已完成的连接队列);</p><p>每个已经完成的三路握手的客户对应其中的一员，这些套接字处于ESTABLISHED状态。</p></li></ul><p>如图<img src="/img/tcpsynlogs.png" srcset="/img/loading.gif" alt="tcpsynlog"></p><p>TCP在未完成队列接收SYN的request，当三次握手完成(established)后，就会将这个request移到已完成的连接队列的尾部;</p><p>除了两个队列模型外，TCP listen path也用无锁方式来改进???（似乎有点问题）4.7内核版本改进</p><h4 id="tcp复用端口">TCP复用端口</h4><p>端口复用最常用的用途应该是防止服务器重启时之前绑定的端口还未释放或者程序突然退出而系统没有释放端口。此时如果设定了端口复用，则新启动的服务器进程可以直接绑定端口;</p><p>一般来说，一个端口释放后会等待两分钟之后或三十秒才能再被使用，<code>SO_REUSEADDR</code>是让端口释放后立即就可以被再次使用;</p><p><code>SO_REUSEADDR</code>套接字选项通知内核，如果TCP状态位于 <code>TIME_WAIT</code>，可以重用端口。如果TCP状态位于其他状态，重用端口时依旧得到一个错误信息，指明&quot;地址已经使用中&quot;。如果你的服务程序停止后想立即重启，而新套接字依旧使用同一端口，此时<code>SO_REUSEADDR</code> 选项非常有用。</p><p>端口复用允许在一个应用程序可以把 n 个套接字绑在一个端口上而不出错。同时，这 n 个套接字发送信息都正常，没有问题;</p><p>但是，这些套接字并不是所有都能读取信息，只有<strong>最后一个</strong>套接字会正常接收数据，这个特性都为后门程序所应用</p><h4 id="dns协议">DNS协议</h4><p>首先这🐔是<strong>应用层</strong>协议！！！功能主要就是<strong>把域名转换为IP地址</strong>然后，主要是在<strong>UDP</strong>上面跑(为什么说主要，因为协议其实规定在超过512B时，应该用<code>TCP</code>进行重试)，端口<strong>53</strong>长度最多是<strong>512Bytes</strong>，若过多要用 <a href="https://en.wikipedia.org/wiki/Extension_mechanisms_for_DNS" target="_blank" rel="noopener">EDNS</a> (最多支持4096B)</p><p>为什么选UDP/TCP?</p><ul><li><p>UDP 协议</p><p>DNS 查询的数据包较小、机制简单；</p><p>UDP 协议的额外开销小、有着更好的性能表现；</p></li><li><p>TCP 协议</p><p>DNS 查询由于 DNSSEC 和 IPv6 的引入迅速膨胀，导致 DNS 响应经常超过 MTU 造成数据的分片和丢失，我们需要依靠更加可靠的 TCP 协议完成数据的传输；</p><p>随着 DNS 查询中包含的数据不断增加，TCP 协议头以及三次握手带来的额外开销比例逐渐降低，不再是占据总传输数据大小的主要部分；</p></li></ul><h4 id="uri-和-url-和域名">URI 和 URL 和域名</h4><p>URI（统一资源标识符） 包括 URL（统一资源定位符）和URN（统一资源名称）</p><p>URI通用模式：</p><div class="hljs"><pre><code class="hljs undefined">scheme：<span class="hljs-comment">[// <span class="hljs-comment">[user：password @]</span> host <span class="hljs-comment">[：port]</span>]</span> <span class="hljs-comment">[/]</span> path <span class="hljs-comment">[？查询]</span> <span class="hljs-comment">[#片段]</span></code></pre></div><p>URL：主要用于连接网页或部分部件，借助访问的协议（http，ftp等）来检索定位资源位置URL包含了</p><ol><li>访问资源的协议</li><li>服务器位置</li><li>端口</li><li>资源在服务器上的位置</li><li>片段标识符（就是锚点#something）</li></ol><p>域名（domain name）指的是任一主机或路由器连接在因特网上都有<strong>唯一</strong>的 <strong>层次结构的名字</strong>,只是一个<strong>逻辑概念</strong></p><h3 id="6-浏览器输入url发生了什么-个人感觉按照这个🐔来复习会比较好">6. 浏览器输入url发生了什么 （个人感觉按照这个🐔来复习会比较好）</h3><ol><li>输入URL，浏览器会解析url，这里面是通过DNS域名解析，找到url对应的服务器地址，其中可能会从HOSTS文件找</li><li>找到主机地址，就会连接主机，这里面tcp三次握手，发送HTTP请求然后封装发送的包，http包放在tcp包里，tcp包放在IP包里，层层往下，每一层会通过网管（gateway）</li><li>到了IP协议处会将其通过ARP解析出相应的在链路层的物理地址（通过路由器），在网络层和以上使用都是IP地址，以下都是硬件地址了，所以数据链路层看不见IP地址了</li><li>网络层通过物理地址找到路由器，把层层封装的包通过网桥（或桥接器等）等发送到链路层上，当前只能看见MAC帧，链路层负责开始传输数据了</li><li>物理层只是把传输数据 变成电信号真正地传到网线上</li></ol>]]></content>
    
    
    
    <tags>
      
      <tag>networking</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Something about LOCK</title>
    <link href="/2019/07/01/Comcon/ReentranceLock/"/>
    <url>/2019/07/01/Comcon/ReentranceLock/</url>
    
    <content type="html"><![CDATA[<h2 id="可重入锁-reentrancelock">可重入锁（ReentranceLock）</h2><p>可以进一步加强锁的封装性，简化了代码的开发，避免死锁;拿别人的一个例子：</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Parent</span></span>&#123;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">synchronized</span> <span class="hljs-keyword">void</span> <span class="hljs-title">parentDo</span><span class="hljs-params">()</span></span>&#123;        <span class="hljs-comment">//....</span>    &#125;&#125;<span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Child</span> <span class="hljs-keyword">extends</span> <span class="hljs-title">Parent</span></span>&#123;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">synchronized</span> <span class="hljs-keyword">void</span> <span class="hljs-title">childDo</span><span class="hljs-params">()</span></span>&#123;        <span class="hljs-keyword">super</span>.parentDo();    &#125;&#125;</code></pre></div><p>如果不可重入，super.parentDo()不可以获得Parent的锁，因为这个锁已经被childDo()持有，从而使线程阻塞，造成死锁所以可以用于：</p><ol><li>递归调用</li><li>此线程调用同一对象其它synchronized或者有同步锁函数。synchronized是可重入锁</li></ol><h3 id="golang中不同">Golang中不同</h3><p>golang设计者对此做过回应<img src="https://stackoverflow.com/questions/14670979/recursive-locking-in-go#14671462" srcset="/img/loading.gif" alt="这里">go不支持可重入锁，只能通过将需要锁的操作作为函数，并以Locked为后缀命名，然后调用它们的时候用锁锁住</p><h2 id="自旋锁-spinlock">自旋锁(SpinLock)</h2><p>线程会反复检查锁变量是否可用。但由于线程此时一直保持执行，所以属于忙等待（busy-waiting），一旦获得了自旋锁，线程会一直保持该锁，直到显式释放;这里说说<strong>互斥锁</strong>，区别就是，互斥锁的调用者一方如果发现锁被拿走了，就会<strong>进入睡眠状态</strong>盗取别人例子：</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">SpinLock</span></span>&#123;    <span class="hljs-keyword">private</span> AtomicReference&lt;Thread&gt; cas = <span class="hljs-keyword">new</span> AtomicReference&lt;Thread&gt;();    <span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> lock&#123;        Thread cur=Thread.currentThread();        <span class="hljs-comment">//CAS</span>        <span class="hljs-comment">//第一个线程得到锁，就会跳过while循环，第二个线程会一直在while循环，直到满足CAS</span>        <span class="hljs-keyword">while</span> (!cas.compareAndSet(<span class="hljs-keyword">null</span>,cur))&#123;            <span class="hljs-comment">//Do sth...</span>        &#125;    &#125;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">unlock</span><span class="hljs-params">()</span></span>&#123;        Thread cur=Thread.currentThread();        cas.compareAndSet(cur,<span class="hljs-keyword">null</span>);    &#125;&#125;</code></pre></div><p>Pros:自旋锁避免上下文切换（？），只要阻塞很短时间的场景下可以使用<br>Cons:</p><ol><li>然而会导致不公平地持有锁，无法满足等待最长时间线程获得锁，会有饥饿状态出现</li><li>越来越多的线程循环等待会导致cpu越来越多</li></ol><p>可重入的自旋锁和不可重入的自旋锁</p>]]></content>
    
    
    
    <tags>
      
      <tag>LOCK</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Something in gRPC</title>
    <link href="/2019/06/29/Go/gRPC/"/>
    <url>/2019/06/29/Go/gRPC/</url>
    
    <content type="html"><![CDATA[<p>常用的rpc框架</p><a id="more"></a><h2 id="整体模型">整体模型</h2><p>就是常见的一种<code>Reactor</code>模型，</p><p>listen线程和处理线程是不同的</p><h2 id="连接池">连接池</h2><h2 id="服务发现和服务注册">服务发现和服务注册</h2><p>很遗憾，grpc只提供了接口,这些是要自己用其他服务发现的中间件来实现</p><h3 id="一般的设计">一般的设计</h3><h4 id="集中式lb">集中式LB</h4><p>就是在server和client端中间加一个proxy来记录注册的服务（一些地址映射表）；</p><ul><li>单点问题(proxy改成分布式组件即可)</li><li>增加了额外中间件，复杂度提高</li></ul><h4 id="进程内lb-balancing-aware-client">进程内LB(Balancing-aware client)</h4><p>因为第一个问题，那么我们把这个proxy放到消费方进程里，这个可以叫软负载或者客户端负载；定期用心跳来同步服务注册表来表明服务存活状态</p><ul><li><p>LB和服务发现分散到每一个服务消费者进程内部</p></li><li><p>同时，消费方和提供方是直接连接，无消耗</p></li><li><p>开发成本高，不同语言的调用方要有不同语言版本的LB</p></li><li><p>升级后，每个都要重新发布</p></li></ul><h4 id="独立进程lb">独立进程LB</h4><p>同进程内LB差不多，都是在消费方进程中，只不过这个进程是独立出来专门负责服务发现和LB的</p><ul><li>因为不同进程，简化服务调用方，不需要重新开发不同语言客户端</li><li>升级不用服务调用方改代码</li></ul><h3 id="grpc的负载均衡设计">grpc的负载均衡设计</h3><p>grpc就属于这种:有通过第三方proxy获得负载均衡列表的方式(<code>grpclb</code>)，也有通过<code>Name Resolver</code>(dns)的方式(都是旧版本):</p><h4 id="新版本中-1-26以上">新版本中(1.26以上)</h4><p>主要通过 <code>atrributes.Attributes</code>来实现:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> Address <span class="hljs-keyword">struct</span> &#123;...<span class="hljs-comment">// Attributes contains arbitrary data about this address intended for</span><span class="hljs-comment">// consumption by the load balancing policy.</span>Attributes *attributes.Attributes&#125;</code></pre></div><p>Atrributes包中我们可以看到,其为一个map，存储:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Attributes is an immutable struct for storing and retrieving generic</span><span class="hljs-comment">// key/value pairs.  Keys must be hashable, and users should define their own</span><span class="hljs-comment">// types for keys.</span><span class="hljs-keyword">type</span> Attributes <span class="hljs-keyword">struct</span> &#123;m <span class="hljs-keyword">map</span>[<span class="hljs-keyword">interface</span>&#123;&#125;]<span class="hljs-keyword">interface</span>&#123;&#125;&#125;</code></pre></div><h4 id="grpclb">grpclb</h4><p>主要属于在客户端处实行负载均衡:</p><ul><li><p>首先client会发起服务器名称解析请求，即把服务器名称解析成若干个IP，这些IP可能是负载均衡器的地址，也可能是实际服务器地址，同时可以设置是使用<code>grpclb</code>还是直接用roundRobin,roundRobinWeighted</p></li><li><p>然后就要实例化负载均衡策略,如果client知道某个ip是负载均衡地址，则client会使用<code>grpclb</code>策略，其他的设置为配置中要求的策略</p></li><li><p>负载均衡策略为每个服务建立一个subChannel（除了grpclb)这里要注意，grpclb下，会在<code>resolver</code>返回的负载均衡器IP上打开一个<strong>流连接</strong>，客户端会通过这个连接，根据名称获得需要的服务器IP但是，如果是非负载均衡器IP的话，会以回调的方式进行，以免没有均衡器</p></li><li><p>当有rpc请求时，就用负载均衡决定的Subchannel接收请求，当可用服务器为空时会被阻塞;</p></li></ul><p>在注释中有一个图描述的比较清楚</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// The parent ClientConn should re-resolve when grpclb loses connection to the</span><span class="hljs-comment">// remote balancer. When the ClientConn inside grpclb gets a TransientFailure,</span><span class="hljs-comment">// it calls lbManualResolver.ResolveNow(), which calls parent ClientConn's</span><span class="hljs-comment">// ResolveNow, and eventually results in re-resolve happening in parent</span><span class="hljs-comment">// ClientConn's resolver (DNS for example).</span><span class="hljs-comment">//</span><span class="hljs-comment">//                          parent</span><span class="hljs-comment">//                          ClientConn</span><span class="hljs-comment">//  +-----------------------------------------------------------------+</span><span class="hljs-comment">//  |             parent          +---------------------------------+ |</span><span class="hljs-comment">//  | DNS         ClientConn      |  grpclb                         | |</span><span class="hljs-comment">//  | resolver    balancerWrapper |                                 | |</span><span class="hljs-comment">//  | +              +            |    grpclb          grpclb       | |</span><span class="hljs-comment">//  | |              |            |    ManualResolver  ClientConn   | |</span><span class="hljs-comment">//  | |              |            |     +              +            | |</span><span class="hljs-comment">//  | |              |            |     |              | Transient  | |</span><span class="hljs-comment">//  | |              |            |     |              | Failure    | |</span><span class="hljs-comment">//  | |              |            |     |  &lt;---------  |            | |</span><span class="hljs-comment">//  | |              | &lt;--------------- |  ResolveNow  |            | |</span><span class="hljs-comment">//  | |  &lt;---------  | ResolveNow |     |              |            | |</span><span class="hljs-comment">//  | |  ResolveNow  |            |     |              |            | |</span><span class="hljs-comment">//  | |              |            |     |              |            | |</span><span class="hljs-comment">//  | +              +            |     +              +            | |</span><span class="hljs-comment">//  |                             +---------------------------------+ |</span><span class="hljs-comment">//  +-----------------------------------------------------------------+</span><span class="hljs-comment">// lbManualResolver is used by the ClientConn inside grpclb. It's a manual</span><span class="hljs-comment">// resolver with a special ResolveNow() function.</span><span class="hljs-comment">//</span><span class="hljs-comment">// When ResolveNow() is called, it calls ResolveNow() on the parent ClientConn,</span><span class="hljs-comment">// so when grpclb client lose contact with remote balancers, the parent</span><span class="hljs-comment">// ClientConn's resolver will re-resolve.</span></code></pre></div><h4 id="name-resolver">Name Resolver</h4><h3 id="grpc服务发现">grpc服务发现</h3><p>在<code>/resolver/</code>文件夹下(在<code>/naming/</code>文件夹下实现的接口已经报废):</p><div class="hljs"><pre><code class="hljs go"></code></pre></div><h2 id="http2">http2</h2><p>见我自己的http2的文章，随便记了一点;</p><h3 id="server端针对不同的帧进行处理">server端针对不同的帧进行处理</h3><p>每个Stream有<strong>唯一</strong>的ID标识，如果是客户端创建的则ID是<code>奇数</code>，服务端创建的ID则是<code>偶数</code>。如果一条连接上的ID使用完了，Client会新建一条连接，Server也会给Client发送一个<code>GOAWAY Frame</code>强制让Client新建一条连接;</p><p>一条grpc连接允许并发的发送和接收多个Stream，而控制的参数便是<code>MaxConcurrentStreams</code>，Golang的服务端默认是100。</p><h3 id="超时问题">超时问题</h3><p>可以带上一个<code>timeout Context</code>，但是里面实际会转换成header frame中的 <code>grpc-timeout</code> ???</p><p>在grpc中，<code>header frame</code>会带上不少信息，比如<code>grpc-status</code>状态等等,server端的处理可以看到<code>processHeaderField</code>方法中解析的多种header：比较重要的有:</p><ul><li><code>grpc-timeout</code>:长连接超时控制</li><li></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(d *decodeState)</span> <span class="hljs-title">processHeaderField</span><span class="hljs-params">(f hpack.HeaderField)</span></span> &#123;<span class="hljs-keyword">switch</span> f.Name &#123;...<span class="hljs-keyword">case</span> <span class="hljs-string">"grpc-timeout"</span>:d.data.timeoutSet = <span class="hljs-literal">true</span><span class="hljs-keyword">var</span> err error<span class="hljs-keyword">if</span> d.data.timeout, err = decodeTimeout(f.Value); err != <span class="hljs-literal">nil</span> &#123;d.data.grpcErr = status.Errorf(codes.Internal, <span class="hljs-string">"transport: malformed time-out: %v"</span>, err)&#125;<span class="hljs-keyword">case</span> <span class="hljs-string">":path"</span>:d.data.method = f.Value<span class="hljs-keyword">case</span> <span class="hljs-string">":status"</span>:code, err := strconv.Atoi(f.Value)<span class="hljs-keyword">if</span> err != <span class="hljs-literal">nil</span> &#123;d.data.httpErr = status.Errorf(codes.Internal, <span class="hljs-string">"transport: malformed http-status: %v"</span>, err)<span class="hljs-keyword">return</span>&#125;d.data.httpStatus = &amp;code<span class="hljs-keyword">case</span> ....&#125;&#125;</code></pre></div><h2 id="如何识别服务以及解析">如何识别服务以及解析</h2><h3 id="识别服务">识别服务</h3><p>识别服务方面比较直接，就是用<code>string</code>判等服务名字即可;</p><p>解析数据主要是借用了<code>proto</code>的<code>generate</code>工具，生成了服务端和客户端的代码，里面使用了自带的解码器，跟pb文件结合起来，<strong>避免了语言层面上的反射的消耗</strong>;</p><p>我们这里只讨论go语法，语言方面其他都是大同小异创建proto文件 <strong>data.proto</strong></p><div class="hljs"><pre><code class="hljs go">syntax = <span class="hljs-string">"proto2"</span>;service Authenticate&#123;rpc login(toServerData) returns (ResponseFromServer)&#123;&#125;rpc home(toServerData) returns(ResponseFromServer)&#123;&#125;rpc logout(toServerData) returns(ResponseFromServer)&#123;&#125;&#125;message toServerData&#123;required <span class="hljs-keyword">int32</span> ctype = <span class="hljs-number">1</span>;required <span class="hljs-keyword">string</span> name =<span class="hljs-number">2</span>;    optional bytes httpdata=<span class="hljs-number">3</span>;&#125;message ResponseFromServer &#123;required <span class="hljs-keyword">bool</span> Success=<span class="hljs-number">1</span>;optional bytes tcpData=<span class="hljs-number">2</span>;<span class="hljs-comment">// Errcode int</span>&#125;</code></pre></div><p>用安装的protoc插件生成</p><div class="hljs"><pre><code class="hljs s">protoc --proto_path=/mypath --go_out=plugins=grpc:. *.proto //当前在mypath路径下，用grpc模式生成proto文件</code></pre></div><h3 id="proto解析">proto解析</h3><p>生成 <strong>data.pb.proto</strong></p><p>-----------这些就直接参照手册理解吧-----------------<br>这里还是记录下这厮的解码方式吧，参考</p><blockquote><blockquote><p>《数据密集型系统设计》</p></blockquote></blockquote><p>protobuf的的编码其实跟 thrift的BinaryCompact编码有点相似就拿上面那个例子:</p><div class="hljs"><pre><code class="hljs pb">message toServerData&#123;required int32 ctype = 1<span class="hljs-comment">;</span>required string name =2<span class="hljs-comment">;</span>    optional bytes httpdata=3<span class="hljs-comment">;</span>repeated string data =4<span class="hljs-comment">;</span>&#125;</code></pre></div><ul><li>有个<code>flag</code>开头表明类型</li><li>紧接着数据长度</li><li>紧接着数据本身</li><li><strong>数组</strong>类型<code>protobuf</code>不提供，只是在二进制中连续排序(在第一个元素<code>flag+length+data</code>后面每个元素都是<code>length+data</code>);</li></ul><h2 id="提供的一些连接方式">提供的一些连接方式</h2><p>单向stream双向stream</p>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>To my lover</title>
    <link href="/2019/06/26/ToMyLover/"/>
    <url>/2019/06/26/ToMyLover/</url>
    
    <content type="html"><![CDATA[<p>I was looking for something until met you,So I found the whole world</p>]]></content>
    
    
    
    <tags>
      
      <tag>Love</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang Context</title>
    <link href="/2019/06/21/Go/Context/"/>
    <url>/2019/06/21/Go/Context/</url>
    
    <content type="html"><![CDATA[<p>Golang出色的协程为其增添不少色彩，而Context在协程间的协作，同步发挥了很大的作用</p><a id="more"></a><p><img src="/img/golangContextMascot.jpg" srcset="/img/loading.gif" alt="Context"></p><h2 id="作用">作用</h2><p>开头已经说道，Context主要用于在goroutine之间传递上下文信息，而这些信息包括key-value pair，cancel信号，timeout信号等http包，sql包里面都用到了context，比如http包里面，API可以由外部执行cancel操作，可以设置timeout信号来cancelhttp请求服务如果过慢，则可以用timeout进行释放资源举例：获取商品的默认库存数量等</p><p><strong><em>参考 go在今日头条的实践</em></strong></p><p>另外，之前的<img src="/Concurrency" srcset="/img/loading.gif" alt="Concurrency"> 有谈到协程之前如何同步（比如 channel和select）但如果要共享一些全局变量，或者需要同时被关闭，就可以用context来实现</p><h2 id="源码">源码</h2><p>可以参考官方blog<img src="https://blog.golang.org/context" srcset="/img/loading.gif" alt="context blog"></p><p>整体提供了：</p><table>  <tr>    <th>Name</th>    <th>Type</th>    <th>Usage</th>    <th>Comment</th>  </tr>  <tr>    <td>Context</td>    <td>Interface</td>    <td>Define four methods:<br><br>Deadline() (deadline time.Time, ok bool)<br>Done()&lt;-chan struct{}<br>Err() error<br>Value(key interface{}) interface{}</td>    <td></td>  </tr>  <tr>    <td>emptyCtx</td>    <td>struct</td>    <td>Also define interface, but it's empty</td>    <td></td>  </tr>  <tr>    <td>CancelFunc</td>    <td>func</td>    <td>cancel func</td>    <td></td>  </tr>  <tr>    <td>CancelCtx</td>    <td>struct</td>    <td>mark as cancelable</td>    <td rowspan="3">都有实现自己的方法<br><br><br>Cancel()</td>  </tr>  <tr>    <td>timerCtx</td>    <td>struct</td>    <td>canceled if timeout</td>  </tr>  <tr>    <td>valueCtx</td>    <td>struct</td>    <td>save K-V pair</td>  </tr>  <tr>    <td>Background</td>    <td>func</td>    <td>Background returns a non-nil, empty Context. It is never canceled, has no<br>values, and has no deadline. It is typically used by the main function,<br> initialization, and tests, and as the top-level Context for incoming requests.</td>    <td>返回空的context，常用做top-level context</td>  </tr>  <tr>    <td>TODO</td>    <td>func</td>    <td>TODO returns a non-nil, empty Context.<br>Code should use context.TODO when it' s unclear which Context to use or it is not yet available<br> (because the surrounding function has not yet been extended to accept a Context<br>parameter). TODO is recognized by static analysis tools that determine<br>whether Contexts are propagated correctly in a program.</td>    <td>返回空的context</td>  </tr>  <tr>    <td>WithCancel</td>    <td>func</td>    <td>Based on parent context, generate a cancelable context</td>    <td>基于父context生成可取消context<br>(自然就会调用下面的propagateCancel)</td>  </tr>  <tr>    <td>newCancelCtx</td>    <td>func</td>    <td>create a cancelable context</td>    <td>返回一个CancelCtx</td>  </tr>  <tr>    <td>propagateCancel</td>    <td>func</td>    <td>propagateCancel arranges for child to be canceled</td>    <td>向下传递context的关系</td>  </tr>  <tr>    <td>parentCancelCtx</td>    <td>func </td>    <td>parentCancelCtx follows a chain of parent references until it finds a<br><br>*cancelCtx. This function understands how each of the concrete types in this<br><br>package represents its parent.</td>    <td>找到第一个可取消的父节点</td>  </tr>  <tr>    <td>removeChild</td>    <td>func</td>    <td>remove child</td>    <td>去掉父节点的孩子节点</td>  </tr>  <tr>    <td>init</td>    <td>func</td>    <td>init this package</td>    <td></td>  </tr>  <tr>    <td>WithDeadLine</td>    <td>func</td>    <td>Create a context with deadline</td>    <td rowspan="3">同理，都是为了创建不同功能的context</td>  </tr>  <tr>    <td>WithTimeout</td>    <td>func</td>    <td></td>  </tr>  <tr>    <td>WIthValue</td>    <td>func</td>    <td></td>  </tr>  <tr>    <td></td>    <td></td>    <td></td>    <td></td>  </tr></table><p>context里面的类图：</p><p><img src="/img/go_context.png" srcset="/img/loading.gif" alt="contextClass"></p><p>如上图所示，展开</p><h3 id="interface">Interface</h3><h4 id="context">Context</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> Context <span class="hljs-keyword">interface</span> &#123;<span class="hljs-comment">//deadline会返回 这个context应该被取消的时间， 如果ok==false，指没有deadline设置（即返回deadline的时间或者返回没有设置deadline）</span>Deadline() (deadline time.Time, ok <span class="hljs-keyword">bool</span>)    <span class="hljs-comment">//返回一个关闭的只读channel ，代表着这个context应该被cancel或者到了deadline</span>    Done() &lt;-<span class="hljs-keyword">chan</span> <span class="hljs-keyword">struct</span>&#123;&#125;    <span class="hljs-comment">//channel Done（）关闭后，返回关闭原因</span>Err() error    <span class="hljs-comment">//获取key对应的value值</span>Value(key <span class="hljs-keyword">interface</span>&#123;&#125;) <span class="hljs-keyword">interface</span>&#123;&#125;&#125;</code></pre></div><p>关于 <strong>Done()</strong> 需要注意的是这个是一个<strong>只读</strong> 的 <strong>channel</strong>！</p><ol><li>只有在其被关闭后，才可以从里面读出值， 而且这个值是相应类型的 <strong>零值</strong>，所以goroutine可以在其关闭后读出零值，判断后继续做后面的事情</li><li>其具有关联性，即所有用到这个context的goroutine，一旦有一方关闭了(Done())，其他的也会被关闭</li></ol><h4 id="canceler">Canceler</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// A canceler is a context type that can be canceled directly. The</span><span class="hljs-comment">// implementations are *cancelCtx and *timerCtx.</span><span class="hljs-keyword">type</span> canceler <span class="hljs-keyword">interface</span> &#123;cancel(removeFromParent <span class="hljs-keyword">bool</span>, err error)Done() &lt;-<span class="hljs-keyword">chan</span> <span class="hljs-keyword">struct</span>&#123;&#125;&#125;</code></pre></div><p>源码说的很清楚了，这个接口会被 *cancelCtx和 *timerCtx 实现</p><p>//todo 为啥 Canceler里面有 cancel()，而Context里面 没有呢,是种设计问题吧？context有一些不会用到cancel，比如emptyCtx？</p><h3 id="struct">struct</h3><h4 id="emptyctx">emptyCtx</h4><p>这个暂时略过，只要知道把这个当成一个占位符即可，有些函数可能以后会用到context，暂时把它当作一个参数传进去而已它的相关方法有background() 和 TODO()</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// An emptyCtx is never canceled, has no values, and has no deadline. It is not</span><span class="hljs-comment">// struct&#123;&#125;, since vars of this type must have distinct addresses.</span></code></pre></div><h4 id="cancelctx">cancelCtx</h4><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// A cancelCtx can be canceled. When canceled, it also cancels any children</span><span class="hljs-comment">// that implement canceler.</span><span class="hljs-keyword">type</span> cancelCtx <span class="hljs-keyword">struct</span> &#123;Contextmu       sync.Mutex            <span class="hljs-comment">// protects following fields</span>done     <span class="hljs-keyword">chan</span> <span class="hljs-keyword">struct</span>&#123;&#125;         <span class="hljs-comment">// created lazily, closed by first cancel call</span>children <span class="hljs-keyword">map</span>[canceler]<span class="hljs-keyword">struct</span>&#123;&#125; <span class="hljs-comment">// set to nil by the first cancel call</span>err      error                 <span class="hljs-comment">// set to non-nil by the first cancel call</span>&#125;</code></pre></div><p>这个cancelCtx 有实现了接口 <strong>Context</strong>我们注意到源码中说到 done是 created lazily，即这个不是初始化就会有;发现这个在其下的Done()方法里面才会初始化：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *cancelCtx)</span> <span class="hljs-title">Done</span><span class="hljs-params">()</span> &lt;-<span class="hljs-title">chan</span> <span class="hljs-title">struct</span></span>&#123;&#125; &#123;c.mu.Lock()<span class="hljs-keyword">if</span> c.done == <span class="hljs-literal">nil</span> &#123;c.done = <span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">struct</span>&#123;&#125;)<span class="hljs-comment">//这里</span>&#125;d := c.donec.mu.Unlock()<span class="hljs-keyword">return</span> d&#125;</code></pre></div><p>前面说到 这个 <strong>&lt;-chan struct{}</strong> 指的是只读 channel，在其他地方如果读取的话（还没关闭）会block住</p><p>紧接着**cancel()**方法</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// cancel closes c.done, cancels each of c's children, and, if</span><span class="hljs-comment">// removeFromParent is true, removes c from its parent's children.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *cancelCtx)</span> <span class="hljs-title">cancel</span><span class="hljs-params">(removeFromParent <span class="hljs-keyword">bool</span>, err error)</span></span> &#123;<span class="hljs-keyword">if</span> err == <span class="hljs-literal">nil</span> &#123;<span class="hljs-built_in">panic</span>(<span class="hljs-string">"context: internal error: missing cancel error"</span>)&#125;c.mu.Lock()<span class="hljs-keyword">if</span> c.err != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">//前面说道err是当timeout或者cancel的时候会添加，所以有err就一定是被取消了（var Canceled = errors.New("context canceled")）</span>c.mu.Unlock()<span class="hljs-keyword">return</span> <span class="hljs-comment">// already canceled</span>&#125;c.err = err<span class="hljs-keyword">if</span> c.done == <span class="hljs-literal">nil</span> &#123;c.done = closedchan <span class="hljs-comment">//var closedchan = make(chan struct&#123;&#125;)</span>&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-built_in">close</span>(c.done)&#125;<span class="hljs-keyword">for</span> child := <span class="hljs-keyword">range</span> c.children &#123;<span class="hljs-comment">//loop所有的children，每个都调用cancel（）</span><span class="hljs-comment">// <span class="hljs-doctag">NOTE:</span> acquiring the child's lock while holding parent's lock.//为啥要锁住呢，golang里面没有可重入锁，所以child和parent的锁都是分开的</span>child.cancel(<span class="hljs-literal">false</span>, err)&#125;c.children = <span class="hljs-literal">nil</span><span class="hljs-comment">//把children字段设为nil</span>c.mu.Unlock()<span class="hljs-keyword">if</span> removeFromParent &#123;<span class="hljs-comment">//从父context移除自己</span>removeChild(c.Context, c)&#125;&#125;</code></pre></div><p>源码注释写着</p><ol><li>cancel会关闭掉 done这个channel，还会cancel掉c的所有children</li><li>如果removeFromparent 是true，最后调用removeChild()</li></ol><h5 id="cancel-方法的流程：">Cancel（）方法的流程：</h5><ol><li>关闭c.Done() channel，然后不断地cancel它的子节点</li><li>并从父节点移除自己（removeFromParent）</li></ol><p>自然地，我们先看看removeChild()函数</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// removeChild removes a context from its parent.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">removeChild</span><span class="hljs-params">(parent Context, child canceler)</span></span> &#123;p, ok := parentCancelCtx(parent)<span class="hljs-comment">//拿出父context，这里是p</span><span class="hljs-keyword">if</span> !ok &#123;<span class="hljs-keyword">return</span>&#125;p.mu.Lock()<span class="hljs-keyword">if</span> p.children != <span class="hljs-literal">nil</span> &#123;<span class="hljs-built_in">delete</span>(p.children, child)<span class="hljs-comment">//直接从map里面删除这个child（删除自己）</span>&#125;p.mu.Unlock()&#125;</code></pre></div><p><del>PS：里面的parentCancelCtx（）后来发现问题出在 valueCtx上面，我们放在 <a href>valueCtx</a> 讲 </del></p><h5 id="withcancel-流程：">WithCancel（）流程：</h5><p>我们回到上面的<strong>cancel()</strong> 方法，输入的removeFromParent什么时候是true or false呢，全局查找后发现在**WithCancel()**里面有用到，同时这也是一个Export出去（大写）的方法，目的是创建一个可cancel的Context：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// A CancelFunc tells an operation to abandon its work.</span><span class="hljs-comment">// A CancelFunc does not wait for the work to stop.</span><span class="hljs-comment">// After the first call, subsequent calls to a CancelFunc do nothing.</span><span class="hljs-keyword">type</span> CancelFunc <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span>// <span class="hljs-title">WithCancel</span> <span class="hljs-title">returns</span> <span class="hljs-title">a</span> <span class="hljs-title">copy</span> <span class="hljs-title">of</span> <span class="hljs-title">parent</span> <span class="hljs-title">with</span> <span class="hljs-title">a</span> <span class="hljs-title">new</span> <span class="hljs-title">Done</span> <span class="hljs-title">channel</span>. <span class="hljs-title">The</span> <span class="hljs-title">returned</span>// <span class="hljs-title">context</span>'<span class="hljs-title">s</span> <span class="hljs-title">Done</span> <span class="hljs-title">channel</span> <span class="hljs-title">is</span> <span class="hljs-title">closed</span> <span class="hljs-title">when</span> <span class="hljs-title">the</span> <span class="hljs-title">returned</span> <span class="hljs-title">cancel</span> <span class="hljs-title">function</span> <span class="hljs-title">is</span> <span class="hljs-title">called</span>// <span class="hljs-title">or</span> <span class="hljs-title">when</span> <span class="hljs-title">the</span> <span class="hljs-title">parent</span> <span class="hljs-title">context</span>'<span class="hljs-title">s</span> <span class="hljs-title">Done</span> <span class="hljs-title">channel</span> <span class="hljs-title">is</span> <span class="hljs-title">closed</span>, <span class="hljs-title">whichever</span> <span class="hljs-title">happens</span> <span class="hljs-title">first</span>.//// <span class="hljs-title">Canceling</span> <span class="hljs-title">this</span> <span class="hljs-title">context</span> <span class="hljs-title">releases</span> <span class="hljs-title">resources</span> <span class="hljs-title">associated</span> <span class="hljs-title">with</span> <span class="hljs-title">it</span>, <span class="hljs-title">so</span> <span class="hljs-title">code</span> <span class="hljs-title">should</span>// <span class="hljs-title">call</span> <span class="hljs-title">cancel</span> <span class="hljs-title">as</span> <span class="hljs-title">soon</span> <span class="hljs-title">as</span> <span class="hljs-title">the</span> <span class="hljs-title">operations</span> <span class="hljs-title">running</span> <span class="hljs-title">in</span> <span class="hljs-title">this</span> <span class="hljs-title">Context</span> <span class="hljs-title">complete</span>.<span class="hljs-title">func</span> <span class="hljs-title">WithCancel</span><span class="hljs-params">(parent Context)</span> <span class="hljs-params">(ctx Context, cancel CancelFunc)</span></span> &#123;c := newCancelCtx(parent)<span class="hljs-comment">//这个就是复制parent</span>propagateCancel(parent, &amp;c)<span class="hljs-keyword">return</span> &amp;c, <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123; c.cancel(<span class="hljs-literal">true</span>, Canceled) &#125;&#125;</code></pre></div><ol><li>传入parent Context（一般是background），返回一个parentcontext的复制，这个parent contex有 <strong>新的Done channel</strong></li><li>这个新（复制）的Done channel会在两种情况被关闭（不管哪个先发生）：返回的cancel CancelFunc （复制的） 被called （注意，<strong>CancelFunc只能被调用一次，接下来的都会do nothing</strong>）parent context的 Done channel被关闭</li></ol><p><strong>那么删除前该怎么办（调用返回的cancel CancelFunc前），直接断开当前context和其parent的链接？</strong>不行，还要把当前context的children全部cancel掉：这里走到propagateCancel()函数</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// propagateCancel arranges for child to be canceled when parent is.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">propagateCancel</span><span class="hljs-params">(parent Context, child canceler)</span></span> &#123;<span class="hljs-keyword">if</span> parent.Done() == <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">//没有初始化操作，即没有调用Done()操作，所以自然不存在cancel</span><span class="hljs-keyword">return</span> <span class="hljs-comment">// parent is never canceled</span>&#125;<span class="hljs-keyword">if</span> p, ok := parentCancelCtx(parent); ok &#123;p.mu.Lock()<span class="hljs-keyword">if</span> p.err != <span class="hljs-literal">nil</span> &#123;            <span class="hljs-comment">// parent has already been canceled</span>            child.cancel(<span class="hljs-literal">false</span>, p.err)<span class="hljs-comment">//这里传入的就是false，因为父context已经被cancel掉了（父与当前child的链接断开），只需要把child和其下面的</span>&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-keyword">if</span> p.children == <span class="hljs-literal">nil</span> &#123;p.children = <span class="hljs-built_in">make</span>(<span class="hljs-keyword">map</span>[canceler]<span class="hljs-keyword">struct</span>&#123;&#125;)&#125;p.children[child] = <span class="hljs-keyword">struct</span>&#123;&#125;&#123;&#125;&#125;p.mu.Unlock()&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//如果没有找到parent context（它自己是空的），就新建一个goroutine监视parent context或者child context的done信号</span><span class="hljs-keyword">go</span> <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-keyword">select</span> &#123;<span class="hljs-keyword">case</span> &lt;-parent.Done():<span class="hljs-comment">//如果找不到父节点，这个就不会调用</span>child.cancel(<span class="hljs-literal">false</span>, parent.Err())<span class="hljs-keyword">case</span> &lt;-child.Done():<span class="hljs-comment">//可能父节点取消了，这个会重复让子节点再取消一次</span>&#125;&#125;()&#125;&#125;</code></pre></div><p>从源码知道，propagateCancel()主要目的是当parent被cancel，把child给取消;（跟<strong>cancel（）方法重叠？？？</strong>）会不断的传播传播下去，把parent的children字段全部设为空的struct</p><h4 id="parentcancelctx特殊性">parentCancelCtx特殊性</h4><p>之前说到的在这个parentCancelCtx() for循环里面，万一把当前context嵌套在一个struct里面，</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// parentCancelCtx follows a chain of parent references until it finds a</span><span class="hljs-comment">// *cancelCtx. This function understands how each of the concrete types in this</span><span class="hljs-comment">// package represents its parent.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">parentCancelCtx</span><span class="hljs-params">(parent Context)</span> <span class="hljs-params">(*cancelCtx, <span class="hljs-keyword">bool</span>)</span></span> &#123;<span class="hljs-keyword">for</span> &#123;<span class="hljs-comment">//很奇怪？？？为啥要for呢？？？</span><span class="hljs-keyword">switch</span> c := parent.(<span class="hljs-keyword">type</span>) &#123;<span class="hljs-keyword">case</span> *cancelCtx:<span class="hljs-keyword">return</span> c, <span class="hljs-literal">true</span><span class="hljs-keyword">case</span> *timerCtx:<span class="hljs-keyword">return</span> &amp;c.cancelCtx, <span class="hljs-literal">true</span><span class="hljs-keyword">case</span> *valueCtx:parent = c.Context<span class="hljs-comment">//for的问题在这里</span><span class="hljs-keyword">default</span>:<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, <span class="hljs-literal">false</span>&#125;&#125;&#125;</code></pre></div><h4 id="valuectx">valueCtx</h4><p>结构体：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// A valueCtx carries a key-value pair. It implements Value for that key and</span><span class="hljs-comment">// delegates all other calls to the embedded Context.</span><span class="hljs-keyword">type</span> valueCtx <span class="hljs-keyword">struct</span> &#123;Contextkey, val <span class="hljs-keyword">interface</span>&#123;&#125;&#125;</code></pre></div><p>一个简单的kv结构，但是一个ctx只支持一对kv，多了的话会构成一颗<strong>树型</strong>的结构</p><p>首先可以确认它是一个<strong>Context</strong> ，它的独立方法只有两个，其他都是继承context：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *valueCtx)</span> <span class="hljs-title">String</span><span class="hljs-params">()</span> <span class="hljs-title">string</span></span> &#123;<span class="hljs-keyword">return</span> fmt.Sprintf(<span class="hljs-string">"%v.WithValue(%#v, %#v)"</span>, c.Context, c.key, c.val)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *valueCtx)</span> <span class="hljs-title">Value</span><span class="hljs-params">(key <span class="hljs-keyword">interface</span>&#123;&#125;)</span> <span class="hljs-title">interface</span></span>&#123;&#125; &#123;<span class="hljs-keyword">if</span> c.key == key &#123;<span class="hljs-keyword">return</span> c.val&#125;<span class="hljs-keyword">return</span> c.Context.Value(key)&#125;</code></pre></div><p>Value（）的方法是取出对应key的value，但返回值是 <strong>c.Context.Value(key)</strong>，明显是递归调用，其会一直往它的parent context查找，key是否等于输入的key，一直到终点（background）前面也说到 <strong>background=new（emptyCtx）</strong>，所以在终点调用的其实是**emptyCtx.Value()**返回的是nil值</p><p>Export出去的创建一个valueCtx的方法：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// WithValue returns a copy of parent in which the value associated with key is</span><span class="hljs-comment">// val.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Use context Values only for request-scoped data that transits processes and</span><span class="hljs-comment">// APIs, not for passing optional parameters to functions.</span><span class="hljs-comment">//这里很明白地说明了，context值是被设计为在进程间或者API间request的存储值，而不是为了传入一些可选的参数给函数</span><span class="hljs-comment">// The provided key must be comparable and should not be of type</span><span class="hljs-comment">// string or any other built-in type to avoid collisions between</span><span class="hljs-comment">// packages using context. Users of WithValue should define their own</span><span class="hljs-comment">// types for keys. </span><span class="hljs-comment">//还要注意这里，WithValue的key必须是可比较的，不能是string或者其他built-in type，目的是避免不同用的context包的冲突</span><span class="hljs-comment">//To avoid allocating when assigning to an</span><span class="hljs-comment">// interface&#123;&#125;, context keys often have concrete type</span><span class="hljs-comment">// struct&#123;&#125;. Alternatively, exported context key variables' static</span><span class="hljs-comment">// type should be a pointer or interface.</span><span class="hljs-comment">//为了避免？？？ context的key一般是有确切的类型struct，或者是exported出去的key静态类型为指针或者interface</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">WithValue</span><span class="hljs-params">(parent Context, key, val <span class="hljs-keyword">interface</span>&#123;&#125;)</span> <span class="hljs-title">Context</span></span> &#123;<span class="hljs-keyword">if</span> key == <span class="hljs-literal">nil</span> &#123;<span class="hljs-built_in">panic</span>(<span class="hljs-string">"nil key"</span>)&#125;<span class="hljs-keyword">if</span> !reflect.TypeOf(key).Comparable() &#123;<span class="hljs-built_in">panic</span>(<span class="hljs-string">"key is not comparable"</span>)&#125;<span class="hljs-keyword">return</span> &amp;valueCtx&#123;parent, key, val&#125;&#125;</code></pre></div><p>WithValue（）适用范围：<strong>只用与在request作用域内的数据，这种数据是在进程或API传输所用，而不是作为函数的parameter使用</strong></p><p>WithValue（）规定了：</p><ol><li>其key不应该是string或者其他内置的类型，主要是为了防止使用context包的其他包之间产生冲突</li><li>所以key应该是用户自己设置的，要自己覆盖Comparable（）方法才行这里给出 **Comparable()**的源码解释</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Methods applicable only to some types, depending on Kind.</span><span class="hljs-comment">// The methods allowed for each kind are:</span><span class="hljs-comment">//</span><span class="hljs-comment">//Int*, Uint*, Float*, Complex*: Bits</span><span class="hljs-comment">//Array: Elem, Len</span><span class="hljs-comment">//Chan: ChanDir, Elem</span><span class="hljs-comment">//Func: In, NumIn, Out, NumOut, IsVariadic.</span><span class="hljs-comment">//Map: Key, Elem</span><span class="hljs-comment">//Ptr: Elem</span><span class="hljs-comment">//Slice: Elem</span>    <span class="hljs-comment">//Struct: Field, FieldByIndex, FieldByName, FieldByNameFunc, NumField</span></code></pre></div><ol start="3"><li>为了防止赋值给interface{}，context的key经常是有具体类型的struct，此外，export出去的key的静态类型应该是指针或者interface</li></ol><h5 id="流程">流程</h5><ol><li>判断key是否为空，key的类型是否合法</li><li>返回一个有parent指针的valueCtx;所以，创建valueCtx是可以从parent开始一级一级往下创建(树)，如下图：<img src="/img/valueCtx.png" srcset="/img/loading.gif" alt="valueCtx"></li></ol><h4 id="timerctx">timerCtx</h4><p>结构体：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// A timerCtx carries a timer and a deadline. It embeds a cancelCtx to</span><span class="hljs-comment">// implement Done and Err. It implements cancel by stopping its timer then</span><span class="hljs-comment">// delegating to cancelCtx.cancel.</span><span class="hljs-keyword">type</span> timerCtx <span class="hljs-keyword">struct</span> &#123;cancelCtxtimer *time.Timer <span class="hljs-comment">// Under cancelCtx.mu.</span>deadline time.Time&#125;</code></pre></div><p>它的cancel方法</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(c *timerCtx)</span> <span class="hljs-title">cancel</span><span class="hljs-params">(removeFromParent <span class="hljs-keyword">bool</span>, err error)</span></span> &#123;c.cancelCtx.cancel(<span class="hljs-literal">false</span>, err)<span class="hljs-keyword">if</span> removeFromParent &#123;<span class="hljs-comment">// Remove this timerCtx from its parent cancelCtx's children.</span>removeChild(c.cancelCtx.Context, c)&#125;c.mu.Lock()<span class="hljs-keyword">if</span> c.timer != <span class="hljs-literal">nil</span> &#123;c.timer.Stop()c.timer = <span class="hljs-literal">nil</span>&#125;c.mu.Unlock()&#125;</code></pre></div><p>流程：</p><ol><li>先调用里面的cancelCtx的cancel（）方法，取消其子节点</li><li>如果要</li></ol><p>哪里用到timerCtx.cancel（）呢，我们看看创建一个timerCtx的函数：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// WithTimeout returns WithDeadline(parent, time.Now().Add(timeout)).</span><span class="hljs-comment">//</span><span class="hljs-comment">// Canceling this context releases resources associated with it, so code should</span><span class="hljs-comment">// call cancel as soon as the operations running in this Context complete:</span><span class="hljs-comment">//</span><span class="hljs-comment">// func slowOperationWithTimeout(ctx context.Context) (Result, error) &#123;</span><span class="hljs-comment">// ctx, cancel := context.WithTimeout(ctx, 100*time.Millisecond)</span><span class="hljs-comment">// defer cancel()  // releases resources if slowOperation completes before timeout elapses</span><span class="hljs-comment">// return slowOperation(ctx)</span><span class="hljs-comment">// &#125;</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">WithTimeout</span><span class="hljs-params">(parent Context, timeout time.Duration)</span> <span class="hljs-params">(Context, CancelFunc)</span></span> &#123;<span class="hljs-keyword">return</span> WithDeadline(parent, time.Now().Add(timeout))&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// WithDeadline returns a copy of the parent context with the deadline adjusted</span><span class="hljs-comment">// to be no later than d. If the parent's deadline is already earlier than d,</span><span class="hljs-comment">// WithDeadline(parent, d) is semantically equivalent to parent. The returned</span><span class="hljs-comment">// context's Done channel is closed when the deadline expires, when the returned</span><span class="hljs-comment">// cancel function is called, or when the parent context's Done channel is</span><span class="hljs-comment">// closed, whichever happens first.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Canceling this context releases resources associated with it, so code should</span><span class="hljs-comment">// call cancel as soon as the operations running in this Context complete.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">WithDeadline</span><span class="hljs-params">(parent Context, d time.Time)</span> <span class="hljs-params">(Context, CancelFunc)</span></span> &#123;<span class="hljs-keyword">if</span> cur, ok := parent.Deadline(); ok &amp;&amp; cur.Before(d) &#123;<span class="hljs-comment">// The current deadline is already sooner than the new one.</span><span class="hljs-keyword">return</span> WithCancel(parent)&#125;c := &amp;timerCtx&#123;cancelCtx: newCancelCtx(parent),deadline:  d,&#125;propagateCancel(parent, c)dur := time.Until(d)<span class="hljs-keyword">if</span> dur &lt;= <span class="hljs-number">0</span> &#123;c.cancel(<span class="hljs-literal">true</span>, DeadlineExceeded) <span class="hljs-comment">// deadline has already passed</span><span class="hljs-keyword">return</span> c, <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123; c.cancel(<span class="hljs-literal">true</span>, Canceled) &#125;&#125;c.mu.Lock()<span class="hljs-keyword">defer</span> c.mu.Unlock()<span class="hljs-keyword">if</span> c.err == <span class="hljs-literal">nil</span> &#123;c.timer = time.AfterFunc(dur, <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123;c.cancel(<span class="hljs-literal">true</span>, DeadlineExceeded)&#125;)&#125;<span class="hljs-keyword">return</span> c, <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123; c.cancel(<span class="hljs-literal">true</span>, Canceled) &#125;&#125;</code></pre></div><p>我们发现WithTimeout和WithDeadline（）都可以创建timerCtx区别： withTimeout第二个参数是传入duration，即距离现在的时间，withDeadline第二个参数指的是绝对时间，即几时几分</p><p>withDeadline（）流程：</p><ol><li><p>检查当前的deadline，如果存在且当前deadline比传入的时间要早，那么就退化成WithCancel（）</p></li><li><p>如果不存在deadline或者传入时间较晚，把当前timerCtx加入到parent Context里面;但是注意，还要计算现在的时间是否大于了传入的时间，如果大于说明已经过了传入的deadline，直接退化成cancel（），并传入exceed错误信息：</p> <div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">var</span> DeadlineExceeded error = deadlineExceededError&#123;&#125;<span class="hljs-keyword">type</span> deadlineExceededError <span class="hljs-keyword">struct</span>&#123;&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(deadlineExceededError)</span> <span class="hljs-title">Error</span><span class="hljs-params">()</span> <span class="hljs-title">string</span></span>   &#123; <span class="hljs-keyword">return</span> <span class="hljs-string">"context deadline exceeded"</span> &#125;</code></pre></div></li><li><p>如果上述都不成立，调用time.AfterFunc()，规定时间后调用cancel()</p></li></ol><h3 id="使用例子">使用例子</h3><p>这个经典的goroutine泄漏：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">repeatGen</span><span class="hljs-params">()</span> &lt;-<span class="hljs-title">chan</span> <span class="hljs-title">int</span></span>&#123;    c:=<span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">int</span>)    <span class="hljs-keyword">go</span> <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span>&#123;        <span class="hljs-keyword">for</span> i:=<span class="hljs-number">0</span>;;i++&#123;            c&lt;-i        &#125;    &#125;()    <span class="hljs-keyword">return</span> c&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    <span class="hljs-keyword">for</span> v:=<span class="hljs-keyword">range</span> repeatGen()&#123;        fmt.Println(v)        <span class="hljs-keyword">if</span> v==<span class="hljs-number">3</span>&#123;            <span class="hljs-keyword">break</span>        &#125;    &#125;&#125;</code></pre></div><p>当v==3的时候，break出来，但这时候repeatGen里面的goroutine仍然在跑，不会被终止,goroutine发生泄漏</p><p>用context改进：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">repeatGen</span><span class="hljs-params">(ctx context.Context)</span> &lt;-<span class="hljs-title">chan</span> <span class="hljs-title">int</span></span>&#123;    c:=<span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">int</span>)    <span class="hljs-keyword">go</span> <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span>&#123;        <span class="hljs-keyword">for</span> i:<span class="hljs-number">0</span>;;i++&#123;            <span class="hljs-keyword">select</span> &#123;                <span class="hljs-keyword">case</span> &lt;-ctx.Done():<span class="hljs-comment">//等待done信号</span>                    <span class="hljs-keyword">return</span>                <span class="hljs-keyword">case</span> c&lt;-i:            &#125;        &#125;    &#125;&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    ctx,cancelFunc:=WithCancel(context.Background())    <span class="hljs-keyword">defer</span> cancelFunc()<span class="hljs-comment">//最后无论怎么样也要调用确保一定能关掉（为保万一而已，不一定要）</span>    <span class="hljs-keyword">for</span> v:=<span class="hljs-keyword">range</span> repeatGen(ctx)&#123;        fmt.Println(v)        <span class="hljs-keyword">if</span> v==<span class="hljs-number">3</span>&#123;            cancelFunc()<span class="hljs-comment">//完成直接调用cancel</span>            <span class="hljs-keyword">break</span>        &#125;    &#125;&#125;</code></pre></div><p><strong><em>参考《go语言圣经》</em></strong></p>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Fmt 输出包</title>
    <link href="/2019/06/21/Go/fmt/"/>
    <url>/2019/06/21/Go/fmt/</url>
    
    <content type="html"><![CDATA[<p>留个坑----</p><a id="more"></a><p>关于这个包，其实有不少很不错的实现</p><p>比如：fmt.Output()</p><h3 id="pool">pool</h3><p>存储暂时的对象，这些对象很可能会被存储以及调用</p><p>使用sync/pool，创建了一个动态长度的buffer，可以根据需求扩展以及收缩,但是不适用于短时间生存的</p><h2 id="sprintf">Sprintf</h2><p>直接看一道题</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> yoyo <span class="hljs-keyword">struct</span>&#123;    Name <span class="hljs-keyword">string</span>&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(y *yoyo)</span> <span class="hljs-title">do</span><span class="hljs-params">()</span> <span class="hljs-title">string</span></span>&#123;    <span class="hljs-keyword">return</span> fmt.Sprintf(<span class="hljs-string">"%+v"</span>,y)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    p:=&amp;yoyo&#123;&#125;    p.do()&#125;</code></pre></div><p>答案是输出stack overflow</p><p>二话不说直接dlv进去看:贴图先不给，大概流程是,fmt.sprintf里面会调用doPrintf(format,a)这个函数就是loop  format里面的字符串（即我们传入的 “%+v”)遇到% ， +v这种特殊字符就停止 调用</p><div class="hljs"><pre><code class="hljs go">printArgs(arg ,verb <span class="hljs-keyword">rune</span>)</code></pre></div><p>这个函数会进行一个简单的判断，arg的类型，有些类型如果用cast (T.(type)这种模式)可以转换，则不需要使用反射，如果要用反射，即上面题目中，传入的指针y;进入 handleMethods(verb rune)</p><ol><li>判断是不是formatter？</li><li>判断这个verb的类型(v,s,x,X,q)中的一种，如果是，会继续判断传入的p.arg.(type)是否是error还是一个Stringer</li></ol><p>进入p.fmtString(v.String(),verb),其中v=p.arg.(type)实际就是最一开始传入的参数 <strong>y</strong></p><p>这时候就会发现，这个v.String()就又回到最初的yoyo.String()方法，就会造成死循环，栈溢出;</p>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Something should be noted in golang&#39;s map</title>
    <link href="/2019/06/19/Go/SortedMap/"/>
    <url>/2019/06/19/Go/SortedMap/</url>
    
    <content type="html"><![CDATA[<p>There is <strong>SortedMap</strong> or <strong>LinkedHashMap</strong> in JavaBoth can ensure the inserted order of elements, but f ** k myself, I found that Go <strong>only supports basic HashMap</strong>which means you have to do SortedMap and LinkedHashMap by yourself.</p><div class="hljs"><pre><code>                                ----From a damn interview test</code></pre></div><a id="more"></a><p>I met a problem in an interview test:</p><p>Given a file containing URLs on every line,you must record all of their occurence times, then save it into a new fileformat like:</p><div class="hljs"><pre><code>url1  2timesurl2  3timesurl3  4times</code></pre></div><p>But it should be noticed that all urls in new file have the same sequence as those in source files;</p><p>其实你要解决三个问题：</p><ol><li>url频次统计（map）</li><li>要按顺序放入新文件？？？//todo</li><li>文件太大怎么办（拆乘多个文件，进行映射，再统计，但这里又不知道怎么保证顺序？？！！难倒只有手动实现LinkedHashMap<del>红黑树</del>不一定是红黑树，但是LinkedHashMap是一定要实现的<a href="LinkedHashMap%E5%AE%9E%E7%8E%B0">这里跳到LinkedHashMap</a>）</li></ol><p>Go blog <a href="https://blog.golang.org/go-maps-in-action" target="_blank" rel="noopener">map in action</a> says:</p><blockquote><blockquote><blockquote><p>When iterating over a map with a range loop, the iteration order is not specified and is not guaranteed to be the same from one iteration to the next. Since the release of Go 1.0, the runtime has randomized map iteration order.</p></blockquote></blockquote></blockquote><h3 id="sortedmap">SortedMap</h3><p>Someone has already implemented itFrom Go blog  <a href="https://blog.golang.org/go-maps-in-action" target="_blank" rel="noopener">map in action</a></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">import</span> <span class="hljs-string">"sort"</span><span class="hljs-keyword">var</span> m <span class="hljs-keyword">map</span>[<span class="hljs-keyword">int</span>]<span class="hljs-keyword">string</span><span class="hljs-keyword">var</span> keys []<span class="hljs-keyword">int</span><span class="hljs-keyword">for</span> k := <span class="hljs-keyword">range</span> m &#123;<span class="hljs-comment">//extract all the damn key out</span>    keys = <span class="hljs-built_in">append</span>(keys, k)&#125;sort.Ints(keys)<span class="hljs-comment">//sort them</span><span class="hljs-keyword">for</span> _, k := <span class="hljs-keyword">range</span> keys &#123;<span class="hljs-comment">//then you can get a sorted map</span>    fmt.Println(<span class="hljs-string">"Key:"</span>, k, <span class="hljs-string">"Value:"</span>, m[k])&#125;</code></pre></div><p>If your <strong>key</strong> is not an int or string or you wanna sort it by yourself:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//we assume a struct is like below:</span><span class="hljs-keyword">type</span> Obj <span class="hljs-keyword">struct</span> &#123;Title    <span class="hljs-keyword">string</span>Sequence <span class="hljs-keyword">int</span>Size     <span class="hljs-keyword">int</span>&#125;<span class="hljs-comment">//</span><span class="hljs-keyword">type</span> By <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(o1, o2 *Obj)</span> <span class="hljs-title">bool</span>//<span class="hljs-title">ObjSorter</span> <span class="hljs-title">joins</span> <span class="hljs-title">a</span> <span class="hljs-title">By</span> <span class="hljs-title">function</span> <span class="hljs-title">and</span> <span class="hljs-title">a</span> <span class="hljs-title">slice</span> <span class="hljs-title">of</span> <span class="hljs-title">Objs</span> <span class="hljs-title">to</span> <span class="hljs-title">be</span> <span class="hljs-title">sorted</span>.<span class="hljs-title">type</span> <span class="hljs-title">ObjSorter</span> <span class="hljs-title">struct</span></span> &#123;<span class="hljs-comment">//to wrap the obj and by function into this struct</span>objs []Objby   <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(o1, o2 *Obj)</span> <span class="hljs-title">bool</span>&#125;<span class="hljs-title">func</span> <span class="hljs-params">(by By)</span> <span class="hljs-title">Sort</span><span class="hljs-params">(objs []Obj)</span></span> &#123;os := &amp;ObjSorter&#123;objs: objs,by:   by,&#125;sort.Sort(os)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(s *ObjSorter)</span> <span class="hljs-title">Less</span><span class="hljs-params">(i, j <span class="hljs-keyword">int</span>)</span> <span class="hljs-title">bool</span></span> &#123;<span class="hljs-keyword">return</span> s.by(&amp;s.objs[i], &amp;s.objs[j])&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(s *ObjSorter)</span> <span class="hljs-title">Swap</span><span class="hljs-params">(i, j <span class="hljs-keyword">int</span>)</span></span> &#123;s.objs[i], s.objs[j] = s.objs[j], s.objs[i]&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(s *ObjSorter)</span> <span class="hljs-title">Len</span><span class="hljs-params">()</span> <span class="hljs-title">int</span></span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-built_in">len</span>(s.objs)&#125;</code></pre></div><p>Then to use it:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">SortedMap</span><span class="hljs-params">(m <span class="hljs-keyword">map</span>[Obj]<span class="hljs-keyword">int</span>)</span></span> &#123;sortByTitle := <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(o1, o2 *Obj)</span> <span class="hljs-title">bool</span></span> &#123; <span class="hljs-comment">//sort by Ttitle field</span><span class="hljs-keyword">return</span> o1.Title &lt; o2.Title&#125;<span class="hljs-comment">// sortBySequence := func(o1, o2 *Obj) bool &#123; //sortBy Sequence field</span><span class="hljs-comment">// return o1.Sequence &lt; o2.Sequence</span><span class="hljs-comment">// &#125;</span>tempObjs := <span class="hljs-built_in">make</span>([]Obj, <span class="hljs-number">0</span>)<span class="hljs-keyword">for</span> k := <span class="hljs-keyword">range</span> m &#123;tempObjs = <span class="hljs-built_in">append</span>(tempObjs, k)&#125;By(sortByTitle).Sort(tempObjs)<span class="hljs-comment">//Sorted this</span><span class="hljs-keyword">for</span> _, k := <span class="hljs-keyword">range</span> tempObjs &#123;<span class="hljs-comment">//operate this </span>fmt.Printf(<span class="hljs-string">"key:%v,value:%v \n"</span>, k, m[k])&#125;&#125;</code></pre></div><p>To test:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    obj1 := &amp;Obj&#123;Title: <span class="hljs-string">"4"</span>&#125;obj2 := &amp;Obj&#123;Title: <span class="hljs-string">"2"</span>&#125;obj3 := &amp;Obj&#123;Title: <span class="hljs-string">"1"</span>&#125;objs := <span class="hljs-built_in">make</span>([]*Obj, <span class="hljs-number">0</span>)objs = <span class="hljs-built_in">append</span>(objs, obj1, obj2, obj3)m := <span class="hljs-built_in">make</span>(<span class="hljs-keyword">map</span>[Obj]<span class="hljs-keyword">int</span>)m[*obj1] = <span class="hljs-number">1</span>m[*obj3] = <span class="hljs-number">1</span>m[*obj2] = <span class="hljs-number">1</span>SortedMap(m)&#125;</code></pre></div><p>Result:</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-string">key:</span>&#123;<span class="hljs-number">1</span> <span class="hljs-number">0</span> <span class="hljs-number">0</span>&#125;,<span class="hljs-string">value:</span><span class="hljs-number">1</span> <span class="hljs-string">key:</span>&#123;<span class="hljs-number">2</span> <span class="hljs-number">0</span> <span class="hljs-number">0</span>&#125;,<span class="hljs-string">value:</span><span class="hljs-number">1</span> <span class="hljs-string">key:</span>&#123;<span class="hljs-number">4</span> <span class="hljs-number">0</span> <span class="hljs-number">0</span>&#125;,<span class="hljs-string">value:</span><span class="hljs-number">1</span></code></pre></div><p>Above example can be found in <a href="https://golang.org/pkg/sort/" target="_blank" rel="noopener">Golang slice doc examples</a>傻了，写到这里才发现，sortedMap有个鸡用，不符合这道题的要求啊！！！</p><h3 id="linkedhashmap">LinkedHashMap</h3><p>To just ensure <strong>inserted order</strong>:you can refer to <a href>Java LinkedHashMap</a>//or arrayMap()</p><p><strong>But</strong>here we won’t achieve the LinkedHashMap, it’s kind of troublesome…</p><p>We do a trade-off between time and space:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//just create a slice to save sequence</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">CalcUrls</span><span class="hljs-params">()</span></span>&#123;    sequence:=<span class="hljs-built_in">make</span>([]<span class="hljs-keyword">string</span>,<span class="hljs-number">0</span>)    m:=<span class="hljs-built_in">make</span>(<span class="hljs-keyword">map</span>[<span class="hljs-keyword">string</span>]<span class="hljs-keyword">int</span>)    <span class="hljs-comment">//.....</span>        url:=fp.ReadLine()    <span class="hljs-keyword">if</span> _,exists:=m[url];exists&#123;        m[url]++    &#125;<span class="hljs-keyword">else</span>&#123;        sequence=<span class="hljs-built_in">append</span>(sequence,url)    &#125;    <span class="hljs-comment">//....</span>    <span class="hljs-keyword">for</span> _,v:=<span class="hljs-keyword">range</span> sequence&#123;        res:=fmt.Sprintf(<span class="hljs-string">"url:%v,times:%v"</span>,v,m[v])        Writer.write(res)<span class="hljs-comment">//write into the new file</span>    &#125;        <span class="hljs-comment">//....</span>&#125;</code></pre></div><p><strong>---------------------------------新增---------------------------------------------</strong>主要是红黑树是为了能便于删除和添加，而我遇到的面试题只考虑添加，所以没必要，但是隔了一天，还是觉得不爽</p><p>自己尝试实现一下基于红黑树的map看看，就当做复习吧：</p><p>首先，红黑树也是AVL树，区别就只是颜色上的判断不同而已，所以AVL的rotate我这里略过，主要就是颜色的判断要符合</p><ol><li>每个节点是黑或者红</li><li>根节点是黑色</li><li>每个叶子节点是黑色（且为nil，添加的时候没用到，删除的时候会用）</li><li>如果一个节点是红色的，它的子节点必须是黑色的</li><li><strong>从一个节点到该节点的所有叶子节点的路径上包含相同数目的黑节点</strong> （这个一行代码可以搞定，但之后很可能会违反第4，所以要紧接着进行处理）</li></ol><h5 id="节点插入时候-检查：">节点插入时候，检查：</h5><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//主要思路，先像AVL一样插入，之后再调整</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(brtree *BRTree)</span> <span class="hljs-title">insertNode</span><span class="hljs-params">(pnode *BRNode)</span></span> &#123;tempRoot := brtree.root<span class="hljs-keyword">var</span> temp *BRNode<span class="hljs-keyword">for</span> tempRoot != <span class="hljs-literal">nil</span> &#123; <span class="hljs-comment">//已有根节点，就往下loop找到插入点</span>temp = tempRoot <span class="hljs-comment">//每次更新</span><span class="hljs-keyword">if</span> pnode.val &gt; tempRoot.val &#123;tempRoot = tempRoot.right&#125; <span class="hljs-keyword">else</span> &#123;tempRoot = tempRoot.left&#125;&#125;pnode.parent = temp <span class="hljs-comment">//找到最后，把pnode的parent指向找到的最后一个点</span><span class="hljs-keyword">if</span> temp != <span class="hljs-literal">nil</span> &#123;    <span class="hljs-comment">//不是根节点</span><span class="hljs-keyword">if</span> temp.val &lt; pnode.val &#123; <span class="hljs-comment">//判断放在左子树还是右子树</span>temp.right = pnode&#125; <span class="hljs-keyword">else</span> &#123;temp.left = pnode&#125;&#125; <span class="hljs-keyword">else</span> &#123; <span class="hljs-comment">//根节点</span>brtree.root = pnode&#125;pnode.color = RED <span class="hljs-comment">//直接把这个颜色先设置为红色！！！主要为了满足：</span><span class="hljs-comment">// 从一个节点到该节点的所有叶子节点的路径上包含相同数目的黑节点（可以试想一下，最下方插入一个红色节点，因为目前红色节点的叶子节点肯定是黑色的nil，所以必会满足这个特性）</span>brtree.insertCheck(pnode) <span class="hljs-comment">//再检查，因为这个时候可能跟上面的第4点违背了，比如在红色的节点下面插一个红色的节点</span>&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//插入时进行检查</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(brtree *BRTree)</span> <span class="hljs-title">insertCheck</span><span class="hljs-params">(pnode *BRNode)</span></span> &#123;<span class="hljs-comment">//1. 该节点没有父节点，即为root（与上面insertNode插入点的检查并不重复，因为接下来会进行递归，这个属于边界条件，所以这个检查root必不可少）</span><span class="hljs-keyword">if</span> pnode.parent == <span class="hljs-literal">nil</span> &#123;brtree.root = pnodebrtree.root.color = BLACK<span class="hljs-keyword">return</span>&#125;<span class="hljs-comment">//2.父节点是黑色直接添加(不用管)，红色接着处理</span><span class="hljs-keyword">if</span> pnode.parent.color == RED &#123;<span class="hljs-comment">//2.1 父，叔叔节点不为空而且其颜色是红色 ,则将该父叔都改为黑色,将祖父改成红色</span><span class="hljs-comment">/*  10,B /     \    6,B    15,B   /   \     /  \4,R   8,R  11,R  19,R/pnode,R==&gt;父叔节点都变成黑色，祖父变成红色  10,B /     \    6,R    15,B   /   \     /  \4,B   8,B  11,R  19,R             /  pnode,R==&gt;递归，从祖父（6）开始，因为此时父节点属于根节点，是黑色，所以完成           10,B /     \    6,R    15,B   /   \     /  \4,B   8,B  11,R  19,R             /  pnode,R*/</span><span class="hljs-keyword">if</span> pnode.getUncle() != <span class="hljs-literal">nil</span> &amp;&amp; pnode.getUncle().color == RED &#123;pnode.parent.color = BLACKpnode.getUncle().color = BLACKpnode.getGrandParent().color = REDbrtree.insertCheck(pnode.getGrandParent()) <span class="hljs-comment">//对该树上的节点递归上去处理</span>&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">//2.2 父节点为红色，叔叔节点不存在或者是黑色</span>isLeft := pnode == pnode.parent.leftisParentLeft := pnode.parent == pnode.getGrandParent().left<span class="hljs-keyword">if</span> isLeft &amp;&amp; isParentLeft &#123;<span class="hljs-comment">//2.2.1 是左子树，其父节点也是左子树</span><span class="hljs-comment">/*       ...  /        8,B    /  \   7,R   /nil  /pnode,R*/</span><span class="hljs-comment">//=》</span><span class="hljs-comment">/*         /     7,B         /   \     pnode,R  8,R               \             /nil*/</span>pnode.parent.color = BLACKpnode.getGrandParent().color = REDbrtree.rotateRight(pnode.getGrandParent())&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> !isLeft &amp;&amp; !isParentLeft &#123;<span class="hljs-comment">//2.2.2 不是左子树，父亲也不是左子树</span><span class="hljs-comment">/*  7,B      /  \2,B/nil   8,R           \           10,pnode*/</span>pnode.parent.color = BLACKpnode.getGrandParent().color = REDbrtree.rotateLeft(pnode.getGrandParent())&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> isLeft &amp;&amp; !isParentLeft &#123;<span class="hljs-comment">//2.2.3 是左子树，但父亲不是左子树</span><span class="hljs-comment">/*8,B    /  \6,B/nil   10,R        /      pnode，R*/</span><span class="hljs-comment">/*pnode,R    /  \6,B/nil  8,B            \            10,R*/</span>brtree.rotateRight(pnode.parent)<span class="hljs-comment">//实际上变成了2.2.2</span>brtree.rotateLeft(pnode.parent) <span class="hljs-comment">//pnode现在在原先pnode的父节点位置</span>pnode.color = BLACKpnode.left.color = REDpnode.right.color = RED&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> !isLeft &amp;&amp; isParentLeft &#123;<span class="hljs-comment">//2.2.4不是左子树，但父亲是左子树</span><span class="hljs-comment">/* \        8,B    /  \   7,R   10,B/nil      \      pnode，R*/</span><span class="hljs-comment">//==&gt;</span><span class="hljs-comment">/* 8，B    /    \  pnode,R  10,B/nil  /7,R*/</span><span class="hljs-comment">/* pnode，R    /    \   8,B   10,B/nil  /7,R*/</span>brtree.rotateLeft(pnode.parent)<span class="hljs-comment">//实际上变成了2.2.1</span>brtree.rotateRight(pnode.parent)pnode.color = BLACKpnode.left.color = REDpnode.right.color = RED&#125;&#125;&#125;&#125;</code></pre></div><h5 id="节点的删除">节点的删除</h5><p>删除节点的时候也分几种情况先把它当作普通二叉查找树处理：</p><ol><li>删除的节点没有子节点，直接删除</li><li>被删除节点只有一个孩子，删除这个节点，并用其孩子代替这个节点</li><li>被删除节点有两个孩子：找出后继节点，用后继节点内容复制给该节点，删除后继节点，再将后继节点删除;就可以考虑后继节点了，被删除的节点（刚刚的后继节点）有两个非空子节点情况下，它的后继节点不可能两个子节点不是空的，意思就是后继节点只有一个或零个儿子，就可以按前面的1,2进行处理</li></ol><p>接着就是按照红黑树的颜色规则来改变和旋转了//todo我实在写不动了</p><div class="hljs"><pre><code class="hljs go"></code></pre></div><p>那么把红黑树放在map上呢：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//todo</span></code></pre></div><h4 id="linkedhashmap实现">LinkedHashMap实现</h4><p>真正的实现，先讲一下：</p><ol><li>定义数据结构node，这个是链表上的每个节点，应该有prev,next,key，val这些</li><li>定义整个map，应该是整个链表lmap，应该有指向头部，尾部的指针，map[key]node,length,</li></ol><p><strong>---------------------------------end-----------------------------------------------</strong>The above code may use O(2n) space, but time complexity reduce to O(n)<strong>!!!Everything seems done!!!</strong></p><p>So what if we use a <strong>large file?</strong></p><h3 id="large-file">Large File</h3><p>Solution: Split the large file into several files, it depends on your MEM:针对大文件，其实普通的思想就是&quot;分而治之&quot;，更多可以参考MapReduce</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">HashFile</span><span class="hljs-params">(url <span class="hljs-keyword">string</span>)</span> <span class="hljs-title">int</span></span>&#123;<span class="hljs-comment">//pass a url</span>    seed:=<span class="hljs-number">131</span>    hash:=<span class="hljs-number">0</span>    <span class="hljs-keyword">for</span> _,v:=<span class="hljs-keyword">range</span> url&#123;        hash=hash*seed+v    &#125;    <span class="hljs-keyword">return</span> hash &amp; <span class="hljs-number">0x7FFFFFFF</span>&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    <span class="hljs-comment">//........</span>    <span class="hljs-comment">//url:=fp.ReadLine()</span>    pos:=HashFile(url)    file:=pos%<span class="hljs-number">10</span><span class="hljs-comment">//we assume that it split into 10 files</span>    bufio.newWriter(file+<span class="hljs-string">".txt"</span>)    Writer.Write(file+<span class="hljs-string">".txt"</span>)<span class="hljs-comment">//write it into specific file</span>    <span class="hljs-comment">//........</span>    urls:=<span class="hljs-built_in">make</span>([]<span class="hljs-keyword">string</span>,<span class="hljs-number">0</span>)    <span class="hljs-keyword">for</span> i:=<span class="hljs-number">0</span>;i&lt;<span class="hljs-number">10</span>;i++&#123;        urlPlusTimes:=LoopEachFile(files)<span class="hljs-comment">//calculate each files' urls</span>        urls=<span class="hljs-built_in">append</span>(urls,urlPlusTimes...)<span class="hljs-comment">// Here comes to the question??? how to ensure sequence???</span>    &#125;&#125;</code></pre></div><h3 id="最终解决办法">最终解决办法</h3><p>几天了，想不出来啊</p><ol><li>主要是拆开多个文件的话，如果每一条记录都是unqiue的，map也要O（n）复杂度，内存会爆</li></ol><p>//todo</p><h3 id="addition">Addition</h3><h4 id="unsafe">unsafe</h4><p>map in golang is <strong>not thread-safe</strong>;So it involves with concurrency situation, you must add <strong>sync.Mutex</strong> or <strong>sync.RWMutex</strong>About that, can refer to <a href="/Concurrency">my previous blog</a></p>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Golang Mux</title>
    <link href="/2019/06/08/Go/GolangMux/"/>
    <url>/2019/06/08/Go/GolangMux/</url>
    
    <content type="html"><![CDATA[<p>开始做笔记…(之前面试被怼了)</p><a id="more"></a><h2 id="路由匹配原理-regexp-in-router">路由匹配原理(Regexp in Router)</h2><p>First, let’s go through its *** usage ***:</p><div class="hljs"><pre><code class="hljs go">router:=mux.router()router.HandlerFunc(<span class="hljs-string">"/api/v&#123;version&#125;/&#123;category&#125;"</span>,handler)</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//then in handler:</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">handler</span><span class="hljs-params">(w http.ResponseWriter,r *http.request)</span></span>&#123;    vars:=mux.Vars(r)<span class="hljs-comment">//Get the variables!</span>    <span class="hljs-comment">//...</span>&#125;</code></pre></div><p>Let’s cut it into two parts:</p><h3 id="handlerfunc-path-handler">HandlerFunc(path,handler)</h3><div class="hljs"><pre><code class="hljs golang"><span class="hljs-comment">// HandleFunc registers a new route with a matcher for the URL path.</span><span class="hljs-comment">// See Route.Path() and Route.HandlerFunc().</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(r *Router)</span> <span class="hljs-title">HandleFunc</span><span class="hljs-params">(path <span class="hljs-keyword">string</span>, f <span class="hljs-keyword">func</span>(http.ResponseWriter,    *http.Request)</span>) *<span class="hljs-title">Route</span></span> &#123;    <span class="hljs-keyword">return</span> r.NewRoute().Path(path).HandlerFunc(f)&#125;</code></pre></div><p>then let’s see  *** Route.Path(path) *** :</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Path -----------------------------------------------------------------------</span><span class="hljs-comment">// Path adds a matcher for the URL path.</span><span class="hljs-comment">// It accepts a template with zero or more URL variables enclosed by &#123;&#125;. The</span><span class="hljs-comment">// template must start with a "/".</span><span class="hljs-comment">// Variables can define an optional regexp pattern to be matched:</span><span class="hljs-comment">//</span><span class="hljs-comment">// - &#123;name&#125; matches anything until the next slash.</span><span class="hljs-comment">//</span><span class="hljs-comment">// - &#123;name:pattern&#125; matches the given regexp pattern.</span><span class="hljs-comment">//</span><span class="hljs-comment">// For example:</span><span class="hljs-comment">//</span><span class="hljs-comment">//     r := mux.NewRouter()</span><span class="hljs-comment">//     r.Path("/products/").Handler(ProductsHandler)</span><span class="hljs-comment">//     r.Path("/products/&#123;key&#125;").Handler(ProductsHandler)</span><span class="hljs-comment">//     r.Path("/articles/&#123;category&#125;/&#123;id:[0-9]+&#125;").</span><span class="hljs-comment">//       Handler(ArticleHandler)</span><span class="hljs-comment">//</span><span class="hljs-comment">// Variable names must be unique in a given route. They can be retrieved</span><span class="hljs-comment">// calling mux.Vars(request).</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(r *Route)</span> <span class="hljs-title">Path</span><span class="hljs-params">(tpl <span class="hljs-keyword">string</span>)</span> *<span class="hljs-title">Route</span></span> &#123;<span class="hljs-comment">//根据传入的url增加正则匹配</span>r.err = r.addRegexpMatcher(tpl, regexpTypePath)<span class="hljs-keyword">return</span> r&#125;</code></pre></div><p>In *** r.err = r.addRegexpMatcher(tpl, regexpTypePath) ***the *** regexpTypePath=0 *** means this is for matching Path instead of Host/Port/Prefix/Query</p><p>Continue, Go into method *** addRegexpMatcher() ***:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// addRegexpMatcher adds a host or path matcher and builder to a route.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(r *Route)</span> <span class="hljs-title">addRegexpMatcher</span><span class="hljs-params">(tpl <span class="hljs-keyword">string</span>, typ regexpType)</span> <span class="hljs-title">error</span></span> &#123;<span class="hljs-keyword">if</span> r.err != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> r.err&#125;<span class="hljs-keyword">if</span> typ == regexpTypePath || typ == regexpTypePrefix &#123;<span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(tpl) &gt; <span class="hljs-number">0</span> &amp;&amp; tpl[<span class="hljs-number">0</span>] != <span class="hljs-string">'/'</span> &#123;<span class="hljs-keyword">return</span> fmt.Errorf(<span class="hljs-string">"mux: path must start with a slash, got %q"</span>, tpl)&#125;<span class="hljs-keyword">if</span> r.regexp.path != <span class="hljs-literal">nil</span> &#123;tpl = strings.TrimRight(r.regexp.path.template, <span class="hljs-string">"/"</span>) + tpl&#125;&#125;rr, err := newRouteRegexp(tpl, typ, routeRegexpOptions&#123;<span class="hljs-comment">// 核心部分</span>strictSlash:    r.strictSlash,useEncodedPath: r.useEncodedPath,&#125;)<span class="hljs-keyword">if</span> err != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> err&#125;<span class="hljs-keyword">for</span> _, q := <span class="hljs-keyword">range</span> r.regexp.queries &#123;<span class="hljs-keyword">if</span> err = uniqueVars(rr.varsN, q.varsN); err != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> err&#125;&#125;<span class="hljs-keyword">if</span> typ == regexpTypeHost &#123;<span class="hljs-keyword">if</span> r.regexp.path != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">if</span> err = uniqueVars(rr.varsN, r.regexp.path.varsN); err != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> err&#125;&#125;r.regexp.host = rr&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-keyword">if</span> r.regexp.host != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">if</span> err = uniqueVars(rr.varsN, r.regexp.host.varsN); err != <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">return</span> err&#125;&#125;<span class="hljs-keyword">if</span> typ == regexpTypeQuery &#123;r.regexp.queries = <span class="hljs-built_in">append</span>(r.regexp.queries, rr)&#125; <span class="hljs-keyword">else</span> &#123;r.regexp.path = rr&#125;&#125;r.addMatcher(rr)<span class="hljs-comment">//这里会把当前matcher加入matcher slice</span><span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>&#125;</code></pre></div><p>We will find that we only enter the first *** if *** block (from line 4 - line 11):Getinto the *** newRouteRegexp() ***</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// newRouteRegexp parses a route template and returns a routeRegexp,</span><span class="hljs-comment">// used to match a host, a path or a query string.</span><span class="hljs-comment">//</span><span class="hljs-comment">// It will extract named variables, assemble a regexp to be matched, create</span><span class="hljs-comment">// a "reverse" template to build URLs and compile regexps to validate variable</span><span class="hljs-comment">// values used in URL building.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Previously we accepted only Python-like identifiers for variable</span><span class="hljs-comment">// names ([a-zA-Z_][a-zA-Z0-9_]*), but currently the only restriction is that</span><span class="hljs-comment">// name and pattern can't be empty, and names can't contain a colon.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">newRouteRegexp</span><span class="hljs-params">(tpl <span class="hljs-keyword">string</span>, typ regexpType, options routeRegexpOptions)</span> <span class="hljs-params">(*routeRegexp, error)</span></span> &#123;    <span class="hljs-comment">//....</span>    <span class="hljs-keyword">return</span> ......&#125;</code></pre></div><p>I will roughly split the above code into serveral parts as below:</p><h4 id="procedure">Procedure :</h4><ol><li>根据括号筛选变量 (braceIndices)It will check the all *** {variable} *** in Path and save them into a slice:</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// braceIndices returns the first level curly brace indices from a string.</span><span class="hljs-comment">// It returns an error in case of unbalanced braces.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">braceIndices</span><span class="hljs-params">(s <span class="hljs-keyword">string</span>)</span> <span class="hljs-params">([]<span class="hljs-keyword">int</span>, error)</span></span> &#123;<span class="hljs-keyword">var</span> level, idx <span class="hljs-keyword">int</span><span class="hljs-keyword">var</span> idxs []<span class="hljs-keyword">int</span><span class="hljs-keyword">for</span> i := <span class="hljs-number">0</span>; i &lt; <span class="hljs-built_in">len</span>(s); i++ &#123;<span class="hljs-keyword">switch</span> s[i] &#123;<span class="hljs-keyword">case</span> <span class="hljs-string">'&#123;'</span>:<span class="hljs-keyword">if</span> level++; level == <span class="hljs-number">1</span> &#123;idx = i&#125;<span class="hljs-keyword">case</span> <span class="hljs-string">'&#125;'</span>:<span class="hljs-keyword">if</span> level--; level == <span class="hljs-number">0</span> &#123;idxs = <span class="hljs-built_in">append</span>(idxs, idx, i+<span class="hljs-number">1</span>)<span class="hljs-comment">//注意：这里只是存了variable开始和结尾的index，而不是string</span>&#125; <span class="hljs-keyword">else</span> <span class="hljs-keyword">if</span> level &lt; <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, fmt.Errorf(<span class="hljs-string">"mux: unbalanced braces in %q"</span>, s)&#125;&#125;&#125;<span class="hljs-keyword">if</span> level != <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, fmt.Errorf(<span class="hljs-string">"mux: unbalanced braces in %q"</span>, s)&#125;<span class="hljs-keyword">return</span> idxs, <span class="hljs-literal">nil</span>&#125;</code></pre></div><p>====&gt;</p><ol start="2"><li>移除斜杠 (Remove endSlash)</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">if</span> has Suffix():    tpl = tpl[:<span class="hljs-built_in">len</span>(tpl)<span class="hljs-number">-1</span>]<span class="hljs-comment">//remove end slash</span></code></pre></div><p>====&gt;</p><ol start="3"><li>traverse idxs(variables)</li></ol><div class="hljs"><pre><code class="hljs go">varsN := <span class="hljs-built_in">make</span>([]<span class="hljs-keyword">string</span>, <span class="hljs-built_in">len</span>(idxs)/<span class="hljs-number">2</span>)<span class="hljs-comment">//多少个变量</span>varsR := <span class="hljs-built_in">make</span>([]*regexp.Regexp, <span class="hljs-built_in">len</span>(idxs)/<span class="hljs-number">2</span>)<span class="hljs-comment">//多少个变量的匹配</span>pattern := bytes.NewBufferString(<span class="hljs-string">""</span>)pattern.WriteByte(<span class="hljs-string">'^'</span>)reverse := bytes.NewBufferString(<span class="hljs-string">""</span>)<span class="hljs-keyword">var</span> end <span class="hljs-keyword">int</span><span class="hljs-keyword">var</span> err error<span class="hljs-keyword">for</span> i := <span class="hljs-number">0</span>; i &lt; <span class="hljs-built_in">len</span>(idxs); i += <span class="hljs-number">2</span> &#123;<span class="hljs-comment">//因为之前说过idxs里面是index,+=2实际是遍历下一个变量</span>    <span class="hljs-comment">// Set all values we are interested in.</span>    raw := tpl[end:idxs[i]]<span class="hljs-comment">//括号之前的string</span>    end = idxs[i+<span class="hljs-number">1</span>]<span class="hljs-comment">//指的是closed bracket</span>    parts := strings.SplitN(tpl[idxs[i]+<span class="hljs-number">1</span>:end<span class="hljs-number">-1</span>], <span class="hljs-string">":"</span>, <span class="hljs-number">2</span>)<span class="hljs-comment">//把变量给弄出来,不过可能有&#123;name:pattern&#125;这种情况，所以要分割开</span>    name := parts[<span class="hljs-number">0</span>]    patt := defaultPattern <span class="hljs-comment">//defaultPattern是 [^/]+,即匹配‘/’多次</span>    <span class="hljs-keyword">if</span> <span class="hljs-built_in">len</span>(parts) == <span class="hljs-number">2</span> &#123;        patt = parts[<span class="hljs-number">1</span>]<span class="hljs-comment">//如果有检测到：，就用后面的正则</span>    &#125;    <span class="hljs-comment">// Name or pattern can't be empty.</span>    <span class="hljs-keyword">if</span> name == <span class="hljs-string">""</span> || patt == <span class="hljs-string">""</span> &#123;        <span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, fmt.Errorf(<span class="hljs-string">"mux: missing name or pattern in %q"</span>,            tpl[idxs[i]:end])    &#125;    <span class="hljs-comment">// Build the regexp pattern.</span>    fmt.Fprintf(pattern, <span class="hljs-string">"%s(?P&lt;%s&gt;%s)"</span>, regexp.QuoteMeta(raw), varGroupName(i/<span class="hljs-number">2</span>), patt)    <span class="hljs-comment">// Build the reverse template.</span>    fmt.Fprintf(reverse, <span class="hljs-string">"%s%%s"</span>, raw)    <span class="hljs-comment">// Append variable name and compiled pattern.</span>    varsN[i/<span class="hljs-number">2</span>] = name<span class="hljs-comment">//！！！！这个就是变量的存储位置</span>    varsR[i/<span class="hljs-number">2</span>], err = regexp.Compile(fmt.Sprintf(<span class="hljs-string">"^%s$"</span>, patt))    <span class="hljs-keyword">if</span> err != <span class="hljs-literal">nil</span> &#123;        <span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>, err    &#125;&#125;</code></pre></div><p>返回后回到 *<em>func (r <em>Route) addRegexpMatcher(tpl string, typ regexpType) error</em></em> func，这个函数最后会把match加入matchers里面</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// addMatcher adds a matcher to the route.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(r *Route)</span> <span class="hljs-title">addMatcher</span><span class="hljs-params">(m matcher)</span> *<span class="hljs-title">Route</span></span> &#123;<span class="hljs-keyword">if</span> r.err == <span class="hljs-literal">nil</span> &#123;r.matchers = <span class="hljs-built_in">append</span>(r.matchers, m)&#125;<span class="hljs-keyword">return</span> r&#125;</code></pre></div><ol start="4"><li></li></ol><h3 id="vars">Vars()</h3><p>The place used by mux to save variables called *** context *** (上下文)And we have to talk about the *** context in ‘net/http’ ***:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//这个是mux里面的context</span><span class="hljs-comment">// ----------------------------------------------------------------------------</span><span class="hljs-comment">// Context </span><span class="hljs-comment">// ----------------------------------------------------------------------------</span><span class="hljs-comment">// RouteMatch stores information about a matched route.</span><span class="hljs-keyword">type</span> RouteMatch <span class="hljs-keyword">struct</span> &#123;Route   *RouteHandler http.HandlerVars    <span class="hljs-keyword">map</span>[<span class="hljs-keyword">string</span>]<span class="hljs-keyword">string</span><span class="hljs-comment">// MatchErr is set to appropriate matching error</span><span class="hljs-comment">// It is set to ErrMethodMismatch if there is a mismatch in</span><span class="hljs-comment">// the request method and route method</span>MatchErr error&#125;<span class="hljs-keyword">type</span> contextKey <span class="hljs-keyword">int</span><span class="hljs-keyword">const</span> (varsKey contextKey = <span class="hljs-literal">iota</span>routeKey)<span class="hljs-comment">// Vars returns the route variables for the current request, if any.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">Vars</span><span class="hljs-params">(r *http.Request)</span> <span class="hljs-title">map</span>[<span class="hljs-title">string</span>]<span class="hljs-title">string</span></span> &#123;<span class="hljs-keyword">if</span> rv := contextGet(r, varsKey); rv != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">//这里会从request的context里面获取key-value pair，然后转换成map</span><span class="hljs-keyword">return</span> rv.(<span class="hljs-keyword">map</span>[<span class="hljs-keyword">string</span>]<span class="hljs-keyword">string</span>)&#125;<span class="hljs-keyword">return</span> <span class="hljs-literal">nil</span>&#125;</code></pre></div><p>Then go into *** contextGet(r,varsKey) ***:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//"mux"</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">contextGet</span><span class="hljs-params">(r *http.Request, key <span class="hljs-keyword">interface</span>&#123;&#125;)</span> <span class="hljs-title">interface</span></span>&#123;&#125; &#123;<span class="hljs-keyword">return</span> r.Context().Value(key)<span class="hljs-comment">//发现用的是net/http里面的context</span>&#125;</code></pre></div><p>Context struct in *** “net/http” ***可以参考自己写的一个<a href="/Context.html">context</a></p><p>一切到最后，handleFunc()会调用ServeHTTP（w,req），里面就会把RouteMatch的vars导出，放入http包里的context里</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// ServeHTTP dispatches the handler registered in the matched route.</span><span class="hljs-comment">//</span><span class="hljs-comment">// When there is a match, the route variables can be retrieved calling</span><span class="hljs-comment">// mux.Vars(request).</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-params">(r *Router)</span> <span class="hljs-title">ServeHTTP</span><span class="hljs-params">(w http.ResponseWriter, req *http.Request)</span></span> &#123;<span class="hljs-keyword">if</span> !r.skipClean &#123;path := req.URL.Path<span class="hljs-keyword">if</span> r.useEncodedPath &#123;path = req.URL.EscapedPath()&#125;<span class="hljs-comment">// Clean path to canonical form and redirect.</span><span class="hljs-keyword">if</span> p := cleanPath(path); p != path &#123;<span class="hljs-comment">// Added 3 lines (Philip Schlump) - It was dropping the query string and #whatever from query.</span><span class="hljs-comment">// This matches with fix in go 1.2 r.c. 4 for same problem.  Go Issue:</span><span class="hljs-comment">// http://code.google.com/p/go/issues/detail?id=5252</span>url := *req.URLurl.Path = pp = url.String()w.Header().Set(<span class="hljs-string">"Location"</span>, p)w.WriteHeader(http.StatusMovedPermanently)<span class="hljs-keyword">return</span>&#125;&#125;<span class="hljs-keyword">var</span> match RouteMatch<span class="hljs-keyword">var</span> handler http.Handler<span class="hljs-keyword">if</span> r.Match(req, &amp;match) &#123;handler = match.Handlerreq = setVars(req, match.Vars)<span class="hljs-comment">//这里，设置vars</span>req = setCurrentRoute(req, match.Route)&#125;<span class="hljs-keyword">if</span> handler == <span class="hljs-literal">nil</span> &amp;&amp; match.MatchErr == ErrMethodMismatch &#123;handler = methodNotAllowedHandler()&#125;<span class="hljs-keyword">if</span> handler == <span class="hljs-literal">nil</span> &#123;handler = http.NotFoundHandler()&#125;handler.ServeHTTP(w, req)&#125;</code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Something about authentication and authorization</title>
    <link href="/2019/06/05/Comcon/LoginSecurity/"/>
    <url>/2019/06/05/Comcon/LoginSecurity/</url>
    
    <content type="html"><![CDATA[<p>Wat the hack is <a href="https://tools.ietf.org/html/rfc6749" target="_blank" rel="noopener">Oauth2</a>, <a href="https://tools.ietf.org/html/rfc7519" target="_blank" rel="noopener">JWT</a>, <a href="https://tools.ietf.org/html/rfc2818" target="_blank" rel="noopener">HTTPS</a>??? Followings are my notes:</p><a id="more"></a><h2 id="oauth2">Oauth2</h2><h3 id="流程">流程</h3><p>主要流程如图：<img src="/img/OauthProcess.png" srcset="/img/loading.gif" alt="OauthProcess" title="Process"></p><h3 id="四种验证方法">四种验证方法</h3><p><img src="/img/Oauth4Methods.png" srcset="/img/loading.gif" alt="Oauth4method" title="four ways of identification"></p><h2 id="jwt">JWT</h2><p>主要流程(盗图)：<img src="/img/JWT.png" srcset="/img/loading.gif" alt="jwt"></p><h3 id="主要格式">主要格式</h3><div class="hljs"><pre><code>header.payload.signature</code></pre></div><p>header结构：</p><div class="hljs"><pre><code class="hljs undefined">&#123;    <span class="hljs-attr">"typ"</span>: <span class="hljs-string">"JWT"</span>,    <span class="hljs-attr">"alg"</span>: <span class="hljs-string">"HS256"</span>&#125;</code></pre></div><p>payload 用于携带你希望向服务端传递的信息。你既可以往里添加官方字段（这里的“字段” (field) 也可以被称作“声明” claims）例如iss(Issuer签发者)，aud（接受jwt的一方），jti（jwt唯一身份标识，主要作为一次性token）, sub(Subject面向的用户), exp(Expiration time)</p><p>也可以塞入自定义的字段，比如 userId:</p><div class="hljs"><pre><code class="hljs undefined">&#123;    <span class="hljs-attr">"userId"</span>: <span class="hljs-string">"yjqweqw0019-aq"</span>//但一般不要敏感信息，因为会被看见&#125;</code></pre></div><p>Signature结构</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-title">secret</span>=<span class="hljs-string">"mhh12121"</span>//存在服务器上的密钥<span class="hljs-class"><span class="hljs-keyword">data</span> = base64( <span class="hljs-title">header</span> ) + “.” + base64( <span class="hljs-title">payload</span> )</span><span class="hljs-title">signature</span> = <span class="hljs-type">Hash</span>( <span class="hljs-class"><span class="hljs-keyword">data</span>, secret )//这里我们用<span class="hljs-type">HMACSHA256</span></span></code></pre></div><p>假设我们的header 和paylaod如下</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-comment">//header</span>&#123;    <span class="hljs-string">"typ"</span>: <span class="hljs-string">"JWT"</span>,    <span class="hljs-string">"alg"</span>: <span class="hljs-string">"HS256"</span>&#125;<span class="hljs-comment">//payload</span>&#123;    <span class="hljs-string">"userId"</span>: <span class="hljs-string">"yjqweqw0019-aq"</span>&#125;<span class="hljs-comment">//服务器上的secret</span>secret=<span class="hljs-string">"mhh12121"</span></code></pre></div><p>可以通过这个网站<img src="https://jwt.io/" srcset="/img/loading.gif" alt="jwtio">验证你的signature准确性</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-comment">//我们的signature应该是这个</span>eyJhbGciOiJIUzI<span class="hljs-number">1</span><span class="hljs-symbol">NiIsInR5</span>cCI<span class="hljs-number">6</span>IkpX<span class="hljs-attr">VCJ9</span>.eyJ<span class="hljs-number">1</span>c<span class="hljs-number">2</span>VySWQiOiJ<span class="hljs-number">5</span>a<span class="hljs-symbol">nF3</span>ZXF<span class="hljs-number">3</span>MDAxOS<span class="hljs-number">1</span>hcSJ<span class="hljs-number">9.7</span>jRvE_F<span class="hljs-number">8</span>lriMHlTZPJulRPg_V<span class="hljs-number">66</span>I<span class="hljs-number">6</span>r-f<span class="hljs-number">7</span>caMuI<span class="hljs-number">82</span>u<span class="hljs-number">1</span>o</code></pre></div><h2 id="tls-https-http-tls-tls1-2-ssl3-3">TLS (HTTPS=HTTP+TLS，TLS1.2=SSL3.3)</h2><p>TLS (Transport layer security) mainly works in Application Layer. TLS is the 3rd edition of SSL(Secure Socket Layer) so that it can work on any application over TCP.</p><p>It provides securty over serverl aspects:</p><ol><li>encryption(加密性)：指提交的数据不被获得</li><li>data Integerity(数据完整性)：指提交的的订单不被修改</li><li>Port Identification (端点鉴别，包括服务端和客户端)：指提交和接收的两边能互相确认身份</li></ol><p>至于流程验证，为了方便了解，我们先简化SSL：</p><p>举例用Bob和Alice</p><h4 id="第一步：">第一步：</h4><h5 id="握手">握手</h5><p>一、 因为这是在TCP之上，所以先进行TCP链接,Bob发起链接，Alice接收，这里略过</p><p>二、 要验证Alice是真正的Alice：1. Bob会发送一个 SSL 验证报文，,下图是 <strong>SSL hello</strong>报文2. Alice 则用她的 <strong>证书</strong>进行回应，证书中包含了她的公钥3. Bob收到证书，因为 证书被 <strong>CA</strong> 证实过了，所以Bob会知道Alice的真实性4. 如果证书是真的，Bob会产生一个 <strong>主密钥（MS）</strong> 该密钥只用于这个SSL会话</p><p>三、 双方生成共同的密钥：1. Bob会用 Alice的公钥加密这个 <strong>主密钥MS</strong> 生成 <strong>加密的主密钥(EMS)</strong>2. Bob发送 <strong>EMS</strong> 给Alice3. Alice收到 <strong>EMS</strong> 并用自己的 <strong>私钥</strong> 解开得到 <strong>密钥（MS）</strong>，这样双方都有一个共同的<strong>密钥</strong></p><h4 id="第二步：">第二步：</h4><h5 id="密钥导出">密钥导出</h5><p>现在双方有一个共同的密钥，已经可以用作通信，但前面说到，要保证3点加密性，数据完整性，端点鉴别而且对于Alice和Bob每个人来说，这些都用不同的密钥才会更安全</p><p>为此，可以让Alice和Bob各生成4个密钥</p><ol><li>E<sub>B</sub> 指Bob发送到Alice的数据的会话加密密钥</li><li>M<sub>B</sub> 指Bob发送到Alice的数据的会话MAC密钥</li><li>E<sub>A</sub> 指Alice发送到Bob的数据的会话加密密钥</li><li>M<sub>A</sub> 指Alice发送到Bob的数据的会话MAC密钥</li></ol><p>每人通过MS生成4个密钥，其中两个用于加密数据，另外两个用于验证数据完整性</p><h4 id="第三步">第三步</h4><h5 id="数据传输">数据传输</h5><p>但在数据传输中，经过TCP链接后，开始发送数据，一次发送了加密的数据，但另一个用于验证数据完整性的MAC去哪里了呢？我们希望一次性就可以把加密数据和验证完整性的MAC都发送出去（即一次发送即可满足加密性和完整性）</p><p>为了解决这个问题，SSL将数据流分割成 <a href="#SSL-%E8%AE%B0%E5%BD%95">记录</a> ，对每一个记录附加一个MAC用于完整性检查，然后加密 <strong>记录+MAC</strong> ：比如，从Bob发送开始， 产生这个MAC， Bob将数据连同密钥M<sub>B</sub>放入一个hash函数，再用自己CA会话加密密钥 <strong>E<sub>B</sub></strong>,最后再传入TCP</p><p>然而，数据完整性仍然得不到保证，万一有中间人能抓取Bob发送的两个报文段，颠倒它们的顺序，调整TCP报文段的序号(seq)，将两个次序颠倒的报文段发送给Alice，假设TCP报文段刚好封装一个记录（流式传输，所以不保证只含一个），Alice会这样做：</p><ol><li>Alice端运行的TCP认为一切正常，传递这两个记录给SSL子层</li><li>SSL解密这两个记录</li><li>SSL会用每个记录中的MAC来验证完整性</li><li>SSL将解密的两条记录的字节流传给应用层，但实际上因为颠倒了报文，次序不正确。</li></ol><p>解决如上问题，主要就要解决TCP的序号问题，所以就可以自己使用一个<strong>序号计数器</strong>：Bob维护一个序号计数器，这个计数器不在记录中，而在MAC的计算中：即</p><blockquote><blockquote><p>MAC=Hash(数据 + MAC密钥M<sub>B</sub> + 当前序号)</p></blockquote></blockquote><p>这样一来，以上颠倒了两个报文段的顺序，Alice解码发现顺序不对，就不会处理这两个报文</p><h3 id="ssl-记录">SSL 记录</h3><p>该记录如下图所示：<img src="/img/SSL.png" srcset="/img/loading.gif" alt="SSL记录格式"></p><p>主要由类型字段，版本字段，长度字段，数据字段和MAC构成，但其前三个字段是不加密的类型字段：指出是握手报文还是有数据的报文，还被用于关闭SSL连接上（后面说到）</p><h3 id="连接关闭时注意的问题">连接关闭时注意的问题</h3><p>一般来讲，关闭连接就会由Client端，即Bob发起TCP FIN报文请求断开，但这个易遭到截断攻击（truncation attack)：如果中间人过早地发送TCP FIN报文，Server端（Alice）会认为已经收到所有Bob的数据。</p><p>解决方案：SSL记录中的<strong>类型字段</strong>指明这个记录是否用于关闭连接。这里还要注意，虽然这个字段是明文，但接收方仍然可以用记录的MAC对它进行鉴别</p><p>***参考 《计算机网络自顶向下》 ***</p><h3 id="https-一般是443端口">HTTPS（一般是443端口）</h3><p>CA证书包含了：</p><ol><li>序号和过期时间</li><li>姓名</li><li>所有者公钥</li><li>域名</li><li>签发机构</li></ol><p>流程如图<img src="/img/HTTPS.png" srcset="/img/loading.gif" alt="https"></p><h4 id="问题：">问题：</h4><ol><li>那么每次client端生成的key放在哪里呢？</li></ol><p>改变环境变量SSLKEYLOGFILE，浏览器会从以下地址记录生成的对称密钥（linux）</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-builtin-name">export</span> <span class="hljs-attribute">SSLKEYLOGFILE</span>=~/tls/key.log</code></pre></div><ol start="2"><li>被劫持咋办？可以从图上的序号开始谈：</li><li>首先3处，可能 <strong>有中间人拦截server端传去client端的请求吗，然后篡改证书？</strong>如果中间人模拟一个自签名证书：浏览器会把这个自签名证书和系统证书匹配，匹配不上，会失败<br>如果中间人假冒颁发机构颁发证书：因为没有颁发机构的私钥，所以证书指纹不能对上，也会失败</li></ol><p>所以，唯一破解就是用户自己安装了一个未知证书，这样系统会认为中间人证书是信任的</p><ol start="2"><li>接着6 处，即是被拦截，中间人没有server的私钥，无法解开</li></ol><p>防止看不懂，还是新增一副图吧（盗图）：<img src="/img/CA.jpeg" srcset="/img/loading.gif" alt="CA"></p><ol start="3"><li>加密都用了啥？</li></ol><p>而下面client<strong>步骤4</strong>则用的是<strong>对称加密</strong>传输的数据，再用<strong>非对称加密</strong> 加密 这个<strong>经过对称加密的数据</strong>（太绕）</p><h2 id="tls1-3">TLS1.3</h2><p>对比TLS1.2，TLS1.3在速度上有了很大的进步,注意到之前TLS1.2在建立连接时用了4次RTT，即4次握手，但是TLS1.3缩减到了2次</p><h3 id="做的修改">做的修改</h3><h4 id="速度加快">速度加快</h4><h4 id="加密算法删减">加密算法删减</h4>]]></content>
    
    
    
    <tags>
      
      <tag>security</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Channel</title>
    <link href="/2019/06/05/Go/Concurrency/"/>
    <url>/2019/06/05/Go/Concurrency/</url>
    
    <content type="html"><![CDATA[<p>其实里面只涉及部分Channel重点问题，只是暂时做个笔记，仍然有大部分问题需要继续深入，保持持续更新</p><a id="more"></a><h2 id="1-channel">1. Channel</h2><h3 id="互斥性">互斥性</h3><p>要先明白一句话</p><blockquote><blockquote><p>Share memory by communication, do not communicate by sharing memory通过通信来分享内存，而不是靠分享内存来通信</p></blockquote></blockquote><p>这句话应该见过无数遍了，但这就是golang channel的核心思想</p><p>作为channel，顾名思义，就像一个管道一样，主要就是控制数据流向（DataFlow），从而可以控制多个协程间的协作，达到互斥，同步等目的其实际就是一个用于同步和通信的<code>有锁队列</code>（现在版本<code>部分路径</code>无锁的，用环型缓存，有人提出了这个无锁实现，但是因为无法达到FIFO特性，所以被搁置）</p><h3 id="channel部分源码解析">Channel部分源码解析</h3><p>应用上，我们经常用作两个goroutine通信，一个写入，一个读出</p><p>这里可以有无缓冲的channel，和有缓冲的channel，</p><p>无缓冲的写入就<strong>必须</strong>要(立即!!!通常写入会放入到一个goroutine中，且该goroutine要在这之前就入队列)读出，否则立即阻塞（阻塞在写入的地方），读出后也会阻塞（阻塞读出）</p><p>有缓冲的在空的时候会阻塞读出，满之后会阻塞写入</p><p>在调用方（其实可以在任何地方close，但是一般在写入方close才符合设计规范）close掉channel，第二个参数会返回false；</p><p>如果里面仍然有值，可以读出，但是写入会引发panic</p><div class="hljs"><pre><code class="hljs go">x,ok:=&lt;-channel1<span class="hljs-keyword">if</span> !ok&#123;<span class="hljs-comment">//channel1已经被关掉且里面没有数据!!!</span>&#125;</code></pre></div><h4 id="基本数据结构">基本数据结构</h4><ul><li><code>qcount</code>为queue内的所有数据总数</li><li><code>dataqsiz</code>为循环队列的长度</li><li><code>buf</code>为缓冲区数据(循环队列内)的指针</li><li><code>sendx</code>为channel发送操作处理到的位置</li><li><code>recvx</code>为channel接收操作处理到的位置</li><li><code>elementsize</code>和<code>elementtype</code>表示当前channel能够收发的数据大小和类型</li><li><code>sendq</code>和<code>recvq</code>存储了当前channel由于缓冲区不足而阻塞的goroutine list，这些等待列表用双向链表<code>runtime.waitq</code>表示，其中每个元素都是<code>runtime.sudog</code>类别</li><li><code>lock</code>会block住所有字段，包括<code>sudog</code>，当这个lock在被使用时，不能改变其他G的状态!!!否则可能在栈的缩容时会导致死锁？？？</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> hchan <span class="hljs-keyword">struct</span> &#123;qcount   <span class="hljs-keyword">uint</span>           <span class="hljs-comment">// total data in the queue</span>dataqsiz <span class="hljs-keyword">uint</span>           <span class="hljs-comment">// size of the circular queue</span>buf      unsafe.Pointer <span class="hljs-comment">// points to an array of dataqsiz elements</span>elemsize <span class="hljs-keyword">uint16</span>closed   <span class="hljs-keyword">uint32</span>elemtype *_type <span class="hljs-comment">// element type</span>sendx    <span class="hljs-keyword">uint</span>   <span class="hljs-comment">// send index</span>recvx    <span class="hljs-keyword">uint</span>   <span class="hljs-comment">// receive index</span>recvq    waitq  <span class="hljs-comment">// list of recv waiters</span>sendq    waitq  <span class="hljs-comment">// list of send waiters</span><span class="hljs-comment">// lock protects all fields in hchan, as well as several</span><span class="hljs-comment">// fields in sudogs blocked on this channel.</span><span class="hljs-comment">//</span><span class="hljs-comment">// Do not change another G's status while holding this lock</span><span class="hljs-comment">// (in particular, do not ready a G), as this can deadlock</span><span class="hljs-comment">// with stack shrinking.</span>lock mutex&#125;<span class="hljs-keyword">type</span> waitq <span class="hljs-keyword">struct</span> &#123;first *sudoglast  *sudog&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// sudog represents a g in a wait list, such as for sending/receiving</span><span class="hljs-comment">// on a channel.</span><span class="hljs-comment">//</span><span class="hljs-comment">// sudog is necessary because the g ↔ synchronization object relation</span><span class="hljs-comment">// is many-to-many. A g can be on many wait lists, so there may be</span><span class="hljs-comment">// many sudogs for one g; and many gs may be waiting on the same</span><span class="hljs-comment">// synchronization object, so there may be many sudogs for one object.</span><span class="hljs-comment">//</span><span class="hljs-comment">// sudogs are allocated from a special pool. Use acquireSudog and</span><span class="hljs-comment">// releaseSudog to allocate and free them.</span><span class="hljs-keyword">type</span> sudog <span class="hljs-keyword">struct</span> &#123;<span class="hljs-comment">// The following fields are protected by the hchan.lock of the</span><span class="hljs-comment">// channel this sudog is blocking on. shrinkstack depends on</span><span class="hljs-comment">// this for sudogs involved in channel ops.</span>g *g<span class="hljs-comment">// isSelect indicates g is participating in a select, so</span><span class="hljs-comment">// g.selectDone must be CAS'd to win the wake-up race.</span>isSelect <span class="hljs-keyword">bool</span>next     *sudogprev     *sudogelem     unsafe.Pointer <span class="hljs-comment">// data element (may point to stack)</span><span class="hljs-comment">// The following fields are never accessed concurrently.</span><span class="hljs-comment">// For channels, waitlink is only accessed by g.</span><span class="hljs-comment">// For semaphores, all fields (including the ones above)</span><span class="hljs-comment">// are only accessed when holding a semaRoot lock.</span>acquiretime <span class="hljs-keyword">int64</span>releasetime <span class="hljs-keyword">int64</span>ticket      <span class="hljs-keyword">uint32</span>parent      *sudog <span class="hljs-comment">// semaRoot binary tree</span>waitlink    *sudog <span class="hljs-comment">// g.waiting list or semaRoot</span>waittail    *sudog <span class="hljs-comment">// semaRoot</span>c           *hchan <span class="hljs-comment">// channel</span>&#125;</code></pre></div><h4 id="channel的创建">channel的创建</h4><p>开始,<code>make</code>关键字会进入类型检查，将当前<code>OMAKE</code>节点变成<code>OMAKECHAN</code></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">typecheck1</span><span class="hljs-params">(n *Node, top <span class="hljs-keyword">int</span>)</span> <span class="hljs-params">(res *Node)</span></span> &#123;<span class="hljs-keyword">switch</span> n.Op &#123;<span class="hljs-keyword">case</span> OMAKE:...<span class="hljs-keyword">switch</span> t.Etype &#123;<span class="hljs-keyword">case</span> TCHAN:l = <span class="hljs-literal">nil</span><span class="hljs-keyword">if</span> i &lt; <span class="hljs-built_in">len</span>(args) &#123; <span class="hljs-comment">// 带缓冲区的异步 Channel</span>...n.Left = l&#125; <span class="hljs-keyword">else</span> &#123; <span class="hljs-comment">// 不带缓冲区的同步 Channel</span>n.Left = nodintconst(<span class="hljs-number">0</span>)&#125;n.Op = OMAKECHAN<span class="hljs-comment">//变成OMAKECHAN</span>&#125;&#125;&#125;</code></pre></div><p>最终这些<code>OMAKECHAN</code>在SSA中间代码生成之前 变成 <code>runtime.makechan</code>或<code>runtime.makechan64</code>,如下:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">walkexpr</span><span class="hljs-params">(n *Node, init *Nodes)</span> *<span class="hljs-title">Node</span></span> &#123;<span class="hljs-keyword">switch</span> n.Op &#123;<span class="hljs-keyword">case</span> OMAKECHAN:size := n.Leftfnname := <span class="hljs-string">"makechan64"</span>argtype := types.Types[TINT64]<span class="hljs-keyword">if</span> size.Type.IsKind(TIDEAL) || maxintval[size.Type.Etype].Cmp(maxintval[TUINT]) &lt;= <span class="hljs-number">0</span> &#123;fnname = <span class="hljs-string">"makechan"</span>argtype = types.Types[TINT]&#125;n = mkcall1(chanfn(fnname, <span class="hljs-number">1</span>, n.Type), n.Type, init, typename(n.Type), conv(size, argtype))&#125;&#125;</code></pre></div><p>其中<code>runtime.makechan64</code>用于构建缓冲区大于2^32的情况，我们只看<code>makechan</code>先:</p><ul><li>先检查元素大小是否 &gt;=2^16 ,如果是就panic,再检查内存对齐情况;</li><li>如果channel不存在缓冲区,只会为<code>runtime.hchan</code>分配一段内存空间;</li><li>如果channel不存在指针，就会为当前channel和底层的数组分配一块连续的内存空间;</li><li>默认情况会为<code>runtime.hchan</code>和缓冲区分配内存;</li><li>最后更新<code>elementsize</code>,<code>elemtype</code>等字段</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">makechan</span><span class="hljs-params">(t *chantype, size <span class="hljs-keyword">int</span>)</span> *<span class="hljs-title">hchan</span></span> &#123;elem := t.elem<span class="hljs-comment">// compiler checks this but be safe.</span><span class="hljs-keyword">if</span> elem.size &gt;= <span class="hljs-number">1</span>&lt;&lt;<span class="hljs-number">16</span> &#123;throw(<span class="hljs-string">"makechan: invalid channel element type"</span>)&#125;<span class="hljs-keyword">if</span> hchanSize%maxAlign != <span class="hljs-number">0</span> || elem.align &gt; maxAlign &#123;throw(<span class="hljs-string">"makechan: bad alignment"</span>)&#125;mem, overflow := math.MulUintptr(elem.size, <span class="hljs-keyword">uintptr</span>(size))<span class="hljs-keyword">if</span> overflow || mem &gt; maxAlloc-hchanSize || size &lt; <span class="hljs-number">0</span> &#123;<span class="hljs-built_in">panic</span>(plainError(<span class="hljs-string">"makechan: size out of range"</span>))&#125;<span class="hljs-comment">// Hchan does not contain pointers interesting for GC when elements stored in buf do not contain pointers.</span><span class="hljs-comment">// buf points into the same allocation, elemtype is persistent.</span><span class="hljs-comment">// SudoG's are referenced from their owning thread so they can't be collected.</span><span class="hljs-comment">// TODO(dvyukov,rlh): Rethink when collector can move allocated objects.</span><span class="hljs-keyword">var</span> c *hchan<span class="hljs-keyword">switch</span> &#123;<span class="hljs-comment">//不存在缓冲区</span><span class="hljs-keyword">case</span> mem == <span class="hljs-number">0</span>:<span class="hljs-comment">// Queue or element size is zero.</span>c = (*hchan)(mallocgc(hchanSize, <span class="hljs-literal">nil</span>, <span class="hljs-literal">true</span>))<span class="hljs-comment">// Race detector uses this location for synchronization.</span>c.buf = c.raceaddr()<span class="hljs-comment">//不存在指针</span><span class="hljs-keyword">case</span> elem.ptrdata == <span class="hljs-number">0</span>:<span class="hljs-comment">// Elements do not contain pointers.</span><span class="hljs-comment">// Allocate hchan and buf in one call.</span>c = (*hchan)(mallocgc(hchanSize+mem, <span class="hljs-literal">nil</span>, <span class="hljs-literal">true</span>))c.buf = add(unsafe.Pointer(c), hchanSize)<span class="hljs-keyword">default</span>:<span class="hljs-comment">// Elements contain pointers.</span>c = <span class="hljs-built_in">new</span>(hchan)c.buf = mallocgc(mem, elem, <span class="hljs-literal">true</span>)&#125;c.elemsize = <span class="hljs-keyword">uint16</span>(elem.size)c.elemtype = elemc.dataqsiz = <span class="hljs-keyword">uint</span>(size)<span class="hljs-keyword">if</span> debugChan &#123;<span class="hljs-built_in">print</span>(<span class="hljs-string">"makechan: chan="</span>, c, <span class="hljs-string">"; elemsize="</span>, elem.size, <span class="hljs-string">"; dataqsiz="</span>, size, <span class="hljs-string">"\n"</span>)&#125;<span class="hljs-keyword">return</span> c&#125;</code></pre></div><h4 id="数据发送">数据发送</h4><p>发送数据时，会将<code>ch&lt;-i</code>这类语句解析为<code>OSEND</code>节点，并<code>cmd/compile/internal/gc.walkexpr</code>转换成<code>runtime.chansend1</code>:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">walkexpr</span><span class="hljs-params">(n *Node, init *Nodes)</span> *<span class="hljs-title">Node</span></span> &#123;<span class="hljs-keyword">switch</span> n.Op &#123;<span class="hljs-keyword">case</span> OSEND:n1 := n.Rightn1 = assignconv(n1, n.Left.Type.Elem(), <span class="hljs-string">"chan send"</span>)n1 = walkexpr(n1, init)n1 = nod(OADDR, n1, <span class="hljs-literal">nil</span>)n = mkcall1(chanfn(<span class="hljs-string">"chansend1"</span>, <span class="hljs-number">2</span>, n.Left.Type), <span class="hljs-literal">nil</span>, init, n.Left, n1)&#125;&#125;</code></pre></div><p>然后<code>chansend1</code>实际调用<code>runtime.chansend</code>，大致分为几步</p><ul><li>先锁住整个channel</li><li>如果目标channel没有关闭并且已经有读等待中的goroutine<code>recvq</code>,直接call<code>runtime.send</code>将数据发送</li><li>如果缓冲区有空闲空间，将发送的数据写入缓冲区<code>buffer</code>中</li><li>当不存在缓冲区或者缓冲区已满时，等待其他 Goroutine 从 Channel 接收数据</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">/* * generic single channel send/recv * If block is not nil, * then the protocol will not * sleep but return if it could * not complete. * * sleep can wake up with g.param == nil * when a channel involved in the sleep has * been closed.  it is easiest to loop and re-run * the operation; we'll see that it's now closed. */</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">chansend</span><span class="hljs-params">(c *hchan, ep unsafe.Pointer, block <span class="hljs-keyword">bool</span>, callerpc <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-title">bool</span></span> &#123;<span class="hljs-comment">//一些检查，是否已经关闭等等</span>...lock(&amp;c.lock)<span class="hljs-keyword">if</span> c.closed != <span class="hljs-number">0</span> &#123;unlock(&amp;c.lock)<span class="hljs-built_in">panic</span>(plainError(<span class="hljs-string">"send on closed channel"</span>))&#125;&#125;</code></pre></div><h5 id="直接发送">直接发送</h5><p>有等待goroutine，</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//接上面</span><span class="hljs-keyword">if</span> sg := c.recvq.dequeue(); sg != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// Found a waiting receiver. We pass the value we want to send</span><span class="hljs-comment">// directly to the receiver, bypassing the channel buffer (if any).</span>send(c, sg, ep, <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123; unlock(&amp;c.lock) &#125;, <span class="hljs-number">3</span>)<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>&#125;</code></pre></div><p>send函数详细:</p><ul><li>数据非空，直接调用<code>sendDirect</code>，copy到类似<code>x=&lt;-c</code>表达式中的变量x上</li><li>然后解锁当前channel，再调用<code>runtime.goready</code>将等待接收数据的goroutine，标记成<code>Grunnable</code>并将其放到<strong>发送方</strong>的P的<code>p.runnext</code>字段上等待执行(注意这里只是把其放在<code>runnext</code>上，并没有立即执行)该P在下一次调度的时候就会立即唤醒数据的接收方;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// send processes a send operation on an empty channel c.</span><span class="hljs-comment">// The value ep sent by the sender is copied to the receiver sg.</span><span class="hljs-comment">// The receiver is then woken up to go on its merry way.</span><span class="hljs-comment">// Channel c must be empty and locked.  send unlocks c with unlockf.</span><span class="hljs-comment">// sg must already be dequeued from c.</span><span class="hljs-comment">// ep must be non-nil and point to the heap or the caller's stack.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">send</span><span class="hljs-params">(c *hchan, sg *sudog, ep unsafe.Pointer, unlockf <span class="hljs-keyword">func</span>()</span>, <span class="hljs-title">skip</span> <span class="hljs-title">int</span>)</span> &#123;<span class="hljs-comment">//race 的判断</span>...<span class="hljs-keyword">if</span> sg.elem != <span class="hljs-literal">nil</span> &#123;sendDirect(c.elemtype, sg, ep)sg.elem = <span class="hljs-literal">nil</span>&#125;gp := sg.gunlockf()gp.param = unsafe.Pointer(sg)<span class="hljs-keyword">if</span> sg.releasetime != <span class="hljs-number">0</span> &#123;sg.releasetime = cputicks()&#125;goready(gp, skip+<span class="hljs-number">1</span>)&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// Sends and receives on unbuffered or empty-buffered channels are the</span><span class="hljs-comment">// only operations where one running goroutine writes to the stack of</span><span class="hljs-comment">// another running goroutine. The GC assumes that stack writes only</span><span class="hljs-comment">// happen when the goroutine is running and are only done by that</span><span class="hljs-comment">// goroutine. Using a write barrier is sufficient to make up for</span><span class="hljs-comment">// violating that assumption, but the write barrier has to work.</span><span class="hljs-comment">// typedmemmove will call bulkBarrierPreWrite, but the target bytes</span><span class="hljs-comment">// are not in the heap, so that will not help. We arrange to call</span><span class="hljs-comment">// memmove and typeBitsBulkBarrier instead.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">sendDirect</span><span class="hljs-params">(t *_type, sg *sudog, src unsafe.Pointer)</span></span> &#123;<span class="hljs-comment">// src is on our stack, dst is a slot on another stack.</span><span class="hljs-comment">// Once we read sg.elem out of sg, it will no longer</span><span class="hljs-comment">// be updated if the destination's stack gets copied (shrunk).</span><span class="hljs-comment">// So make sure that no preemption points can happen between read &amp; use.</span>dst := sg.elemtypeBitsBulkBarrier(t, <span class="hljs-keyword">uintptr</span>(dst), <span class="hljs-keyword">uintptr</span>(src), t.size)<span class="hljs-comment">// No need for cgo write barrier checks because dst is always</span><span class="hljs-comment">// Go memory.</span>memmove(dst, src, t.size)&#125;</code></pre></div><h5 id="有缓冲区的发送">有缓冲区的发送</h5><p>如果有缓冲区:</p><ul><li><code>chanbuf</code>计算下一个可以存储数据的位置</li><li>用<code>typedmemmove</code>将发送的数据copy到buffer中，并增加<code>sendx</code>和<code>qcount</code>计数器</li><li>如果当前缓冲区未满，向channel发送的数据会存在channel中的<code>sendx</code>索引中，并将<code>sendx++</code></li><li>由于缓冲区是一个环形数组，<code>c.sendx==c.dataqsiz</code>的时候, c.sendx=0</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//接上面</span><span class="hljs-keyword">if</span> c.qcount &lt; c.dataqsiz &#123;<span class="hljs-comment">// Space is available in the channel buffer. Enqueue the element to send.</span>qp := chanbuf(c, c.sendx)<span class="hljs-keyword">if</span> raceenabled &#123;raceacquire(qp)racerelease(qp)&#125;typedmemmove(c.elemtype, qp, ep)c.sendx++<span class="hljs-keyword">if</span> c.sendx == c.dataqsiz &#123;c.sendx = <span class="hljs-number">0</span>&#125;c.qcount++unlock(&amp;c.lock)<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>&#125;</code></pre></div><h5 id="阻塞发送">阻塞发送</h5><p>当channel无接收者可以接收数据时，向channel发数据就会被阻塞:</p><ul><li><p>首先<code>getg()</code>获得当前发送的goroutine</p></li><li><p><code>acquireSudo()</code>获得<code>runtime.sudog</code>结构体并设置这一次的阻塞信号，比如待发送的数据内存地址<code>mysg.elem</code>, 发送的channel <code>mysg.c</code>, 是否在select内<code>mysg.isSelect</code>(在select内可以不阻塞)等等;</p></li><li><p>然后入队，进入<code>sendq</code>等待发送队列，并设置到当前的goroutine的等待字段上面<code>gp.wating=mysg</code>，表示当前goroutine在等待这个sudog完成;</p></li><li><p>用<code>runtime.gopark()</code>将当前gorotuine睡眠；</p></li><li><p>然后是一些初始化工作，主要归零一些属性，并释放<code>runtime.sudog</code>结构体,最后返回true;</p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//接上面</span><span class="hljs-keyword">if</span> !block &#123;unlock(&amp;c.lock)<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;<span class="hljs-comment">// Block on the channel. Some receiver will complete our operation for us.</span>gp := getg()mysg := acquireSudog()mysg.releasetime = <span class="hljs-number">0</span><span class="hljs-keyword">if</span> t0 != <span class="hljs-number">0</span> &#123;mysg.releasetime = <span class="hljs-number">-1</span>&#125;<span class="hljs-comment">// No stack splits between assigning elem and enqueuing mysg</span><span class="hljs-comment">// on gp.waiting where copystack can find it.</span>mysg.elem = epmysg.waitlink = <span class="hljs-literal">nil</span>mysg.g = gpmysg.isSelect = <span class="hljs-literal">false</span>mysg.c = cgp.waiting = mysggp.param = <span class="hljs-literal">nil</span>c.sendq.enqueue(mysg)gopark(chanparkcommit, unsafe.Pointer(&amp;c.lock), waitReasonChanSend, traceEvGoBlockSend, <span class="hljs-number">2</span>)<span class="hljs-comment">// Ensure the value being sent is kept alive until the</span><span class="hljs-comment">// receiver copies it out. The sudog has a pointer to the</span><span class="hljs-comment">// stack object, but sudogs aren't considered as roots of the</span><span class="hljs-comment">// stack tracer.</span>KeepAlive(ep)<span class="hljs-comment">// someone woke us up.</span><span class="hljs-keyword">if</span> mysg != gp.waiting &#123;throw(<span class="hljs-string">"G waiting list is corrupted"</span>)&#125;gp.waiting = <span class="hljs-literal">nil</span>gp.activeStackChans = <span class="hljs-literal">false</span><span class="hljs-keyword">if</span> gp.param == <span class="hljs-literal">nil</span> &#123;<span class="hljs-keyword">if</span> c.closed == <span class="hljs-number">0</span> &#123;throw(<span class="hljs-string">"chansend: spurious wakeup"</span>)&#125;<span class="hljs-built_in">panic</span>(plainError(<span class="hljs-string">"send on closed channel"</span>))&#125;gp.param = <span class="hljs-literal">nil</span><span class="hljs-keyword">if</span> mysg.releasetime &gt; <span class="hljs-number">0</span> &#123;blockevent(mysg.releasetime-t0, <span class="hljs-number">2</span>)&#125;mysg.c = <span class="hljs-literal">nil</span>releaseSudog(mysg)<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span></code></pre></div><h4 id="数据接收">数据接收</h4><p>与上面的发送一一对应</p><h5 id="直接接收">直接接收</h5><p>最终都是从<code>chanrecv</code>函数开始:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">chanrecv</span><span class="hljs-params">(c *hchan, ep unsafe.Pointer, block <span class="hljs-keyword">bool</span>)</span> <span class="hljs-params">(selected, received <span class="hljs-keyword">bool</span>)</span></span> &#123;<span class="hljs-comment">//一些race操作等</span>...lock(&amp;c.lock)<span class="hljs-keyword">if</span> c.closed != <span class="hljs-number">0</span> &amp;&amp; c.qcount == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> raceenabled &#123;raceacquire(c.raceaddr())&#125;unlock(&amp;c.lock)<span class="hljs-keyword">if</span> ep != <span class="hljs-literal">nil</span> &#123;typedmemclr(c.elemtype, ep)&#125;<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>, <span class="hljs-literal">false</span>&#125;<span class="hljs-keyword">if</span> sg := c.sendq.dequeue(); sg != <span class="hljs-literal">nil</span> &#123;recv(c, sg, ep, <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123; unlock(&amp;c.lock) &#125;, <span class="hljs-number">3</span>)<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>, <span class="hljs-literal">true</span>&#125;...&#125;</code></pre></div><p>其中检查待发送goroutine<code>sendq</code>队列中出队，然后调用<code>recv</code>方法;该<code>recv</code>函数会根据缓冲区大小分为不同情况:</p><ol><li>存在缓冲区</li></ol><ul><li>将队列数据copy到接收方的内存地址；</li><li>将发送队列头的数据copy到缓冲区，释放一个阻塞的发送方;</li></ul><ol start="2"><li>不存在缓冲区</li></ol><ul><li>直接调用<code>runtime.recvDirect</code>将channel发送队列中Goroutine存储的<code>elem</code>数据copy到目标内存地址中;</li></ul><p>但是到最后都会调用<code>runtime.goready</code>将当前P的runnext设置为发送数据的goroutine，等待下一次调度唤醒;</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// recv processes a receive operation on a full channel c.</span><span class="hljs-comment">// There are 2 parts:</span><span class="hljs-comment">// 1) The value sent by the sender sg is put into the channel</span><span class="hljs-comment">//    and the sender is woken up to go on its merry way.</span><span class="hljs-comment">// 2) The value received by the receiver (the current G) is</span><span class="hljs-comment">//    written to ep.</span><span class="hljs-comment">// For synchronous channels, both values are the same.</span><span class="hljs-comment">// For asynchronous channels, the receiver gets its data from</span><span class="hljs-comment">// the channel buffer and the sender's data is put in the</span><span class="hljs-comment">// channel buffer.</span><span class="hljs-comment">// Channel c must be full and locked. recv unlocks c with unlockf.</span><span class="hljs-comment">// sg must already be dequeued from c.</span><span class="hljs-comment">// A non-nil ep must point to the heap or the caller's stack.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">recv</span><span class="hljs-params">(c *hchan, sg *sudog, ep unsafe.Pointer, unlockf <span class="hljs-keyword">func</span>()</span>, <span class="hljs-title">skip</span> <span class="hljs-title">int</span>)</span> &#123;<span class="hljs-comment">//1. 不存在缓冲区</span><span class="hljs-keyword">if</span> c.dataqsiz == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> raceenabled &#123;racesync(c, sg)&#125;<span class="hljs-keyword">if</span> ep != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// copy data from sender</span>recvDirect(c.elemtype, sg, ep)&#125;&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">// Queue is full. Take the item at the</span><span class="hljs-comment">// head of the queue. Make the sender enqueue</span><span class="hljs-comment">// its item at the tail of the queue. Since the</span><span class="hljs-comment">// queue is full, those are both the same slot.</span>qp := chanbuf(c, c.recvx)<span class="hljs-keyword">if</span> raceenabled &#123;raceacquire(qp)racerelease(qp)raceacquireg(sg.g, qp)racereleaseg(sg.g, qp)&#125;<span class="hljs-comment">// copy data from queue to receiver</span><span class="hljs-keyword">if</span> ep != <span class="hljs-literal">nil</span> &#123;typedmemmove(c.elemtype, ep, qp)&#125;<span class="hljs-comment">// copy data from sender to queue</span>typedmemmove(c.elemtype, qp, sg.elem)c.recvx++<span class="hljs-keyword">if</span> c.recvx == c.dataqsiz &#123;c.recvx = <span class="hljs-number">0</span>&#125;c.sendx = c.recvx <span class="hljs-comment">// c.sendx = (c.sendx+1) % c.dataqsiz</span>&#125;sg.elem = <span class="hljs-literal">nil</span>gp := sg.gunlockf()gp.param = unsafe.Pointer(sg)<span class="hljs-keyword">if</span> sg.releasetime != <span class="hljs-number">0</span> &#123;sg.releasetime = cputicks()&#125;goready(gp, skip+<span class="hljs-number">1</span>)&#125;</code></pre></div><h5 id="有缓存的接收">有缓存的接收</h5><p>整个过程比较镜像:</p><ul><li>先锁住channel</li><li>如果有数据，直接从<code>recvx</code>的索引处拿出数据</li><li>如果接收数据的地址不为空，则直接使用<code>runtime.typedmemmove</code>将缓冲区的数据copy到内存中，清空队列中的数据;</li><li>最后也是初始化清空，递增<code>recvx</code>索引，如果其超过了channel容量，还要归零，减少<code>qcount</code>计数器并释放channel锁;</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">// chanrecv receives on channel c and writes the received data to ep.</span><span class="hljs-comment">// ep may be nil, in which case received data is ignored.</span><span class="hljs-comment">// If block == false and no elements are available, returns (false, false).</span><span class="hljs-comment">// Otherwise, if c is closed, zeros *ep and returns (true, false).</span><span class="hljs-comment">// Otherwise, fills in *ep with an element and returns (true, true).</span><span class="hljs-comment">// A non-nil ep must point to the heap or the caller's stack.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">chanrecv</span><span class="hljs-params">(c *hchan, ep unsafe.Pointer, block <span class="hljs-keyword">bool</span>)</span> <span class="hljs-params">(selected, received <span class="hljs-keyword">bool</span>)</span></span> &#123;<span class="hljs-comment">//</span>...lock(&amp;c.lock)<span class="hljs-keyword">if</span> c.closed != <span class="hljs-number">0</span> &amp;&amp; c.qcount == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">if</span> raceenabled &#123;raceacquire(c.raceaddr())&#125;unlock(&amp;c.lock)<span class="hljs-keyword">if</span> ep != <span class="hljs-literal">nil</span> &#123;typedmemclr(c.elemtype, ep)&#125;<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>, <span class="hljs-literal">false</span>&#125;<span class="hljs-keyword">if</span> sg := c.sendq.dequeue(); sg != <span class="hljs-literal">nil</span> &#123;<span class="hljs-comment">// Found a waiting sender. If buffer is size 0, receive value</span><span class="hljs-comment">// directly from sender. Otherwise, receive from head of queue</span><span class="hljs-comment">// and add sender's value to the tail of the queue (both map to</span><span class="hljs-comment">// the same buffer slot because the queue is full).</span>recv(c, sg, ep, <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">()</span></span> &#123; unlock(&amp;c.lock) &#125;, <span class="hljs-number">3</span>)<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>, <span class="hljs-literal">true</span>&#125;<span class="hljs-keyword">if</span> c.qcount &gt; <span class="hljs-number">0</span> &#123;<span class="hljs-comment">// Receive directly from queue</span>qp := chanbuf(c, c.recvx)<span class="hljs-keyword">if</span> raceenabled &#123;raceacquire(qp)racerelease(qp)&#125;<span class="hljs-keyword">if</span> ep != <span class="hljs-literal">nil</span> &#123;typedmemmove(c.elemtype, ep, qp)&#125;typedmemclr(c.elemtype, qp)c.recvx++<span class="hljs-keyword">if</span> c.recvx == c.dataqsiz &#123;c.recvx = <span class="hljs-number">0</span>&#125;c.qcount--unlock(&amp;c.lock)<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>, <span class="hljs-literal">true</span>&#125;...&#125;</code></pre></div><h5 id="阻塞接收">阻塞接收</h5><p>同样比较镜像:</p><ul><li><p>当Channel的发送队列中不存在等待的goroutine并且缓冲区中也不存在任何数据时，从管道中接收会阻塞（当然与select一起的就不一定阻塞）</p></li><li><p>如果接收到，会将任务包装成<code>sudog</code>结构体，进入<code>recvq</code>队列</p></li><li><p>入队后，立即call <code>gopark</code>将当前goroutine状态变成<code>Gwaiting</code></p></li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//接上面</span><span class="hljs-keyword">if</span> !block &#123;unlock(&amp;c.lock)<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>, <span class="hljs-literal">false</span>&#125;<span class="hljs-comment">// no sender available: block on this channel.</span>gp := getg()mysg := acquireSudog()mysg.releasetime = <span class="hljs-number">0</span><span class="hljs-keyword">if</span> t0 != <span class="hljs-number">0</span> &#123;mysg.releasetime = <span class="hljs-number">-1</span>&#125;<span class="hljs-comment">// No stack splits between assigning elem and enqueuing mysg</span><span class="hljs-comment">// on gp.waiting where copystack can find it.</span>mysg.elem = epmysg.waitlink = <span class="hljs-literal">nil</span>gp.waiting = mysgmysg.g = gpmysg.isSelect = <span class="hljs-literal">false</span>mysg.c = cgp.param = <span class="hljs-literal">nil</span>c.recvq.enqueue(mysg)gopark(chanparkcommit, unsafe.Pointer(&amp;c.lock), waitReasonChanReceive, traceEvGoBlockRecv, <span class="hljs-number">2</span>)<span class="hljs-comment">// someone woke us up</span><span class="hljs-keyword">if</span> mysg != gp.waiting &#123;throw(<span class="hljs-string">"G waiting list is corrupted"</span>)&#125;gp.waiting = <span class="hljs-literal">nil</span>gp.activeStackChans = <span class="hljs-literal">false</span><span class="hljs-keyword">if</span> mysg.releasetime &gt; <span class="hljs-number">0</span> &#123;blockevent(mysg.releasetime-t0, <span class="hljs-number">2</span>)&#125;closed := gp.param == <span class="hljs-literal">nil</span>gp.param = <span class="hljs-literal">nil</span>mysg.c = <span class="hljs-literal">nil</span>releaseSudog(mysg)<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>, !closed</code></pre></div><h5 id="触发调度时机">触发调度时机</h5><p>从channel<strong>接收</strong>数据时会触发调度:</p><ul><li>channel为空的时候</li><li>缓冲区不存在数据并且也不存在数据的发送者时</li></ul><h4 id="接收后的关闭">接收后的关闭</h4><h3 id="slowpath和fastpath">SlowPath和fastPath</h3><p>在<code>chansend</code>和<code>chanrecv</code>上有两种路径，一个是fastPath，一个是slowPath(要加上<code>atomic</code>):</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//chansend()</span><span class="hljs-comment">// Fast path: check for failed non-blocking operation without acquiring the lock.</span><span class="hljs-comment">//</span><span class="hljs-comment">// After observing that the channel is not closed, we observe that the channel is</span><span class="hljs-comment">// not ready for sending. Each of these observations is a single word-sized read</span><span class="hljs-comment">// (first c.closed and second c.recvq.first or c.qcount depending on kind of channel).</span><span class="hljs-comment">// Because a closed channel cannot transition from 'ready for sending' to</span><span class="hljs-comment">// 'not ready for sending', even if the channel is closed between the two observations,</span><span class="hljs-comment">// they imply a moment between the two when the channel was both not yet closed</span><span class="hljs-comment">// and not ready for sending. We behave as if we observed the channel at that moment,</span><span class="hljs-comment">// and report that the send cannot proceed.</span><span class="hljs-comment">//</span><span class="hljs-comment">// It is okay if the reads are reordered here: if we observe that the channel is not</span><span class="hljs-comment">// ready for sending and then observe that it is not closed, that implies that the</span><span class="hljs-comment">// channel wasn't closed during the first observation.</span><span class="hljs-keyword">if</span> !block &amp;&amp; c.closed == <span class="hljs-number">0</span> &amp;&amp; ((c.dataqsiz == <span class="hljs-number">0</span> &amp;&amp; c.recvq.first == <span class="hljs-literal">nil</span>) ||(c.dataqsiz &gt; <span class="hljs-number">0</span> &amp;&amp; c.qcount == c.dataqsiz)) &#123;<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//chanrecv</span><span class="hljs-comment">// Fast path: check for failed non-blocking operation without acquiring the lock.</span><span class="hljs-comment">//</span><span class="hljs-comment">// After observing that the channel is not ready for receiving, we observe that the</span><span class="hljs-comment">// channel is not closed. Each of these observations is a single word-sized read</span><span class="hljs-comment">// (first c.sendq.first or c.qcount, and second c.closed).</span><span class="hljs-comment">// Because a channel cannot be reopened, the later observation of the channel</span><span class="hljs-comment">// being not closed implies that it was also not closed at the moment of the</span><span class="hljs-comment">// first observation. We behave as if we observed the channel at that moment</span><span class="hljs-comment">// and report that the receive cannot proceed.</span><span class="hljs-comment">//</span><span class="hljs-comment">// The order of operations is important here: reversing the operations can lead to</span><span class="hljs-comment">// incorrect behavior when racing with a close.</span><span class="hljs-keyword">if</span> !block &amp;&amp; (c.dataqsiz == <span class="hljs-number">0</span> &amp;&amp; c.sendq.first == <span class="hljs-literal">nil</span> ||c.dataqsiz &gt; <span class="hljs-number">0</span> &amp;&amp; atomic.Loaduint(&amp;c.qcount) == <span class="hljs-number">0</span>) &amp;&amp;atomic.Load(&amp;c.closed) == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span>&#125;</code></pre></div><p>A: 这个这两个 fast path 其实炫技的成分太高了，我们需要先理解这两个 fast path 才能理解为什么这里一个需要<code>atomic</code>操作而另一个不需要。</p><ul><li><p>首先，他们是针对<code>select</code>语句中<strong>非阻塞 channel</strong>操作的的一种优化，也就是说要求不在channel上发生阻塞（能失败则立刻失败）;</p><p>这时候我们要考虑关于<code>channel</code>的这样两个事实，</p><ol><li><p>如果<code>channel</code>没有被 close，那么不能进行<strong>发送</strong>的条件只可能是：</p><p>unbuffered channel 没有接收方（ <code>dataqsiz</code>为空且接受队列为空时），要么 buffered channel 缓存已满（<code>dataqsiz != 0 &amp;&amp; qcount == dataqsize</code>）</p></li><li><p>那么不能进行<strong>接收</strong>的条件只可能是：</p><p><code>unbuffered channel</code>没有发送方（ <code>dataqsiz</code> 为空且发送队列为空），要么<code>buffered channel</code>缓存为空（<code>dataqsiz != 0 &amp;&amp; qcount == 0</code>）</p></li></ol></li><li><p>理解是否需要 atomic 操作的关键在于：<code>atomic 操作保证了代码的内存顺序</code>，是否发生指令重排！！！！！</p></li></ul><p>由于 channel 只能由未关闭状态转换为关闭状态，因此在<code>!block</code> 的异步操作中，</p><p>第一种情况下，channel 未关闭和 channel 不能进行发送之间的指令重排是能够保证代码的正确性的，因为：在不发生重排时，「不能进行发送」同样适用于 channel 已经 close。如果 closed 的操作被重排到不能进行发送之后，依然隐含着在判断「不能进行发送」这个条件时候 channel 仍然是未 closed 的。</p><p>但第二种情况中，如果「不能进行接收」和 channel 未关闭发生重排，我们无法保证在观察 channel 未关闭之后，得到的 「不能进行接收」是 channel 尚未关闭得到的结果，这时原本应该得到「已关闭且 buf 空」的结论（chanrecv 应该返回 true, false），却得到了「未关闭且 buf 空」（返回值 false, false），从而报告错误的状态。因此必须使此处的 qcount 和 closed 的读取操作的顺序通过原子操作得到顺序保障。</p><h3 id="使用注意">使用注意</h3><p>在Linux下，<code>channel</code>是用<code>futex</code>实现的，不会导致上下文切换；可以与<code>Sync.Mutex</code>进行比较，mutex是用信号量进行处理，其中涉及了<code>gopark</code></p><p>当把channel当作传入参数的时候要先确定一下箭头方向</p><p>chan&lt;-string：指的是可以入可以出的channel</p><p>&lt;-chan string：指的是receive-only channnel</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//gobyexample 例子</span><span class="hljs-comment">// This `ping` function only accepts a channel for sending</span><span class="hljs-comment">// values. It would be a compile-time error to try to</span><span class="hljs-comment">// receive on this channel.</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">ping</span><span class="hljs-params">(pings <span class="hljs-keyword">chan</span>&lt;- <span class="hljs-keyword">string</span>, msg <span class="hljs-keyword">string</span>)</span></span> &#123;    pings &lt;- msg&#125;<span class="hljs-comment">// The `pong` function accepts one channel for receives</span><span class="hljs-comment">// (`pings`) and a second for sends (`pongs`).</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">pong</span><span class="hljs-params">(pings &lt;-<span class="hljs-keyword">chan</span> <span class="hljs-keyword">string</span>, pongs <span class="hljs-keyword">chan</span>&lt;- <span class="hljs-keyword">string</span>)</span></span> &#123;    msg := &lt;-pings    pongs &lt;- msg&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span> &#123;    pings := <span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">string</span>, <span class="hljs-number">1</span>)    pongs := <span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">string</span>, <span class="hljs-number">1</span>)    ping(pings, <span class="hljs-string">"passed message"</span>)    pong(pings, pongs)    fmt.Println(&lt;-pongs)&#125;</code></pre></div><h3 id="happens-before问题">Happens-before问题</h3><p>在<a href="https://golang.org/ref/mem" target="_blank" rel="noopener">goMemory</a>里面有提到这个happens-before问题,其实就是指令重排(特么终于解决我的疑问了)channel的一些问题：</p><ol><li>带缓冲的channel的写操作在其相应的读操作之前</li><li>不带缓冲的channel发生在其相应的写操作之前</li><li>如果你关闭channel，之后才会读其channel最后的返回值0(这个其实在Context里面发现过！)</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">var</span> temp=<span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">int</span>)<span class="hljs-comment">//不带缓冲</span><span class="hljs-keyword">var</span> a=<span class="hljs-string">"123"</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">foo</span><span class="hljs-params">()</span></span>&#123;    fmt.Println(<span class="hljs-string">"a:"</span>,a)<span class="hljs-comment">//1 </span>    &lt;-temp<span class="hljs-comment">// 2</span>&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    <span class="hljs-keyword">go</span> foo()    temp&lt;<span class="hljs-number">-1</span><span class="hljs-comment">//3</span>    fmt.Println(<span class="hljs-string">"a main:"</span>,a)<span class="hljs-comment">//4</span>&#125;</code></pre></div><p>输出</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-selector-tag">a</span>: <span class="hljs-number">123</span><span class="hljs-selector-tag">a</span> main: <span class="hljs-number">123</span></code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">var</span> temp=<span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">int</span>,<span class="hljs-number">10</span>)<span class="hljs-comment">//带缓冲</span><span class="hljs-keyword">var</span> a=<span class="hljs-string">"123"</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">foo</span><span class="hljs-params">()</span></span>&#123;    fmt.Println(<span class="hljs-string">"a:"</span>,a)<span class="hljs-comment">//1 </span>    temp&lt;<span class="hljs-number">-1</span><span class="hljs-comment">// 2</span>&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;    <span class="hljs-keyword">go</span> foo()    &lt;-temp<span class="hljs-comment">//3,可能先发生</span>    fmt.Println(<span class="hljs-string">"a main:"</span>,a)<span class="hljs-comment">//4</span>&#125;</code></pre></div><p>不能保证1, 2 和 3 的发生顺序，就很有可能只输出</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-selector-tag">a</span> main: <span class="hljs-number">123</span></code></pre></div><blockquote><blockquote><p>The kth receive on a channel with capacity C happens before the k+Cth send from that channel completes.一个容量为C的channel接到的第k个值会发生在 第K+C个值发出完成 之前</p></blockquote></blockquote><p>用一个官方例子：下面这个例子限制了每时每刻最多有三个work在工作</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">var</span> limit = <span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">int</span>, <span class="hljs-number">3</span>)<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span> &#123;<span class="hljs-keyword">for</span> _, w := <span class="hljs-keyword">range</span> work &#123;<span class="hljs-keyword">go</span> <span class="hljs-function"><span class="hljs-keyword">func</span><span class="hljs-params">(w <span class="hljs-keyword">func</span>()</span>)</span> &#123;limit &lt;- <span class="hljs-number">1</span>w()&lt;-limit&#125;(w)&#125;<span class="hljs-keyword">select</span>&#123;&#125;&#125;</code></pre></div><h2 id="2-sync-waitgroup">2.Sync.WaitGroup</h2><p>见<a href="../sync.md">sync文章</a></p><h3 id="普通使用">普通使用</h3><p>Sync.WaitGroup有三个methods:</p><ol><li>Add(delta int):将你要等待的协程加入，delta即加入的数量</li><li>Done() : 代表当前协程完成</li><li>Wait() : 等待所有协程完成(调用Done())，完成后即返回，否则一直阻塞</li></ol><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//举个例子：</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">(number <span class="hljs-keyword">int</span>)</span></span>&#123;    <span class="hljs-keyword">var</span> wg Sync.WaitGroup    wg.Add(number)<span class="hljs-comment">//要同步的协程数</span>    <span class="hljs-keyword">for</span> i:=<span class="hljs-number">0</span>;i&lt;number;i++&#123;        <span class="hljs-keyword">go</span> doSth(&amp;wg,i)    &#125;    begin:=Time.Now()    wg.Wait()<span class="hljs-comment">//完成后继续往下跑</span>    end:=Time.Now()<span class="hljs-comment">//这里还可以这样进行批量测试</span>    dur:=Time.Duration(end-begin)&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">doSth</span><span class="hljs-params">(wg *Sync.WaitGroup,num <span class="hljs-keyword">int</span>)</span></span>&#123;    fmt.Println(num)    wg.Done()<span class="hljs-comment">//完成就Done</span>&#125;</code></pre></div><h2 id="3-sync-once-单例">3. Sync.Once 单例</h2><p>见<a href="../sync.md">sync文章</a></p><h2 id="4-mutex互斥量">4. Mutex互斥量</h2><h3 id="01互斥量">01互斥量</h3><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">var</span> sema=<span class="hljs-built_in">make</span>(<span class="hljs-keyword">chan</span> <span class="hljs-keyword">struct</span>&#123;&#125;,<span class="hljs-number">1</span>)</code></pre></div><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//每次使用前</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">Deposit</span><span class="hljs-params">(amount <span class="hljs-keyword">int</span>)</span></span>&#123;    sema&lt;-<span class="hljs-keyword">struct</span>&#123;&#125;&#123;&#125;<span class="hljs-comment">//锁住，往里面加一个</span>    balance+=amount    &lt;-sema<span class="hljs-comment">//释放</span>&#125;</code></pre></div><h3 id="sync-mutex-互斥量">Sync.Mutex 互斥量</h3><p>注意： golang的锁都不是可重入锁(ReentranLock),参考一下Java的 <a href="../Java-Concurrency.html">可重入锁</a></p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">var</span> mu Sync.Mutex<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">Deposit</span><span class="hljs-params">(amount <span class="hljs-keyword">int</span>)</span></span> &#123;mu.Lock()balance = balance + amountmu.Unlock()&#125;<span class="hljs-comment">// func Withdraw() int &#123;</span><span class="hljs-comment">// return &lt;-balances</span><span class="hljs-comment">// &#125;</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">Balance</span><span class="hljs-params">()</span> <span class="hljs-title">int</span></span> &#123;    mu.Lock()    <span class="hljs-keyword">defer</span> mu.Unlockb := balance<span class="hljs-keyword">return</span> b&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">Withdraw</span><span class="hljs-params">(amount <span class="hljs-keyword">int</span>)</span> <span class="hljs-title">bool</span></span> &#123;    mu.Lock()    <span class="hljs-keyword">defer</span> mu.Unlock()Deposit(-amount)<span class="hljs-comment">//这里，重用了mu的锁，但是，golang不支持重入锁，所以这里会进行阻塞</span><span class="hljs-keyword">if</span> Balance() &lt; <span class="hljs-number">0</span> &#123;Deposit(amount)<span class="hljs-keyword">return</span> <span class="hljs-literal">false</span>&#125;<span class="hljs-keyword">return</span> <span class="hljs-literal">true</span>&#125;</code></pre></div><h3 id="rwmutex-读写互斥量">* RWMutex 读写互斥量</h3><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">var</span> mu Sync.RWMutex</code></pre></div><p><strong>写操作 Lock(), UnLock()</strong></p><p><strong>读操作 RLock(), RUnlock()</strong></p><p>读锁即是 同一时间允许多个读的协程，但只允许一个写的协程</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//重写Balance()</span><span class="hljs-keyword">var</span> mu Sync.RWMutex<span class="hljs-keyword">var</span> balance <span class="hljs-keyword">int</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">Balance</span><span class="hljs-params">()</span> <span class="hljs-title">int</span></span>&#123;    mu.RLock()    <span class="hljs-keyword">defer</span> mu.RUnlock()    b:=balance    <span class="hljs-keyword">return</span> b&#125;</code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Something about Slice in Go</title>
    <link href="/2019/06/05/Go/Slice/"/>
    <url>/2019/06/05/Go/Slice/</url>
    
    <content type="html"><![CDATA[<p>Golang’s Slice is kinda diffrent===&gt;</p><a id="more"></a><p>当前版本go 1.13<img src="../../img/golangusergroups.png" srcset="/img/loading.gif" alt="group"></p><p>Related article [Slice]<a href="https://blog.golang.org/slices" target="_blank" rel="noopener">https://blog.golang.org/slices</a></p><p>讨论到一种数据结构，我们很自然就从以下：</p><ul><li>结构体本身</li><li>初始化(constructor)</li><li>使用详情（包含各种数据增删改等情况）</li><li>销毁(deconstructor)</li></ul><h2 id="1-结构体">1. 结构体</h2><p>slice其实是一个结构体，并不是简单的数组或者链表</p> <div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//actually its not visible to programmer</span><span class="hljs-comment">//我自己臆想出来的,但可以从src/runtime/slice.go找出,或者走这个链接：https://golang.org/src/runtime/slice.go</span><span class="hljs-keyword">type</span> slice <span class="hljs-keyword">struct</span>&#123;    length <span class="hljs-keyword">int</span> <span class="hljs-comment">//length</span>    pointer <span class="hljs-keyword">interface</span>&#123;&#125; <span class="hljs-comment">//point to first element, the type depends on the element</span>    Capacity <span class="hljs-keyword">int</span><span class="hljs-comment">//max容量</span>&#125;</code></pre></div><p>实际上是这样的:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-keyword">type</span> slice <span class="hljs-keyword">struct</span>&#123;    array unsafe.Pointer    <span class="hljs-built_in">len</span> <span class="hljs-keyword">int</span>    <span class="hljs-built_in">cap</span> <span class="hljs-keyword">int</span>&#125;</code></pre></div><p>来到这里你可能会想到，好像一个对象耶，那么这玩意儿传入func里面是不是传指针进去呢？（即里面的修改会影响到origianl？）答案是不会</p><p>Example:</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">SubtractOneFromLength</span><span class="hljs-params">(slice []<span class="hljs-keyword">byte</span>)</span> []<span class="hljs-title">byte</span></span> &#123;    slice = slice[<span class="hljs-number">0</span> : <span class="hljs-built_in">len</span>(slice)<span class="hljs-number">-1</span>]    <span class="hljs-keyword">return</span> slice&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span> &#123;    fmt.Println(<span class="hljs-string">"Before: len(slice) ="</span>, <span class="hljs-built_in">len</span>(slice))    newSlice := SubtractOneFromLength(slice)<span class="hljs-comment">//you haven't passed the slice's header into it</span>    fmt.Println(<span class="hljs-string">"After:  len(slice) ="</span>, <span class="hljs-built_in">len</span>(slice))    fmt.Println(<span class="hljs-string">"After:  len(newSlice) ="</span>, <span class="hljs-built_in">len</span>(newSlice))&#125;</code></pre></div><p>You <strong>Must</strong> do sth like:</p><div class="hljs"><pre><code class="hljs go">newSlice := SubtractOneFromLength(&amp;slice)<span class="hljs-comment">//Like this will work!</span></code></pre></div><h2 id="2-初始化">2. 初始化</h2><div class="hljs"><pre><code class="hljs golang"><span class="hljs-comment">//顶层</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">makeslice64</span><span class="hljs-params">(et *_type, len64, cap64 <span class="hljs-keyword">int64</span>)</span> <span class="hljs-title">unsafe</span>.<span class="hljs-title">Pointer</span></span>&#123;&#125;<span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">makeslice</span><span class="hljs-params">(et *_type, <span class="hljs-built_in">len</span>, <span class="hljs-built_in">cap</span> <span class="hljs-keyword">int</span>)</span> <span class="hljs-title">unsafe</span>.<span class="hljs-title">Pointer</span></span>&#123;&#125;</code></pre></div><p>以上函数会先进行一个<strong>溢出判断</strong> （todo，这里涉及到编译器平台问题），如果cap设定太大会panic:cap out of range</p><p>然后便会开始分配空间，这里用的是runtime/malloc.go里面的mallocgc方法:</p><ol><li>小对象会从每个<strong>P(GPM模型中process)中的cache</strong>的可用队列中拿到空间</li><li>大对象(&gt;32KB)则会从全局<strong>堆</strong>中拿到空间</li></ol><p>详情可以看下<a href="./gc.md">之前的那篇笔记gc</a></p><h2 id="3-enlarge-capcity">3. Enlarge Capcity</h2><p>你可以理解slice是动态列表，到达某个值后很自然就会扩容，扩容的大小文档里面也写了，小于1024长度是直接 <strong>×2</strong>，或者是超过了1024的只会库容1.25倍（也不一定奥，可以接着看）：</p><div class="hljs"><pre><code class="hljs golang"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">growslice</span><span class="hljs-params">(et *_type, old slice, <span class="hljs-built_in">cap</span> <span class="hljs-keyword">int</span>)</span> <span class="hljs-title">slice</span></span> &#123;<span class="hljs-comment">// et 指的是？？？？， old是老的slice，cap是申请的容量</span>    <span class="hljs-comment">//前面是一些判断racecodition以及调试，防止cap设置不合理的判断</span>    ........    <span class="hljs-comment">//-------------这里开始计算扩容数量-----------------</span>    newcap := old.<span class="hljs-built_in">cap</span>    doublecap := newcap + newcap    <span class="hljs-keyword">if</span> <span class="hljs-built_in">cap</span> &gt; doublecap &#123;<span class="hljs-comment">//申请容量 &gt; 2 * 旧的容量</span>        newcap = <span class="hljs-built_in">cap</span>    &#125; <span class="hljs-keyword">else</span> &#123;        <span class="hljs-keyword">if</span> old.<span class="hljs-built_in">len</span> &lt; <span class="hljs-number">1024</span> &#123;<span class="hljs-comment">//老容量 &lt; 1024，直接扩成旧的两倍</span>            newcap = doublecap        &#125; <span class="hljs-keyword">else</span> &#123;            <span class="hljs-comment">// Check 0 &lt; newcap to detect overflow</span>            <span class="hljs-comment">// and prevent an infinite loop.</span>            <span class="hljs-keyword">for</span> <span class="hljs-number">0</span> &lt; newcap &amp;&amp; newcap &lt; <span class="hljs-built_in">cap</span> &#123;<span class="hljs-comment">//好像看见有文章說是1.25倍，但实际并不是，可以往下继续看capmen变量</span>                newcap += newcap / <span class="hljs-number">4</span>            &#125;            <span class="hljs-comment">// Set newcap to the requested cap when</span>            <span class="hljs-comment">// the newcap calculation overflowed.</span>            <span class="hljs-keyword">if</span> newcap &lt;= <span class="hljs-number">0</span> &#123;<span class="hljs-comment">//溢出的话就使之等于申请的容量</span>                newcap = <span class="hljs-built_in">cap</span>            &#125;        &#125;    &#125;</code></pre></div><p>但是，那些只针对于没有定义 *** cap *** 字段的slice，万一规定了，像</p><div class="hljs"><pre><code class="hljs go">slice := <span class="hljs-built_in">make</span>([]<span class="hljs-keyword">int</span>, <span class="hljs-number">10</span>, <span class="hljs-number">15</span>)</code></pre></div><p>上限就是15了，如果强行append。。。。。。*** 也没关系! ***，只是这个时候会进行扩容，然后,原slice的地址（即第一个元素的地址）会进行改变</p><div class="hljs"><pre><code class="hljs go">slice := <span class="hljs-built_in">make</span>([]<span class="hljs-keyword">int</span>, <span class="hljs-number">1</span>, <span class="hljs-number">2</span>)slice[<span class="hljs-number">0</span>]=<span class="hljs-number">1</span>slice=<span class="hljs-built_in">append</span>(slice,<span class="hljs-number">3</span>)fmt.Println(&amp;slice[<span class="hljs-number">0</span>])<span class="hljs-comment">//0x414020</span>slice=<span class="hljs-built_in">append</span>(slice,<span class="hljs-number">4</span>)fmt.Println(&amp;slice[<span class="hljs-number">0</span>])<span class="hljs-comment">//0x414040</span></code></pre></div><p>同样，<strong><em>这里有另外几个坑</em></strong>：</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-comment">//example in docs</span>    slice := <span class="hljs-built_in">make</span>([]<span class="hljs-keyword">int</span>, <span class="hljs-number">10</span>, <span class="hljs-number">15</span>)    fmt.Printf(<span class="hljs-string">"len: %d, cap: %d\n"</span>, <span class="hljs-built_in">len</span>(slice), <span class="hljs-built_in">cap</span>(slice))<span class="hljs-comment">//15</span>    newSlice := <span class="hljs-built_in">make</span>([]<span class="hljs-keyword">int</span>, <span class="hljs-built_in">len</span>(slice), <span class="hljs-number">2</span>*<span class="hljs-built_in">cap</span>(slice))    <span class="hljs-keyword">for</span> i := <span class="hljs-keyword">range</span> slice &#123;        newSlice[i] = slice[i]    &#125;    slice = newSlice    fmt.Printf(<span class="hljs-string">"len: %d, cap: %d\n"</span>, <span class="hljs-built_in">len</span>(slice), <span class="hljs-built_in">cap</span>(slice))<span class="hljs-comment">//30</span></code></pre></div><ol><li>究竟是哪个slice</li></ol><p>简单的 = 号其实属于一种浅复制</p><div class="hljs"><pre><code class="hljs go">a:=<span class="hljs-built_in">make</span>([]<span class="hljs-keyword">int</span>,<span class="hljs-number">3</span>,<span class="hljs-number">4</span>)a[<span class="hljs-number">0</span>]=<span class="hljs-number">1</span>a[<span class="hljs-number">1</span>]=<span class="hljs-number">2</span>a[<span class="hljs-number">2</span>]=<span class="hljs-number">3</span>b:=<span class="hljs-built_in">append</span>(a,<span class="hljs-number">4</span>)c:=<span class="hljs-built_in">append</span>(a,<span class="hljs-number">100</span>)fmt.Println(&amp;a[<span class="hljs-number">0</span>], &amp;b[<span class="hljs-number">0</span>], &amp;c[<span class="hljs-number">0</span>]) <span class="hljs-comment">//0xc0000125c0 0xc0000125c0 0xc0000125c0</span><span class="hljs-comment">//  可以看到这里其实用的是同一个slice</span>fmt.Println(a, b, c)<span class="hljs-comment">//[1 2 3] [1 2 3 100] [1 2 3 100]</span>c[<span class="hljs-number">0</span>] = <span class="hljs-number">101</span>fmt.Println(a, b, c)<span class="hljs-comment">//[101 2 3] [101 2 3 100] [101 2 3 100]</span></code></pre></div><ol start="2"><li>值传递？引用传递？</li></ol><p>首先明确，slice是一种struct,struct本身就是值传递</p><div class="hljs"><pre><code class="hljs go">y:=[]<span class="hljs-keyword">int</span>&#123;<span class="hljs-number">0</span>,<span class="hljs-number">0</span>,<span class="hljs-number">0</span>&#125;add(y)<span class="hljs-comment">//你可能这样子推论：go里面都是值传递==&gt;所以这里面的改动不会影响到y。可惜，这是错的</span>fmt.Println(y)<span class="hljs-comment">//&#123;1,1,1&#125;  ///wtf？？？？？</span><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">add</span><span class="hljs-params">(y []<span class="hljs-keyword">int</span>)</span></span>&#123;    <span class="hljs-comment">//这个就不会改变，因为这个v只是值的拷贝</span>    <span class="hljs-comment">// for _,v:=range y&#123; </span>    <span class="hljs-comment">//     v++</span>    <span class="hljs-comment">// &#125;</span>    <span class="hljs-comment">//但这个会被改变</span>    <span class="hljs-keyword">for</span> i:=<span class="hljs-keyword">range</span> y&#123;        y[i]++    &#125;&#125;</code></pre></div><h5 id="注意：">注意：</h5><p>其实上面已经提到，slice是一个struct，传入的时候如果仅仅是修改一下元素的内容，是不会对其头部地址进行改变，所以，传入修改其值是可以的;</p><p>但是，做一些比如append之类的操作，这样会使整个slice发生变化，<strong>其头部指针指向一个新的slice</strong>，所以原来的slice就不会被改变</p><p>针对以上问题的答案也有了：</p><ol><li><p>因为a,b,c的头指针地址都一样，所以其实它们都指向同一个slice，所以后面对任意一个进行改变，都会覆盖其他的改变;</p></li><li><p>slice作为参数传递进去，其实可以改变其中的元素，在不重新分配内存的情况下会影响到自身。但如果需要（保险为先），必须从返回值或传指针进行修改;</p></li></ol><h3 id="但是-扩容其实没有那么简单-我注意到growslice下面还要一段源码"><strong>但是，扩容其实没有那么简单，我注意到growslice下面还要一段源码！</strong></h3><h3 id="扩容growslice">扩容growslice</h3><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">growslice</span><span class="hljs-params">(et *_type, old slice, <span class="hljs-built_in">cap</span> <span class="hljs-keyword">int</span>)</span> <span class="hljs-title">slice</span></span> &#123;<span class="hljs-comment">// et 是_type指针，详情可以看type文章， old是老的slice，cap是申请的容量</span><span class="hljs-comment">//新的slice的length会设为旧的slice的length</span>    ........    <span class="hljs-comment">//-------------以上计算扩容数量结束--------------------</span>    newcap := old.<span class="hljs-built_in">cap</span>doublecap := newcap + newcap<span class="hljs-keyword">if</span> <span class="hljs-built_in">cap</span> &gt; doublecap &#123;newcap = <span class="hljs-built_in">cap</span>&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-keyword">if</span> old.<span class="hljs-built_in">len</span> &lt; <span class="hljs-number">1024</span> &#123;            newcap = doublecap            <span class="hljs-comment">//长度小于1024，新cap直接= len*2</span>&#125; <span class="hljs-keyword">else</span> &#123;<span class="hljs-comment">// Check 0 &lt; newcap to detect overflow</span>            <span class="hljs-comment">// and prevent an infinite loop.</span>            <span class="hljs-comment">//这里有个检测overflow的小技巧；还有</span><span class="hljs-keyword">for</span> <span class="hljs-number">0</span> &lt; newcap &amp;&amp; newcap &lt; <span class="hljs-built_in">cap</span> &#123;newcap += newcap / <span class="hljs-number">4</span>&#125;<span class="hljs-comment">// Set newcap to the requested cap when</span><span class="hljs-comment">// the newcap calculation overflowed.</span><span class="hljs-keyword">if</span> newcap &lt;= <span class="hljs-number">0</span> &#123;newcap = <span class="hljs-built_in">cap</span>&#125;&#125;&#125;    <span class="hljs-comment">//-------------!!!!以下开始计算内存位置，不但继续计算新的容量大小，还要决定扩容后是否要重新划分内存------------------</span>    <span class="hljs-keyword">var</span> overflow <span class="hljs-keyword">bool</span>    <span class="hljs-keyword">var</span> lenmem, newlenmem, capmem <span class="hljs-keyword">uintptr</span>    <span class="hljs-comment">// Specialize for common values of et.size.</span>    <span class="hljs-comment">// For 1 we don't need any division/multiplication.</span>    <span class="hljs-comment">// For sys.PtrSize, compiler will optimize division/multiplication into a shift by a constant.</span>    <span class="hljs-comment">// For powers of 2, use a variable shift.</span>    <span class="hljs-comment">//1 不用做任何处理</span>    <span class="hljs-comment">//2 是指针大小会，进行roundup</span>    <span class="hljs-comment">//3 2的幂次方，会使用移位来进行计算</span>    <span class="hljs-keyword">switch</span> &#123;    <span class="hljs-keyword">case</span> et.size == <span class="hljs-number">1</span>:        lenmem = <span class="hljs-keyword">uintptr</span>(old.<span class="hljs-built_in">len</span>)        newlenmem = <span class="hljs-keyword">uintptr</span>(<span class="hljs-built_in">cap</span>)        capmem = roundupsize(<span class="hljs-keyword">uintptr</span>(newcap))<span class="hljs-comment">//这个就是计算新的capmen，当newcap不符合规定内存的大小规格时，会进行roundup内存对齐!!!</span>        overflow = <span class="hljs-keyword">uintptr</span>(newcap) &gt; maxAlloc        newcap = <span class="hljs-keyword">int</span>(capmem)    <span class="hljs-keyword">case</span> et.size == sys.PtrSize:<span class="hljs-comment">//是一个指针大小</span>        lenmem = <span class="hljs-keyword">uintptr</span>(old.<span class="hljs-built_in">len</span>) * sys.PtrSize        newlenmem = <span class="hljs-keyword">uintptr</span>(<span class="hljs-built_in">cap</span>) * sys.PtrSize        <span class="hljs-comment">//这里注意要roundup以下，对应sizetoclass表</span>        capmem = roundupsize(<span class="hljs-keyword">uintptr</span>(newcap) * sys.PtrSize)        overflow = <span class="hljs-keyword">uintptr</span>(newcap) &gt; maxAlloc/sys.PtrSize        newcap = <span class="hljs-keyword">int</span>(capmem / sys.PtrSize)<span class="hljs-comment">//sys.PtrSize指的是一个指针的size，64位的机器就是8</span>    <span class="hljs-keyword">case</span> isPowerOfTwo(et.size):<span class="hljs-comment">//类型为2次幂会用variable shift，比如</span>        <span class="hljs-keyword">var</span> shift <span class="hljs-keyword">uintptr</span>        <span class="hljs-keyword">if</span> sys.PtrSize == <span class="hljs-number">8</span> &#123;            <span class="hljs-comment">// Mask shift for better code generation.</span>            shift = <span class="hljs-keyword">uintptr</span>(sys.Ctz64(<span class="hljs-keyword">uint64</span>(et.size))) &amp; <span class="hljs-number">63</span>        &#125; <span class="hljs-keyword">else</span> &#123;            shift = <span class="hljs-keyword">uintptr</span>(sys.Ctz32(<span class="hljs-keyword">uint32</span>(et.size))) &amp; <span class="hljs-number">31</span>        &#125;        lenmem = <span class="hljs-keyword">uintptr</span>(old.<span class="hljs-built_in">len</span>) &lt;&lt; shift        newlenmem = <span class="hljs-keyword">uintptr</span>(<span class="hljs-built_in">cap</span>) &lt;&lt; shift        capmem = roundupsize(<span class="hljs-keyword">uintptr</span>(newcap) &lt;&lt; shift)        overflow = <span class="hljs-keyword">uintptr</span>(newcap) &gt; (maxAlloc &gt;&gt; shift)        newcap = <span class="hljs-keyword">int</span>(capmem &gt;&gt; shift)    <span class="hljs-keyword">default</span>:<span class="hljs-comment">//其他的情况就直接除以et.size</span>        lenmem = <span class="hljs-keyword">uintptr</span>(old.<span class="hljs-built_in">len</span>) * et.size        newlenmem = <span class="hljs-keyword">uintptr</span>(<span class="hljs-built_in">cap</span>) * et.size        capmem = roundupsize(<span class="hljs-keyword">uintptr</span>(newcap) * et.size)        overflow = <span class="hljs-keyword">uintptr</span>(newcap) &gt; maxSliceCap(et.size)        newcap = <span class="hljs-keyword">int</span>(capmem / et.size)    &#125;    <span class="hljs-comment">// The check of overflow (uintptr(newcap) &gt; maxSliceCap(et.size))</span>    <span class="hljs-comment">// in addition to capmem &gt; _MaxMem is needed to prevent an overflow</span>    <span class="hljs-comment">// which can be used to trigger a segfault on 32bit architectures</span>    <span class="hljs-comment">// with this example program:</span>    <span class="hljs-comment">//</span>    <span class="hljs-comment">// type T [1&lt;&lt;27 + 1]int64</span>    <span class="hljs-comment">//</span>    <span class="hljs-comment">// var d T</span>    <span class="hljs-comment">// var s []T</span>    <span class="hljs-comment">//</span>    <span class="hljs-comment">// func main() &#123;</span>    <span class="hljs-comment">//   s = append(s, d, d, d, d)</span>    <span class="hljs-comment">//   print(len(s), "\n")</span>    <span class="hljs-comment">// &#125;</span>    <span class="hljs-keyword">if</span> <span class="hljs-built_in">cap</span> &lt; old.<span class="hljs-built_in">cap</span> || overflow || capmem &gt; maxAlloc &#123;        <span class="hljs-built_in">panic</span>(errorString(<span class="hljs-string">"growslice: cap out of range"</span>))    &#125;        <span class="hljs-keyword">var</span> p unsafe.Pointer         <span class="hljs-comment">//这个应该是地址了</span>    <span class="hljs-keyword">if</span> et.kind&amp;kindNoPointers != <span class="hljs-number">0</span> &#123;        p = mallocgc(capmem, <span class="hljs-literal">nil</span>, <span class="hljs-literal">false</span>)<span class="hljs-comment">//申请内存空间</span>        memmove(p, old.array, lenmem)        <span class="hljs-comment">// The append() that calls growslice is going to overwrite from old.len to cap (which will be the new length).</span>        <span class="hljs-comment">// Only clear the part that will not be overwritten.</span>        memclrNoHeapPointers(add(p, newlenmem), capmem-newlenmem)    &#125; <span class="hljs-keyword">else</span> &#123;        <span class="hljs-comment">// Note: can't use rawmem (which avoids zeroing of memory), because then GC can scan uninitialized memory.</span>        p = mallocgc(capmem, et, <span class="hljs-literal">true</span>)        <span class="hljs-keyword">if</span> !writeBarrier.enabled &#123;            memmove(p, old.array, lenmem)        &#125; <span class="hljs-keyword">else</span> &#123;            <span class="hljs-keyword">for</span> i := <span class="hljs-keyword">uintptr</span>(<span class="hljs-number">0</span>); i &lt; lenmem; i += et.size &#123;                typedmemmove(et, add(p, i), add(old.array, i))            &#125;        &#125;    &#125;    <span class="hljs-keyword">return</span> slice&#123;p, old.<span class="hljs-built_in">len</span>, newcap&#125;&#125;</code></pre></div><p>这里可以参照<a href="./memManage.md">之前写的日志memManger</a>里面有关go的内存管理中</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">roundupsize</span><span class="hljs-params">(size <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-title">uintptr</span></span>&#123;&#125;</code></pre></div><p>即会对传入类型进行内存对齐(并roundup),这也可能会导致扩容的容量跟之前说的*2或1.25倍不同！</p><p>我们做一个实验:</p><div class="hljs"><pre><code class="hljs go">t := <span class="hljs-built_in">make</span>([]<span class="hljs-keyword">int</span>, <span class="hljs-number">1000</span>, <span class="hljs-number">1000</span>)log.Printf(<span class="hljs-string">"%+v"</span>, <span class="hljs-built_in">cap</span>(t))t = <span class="hljs-built_in">append</span>(t, <span class="hljs-number">1</span>,<span class="hljs-number">2</span>,<span class="hljs-number">3</span>,<span class="hljs-number">4</span>,<span class="hljs-number">5</span>)log.Printf(<span class="hljs-string">"%+v"</span>, <span class="hljs-built_in">cap</span>(t))</code></pre></div><p>结果得出的是1000和2048，符合&lt;1024 则*2</p><div class="hljs"><pre><code class="hljs go">t := <span class="hljs-built_in">make</span>([]<span class="hljs-keyword">int</span>, <span class="hljs-number">1024</span>, <span class="hljs-number">1024</span>)log.Printf(<span class="hljs-string">"%+v"</span>, <span class="hljs-built_in">cap</span>(t))t = <span class="hljs-built_in">append</span>(t, <span class="hljs-number">1</span>,<span class="hljs-number">2</span>,<span class="hljs-number">3</span>,<span class="hljs-number">4</span>,<span class="hljs-number">5</span>)log.Printf(<span class="hljs-string">"%+v"</span>, <span class="hljs-built_in">cap</span>(t))</code></pre></div><p>结果是1024和1280 ，符合&gt;1024 则 ×1.25倍</p><div class="hljs"><pre><code class="hljs go">t := <span class="hljs-built_in">make</span>([]<span class="hljs-keyword">int</span>, <span class="hljs-number">1025</span>, <span class="hljs-number">1025</span>)log.Printf(<span class="hljs-string">"%+v"</span>, <span class="hljs-built_in">cap</span>(t))t = <span class="hljs-built_in">append</span>(t, <span class="hljs-number">1</span>,<span class="hljs-number">2</span>,<span class="hljs-number">3</span>,<span class="hljs-number">4</span>,<span class="hljs-number">5</span>)log.Printf(<span class="hljs-string">"%+v"</span>, <span class="hljs-built_in">cap</span>(t))</code></pre></div><p>1025,1360 ,符合</p><p>扩容不符合</p><div class="hljs"><pre><code class="hljs go">b := []<span class="hljs-keyword">int</span>&#123;<span class="hljs-number">23</span>, <span class="hljs-number">51</span>&#125;b = <span class="hljs-built_in">append</span>(b, <span class="hljs-number">4</span>, <span class="hljs-number">5</span>, <span class="hljs-number">6</span>)fmt.Println(<span class="hljs-string">"cap of b is "</span>,<span class="hljs-built_in">cap</span>(b))</code></pre></div><p>上面输出</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-built_in">cap</span> of b is <span class="hljs-number">6</span></code></pre></div><p>因为，size=int,即是8Bytes，里面计算cap是<code>capmem = roundupsize(uintptr(newcap) * sys.PtrSize)</code>计算出来的，其中newcap=5(比oldcap<em>2=2</em>2=4要大)uintptr(newcap)<em>sys.PtrSize=5</em>8=40,但是roundup会使用<code>sizetoclass</code>表，所以会变成48;</p><h2 id="4-复制切片">4. 复制切片</h2><h3 id="使用函数">使用函数</h3><p>当使用<code>copy</code>关键字进行slice的复制时，<code>cmd/compile/internal/gc.copyany</code>函数会有两种情况:</p><ul><li>如果当前<code>copy</code>不是在runtime调用,<code>copy</code>会直接转换成以下代码:</li></ul><div class="hljs"><pre><code class="hljs go">n := <span class="hljs-built_in">len</span>(a)<span class="hljs-keyword">if</span> n &gt; <span class="hljs-built_in">len</span>(b) &#123;    n = <span class="hljs-built_in">len</span>(b)&#125;<span class="hljs-keyword">if</span> a.ptr != b.ptr &#123;    memmove(a.ptr, b.ptr, n*sizeof(elem(a))) &#125;</code></pre></div><ul><li>如果是在runtime下调用，则会使用<code>runtime.slicecopy</code>函数替换运行期间调用的<code>copy</code>:</li></ul><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">slicecopy</span><span class="hljs-params">(to, fm slice, width <span class="hljs-keyword">uintptr</span>)</span> <span class="hljs-title">int</span></span> &#123;<span class="hljs-keyword">if</span> fm.<span class="hljs-built_in">len</span> == <span class="hljs-number">0</span> || to.<span class="hljs-built_in">len</span> == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span> <span class="hljs-number">0</span>&#125;n := fm.<span class="hljs-built_in">len</span><span class="hljs-keyword">if</span> to.<span class="hljs-built_in">len</span> &lt; n &#123;n = to.<span class="hljs-built_in">len</span>&#125;<span class="hljs-keyword">if</span> width == <span class="hljs-number">0</span> &#123;<span class="hljs-keyword">return</span> n&#125;    <span class="hljs-comment">//race代码</span>...size := <span class="hljs-keyword">uintptr</span>(n) * width<span class="hljs-keyword">if</span> size == <span class="hljs-number">1</span> &#123; <span class="hljs-comment">// common case worth about 2x to do here</span><span class="hljs-comment">// <span class="hljs-doctag">TODO:</span> is this still worth it with new memmove impl?</span>*(*<span class="hljs-keyword">byte</span>)(to.array) = *(*<span class="hljs-keyword">byte</span>)(fm.array) <span class="hljs-comment">// known to be a byte pointer</span>&#125; <span class="hljs-keyword">else</span> &#123;memmove(to.array, fm.array, size)&#125;<span class="hljs-keyword">return</span> n&#125;</code></pre></div><h3 id="现象">现象</h3><p>第一个，切片:</p><div class="hljs"><pre><code class="hljs go">   a := []<span class="hljs-keyword">int</span>&#123;<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>&#125;<span class="hljs-keyword">for</span> k, v := <span class="hljs-keyword">range</span> a &#123;<span class="hljs-keyword">if</span> k == <span class="hljs-number">0</span> &#123;a[<span class="hljs-number">0</span>], a[<span class="hljs-number">1</span>] = <span class="hljs-number">100</span>, <span class="hljs-number">200</span>fmt.Print(a)&#125;a[k] = <span class="hljs-number">100</span> + v&#125;fmt.Print(a)</code></pre></div><div class="hljs"><pre><code class="hljs go">[<span class="hljs-number">100</span>,<span class="hljs-number">200</span>,<span class="hljs-number">3</span>][<span class="hljs-number">101</span>,<span class="hljs-number">300</span>,<span class="hljs-number">103</span>]</code></pre></div><p>第二个，传入是数组:</p><div class="hljs"><pre><code class="hljs go">   a := [<span class="hljs-number">3</span>]<span class="hljs-keyword">int</span>&#123;<span class="hljs-number">1</span>, <span class="hljs-number">2</span>, <span class="hljs-number">3</span>&#125;<span class="hljs-keyword">for</span> k, v := <span class="hljs-keyword">range</span> a &#123;<span class="hljs-keyword">if</span> k == <span class="hljs-number">0</span> &#123;a[<span class="hljs-number">0</span>], a[<span class="hljs-number">1</span>] = <span class="hljs-number">100</span>, <span class="hljs-number">200</span>fmt.Print(a)&#125;a[k] = <span class="hljs-number">100</span> + v&#125;fmt.Print(a)</code></pre></div><div class="hljs"><pre><code class="hljs go">[<span class="hljs-number">100</span>,<span class="hljs-number">200</span>,<span class="hljs-number">3</span>][<span class="hljs-number">101</span>,<span class="hljs-number">102</span>,<span class="hljs-number">103</span>]</code></pre></div><p>原因是针对数组而言,在loop内修改其元素数据不会第一时间反应在数据源中，这就是为什么<code>a[k]=100+v</code>覆盖了<code>k==0</code>时<code>a[1]=200</code>的条件;而针对slice，修改元素会真正修改数据源，因为slice实际是一个<code>struct</code>,其header保存着底层数组的数据;</p><h2 id="5-回收">5. 回收</h2><p>//todo</p><h2 id="再看现象">再看现象</h2><p>将一个slice y等于另外一个slice x(x已经到达其最大的capacity)的截断片，再append</p><div class="hljs"><pre><code class="hljs go"><span class="hljs-function"><span class="hljs-keyword">func</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;x:=<span class="hljs-built_in">make</span>([]<span class="hljs-keyword">int</span>,<span class="hljs-number">5</span>,<span class="hljs-number">5</span>)x[<span class="hljs-number">0</span>]=<span class="hljs-number">1</span>x[<span class="hljs-number">1</span>]=<span class="hljs-number">2</span>x[<span class="hljs-number">2</span>]=<span class="hljs-number">3</span>x[<span class="hljs-number">3</span>]=<span class="hljs-number">4</span>x[<span class="hljs-number">4</span>]=<span class="hljs-number">5</span>y:=x[<span class="hljs-number">2</span>:<span class="hljs-number">4</span>]y=<span class="hljs-built_in">append</span>(y,<span class="hljs-number">9</span>)    log.Printf(<span class="hljs-string">"%+v"</span>,y)&#125;</code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>golang</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>class loading</title>
    <link href="/2019/06/05/java/Java-Class/"/>
    <url>/2019/06/05/java/Java-Class/</url>
    
    <content type="html"><![CDATA[<a id="more"></a><p>##类的加载</p><h3 id="双亲委托模型">双亲委托模型</h3><p>双亲委派指的就是类加载器需要加载类的时候，会先把请求委托给父类加载器，依次递归，如果父类能完成，就返回，只有父类或祖类等无法完成，才自己加载;</p><p>JVM预定义的三个类型的加载器（classloader）</p><ol><li>Bootstrap加载器一般在&lt;JAVA_RUNTIME_HOME&gt;/lib下的类库加载到内存中，因为其涉及JVM本身实现细节，使用C++实现，所以开发者没法获得其引用</li><li>Extension类加载器sun的ExtClassloader负责&lt;JAVA_RUNTIME_HOME&gt;/lib/ext下或由java.ext.dir指定位置中的类库加载到内存中;</li><li>System类加载器AppClassloader实现，负责把classpath中指定的类加载;</li></ol><p>####目的防止内存出现多个同样的字节码，也能保证用到真正JDK里面提供的类;</p><p>然而这个<strong>并不是强制模型</strong></p><h4 id="需要注意：">需要注意：</h4><ol><li>加载的时候，会派出<strong>当前线程</strong>的类加载器（当前类加载器通过Thread.getContextClassLoader()，也可以通过setContextLoader()设置类加载器</li><li>如果加载该类的时候，有其他类引用此类，其他类也会被该类加载器加载</li><li>可直接调用Classloader.loadClass()指定某个类加载器</li></ol><p>SPI（service Provider Interface）//todo</p>]]></content>
    
    
    
    <tags>
      
      <tag>Java</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Notes about JVM</title>
    <link href="/2019/06/05/java/Java-JVM/"/>
    <url>/2019/06/05/java/Java-JVM/</url>
    
    <content type="html"><![CDATA[<p>不得不说，特么这兄弟归纳的太好了<img src="https://github.com/caison/java-knowledge-mind-map" srcset="/img/loading.gif" alt="地址"></p><a id="more"></a><p>字符串常量池</p><h2 id="内存结构">内存结构</h2><p><img src="/img/jvm_memStruct.jpg" srcset="/img/loading.gif" alt="旧的图" title="jvm Memstruct"></p><p><img src="/img/MemStruct.png" srcset="/img/loading.gif" alt="更特么详细的">上面的线程共享区，因为所有对象都在这里分配内存，所以也是GC的主要区域</p><p><img src="/img/HeapStruct.png" srcset="/img/loading.gif" alt="堆结构"></p><p>关于堆结构，上图是JDK8的示意图###堆</p><h4 id="新生代：">新生代：</h4><p>所有新生成的对象首先都是存放在新生代</p><h5 id="eden区">Eden区</h5><p>最主要包含了刚创建的对象，该区域对象大部分都是短期内死亡，所以垃圾回收器主要用标记-整理算法回收该区域</p><h5 id="survivor区">Survivor区</h5><p>又分为平等的两个区：</p><ol><li>From Survivor（s0）</li><li>To Survivor（S1）采用复制算法，每次只使用其中一块;</li></ol><p>Eden：Survivor=8：1</p><h4 id="老年代">老年代</h4><p>一般在Survivor中没有被清除出去的对象才会进入该区域，主要使用标记-清除（mark-sweep）算法</p><h4 id="操作">操作</h4><p>可-Xms 数字或-Xmx 数字 两个jvm参数来指定一个程序的堆内存大小，第一个是起始值，第二个是最大值</p><div class="hljs"><pre><code class="hljs java">java -Xms1M -Xmx2M helloworld</code></pre></div><h3 id="方法区-method-area">方法区（method area）</h3><p>用于存放已被加载的类信息，常量，静态变量，即时编译器编译（jit）后的代码等然而对其进行垃圾回收的话主要是对<strong>常量池</strong>和类的卸载</p><p>Hotspot虚拟机把它当作永久代进行垃圾回收，但因为很少确定永久代的大小JDK1.8开始，把永久代移除，并把方法区移至元空间（位于本地内存，不是JVM内存）</p><h4 id="运行时常量池">运行时常量池</h4><p>是方法区一部分，Class文件中的常量池（编译器生成的各种字面量和符号引用）会在类加载后被放入这个区域除了在编译期生成的常量，还允许动态生成，如string.Intern()PS：</p><ol><li>用双引号声明出来的String对象会被存储在常量池里</li><li>如果不是用双引号，intern方法会<strong>从字符串常量池中查询字符串是否存在，如果不存在就会将当前字符串放入常量池</strong></li></ol><h5 id="常见面试题">常见面试题</h5><div class="hljs"><pre><code class="hljs java">String s=<span class="hljs-keyword">new</span> String(<span class="hljs-string">"abc"</span>);</code></pre></div><p>如上创建了多少个对象？答案就是2个，一个是String在堆上的对象，一个是常量池里面的字符串”abc“</p><h4 id="操作-v2">操作</h4><p>-XX:PermSize最小空间-XX:MaxPermSize最大空间</p><h4 id="异常">异常</h4><p>和heap一样不需要连续的内存，而且动态扩展，失败会抛出OutOfMemory异常</p><h3 id="jvm-stack">JVM stack</h3><p>每个Java方法在执行的同时会创建一个栈帧用于存储局部变量表，操作数栈，常量池引用等信息;<strong>从调用到执行完成的过程，就对应了一个栈帧在jvm栈中的入栈和出栈</strong></p><h4 id="操作-v3">操作</h4><p>可以通过-Xss 设置jvm栈大小</p><div class="hljs"><pre><code class="hljs java">java -Xss512M helloworld</code></pre></div><h4 id="异常-v2">异常</h4><p>当请求栈深度超过最大值，会抛出stackoverflow异常</p><h3 id="本地方法栈">本地方法栈</h3><p>本地方法栈与JVM栈类似，只不过JVM栈为java方法（字节码）服务，本地方法为JVM的native方法服务本地方法一般由c++，汇编等编写</p><h4 id="操作-v4">操作</h4><p>Sun jdk中和jvm栈和本地方法栈是同一个，因此也可以用-Xss控制每个线程的大小</p><h3 id="程序计数器">程序计数器</h3><p>记录正在执行的虚拟机字节码指令的地址（如果正在执行的是本地方法则为空）</p>]]></content>
    
    
    
    <tags>
      
      <tag>Java</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Notes about HTTP2</title>
    <link href="/2018/07/15/Comcon/HTTP2/"/>
    <url>/2018/07/15/Comcon/HTTP2/</url>
    
    <content type="html"><![CDATA[<p>HTTP2</p><a id="more"></a><h2 id="结构">结构</h2><h2 id="相对于http1-1-改变了啥子">相对于HTTP1.1 改变了啥子</h2><h3 id="http1-1提供的特性">HTTP1.1提供的特性</h3><p>HTTP1.1有一个关键的特性：keep-alive头字段以前的HTTP协议都是规定 <strong>每次请求都要建立一次连接（3次握手）</strong>，更别提慢启动，拥塞控制这些捞东西了;</p><p>而HTTP1.1的keep-alive字段就可以保证 <strong>一定时间内，同一域名多次请求数据，只建立一次HTTP请求</strong>，其他请求可以复用这次建立的通道，这里一定时间可以通过工具配置（nginx，apache）</p><h3 id="http1-1仍然有缺陷">HTTP1.1仍然有缺陷</h3><ol><li>连接数过多，浏览器一般就会因为这个设置TCP连接限制;一般是6-8个，假设是6,如果apache最大并发数为300,服务器每次承载最多只有300/6=50而已，多过这个数就要等待</li><li>文件传输只能是<strong>串行</strong>的（在一个通道里面）</li></ol><h3 id="http2结构">HTTP2结构</h3><h4 id="hpack">HPACK</h4><p>头部压缩HTTP /1 的请求头较大，而且是以纯文本发送，HTTP/2 对消息头进行了压缩，采用的是 <code>HPACK</code> 算法；能够节省消息头占用的网络流量，其主要是在两端建立了索引表，消息头在传输时可以采用索引，而 HTTP/1.x 每次请求，都会携带大量冗余头信息，浪费了很多带宽资源。</p><h4 id="frame">Frame</h4><p>HTTP2 传输的最小单位是 Frame（帧）。HTTP2 的帧包含很多类型：<code>DATA Frame</code>、<code>HEADERS Frame</code>、<code>PRIORITY Frame</code>、<code>RST_STREAM Frame</code>、<code>CONTINUATON Frame</code> 等;</p><p>这也是多路复用的关键: 一个 HTTP2 请求/响应可以被拆成多个帧并行发送，每一帧都有一个 StreamID 来标记属于哪个 Stream。服务端收到 Frame 后，根据 StreamID 组装出原始请求数据,同一个 stream 内 frame 必须是有序的,<code>SETTINGS_MAX_CONCURRENT_STREAMS</code> 控制着最大并发数(并且这个设置仅适用于接收设置的对端);</p><p>如图:<img src="/img/http2header.png" srcset="/img/loading.gif" alt="tu"></p><ul><li>所有帧都以固定的 9 字节头开头，后跟可变长度的有效载荷，组成如下：</li><li>长度：帧有效负载的长度表示为无符号的 24 位整数</li><li>类型：8 位类型的帧，帧类型确定帧的格式和语义</li><li>标志：为特定于帧类型的布尔标志保留的 8 位字段</li><li>R：保留的 1 位字段。该位的语义未定义</li><li>流标识符：流标识符，表示为无符号 31 位整数，客户端发起流标识符必须时<strong>奇数</strong>，服务端发起的流标识符必须是<strong>偶数</strong>,流标识符零(0x0)用于<strong>连接控制消息</strong>,零流标识符不能用于建立新的 stream 流</li></ul><p>24+8+8+1+31=9B*8=72bit</p><blockquote><blockquote><p>常用的标志位有 <code>END_HEADERS</code> 表示头数据结束，相当于 HTTP/1 里头后的空行（“\r\n”），<code>END_STREAM</code> 表示单方向数据发送结束（即 EOS，End of Stream），相当于 HTTP/1 里 Chunked 分块结束标志（“0\r\n\r\n”）</p></blockquote></blockquote><h3 id="http2多路复用">HTTP2多路复用</h3><ol><li><p>连接数过多的问题：HTTP2在<strong>同一域名</strong>下所有连接都基于<strong>流</strong>，都在同一个连接，比如上面的并发为300,用HTTP2就能达到300;</p></li><li><p>因为HTTP1.1传输的request和response都是<strong>基于文本的</strong>，所以所有数据必须按<strong>顺序传输</strong>才能保证可靠性（TCP）;然而HTTP2前面说到利用的是<strong>二进制数据帧</strong>和<strong>流</strong>（TCP其实也是流），帧就是标识了数据的顺序！即不必要传完一个request，等到response再继续下一个;</p></li><li><p>个人认为本质上，HTTP2为了解决<code>head of line blocking</code>这种问题，可以理解为将 HTTP1.1 这种<code>一条通道</code>的 切割成<code>多个通道</code>(现在大多数的stream数目为<code>100</code>)，每个通道放不同的stream，达到了<code>multiplexing</code>的效果, 上面的flow control就在应用层上细化了粒度，管理这样不同<code>通道</code>的速度（TCP可以理解为为管理整个连接）</p></li></ol><h2 id="question">Question</h2><h3 id="why-not-use-http2-as-download-protocol">why not use HTTP2 as download protocol?</h3><p><a href="https://stackoverflow.com/questions/44019565/http2-file-download" target="_blank" rel="noopener">StackOverflow上有大佬提出</a></p><p>主要是两个点：</p><ol><li>frame overhead</li></ol><p>HTTP1.1 每次传输实际都是content的bytes，但是HTTP2每次传输<code>DATA</code>frame额外的<code>9 bytes</code>的header</p><ol start="2"><li>flow control</li></ol><p>http2的<code>server</code>端会控制给每一个session和每一个在这个session里的stream的发送窗口,默认是<code>65535</code>bytes，如果要开启下载，一定要扩大该窗口;然而问题来了，扩大该窗口是<code>client</code>端发送<code>WINDOW_UPDATE</code>进行扩大的，万一遇到延迟，这个传输无效，那么窗口只可以保持默认值大小</p><h3 id="造成的问题-something-not-compatible-with-http2">造成的问题? Something not compatible with HTTP2?</h3><ol><li>JS文件合并:以前多个模块（文件）会合并成一个文件，当有模块要修改的时候，会全部上传一遍;</li><li>多域名进行文件传输(domain sharding)，会导致<strong>DNS解析时间过长</strong>，<strong>增加服务端压力</strong>等</li></ol>]]></content>
    
    
    
    <tags>
      
      <tag>networking</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Notes about Raft and Paxos</title>
    <link href="/2018/07/10/Comcon/DistributeProtocol/"/>
    <url>/2018/07/10/Comcon/DistributeProtocol/</url>
    
    <content type="html"><![CDATA[<p>复习一些分布式理论: Raft和Paxos(暂时只讨论算法情况，工程情况留坑)</p><a id="more"></a><h2 id="paxos">PAXOS</h2><p>paxos实际上是一系列的协议，有basic-paxos, multi-paxos</p><p>目的：让多个参与者达成一致，共识</p><p><strong>一个原则</strong>：参与者如果达成一致，这个一致的观点在传递中永不会改变</p><h3 id="basic-paxos">basic-Paxos</h3><h4 id="一些角色和名词">一些角色和名词</h4><ol><li>client</li></ol><p>只是负责发起request到分布式系统并等待相应</p><ol start="2"><li><p>Proposer（发起者）每个Proposer也是一个Acceptor它负责从client 发起请求，尝试让Acceptor来同意这个请求，也会在冲突发生的时候作为一个合作者来使协议继续执行（propser可以不断发起提议，不用等提议完成）</p></li><li><p>Acceptor（Voter）每个Acceptor也是一个Proposer<strong>多个Acceptor</strong>组成一个Quorums（法定多数），所有发向一个Acceptor的必须要发给Acceptor的一个Quorum,不会同意比自己以前接受过的提案编号小的提案，任何从一个Acceptor来的信息会被忽略，除非能接受到一个从这个Quorum里所有的Acceptor来的拷贝？？？</p></li><li><p>Quorums（多数派）</p></li></ol><p>Quorums被定义为某些Accetors的集合的子集，即任何两个Quorums都会有至少一个Acceptor交集，而且Quorums总会是包含大部分的Acceptor;比如：有一个Accetpros的集合{A，B，C，D}，大部分的Quorums可以是任意三个Acceptor组成：{A，B，C}，{A，C，D}等等而经常Acceptor还会携带不同的权值，但保证的是Quorums所包含的所有Acceptors的权值和都会<strong>大于</strong>总的权值和（所有的Acceptors）的一半;</p><ol start="5"><li>Learner</li></ol><p>只是作为协议的复制因素，一旦client的request被Acceptor同意，learner就会开始执行这个request以及返回给client一个响应;一般为了提高协议的可用性，可以额外增加多个learner</p><h4 id="基本流程">基本流程</h4><p>每个basic-paxos的实例（或执行者）决定于一个输出值。该协议在多轮通信中进行;成功的有两轮，每轮有a，b，我们假定是在一个异步模型里面，即一个processor可能在第一轮，另一个可能在第二轮</p><p>首先进行典型的2PC 协议</p><h5 id="phase-1：">Phase 1：</h5><ol><li>1a：Prepare</li></ol><p><code>Proposer</code>创建一个信息，我们称为<strong>Prepare</strong>，附带一个<strong>唯一标识数字 n</strong> :</p><ul><li><p>而且当前的 n 要大于这个Proposer之前发的信息附带的所有 nX;</p></li><li><p>每次n = ++maxProposal;</p></li><li><p><code>Porcessor</code>把这个带有 n 的Prepare信息（这里只带有n(id)，<strong>实际上它不用带其他信息</strong>，比如提议内容）发到在一个Quorum的<code>Acceptor</code>(实际就是集群上的所有机器)上（哪些Acceptor在Quorum里面是由Proposer决定的???)</p></li></ul><p>如果Proposer不能与至少一个Quorum通信则不应该初始化Paxos</p><ol start="2"><li>1b：Promise（两个承诺，一个应答）</li></ol><p>每一个<code>Acceptor</code>会等待<code>Proposer</code>的<code>Prepare</code>信息，如果一个Acceptor收到了，它会查看附带的n标识(id)，会有两种情况：</p><ol><li><p>如果<strong>n出现&lt;=之前收到的任何一个proposal n</strong> (不再应答proposal n <strong>&lt;=</strong> 当前请求的<code>Propose</code>)!!!</p><ul><li>则不接受proposal n&lt;=当前请求的prepare请求，<code>Acceptor</code>能忽略这个提议;</li><li>为了优化，还是可回一个denial response，提示Proposer可以停止创建带有n的提议了，且该response可以带上一个当前Acceptor的promiseProposal，以便Proposer的更新;</li></ul></li><li><p>如果<strong>n比之前收到的所有proposal n都大</strong> (不再应答proposal n <strong>&lt;</strong> 当前请求的<code>Accept</code> 请求 )!!</p><ul><li>这个<code>Acceptor</code>一定要返回一个信息Promise给对应的<code>Proposer</code>，然后会忽视所有未来的附带标识小于n的提议（信息）;</li><li>如果这个<code>Acceptor</code>以前接收过<strong>其他提议</strong>，返回给这个Proposer的response一定要包含之前的提议编号m，和对应的值w，没有就返回空;（这里Acceptor还会<strong>持久化</strong>该proposal值）;</li></ul></li></ol><p>注意上面两者的比较, <code>Acceptor</code>一定要接收Propose &gt; 当前n的请求 以及 可以接收 &gt;= 当前Accept的n 的请求</p><h5 id="phase-2：">Phase 2：</h5><ol><li>2a： Accept(propose)</li></ol><p>如果一个<code>Proposer</code>收到了的从Quorum返回的response，</p><ol><li>如果未超过一半的<code>Acceptors</code>同意，提议失败</li><li>如果超过了一半：</li></ol><div class="hljs"><pre><code>- 如果有**部分**`Acceptor`接到过内容，会从所有`Acceptor`接受过的内容(response)中，**选择proposal n最大**的内容作为真正的内容(value)，提议编号仍然为 n，**但这时Proposer就不能提议自己的内容，只能信任Acceptor通过的内容**;- 如果**所有**`Acceptor`应答的proposal内容都为null，即可随意决定发起proposal的内容(Value自己定)，然后带上当前的**proposal n**，向所有Acceptor再发送提议;</code></pre></div><ol start="2"><li>2b：Accepted</li></ol><p>如果<code>Acceptor</code>接收到了提议后，他必须遵循：</p><div class="hljs"><pre><code>- 如果有且仅有不违背 **Phase1b（两个承诺）**情况下（即该提议n等于之前Phase1保存的编号），记录下(**持久化**）当前proposal n 和内容(value);</code></pre></div><p>最后<code>Proposer</code>收到Quorum返回的Accept response后，形成决议</p><h4 id="图解在算法异常的情况下-工程下更加复杂-先占坑-：">图解在算法异常的情况下（工程下更加复杂，先占坑）：</h4><ol><li>没有失败的情况下</li></ol><p>有一个client，1一个proposer，3个Acceptor，和两个learnner;</p><div class="hljs"><pre><code class="hljs undefined">Client   Proposer      Acceptor     Learner   |<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|   X--------&gt;|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Request   </span>|<span class="hljs-string">         X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(1)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">&lt;---------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Promise(1,&#123;Va,Vb,Vc&#125;)   </span>|<span class="hljs-string">         X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Accept!(1,V)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">&lt;---------X--X--X------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">  Accepted(1,V)   </span>|<span class="hljs-string">&lt;---------------------------------X--X  Response   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|Here, V is the last of (Va, Vb, Vc).</code></pre></div><ol start="2"><li><p>Acceptor失败的情况</p></li><li><p>Quorum里面有一个Acceptor失败了，所以整个Quorum大小会变成2，整个paxos还是成功的（此时Quorum数目&gt; Acceptors/2)</p></li><li><p>如果有两个或三个失败，则直接返回失败;或Proposer重新发起提议（内容一样，提议号n+1）但是，对第一次已经成功接收的acceptor不会修改，其余上一次失败的acceptors才会接收提议</p></li></ol><div class="hljs"><pre><code class="hljs undefined">Client   Proposer      Acceptor     Learner   |<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|   X--------&gt;|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Request   </span>|<span class="hljs-string">         X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(1)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! FAIL !!   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">&lt;---------X--X          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Promise(1,&#123;Va, Vb, null&#125;)   </span>|<span class="hljs-string">         X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Accept!(1,V)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">&lt;---------X--X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">  Accepted(1,V)   </span>|<span class="hljs-string">&lt;---------------------------------X--X  Response   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|</code></pre></div><ol start="3"><li>Leaner也可能会失败</li></ol><p>虽然有一个learner失败了，但整个paxos还是成功的</p><div class="hljs"><pre><code class="hljs undefined">Client Proposer         Acceptor     Learner   |<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|   X--------&gt;|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Request   </span>|<span class="hljs-string">         X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(1)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">&lt;---------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Promise(1,&#123;Va,Vb,Vc&#125;)   </span>|<span class="hljs-string">         X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Accept!(1,V)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">&lt;---------X--X--X------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">  Accepted(1,V)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  !  !! FAIL !!   </span>|<span class="hljs-string">&lt;---------------------------------X     Response   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|</code></pre></div><ol start="4"><li>Proposer失败</li></ol><p>在Proposer接收到proposal值（提议内容）之后，发回的Accept阶段失败了，只有一个Acceptor接到提议;同时，一个新的Proposer被选举出来：</p><div class="hljs"><pre><code class="hljs undefined">Client  Proposer        Acceptor     Learner   |<span class="hljs-string">      </span>|<span class="hljs-string">             </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|   X-----&gt;|<span class="hljs-string">             </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Request   </span>|<span class="hljs-string">      X------------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(1)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">&lt;------------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Promise(1,&#123;Va, Vb, Vc&#125;)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">             </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|   |<span class="hljs-string">      </span>|<span class="hljs-string">             </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! Leader fails during broadcast !!   </span>|<span class="hljs-string">      X------------&gt;</span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Accept!(1,V)   </span>|<span class="hljs-string">      !             </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|   |<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! NEW LEADER !!   </span>|<span class="hljs-string">         X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(2)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">&lt;---------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Promise(2,&#123;V, null, null&#125;)   </span>|<span class="hljs-string">         X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Accept!(2,V)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">&lt;---------X--X--X------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">  Accepted(2,V)   </span>|<span class="hljs-string">&lt;---------------------------------X--X  Response   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|</code></pre></div><ol start="5"><li>多个Proposer冲突 ！！！！锁住了对方的Accept，导致prepare的值全部作废这里问题大了。。。。（其实就引出了multi-paxos）</li></ol><div class="hljs"><pre><code class="hljs undefined">Client   Leader         Acceptor     Learner   |<span class="hljs-string">      </span>|<span class="hljs-string">             </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|   X-----&gt;|<span class="hljs-string">             </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Request   </span>|<span class="hljs-string">      X------------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(1)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">&lt;------------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Promise(1,&#123;null,null,null&#125;)   </span>|<span class="hljs-string">      !             </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! LEADER FAILS   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! NEW LEADER (knows last number was 1)   </span>|<span class="hljs-string">         X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(2)   </span>|<span class="hljs-string">         </span>|<span class="hljs-string">&lt;---------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Promise(2,&#123;null,null,null&#125;)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! OLD LEADER recovers   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! OLD LEADER tries 2, denied   </span>|<span class="hljs-string">      X------------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(2)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">&lt;------------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Nack(2)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! OLD LEADER tries 3   </span>|<span class="hljs-string">      X------------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(3)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">&lt;------------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Promise(3,&#123;null,null,null&#125;)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! NEW LEADER proposes, denied   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Accept!(2,Va)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  </span>|<span class="hljs-string">&lt;---------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Nack(3)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! NEW LEADER tries 4   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  X---------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Prepare(4)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  </span>|<span class="hljs-string">&lt;---------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Promise(4,&#123;null,null,null&#125;)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  !! OLD LEADER proposes, denied   </span>|<span class="hljs-string">      X------------&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">-&gt;</span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Accept!(3,Vb)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">&lt;------------X--X--X       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  Nack(4)   </span>|<span class="hljs-string">      </span>|<span class="hljs-string">  </span>|<span class="hljs-string">          </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  </span>|<span class="hljs-string">       </span>|<span class="hljs-string">  </span>|<span class="hljs-string">  ... and so on ...</span></code></pre></div><h2 id="multi-paxos">Multi-Paxos</h2><p>multi-Paxos将集群分为两种状态</p><p>以上参考 *** <img src="https://webcache.googleusercontent.com/search?q=cache:zXcryn67tFcJ:https://en.wikipedia.org/wiki/Paxos_(computer_science)+&amp;cd=2&amp;hl=en&amp;ct=clnk" srcset="/img/loading.gif" alt="wiki-Paxos"> ***</p><h2 id="raft">Raft</h2><p>解决一致性问题的三个子问题</p><p>比较简单，可以从几个方面进行理解，为</p><ul><li>主从筛选(leader election)</li><li>日志复制(log replication)</li><li>安全性(Safety,leader变更时)</li></ul><h3 id="主从筛选">主从筛选</h3><p>保证任何时期最多只有一个leader，leader节点有所有已提交的日志</p><h3 id="日志复制">日志复制</h3><p>指的是Raft保证每个副本日志append的<strong>连续性</strong></p><p>leader会为每个follower维护一个nextIndex，表示leader给各个follower发送的下一条log entry在log中的index。</p><p>如果日志append的时候前一个日志还没有append，则必须等到前一个日志append后才能append。也就是说不同副本上相同index的日志，只要term相同，那么这两条日志必然相同，且这之前的日志必然也相同。</p><h3 id="安全性">安全性</h3><p>Leader只能附加的原则，只允许leader commit被大部分append的log entry;即如果当前的log已被commit，证明在这之前的所有log都被提交</p><h4 id="节点错误处理">节点错误处理</h4><p>宕机等有几种情况：</p><ol><li><p>Followers 或 CandidatesFollowers 或者candidates 崩溃, 解决办法只需要leader不断重试发送请求即可， 再不行就重启该崩溃的服务器，就能收到AppendEntriesRPC和Requestvotes请求</p></li><li><p>Leader但是Leader 崩溃，如果直接重启服务器：在完成了一次RPC发送但没有接收response的时候重启，他会再次收到一个相同的RPC，但是因为Raft RPC是幂等的，所以这个没有关系，举个例子，如果一个follower接到了AppendEntries请求，而且发现这个请求的log entries已经在自己的log里面了，他会忽略这个请求</p></li><li><p>网络分区，多数派的leader照常工作，少数派按照道理是不能选出leader，所以没有工作，恢复后少数派没有最新的log，所以肯定是成为follower，leader只需要appendEntries回复这些followers的log即可</p></li></ol><h4 id="时间和可用性">时间和可用性</h4><p>时间指timing，这里尤其是leader的选举，是时间敏感的，Raft有个公式，可以保证能选举和维护一个稳定的leader:</p><blockquote><blockquote><p>broadcastTime&lt;&lt; electionTimeout &lt;&lt; MTBF</p></blockquote></blockquote><p>broadcastTime: 广播的平均时间（某个节点）electionTimeout: 选举的时间MTBF: 前一次失败和这一次失败间隔的平均时间（某个节点）</p><p>broadcastTime&lt;&lt; electionTimeout 可以保证leader发送heartbeat给其他followers，防止他们开始选举</p><p>electionTimeout&lt;&lt; MTBF 可以保证系统平稳进步？</p><p>一般来说， broadcastTime只会在0.5ms ~ 20ms取决于持久化的技术，所以相应的，electionTimeout就会是在10ms~500ms之间， 比较重要的节点的MTBF则取几个月或更多</p><h3 id="节点成员改变">节点成员改变</h3><p>我们之前都是假定节点配置是不变的，但实际上当有server crash的时候就需要替换他们虽然我们可以把所有接地都下线，更新配置，然后上线，但这明显有问题</p><ol><li>保证配置的安全， 一个term期间，在有可能有两个leader被选举的时候进行配置的传输，但在转移配置的时候，不能保证所有server都能第一时间拿到新的配置，可能会导致节点分裂成两部分，两个独立的大多数</li></ol><p>为了保证安全，只能分为 <strong>两步提交</strong>一些系统就会做 第一步：关闭旧的配置，第二开启新配置</p><p>在raft里面，节点第一次切换到新的配置，我们称之为joint consensus（交叉共识），一旦交叉共识被commit，整个系统就都转到了新的配置上</p><h4 id="leadercrash">LeaderCrash</h4><p>leader挂了的情况:</p><h4 id="leaderstickness">LeaderStickness</h4><p>Leader(或者某个服务器)被隔绝开集群</p><ul><li>自己就会选自己为leader，<strong>不断增加</strong>term，</li><li>再次加入到集群中，自己的term远远大于当前集群的term，leader stepDown</li></ul><p>解决方法:Pre-Vote:</p><ul><li>新增一种<code>Pre-Vote</code> RPC( curTerm+1, lastLogIndex, lastLogTerm )</li><li>不修改server的任何状态</li><li>Pre-Vote RPC多数派返回成功，本地Term++，将节点状态转换为Candidate，发出真正的Vote RPC</li><li>分区后节点，Pre-Vote不会成功，term不会增加，再次加入就不会导致集群异常;</li></ul><p>缺点:???</p><ul><li>个人认为增加了一个RPC，有延迟</li></ul><h4 id="分区">分区</h4><p>增加<code>preVote</code>的原因主要是在于，B被分区后每次electiontimeout后，<code>term</code>都会增加，网络分区恢复后，如果leader收到比自己大的<code>term</code>消息，leader就会认为自己已经不是leader，然后发起选举流程，但其实是没必要的。为了让B 的term不增加，所以加了个<code>preVote</code>流程。</p><h4 id="joint-consensus">Joint consensus</h4><p>交叉共识包含log entries会被复制到新旧配置中的所有server任何节点在任一配置中都能被当做leader共识（这里针对选举和提交）要分开的大多数配置通过 （旧的大多数和新的大多数）</p><h2 id="raft缺点">Raft缺点</h2><p>每次都是串行投票,串行apply</p><p><a href="https://github.com/maemual/raft-zh_cn/blob/master/raft-zh_cn.md" target="_blank" rel="noopener">大牛的证明（中文）</a></p>]]></content>
    
    
    
    <tags>
      
      <tag>Distributed</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Notes about Design Pattern（java）</title>
    <link href="/2018/04/30/Comcon/DesignPattern/"/>
    <url>/2018/04/30/Comcon/DesignPattern/</url>
    
    <content type="html"><![CDATA[<a id="more"></a><p>For reviewing some most likely used design pattern:</p><h2 id="singleton"><strong>Singleton</strong></h2><p>Most commonly</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Singleton</span></span>&#123;    <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">final</span> Singleton singleton = <span class="hljs-keyword">new</span> Singleton();    <span class="hljs-function"><span class="hljs-keyword">private</span> <span class="hljs-title">Singleton</span><span class="hljs-params">()</span></span>&#123;&#125;<span class="hljs-comment">//in case of creating a new object</span>    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> Singleton <span class="hljs-title">getSingleton</span><span class="hljs-params">()</span></span>&#123;        <span class="hljs-keyword">return</span> singleton;    &#125;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-title">other</span><span class="hljs-params">()</span></span>&#123;        <span class="hljs-comment">//other method</span>    &#125;&#125;</code></pre></div><p>Pros:</p><ol><li><p>Because only one instance exists in memory, it greatly decrease the usage of memory.Especially in the case that an object needs to be initialized and deleted frequently,while the instantiation and deletion can’t be optimized, Singleton pattern is the best choice.</p></li><li><p>Prevent multiple domination towards the same source.For example, an operation of writing in a file.</p></li><li></li></ol><p>Cons:</p><ol><li>Obviously, because singleton has instatiated itself, so it’s impossible to extend interface to it (abstract class,interface will not be instantiated )</li></ol><p>Scenes:</p><ol><li>generate unique key;</li><li>IO;</li><li>global stored in memory for calculating purpose.For exmaple, Spring Bean is set as a singleton by default so that spring can manage this bean life cycle(but if Bean use non-singleton like prototype pattern,after the initialization of Bean, it will be transfered to J2EE container,Spring no longer manages this Bean)</li></ol><h2 id="proxy"><strong>Proxy</strong></h2><h3 id="static-proxy"><strong>static proxy</strong></h3><p>We assume there is an interface:</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">interface</span> <span class="hljs-title">IUser</span></span>&#123;    <span class="hljs-function"><span class="hljs-keyword">void</span> <span class="hljs-title">login</span><span class="hljs-params">()</span></span>;&#125;</code></pre></div><p>Our target UserDAO:</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">User</span> <span class="hljs-keyword">implements</span> <span class="hljs-title">IUser</span></span>&#123;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">login</span><span class="hljs-params">()</span></span>&#123;        System.out.Println(<span class="hljs-string">"you've logined"</span>);    &#125;&#125;</code></pre></div><p>Proxy Object: UserProxy</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">UserProxy</span> <span class="hljs-keyword">implements</span> <span class="hljs-title">IUser</span></span>&#123;    <span class="hljs-keyword">private</span> User target;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-title">UserProxy</span> <span class="hljs-params">(User target)</span></span>&#123;        <span class="hljs-keyword">this</span>.target=target;    &#125;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">login</span><span class="hljs-params">()</span></span>&#123;        System.out.println(<span class="hljs-string">"begin login"</span>);        target.login();        System.out.println(<span class="hljs-string">"login end"</span>);    &#125;&#125;</code></pre></div><p><strong>How to Use:</strong></p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Test</span></span>&#123;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">main</span><span class="hljs-params">(String[] args)</span></span>&#123;        User target=<span class="hljs-keyword">new</span> User();        UserProxy p=<span class="hljs-keyword">new</span> UserProxy(target);        p.login();    &#125;&#125;</code></pre></div><h4 id="pros">Pros:</h4><p>It can extend the target’s function without change target’s code</p><h4 id="cons">Cons:</h4><p>If methods in interface increase, both proxy class and target class have to be recodedProxy class <strong>must achieve the same interface</strong> as target class’s, so it may cause loads of proxy class</p><h3 id="dynamic-proxy"><strong>dynamic proxy</strong></h3><h4 id="jdk">JDK</h4><p>Spring 默认使用JDK，<strong>proxy-target-class=false</strong>时使用jdk，true的时候用CGllib代理Able to solve cons in Static proxy:</p><ol><li>you <strong>don’t have to implements interface{} in Proxy class(But the target class must implement interface)</strong>动态代理的区别主要就是，代理的proxy class并<strong>不用自己实现接口</strong>，而是在runtime的时候动态在内存生成</li></ol><p>Creating a dynamic proxy is a static method in <strong>java.lang.reflect.Proxy</strong> package</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-function"><span class="hljs-keyword">static</span> Object <span class="hljs-title">newProxyInstance</span><span class="hljs-params">(ClassLoader loader, Class&lt;?&gt;[] interfaces,InvocationHandler h)</span></span>&#123;&#125;</code></pre></div><p>用法：</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">GamePlayIH</span> <span class="hljs-keyword">implements</span> <span class="hljs-title">InvocationHandler</span> </span>&#123;     <span class="hljs-comment">//被代理者</span>     Class cls =<span class="hljs-keyword">null</span>;     <span class="hljs-comment">//被代理的实例</span>     Object obj = <span class="hljs-keyword">null</span>;     <span class="hljs-comment">//我要代理谁</span>     <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-title">GamePlayIH</span><span class="hljs-params">(Object _obj)</span></span>&#123;             <span class="hljs-keyword">this</span>.obj = _obj;     &#125;     <span class="hljs-comment">//调用被代理的方法</span>     <span class="hljs-function"><span class="hljs-keyword">public</span> Object <span class="hljs-title">invoke</span><span class="hljs-params">(Object proxy, Method method, Object[] args)</span>                    <span class="hljs-keyword">throws</span> Throwable </span>&#123;                                   Object result = method.invoke(<span class="hljs-keyword">this</span>.obj, args);             <span class="hljs-keyword">return</span> result;     &#125;&#125;</code></pre></div><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Test</span></span>&#123;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;        User u = <span class="hljs-keyword">new</span> User();<span class="hljs-comment">//注意要有一个实例才可以</span>        InvocationHandler i=<span class="hljs-keyword">new</span> GamePlayIH(u);                IUser dynamicUser=(IUser) Proxy.newProxyInstance(User.class.getClassLoader(),User.class.getInterfaces(),i);        dynamicUser.login();    &#125;&#125;</code></pre></div><p>看下newProxyInstance源码：</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> Object <span class="hljs-title">newProxyInstance</span><span class="hljs-params">(ClassLoader loader,Class&lt;?&gt;[] interfaces,InvocationHandler h)</span> <span class="hljs-keyword">throws</span> IllegalArgumentException    </span>&#123;        Objects.requireNonNull(h);        <span class="hljs-keyword">final</span> Class&lt;?&gt;[] intfs = interfaces.clone();        <span class="hljs-comment">/*         * Look up or generate the designated proxy class.         */</span>        Class&lt;?&gt; cl = getProxyClass0(loader, intfs);        <span class="hljs-comment">/*         * Invoke its constructor with the designated invocation handler.         */</span>        <span class="hljs-keyword">try</span> &#123;            <span class="hljs-keyword">final</span> Constructor&lt;?&gt; cons = cl.getConstructor(constructorParams);            <span class="hljs-keyword">final</span> InvocationHandler ih = h;            <span class="hljs-keyword">if</span> (!Modifier.isPublic(cl.getModifiers())) &#123;                AccessController.doPrivileged(<span class="hljs-keyword">new</span> PrivilegedAction&lt;Void&gt;() &#123;                    <span class="hljs-function"><span class="hljs-keyword">public</span> Void <span class="hljs-title">run</span><span class="hljs-params">()</span> </span>&#123;                        cons.setAccessible(<span class="hljs-keyword">true</span>);                        <span class="hljs-keyword">return</span> <span class="hljs-keyword">null</span>;                    &#125;                &#125;);            &#125;            <span class="hljs-keyword">return</span> cons.newInstance(<span class="hljs-keyword">new</span> Object[]&#123;h&#125;);        &#125; <span class="hljs-keyword">catch</span> (IllegalAccessException|InstantiationException e) &#123;            <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> InternalError(e.toString(), e);        &#125; <span class="hljs-keyword">catch</span> (InvocationTargetException e) &#123;            Throwable t = e.getCause();            <span class="hljs-keyword">if</span> (t <span class="hljs-keyword">instanceof</span> RuntimeException) &#123;                <span class="hljs-keyword">throw</span> (RuntimeException) t;            &#125; <span class="hljs-keyword">else</span> &#123;                <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> InternalError(t.toString(), t);            &#125;        &#125; <span class="hljs-keyword">catch</span> (NoSuchMethodException e) &#123;            <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> InternalError(e.toString(), e);        &#125;    &#125;</code></pre></div><p>So how to generate the Proxy Class? <strong>getProxyClass0</strong></p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> Class&lt;?&gt; getProxyClass0(ClassLoader loader， Class&lt;?&gt;... interfaces) &#123;    <span class="hljs-keyword">if</span> (interfaces.length &gt; <span class="hljs-number">65535</span>) &#123;<span class="hljs-comment">//为啥不能超过65535？数字怎么来？//网络MTU？？jvm限制方法数目不能大于这个数</span>        <span class="hljs-keyword">throw</span> <span class="hljs-keyword">new</span> IllegalArgumentException(<span class="hljs-string">"interface limit exceeded"</span>);    &#125;    <span class="hljs-comment">// If the proxy class defined by the given loader implementing</span>    <span class="hljs-comment">// the given interfaces exists, this will simply return the cached copy;</span>    <span class="hljs-comment">// otherwise, it will create the proxy class via the ProxyClassFactory</span>    <span class="hljs-keyword">return</span> proxyClassCache.get(loader, interfaces);<span class="hljs-comment">//proxyClass的获取可以从Cache里面得到，也可以从proxyFactory里面得到：</span>&#125;</code></pre></div><p>COntinue to <strong>proxyFactory</strong></p><div class="hljs"><pre><code class="hljs java">/**     * A factory function that generates, defines and returns the proxy class given     * the ClassLoader and array of interfaces.     */    private static final class ProxyClassFactory implements BiFunction&lt;ClassLoader, Class&lt;?&gt;[], Class&lt;?&gt;&gt;    &#123;        // Proxy class会附带前缀 $Proxy        private static final String proxyClassNamePrefix = "$Proxy";        // next number to use for generation of unique proxy class names        private static final AtomicLong nextUniqueNumber = new AtomicLong();        @Override        public Class&lt;?&gt; apply(ClassLoader loader, Class&lt;?&gt;[] interfaces) &#123;            Map&lt;Class&lt;?&gt;, Boolean&gt; interfaceSet = new IdentityHashMap&lt;&gt;(interfaces.length);            for (Class&lt;?&gt; intf : interfaces) &#123;                /*                 * Verify that the class loader resolves the name of this                 * interface to the same Class object.                 */                Class&lt;?&gt; interfaceClass = null;                try &#123;                    interfaceClass = Class.forName(intf.getName(), false, loader);                &#125; catch (ClassNotFoundException e) &#123;                &#125;                if (interfaceClass != intf) &#123;//可能会对classloader隐藏                    throw new IllegalArgumentException(                        intf + " is not visible from class loader");                &#125;                /*                 * Verify that the Class object actually represents an                 * interface.                 */                if (!interfaceClass.isInterface()) &#123;                    throw new IllegalArgumentException(                        interfaceClass.getName() + " is not an interface");                &#125;                /*                 * Verify that this interface is not a duplicate.                 */                if (interfaceSet.put(interfaceClass, Boolean.TRUE) != null) &#123;                    throw new IllegalArgumentException(                        "repeated interface: " + interfaceClass.getName());                &#125;            &#125;            String proxyPkg = null;     // package to define proxy class in            int accessFlags = Modifier.PUBLIC | Modifier.FINAL;            /*             * Record the package of a non-public proxy interface so that the             * proxy class will be defined in the same package.  Verify that             * all non-public proxy interfaces are in the same package.             */            for (Class&lt;?&gt; intf : interfaces) &#123;                int flags = intf.getModifiers();                if (!Modifier.isPublic(flags)) &#123;                    accessFlags = Modifier.FINAL;                    String name = intf.getName();                    int n = name.lastIndexOf('.');                    String pkg = ((n == -1) ? "" : name.substring(0, n + 1));                    if (proxyPkg == null) &#123;                        proxyPkg = pkg;                    &#125; else if (!pkg.equals(proxyPkg)) &#123;                        throw new IllegalArgumentException(                            "non-public interfaces from different packages");                    &#125;                &#125;            &#125;            if (proxyPkg == null) &#123;                // if no non-public proxy interfaces, use com.sun.proxy package                proxyPkg = ReflectUtil.PROXY_PACKAGE + ".";            &#125;            /*             * Choose a name for the proxy class to generate.             */            long num = nextUniqueNumber.getAndIncrement();            String proxyName = proxyPkg + proxyClassNamePrefix + num;//这里就是动态生成proxy名字的地方            /*             * Generate the specified proxy class.             */             //这里就是生成proxy class的地方            byte[] proxyClassFile = ProxyGenerator.generateProxyClass(                proxyName, interfaces, accessFlags);            try &#123;                return defineClass0(loader, proxyName,//defineClass0似乎是到字节码的代码了，没法进入                                    proxyClassFile, 0, proxyClassFile.length);            &#125; catch (ClassFormatError e) &#123;                /*                 * A ClassFormatError here means that (barring bugs in the                 * proxy class generation code) there was some other                 * invalid aspect of the arguments supplied to the proxy                 * class creation (such as virtual machine limitations                 * exceeded).                 */                throw new IllegalArgumentException(e.toString());            &#125;        &#125;    &#125;</code></pre></div><h4 id="cglib-code-generation-library">CGLIB（Code Generation Library)</h4><p><strong>If the target object doesn’t implement any interface{}</strong>you can use <strong>Cglib proxy</strong></p><p>Its principle is that it will create an object <strong>inherit</strong> to target class整个CGlib是基于ASM字节码的生成和转换的库也是用上面的例子,没有implement任何interface</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">UserNo</span></span>&#123;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">login</span><span class="hljs-params">()</span></span>&#123;        System.out.println(<span class="hljs-string">"login without interface"</span>);    &#125;&#125;</code></pre></div><p>How to use:</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">UserMethodInterceptor</span> <span class="hljs-keyword">implements</span> <span class="hljs-title">MethodInterceptor</span></span>&#123;    <span class="hljs-comment">//..</span>    <span class="hljs-meta">@Override</span>    <span class="hljs-function"><span class="hljs-keyword">public</span> Object <span class="hljs-title">intercept</span><span class="hljs-params">(Obj obj, Method method, Object[] args,MethodProxy proxy)</span> <span class="hljs-keyword">throws</span> Throwable</span>&#123;        System.out.println(<span class="hljs-string">"method name:"</span>+ method.getName());        System.out.println(<span class="hljs-string">"method declaring CLass:"</span>+ method.getDeclaringClass());        System.out.println(<span class="hljs-string">"this is "</span>+ (String)proxy.invokeSuper(obj,args));        System.out.println(<span class="hljs-string">"method name:"</span>+ method.getName());            &#125;&#125;</code></pre></div><div class="hljs"><pre><code class="hljs java"><span class="hljs-keyword">public</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Test</span></span>&#123;    <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">void</span> <span class="hljs-title">main</span><span class="hljs-params">()</span></span>&#123;        Enhancer enhancer=<span class="hljs-keyword">new</span> Enhancer();        enhancer.setSuperClass(User.class);<span class="hljs-comment">//根本不用类的实例传入（与JDK有点不一样）</span>        enhancer.setCallback(<span class="hljs-keyword">new</span> UserMethodINterceptor);        User u=(User) enhancer.create();                System.out.println(<span class="hljs-string">"login proxy:"</span>);        u.login();    &#125;&#125;</code></pre></div><p>前面说到因为这是基于继承的动态代理，所以因为 <strong>final</strong> 修饰的方法等是无法被继承的，所以像getClass(),wait()等并不会进行代理如果要强行代理，便会出错：</p><div class="hljs"><pre><code class="hljs java">java.lang.IllegalArgumentException: Cannot subclass <span class="hljs-keyword">final</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">cglib</span>.<span class="hljs-title">HelloConcrete</span></span></code></pre></div><p>源码在<strong>java.net.sf.cglib</strong></p><p>里面暂时注意Enhancer class和MethodInterceptor Class</p>]]></content>
    
    
    
    <tags>
      
      <tag>Design Pattern</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Java-Concurrency</title>
    <link href="/2018/03/07/java/Java-Concurrency/"/>
    <url>/2018/03/07/java/Java-Concurrency/</url>
    
    <content type="html"><![CDATA[<p>Actually we’ve learned the concept about multithread and the problem may cause.For instance, deadlock.</p><a id="more"></a><p>Following is some notes about the avoiding or solving the deadlock.</p><h2 id="monitor">**Monitor **</h2><p>At the beginning, we can look at a picture:<img src="/img/java-monitor-associate-with-object.jpg" srcset="/img/loading.gif" alt="Monitor"></p><p>而一个锁就像一种任何时候只允许一个线程拥有的特权.<br>一个线程可以允许多次对同一对象上锁.对于每一个对象来说,java虚拟机维护一个计数器,记录对象被加了多少次锁,没被锁的对象的计数器是0,线程每加锁一次,计数器就加1,每释放一次,计数器就减1.当计数器跳到0的  时候,锁就被完全释放了.</p><p>java虚拟机中的一个线程在它到达监视区域开始处的时候请求一个锁.JAVA程序中每一个监视区域都和一个对象引用相关联.<br>The program is like a building. And it stores some data meanwhile. But every time this building can only be occupied by only <span style="color:#f92672"><strong>one</strong> thread.If the thread enter this building, it’s called <strong>enter monitor</strong>;If the thread enter the special room in this building, it’s called <strong>get monitor</strong>;If the thread occupy the room, it’s called <strong>occupy monitor</strong>;If the thread leave the room, it’s called <strong>release monitor</strong>;If the thread leave the building, it’s called <strong>exit monitor</strong>;</span></p><p>Then,a <strong>LOCK</strong> is an authority only for single thread in every time.Single thread is allowed to multiplically add the lock to the same <a href="http://object.As" target="_blank" rel="noopener">object.As</a> for an object, JVM maintain a counter to record how many times the object was added lock.</p><p>In Java, to implement a monitor, we’re supposed to use <strong>Lock</strong> and <strong>Condition</strong> class,</p><h2 id="semaphore"><strong>Semaphore</strong></h2><p>In *** official docs ***:</p><blockquote><blockquote><p>A counting semaphore. Conceptually, a semaphore maintains a set of permits.Each acquire() blocks if necessary until a permit is available, and then takes it.Each release() adds a permit, potentially releasing a blocking acquirer.However, no actual permit objects are used; the Semaphore just keeps a count of the number available and acts accordingly.</p></blockquote></blockquote><p>Semaphores are often used to restrict the number of threads than can access some (physical or logical) resource. For example, here is a class that uses a semaphore to control access to a pool of items:</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Pool</span> </span>&#123;  <span class="hljs-keyword">private</span> <span class="hljs-keyword">static</span> <span class="hljs-keyword">final</span> <span class="hljs-keyword">int</span> MAX_AVAILABLE = <span class="hljs-number">100</span>;  <span class="hljs-keyword">private</span> <span class="hljs-keyword">final</span> Semaphore available = <span class="hljs-keyword">new</span> Semaphore(MAX_AVAILABLE, <span class="hljs-keyword">true</span>);  <span class="hljs-function"><span class="hljs-keyword">public</span> Object <span class="hljs-title">getItem</span><span class="hljs-params">()</span> <span class="hljs-keyword">throws</span> InterruptedException </span>&#123;    available.acquire();    <span class="hljs-keyword">return</span> getNextAvailableItem();  &#125;  <span class="hljs-function"><span class="hljs-keyword">public</span> <span class="hljs-keyword">void</span> <span class="hljs-title">putItem</span><span class="hljs-params">(Object x)</span> </span>&#123;    <span class="hljs-keyword">if</span> (markAsUnused(x))      available.release();  &#125;  <span class="hljs-comment">// Not a particularly efficient data structure; just for demo</span>  <span class="hljs-keyword">protected</span> Object[] items = ... <span class="hljs-comment">//whatever kinds of items being managed</span>  <span class="hljs-keyword">protected</span> <span class="hljs-keyword">boolean</span>[] used = <span class="hljs-keyword">new</span> <span class="hljs-keyword">boolean</span>[MAX_AVAILABLE];  <span class="hljs-function"><span class="hljs-keyword">protected</span> <span class="hljs-keyword">synchronized</span> Object <span class="hljs-title">getNextAvailableItem</span><span class="hljs-params">()</span> </span>&#123;    <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i &lt; MAX_AVAILABLE; ++i) &#123;      <span class="hljs-keyword">if</span> (!used[i]) &#123;         used[i] = <span class="hljs-keyword">true</span>;         <span class="hljs-keyword">return</span> items[i];      &#125;    &#125;    <span class="hljs-keyword">return</span> <span class="hljs-keyword">null</span>; <span class="hljs-comment">// not reached</span>  &#125;  <span class="hljs-function"><span class="hljs-keyword">protected</span> <span class="hljs-keyword">synchronized</span> <span class="hljs-keyword">boolean</span> <span class="hljs-title">markAsUnused</span><span class="hljs-params">(Object item)</span> </span>&#123;    <span class="hljs-keyword">for</span> (<span class="hljs-keyword">int</span> i = <span class="hljs-number">0</span>; i &lt; MAX_AVAILABLE; ++i) &#123;      <span class="hljs-keyword">if</span> (item == items[i]) &#123;         <span class="hljs-keyword">if</span> (used[i]) &#123;           used[i] = <span class="hljs-keyword">false</span>;           <span class="hljs-keyword">return</span> <span class="hljs-keyword">true</span>;         &#125; <span class="hljs-keyword">else</span>           <span class="hljs-keyword">return</span> <span class="hljs-keyword">false</span>;      &#125;    &#125;    <span class="hljs-keyword">return</span> <span class="hljs-keyword">false</span>;  &#125;&#125;</code></pre></div><h2 id="threadlocal"><strong>ThreadLocal</strong></h2><p>该修饰符主要修饰变量，使该变量变为线程局部变量，同一个ThreadLocal所包含的对象，在不同thread中有不同的副本所以可以引出：</p><ol><li>每个Thread有不同副本，所以其他thread不可访问，就不会有线程问题</li><li>因为只在一个线程内使用，其<strong>一般</strong>会被<strong>private static</strong>修饰;static主要是因为其保证了多个实例只会有一个，否则同一个线程可能会访问同一个类的不同实例，即使不错误也会导致浪费（重复创建了一样的对象）//？？？？</li></ol><h3 id="结构">结构</h3><p>其内有一个 ThreadlocalMap的内部类，使用的是线性探测法（够慢，不过也够）它的key为ThreadLocal对象，而且还是弱引用</p><div class="hljs"><pre><code class="hljs java"><span class="hljs-comment">/**     * ThreadLocalMap is a customized hash map suitable only for     * maintaining thread local values. No operations are exported     * outside of the ThreadLocal class. The class is package private to     * allow declaration of fields in class Thread.  To help deal with     * very large and long-lived usages, the hash table entries use     * WeakReferences for keys. However, since reference queues are not     * used, stale entries are guaranteed to be removed only when     * the table starts running out of space.     */</span>    <span class="hljs-keyword">static</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">ThreadLocalMap</span> </span>&#123;        <span class="hljs-comment">/**         * The entries in this hash map extend WeakReference, using         * its main ref field as the key (which is always a         * ThreadLocal object).  Note that null keys (i.e. entry.get()         * == null) mean that the key is no longer referenced, so the         * entry can be expunged from table.  Such entries are referred to         * as "stale entries" in the code that follows.         */</span>        <span class="hljs-keyword">static</span> <span class="hljs-class"><span class="hljs-keyword">class</span> <span class="hljs-title">Entry</span> <span class="hljs-keyword">extends</span> <span class="hljs-title">WeakReference</span>&lt;<span class="hljs-title">ThreadLocal</span>&lt;?&gt;&gt; </span>&#123;            <span class="hljs-comment">/** The value associated with this ThreadLocal. */</span>            Object value;            Entry(ThreadLocal&lt;?&gt; k, Object v) &#123;                <span class="hljs-keyword">super</span>(k);                value = v;            &#125;        &#125;        <span class="hljs-comment">//....</span>    &#125;</code></pre></div><h3 id="注意">注意</h3><p>ThreadLocal是个浅copy，如果变量是一个引用类型，那么就要考虑其内部状态是否会被改变，想要解决只能通过重写initialValue（）方法来自己实现深copy;其思路和锁也不一样，锁是强调如何同步多个线程去正确共享一个变量，而threadlocal是为了解决同一个变量如何不被多个线程共享;</p><h3 id="使用场景：">使用场景：</h3><ol><li>每个线程需要有自己单独的实例</li><li>实例需要在多个方法中共享，但不希望被多线程共享</li></ol>]]></content>
    
    
    
    <tags>
      
      <tag>Java</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Java-Collections</title>
    <link href="/2018/01/07/java/Java-Collection/"/>
    <url>/2018/01/07/java/Java-Collection/</url>
    
    <content type="html"><![CDATA[<p>Following is the summarize of the Java Collections://todo</p><a id="more"></a> <p>#类别List,Set,Map</p><p>##Wildcard</p><p>Lets look at an interesting stuff:<div style="color:#f92672">Collections</div>the <mark>supertype</mark> of all kinds of collections is Collection&lt;?&gt;(? is called “unknown”)</p><div class="hljs"><pre><code class="hljs java">Collection&lt;?&gt; c = <span class="hljs-keyword">new</span> ArrayList&lt;String&gt;();c.add(<span class="hljs-keyword">new</span> Object());<span class="hljs-comment">//Compile time error;</span></code></pre></div><h2 id="map">Map</h2><h3 id="linkedhashmap">LinkedHashMap</h3><p>首先我们来看下LinkedHashMap的结构（这里盗一张别人的图）<img src="/img/linkedHashMap.jpg" srcset="/img/loading.gif" alt="linkedhashmap"></p><p>由此知道 LinkedHashMap是继承于HashMap的，所以hash算法，红黑树这些玩意儿都会有，但这里主要是它特殊的性质：</p><ol><li>其底层是双向链表，<strong>保证了遍历顺序和插入顺序一致的情况</strong></li><li>实现了LRU (即对访问顺序有相关实现)</li></ol><h3 id="hashmap">HashMap</h3><p>HashTable暂且不谈，jdk8已经不推荐使用，但其是线程安全，因为用了synchronized修饰</p>]]></content>
    
    
    
    <tags>
      
      <tag>Java</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>正则表达式</title>
    <link href="/2018/01/05/Regexp/"/>
    <url>/2018/01/05/Regexp/</url>
    
    <content type="html"><![CDATA[<p>曾经认为，这玩意儿要匹配的话扔网上工具转换就行，但现在感觉还是记住一些比较重要的好一点，所以就当练打字吧~</p><p>这里的正则表达式就暂时在javascript上讨论吧。</p><a id="more"></a><p>参考<a href="http://www.runoob.com/jsref/jsref-obj-regexp.html" target="_blank" rel="noopener">网上随便找的中文教程</a></p><p>再附上一个<a href="https://regexr.com/" target="_blank" rel="noopener">贼鸡儿好用的在线匹配</a></p><blockquote><blockquote><blockquote><p>在某些语言里面记得用` ` 字符进行传递</p></blockquote></blockquote></blockquote><p>这就不得不提到<span style="color:#f92672;font-size:2rem">RegExp</span><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp" target="_blank" rel="noopener">MDN手册</a>这个对象</p><h3 id="语法：">语法：</h3><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">var</span> patt = <span class="hljs-keyword">new</span> <span class="hljs-built_in">RegExp</span>(pattern[, flags])<span class="hljs-keyword">var</span> patt = <span class="hljs-regexp">/pattern/</span>flags;</code></pre></div><p><strong>pattern:就是写正则的地方</strong><strong>flags取值(可以是下面的任意组合):</strong></p><div class="hljs"><pre><code class="hljs undefined">g:<span class="hljs-keyword">global</span> <span class="hljs-keyword">match</span> <span class="hljs-comment">//查找所有匹配而不是在找到第一个匹配后停止</span>i:ignore case <span class="hljs-comment">//忽略大小写</span>m:multiline;treat begining <span class="hljs-keyword">and</span> <span class="hljs-keyword">end</span> characters(^ <span class="hljs-keyword">and</span> $)<span class="hljs-keyword">as</span> working over multiple lines <span class="hljs-comment">//多行匹配,用这个属性后$会匹配'\n'或'\r'.要匹配$本身，用\$</span></code></pre></div><h3 id="特殊字符">特殊字符</h3><table><thead><tr><th style="text-align:center">特殊字符</th><th style="text-align:center">简单说明(无特殊说明，匹配本身都是用\转义)</th></tr></thead><tbody><tr><td style="text-align:center">$</td><td style="text-align:center">匹配输入字符串的结尾位置。</td></tr><tr><td style="text-align:center">()</td><td style="text-align:center">标记一个子表达式的开始和结束位置。子表达式可以获取供以后使用。</td></tr><tr><td style="text-align:center">*</td><td style="text-align:center">(限定符)匹配前面的子表达式零次或多次。</td></tr><tr><td style="text-align:center">+</td><td style="text-align:center">(限定符)匹配前面的子表达式一次或多次。</td></tr><tr><td style="text-align:center">.</td><td style="text-align:center">匹配除换行符\n之外的任何单字符;匹配本身用\。</td></tr><tr><td style="text-align:center">?</td><td style="text-align:center">匹配前面的子表达式零次或一次，或知名一个非贪婪限定符。</td></tr><tr><td style="text-align:center">[</td><td style="text-align:center">标记一个中括号表达式的开始。</td></tr><tr><td style="text-align:center">^</td><td style="text-align:center">匹配输入字符串的开始位置，除非在方括号表达式中使用，此时它表示不接受该字符集合。</td></tr><tr><td style="text-align:center">{</td><td style="text-align:center">标记限定符表达式的开始。</td></tr><tr><td style="text-align:center">¦</td><td style="text-align:center">很明显，&quot;或&quot;符号。</td></tr></tbody></table><!--<table><tr><th>特殊字符</th><th>简单说明（没特殊说明的，匹配本身都是直接用\转义）</th></tr><tr><td>$</td><td>匹配输入字符串的结尾位置。</td></tr><tr><td>()</td><td>标记一个子表达式的开始和结束位置。子表达式可以获取供以后使用。</td></tr><tr><td>*</td><td>(限定符)匹配前面的子表达式零次或多次。</td></tr><tr><td>+</td><td>(限定符)匹配前面的子表达式一次或多次。</td></tr><tr><td>.</td><td>匹配除换行符\n之外的任何单字符;匹配本身用\。</td></tr><tr><td>?</td><td>匹配前面的子表达式零次或一次，或知名一个非贪婪限定符。</td></tr><tr><td>[</td><td>标记一个中括号表达式的开始。</td></tr><tr><td>^</td><td>匹配输入字符串的开始位置，除非在方括号表达式中使用，此时它表示不接受该字符集合。</td></tr><tr><td>{</td><td>标记限定符表达式的开始</td></tr><tr><td>|</td><td>很明显，"或"符号</td></tr></table>--><h3 id="限定符">限定符</h3><table><thead><tr><th style="text-align:center">字符</th><th style="text-align:left">简单说明</th></tr></thead><tbody><tr><td style="text-align:center">()</td><td style="text-align:left">标记一个子表达式的开始和结束位置。子表达式可以获取供以后使用。</td></tr><tr><td style="text-align:center">*</td><td style="text-align:left">(限定符)匹配前面的子表达式零次或多次。</td></tr><tr><td style="text-align:center">+</td><td style="text-align:left">(限定符)匹配前面的子表达式一次或多次。</td></tr><tr><td style="text-align:center">?</td><td style="text-align:left">匹配前面的子表达式零次或一次，或指明一个非贪婪限定符。</td></tr></tbody></table><div style="color:#f92672;">这里顺带说一声，如果在markdown中直接用table标签的话要所有html代码压缩才可以，要不会多出很多&lt;br/&gt;</div><h3 id="其他元字符">其他元字符</h3><table><thead><tr><th style="text-align:center">字符</th><th>简单说明</th></tr></thead><tbody><tr><td style="text-align:center">[xyz]</td><td>字符集合。匹配所包含的任意一个字符。例如， ‘[abc]’ 可以匹配 “plain” 中的 ‘a’。</td></tr><tr><td style="text-align:center">[^xyz]</td><td>负值字符集合。匹配未包含的任意字符。例如， ‘[^abc]’ 可以匹配 “plain” 中的’p’、‘l’、‘i’、‘n’。</td></tr><tr><td style="text-align:center">\b</td><td>匹配一个单词边界，也就是指单词和空格间的位置。‘er\b’ 可以匹配&quot;never&quot; 中的 ‘er’，但不能匹配 “verb” 中的 ‘er’。</td></tr><tr><td style="text-align:center">\B</td><td>匹配非单词边界。‘er\B’ 能匹配 “verb” 中的 ‘er’，但不能匹配 “never” 中的 ‘er’。</td></tr><tr><td style="text-align:center">\d</td><td>匹配一个数字字符。等价于 [0-9]。 \D同上。</td></tr><tr><td style="text-align:center">\s</td><td>匹配任何空白字符，包括空格、制表符、换页符等等。等价于 [ \f\n\r\t\v]。\S同上。</td></tr><tr><td style="text-align:center">\w</td><td>匹配包括下划线的任何单词字符。等价于’[A-Za-z0-9_]’。\W同上。</td></tr><tr><td style="text-align:center">x{n}</td><td>n 是一个非负整数。匹配x的 n 次。例如，‘o{2}’ 不能匹配 “Bob” 中的 ‘o’，但是能匹配 “food” 中的两个 o。</td></tr><tr><td style="text-align:center">x{n,}</td><td>n 是一个非负整数。至少匹配x的n 次。例如，‘o{2,}’ 不能匹配 “Bob” 中的 ‘o’，但能匹配 “foooood” 中的所有 o。‘o{1,}’ 等价于 ‘o+’。‘o{0,}’ 则等价于 ‘o*’。</td></tr></tbody></table><!--<table><thead><tr><th>字符</th><th>简单说明</th></tr></thead><tbody><tr><td>()</td><td>标记一个子表达式的开始和结束位置。子表达式可以获取供以后使用。</td></tr><tr><td>*</td><td>(限定符)匹配前面的子表达式零次或多次。</td></tr><tr><td>+</td><td>(限定符)匹配前面的子表达式一次或多次。</td></tr><tr><td>?</td><td>匹配前面的子表达式零次或一次，或知名一个非贪婪限定符。</td></tr></tbody></table>--><h3 id="使用正则的方法">使用正则的方法:</h3><h4 id="1-exec">1. exec</h4><p>搜索字符串制定的值，返回找到的值，如果没有匹配，返回Null</p><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">var</span> myRe = <span class="hljs-regexp">/d(b+)d/g</span>;<span class="hljs-comment">//使用g全局匹配,符合d开始,d结尾</span><span class="hljs-keyword">var</span> myArray = myRe.exec(<span class="hljs-string">"cdbbdbsbz"</span>);<span class="hljs-comment">/*["dbbd", "bb"]*/</span><span class="hljs-keyword">var</span> myArray2 = myRe.exec(<span class="hljs-string">"a"</span>);<span class="hljs-comment">/*null*/</span></code></pre></div><h4 id="2-test">2. test()</h4><p>搜索字符串指定的值，返回真假</p><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">var</span> myRe = <span class="hljs-keyword">new</span> <span class="hljs-built_in">RegExp</span>(<span class="hljs-string">"e"</span>);<span class="hljs-built_in">console</span>.log(myRe.test(<span class="hljs-string">"The best"</span>));<span class="hljs-comment">/*true*/</span><span class="hljs-built_in">console</span>.log(myRe.test(<span class="hljs-string">"a"</span>));<span class="hljs-comment">/*false*/</span></code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>Regular Expression</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>从DocumentFragment看到GUI渲染</title>
    <link href="/2017/02/05/js/gui_engine/"/>
    <url>/2017/02/05/js/gui_engine/</url>
    
    <content type="html"><![CDATA[<p>这里主要说说<span style="color:#f92672">重绘(repaint)</span>和<span style="color:#f92672">回流(reflow)</span>的问题</p><a id="more"></a><h4 id="犹记得某个项目要求使用经典的下拉刷新功能-其中就有ajax在后面添加某些信息；">犹记得某个项目要求使用经典的下拉刷新功能，其中就有ajax在后面添加某些信息；</h4><h4 id="其中要使用documentfragmentmdn手册">其中要使用DocumentFragment<a href="https://developer.mozilla.org/en-US/docs/Web/API/DocumentFragment" target="_blank" rel="noopener">MDN手册</a></h4><p>先看下以下代码</p><div class="hljs"><pre><code class="language-javascript">window.onload = function(){/*一个大致的ajax动态获取信息并添加到DOM中的代码*//*ajax*/var xhr;if(window.XMLHttpRequest){xhr = new window.XMLHttpRequest();}else{xhr = new ActiveXObject('Microsoft.XMLHttp');/*IE6没什么X用*/}xhr.open('get',this.getAtrribute('data-url'),true);xhr.send(null);xhr.onreadystatechange = function(){if(xhr.readyState == 4 &amp;&amp; xhr.status == 200){/*核心*/var frag = document.createDocumentFragment();/*创建一个文档节点，其实和用一串字符串存储在一个变量中一样*/for(var x = 0; x&lt;10;x++){var li = document.createElement(&quot;li&quot;);li.innerHTML = &quot;List item&quot;+x;frag.appendChild(li);}Node.appendChild(frag);frag = null;/*清空Documentfragment缓存*/}}}</code></pre></div><p>上面这样写的好处就是：创建一个DocumentFragment(),这样在appendChild()的时候就只会触发一次reflow</p><p>因为暂时没有现成网站，暂时先当作一个笔记吧：</p><h3 id="回流-reflow">回流(reflow)</h3><p>Render Tree中一部分或全部因为元素尺寸、布局、隐藏等改变而需要重新构建，每个页面在一开始加载的时候都会回流，以下一些建议：1. 如要修改，尽量修改DOM中下层的class，避免上层父级class改变2. 避免使用table（一开始接触就听说这玩意有问题）3. 避免在html中设置style，尽量在外部css设置style，跟documentFragment()是一个道理</p><h3 id="重绘-repaint">重绘(repaint)</h3><p>Render Tree中一些只影响颜色、风格，最重要跟以上reflow不同的关键就是不影响DOM结构，比如一些absolute或者fixed定位元素就基本可以随便移动（假设他们的父元素不变啦）</p>]]></content>
    
    
    
    <tags>
      
      <tag>css</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>Markdown学习记录</title>
    <link href="/2017/02/01/markdown/"/>
    <url>/2017/02/01/markdown/</url>
    
    <content type="html"><![CDATA[<p>很久不用，今天就从这里开始重新学习吧！</p><a id="more"></a><div style="color:#f92672">注意：Markdown是可以内嵌html的，所以忘了或有特殊样式就直接套html吧,比如这个</div><div style="color:#4d8fc0">注意：内嵌了html标签的，要空一行才可以接markdown,否则会出问题</div><h2 id="标题"><strong>标题</strong></h2><h3 id="有两种：">有两种：</h3><h3 id="①setext类：">①Setext类：</h3><h4 id="写法">写法:</h4><div class="hljs"><pre><code>This is an H1=============This is an H2-------------</code></pre></div><h4 id="效果">效果:</h4><h1>This is an H1</h1><h2 id="this-is-an-h2">This is an H2</h2><h3 id="②atx类：">②ATX类：</h3><h4 id="写法：行首插入1到6个-注意-后面要加一个空格">写法：行首插入1到6个#,注意#后面要加<strong>一个空格</strong>!</h4><div class="hljs"><pre><code># 这个是一级标题，前面一个`#`## 这个是二级标题，前面两个`##`</code></pre></div><h4 id="效果-v2">效果:</h4><h1>这个是一级标题，前面一个<code>#</code></h1><h2 id="这个是二级标题-前面两个">这个是二级标题，前面两个<code>##</code></h2><h2 id="列表"><strong>列表</strong></h2><h3 id="无序列表">无序列表</h3><h4 id="写法：前面用星号或加号或减号来标记">写法：前面用星号或加号或减号来标记</h4><h4 id="效果：">效果：</h4><ul><li>Red</li><li>Green</li><li>Blue</li></ul><h3 id="有序列表">有序列表</h3><h4 id="写法-v2">写法:</h4><div class="hljs"><pre><code>1. Red2. Green3. Blue</code></pre></div><h4 id="效果-v3">效果:</h4><ol><li>Red</li><li>Green</li><li>Blue</li></ol><h2 id="图片"><strong>图片</strong></h2><h4 id="写法-v3">写法:</h4><div class="hljs"><pre><code class="hljs undefined">![<span class="hljs-string">Alt text</span>](<span class="hljs-link">/path/to/img.jpg</span>)![<span class="hljs-string">Alt text</span>](<span class="hljs-link">/path/to/img.jpg "Optional title"</span>)</code></pre></div><h4 id="即：">即：</h4><ul><li>一个惊叹号 !</li><li>接着一个方括号，里面放上图片的替代文字</li><li>接着一个普通括号，里面放上图片的网址，最后还可以用引号包住并加上 选择性的 ‘title’ 文字。</li></ul><p>这里的图片是基于这个hexo固定文件生成结构的,位于<span style="color:#f92672">public/Year/Month/Day/article_title/</span>中，所以，我只能用相对路径找到public内的css/images/图片<span style="color:#f92672">…/…/…/…/css/images/图片</span></p><h4 id="效果-v4">效果:</h4><p><img src="../../../../css/images/mp1.jpg" srcset="/img/loading.gif" alt="显示在下面的文字"></p><h2 id="代码"><strong>代码</strong></h2><h4 id="写法-v4">写法:</h4><p>1、可以用三个```包裹一段代码,第一个```后面加个<span style="color:#f92672">空格</span>，再加上代码类型：比如javascript、python等,可以高亮</p><div class="hljs"><pre><code class="hljs javascript">&lt;div style=<span class="hljs-string">"color:#eee;"</span>&gt;foo&lt;<span class="hljs-regexp">/div&gt;</span></code></pre></div><div class="hljs"><pre><code class="hljs python"><span class="hljs-function"><span class="hljs-keyword">def</span> <span class="hljs-title">fun</span><span class="hljs-params">()</span>:</span><span class="hljs-keyword">return</span> <span class="hljs-number">0</span></code></pre></div><p>2、可以用一个`包裹一段代码<code>&lt;div style=&quot;color:#eee;&quot;&gt;foo&lt;/div&gt;</code></p><div style="text-decoration:line-through"><h4> 3、可以用4空格缩进</h4></div><div style="color:#f92672">这里很奇怪,不知道为啥↑,初步估计是html标签是不能直接tab的,如下<span style="color: #4d8fc0">链接</span>写法就可以</div><h2 id="引用"><strong>引用</strong></h2><h4 id="写法-v5">写法:</h4><p>加上 &gt;&gt;</p><blockquote><blockquote><p>This is a quote</p></blockquote></blockquote><h2 id="链接"><strong>链接</strong></h2><h4 id="写法-v6">写法:</h4><p>1、文字连接:<a href="http://xn--ses510ahtve0t" target="_blank" rel="noopener">链接文字</a>2、网址链接:<a href="http://xn--ses510ahtve0t" target="_blank" rel="noopener">http://链接网址</a>3、链接到本页锚处:<a href="#anchor_lowercase_name">link</a></p><p><span style="color:#f92672">PS:一般只对header有用,还会遵循以下原则（不清楚为什么部分不生效）:</span></p><ol><li>标点符号会被丢弃</li><li>首空格会被丢弃</li><li>大写字母会被转换为小写字母（并不会。。。）</li><li>字母之间的空格会被转换为 -  (实际上.也会被转化)</li></ol><div class="hljs"><pre><code class="hljs undefined">[<span class="hljs-string">link1</span>](<span class="hljs-link">#hello</span>)[<span class="hljs-string">link2</span>](<span class="hljs-link">#new-hello</span>)<span class="hljs-section">#Hello or &lt;a name="hello"&gt;&lt;/a&gt;</span><span class="hljs-section">#New Hello</span></code></pre></div><h4 id="效果-v5">效果:</h4><p><a href="https://www.baidu.com" target="_blank" rel="noopener">百度</a><a href="https://www.baidu.com" target="_blank" rel="noopener">https://www.baidu.com</a></p><p><a href="#1-12-Test">GoToTest</a>............test</p><h3 id="1-12-test">1.12 Test</h3><div style="color: #f92672">暂时记录到这里吧，差不多够用了，这些都是从网上找的教程，在此感谢那些博客主Orz</div>]]></content>
    
    
    
    <tags>
      
      <tag>Markdown</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>My Understanding towards Promise/A+</title>
    <link href="/2017/01/07/js/js-Promise/"/>
    <url>/2017/01/07/js/js-Promise/</url>
    
    <content type="html"><![CDATA[<p>Personal comprehension about ES6 promise,including simulated promise implementation</p><a id="more"></a><h2 id="for-a-stupid-instance">For a stupid Instance</h2><p>There’s a scene that you’re going to order a cup of coffee.The pretty waitress may say ‘Please come to get your coffee later’.So this is a <div style="color:#f92672">Promise</div>.She promised that you can get a coffee soon.Besides, your waiting process is called <div style="color:#4d8fc0">pending</div>.After you sucessfully get your coffee from the waitress,this’s called <div style="color:#4d8fc0">resolved(fulfilled)</div>.But it’s possible that her coffee beans are out of <a href="http://stock.As" target="_blank" rel="noopener">stock.As</a> a result you can’t get your coffee,so this is <div style="color:#4d8fc0">rejected</div>.Furthermore,after you got the promise,the following operation is called  <div style="color:#f92672">then</div>;</p><h2 id="definition">Definition</h2><p>The Promise object represents the eventual completion (or failure) of an asynchronous operation, and its resulting value.<a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise" target="_blank" rel="noopener">MDN</a></p><p>Essentially, a promise is a returned object to which you attach callbacks, instead of passing callbacks into a function.(That’s why it can avoid callback hell,once you call then() method,it will implicitly <div style="color:#f92672">create a new promise object and return it;</div> ) <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Using_promises" target="_blank" rel="noopener">MDN</a></p><h3 id="more-advantages">More advantages:</h3><p>Unlike old-style passed-in callbacks, a promise comes with some guarantees:</p><div class="hljs"><pre><code>1. Callbacks will never be called before the completion of the current run of the JavaScript event loop.2. Callbacks added with .then even after the success or failure of the asynchronous operation, will be called, as above.3. Multiple callbacks may be added by calling .then several times, to be executed independently in insertion order.</code></pre></div><p>But the most immediate benefit of promises is chaining.</p><p>For more detail:<a href="https://promisesaplus.com/" target="_blank" rel="noopener">Promise A+</a></p><p>Following my code about a simple promise implementation:</p><div class="hljs"><pre><code class="language-javascript"></code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>Javascript</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>From SetTimeout to Js Scope</title>
    <link href="/2017/01/01/js/js-Scope/"/>
    <url>/2017/01/01/js/js-Scope/</url>
    
    <content type="html"><![CDATA[<p>There’s a chinese word ‘温故而知新’.</p><a id="more"></a><p>Today I saw a code:</p><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">for</span> (<span class="hljs-keyword">var</span> i=<span class="hljs-number">1</span>;i&lt;=<span class="hljs-number">5</span>;i++)&#123;    setTimeout(<span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">timer</span>(<span class="hljs-params"></span>)</span>&#123;        <span class="hljs-built_in">console</span>.log(i);    &#125;,i*<span class="hljs-number">1000</span>);&#125;</code></pre></div><p>And the result is:</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-number">34</span><span class="hljs-comment">//a random number,it's different any time after u reopen your browser</span><span class="hljs-number">6</span><span class="hljs-comment">//run 5 times</span></code></pre></div><p>Directly go to <a href="#The-1stQ-solution">Solution</a> !!!</p><p>To explain above,We begin with some of the characteristics of <a href="https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout" target="_blank" rel="noopener">setTimeout</a>##Syntax</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-keyword">var</span> timeoutID = scope.setTimeout(<span class="hljs-function"><span class="hljs-keyword">function</span>[,<span class="hljs-title">delay</span>,<span class="hljs-title">param1</span>,<span class="hljs-title">param2</span>]);</span></code></pre></div><h2 id="return-value">Return value</h2><p>we should know about the setTimeout() return a <strong>timeoutID</strong>.</p><ol><li>this is an positive integer value</li><li>this value identifies the timer created by setTimeout()(this is the reason why it’s different all the time)</li><li>setTimeout() and setInterval() share the same pool of IDs,which means setInterval() may effect the timer if u use them at the same object(window or a worker);</li></ol><h2 id="this-problem">“This” problem</h2><p>Let’s first see a simpler example:</p><div class="hljs"><pre><code class="hljs javascript">(<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) </span>&#123;    <span class="hljs-keyword">this</span>.fuck = <span class="hljs-literal">false</span>;    Console.log(<span class="hljs-string">"OMG!"</span>);    <span class="hljs-keyword">var</span> timer = setTimeout((<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) </span>&#123;        <span class="hljs-keyword">this</span>.fuck = <span class="hljs-literal">true</span>;        Console.log(<span class="hljs-string">"timeout Fuck!"</span>);    &#125;),<span class="hljs-number">3000</span>);&#125;)</code></pre></div><p>The problem of above example is :The <span style="color:#f92672">this</span> always refers to the<span style="color:#f92672">this</span> of the current scope,which changes any time I wrap something in <span style="color:#f92672">function(){}</span></p><p>So there is###one solution:</p><div class="hljs"><pre><code class="hljs javascript">(<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) </span>&#123;    <span class="hljs-keyword">var</span> symbol = <span class="hljs-keyword">this</span>;<span class="hljs-comment">//now refers to the Window object </span>    symbol.fuck = <span class="hljs-literal">false</span>;    Console.log(<span class="hljs-string">"OMG!"</span>);    <span class="hljs-keyword">var</span> timer = setTimeout((<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) </span>&#123;        symbol.fuck = <span class="hljs-literal">true</span>;<span class="hljs-comment">//still refers to the Window Object</span>        Console.log(<span class="hljs-string">"timeout Fuck!"</span>);    &#125;),<span class="hljs-number">3000</span>);&#125;)</code></pre></div><p>###SecondES5 Function.prototype.bind()</p><div class="hljs"><pre><code class="hljs javascript">(<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) </span>&#123;    <span class="hljs-keyword">this</span>.fuck = <span class="hljs-literal">false</span>;    Console.log(<span class="hljs-string">"OMG!"</span>);    <span class="hljs-keyword">var</span> timer = setTimeout((<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) </span>&#123;        <span class="hljs-keyword">this</span>.fuck = <span class="hljs-literal">true</span>;        Console.log(<span class="hljs-string">"timeout Fuck!"</span>);    &#125;).bind(<span class="hljs-keyword">this</span>),<span class="hljs-number">3000</span>);<span class="hljs-comment">//bind the scope</span>&#125;)</code></pre></div><p>###ThirdES6 [Arrow Function()]</p><div class="hljs"><pre><code class="hljs javascript">(<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>) </span>&#123;    <span class="hljs-keyword">this</span>.fuck = <span class="hljs-literal">false</span>;    Console.log(<span class="hljs-string">"OMG!"</span>);    <span class="hljs-keyword">var</span> timer = setTimeout(<span class="hljs-function"><span class="hljs-params">()</span>=&gt;</span> &#123;        <span class="hljs-keyword">this</span>.fuck = <span class="hljs-literal">true</span>;        Console.log(<span class="hljs-string">"timeout Fuck!"</span>);    &#125;,<span class="hljs-number">3000</span>);<span class="hljs-comment">//no binding in Arrow Functions!!!</span>&#125;)</code></pre></div><p>I think now it’s clear that why the first code resulted in that,The solution:</p><p>###The 1stQ solution</p><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">for</span> (<span class="hljs-keyword">var</span> i=<span class="hljs-number">1</span>;i&lt;=<span class="hljs-number">5</span>;i++)&#123;        (<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params">i</span>)</span>&#123;<span class="hljs-comment">//1.set the self-invoking anonymous function to set the scope</span>        setTimeout(<span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">timer</span>(<span class="hljs-params"></span>)</span>&#123;            <span class="hljs-built_in">console</span>.log(i);        &#125;,i*<span class="hljs-number">1000</span>);    &#125;)(i)<span class="hljs-comment">//2.Then call the value of i </span>&#125;</code></pre></div><p>Also u can try this,result is the same:</p><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">for</span> (<span class="hljs-keyword">var</span> i=<span class="hljs-number">1</span>;i&lt;=<span class="hljs-number">5</span>;i++)&#123;    setTimeout((<span class="hljs-function"><span class="hljs-keyword">function</span> <span class="hljs-title">timer</span>(<span class="hljs-params">i</span>)</span>&#123;        <span class="hljs-keyword">return</span> <span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>)</span>&#123;<span class="hljs-comment">//set return value as an anonymous function</span>            <span class="hljs-built_in">console</span>.log(i);        &#125;    &#125;)(i),i*<span class="hljs-number">1000</span>);&#125;</code></pre></div>]]></content>
    
    
    
    <tags>
      
      <tag>javascript</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>关于javascript中时间的一系列函数</title>
    <link href="/2017/01/01/js/js-time/"/>
    <url>/2017/01/01/js/js-time/</url>
    
    <content type="html"><![CDATA[<p>很久以前就很混乱对这个，最近实习的时候突然要用到，现在来做个总结</p><a id="more"></a><p>总结之前先说明几个名词:</p><h4 id="1-span-style-color-f92672-时间戳-span">1、<span style="color:#f92672">时间戳</span></h4><p>指的是<div style="color:#f92672">格林威治时间1970年01月01日00时00分00秒(北京时间1970年01月01日08时00分00秒)起至现在的总秒数。</div>而这个东西的用处挺大的，比如：在书面合同中，文件签署的日期和签名一样均是十分重要的防止文件被伪造和篡改的关键性内容。数字时间戳服务（DTS：digital time stamp service）是网上电子商务安全服务项目之一，能提供电子文件的日期和时间信息的安全保护等等。</p><h4 id="2-span-style-color-f92672-utc-span">2、<span style="color:#f92672">UTC</span></h4><p>协调世界时（英：Coordinated Universal Time ，法：Temps Universel Coordonné），又称世界统一时间，世界标准时间，国际协调时间。英文（CUT）和法文（TUC）的缩写不同，作为妥协，简称UTC。中国大陆、中国香港、中国澳门、中国台湾、蒙古国、新加坡、马来西亚、菲律宾、西澳大利亚州的时间与UTC的时差均为+8，也就是UTC+8。</p><h4 id="3-span-style-color-f92672-gmt-span">3、<span style="color:#f92672">GMT</span></h4><p>格林尼治标准时间（Greenwich Mean Time，GMT）是指位于伦敦郊区的皇家格林尼治天文台的标准时间，因为本初子午线被定义在通过那里的经线。</p><p>说了那么多，现在就先来看看javascript中<span style="color:#f92672">Date</span><a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date" target="_blank" rel="noopener">MDN手册</a>这个对象吧。</p><h4 id="创建对象"><strong>创建对象</strong></h4><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">var</span> myDate = <span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>();<span class="hljs-comment">//返回当天的日期和时间</span></code></pre></div><h5 id="语法">语法</h5><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>(year, month[, day[, hour[, minutes[, seconds[, milliseconds]]]]]);<span class="hljs-keyword">var</span> myDate = <span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>(<span class="hljs-number">2016</span>,<span class="hljs-number">2</span>,<span class="hljs-number">1</span>,<span class="hljs-number">1</span>,<span class="hljs-number">10</span>)<span class="hljs-comment">//for example</span></code></pre></div><h5 id="结果：">结果：</h5><div class="hljs"><pre><code>Tue Mar 01 2016 01:10:00 GMT+0800 (中国标准时间)</code></pre></div><p><span style="color:#f92672">你可能已经注意到<strong>月份</strong>中是要加1，才是最终的时间,因为原本的Date的月份是从0开始算的</span></p><p>有趣的是：溢出了也可以用，比如：</p><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">var</span> myDate = <span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>(<span class="hljs-number">2016</span>,<span class="hljs-number">2</span>,<span class="hljs-number">1</span>,<span class="hljs-number">0</span>,<span class="hljs-number">70</span>)<span class="hljs-comment">//Second overflow</span><span class="hljs-keyword">var</span> myDate2 = <span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>(<span class="hljs-number">2015</span>,<span class="hljs-number">14</span>,<span class="hljs-number">1</span>,<span class="hljs-number">1</span>,<span class="hljs-number">10</span>)<span class="hljs-comment">//Month overflow</span></code></pre></div><p>结果和上面的是一模一样</p><h4 id="常用方法">常用方法:</h4><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">var</span> Date1 = <span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>();Date1.getDate();<span class="hljs-comment">//返回一个月中的某一天(1~31)</span>Date1.getDay();<span class="hljs-comment">//返回一周中的某一天(0~6)</span>Date1.getMonth();<span class="hljs-comment">//返回月份(0~11)</span>Date1.getFullYear();<span class="hljs-comment">//四位数返回年份</span><span class="hljs-comment">/*还可以根据世界时返回时间*/</span>Date1.getUTCDate();<span class="hljs-comment">//同上，多余就不写了</span></code></pre></div><h4 id="重要方法">重要方法:</h4><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">var</span> Date2 = <span class="hljs-keyword">new</span> <span class="hljs-built_in">Date</span>();Date2.parse();<span class="hljs-comment">//这个主要用来计算时间戳,得出单位是ms</span><span class="hljs-keyword">var</span> d = <span class="hljs-built_in">Date</span>.UTC(<span class="hljs-number">2005</span>,<span class="hljs-number">7</span>,<span class="hljs-number">8</span>);<span class="hljs-comment">//这个Date.UTC()是静态方法，不可以通过Date对象调用</span><span class="hljs-comment">//只能用构造函数Date()调用</span><span class="hljs-comment">//也是计算时间戳，得出单位也是ms</span>Date2.toString();<span class="hljs-comment">//挺明显的，转换成string</span>Date2.toLocaleString();<span class="hljs-comment">//根据本地时间，转换成字符串</span><span class="hljs-comment">//举例:</span><span class="hljs-comment">//var born = new Date("July 21, 1983 01:15:00");</span><span class="hljs-comment">//结果:</span><span class="hljs-comment">//1983/7/21 上午1:15:00</span></code></pre></div><h5 id="div-style-color-f92672-其实实际工作运用中就很少用到原生的方法-一般直接套用别人的format函数-而这些format函数中用到一些-strong-正则匹配-strong-这个下次再说-div"><div style="color:#f92672">  其实实际工作运用中就很少用到原生的方法，一般直接套用别人的format函数，而这些format函数中用到一些<strong>正则匹配</strong>，这个下次再说…</div></h5>]]></content>
    
    
    
    <tags>
      
      <tag>Javascript</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>css-clip问题</title>
    <link href="/2016/01/07/js/css-clip/"/>
    <url>/2016/01/07/js/css-clip/</url>
    
    <content type="html"><![CDATA[<p>今天在网上看见一个很好玩的属性<span style="color:#f92672;font-weight: bold;font-size: 2rem;">clip</span> <a href="https://developer.mozilla.org/en-US/docs/Web/CSS/clip" target="_blank" rel="noopener">MDN手册</a></p><a id="more"></a><p>主要是看见这个东西可以用在图片切换上面，实现一个简单的裁剪切换效果</p><h3 id="summary">Summary:</h3><blockquote><p>The clip CSS property defines what portion of an element is visible. The clip property applies only to absolutely positioned elements, that is elements with position:absolute or position:fixed.</p></blockquote><h5 id="意思就是这玩意儿只能用在-span-style-color-f92672-绝对定位-span-上面">意思就是这玩意儿只能用在<span style="color:#f92672">绝对定位</span>上面.</h5><h4 id="语法">语法:</h4><div class="hljs"><pre><code class="hljs undefined">clip:rect(<span class="hljs-built_in">top</span> <span class="hljs-built_in">right</span> <span class="hljs-built_in">bottom</span> <span class="hljs-built_in">left</span>)</code></pre></div><p>顺时针，是不是和<strong>margin</strong>、<strong>padding</strong>的一样呢？</p><div style="color:#f92672">注意：要满足 top < bottom 和left < right </div><div style="color:#4d8fc0">注意：其中bottom和top一样，是以上边缘为开始计算距离;同样，right是和left一样，都是从左边缘算起</div><p>同时，取值<strong>auto</strong>或不满足上面条件的时候，rect就不显示。</p><p>从以上特性看来，好像是可以实现css sprite的效果，而且不用background-position，兼容性得到大幅度提升!</p><p>然而我并不想做那个，我自己做了一个图片切换的demo:<a href="https://codepen.io/doujohner/pen/QNrGZw" target="_blank" rel="noopener">点我</a></p><h5 id="原理很简单-就是从横截面减少right值">原理很简单，就是从横截面减少right值。</h5>]]></content>
    
    
    
    <tags>
      
      <tag>css</tag>
      
    </tags>
    
  </entry>
  
  
  
  <entry>
    <title>从setTimeout()看到js异步机制</title>
    <link href="/2016/01/05/js/js-async/"/>
    <url>/2016/01/05/js/js-async/</url>
    
    <content type="html"><![CDATA[<h3 id="格式">格式</h3><div class="hljs"><pre><code>var timeoutID = window.setTimeout(func, [delay, param1, param2, ...])</code></pre></div><p>指的是若干毫秒后执行func，大概是一个闹钟的功能; <a href="https://developer.mozilla.org/en-US/docs/Web/API/WindowTimers/setTimeout" target="_blank" rel="noopener">MDN手册</a></p><a id="more"></a><h4 id="值得注意的问题">值得注意的问题</h4><p>以前想过setTimeout(func,0)指的是不是立即执行呢？下面看个例子:</p><div class="hljs"><pre><code class="hljs javascript">alert(<span class="hljs-number">1</span>);setTimeout(<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>)</span>&#123;alert(<span class="hljs-number">2</span>);&#125;,<span class="hljs-number">0</span>);alert(<span class="hljs-number">3</span>);</code></pre></div><p>结果却是</p><div class="hljs"><pre><code class="hljs undefined"><span class="hljs-number">1</span><span class="hljs-number">3</span><span class="hljs-number">2</span></code></pre></div><p>是不是感到很奇怪，这个意思就是setTimeout(func,0)并不是立即执行;</p><p>在网上查了资料，发现这个要从<span style="color:#f92672">js引擎</span>说起</p><h4 id="js引擎的单线程性">js引擎的单线程性</h4><p>首先区分两个概念:<span style="color:#f92672">js引擎</span>和<span style="color:#f92672">浏览器内核</span></p><p>js引擎是单线程的这个没错，但是，就因为js引擎是单线程的，它没法再为其他功能服务；同时，<span style="color:#f92672">浏览器是多线程的</span>,这就说的通了：比如浏览器事件触发、定时器计时、网络请求、渲染等都是由浏览器内核分配的其他线程完成的，js引擎只占其中一个线程。</p><p>再举个经典例子：</p><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-keyword">var</span> end = ture;<span class="hljs-built_in">window</span>.setTimeout(<span class="hljs-function"><span class="hljs-keyword">function</span>(<span class="hljs-params"></span>)</span>&#123;end = <span class="hljs-literal">false</span>;&#125;,<span class="hljs-number">1000</span>);<span class="hljs-keyword">while</span>(end);alert(<span class="hljs-string">'end'</span>);</code></pre></div><p>上面的例子结果就是 while(end)处死循环，永远到不了alert(‘end’)，意思就是setTimeout里边的函数没有执行;</p><p>下面就引入这个事件队列的问题，用一张图表示：<img src="../../../../css/images/jsengine.jpg" srcset="/img/loading.gif" alt="js事件队列"></p><p>js是基于事件驱动的语言,所以遵循一个叫事件队列的机制。从图中可看出浏览器的各种各样的线程，比如事件触发器、网络请求、定时器等，js引擎处理到与其他线程相关的代码，会把他们分发到其他线程，他们处理完后需要js引擎计算时就是在事件队列添加一个任务。然而在这个过程中，js并不会阻塞代码等待其他线程执行完毕，而是直接跳过这个分发给别的进程的代码，执行下一串代码。</p><p>很明显上面while(end)的死循环问题就得到解决，执行到setTimeout的时候，把这个东西（定时器）扔到了事件队列后边，然后执行while(end)但这时end是true，所以造成死循环，与第一个setTimeout(func,0)的例子一样得到解释。</p><div style="color:#f92672">PS:实现异步加载</div>1.现代浏览器prefetch（预加载）优化，即浏览器会另开新线程，提前下载js、css文件，prefetch不会改变dom结构.<div class="hljs"><pre><code>&lt;link rel=&quot;prefetch&quot; href=&quot;http://&quot;&gt; &lt;!--利用空余时间加载剩下的网页--&gt;</code></pre></div><p>2.HTML5的defer async标签实现异步加载</p><div class="hljs"><pre><code class="hljs javascript">&lt;script defer=<span class="hljs-string">"true"</span> type=<span class="hljs-string">"text/javascript"</span>&gt;<span class="xml"><span class="hljs-tag">&lt;/<span class="hljs-name">script</span>&gt;</span></span><span class="xml"><span class="hljs-comment">&lt;!--不等js加载完成就加载后边的图片--&gt;</span><span class="hljs-tag">&lt;<span class="hljs-name">script</span> <span class="hljs-attr">async</span>=<span class="hljs-string">"true"</span> <span class="hljs-attr">type</span>=<span class="hljs-string">"text/javascript"</span>&gt;</span><span class="hljs-tag">&lt;/<span class="hljs-name">script</span>&gt;</span></span></code></pre></div><p>3.动态加载js</p><div class="hljs"><pre><code class="hljs javascript"><span class="hljs-built_in">document</span>.write(<span class="hljs-string">"&lt;script src="</span>a.js<span class="hljs-string">"&gt;&lt;/script&gt;"</span>);<span class="hljs-comment">/*还有修改src路径、创建script标签再将其放进DOM中，这些都是对DOM操作*/</span></code></pre></div><h4 id="settimeout-func-0-特殊功效">setTimeout(func,0)特殊功效</h4><p><span style="color:#f92672;font-size:1.8rem">换个角度想，是不是可以用setTimeout(func,0)强行将func放到队列后边延迟执行呢？！这也算是个小技巧吧？</span></p><p>而这个技巧在同样情况下（打开一个标签，就测试环境差不多）不同浏览器上出现的时间会不同，我个人认为这也可以看出浏览器的性能;</p>]]></content>
    
    
    
    <tags>
      
      <tag>javascript</tag>
      
    </tags>
    
  </entry>
  
  
  
  
</search>