CPU Profiling with perf
- Add the following settings to Cargo.toml
1 | [profile.release] |
Also run the code in release mode.
- Install perf + FlameGraph
1 | sudo pacman -Syu perf |
1 | [profile.release] |
Also run the code in release mode.
1 | sudo pacman -Syu perf |
1 | [profile.release] |
并且以 release 模式运行代码
1 | sudo pacman -Syu perf |
Author: suikammd ihciah
An HTTP Server contains many internal components: protocol implementation (h1, h2, compression, and so on), connection state management (keepalive), request dispatching, middleware, business logic, and more. Users could implement all of these themselves, but everything other than the business logic is a fairly general capability. Decoupling these general capabilities from the user’s business logic gives us an HTTP framework.
Because hyper already provides a fairly complete HTTP protocol implementation in the Rust ecosystem, building an HTTP framework on hyper only requires providing capabilities such as routing, shared state, and middleware.
From the perspective of HTTP framework design, this article uses the new version of Axum as an example to analyze how an HTTP framework in Rust can provide sound abstractions and type constraints. Rust’s powerful type system allows us to write efficient and correct code.
Author: suikammd ihciah
一个 HTTP Server 内部包括很多部分:协议实现(h1、h2、压缩等)、连接状态管理(keepalive)、请求分发、中间件、业务逻辑等。用户可以全部自行实现,但其实除了业务逻辑之外的部分都是较为通用的能力。我们将这些通用能力与用户的业务逻辑解耦,即 HTTP 框架。
由于 Rust 生态中 hyper 已经提供了较为完备的 HTTP 协议实现,基于 hyper 实现 HTTP 框架就只需要提供包括路由、共享状态、中间件等能力。
本文从 HTTP 框架的设计角度,以新版本 Axum 作为例子,分析 Rust 下 HTTP 框架如何提供合理抽象与类型约束。基于 Rust 强大的类型系统,我们可以写出高效且正确的代码。
Read moreThe core data structure for concurrent programming in Golang is the channel. Its central idea is to share memory by communicating rather than communicate by sharing memory. Channels are indeed useful and avoid the need to write various complex communication primitives directly. Rust naturally has corresponding implementations as well, and Tokio’s channel implementation is extremely clever and well worth studying. This article briefly introduces the basic concepts of channels, then implements a minimal mpsc unbuffered channel, and finally examines Tokio’s channel implementation.
This section implements a minimal channel. This channel has an unlimited buffer, allows multiple writers, and never blocks writes to the channel. However, if the channel is empty, consumers attempting to consume data from it will be blocked. In other words, this is an mpsc unbuffered channel. This section refers to Chen Tian: Rust Programming First Course, Lesson 35. Chen Tian’s lesson is highly accessible and uses TDD to complete the entire channel design step by step, starting with interface test cases. I will take the same approach here, using TDD and aligning with the standard library’s mspc interface.
Golang 并发编程的核心数据结构就是 channel 了,核心观点是通过通信来共享内存,而不是通过共享内存来通信。channel 确实好用,避免了直接写各种复杂的通信原语,在 rust 中当然也有相应的实现,tokio channel 的实现非常巧妙,很值得学习。本文目的是简单介绍下 channel 的基本概念,然后实现一个最简版 mpsc unbuffered channel ,最后再介绍下 tokio 里 channel 的实现
这一部分实现一个最简版 channel ,这里的 channel 是一个有着无限 buffer 的 channel ,允许多个写入者,并且写入 channel 的操作永远不会阻塞,但是如果 channel 是空的,从 channel 消费数据的消费者会被阻塞,这里其实就是实现一个 mpsc unbuffered channel。本部分参考了 陈天 · Rust编程第一课 35 讲,陈天这一课讲得非常通俗易懂,以 TDD 的模式,从接口测试样例开始,逐步完成整个 channel 的设计。这里同样也这么做,采用 TDD 模式,并且对齐标准库的 mspc 的接口。
Read moreMost of this article is based on aysnc-book and explains how to implement a simple executor. The project is available at simple-executor.
Asynchronous programming in Rust involves three important concepts: Future, executor, and reactor. When an executor polls a Future, it passes a waker to the Future. The reactor then waits for the events involved in the Future and, when an event completes, uses this waker to wake the executor so that it can continue executing the Future.
The core of asynchronous programming in Rust is the Future trait. The Future trait has an associated type, Output, which is the result returned when the Future completes. The executor polls the Future step by step until it completes.
1 | pub trait Future { |
When a Future is submitted to the executor, the executor polls it. If it returns Poll::Ready(Result), the Future is finished. If it returns Poll::Pending, the Future needs to be submitted to the executor again the next time it can be polled. This is done by calling Waker’s wake method.
本文大部分内容参考 aysnc-book,介绍了如何实现一个简单的 executor,项目地址 simple-executor。
Rust 异步编程涉及到三个比较重要的概念: Future 、 executor 和 reactor 。executor 在 poll Future 的时候,会传递给 Future 一个 waker ,之后由reactor 会等待 Future 涉及的 event ,在 event 完成的时候再通过这个 waker 唤醒 executor 来继续执行这个 Future。
Rust 异步编程的核心是 Future trait。Future trait 有一个关联类型 Output ,它就是 Future 完成的时返回的结果。executor 会逐步 poll 这个 Future,直到 Future 完成
1 | pub trait Future { |
当提交 Future 给 executor 的时候,executor 会 poll 这个 Future,如果返回 Poll::Ready(Result),这个 Future 就结束了,否则返回 Poll::Pending 的话,就需要在 Future 下次可以 poll 的时候重新把 Future 提交给 executor,做这件事的其实就是通过调用 Waker 的 wake 方法
The Tokio library provides the copy_bidirectional method, which makes it easy to copy data bidirectionally for data structures that implement AsyncRead+AsyncWrite. In my previous article (which is still being revised), I implemented a simple proxy based on WebSocket, using tokio_tungstenite as the WebSocket library.
Originally, I customized a function similar to bidirectional copying on the client side instead of directly using copy_bidirectional. This was because the WebSocketStream built on TcpStream by tokio_tungstenite did not implement AsyncRead+AsyncWrite. In addition, the WebSocketStream returned by tokio_tungstenite does not expose convenient read and write methods; it only implements Sink and Stream. However, it provides a split method that separates the stream’s reads and writes. The split read half blindly writes the data it reads to TcpStream, while the split write half blindly writes data read from TcpStream to its own stream. The resulting problem is that when reads and writes are separated, there is no graceful way to detect a Close packet and exit the read and write flows described above. Here is a simple illustration of the original implementation:
tokio 库提供了 copy_bidirectional 方法,可以很方便地对于实现了 AsyncRead+AsyncWrite 的数据结构实现双向拷贝。在上一篇文章(还在持续修改)中,我基于 websocket 实现了一个简单代理,使用的 websocket 库是 tokio_tungstenite 。
原本在client端我自定义了类似双向拷贝的函数,而不是直接使用 copy_bidirectional ,这是由于 tokio_tungstenite 基于 TcpStream 构建的 WebSocketStream 并没有实现 AsyncRead+AsyncWrite 。另外,由于 tokio_tungstenite 返回的的 WebSocketStream 没有对外暴露好用的 read 和 write 方法,只是实现了 Sink 和 Stream,不过它对外提供了 split 方法,可以将 stream 的读写分别拆分开来,拆分出来的读无脑将读到的数据写到 TcpStream ,拆分出来的写无脑将从 TcpStream 读到的写到自己的 stream 。这样带来的问题是,在读写分离的情况下,不能够优雅的感知 Close 包,无法优雅的退出上述读写的流程。简单展示下原本的实现:
The first step toward becoming familiar with a language might be writing a proxy with it. After studying Rust syntax for a long time, I still had little opportunity to use it at work, so I had to create a need for it myself. I hope to find a job writing Rust in the future, or one that gives me the freedom to learn new things.
The main purpose is to practice writing Rust. Rust is also safe and efficient.
Getting past the mysterious wall requires an encrypted connection; otherwise, it can easily get killed. A common approach, like the one used by Shadowsocks, is to define a custom encryption protocol, but that is rather troublesome. To keep things simple, I used TLS WebSocket directly for this purpose.
The benefits of connection pooling hardly need explaining. Eliminating connection establishment time makes the proxy faster.
The overall proxy flow looks like this:

Source code: https://github.com/suikammd/rust-proxy
熟悉一门语言的第一步可能是用它写个代理吧。看了很久 rust 的语法,工作上派不上啥用场,只能自己制造需求了,希望以后能找份写 rust 的工作,或者可以自由学东西的工作。
主要目的是练习写 rust ,另外 rust 安全高效
过神秘的墙需要加密连接,否则容易毙掉。常见的做法类似 shadowsocks ,自定义加密协议,但是这种比较麻烦,偷懒直接使用 TLS websocket 来做这件事
池化连接的好处不容多说,少了建立连接的时间,代理速度更快
代理大体流程是这样的:

代码地址:https://github.com/suikammd/rust-proxy
Read moreA beginner’s first CTF experience. Many thanks to Hai for the patient guidance!
The problem categories included SQL injection, SSRF, XSS, privilege bypass, information disclosure, and Android. Most categories created by ByteDance contained multiple problems of progressively increasing difficulty. For example, in SQL injection, the first problem showed you the Golang source code, so you could directly see how Gorm concatenated strings. The later problems gradually revealed less information to participants.
Useful tools:
小白的CTF初体验,感谢海大佬的耐心指导!
题型包括SQL注入、SSRF、XSS、权限绕过、信息泄漏、安卓。字节出的大部分题型都包含多道题,题目的难度也是由浅入深的。比如SQL注入,你可以看到第一道题的golang源码,直接就可以看到gorm是如何拼接字符串的,接下来的几道题目会逐渐减少对参赛者暴露的信息。
优秀工具介绍:
Rate limiters primarily prevent highly concurrent traffic from overwhelming a service. Common rate-limiting algorithms include counters, token buckets, and leaky buckets. A token bucket directly rejects requests when there are insufficient tokens, while a leaky bucket blocks requests when there are insufficient requests. This article implements a rate limiter that allows a moderate amount of burst traffic. This is just for fun, so please don’t criticize my crude implementation.
限流器主要是防止高并发流量打垮服务,常见的限流器算法有计数器、令牌桶和漏桶,令牌桶在令牌不足时会直接拒绝请求,漏桶在请求不足时会阻塞请求。本文实现一个允许适量流量激增的限流器。只是写着玩,不要喷我写的挫谢谢。
Read moreWhen I first started learning Go, what puzzled me most was passing slices to functions: had the slice changed or not? I recently discovered that some of my colleagues were not clear about this either, so I decided to take a closer look at whether modifying a slice passed to a function changes the original slice.
I’ll start with the structure of a slice. As you can see, a slice is actually backed by an array.
1 | type slice struct { |
开始看golang的时候,最纠结的就是函数传递slice,这个slice到底有没有发生变化。最近发现周围的同事也没有弄清楚,于是打算仔细看下,在函数里修改传递的slice会不会改变原来的slice。
首先从slice的结构入手,可以看到slice底层其实就是一个数组
1 | type slice struct { |
Recently, I have often worked with ANTLR. At first, I only half understood it and got things to work through all kinds of blind trial and error. After finishing that work, I wanted to systematically understand what a compiler frontend actually does: what lexical analysis and syntax analysis do, as well as the syntax of ANTLR lexers and parsers.
From a fairly high-level perspective, a compiler frontend processes a string in two steps. The first step is lexical analysis, which parses the string into a sequence of tokens. The second step is syntax analysis, which matches the corresponding grammar rules according to the grammar.
Lexical analysis
Take the SPL query search id |stats count by id as an example. Lexical analysis parses this string into the token sequence search, id, stats, count, by, and |, while also obtaining the type corresponding to each token.
Syntax analysis
The result of lexical analysis serves as the input to the parser. After syntax analysis, the AST for this SPL query is determined.
最近工作经常跟antlr打交道,一开始半懂不懂,各种瞎写胡试,也能work。工作做完之后就想系统了解下编译器前端到底做了哪些事情,也就是词法分析和语法分析究竟做了什么,以及antlr的lexer和parser的语法。
从一个比较high level的角度看待编译器前端对字符串的处理流程,第一步是做词法解析,将字符串解析成Token序列,第二步是做语法分析,根据语法匹配命中相应的语法规则。
词法分析
以这句SPL查询search id |stats count by id 为例,词法分析做的事情就是把这个字符串解析成Token序列:search、id、stats、count、by、|,同时也会拿到这些token对应的类型。
语法分析
词法分析的结果作为语法分析器的输入,经过语法分析之后会确定这个SPL的AST语法树。