The Tokio library provides the copy_bidirectional method, which makes it easy to copy data bidirectionally for data structures that implement AsyncRead+AsyncWrite. In my previous article (which is still being revised), I implemented a simple proxy based on WebSocket, using tokio_tungstenite as the WebSocket library.
Originally, I customized a function similar to bidirectional copying on the client side instead of directly using copy_bidirectional. This was because the WebSocketStream built on TcpStream by tokio_tungstenite did not implement AsyncRead+AsyncWrite. In addition, the WebSocketStream returned by tokio_tungstenite does not expose convenient read and write methods; it only implements Sink and Stream. However, it provides a split method that separates the stream’s reads and writes. The split read half blindly writes the data it reads to TcpStream, while the split write half blindly writes data read from TcpStream to its own stream. The resulting problem is that when reads and writes are separated, there is no graceful way to detect a Close packet and exit the read and write flows described above. Here is a simple illustration of the original implementation:
1 | // copy data from websocket to tcp |
Discovering that the previously working curl hung
I wanted to wrap this WebSocketStream myself and implement AsyncRead + AsyncWrite for it, so that I could use copy_bidirectional’s capabilities to handle some corner cases for me, such as connections disconnecting abnormally. In short, this would allow me to use copy_bidirectional directly, with inbound as TcpStream and outbound as WebSocketStream.
1 | let _ = copy_bidirectional(&mut inbound, &mut outbound).await |
After painstakingly finishing the implementation, I expected it to work right away (hahaha, inexplicable confidence). I continued using curl to test the proxy.
1 | curl -vvv --socks5 127.0.0.1:8081 t.cn |
I expected a 302 response, but curl hung instead.
1 | * Trying 127.0.0.1:8081... |
Could there be a problem with the AsyncRead + AsyncWrite implementation? I briefly reviewed the code but did not find any particularly obvious issues. Could it be a library problem? Once this thought occurred to me, I embarked on a long and indirect journey in search of a workaround.
More specific symptoms
The code changes were only on the client side, but just in case, I still tried capturing packets on both the client and server to see whether either had a problem. Since the port of the WebSocket connection established between the client and server was difficult to determine, I decided to directly capture packets received and sent by the server.
1 | sudo tcpdump -i lo -X dst port 8080 |
I found that the WebSocket handshake was fine, and the server also sent the 302 packet received from t.cn to the client. Since there was data on the connection, why couldn’t the client read it? Below is the client’s AsyncRead implementation (with the concrete implementation omitted). I added a log line before the actual implementation, so that poll..... should be printed during poll_read.
1 | impl AsyncRead for WebSocketStreamConnection { |
When I reran the code, the expected log was not printed. I then swapped the positions of inbound and outbound in copy_bidrectional and ran the code again. This time, the expected log was printed (at this point, I realized that the thread might be deadlocked rather than stuck in an infinite loop). Could Tokio’s implementation have a problem? Taking it one step further, could tokio_tungstenite’s implementation have a problem?
Perhaps Tokio’s copy_bidirectional had gone wrong
(At this stage, I dug into the details of Tokio’s copy_bidirectional. Because I wanted to share this experience, I also looked at tokio_tungstenite’s implementation.)
tokio copy_bidrectional
1 | let _ = copy_bidirectional(&mut inbound, &mut outbound).await |
copy_bidirectional constructs a CopyBidirectional Future.
1 | struct CopyBidirectional<'a, A: ?Sized, B: ?Sized> { |
The concrete poll implementation performs equivalent bidirectional copying for inbound and outbound.
1 | let a_to_b = transfer_one_direction(cx, a_to_b, &mut *a, &mut *b)?; |
The transfer_one_direction implementation uses a large loop to implement the TransferState state transitions. TransferState changes from Running to ShuttingDown and finally to Done. The core logic is in the Running step, where CopyBuffer avoids the overhead of initializing the buffer multiple times during bidirectional copying. CopyBuffer‘s poll_copy method copies data from reader to writer .
1 | let count = ready!(buf.poll_copy(cx, r.as_mut(), w.as_mut(), prefix.clone()))?; |
poll_copy also contains a large loop that repeatedly calls poll_read on the reader and writes the data it reads to the writer, while updating CopyBuffer’s state. For example, it sets read_done to true when reading is complete, and updates pos and cap after data is normally read and written to the writer. The data between pos and cap is the data expected to be written to the writer.
At this point, I have covered most of the logic of copy_directional. It looked as though swapping the arguments to copy_bidirectional should make the two directions completely equivalent (of course, if I had been a little more careful at this point… the rest of this article would not exist, haha).
Could the thread be stuck in poll_copy?
- The first step was to see whether the thread was stuck copying from inbound (TcpStream) to outbound (a -> b), or from outbound to inbound (b -> a). Therefore, I added
prefix: Stringto thetransfer_one_directionsignature, and also modified buf’spoll_copysignature to addprefix: String.
1 | fn transfer_one_direction<A, B>( |
1 | pub(super) fn poll_copy<R, W>( |
This step required patching Tokio’s code so that I could easily modify the copy_bidirectional logic while debugging.
1 | [patch.crates-io] |
After rerunning it, I found that it was frantically printing a -> b, which meant it was stuck in the logic copying data from TcpStream to WebSocketStream.
- I then added many log lines inside the
poll_copyloop.
1 | pub(super) fn poll_copy<R, W>( |
After rerunning it, I found that it was frantically printing the #4 log line, and pos was 69 while cap was 68! I was secretly delighted. Could this be a Tokio bug? But a simple echo server I had written earlier worked normally, so it still had to be my problem. I then carefully examined the following code and suddenly remembered that when I implemented poll_write for WebSocketStream, the length I returned was buf.len() + 1 (1 was the length of my custom packet header). The problem was immediately obvious.
1 | // If our buffer has some data, let's write it out! |
At the time, I found it hilarious. Nevertheless, the debugging process was very meaningful. At moments like this, I am reminded of something someone said: you can’t master it if you don’t understand it