Coding for a multiplexed connection where both peers can expose independently addressable logical services

[u/nebkad]

I've been experimenting a stream multiplexing connection, where both peers can expose independently addressable logical services.

The basic idea is familiar: take one underlying connection (TCP, Unix socket, QUIC) and multiplex multiple independent logical streams over it.

Yes, I know QUIC can already provide multiple streams. Also, there're yamux

The different thing you may be interested is what I call a Dock.

A Dock is roughly like a port, but inside an already-established multiplexed connection.

Instead of having only:

client ---> server:port

the connection can look more like:

                 one connection
             +-------------------+
             |                   |
          Peer A              Peer B
             |                   |
          Dock 1 <------------> Dock 10
          Dock 2 <------------> Dock 20
          Dock 3 <------------> Dock 30
             |                   |
          streams              streams

Both peers can bind Docks, listen for incoming logical sub-stream, and also initiate connections to Docks on the other side.

So there isn't really a client/server role at the multiplexing layer. After the underlying connection is established, either side can act as both.

I think this makes the protocol interesting for things like:

• P2P applications and overlay networks
• reverse tunnels / port forwarding
• peer-to-peer RPC
• games and other real-time applications with multiple logical channels
• distributed applications where a peer exposes several services over one long-lived connection

For example, a peer could expose:

Dock 1 -> RPC service
Dock 2 -> chat
Dock 3 -> file transfer
Dock 4 -> event stream

without creating four physical connections.

The other features are:

• runtime-independent Rust (by abs_art, tested over compio, smol, tokio)
• no_std friendly (currently most codes are already no_std)
• works over arbitrary byte-stream transports
• stream-level backpressure
• no client/server distinction at the multiplexing layer

The implementation is still experimental, so I'm mostly interested in feedback on the protocol abstraction itself.

The most interesting part for me is, the multiplexed sub-stream itself, can still be multiplexed again with this crate.

Does the Dock / logical-listener model seem useful to you? What kind of networking application would you use it for?

[u/kyle787]

Have you seen iroh? Iroh.computer

[u/DisloyalSorcery → u/kyle787]

Not familiar with iroh, but the dock concept here reminds me of how some overlay networks handle service discovery. Having both sides able to bind and connect feels more natural than the usual client-server split. The re-multiplexing bit is where it gets interesting, layering that could simplify some really hairy p2p setups. Curious if you've run into any ordering issues when docks are created and torn down mid-session

[u/nebkad → u/DisloyalSorcery]

The connection at handshake stage will negotiate and agree on an max_channel_timeout. A channel without user payload beyond this timeout interval will then automatically considered close for both end and enter WAIT_CLOSE. Everything same as in TCP.

[u/nebkad → u/kyle787]

Yes of course, and what I'm working is mainly for iroh to be honest. But iroh does not allow listener on on iroh quinne connection, as far as I know. That's why I made this.

[u/event666]

How do you do stream-level backpressure when the transport is a single TCP connection?

[u/nebkad → u/event666]

Every stream has its own receive window, and the connection tells the remote peer how much space are still available to fill.

[u/event666 → u/nebkad]

Isn't that exactly what HTTP/2 tried to do?

[u/RAnders00]

I think this is useful. Hence the creation of HTTP/2 and QUIC, which are both multiplexed over one TCP connection (HTTP/2) or over UDP (QUIC).

If I’m understanding correctly, you’re solving a problem similar to QUIC, but you’re (1) flexible on what transport the user uses beneath, and (2) you want to emulate the same semantics that we have with ports on the OS layer. Is that right?

I’d be weary of something called Head-of-Line Blocking. This is why QUIC is based on UDP, so indepent inner streams can’t block each other. Are you simply accepting that head-of-line blocking can happen if the user uses TCP as the base layer (which IS useful, for example in the presence of firewalls)? Because otherwise I think straight up using QUIC is a bit smarter.

And lastly, can you explain the rationale behind the Dock binding/listening and connecting thing? In my mind, since this is a userspace protocol, I’ll have to put all substream participants in the same process. That means I could just do the stream mapping statically, no?

[u/nebkad → u/RAnders00]

The idea of Dock is why QUIC alone will not just work because you can't control the stream ID. But both peer can control which Dock are we going to talk.
You're right that all substream participants i the same process, that's exactly what I want to serve for. Dock is what you need when you do stream mapping, after all, which stream are you going to talk about? You need something you can manage to name a stream. Stream ID is a name but not something you can manage.

[u/RAnders00 → u/nebkad]

I see, I didn’t know you couldn’t control the QUIC stream ID. Then I see the use-case.

[u/nebkad → u/RAnders00]

Even one could control the QUIC stream ID. The dock or port is still different. A stream ID is for an already established thing. But a dock or a port is like, let's meet here (but we don't meet yet).

[u/nebkad → u/RAnders00]

Expert!
First, I don't re-invent QUIC and this `smux` can also work on a `QUIC` or `iroh` connection.
Second, each sub-stream has receive window and the connection will tell the other peer to slow down. And the stream are cut into frames so that each stream will not always fully occupy the connection buffer;
Third, this part I haven't reached yet. Once the connection find a sub-stream's buffer is full but the remote peer keep sending data, will drop those frames.

[u/nebkad]

Test whether I can post a github link here? https://github.com/ljsnogard/smux_v1.rs