Multipath Extension for QUIC

1.1. Basic Design Points

This proposal is based on several basic design points:¶

• Re-use as much as possible mechanisms of QUIC version 1. In particular, this proposal uses path validation as specified for QUIC version 1 and aims to re-use as much as possible of QUIC's connection migration.¶

• Use the same packet header formats as QUIC version 1 to minimize the difference between multipath and non-multipath traffic being exposed on wire.¶

• Congestion Control must be per-path (following [QUIC-TRANSPORT]) which usually also requires per-path RTT measurements¶

• PMTU discovery should be performed per-path¶

• The use of this multipath extension requires the use of non-zero length connection IDs in both directions.¶

• Connection IDs are associated with a path ID. The path initiation associates that path ID with a 4-tuple of source and destination IP address as well as source and destination port.¶

• Migration is detected without ambiguity when a packet arrives with a connection ID pointing to an existing path ID, but the connection ID and/or the 4-tuple are different from the value currently used for that path.¶

• Paths can be closed at any time, as specified in Section 3.3.¶

• It is possible to create multiple paths sharing the same 4-tuple. Each of these paths can be closed at any time, like any other path.¶

Further the design of this extension introduces an explicit path identifier and use of multiple packet number spaces as further explained in the next sections.¶

1.2. Introduction of an Explicit Path Identifier

This extension specifies a new path identifier (Path ID), which is an integer between 0 and 2^32-1 (inclusive). Path identifies are generated monotonically increasing and cannot be reused. The connection ID of a packet binds the packet to a path identifier, and therefore to a packet number space.¶

The same Path ID is used in both directions to address a path in the new multipath control frames, such as PATH_ABANDON Section 7.2, PATH_STANDBY Section 7.3}, PATH_AVAILABLE Section 7.3 as well as PATH_ACK Section 7.1. Further, connection IDs are issued per Path ID using the PATH_NEW_CONNECTION_ID frame (see Section 7.4). That means each connection ID is associated with exactly one path identifier but multiple connection IDs are usually issued for each path identifier.¶

The Path ID of the initial path is 0. Connection IDs which are issued by a NEW_CONNECTION_ID frame Section 19.15. of [QUIC-TRANSPORT] respectively are associated with Path ID 0. Also, the Path ID for the connection ID specified in the "preferred address" transport parameter is 0. Use of the "preferred address" is considered as a migration event that does not change the Path ID.¶

1.3. Use of Multiple Packet Number Spaces

This extension uses multiple packet number spaces, one for each path. As such, each path maintains distinct packet number states for sending and receiving packets, as in [QUIC-TRANSPORT]. Using multiple packet number spaces enables direct use of the loss recovery and congestion control mechanisms defined in [QUIC-RECOVERY] on a per-path basis.¶

Each Path ID-specific packet number space starts at packet number 0. When following the packet number encoding algorithm described in Appendix A.2 of [QUIC-TRANSPORT], the largest packet number (largest_acked) that has been acknowledged by the peer in the Path ID-specific packet number space is initially set to "None".¶

Using multiple packet number spaces requires changes in the way AEAD is applied for packet protection, as explained in Section 4. More concretely, the Path ID is used to construct the packet protection nonce in addition to the packet number in order to enable use of the same packet number on different paths. Respectively, the Path ID is limited to 32 bits to ensure a unique nonce. Additional consideration on key updates are explained in Section 4.2.¶

1.4. Conventions and Definitions

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.¶

We assume that the reader is familiar with the terminology used in [QUIC-TRANSPORT]. When this document uses the term "path", it refers to the notion of "network path" used in [QUIC-TRANSPORT].¶

3.1. Path Initiation

Opening a new path requires the use of a new connection ID (see Section 9.5 of [QUIC-TRANSPORT]) mapped to an unused Path ID (see Section 1.2). Instead of NEW_CONNECTION_ID frame as specified in Section 19.15 of [QUIC-TRANSPORT], each endpoint uses the PATH_NEW_CONNECTION_ID frame as specified in this extension to issue Path ID-specific connections IDs. The same Path ID is used in both directions. As such to open a new path, both sides need at least one connection ID (see Section 5.1.1 of [QUIC-TRANSPORT]), which is associated with the same, unused Path ID. When the peer receives the PATH_CHALLENGE, it MUST pick a Connection ID with the same Path ID for sending the PATH_RESPONSE.¶

When the multipath extension is negotiated, a client that wants to use an additional path MUST validate the peer's address before sending any data packets as described in (Section 8.2 of [QUIC-TRANSPORT]), unless it has previously validated the 4-tuple used for that path.¶

After receiving packets from the client on a new path, if the server decides to use the new path, the server MUST validate the peer's address before sending any data packets as described in (Section 8.2 of [QUIC-TRANSPORT]), unless it has previously validated the 4-tuple used for that path. Until the client's address is validated, the anti-amplification limit from Section 8 of [QUIC-TRANSPORT] applies.¶

The server may receive packets for a yet unused Path ID that do not contain a path challenge. Such packets are valid if they can be properly decrypted and if they contain a valid connection ID.¶

Each endpoint MUST also validate that a minimum MTU of 1200 bytes is supported on the path. This can be done during initial path validation or separately later if the amplification limit prevents it initially, as specified in Section 8.2.1 of RFC9000.¶

An endpoint that receives packets on a new path and does not want to establish this path is expected to close the path by sending a PATH_ABANDON on another path, as specified in section Section 3.3.¶

An endpoint that has no active connection ID for this path or lacks other resource to immediately configure a new path could delay sending the PATH_RESPONSE until sufficient resource are available. Long delays may cause the peer to repeat the PATH_CHALLENGE and eventually send a PATH_ABANDON, in which case the procedures specified in Section Section 3.3 apply.¶

PATH_ACK frames (defined in Section 7.1) can be returned on any path. If the PATH_ACK is preferred to be sent on the same path as the acknowledged packet (see Section 6.3 for further guidance), it can be beneficial to bundle a PATH_ACK frame with the PATH_RESPONSE frame during path validation.¶

If validation succeeds, the client can continue to use the path. If validation fails, the client MUST NOT use the path and can remove any status associated to the path initiation attempt. However, as the used Path ID is anyway consumed, the endpoint MUST explicitly close the path as specified in Section 3.3.¶

Section 9.1 of [QUIC-TRANSPORT] introduces the concept of "probing" and "non-probing" frames. A packet that contains at least one "non-probing" frame is a "non-probing" packet. When the multipath extension is negotiated, the reception of a "non-probing" packet on a new path with a new so far unused Path ID does not impact the path status of any existing path. Therefore, any frame can be sent on a new path at any time as long as the anti-amplification limits (Section 21.1.1.1 of [QUIC-TRANSPORT]) and the congestion control limits for this path are respected.¶

Connection ID changes as specified in Section 5.1.2 of [QUIC-TRANSPORT] and in Section 9 of [QUIC-TRANSPORT] for connection migration apply for connection IDs associated to the same Path ID. With the successful negotiation of the extension specified in this draft, endpoints have to consider the reception of a packet with a connection ID associated to an so far unused Path ID as an attempt to establish a new path.¶

As specified in Section 9.3 of [QUIC-TRANSPORT], the server is expected to send a new address validation token to a client following the successful validation of a new client address. The client will receive several tokens. When considering using a token for subsequent connections, it might be difficult for the client to pick the "right" token among multiple tokens obtained in a previous connection. The client is likely to fall back to the strategy specified in Section 8.1.3 of [QUIC-TRANSPORT], i.e., pick the last received token. To avoid issues when clients make the "wrong" choice, a server should issue a token that is capable of validating any of the previously validated addresses. Further guidance on token usage can be found in Section 8.1.3 of [QUIC-TRANSPORT].¶

3.2. Path Status Management

An endpoint uses the PATH_BACKUP and PATH_AVAILABLE frames to inform the peer that it should send packets with the preference expressed by these frames. Note that an endpoint might not follow the peer’s advertisements, but these frames are still a clear signal of the peer's preference of path usage. Each peer indicates its preference of path usage independently of the other peer. That means that peers may have different usage preferences for the same path. Depending on the data sender's decisions, this may lead to usage of paths that have been indicated as "standby" by the peer or non-usage of some locally available paths.¶

PATH_AVAILABLE indicates that a path is "available", i.e., it suggests to the peer to use its own logic to split traffic among available paths.¶

PATH_BACKUP marks a path as "standby", i.e., it suggests that no traffic should be sent on that path if another path is available. If all established paths are marked as "standby", no guidance is provided about which path should be used.¶

If an endpoint starts using a path that was marked as "standby" by its peer because it has detected issues on the paths marked as "available", it is RECOMMENDED to update its own path state signaling such that the peer avoids using the broken path. An endpoint that detects a path breakage can also explicitly close the path by sending a PATH_ABANDON frame (see Section 3.3) in order to avoid that its peer keeps using it and enable faster switch over to a standby path. If the endpoints do not want to close the path immediately, as connectivity could be re-established, PING frames can potentially be used to quickly detect connectivity changes and switch back in a timely way.¶

If no frame indicating a path usage preference was received for a certain path, the preference of the peer is unknown and the sender needs to decide based on it own local logic if the path should be used.¶

Endpoints use the Path ID in these frames to identify which path state is going to be changed. Note that both frames can be sent via a different path and therefore might arrive in different orders. The PATH_AVAILABLE and PATH_BACKUP frames share a common sequence number space to detect and ignore outdated information.¶

3.3. Path Close

Each endpoint manages the set of paths that are available for transmission. At any time in the connection, each endpoint can decide to abandon one of these paths, for example following changes in local connectivity or local preferences. After an endpoint abandons a path, the peer can expect to not receive any more packets on that path.¶

Note that other explicit closing mechanisms of [QUIC-TRANSPORT] still apply on the whole connection. In particular, the reception of either a CONNECTION_CLOSE (Section 10.2 of [QUIC-TRANSPORT]) or a Stateless Reset (Section 10.3 of [QUIC-TRANSPORT]) closes the connection.¶

An endpoint that wants to close a path MUST explicitly terminate the path by sending a PATH_ABANDON frame. Note that while abandoning a path will cause connection ID retirement, the inverse is not true: retiring the associated connection IDs does not indicate path abandonment (see further Section 3.4). This is true whether the decision to close the path results from implicit signals such as idle time or packet losses (see Section 3.3.3) or for any other reason, such as management of local resources. It is also possible to abandon a path for which no packet has been sent (see Section 3.3.4).¶

When an endpoint receives a PATH_ABANDON frame, it MUST send a corresponding PATH_ABANDON frame if it has not already done so. It MUST stop sending any new packet on the abandoned path, and it MUST treat all connection identifiers received from the peer for that path as immediately retired. However, knowledge of the connection identifiers received from the peer and of the state of the number space associated to the path SHOULD be retained while packets from the peer might still be in transit, i.e., for a delay of 3 PTO after the PATH_ABANDON frame has been received from the peer, both to avoid generating spurious stateless packets as specified in Section 3.3.1 and to be able to acknowledge the last packets received from the peer as specified in Section 3.3.2.¶

After receiving or sending a PATH_ABANDON frame, the endpoints SHOULD promptly send PATH_ACK frames to acknowledge all packets received on the path and not yet acknowledged, as specified in Section 3.3.2. When an endpoint finally deletes all resource associated with the path, the packets sent over the path and not yet acknowledged MUST be considered lost.¶

After a path is abandoned, the Path ID MUST NOT be reused for new paths, as the Path ID is part of the nonce calculation Section 4.¶

PATH_ABANDON frames can be sent on any path, not only the path that is intended to be closed. Thus, even if connectivity on that path is already broken but there is still another usable path, it is RECOMMENDED to send the PATH_ABANDON frames on another path.¶

If a PATH_ABANDON frame is received for the only open path of a QUIC connection, the receiving peer SHOULD send a CONNECTION_CLOSE frame and enter the closing state. If the client received a PATH_ABANDON frame for the last open path, it MAY instead try to open a new path, if available, and only initiate connection closure if path validation fails or a CONNECTION_CLOSE frame is received from the server. Similarly the server MAY wait for a short, limited time such as one PTO if a path probing packet is received on a new path before sending the CONNECTION_CLOSE frame.¶

3.4. Allocating, Consuming, and Retiring Connection IDs

With the multipath extension, each connection ID is associated with one path that is identified by the Path ID that is specified in the Path Identifier field of the PATH_NEW_CONNECTION_ID frame Section 7.4. The Path ID 0 indicates the initial path of the connection. Respectively, the connection IDs used during the handshake belong to the initial path with Path ID 0. The PATH_NEW_CONNECTION_ID frame is used to issue new connection IDs for all paths. In order to let the peer open new paths, it is RECOMMENDED to proactively issue a Connection ID for at least one unused Path ID, as long as it remains compatible with the peer's Maximum Path ID limit.¶

When issuing path-specific connection IDs, an endpoint associates a sequence number as specified in Section 5.1.1 of [QUIC-TRANSPORT]. Each Path ID has its own connection ID sequence number space whose initial value is 0. On non-initial paths (i.e., Path ID different from 0), the initial path connection ID is provided by a PATH_NEW_CONNECTION_ID frame with a sequence number value of 0.¶

Each endpoint maintains the set of connection IDs received from its peer for each path, any of which it can use when sending packets on that path; see also Section 5.1 of [QUIC-TRANSPORT]. Usually, it is desired to provide at least one additional connection ID for all used paths, to allow for migration. As further specified in Section 5.1 of [QUIC-TRANSPORT] connection IDs cannot be issued more than once on the same connection and therefore are unique for the scope of the connection, regardless of the associated Path ID.¶

Section 5.1.2 of [QUIC-TRANSPORT] indicates that an endpoint can change the connection ID it uses to another available one at any time during the connection. For the extension specified in this draft, endpoints MUST only rotate to another connection IDs associated to the same Path ID. Use of a connection ID associated with another Path ID will be considered as an attempt to open a new path instead.¶

Over a given path, both endpoints use connection IDs associated to a given Path ID. To initiate a path, each endpoint needs to advertise at least one connection ID for a given Path ID to its peer. Endpoints SHOULD NOT introduce discontinuity in the issuing of Path IDs through their connection ID advertisements as path initiation requires available connection IDs for the same Path ID on both sides. For instance, if the maximum Path ID limit is 2 and the endpoint wants to provide connection IDs for only one Path ID inside range [1, 2], it should select Path ID 1 (and not Path ID 2). Similarly, endpoints SHOULD consume Path IDs in a continuous way, i.e., when creating paths. However, endpoints cannot expect to receive new connection IDs or path initiation attempts with in order use of Path IDs due to out-of-order delivery or path validation failure.¶

Section 5.1.2. of [QUIC-TRANSPORT] specifies the retirement of connection IDs. In order to identify a connection ID correctly when the multipath extension is used, endpoints have to use the PATH_RETIRE_CONNECTION_ID frame instead of the RETIRE_CONNECTION_ID frame to indicate the respective Path ID together with the connection ID sequence number, at least for all paths with a Path ID other than 0. Endpoints can also use PATH_NEW_CONNECTION_ID and PATH_RETIRE_CONNECTION_ID for the initial path with Path ID 0, however, the use of NEW_CONNECTION_ID and RETIRE_CONNECTION_ID is still valid as well and endpoints need to process these frames accordingly as corresponding to Path ID 0.¶

Endpoints MUST NOT issue new connection IDs with Path IDs greater than the Maximum Path Identifier field in MAX_PATH_ID frames (see Section 7.6) or the value of initial_max_path_id transport parameter if no MAX_PATH_ID frame was received yet. Receipt of a frame with a greater Path ID is a connection error as specified in Section 7. When an endpoint finds it has not enough available unused path identifiers, it SHOULD either send a MAX_PATH_ID frame to increase the active path limit (when limited by the sender) or a PATHS_BLOCKED frame (see Section Section 7.7) to inform the peer that a new path identifier was needed but the current limit set by the peer prevented the creation of the new path.¶

If the client has consumed all the allocated connection IDs for a path, it is supposed to retire those that are not used anymore, and the server is supposed to provide replacements for that path, see Section 5.1.2. of [QUIC-TRANSPORT]. Sending a PATH_RETIRE_CONNECTION_ID frame indicates that the connection ID will not be used anymore. In response, if the path is still open, the peer SHOULD provide new connection IDs using PATH_NEW_CONNECTION_ID frames.¶

Retirement of connection IDs will not retire the Path ID that corresponds to the connection ID or any other path resources as the packet number space is associated with a path.¶

The peer that sends the PATH_RETIRE_CONNECTION_ID frame can keep sending data on the path that the retired connection ID was used on but has to use a different connection ID for the same Path ID when doing so.¶

4.1. Nonce Calculation

Section 5.3 of [QUIC-TLS] specifies AEAD usage, and in particular the use of a nonce, N, formed by combining the packet protection IV with the packet number. When multiple packet number spaces are used, the packet number alone would not guarantee the uniqueness of the nonce. Therefore, the nonce N is calculated by combining the packet protection IV with the packet number and with the least significant 32 bits of the Path ID. In order to guarantee the uniqueness of the nonce, the Path ID is limited to a max value of 2^32-1.¶

To calculate the nonce, a 96-bit path-and-packet-number is composed of the least significant 32 bits of the Path ID in network byte order, two zero bits, and the 62 bits of the reconstructed QUIC packet number in network byte order. If the IV is larger than 96 bits, the path-and-packet-number is left-padded with zeros to the size of the IV. The exclusive OR of the padded packet number and the IV forms the AEAD nonce.¶

For example, assuming the IV value is 6b26114b9cba2b63a9e8dd4f, the Path ID is 3, and the packet number is aead, the nonce will be set to 6b2611489cba2b63a9e873e2.¶

4.2. Key Update

The Key Phase bit update process is specified in Section 6 of [QUIC-TLS]. The general principles of key update are not changed in this specification. Following [QUIC-TLS], the Key Phase bit is used to indicate which packet protection keys are used to protect the packet. The Key Phase bit is toggled to signal each subsequent key update.¶

Because of network delays, packets protected with the older key might arrive later than the packets protected with the new key, however receivers can solely rely on the Key Phase bit to determine the corresponding packet protection key, assuming that there is sufficient interval between two consecutive key updates (Section 6.5 of [QUIC-TLS]).¶

When this specification is used, endpoints SHOULD wait for at least three times the largest PTO among all the paths before initiating a new key update after receiving an acknowledgment that confirms the receipt of the previous key update. This interval is different from that in [QUIC-TLS] which used three times the PTO of the sole single path.¶

The choice of three times the largest PTO is a trade-off: Packets that arrive after their decryption key has been discarded will be dropped. Longer delays reduce the probability of losing packets but keeping old keys longer can negatively impact the security of the protocol. The use of three times the largest PTO aims to minimize packet lost for all paths and therefore limits the impact on performance.¶

Following Section 5.4 of [QUIC-TLS], the Key Phase bit is protected, so sending multiple packets with Key Phase bit flipping at the same time should not cause linkability issue.¶

5.1. Path Establishment

Figure 1 illustrates an example of new path establishment using multiple packet number spaces.¶

In this example it is assumed that both endpoints have indicated an initial_max_path_id value of at least 2, which means both endpoints can use Path IDs 0, 1, and 2. Note that Path ID 0 is already used for the initial path.¶

   Client                                                  Server

   (Exchanges start on default path)
   1-RTT[]: PATH_NEW_CONNECTION_ID[C1, Seq=0, PathID=1] -->
             <-- 1-RTT[]: PATH_NEW_CONNECTION_ID[S1, Seq=0, PathID=1]
             <-- 1-RTT[]: PATH_NEW_CONNECTION_ID[S2, Seq=0, PathID=2]
   ...
   (starts new path)
   1-RTT[0]: DCID=S1, PATH_CHALLENGE[X] -->
                           Checks AEAD using nonce(Path ID 1, PN 0)
        <-- 1-RTT[0]: DCID=C1, PATH_RESPONSE[X], PATH_CHALLENGE[Y],
                                             PATH_ACK[PathID=1, PN=0]
   Checks AEAD using nonce(Path ID 1, PN 0)
   1-RTT[1]: DCID=S1, PATH_RESPONSE[Y],
            PATH_ACK[PathID=1, PN=0], ... -->

In Figure 1, the endpoints first exchange new available connection IDs with the NEW_CONNECTION_ID frame. In this example, the client provides one connection ID (C1 with Path ID 1), and server provides two connection IDs (S1 with Path ID 1, and S2 with Path ID 2).¶

Before the client opens a new path by sending a packet on that path with a PATH_CHALLENGE frame, it has to check whether there is an unused connection IDs for the same unused Path ID available for each side. In this example the Path ID 1 is used which is the smallest unused Path ID available as recommended in Section 3.4. Respectively, the client chooses the connection ID S1 as the Destination Connection ID of the new path.¶

5.2. Path Closure

Figure 2 illustrates an example of path closure.¶

In this example, the client wants to close the path with Path ID 1. It sends the PATH_ABANDON frame to terminate the path. After receiving the PATH_ABANDON frame with Path ID 1, the server also send a PATH_ABANDON frame with Path ID 1.¶

Client                                                      Server

(client tells server to abandon a path with Path ID 1)
1-RTT[X]: DCID=S1 PATH_ABANDON[Path ID=1]->
                           (server tells client to abandon a path)
                    <-1-RTT[Y]: DCID=C1 PATH_ABANDON[Path ID=1],
                                           PATH_ACK[PATH ID=1, PN=X]
1-RTT[U]: DCID=S2 PATH_ACK[Path ID=1, PN=Y] ->

Note that the last acknowledgment needs to be send on a different path. This examples assumes another path which uses connection ID S2 exists.¶

6.1. Number Spaces

As stated in Section 1, when the multipath extension is negotiated, each path uses a separate packet number space. This is a major difference from [QUIC-TRANSPORT], which only defines three number spaces (Initial, Handshake and Application packets).¶

The relation between packet number spaces and paths is fixed. Connection IDs are separately allocated for each Path ID. Rotating the connection ID on a path does not change the Path ID. NAT rebinding, though it changes the 4-tuple of the path, also does not change the path identifier. The packet number space does not change when connection ID rotation happens within a given Path ID.¶

Data associated with the transmission and reception such RTT measurements, congestion control state, or loss recovery are maintained per packet number space and as such per Path ID.¶

6.2. Congestion Control

When the QUIC multipath extension is used, senders manage per-path congestion status as required in Section 9.4 of [QUIC-TRANSPORT]. However, in [QUIC-TRANSPORT] only one path is assumed and as such the requirement is to reset the congestion control status on path migration. With the multipath extension, multiple paths can be used simultaneously, therefore separate congestion control state is maintained for each path. This means a sender is not allowed to send more data on a given path than congestion control for that path indicates.¶

When a Multipath QUIC connection uses two or more paths, there is no guarantee that these paths are fully disjoint. When two (or more paths) share the same bottleneck, using a standard congestion control scheme could result in an unfair distribution of the bandwidth with the multipath connection getting more bandwidth than competing single paths connections. Multipath TCP uses the linked increased algorithm (LIA) congestion control scheme specified in [RFC6356] to solve this problem. This scheme can immediately be adapted to Multipath QUIC. Other coupled congestion control schemes have been proposed for Multipath TCP such as [OLIA].¶

6.3. Computing Path RTT

Acknowledgment delays are the sum of two one-way delays, the delay on the packet sending path and the delay on the return path chosen for the acknowledgments. When different paths have different characteristics, this can cause acknowledgment delays to vary widely. Consider for example a multipath transmission using both a terrestrial path, with a latency of 50ms in each direction, and a geostationary satellite path, with a latency of 300ms in both directions. The acknowledgment delay will depend on the combination of paths used for the packet transmission and the ACK transmission, as shown in Table 1.¶

The PATH_ACK frames describe packets that were sent on the specified path, but they may be received through any available path. There is an understandable concern that if successive acknowledgments are received on different paths, the measured RTT samples will fluctuate widely, and that might result in poor performance. While this may be a concern, the actual behavior is complex.¶

The computed values reflect both the state of the network path and the scheduling decisions by the sender of the PATH_ACK frames. In the example above, we may assume that the PATH_ACK will be sent over the terrestrial link, because that provides the best response time. In that case, the computed RTT value for the satellite path will be about 350ms. This lower than the 600ms that would be measured if the PATH_ACK came over the satellite channel, but it is still the right value for computing for example the PTO timeout: if a PATH_ACK is not received after more than 350ms, either the data packet or its PATH_ACK were probably lost.¶

The simplest implementation is to compute smoothed_rtt and rttvar per Section 5.3 of [QUIC-RECOVERY] regardless of the path through which PATH_ACK frames are received. This algorithm will provide good results, except if the set of paths changes and the PATH_ACK sender revisits its sending preferences. This is not very different from what happens on a single path if the routing changes. The RTT, RTT variance and PTO estimates will rapidly converge to reflect the new conditions. There is however an exception: some congestion control functions rely on estimates of the minimum RTT. It might be prudent for nodes to remember the path over which the PATH_ACK that produced the minimum RTT was received, and to restart the minimum RTT computation if that path is abandoned.¶

6.4. Packet Scheduling

The transmission of QUIC packets on a regular QUIC connection is regulated by the arrival of data from the application and the congestion control scheme. QUIC packets that increase the number of bytes in flight can only be sent when the congestion window allows it. Multipath QUIC implementations also need to include a packet scheduler that decides, among the paths whose congestion window is open, the path over which the next QUIC packet will be sent. Most frames, including control frames (PATH_CHALLENGE and PATH_RESPONSE being the notable exceptions), can be sent and received on any open path. The scheduling is a local decision, based on the preferences of the application and the implementation.¶

Note that this implies that an endpoint may send and receive PATH_ACK frames on a path different from the one that carried the acknowledged packets. As noted in Section 6.3 the values computed using the standard algorithm reflect both the characteristics of the path and the scheduling algorithm of PATH_ACK frames. The estimates will converge faster if the scheduling strategy is stable, but besides that implementations can choose between multiple strategies such as sending PATH_ACK frames on the path they acknowledge packets, or sending PATH_ACK frames on the shortest path, which results in shorter control loops and thus better performance.¶

6.5. Retransmissions

Simultaneous use of multiple paths enables different retransmission strategies to cope with losses such as: a) retransmitting lost frames over the same path, b) retransmitting lost frames on a different or dedicated path, and c) duplicate lost frames on several paths (not recommended for general purpose use due to the network overhead). While this document does not preclude a specific strategy, more detailed specification is out of scope.¶

As noted in Section 2.2 of [QUIC-TRANSPORT], STREAM frame boundaries are not expected to be preserved when data is retransmitted. Especially when STREAM frames have to be retransmitted over a different path with a smaller MTU limit, new smaller STREAM frames might need to be sent instead.¶

6.6. Handling PTO

An implementation should follow the mechanism specified in [QUIC-RECOVERY] for detecting packet loss on each individual path. A special case happens when the PTO timer expires. According to [QUIC-RECOVERY], no packet will be declared lost until either the packet sender receives a new acknowledgement for this path, or the path itself is finally declared broken. This cautious process minimizes the risk of spurious retransmissions, but it may cause significant delivery delay for the frames contained in these "lost packets".¶

Endpoints could take advantage of the multipath extension, and retransmit the content of the delayed packets on other available paths if the congestion control window on these paths allows.¶

6.7. Handling different PMTU sizes

An implementation should take care to handle different PMTU sizes across multiple paths. As specified in Section 14.3 of [QUIC-TRANSPORT] the DPLPMTUD Maximum Packet Size (MPS) is maintained for each combination of local and remote IP addresses. One simple option, if the PMTUs are relatively similar, is to apply the minimum PMTU of all paths to each path. The benefit of such an approach is to simplify retransmission processing as the content of lost packets initially sent on one path can be sent on another path without further frame scheduling adaptations.¶

6.8. Keep Alive

The QUIC specification defines an optional keep alive process, see Section 5.3 of [QUIC-TRANSPORT]. Implementations of the multipath extension should map this keep alive process to a number of paths. Some applications may keep only one path alive, while others could prefer to maintain liveliness on two or more paths during the connection lifetime. Different applications will likely require different strategies. Once the implementation has decided which paths to keep alive, it can do so by sending Ping frames on each of these paths before the idle timeout expires.¶

6.9. Connection ID Changes

Section 5.1.2 of [QUIC-TRANSPORT] indicates that an endpoint can change the connection ID it uses to another available one at any time during the connection. As such a sole change of the Connection ID without any change in the address does not indicate a path change and the endpoint can keep the same congestion control and RTT measurement state.¶

6.10. Migration and NAT Rebindings

While endpoints assign a connection ID to a specific sending 4-tuple, networks events such as NAT rebinding may make the packet's receiver observe a different 4-tuple. Servers observing a 4-tuple change will perform path validation (see Section 9 of [QUIC-TRANSPORT]). If path validation process succeeds, the endpoints set the path's congestion controller and round-trip time estimator according to Section 9.4 of [QUIC-TRANSPORT].¶

Section 9.3 of [QUIC-TRANSPORT] allows an endpoint to skip validation of a peer address if that address has been seen recently. However, when the multipath extension is used and an endpoint has multiple addresses that could lead to switching between different paths, it should rather maintain multiple open paths instead.¶

More generally, while migration cannot be avoided in case of network-based NAT rebindings, opening a new path instead of active client migration should be strongly preferred when the multipath extension is supported. This enables a smoother handover and allows a simplified migration handling at the server as NAT rebindings imply immediate loss of the old address.¶

6.11. Using multiple paths on the same 4-tuple

As noted in Section 1.1, it is possible to create paths that refer to the same 4-tuple. For example, the endpoints may want to create paths that use different Differentiated Service [RFC2475] markings. This could be done in conjunction with scheduling algorithms that match streams to paths, so that for example data frames for low priority streams are sent over low priority paths. Since these paths use different path IDs, they can be managed independently to suit the needs of the application.¶

There may be cases in which paths are created with different 4-tuples, but end up using the same 4-tuples as a consequence of path migrations. For example:¶

• Client starts path 1 from address 192.0.2.1 to server address 198.51.100.1¶

• Client starts path 2 from address 192.0.2.2 to server address 198.51.100.1¶

• both paths are used for a while.¶

• Server sends packet from address 198.51.100.1 to client address 192.0.2.1, with CID indicating path=2.¶

• Client receives packet, recognizes a path migration, update source address of path 2 to 192.0.2.1.¶

Such unintentional use of the same 4-tuple on different paths ought to be rare. When they happen, the two paths would be redundant, and the endpoint will want to close one of them. Uncoordinated Abandon from both ends of the connection may result in deleting two paths instead of just one. To avoid this pitfall, endpoints could adopt a simple coordination rule, such as only letting the client initiate closure of duplicate paths, or perhaps relying on the application protocol to decide which paths should be closed.¶

7.1. PATH_ACK Frame

The PATH_ACK frame (types TBD-00 and TBD-01) is an extension of the ACK frame specified in Section 19.3 of [QUIC-TRANSPORT]. It is used to acknowledge packets that were sent on different paths, as each path as its own packet number space. If the frame type is TBD-01, PATH_ACK frames also contain the sum of QUIC packets with associated ECN marks received on the acknowledged packet number space up to this point.¶

PATH_ACK frame is formatted as shown in Figure 3.¶

  PATH_ACK Frame {
    Type (i) = TBD-00..TBD-01
         (experiments use  0x15228c00-0x15228c01),
    Path Identifier (i),
    Largest Acknowledged (i),
    ACK Delay (i),
    ACK Range Count (i),
    First ACK Range (i),
    ACK Range (..) ...,
    [ECN Counts (..)],
  }

Compared to the ACK frame specified in [QUIC-TRANSPORT], the following field is added.¶

Path Identifier:

The Path ID associated with the packet number space of the 0-RTT and 1-RTT packets which are acknowledged by the PATH_ACK frame.¶

7.2. PATH_ABANDON Frame

The PATH_ABANDON frame informs the peer to abandon a path.¶

PATH_ABANDON frames are formatted as shown in Figure 4.¶

  PATH_ABANDON Frame {
    Type (i) = TBD-02 (experiments use 0x15228c05),
    Path Identifier (i),
    Error Code (i),
  }

PATH_ABANDON frames contain the following fields:¶

Path Identifier:

The Path ID to abandon.¶

Error Code:

A variable-length integer that indicates the reason for abandoning this path. NO_ERROR(0x0) indicates that the path is being abandoned without any error being encountered. Other error codes can be found in Section 8.¶

PATH_ABANDON frames are ack-eliciting. If a packet containing a PATH_ABANDON frame is considered lost, the peer SHOULD repeat it.¶

Use of the PATH_ABANDON frame is specified in section Section 3.3.¶

7.3. PATH_BACKUP and PATH_AVAILABLE frames

PATH_AVAILABLE frames are used by endpoints to inform the peer that the indicated path is available for sending. PATH_AVAILABLE frames are formatted as shown in Figure 5.¶

  PATH_AVAILABLE Frame {
    Type (i) = TBD-04 (experiments use 0x15228c08),
    Path Identifier (i),
    Path Status Sequence Number (i),
  }

PATH_BACKUP frames are used by endpoints to inform the peer about its preference to not use the indicated path for sending. PATH_BACKUP frames are formatted as shown in Figure 6.¶

  PATH_BACKUP Frame {
    Type (i) = TBD-03 (experiments use 0x15228c07)
    Path Identifier (i),
    Path Status Sequence Number (i),
  }

Both PATH_AVAILABLE and PATH_BACKUP Frames contain the following fields:¶

Path Identifier:

The Path ID the status update corresponds to. All Path IDs that have been issued MAY be specified, even if they are not yet in use over a path.¶

Path Status Sequence Number:

A variable-length integer specifying the per-path sequence number assigned for this frame.¶

The sequence number space is common to the two frame types, and monotonically increasing values MUST be used when sending PATH_AVAILABLE or PATH_BACKUP frames for a given Path ID.¶

Frames may be received out of order. A peer MUST ignore an incoming PATH_AVAILABLE or PATH_BACKUP frame if it previously received another PATH_BACKUP frame or PATH_AVAILABLE frame for the same Path ID with a Path Status sequence number equal to or higher than the Path Status sequence number of the incoming frame.¶

The requirement of monotonically increasing sequence numbers is per path. Receivers could very well receive the same sequence number for PATH_AVAILABLE or PATH_STANDBY Frames on different paths. The receiver of the PATH_AVAILABLE or PATH_BACKUP frame needs to use and compare the sequence numbers separately for each Path ID.¶

PATH_BACKUP frames are ack-eliciting. If a packet containing a PATH_BACKUP frame is considered lost, the peer SHOULD resend the frame only if it contains the last status sent for that path -- as indicated by the sequence number.¶

A PATH_BACKUP or a PATH_AVAILABLE frame MAY be bundled with a PATH_NEW_CONNECTION_ID frame or a PATH_RESPONSE frame in order to indicate the preferred path usage before or during path initiation.¶

7.4. PATH_NEW_CONNECTION_ID frames

The PATH_NEW_CONNECTION_ID frame (type=0x15228c09) is an extension of the NEW_CONNECTION_ID frame specified in Section 19.15 of [QUIC-TRANSPORT]. It is used to provide its peer with alternative connection IDs for 1-RTT packets for a specific path. The peer can then use a different connection ID on the same path to break linkability when migrating on that path; see also Section 9.5 of [QUIC-TRANSPORT].¶

PATH_NEW_CONNECTION_ID frames are formatted as shown in Figure 7.¶

PATH_NEW_CONNECTION_ID Frame {
  Type (i) = TBD-05 (experiments use 0x15228c09),
  Path Identifier (i),
  Sequence Number (i),
  Retire Prior To (i),
  Length (8),
  Connection ID (8..160),
  Stateless Reset Token (128),
}

Compared to the NEW_CONNECTION_ID frame specified in Section 19.15 of [QUIC-TRANSPORT], the following field is added:¶

Path Identifier:

The Path ID associated with the connection ID. This means the provided connection ID can only be used on the corresponding path.¶

Note that, other than for the NEW_CONNECTION_ID frame of Section 19.15 of [QUIC-TRANSPORT], the sequence number applies on a per-path context. This means different connection IDs on different paths may have the same sequence number value.¶

The Retire Prior To field indicates which connection IDs should be retired among those that share the Path ID in the Path Identifier field. Connection IDs associated with different path IDs are not affected.¶

Note that the NEW_CONNECTION_ID frame can only be used to issue or retire connection IDs for the initial path with Path ID 0.¶

The last paragraph of Section 5.1.2 of [QUIC-TRANSPORT] specifies how to verify the Retire Prior To field of an incoming NEW_CONNECTION_ID frame. The same rule applies for PATH_RETIRE_CONNECTION_ID frames, but it applies per path. After the multipath extension is negotiated successfully, the rule for RETIRE_CONNECTION_ID frame is only applied for Path ID 0.¶

7.5. PATH_RETIRE_CONNECTION_ID frames

The PATH_RETIRE_CONNECTION_ID frame (type=0x15228c0a) is an extension of the RETIRE_CONNECTION_ID frame specified in Section 19.16 of [QUIC-TRANSPORT]. It is used to indicate that it will no longer use a connection ID for a specific path that was issued by its peer. To retire the connection ID used during the handshake on the initial path, Path ID 0 is used. Sending a PATH_RETIRE_CONNECTION_ID frame also serves as a request to the peer to send additional connection IDs for this path (see also Section 5.1 of [QUIC-TRANSPORT]), unless the path specified by the Path ID has been abandoned. New path-specific connection IDs can be delivered to a peer using the PATH_NEW_CONNECTION_ID frame (see Section 7.4).¶

PATH_RETIRE_CONNECTION_ID frames are formatted as shown in Figure 8.¶

PATH_RETIRE_CONNECTION_ID Frame {
  Type (i) = TBD-06 (experiments use 0x15228c0a),
  Path Identifier (i),
  Sequence Number (i),
}

Compared to the RETIRE_CONNECTION_ID frame specified in Section 19.16 of [QUIC-TRANSPORT], the following field is added:¶

Path Identifier:

The Path ID associated with the connection ID to retire.¶

Note that the RETIRE_CONNECTION_ID frame can only be used to retire connection IDs for the initial path with Path ID 0.¶

As the PATH_NEW_CONNECTION_ID frames applies the sequence number per path, the sequence number in the PATH_RETIRE_CONNECTION_ID frame is also per path. The PATH_RETIRE_CONNECTION_ID frame retires the Connection ID with the specified Path ID and sequence number.¶

The processing of an incoming RETIRE_CONNECTION_ID frame is described in Section 19.17 of [QUIC-TRANSPORT]. The same processing applies for PATH_RETIRE_CONNECTION_ID frames per path, while the processing of a RETIRE_CONNECTION_ID frame is only applied for Path ID 0.¶

7.6. MAX_PATH_ID frames

A MAX_PATH_ID frame (type=0x15228c0c) informs the peer of the maximum path identifier it is permitted to use.¶

MAX_PATH_ID frames are formatted as shown in Figure 9.¶

MAX_PATH_ID Frame {
  Type (i) = TBD-07 (experiments use 0x15228c0c),
  Maximum Path Identifier (i),
}

MAX_PATH_ID frames contain the following field:¶

Maximum Path Identifier:

The maximum path identifier that the sending endpoint is willing to accept. This value MUST NOT exceed 2^32-1, the maximum allowed value for the Path ID due to restrictions on the nonce calculation (see Section 4). The Maximum Path Identifier value MUST NOT be lower than the value advertised in the initial_max_path_id transport parameter.¶

Receipt of an invalid Maximum Path Identifier value MUST be treated as a connection error of type PROTOCOL_VIOLATION.¶

Loss or reordering can cause an endpoint to receive a MAX_PATH_ID frame with a smaller Maximum Path Identifier value than was previously received. MAX_PATH_ID frames that do not increase the path limit MUST be ignored.¶

7.7. PATHS_BLOCKED and PATH_CIDS_BLOCKED frames

A sender can send a PATHS_BLOCKED frame (type=0x15228c0d) when it wishes to open a path but is unable to do so due to the maximum path identifier limit set by its peer.¶

A sender can send a PATH_CIDS_BLOCKED frame (type=0x15228c0e) when it wishes to open a path with a valid Path ID or change the CID on an established path but is unable to do so because there are no unused connection IDs available for the corresponding Path ID.¶

Note that PATHS_BLOCKED and PATH_CIDS_BLOCKED frames are informational. Sending a PATHS_BLOCKED or a PATH_CIDS_BLOCKED frame does not imply a particular action from the peer like updating the new Max Path ID value, but informs the peer that the maximum path identifier limit or the absence of unused connection IDs prevented the creation or the usage of paths.¶

PATHS_BLOCKED frames are formatted as shown in Figure 10.¶

PATHS_BLOCKED Frame {
  Type (i) = TBD-08 (experiments use 0x15228c0d),
  Maximum Path Identifier (i),
}

PATHS_BLOCKED frames contain the following field:¶

Maximum Path Identifier:

A variable-length integer indicating the maximum path identifier that was allowed at the time the frame was sent. If the received value is lower than the currently allowed maximum value, this frame can be ignored.¶

PATH_CIDS_BLOCKED frames are formatted as shown in Figure 11.¶

PATH_CIDS_BLOCKED Frame {
  Type (i) = TBD-09 (experiments use 0x15228c0e),
  Path Identifier (i),
}

Path Identifier:

Identifier of the path for which unused connection IDs are not available.¶

Receipt of a value of Maximum Path Identifier or Path Identifier that is higher than the local maximum value MUST be treated as a connection error of type PROTOCOL_VIOLATION.¶

10.1. Memory Allocation for Per-Path Resources

The initial_max_path_id transport parameter and the Max Path ID field in the MAX_PATH_ID frame limit the number of paths an endpoint is willing to maintain and accordingly limit the associated path resources.¶

Furthermore, as connection IDs have to be issued by both endpoints for the same path ID before an endpoint can open a path, each endpoint can further control the per-path resource usage (beyond the connection IDs) by limiting the number of Path ID that it issues connection IDs for.¶

Therefore, to avoid unnecessarily resource usage, that potentially could be exploited in a resource exhaustion attack, endpoints should allocate those additional path resource, such as e.g. for packet number handling, only after path validation has successfully completed.¶

10.2. Request Forgery with Spoofed Address

The path validation mechanism as specified in Section 8.2. of [QUIC-TRANSPORT] for migration is used unchanged for initiation of new paths in this extension. Therefore, the security considerations on source address spoofing as outlined in Section 21.5.4 of [QUIC-TRANSPORT] equally apply. Similarly, the anti-amplification limits as specified in Section 8 of [QUIC-TRANSPORT] need to be followed to limit the amplification risk.¶

However, while [QUIC-TRANSPORT] only allows the use of one path simultaneously and therefore only one path migration at the time should be validated, this extension allows for multiple open paths, that could in theory be migrated all at the same time, and it allows for multiple paths that could be initialized simultaneously. Therefore, each path could be used to further amplify an attack. Endpoints needs limit the number of maximum paths and might consider additional measures to limit the number of concurrent path validation processes e.g. by pacing them out or limiting the number of path initiation attempts over a certain time period.¶

10.3. Use of Transport Layer Security and the AEAD Encryption Nonce

The multipath extension as specified in this document is only enabled after a successful handshake when both endpoints indicate support for this extension. Respectively, all new frames defined in this extension are only used in 1-RTT packets. As the handshake is not changed by this extension, the transport security mechanisms as specified in [QUIC-TLS], such as encryption key exchange and peer authentication, remain unchanged as well and the respective security considerations in [QUIC-TLS] applied unaltered. Note that with the use of this extension, multiple nonces can be in use simultaneously for the same AEAD key.¶

Further note, that the limits as discussed on Appendix B of [QUIC-TLS] apply to the total number of packets sent on all paths.¶

This specification changes the AEAD calculation by using the path identifier as part of AEAD encryption nonce (see Section 4). To ensure a unique nonce, path identifiers are limited to 32 bits and cannot be reused for another path in the same connection.¶

13.1. Normative References

[QUIC-RECOVERY]

Iyengar, J., Ed. and I. Swett, Ed., "QUIC Loss Detection and Congestion Control", RFC 9002, DOI 10.17487/RFC9002, May 2021, <https://www.rfc-editor.org/rfc/rfc9002>.

[QUIC-TLS]

Thomson, M., Ed. and S. Turner, Ed., "Using TLS to Secure QUIC", RFC 9001, DOI 10.17487/RFC9001, May 2021, <https://www.rfc-editor.org/rfc/rfc9001>.

[QUIC-TRANSPORT]

Iyengar, J., Ed. and M. Thomson, Ed., "QUIC: A UDP-Based Multiplexed and Secure Transport", RFC 9000, DOI 10.17487/RFC9000, May 2021, <https://www.rfc-editor.org/rfc/rfc9000>.

[RFC2119]

Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, <https://www.rfc-editor.org/rfc/rfc2119>.

[RFC8174]

Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, <https://www.rfc-editor.org/rfc/rfc8174>.

13.2. Informative References

[OLIA]

Khalili, R., Gast, N., Popovic, M., Upadhyay, U., and J. Le Boudec, "MPTCP is not pareto-optimal: performance issues and a possible solution", Proceedings of the 8th international conference on Emerging networking experiments and technologies, ACM , 2012.

[RFC2475]

Blake, S., Black, D., Carlson, M., Davies, E., Wang, Z., and W. Weiss, "An Architecture for Differentiated Services", RFC 2475, DOI 10.17487/RFC2475, December 1998, <https://www.rfc-editor.org/rfc/rfc2475>.

[RFC6356]

Raiciu, C., Handley, M., and D. Wischik, "Coupled Congestion Control for Multipath Transport Protocols", RFC 6356, DOI 10.17487/RFC6356, October 2011, <https://www.rfc-editor.org/rfc/rfc6356>.