Re: [PATCH 0/4] Some improvements to the tap send path

On 3/8/24 07:53, David Gibson wrote:
...
  This series has a handful of small improvements to the
tap send path.
 See individual commit messages for the details.
 
 I expect this will conflict with Laurent's upcoming work.  I hope the
 conflicts won't be too bad, and indeed will set us up for less
 duplication there in the end. 
I'm working on patch that devides TCP buffers in several buffers pointed out by an
IOV 
arrays and then provided to tap_send_frames(). I'm going to base my patch on this
series.

The idea is:

A frame is made with 4 iovecs:

#define TCP_IOV_VNET    0
#define TCP_IOV_ETH     1
#define TCP_IOV_IP      2
#define TCP_IOV_PAYLOAD 3
#define TCP_IOV_NUM     4
typedef struct iovec tap_iovec_t[TCP_IOV_NUM];

The payload can be TCP + data or TCP + flags:

struct tcp_l2_flags_t {
         struct tcphdr th;
         char opts[OPT_MSS_LEN + OPT_WS_LEN + 1];
};

struct tcp_l2_payload_t {
         struct tcphdr th;       /*    20 bytes */
         uint8_t data[MSS];      /* 65516 bytes */
#ifdef __AVX2__
} __attribute__ ((packed, aligned(32)));
#else
} __attribute__ ((packed, aligned(__alignof__(unsigned int))));
#endif

static struct ethhdr            tcp4_eth_src;

static struct iphdr             tcp4_l2_ip[TCP_FRAMES_MEM];
static struct tcp_l2_payload_t  tcp4_l2_payload[TCP_FRAMES_MEM];

static struct tcp_buf_seq_update tcp4_l2_buf_seq_update[TCP_FRAMES_MEM];
static unsigned int tcp4_l2_buf_used;

static tap_iovec_t      tcp4_l2_iov             [TCP_FRAMES_MEM];

Initialization looks like:

	tcp4_eth_src.h_proto = htons_constant(ETH_P_IP);
	for (i = 0; i < TCP_FRAMES_MEM; i++) {
		...
                 /* headers */
                 tcp4_l2_ip[i] = iph;
                 tcp4_l2_payload[i].th = (struct tcphdr){
                                         .doff = sizeof(struct tcphdr) / 4,
                                         .ack = 1
                                 };
		...
		/* iovecs */
		iov = tcp4_l2_iov[i];
                 iov[TCP_IOV_ETH].iov_base = &tcp4_eth_src;
                 iov[TCP_IOV_ETH].iov_len = sizeof(struct ethhdr);
                 iov[TCP_IOV_IP].iov_base = &tcp4_l2_ip[i];
                 iov[TCP_IOV_IP].iov_len = sizeof(struct iphdr);
                 iov[TCP_IOV_PAYLOAD].iov_base = &tcp4_l2_payload[i];
		...
	}

Then to fill the payload header (data are received by tcp_data_from_sock()):

	iov = tcp4_l2_iov[tcp4_l2_buf_used++];
	iov[TCP_IOV_PAYLOAD].iov_len = tcp_l2_buf_fill_headers(c, conn,
                                                        iov, plen, check, seq);

And the frame is sent using:

	m = tap_send_iov(c, tcp4_l2_iov, tcp4_l2_buf_used);

For the moment (I'll rebase on your series), tap_send_iov_passt() looks like:

static size_t tap_send_iov_passt(const struct ctx *c, tap_iovec_t *tap_iov,
                                  size_t n)
{
         unsigned int i;

         for (i = 0; i < n; i++) {
                 uint32_t vnet_len;
                 int j;

                 vnet_len = 0;
                 for (j = TCP_IOV_ETH; j < TCP_IOV_NUM; j++)
                         vnet_len += tap_iov[i][j].iov_len;

                 tap_iov[i][TCP_IOV_VNET].iov_base = &vnet_len;
                 tap_iov[i][TCP_IOV_VNET].iov_len = sizeof(vnet_len);

                 if (!tap_send_frames_passt(c, tap_iov[i], TCP_IOV_NUM))
                         break;
         }

         return i;

}

Thanks,
Laurent
...
  
 This is based on Laurent's patch fixing pcap_multiple() not to capture
 frames we failed to send.
 
 David Gibson (4):
    tap: Extend tap_send_frames() to allow multi-buffer frames
    tap: Simplify some casts in the tap "slow path" functions
    tap: Implement tap_send() "slow path" in terms of fast path
    tap: Rename tap_iov_{base,len}
 
   arp.c |   4 +-
   tap.c | 158 +++++++++++++++++++++++++++++++---------------------------
   tap.h |  19 +++----
   tcp.c |  20 ++++----
   udp.c |  10 ++--
   5 files changed, 111 insertions(+), 100 deletions(-)