123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172737475767778798081828384858687888990919293949596979899100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135136137138139140141142143144145146147148149150151152153154155156157158159 |
- =================================================================
- Intel Omni-Path (OPA) Virtual Network Interface Controller (VNIC)
- =================================================================
- Intel Omni-Path (OPA) Virtual Network Interface Controller (VNIC) feature
- supports Ethernet functionality over Omni-Path fabric by encapsulating
- the Ethernet packets between HFI nodes.
- Architecture
- =============
- The patterns of exchanges of Omni-Path encapsulated Ethernet packets
- involves one or more virtual Ethernet switches overlaid on the Omni-Path
- fabric topology. A subset of HFI nodes on the Omni-Path fabric are
- permitted to exchange encapsulated Ethernet packets across a particular
- virtual Ethernet switch. The virtual Ethernet switches are logical
- abstractions achieved by configuring the HFI nodes on the fabric for
- header generation and processing. In the simplest configuration all HFI
- nodes across the fabric exchange encapsulated Ethernet packets over a
- single virtual Ethernet switch. A virtual Ethernet switch, is effectively
- an independent Ethernet network. The configuration is performed by an
- Ethernet Manager (EM) which is part of the trusted Fabric Manager (FM)
- application. HFI nodes can have multiple VNICs each connected to a
- different virtual Ethernet switch. The below diagram presents a case
- of two virtual Ethernet switches with two HFI nodes::
- +-------------------+
- | Subnet/ |
- | Ethernet |
- | Manager |
- +-------------------+
- / /
- / /
- / /
- / /
- +-----------------------------+ +------------------------------+
- | Virtual Ethernet Switch | | Virtual Ethernet Switch |
- | +---------+ +---------+ | | +---------+ +---------+ |
- | | VPORT | | VPORT | | | | VPORT | | VPORT | |
- +--+---------+----+---------+-+ +-+---------+----+---------+---+
- | \ / |
- | \ / |
- | \/ |
- | / \ |
- | / \ |
- +-----------+------------+ +-----------+------------+
- | VNIC | VNIC | | VNIC | VNIC |
- +-----------+------------+ +-----------+------------+
- | HFI | | HFI |
- +------------------------+ +------------------------+
- The Omni-Path encapsulated Ethernet packet format is as described below.
- ==================== ================================
- Bits Field
- ==================== ================================
- Quad Word 0:
- 0-19 SLID (lower 20 bits)
- 20-30 Length (in Quad Words)
- 31 BECN bit
- 32-51 DLID (lower 20 bits)
- 52-56 SC (Service Class)
- 57-59 RC (Routing Control)
- 60 FECN bit
- 61-62 L2 (=10, 16B format)
- 63 LT (=1, Link Transfer Head Flit)
- Quad Word 1:
- 0-7 L4 type (=0x78 ETHERNET)
- 8-11 SLID[23:20]
- 12-15 DLID[23:20]
- 16-31 PKEY
- 32-47 Entropy
- 48-63 Reserved
- Quad Word 2:
- 0-15 Reserved
- 16-31 L4 header
- 32-63 Ethernet Packet
- Quad Words 3 to N-1:
- 0-63 Ethernet packet (pad extended)
- Quad Word N (last):
- 0-23 Ethernet packet (pad extended)
- 24-55 ICRC
- 56-61 Tail
- 62-63 LT (=01, Link Transfer Tail Flit)
- ==================== ================================
- Ethernet packet is padded on the transmit side to ensure that the VNIC OPA
- packet is quad word aligned. The 'Tail' field contains the number of bytes
- padded. On the receive side the 'Tail' field is read and the padding is
- removed (along with ICRC, Tail and OPA header) before passing packet up
- the network stack.
- The L4 header field contains the virtual Ethernet switch id the VNIC port
- belongs to. On the receive side, this field is used to de-multiplex the
- received VNIC packets to different VNIC ports.
- Driver Design
- ==============
- Intel OPA VNIC software design is presented in the below diagram.
- OPA VNIC functionality has a HW dependent component and a HW
- independent component.
- The support has been added for IB device to allocate and free the RDMA
- netdev devices. The RDMA netdev supports interfacing with the network
- stack thus creating standard network interfaces. OPA_VNIC is an RDMA
- netdev device type.
- The HW dependent VNIC functionality is part of the HFI1 driver. It
- implements the verbs to allocate and free the OPA_VNIC RDMA netdev.
- It involves HW resource allocation/management for VNIC functionality.
- It interfaces with the network stack and implements the required
- net_device_ops functions. It expects Omni-Path encapsulated Ethernet
- packets in the transmit path and provides HW access to them. It strips
- the Omni-Path header from the received packets before passing them up
- the network stack. It also implements the RDMA netdev control operations.
- The OPA VNIC module implements the HW independent VNIC functionality.
- It consists of two parts. The VNIC Ethernet Management Agent (VEMA)
- registers itself with IB core as an IB client and interfaces with the
- IB MAD stack. It exchanges the management information with the Ethernet
- Manager (EM) and the VNIC netdev. The VNIC netdev part allocates and frees
- the OPA_VNIC RDMA netdev devices. It overrides the net_device_ops functions
- set by HW dependent VNIC driver where required to accommodate any control
- operation. It also handles the encapsulation of Ethernet packets with an
- Omni-Path header in the transmit path. For each VNIC interface, the
- information required for encapsulation is configured by the EM via VEMA MAD
- interface. It also passes any control information to the HW dependent driver
- by invoking the RDMA netdev control operations::
- +-------------------+ +----------------------+
- | | | Linux |
- | IB MAD | | Network |
- | | | Stack |
- +-------------------+ +----------------------+
- | | |
- | | |
- +----------------------------+ |
- | | |
- | OPA VNIC Module | |
- | (OPA VNIC RDMA Netdev | |
- | & EMA functions) | |
- | | |
- +----------------------------+ |
- | |
- | |
- +------------------+ |
- | IB core | |
- +------------------+ |
- | |
- | |
- +--------------------------------------------+
- | |
- | HFI1 Driver with VNIC support |
- | |
- +--------------------------------------------+
|