NAME

fi_arch - libfabric architecture

OVERVIEW

Libfabric APIs define application facing communication semantics without mandating the underlying implementation or wire protocols. It is architected such that applications can have direct access to network hardware without operating system intervention, but does not mandate such an implementation. The APIs have been defined specifically to allow multiple implementations.

The following diagram highlights the general architecture of the interfaces exposed by libfabric.

                 Applications and Middleware
       [MPI]   [SHMEM]   [PGAS]   [Storage]   [Other]

--------------------- libfabric API ---------------------

/  Core  \ + /Communication\ + /  Data  \ + <Completion>
\Services/   \    Setup    /   \Transfer/

----------------- libfabric Provider API ----------------

                    libfabric providers
   [TCP]   [UDP]   [Verbs]    [EFA]    [SHM]   [Other]

---------------------------------------------------------

     Low-level network hardware and software interfaces

Details of each libfabric component is described below.

Core Services

libfabric can be divided between the libfabric core and providers. The core defines defines the APIs that applications use and implements what is referred to as discovery services. Discovery services are responsible for identifying what hardware is available in the system, platform features, operating system features, associated communication and computational libraries, and so forth. Providers are optimized implementations of the libfabric API. One of the goals of the libfabric core is to link upper level applications and middleware with specific providers best suited for their needs.

From the viewpoint of an application, the core libfabric services are accessed primarily by the fi_getinfo() API. See fi_getinfo(3).

Providers

Unlike many libraries, the libfabric core does not implement most of the APIs called by its users. Instead, that responsibility instead falls to what libfabric calls providers. The bulk of the libfabric API is implemented by each provider. When an application calls a libfabric API, that function call is routed directly into a specific provider. This is done using function pointers associated with specific libfabric defined objects. The object model is describe in more detail below.

The benefit of this approach is that each provider can optimize the libfabric defined communication semantics according to their available network hardware, operating system, platform, and network protocol features.

In general, each provider focuses on supporting the libfabric API over a specific lower-level communication API or NIC. See fi_provider(7) for a discussion on the different types of providers available and the provider architecture.

Communication Setup

At a high-level, communication via libfabric may be either connection- oriented or connectionless. This is similar to choosing between using TCP or UDP sockets, though libfabric supports reliable-connectionless communication semantics. Communication between two processes occurs over a construct known as an endpoint. Conceptually, an endpoint is equivalent to a socket in the socket API world.

Specific APIs and libfabric objects are designed to manage and setup communication between nodes. It includes calls for connection management (CM), as well as functionality used to address connection-less endpoints.

The CM APIs are modeled after APIs used to connect TCP sockets: connect(), bind(), listen(), and accept(). A main difference is that libfabric calls are designed to always operate asynchronously. CM APIs are discussed in fi_cm(3).

For performance and scalability reasons discussed in the fi_intro(7) page, connection-less endpoints use a unique model to setup communication. These are based on a concept referred to as address vectors, where the term vector means table or array. Address vectors are discussed in detail later, but target applications needing to talk with potentially thousands to millions of peers.

Data Transfer Services

libfabric provides several data transfer semantics to meet different application requirements. There are five basic sets of data transfer APIs: messages, tagged messages, RMA, atomics, and collectives.

Messages: Message APIs expose the ability to send and receive data with message boundaries being maintained. Message transfers act as FIFOs, with sent messages matched with receive buffers in the order that messages are received at the target. The message APIs are modeled after socket APIs, such as send(). sendto(), sendmsg(), recv(), recvmsg(), etc. For more information see fi_msg(3).
Tagged Messages: Tagged messages are similar to messages APIs, with the exception of how messages are matched at the receiver. Tagged messages maintain message boundaries, same as the message API. The tag matching APIs differ from the message APIs in that received messages are directed into buffers based on small steering tags that are specified and carried in the sent message. All message buffers, posted to send or receive data, are associated with a tag value. Sent messages are matched with buffers at the receiver that have the same tag. For more information, see fi_tagged(3).
RMA: RMA stands for remote memory access. RMA transfers allow an application to write data directly into a specific memory location in a target process or to read memory from a specific address at the target process and return the data into a local buffer. RMA is also known as RDMA (remote direct memory access); however, RDMA originally defined a specific transport implementation of RMA. For more information, see fi_rma(3).
Atomics: Atomic operations add arithmetic operations to RMA transfers. Atomics permit direct access and manipulation of memory on the target process. libfabric defines a wide range of arithmetic operations that may be invoked as part of a data transfer operation. For more information, see fi_atomic(3).
Collectives: The above data transfer APIs perform point-to-point communication. Data transfers occur between exactly one initiator and one target. Collective operations are coordinated atomic operations among an arbitrarily large number of peers. For more information, see fi_collective(3).

Memory Registration

One of the objective of libfabric is to allow network hardware direct access to application data buffers. This is accomplished through an operation known as memory registration.

In order for a NIC to read or write application memory directly, it must access the physical memory pages that back the application’s address space. Modern operating systems employ page files that swap out virtual pages from one process with the virtual pages from another. As a result, a physical memory page may map to different virtual addresses depending on when it is accessed. Furthermore, when a virtual page is swapped in, it may be mapped to a new physical page. If a NIC attempts to read or write application memory without being linked into the virtual address manager, it could access the wrong data, possibly corrupting an application’s memory. Memory registration can be used to avoid this situation from occurring. For example, registered pages can be marked such that the operating system locks the virtual to physical mapping, avoiding any possibility of the virtual page being paged out or remapped.

Memory registration is also the security mechanism used to grant a remote peer access to local memory buffers. Registered memory regions associate memory buffers with permissions granted for access by fabric resources. A memory buffer must be registered before it can be used as the target of an RMA or atomic data transfer. Memory registration provides a simple protection mechanism. (Advanced scalable networks employ other mechanisms, which are considered out of scope for the purposes of this discussion.) After a memory buffer has been registered, that registration request (buffer’s address, buffer length, and access permission) is given a registration key. Peers that issue RMA or atomic operations against that memory buffer must provide this key as part of their operation. This helps protects against unintentional accesses to the region.

Completion Services

libfabric data transfers operate asynchronously. Completion services are used to report the results of submitted data transfer operations. Completions may be reported using the cleverly named completions queues, which provide details about the operation that completed. Or, completions may be reported using completion counters that simply return the number of operations that have completed.

Completion services are designed with high-performance, low-latency in mind. The calls map directly into the providers, and data structures are defined to minimize memory writes and cache impact. Completion services do not have corresponding socket APIs. However, for Windows developers, they are similar to IO completion ports.

Object Model

libfabric follows an object-oriented design model. Although the interfaces are written in C, the structures and implementation have a C++ feel to them. The following diagram shows a high-level view of notable libfabric objects and object dependencies.

/ Passive \ ---> <Fabric> <--- /Event\
\Endpoints/         ^          \Queue/
                    |
  /Address\ ---> <Domain> <--- /Completion\
  \Vector /       ^  ^         \  Queue   /
                  |  |
      /Memory\ ---    --- / Active \
      \Region/            \Endpoint/

Fabric: A fabric represents a collection of hardware and software resources that access a single physical or virtual network. For example, a fabric may be a single network subnet or cluster. All network ports on a system that can communicate with each other through the fabric belong to the same fabric. A fabric shares network addresses and can span multiple providers. Fabrics are the top level object from which other objects are allocated.
Domain: A domain represents a logical connection into a fabric. In the simplest case, a domain may correspond to a physical or virtual NIC; however a domain could include multiple NICs (in the case of a multi-rail provider), or no NIC at all (in the case of shared memory). A domain defines the boundary within which other resources may be associated. Active endpoints and completion queues must be part of the same domain in order to be related to each other.
Passive Endpoint: Passive endpoints are used by connection-oriented protocols to listen for incoming connection requests. Passive endpoints often map to software constructs and may span multiple domains. They are best represented by a listening socket.
Event Queues: Event queues (EQs) are used to collect and report the completion of asynchronous operations and events. Event queues handle control events, that is, operations which are not directly associated with data transfer operations. The reason for separating control events from data transfer events is for performance reasons. Event queues are often implemented entirely in software using operating system constructs. Control events usually occur during an application’s initialization phase, or at a rate that’s several orders of magnitude smaller than data transfer events. Event queues are most commonly used by connection-oriented protocols for notification of connection request or established events.
Active Endpoint: Active endpoints are data transfer communication portals. They are conceptually similar to a TCP or UDP socket. Active endpoints are used to perform data transfers. Active endpoints implement the network protocol.
Completion Queue: Completion queues (CQs) are high-performance queues used to report the completion of data transfer operations. Unlike event queues, completion queues are often fully or partially implemented in hardware. Completion queue interfaces are designed to minimize software overhead.
Memory Region: Memory regions describe application’s local memory buffers. In order for fabric resources to access application memory, the application must first grant permission to the fabric provider by constructing a memory region. Memory regions are required for specific types of data transfer operations, such as RMA and atomic operations.
Address Vectors: Address vectors are used by connection-less endpoints. They map higher level addresses, such as IP addresses or hostnames, which may be more natural for an application to use, into fabric specific addresses. The use of address vectors allows providers to reduce the amount of memory required to maintain large address look-up tables, and eliminate expensive address resolution and look-up methods during data transfer operations.

Communication Model

Endpoints represent communication portals, and all data transfer operations are initiated on endpoints. libfabric defines the conceptual model for how endpoints are exposed to applications. It supports three main communication endpoint types. The endpoint names are borrowed from socket API naming.

FI_EP_MSG: Reliable-connected
FI_EP_DGRAM: Unreliable datagram
FI_EP_RDM: Reliable-unconnected

Communication setup is based on whether the endpoint is connected or unconnected. Reliability is a feature of the endpoint’s data transfer protocol.

Connected Communications

The following diagram highlights the general usage behind connection-oriented communication. Connected communication is based on the flow used to connect TCP sockets, with improved asynchronous support.

         1 listen()              2 connect()
             |                      |
         /Passive \  <---(3)--- / Active \
         \Endpoint/             \Endpoint/
         /                               \
        / (4 CONNREQ)                     \
/Event\                                     /Event\
\Queue/                                     \Queue/
                                           /
         5 accept()         (8 CONNECTED) /
             |                           /
         / Active \  ------(6)--------->
         \Endpoint/  <-----(7)----------
         /
        / (9 CONNECTED)
/Event\
\Queue/

Connections require the use of both passive and active endpoints. In order to establish a connection, an application must first create a passive endpoint and associate it with an event queue. The event queue will be used to report the connection management events. The application then calls listen on the passive endpoint. A single passive endpoint can be used to form multiple connections.

The connecting peer allocates an active endpoint, which is also associated with an event queue. Connect is called on the active endpoint, which results in sending a connection request (CONNREQ) message to the passive endpoint. The CONNREQ event is inserted into the passive endpoint’s event queue, where the listening application can process it.

Upon processing the CONNREQ, the listening application will allocate an active endpoint to use with the connection. The active endpoint is bound with an event queue. Although the diagram shows the use of a separate event queue, the active endpoint may use the same event queue as used by the passive endpoint. Accept is called on the active endpoint to finish forming the connection. It should be noted that the OFI accept call is different than the accept call used by sockets. The differences result from OFI supporting process direct I/O.

libfabric does not define the connection establishment protocol, but does support a traditional three-way handshake used by many technologies. After calling accept, a response is sent to the connecting active endpoint. That response generates a CONNECTED event on the remote event queue. If a three-way handshake is used, the remote endpoint will generate an acknowledgment message that will generate a CONNECTED event for the accepting endpoint. Regardless of the connection protocol, both the active and passive sides of the connection will receive a CONNECTED event that signals that the connection has been established.

Connectionless Communications

Connectionless communication allows data transfers between active endpoints without going through a connection setup process. The diagram below shows the basic components needed to setup connection-less communication. Connectionless communication setup differs from UDP sockets in that it requires that the remote addresses be stored with libfabric.

  1 insert_addr()              2 send()
         |                        |
     /Address\ <--3 lookup--> / Active \
     \Vector /                \Endpoint/

libfabric requires the addresses of peer endpoints be inserted into a local addressing table, or address vector, before data transfers can be initiated against the remote endpoint. Address vectors abstract fabric specific addressing requirements and avoid long queuing delays on data transfers when address resolution is needed. For example, IP addresses may need to be resolved into Ethernet MAC addresses. Address vectors allow this resolution to occur during application initialization time. libfabric does not define how an address vector be implemented, only its conceptual model.

All connection-less endpoints that transfer data must be associated with an address vector.

Endpoints

At a low-level, endpoints are usually associated with a transmit context, or queue, and a receive context, or queue. Although the terms transmit and receive queues are easier to understand, libfabric uses the terminology context, since queue like behavior of acting as a FIFO (first-in, first-out) is not guaranteed. Transmit and receive contexts may be implemented using hardware queues mapped directly into the process’s address space. An endpoint may be configured only to transmit or receive data. Data transfer requests are converted by the underlying provider into commands that are inserted into hardware transmit and/or receive contexts.

Endpoints are also associated with completion queues. Completion queues are used to report the completion of asynchronous data transfer operations.

Shared Contexts

An advanced usage model allows for sharing resources among multiple endpoints. The most common form of sharing is having multiple connected endpoints make use of a single receive context. This can reduce receive side buffering requirements, allowing the number of connected endpoints that an application can manage to scale to larger numbers.

Data Transfers

Obviously, a primary goal of network communication is to transfer data between processes running on different systems. In a similar way that the socket API defines different data transfer semantics for TCP versus UDP sockets, that is, streaming versus datagram messages, libfabric defines different types of data transfers. However, unlike sockets, libfabric allows different semantics over a single endpoint, even when communicating with the same peer.

libfabric uses separate API sets for the different data transfer semantics; although, there are strong similarities between the API sets. The differences are the result of the parameters needed to invoke each type of data transfer.

Message transfers

Message transfers are most similar to UDP datagram transfers, except that transfers may be sent and received reliably. Message transfers may also be gigabytes in size, depending on the provider implementation. The sender requests that data be transferred as a single transport operation to a peer. Even if the data is referenced using an I/O vector, it is treated as a single logical unit or message. The data is placed into a waiting receive buffer at the peer, with the receive buffer usually chosen using FIFO ordering. Note that even though receive buffers are selected using FIFO ordering, the received messages may complete out of order. This can occur as a result of data between and within messages taking different paths through the network, handling lost or retransmitted packets, etc.

Message transfers are usually invoked using API calls that contain the string “send” or “recv”. As a result they may be referred to simply as sent or received messages.

Message transfers involve the target process posting memory buffers to the receive (Rx) context of its endpoint. When a message arrives from the network, a receive buffer is removed from the Rx context, and the data is copied from the network into the receive buffer. Messages are matched with posted receives in the order that they are received. Note that this may differ from the order that messages are sent, depending on the transmit side’s ordering semantics.

Conceptually, on the transmit side, messages are posted to a transmit (Tx) context. The network processes messages from the Tx context, packetizing the data into outbound messages. Although many implementations process the Tx context in order (i.e. the Tx context is a true queue), ordering guarantees specified through the libfabric API determine the actual processing order. As a general rule, the more relaxed an application is on its message and data ordering, the more optimizations the networking software and hardware can leverage, providing better performance.

Tagged messages

Tagged messages are similar to message transfers except that the messages carry one additional piece of information, a message tag. Tags are application defined values that are part of the message transfer protocol and are used to route packets at the receiver. At a high level, they are roughly similar to message ids. The difference is that tag values are set by the application, may be any value, and duplicate tag values are allowed.

Each sent message carries a single tag value, which is used to select a receive buffer into which the data is copied. On the receiving side, message buffers are also marked with a tag. Messages that arrive from the network search through the posted receive messages until a matching tag is found.

Tags are often used to identify virtual communication groups or roles. In practice, message tags are typically divided into fields. For example, the upper 16 bits of the tag may indicate a virtual group, with the lower 16 bits identifying the message purpose. The tag message interface in libfabric is designed around this usage model. Each sent message carries exactly one tag value, specified through the API. At the receiver, buffers are associated with both a tag value and a mask. The mask is used as part of the buffer matching process. The mask is applied against the received tag value carried in the sent message prior to checking the tag against the receive buffer. For example, the mask may indicate to ignore the lower 16-bits of a tag. If the resulting values match, then the tags are said to match. The received data is then placed into the matched buffer.

For performance reasons, the mask is specified as ‘ignore’ bits. Although this is backwards from how many developers think of a mask (where the bits that are valid would be set to 1), the definition ends up mapping well with applications. The actual operation performed when matching tags is:

send_tag | ignore == recv_tag | ignore

/* this is equivalent to:
 * send_tag & ~ignore == recv_tag & ~ignore
 */

Tagged messages are equivalent of message transfers if a single tag value is used. But tagged messages require that the receiver perform a matching operation at the target, which can impact performance versus untagged messages.

RMA

RMA operations are architected such that they can require no processing by the CPU at the RMA target. NICs which offload transport functionality can perform RMA operations without impacting host processing. RMA write operations transmit data from the initiator to the target. The memory location where the data should be written is carried within the transport message itself, with verification checks at the target to prevent invalid access.

RMA read operations fetch data from the target system and transfer it back to the initiator of the request, where it is placed into memory. This too can be done without involving the host processor at the target system when the NIC supports transport offloading.

The advantage of RMA operations is that they decouple the processing of the peers. Data can be placed or fetched whenever the initiator is ready without necessarily impacting the peer process.

Because RMA operations allow a peer to directly access the memory of a process, additional protection mechanisms are used to prevent unintentional or unwanted access. RMA memory that is updated by a write operation or is fetched by a read operation must be registered for access with the correct permissions specified.

Atomic operations

Atomic transfers are used to read and update data located in remote memory regions in an atomic fashion. Conceptually, they are similar to local atomic operations of a similar nature (e.g. atomic increment, compare and swap, etc.). The benefit of atomic operations is they enable offloading basic arithmetic capabilities onto a NIC. Unlike other data transfer operations, which merely need to transfer bytes of data, atomics require knowledge of the format of the data being accessed.

A single atomic function operates across an array of data, applying an atomic operation to each entry. The atomicity of an operation is limited to a single data type or entry, however, not across the entire array. libfabric defines a wide variety of atomic operations across all common data types. However support for a given operation is dependent on the provider implementation.

Collective operations

In general, collective operations can be thought of as coordinated atomic operations between a set of peer endpoints, almost like a multicast atomic request. A single collective operation can result in data being collected from multiple peers, combined using a set of atomic primitives, and the results distributed to all peers. A collective operation is a group communication exchange. It involves multiple peers exchanging data with other peers participating in the collective call. Collective operations require close coordination by all participating members, and collective calls can strain the fabric, as well as local and remote data buffers.

Collective operations are an area of heavy research, with dedicated libraries focused almost exclusively on implementing collective operations efficiently. Such libraries are a specific target of libfabric. The main object of the libfabric collection APIs is to expose network acceleration features for implementing collectives to higher-level libraries and applications. It is recommended that applications needing collective communication target higher-level libraries, such as MPI, instead of using libfabric collective APIs for that purpose.

fi_arch(7) Libfabric Programmer's Guide - Architecture