Use to exchange data with a remote peer. The IND2Adapter::CreateQueuePair method returns this interface.
The IND2QueuePair interface inherits the methods of the IUnknown interface. In addition, IND2QueuePair defines the following methods.
- Flush - Cancels all outstanding requests in the inbound and outbound completion queues.
- Send - Sends data to a remote peer.
- Receive - Receives data from a remote peer.
- Bind - Binds a memory window to a buffer that is within the registered memory.
- Invalidate - Invalidates a local memory window.
- Read - Initiates an RDMA Read request.
- Write - Initiates an RDMA Write request.
Remarks:
If you do not retrieve the outstanding requests from the completion queue before releasing your last reference to this queue pair, you may get back requests from the completion queue that were issued on a now-closed queue pair.
Cancels all outstanding requests in the Receive and Initiator queues.
HRESULT Flush();
Return Value
When you implement this method, you should return the following return values. If you return others, try to use well-known values to aid in debugging issues.
- ND_SUCCESS - The operation succeeded. Note that all pending requests may not have completed when this call returns.
Remarks:
The status for flushed requests is set to ND_CANCELED.
Note that the IND2Connector::Disconnect method implicitly flushes all outstanding requests. The Flush method gives the client the opportunity to stop I/O processing before disconnecting (if needed).
If the completion queues are used by more than one queue pair, only this queue pair's requests are canceled.
Represents local memory segments that are used in Send, Receive, Read, and Write requests.
typedef struct _ND2_SGE {
void *Buffer;
ULONG BufferLength;
UINT32 MemoryRegionToken;
} ND2_SGE;
Members:
-
Buffer
The address of the buffer. The buffer is located within the memory region indicated by MemoryRegionToken. When using Send or Write requests with the ND_OP_FLAG_INLINE flag, the memory can be defined inside or outside a memory region.
-
BufferLength
The length, in bytes, of the buffer.
-
MemoryRegionToken
Opaque token value that is returned by a call to IND2MemoryRegion::GetLocalToken. The token value controls hardware access to registered memory because the provider associated the token to the memory region when memory was registered.
Sends data to a remote peer.
HRESULT Send(
[in] VOID *requestContext,
[in] const ND2_SGE sge[],
[in] ULONG nSge,
[in] ULONG flags
);
Parameters:
-
requestContext [in]
A context value to associate with the request, returned in the RequestContext member of the ND2_RESULT structure that is returned when the request completes.
-
sge [in]
An array of ND2_SGE structures that describe the buffers that contain the data to send to the peer. May be nullptr if nSge is zero, resulting in a zero-byte Send request and a corresponding zero-byte Receive request on the remote peer. The SGE array can be stack-based, and it is only used in the context of this call.
-
nSge [in]
The number of entries in the sge array. May be zero.
-
flags [in]
The following flags control the behavior of the Send request. You can specify one or more of the following flags:
-
ND_OP_FLAG_SILENT_SUCCESS
The successful completion of the request does not generate a completion on the outbound completion queue. Requests that fail will generate a completion.
-
ND_OP_FLAG_READ_FENCE
All prior Read requests must be complete before the hardware begins processing the Send request.
-
ND_OP_FLAG_SEND_AND_SOLICIT_EVENT
You can use this flag if you issue multiple, related Send requests. Set this flag only on the last, related Send request. The peer will receive all the Send requests as normal—the only difference is that when they receive the last Send request (the request with the ND_OP_FLAG_SEND_AND_SOLICIT_EVENT flag set), the completion queue for the peer generates a notification. The notification is generated after the Receive request completes.
This flag has no meaning to the receiver (peer) unless it has previously called the IND2CompletionQueue::Notify method with the notification type set to ND_CQ_NOITFY_SOLICITED. Note that errors always satisfy a Notify request.
-
ND_OP_FLAG_INLINE
Indicates that the memory referenced by the scatter/gather list should be transferred inline, and the MemoryRegionToken in the ND2_SGE entries may be invalid. Inline requests do not need to limit the number of entries in the scatter/gather list to the maxInitiatorRequestSge that is specified when the queue pair was created. The amount of memory transferred inline must be within the inline data limits of the queue pair. Please see IND2Adapter::Query method and MaxInlineDataSize field of ND2_ADAPTER_INFO.
-
Return Value:
When you implement this method, you should return the following return values. If you return others, try to use well-known values to aid in debugging issues.
- ND_SUCCESS - The operation succeeded. Completion status will be returned through the outbound completion queue that is associated with the queue pair.
- ND_CONNECTION_INVALID - The queue pair is not connected.
- ND_BUFFER_OVERFLOW - The send request referenced more data than is supported by the underlying hardware.
- ND_NO_MORE_ENTRIES - The request would have exceeded the number of initiator requests allowed on the queue pair. The initiatorQueueDepth parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
- ND_DATA_OVERRUN - The number of scatter/gather entries in the scatter/gather list of the request exceeded the number allowed on the queue pair. The maxInitiatorRequestSge parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
- ND_INVALID_PARAMETER_4 - Invalid flags.
Remarks:
If the method fails, the queue pair can still process existing or future requests. However, if the IND2CompletionQueue::GetResults method returns an ND2_RESULT with an error status, the connection is terminated.
Receives data from a peer.
HRESULT Receive(
[in] VOID *requestContext,
[in] ND2_SGE sge[],
[in] ULONG nSge
);
Parameters:
-
requestContext [in]
A context value to associate with the request, returned in the RequestContext member of the ND2_RESULT structure that is returned when the request completes.
-
sge [in]
A pointer to an array of ND2_SGE structures that describe the buffers to which data sent by the peer is written. May be nullptr if nSge is zero, resulting in a zero-byte Receive request. The SGE array can be stack-based, and it is only used in the context of this call.
-
nSge [in]
The number of entries in the sge array. May be zero.
Return Value:
When you implement this method, you should return the following return values. If you return others, try to use well-known values to aid in debugging issues.
- ND_SUCCESS - The operation succeeded. Completion status will be returned through the outbound completion queue associated with the queue pair.
- ND_BUFFER_OVERFLOW - The request referenced more data than is supported by the underlying hardware.
- ND_NO_MORE_ENTRIES - The request would have exceeded the number of Receive requests allowed on this queue pair. The receiveQueueDepth parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
- ND_DATA_OVERRUN - The number of scatter/gather entries in the scatter/gather list exceeded the number allowed on the queue pair. The maxReceiveRequestSge parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
Remarks:
You can call this method at any time, the queue pair does not have to be connected.
You must post a Receive request before the peer posts a Send request. The buffers that you specify must be large enough to receive the sent data. If not, the Send request fails and the connection is terminated; the completion status for the Receive request is ND_BUFFER_OVERFLOW. If the Receive request fails, the queue pair can still process existing or future requests.
The protocol determines how many Receive requests you must post and the buffer size that is required for each post.
If the method fails, the queue pair can still process existing or future requests. However, if the IND2CompletionQueue::GetResults method returns an ND2_RESULT with an error status, the connection is terminated and any further requests are completed with ND_CANCELED status.
Binds a memory window to a buffer that is within the registered memory.
HRESULT Bind(
[in] VOID *requestContext,
[in] IUnknown *pMemoryRegion,
[in] IUnknown *pMemoryWindow,
[in] const VOID *pBuffer,
[in] SIZE_T cbBuffer,
[in] ULONG flags
);
Parameters:
-
requestContext [in]
A context value to associate with the request, returned in the RequestContext member of the ND2_RESULT structure that is returned when the request completes.
-
pMemoryRegion [in]
The memory region to which to bind the memory window. The IND2Adapter::CreateMemoryRegion method returns this interface.
-
pMemoryWindow [in]
The memory window to bind to the registered memory. The IND2Adapter::CreateMemoryWindow method returns this interface.
-
pBuffer [in]
A buffer within the registered memory to which the memory window is bound. This buffer must be entirely within the bounds of the specified registered memory.
-
cbBuffer [in]
The size, in bytes, of the memory window buffer.
-
flags [in]
The following flags control the operations that the remote peer can perform against the memory. You can specify one or more of the following flags (you must specify at least ND_OP_FLAG_ALLOW_READ or ND_OP_FLAG_ALLOW_WRITE):
-
ND_OP_FLAG_SILENT_SUCCESS
If the request completes successfully, do not generate an entry in the completion queue. Requests that fail will generate an entry in the completion queue.
-
ND_OP_FLAG_READ_FENCE
All prior Read requests must be complete before the hardware processes new requests.
-
ND_OP_FLAG_ALLOW_READ
A remote peer can perform Read operations against the memory window.
-
ND_OP_FLAG_ALLOW_WRITE
A remote peer can perform Write operations against the memory window.
-
Return Value:
When you implement this method, you should return the following return values. If you return others, try to use well-known values to aid in debugging issues.
- ND_SUCCESS - The operation succeeded. Completion status will be returned through the outbound completion queue associated with the queue pair.
- ND_CONNECTION_INVALID - The queue pair is not connected.
- ND_NO_MORE_ENTRIES - The request would have exceeded the number of outbound requests allowed on the queue pair. The initiatorQueueDepth parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
- ND_ACCESS_VIOLATION - The memory region does not allow the type of access requested for the memory window. ND_OP_FLAG_ALLOW_WRITE requires a memory region registered with ND_MR_FLAG_ALLOW_LOCAL_WRITE.
Remarks:
The remote peer can use the window for Read and Write operations after you send the window descriptor to the remote peer. Typically, you send the descriptor as part of a Send request.
The memory window can be bound to only one queue pair at a time. To unbind the window, you can call the IND2QueuePair::Invalidate method.
If the method fails, the queue pair can still process existing or future requests. However, if the IND2CompletionQueue::GetResults method returns an ND2_RESULT with an error status, the connection is terminated and any further requests are completed with ND_CANCELED status.
The token for accessing the remote memory can be retrieved through the IND2MemoryWindow::GetRemoteToken method as soon as this method returns, before the Bind operation completion is reported through the IND2CompletionQueue::GetResults method.
Invalidates a local memory window. Any in-progress data transfers that reference the memory window will fail.
HRESULT Invalidate(
[in] VOID *requestContext,
[in] IUnknown *pMemoryWindow,
[in] ULONG flags
);
Parameters:
-
requestContext [in]
A client-defined opaque value, returned in the RequestContext member of an ND2_RESULT structure that is filled in by IND2CompletionQueue::GetResults method.
-
pMemoryWindow [in]
An interface of the memory window to invalidate. Invalidating the window lets the window be bound again to a different region of registered memory.
-
flags [in]
The following flags control the behavior of the operation. You can specify one or both of the flags:
-
ND_OP_FLAG_SILENT_SUCCESS
The successful completion of the request does not generate a completion in the outbound completion queue. However, requests that fail will generate a completion in the completion queue.
-
ND_OP_FLAG_READ_FENCE
All prior Read requests must be complete before the hardware begins processing the request.
-
Return Value:
When you implement this method, you should return the following return values. If you return others, try to use well-known values to aid in debugging issues.
- ND_SUCCESS - The operation succeeded. Completion status will be returned through the outbound completion queue associated with this queue pair.
- ND_CONNECTION_INVALID - The queue pair is not connected.
- ND_NO_MORE_ENTRIES - The request would have exceeded the number of outbound requests allowed on the queue pair. The initiatorQueueDepth parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
Remarks:
A queue pair can call this method after receiving notification from a peer that it is done with the memory window. Invalidating a memory window prevents further access to the window. Further references to the memory window in an RDMA Read or Write would result in a fatal error on the queue pair and the connection will be terminated.
If the method fails, the queue pair can still process existing or future requests. However, if the IND2CompletionQueue::GetResults method returns an ND2_RESULT with an error status, the connection is terminated and any further requests are completed with ND_CANCELED status.
Initiates an RDMA Read request.
HRESULT Read(
[in] VOID *requestContext,
[in] ND2_SGE sge[],
[in] ULONG nSge,
[in] UINT64 remoteAddress,
[in] UINT32 remoteToken,
[in] ULONG flags
);
Parameters:
-
requestContext [in]
A client-defined opaque value, returned in the RequestContext member of an ND2_RESULT structure that is filled in by IND2CompletionQueue::GetResults method.
-
sge [in]
An array of ND2_SGE structures that describe the local buffers to which data that is read from the remote memory is written. The length of all the scatter/gather entry buffers in the list determines how much data is read from the remote memory. The length of all the buffers must not extend past the boundary of the remote memory.
May be nullptr if nSge is zero. You can use a zero byte RDMA Read request to detect if the target hardware is working. The SGE array can be stack-based, and it is only used in the context of this call.
-
nSge [in]
The number of entries in the scatter/gather list. May be zero.
-
remoteAddress [in]
The remote address on the remote peer, in host order, from which to read.
-
remoteToken [in]
Opaque token value provided by the remote peer granting read access to the remote memory specified by remoteAddress. Please see IND2MemoryRegion::GetRemoteToken.
-
flags [in]
The following flags control the behavior of the Read request. You can specify one or both of the flags:
-
ND_OP_FLAG_SILENT_SUCCESS
The successful completion of the request does not generate a completion in the outbound completion queue. However, requests that fail will generate a completion in the completion queue.
-
ND_OP_FLAG_READ_FENCE
All prior Read requests must be complete before the hardware begins processing this request.
-
Return Value:
When you implement this method, you should return the following return values. If you return others, try to use well-known values to aid in debugging issues.
- ND_SUCCESS - The operation succeeded. Completion status will be returned through the outbound completion queue associated with this queue pair.
- ND_CONNECTION_INVALID - The queue pair is not connected.
- ND_BUFFER_OVERFLOW - The request referenced more data than is supported by the underlying hardware.
- ND_NO_MORE_ENTRIES - The request exceeded the number of outbound requests allowed on this queue pair. The initiatorQueueDepth parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
- ND_DATA_OVERRUN - The number of scatter/gather entries in the scatter/gather list exceeded the number allowed on this queue pair. The maxInitiatorRequestSge parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
- ND_REMOTE_ERROR - The request tried to read beyond the size of the remote memory.
Remarks:
You can read the data in a single scatter/gather entry or in multiple entries. For example, you could use multiple entries to read message headers in one entry and the payload in the other.
If the method fails, the queue pair can still process existing or future requests. However, if the IND2CompletionQueue::GetResults method returns an ND2_RESULT with an error status, the connection is terminated and any further requests are completed with ND_CANCELED status.
Initiates a one-sided write operation.
HRESULT Write(
[in] VOID *requestContext,
[in] const ND2_SGE sge[],
[in] ULONG nSge,
[in] UINT64 remoteAddress,
[in] UINT32 remoteToken,
[in] ULONG flags
);
Parameters:
-
requestContext [in]
A client-defined opaque value, returned in the RequestContext member of an ND2_RESULT structure that is filled in by IND2CompletionQueue::GetResults method.
-
sge [in]
An array of ND2_SGE structures that describe the local buffers to write to the remote memory. May be nullptr if nSge is zero, resulting in a zero-byte write. The SGE array can be stack-based, and it is only used in the context of this call.
-
nSge [in]
The number of entries in the scatter/gather list. May be zero.
-
remoteAddress [in]
The remote address on the remote peer, in host order, to which to write.
-
remoteToken [in]
Opaque token value provided by the remote peer that is granting Write access to the remote memory specified by remoteAddress.
-
remoteAddress [in]
The remote address on the remote peer, in host order, to which to write.
-
remoteToken [in]
Opaque token value provided by the remote peer that is granting Write access to the remote memory specified by remoteAddress. Please see IND2MemoryRegion::GetRemoteToken.
-
flags [in]
The following flags control the behavior of the Write operation. You can specify one or both of the following flags:
-
ND_OP_FLAG_SILENT_SUCCESS
The successful completion of the Write request does not generate a completion on the associated completion queue. Work requests that fail processing always generate a work completion.
-
ND_OP_FLAG_READ_FENCE
All prior Read requests must be complete before the hardware begins processing the Write request.
-
ND_OP_FLAG_INLINE
Indicates that the memory referenced by the scatter/gather list should be transferred inline, and the MemoryRegionToken in the ND2_SGE entries may be invalid. Inline requests do not need to limit the number of entries in the scatter/gather list to the maxInitiatorRequestSge parameter specified when the queue pair was created. The amount of memory transferred inline must be within the inline data limits of the queue pair.
-
Return Value:
When you implement this method, you should return the following return values. If you return others, try to use well-known values to aid in debugging issues.
- ND_SUCCESS - The operation succeeded. Completion status will be returned through the outbound completion queue associated with the queue pair.
- ND_CONNECTION_INVALID - The queue pair is not connected.
- ND_BUFFER_OVERFLOW - The request referenced more data than is supported by the underlying hardware.
- ND_NO_MORE_ENTRIES - The request would have exceeded the number of outbound requests allowed on the queue pair. The initiatorQueueDepth parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
- ND_DATA_OVERRUN - The number of scatter/gather entries in the scatter/gather list of the request exceeded the number allowed on the queue pair. The maxInitiatorRequestSge parameter of the IND2Adapter::CreateQueuePair method specifies the limit.
- ND_INVALID_PARAMETER_6 - Invalid flags.
Remarks:
If the method fails, the queue pair can still process existing or future requests. However, if the IND2CompletionQueue::GetResults method returns an ND2_RESULT with an error status, the connection is terminated and any further requests are completed with ND_CANCELED status.