Management for memory buffers registered to the network. More...

Collaboration diagram for LCI memory management:

Typedefs
typedef struct LCII_mr_t *	LCI_segment_t
	LCI memory segment. More...

Functions
LCI_API LCI_error_t	LCI_memory_register (LCI_device_t device, void address, size_t length, LCI_segment_t segment)
	Register a memory region to a device. More...

LCI_API LCI_error_t	LCI_memory_deregister (LCI_segment_t *segment)
	Deregister a memory region. More...

LCI_API LCI_error_t	LCI_mbuffer_alloc (LCI_device_t device, LCI_mbuffer_t *mbuffer)
	Get a pre-registered memory buffer of size LCI_MEDIUM_SIZE. More...

LCI_API LCI_error_t	LCI_mbuffer_free (LCI_mbuffer_t mbuffer)
	Return a pre-registered memory buffer to the LCI runtime. More...

LCI_API LCI_error_t	LCI_lbuffer_alloc (LCI_device_t device, size_t size, LCI_lbuffer_t *lbuffer)
	Allocate a memory buffer of arbitrary size and register it. More...

LCI_API LCI_error_t	LCI_lbuffer_memalign (LCI_device_t device, size_t size, size_t alignment, LCI_lbuffer_t *lbuffer)
	Allocate a memory buffer of arbitrary size with a specific alignment and register it. More...

LCI_API LCI_error_t	LCI_lbuffer_free (LCI_lbuffer_t lbuffer)
	Deregister and free a memory buffer. More...

Variables
int	LCI_PACKET_RETURN_THRESHOLD
	The packet returning threshold. More...

int	LCI_IBV_USE_ODP
	Whether or what to use IBV on-demand paging. More...

int	LCI_TOUCH_LBUFFER
	Whether touch every page of lbuffers allocated by LCI_lbuffer_alloc. More...

int	LCI_USE_DREG
	Whether or what to use LCI-provided registration cache. More...

int	LCI_IBV_USE_PREFETCH
	Whether or what to use IBV prefetch. More...

int	LCI_PACKET_SIZE
	Size of each packet. More...

int	LCI_SERVER_NUM_PKTS
	Total number of packets. More...

Detailed Description

Management for memory buffers registered to the network.

In most of the cases, memory buffers participated in a network operations need to be registered to the network. The LCI runtime can do this for users through either memory copies to a pre-registered buffers or dynamical registering and deregistering. LCI also exposes relevant functions regarding registered memory allocation and memory registration to the users so users can do communication operations more efficiently.

For small messages, the send data does not need to be registered since the data will be packed into the low-level network operation descriptors.

For medium messages, the send and receive buffers need to be registered. If users are using the "copy" version of operations (e.g. LCI_sendm, LCI_putm, LCI_recvm), the data will be copied to/from pre-registered buffers by the runtime. If users are using the "no-copy"version of operations (e.g. LCI_sendmn, LCI_putmn, LCI_recvmn), users will directly get access to the LCI internal pre-registered buffers through either LCI_mbuffer_alloc (for send buffers) or completion mechanisms (for receive buffers).

For long messages, the send and receive buffers need to be registered, and the cost of memory copies to pre-registered buffers is considered unworthy. As a result, either users or the runtime needs to explicitly register the buffers.

Typedef Documentation

◆ LCI_segment_t

typedef struct LCII_mr_t* LCI_segment_t

LCI memory segment.

All LCI communication must take place in memory segments, which represent memory regions registered to devices.

Function Documentation

◆ LCI_lbuffer_alloc()

LCI_API LCI_error_t LCI_lbuffer_alloc	(	LCI_device_t	device,
		size_t	size,
		LCI_lbuffer_t *	lbuffer
	)

Allocate a memory buffer of arbitrary size and register it.

Note: This is just a convenient function for now.

Parameters

[in]	device	The device to register the buffer.
[in]	size	The size of the buffer to allocate.
[out]	lbuffer	The resulting buffer.

Returns: Should always be LCI_OK. All the other errors are fatal as defined by LCI_error_t.

◆ LCI_lbuffer_free()

LCI_API LCI_error_t LCI_lbuffer_free ( LCI_lbuffer_t lbuffer )

Deregister and free a memory buffer.

Note: This is just a convenient function for now.

Parameters

[in] lbuffer The buffer to deregister and free.

Returns: Should always be LCI_OK. All the other errors are fatal as defined by LCI_error_t.

◆ LCI_lbuffer_memalign()

LCI_API LCI_error_t LCI_lbuffer_memalign	(	LCI_device_t	device,
		size_t	size,
		size_t	alignment,
		LCI_lbuffer_t *	lbuffer
	)

Allocate a memory buffer of arbitrary size with a specific alignment and register it.

Note: This is just a convenient function for now.

Parameters

[in]	device	The device to register the buffer.
[in]	size	The size of the buffer to allocate.
[in]	alignment	The alignment of the buffer to allocate.
[out]	lbuffer	The resulting buffer.

Returns: Should always be LCI_OK. All the other errors are fatal as defined by LCI_error_t.

◆ LCI_mbuffer_alloc()

LCI_API LCI_error_t LCI_mbuffer_alloc	(	LCI_device_t	device,
		LCI_mbuffer_t *	mbuffer
	)

Get a pre-registered memory buffer of size LCI_MEDIUM_SIZE.

Parameters

[in]	device	The device the buffer is registered to.
[out]	mbuffer	The pre-registered memory buffer.

Returns: LCI_OK if successfully get one. LCI_ERR_RETRY if buffers are temporarily unavailable. All the other errors are fatal as defined by LCI_error_t.

◆ LCI_mbuffer_free()

LCI_API LCI_error_t LCI_mbuffer_free ( LCI_mbuffer_t mbuffer )

Return a pre-registered memory buffer to the LCI runtime.

Parameters

[in] mbuffer The pre-registered memory buffer to return.

Returns: Should always be LCI_OK. All the other errors are fatal as defined by LCI_error_t.

◆ LCI_memory_deregister()

LCI_API LCI_error_t LCI_memory_deregister ( LCI_segment_t * segment )

Deregister a memory region.

Parameters

[in,out] segment The registration descriptor to deregister.

Returns: Should always be LCI_OK. All the other errors are fatal as defined by LCI_error_t.

◆ LCI_memory_register()

LCI_API LCI_error_t LCI_memory_register	(	LCI_device_t	device,
		void *	address,
		size_t	length,
		LCI_segment_t *	segment
	)

Register a memory region to a device.

Parameters

[in]	device	The device to register.
[in]	address	The starting address of the memory region.
[in]	length	The length of the memory region.
[out]	segment	The registration descriptor.

Returns: Should always be LCI_OK. All the other errors are fatal as defined by LCI_error_t.

Variable Documentation

◆ LCI_IBV_USE_ODP

int LCI_IBV_USE_ODP

Whether or what to use IBV on-demand paging.

0: Don't use ODP 1: Use explicit ODP 2: Use implicit ODP

◆ LCI_IBV_USE_PREFETCH

int LCI_IBV_USE_PREFETCH

Whether or what to use IBV prefetch.

0: Don't use. 1: Use. (Only make sense when LCI_IBV_USE_ODP is 1 or 2)

◆ LCI_PACKET_RETURN_THRESHOLD

int LCI_PACKET_RETURN_THRESHOLD

The packet returning threshold.

Apply to LCI_sendm and LCI_sendmn. if size > LCI_PACKET_RETURN_THRESHOLD: the packet will be returned to the compute thread's pool. else: the packet will be returned to the progress thread's pool.

◆ LCI_PACKET_SIZE

int LCI_PACKET_SIZE

Size of each packet.

◆ LCI_SERVER_NUM_PKTS

int LCI_SERVER_NUM_PKTS

Total number of packets.

◆ LCI_TOUCH_LBUFFER

int LCI_TOUCH_LBUFFER

Whether touch every page of lbuffers allocated by LCI_lbuffer_alloc.

◆ LCI_USE_DREG

int LCI_USE_DREG

Whether or what to use LCI-provided registration cache.

0: Don't use. 1: Use.

Typedefs

Functions

Variables

Detailed Description

Typedef Documentation

◆ LCI_segment_t

Function Documentation

◆ LCI_lbuffer_alloc()

◆ LCI_lbuffer_free()

◆ LCI_lbuffer_memalign()

◆ LCI_mbuffer_alloc()

◆ LCI_mbuffer_free()

◆ LCI_memory_deregister()

◆ LCI_memory_register()

Variable Documentation

◆ LCI_IBV_USE_ODP

◆ LCI_IBV_USE_PREFETCH

◆ LCI_PACKET_RETURN_THRESHOLD

◆ LCI_PACKET_SIZE

◆ LCI_SERVER_NUM_PKTS

◆ LCI_TOUCH_LBUFFER

◆ LCI_USE_DREG