This class virtualize the cluster of PC as a set of processes that communicate. More...

Detailed Description

This class virtualize the cluster of PC as a set of processes that communicate.

At the moment it is an MPI-like interface, with a more type aware, and simple, interface. It also give some more complex communication functionalities like Dynamic Sparse Data Exchange

Actually VCluster expose a Computation driven parallelism (MPI-like), with a plan of extending to communication driven parallelism

In computation driven parallelism, the program compute than communicate to the other processors
In a communication driven parallelism, (Charm++ or HPX), the program receive messages, this receiving messages trigger computation

An example of sending and receive plain buffers

    // Send to 8 processors
    for (size_t i = 0 ; i < 8 ; i++)
        vcl.send( mod(vcl.getProcessUnitID() + i * P_STRIDE, vcl.getProcessingUnits()) ,i,v_send.getPointer(),v_send.size()*sizeof(T));
    openfpm::vector<openfpm::vector<T> > pt_buf;
    pt_buf.resize(8);
    // Recv from 8 processors
    for (size_t i = 0 ; i < 8 ; i++)
    {
        pt_buf.get(i).resize(n);
        vcl.recv( mod( (vcl.getProcessUnitID() - i * P_STRIDE), vcl.getProcessingUnits()) ,i,pt_buf.get(i).getPointer(),pt_buf.get(i).size()*sizeof(T));
    }
    vcl.execute();

An example of sending vectors of primitives with (T=float,double,lont int,...)

    // Send to 8 processors
    for (size_t i = 0 ; i < 8 ; i++)
        vcl.send( mod(vcl.getProcessUnitID() + i * P_STRIDE, vcl.getProcessingUnits()) ,i,v_send);
    openfpm::vector<openfpm::vector<T> > pt_buf;
    pt_buf.resize(8);
    // Recv from 8 processors
    for (size_t i = 0 ; i < 8 ; i++)
    {
        pt_buf.get(i).resize(n);
        vcl.recv( mod( (vcl.getProcessUnitID() - i * P_STRIDE), vcl.getProcessingUnits()) ,i,pt_buf.get(i));
    }
    vcl.execute();

An example of sending vectors of complexes object

    // Point test typedef
    typedef Point_test<float> p;
    openfpm::vector<Point_test<float>> v_send = allocate_openfpm_fill(n,vcl.getProcessUnitID());
    // Send to 8 processors
    for (size_t i = 0 ; i < 8 ; i++)
        vcl.send( mod(vcl.getProcessUnitID() + i * P_STRIDE, vcl.getProcessingUnits()) ,i,v_send);
    openfpm::vector<openfpm::vector<Point_test<float>> > pt_buf;
    pt_buf.resize(8);
    // Recv from 8 processors
    for (size_t i = 0 ; i < 8 ; i++)
    {
        pt_buf.get(i).resize(n);
        vcl.recv( mod( (vcl.getProcessUnitID() - i * P_STRIDE), vcl.getProcessingUnits()) ,i,pt_buf.get(i));
    }
    vcl.execute();

An example of gathering numbers from all processors

    openfpm::vector<T> clt;
    T data = vcl.getProcessUnitID();
    vcl.allGather(data,clt);
    vcl.execute();
    for (size_t i = 0 ; i < vcl.getProcessingUnits() ; i++)
        BOOST_REQUIRE_EQUAL(i,(size_t)clt.get(i));

Definition at line 103 of file VCluster_base.hpp.

#include <VCluster_base.hpp>

Inheritance diagram for Vcluster_base:

Public Member Functions
	Vcluster_base (int argc, char **argv)
	Virtual cluster constructor. More...

MPI_Comm	getMPIComm ()
	Get the MPI_Communicator (or processor group) this VCluster is using. More...

size_t	getProcessingUnits ()
	Get the total number of processors. More...

size_t	size ()
	Get the total number of processors. More...

size_t	getProcessUnitID ()
	Get the process unit id. More...

size_t	rank ()
	Get the process unit id. More...

template<typename T >
void	sum (T &num)
	Sum the numbers across all processors and get the result. More...

template<typename T >
void	max (T &num)
	Get the maximum number across all processors (or reduction with infinity norm) More...

template<typename T >
void	min (T &num)
	Get the minimum number across all processors (or reduction with insinity norm) More...

template<typename T >
void	sendrecvMultipleMessagesNBX (openfpm::vector< size_t > &prc, openfpm::vector< T > &data, openfpm::vector< size_t > prc_recv, openfpm::vector< size_t > &recv_sz, void (msg_alloc)(size_t, size_t, size_t, size_t, size_t, void ), void ptr_arg, long int opt=NONE)
	Send and receive multiple messages. More...

template<typename T >
void	sendrecvMultipleMessagesNBX (openfpm::vector< size_t > &prc, openfpm::vector< T > &data, void (msg_alloc)(size_t, size_t, size_t, size_t, size_t, void ), void ptr_arg, long int opt=NONE)
	Send and receive multiple messages. More...

void	sendrecvMultipleMessagesNBX (size_t n_send, size_t sz[], size_t prc[], void ptr[], size_t n_recv, size_t prc_recv[], size_t sz_recv[], void (msg_alloc)(size_t, size_t, size_t, size_t, size_t, void ), void *ptr_arg, long int opt=NONE)
	Send and receive multiple messages. More...

void	sendrecvMultipleMessagesNBX (size_t n_send, size_t sz[], size_t prc[], void ptr[], size_t n_recv, size_t prc_recv[], void (msg_alloc)(size_t, size_t, size_t, size_t, size_t, void ), void *ptr_arg, long int opt=NONE)
	Send and receive multiple messages. More...

void	sendrecvMultipleMessagesNBX (size_t n_send, size_t sz[], size_t prc[], void ptr[], void (msg_alloc)(size_t, size_t, size_t, size_t, size_t, void ), void *ptr_arg, long int opt=NONE)
	Send and receive multiple messages. More...

bool	send (size_t proc, size_t tag, const void *mem, size_t sz)
	Send data to a processor. More...

template<typename T , typename Mem , typename gr >
bool	send (size_t proc, size_t tag, openfpm::vector< T, Mem, gr > &v)
	Send data to a processor. More...

bool	recv (size_t proc, size_t tag, void *v, size_t sz)
	Recv data from a processor. More...

template<typename T , typename Mem , typename gr >
bool	recv (size_t proc, size_t tag, openfpm::vector< T, Mem, gr > &v)
	Recv data from a processor. More...

template<typename T , typename Mem , typename gr >
bool	allGather (T &send, openfpm::vector< T, Mem, gr > &v)
	Gather the data from all processors. More...

template<typename T , typename Mem , typename gr >
bool	Bcast (openfpm::vector< T, Mem, gr > &v, size_t root)
	Broadcast the data to all processors. More...

void	execute ()
	Execute all the requests. More...

void	clear ()
	Release the buffer used for communication. More...

Data Fields
openfpm::vector< size_t >	sz_recv_tmp

Protected Attributes
openfpm::vector< BHeapMemory >	recv_buf
	Receive buffers.

Private Member Functions
Vcluster_base &	operator= (const Vcluster_base &)
	disable operator=

	Vcluster_base (const Vcluster_base &)
	disable copy constructor

Private Attributes
Vcluster_log	log
	log file

size_t	NBX_cnt

openfpm::vector< size_t >	proc_com

openfpm::vector< int >	map_scatter
	vector that contain the scatter map (it is basically an array of one)

openfpm::vector< MPI_Request >	req
	vector of MPI requests

openfpm::vector< MPI_Status >	stat
	vector of MPI status

std::vector< int >	post_exe
	vector of functions to execute after all the request has been performed

int	m_size
	number of processes

int	m_rank
	actual rank

int	numPE = 1
	number of processing unit per process

std::vector< red >	r

openfpm::vector< void * >	ptr_send
	vector of pointers of send buffers

openfpm::vector< size_t >	sz_send
	vector of the size of send buffers

MPI_Request	bar_req
	barrier request

MPI_Status	bar_stat
	barrier status

Constructor & Destructor Documentation

Vcluster_base::Vcluster_base	(	int *	argc,
		char ***	argv
	)

inline

Virtual cluster constructor.

Parameters

argc	pointer to arguments counts passed to the program
argv	pointer to arguments vector passed to the program

Definition at line 219 of file VCluster_base.hpp.

Member Function Documentation

template<typename T , typename Mem , typename gr >

bool Vcluster_base::allGather	(	T &	send,
		openfpm::vector< T, Mem, gr > &	v
	)

inline

Gather the data from all processors.

send a primitive data T receive the same primitive T from all the other processors

Warning: operation is asynchronous execute must be called to ensure they are executed

Parameters

v	vector to receive (automaticaly resized)
send	data to send

Returns: true if succeed false otherwise

Definition at line 1002 of file VCluster_base.hpp.

template<typename T , typename Mem , typename gr >

bool Vcluster_base::Bcast	(	openfpm::vector< T, Mem, gr > &	v,
		size_t	root
	)

inline

Broadcast the data to all processors.

broadcast a vector of primitives.

Warning: operation is asynchronous execute must be called to ensure the operation is executed; the non-root processor must resize the vector to the exact receive size. This mean the each processor must known a priory the receiving size

Parameters

v	vector to send in the case of the root processor and vector where to receive in the case of non-root
root	processor (who broadcast)

Returns: true if succeed false otherwise

Definition at line 1036 of file VCluster_base.hpp.

void Vcluster_base::clear ( )

inline

Release the buffer used for communication.

Definition at line 1073 of file VCluster_base.hpp.

void Vcluster_base::execute ( )

inline

Execute all the requests.

Definition at line 1054 of file VCluster_base.hpp.

MPI_Comm Vcluster_base::getMPIComm ( )

inline

Get the MPI_Communicator (or processor group) this VCluster is using.

Returns: MPI comunicator

Definition at line 322 of file VCluster_base.hpp.

size_t Vcluster_base::getProcessingUnits ( )

inline

Get the total number of processors.

Returns: the total number of processors

Definition at line 332 of file VCluster_base.hpp.

size_t Vcluster_base::getProcessUnitID ( )

inline

Get the process unit id.

Returns: the process ID

Definition at line 356 of file VCluster_base.hpp.

template<typename T >

void Vcluster_base::max ( T & num )

inline

Get the maximum number across all processors (or reduction with infinity norm)

Parameters

num to reduce

Definition at line 402 of file VCluster_base.hpp.

template<typename T >

void Vcluster_base::min ( T & num )

inline

Get the minimum number across all processors (or reduction with insinity norm)

Parameters

num to reduce

Definition at line 422 of file VCluster_base.hpp.

size_t Vcluster_base::rank ( )

inline

Get the process unit id.

It is the same as getProcessUnitID()

See Also: getProcessUnitID()

Returns: the process ID

Definition at line 370 of file VCluster_base.hpp.

bool Vcluster_base::recv	(	size_t	proc,
		size_t	tag,
		void *	v,
		size_t	sz
	)

inline

Recv data from a processor.

Warning: In order to avoid deadlock every recv must be coupled with a send in case you want to send data without knowledge from the other side consider to use or sendrecvMultipleMessagesNBX; operation is asynchronous execute must be called to ensure they are executed

See Also: sendrecvMultipleMessagesNBX

Parameters

proc	processor id
tag	id
v	buffer to send
sz	size of the buffer

Returns: true if succeed false otherwise

Definition at line 943 of file VCluster_base.hpp.

template<typename T , typename Mem , typename gr >

bool Vcluster_base::recv	(	size_t	proc,
		size_t	tag,
		openfpm::vector< T, Mem, gr > &	v
	)

inline

Recv data from a processor.

Warning: In order to avoid deadlock every recv must be coupled with a send in case you want to send data without knowledge from the other side consider to use sendrecvMultipleMessagesNBX; operation is asynchronous execute must be called to ensure they are executed

See Also: sendrecvMultipleMessagesNBX

Parameters

proc	processor id
tag	id
v	vector to send

Returns: true if succeed false otherwise

Definition at line 973 of file VCluster_base.hpp.

bool Vcluster_base::send	(	size_t	proc,
		size_t	tag,
		const void *	mem,
		size_t	sz
	)

inline

Send data to a processor.

Warning: In order to avoid deadlock every send must be coupled with a recv in case you want to send data without knowledge from the other side consider to use sendRecvMultipleMessages; operation is asynchronous execute must be called to ensure they are executed

See Also: sendRecvMultipleMessages

Parameters

proc	processor id
tag	id
mem	buffer with the data to send
sz	size

Returns: true if succeed false otherwise

Definition at line 877 of file VCluster_base.hpp.

template<typename T , typename Mem , typename gr >

bool Vcluster_base::send	(	size_t	proc,
		size_t	tag,
		openfpm::vector< T, Mem, gr > &	v
	)

inline

Send data to a processor.

Warning: In order to avoid deadlock every send must be coupled with a recv in case you want to send data without knowledge from the other side consider to use sendRecvMultipleMessages; operation is asynchronous execute must be called to ensure they are executed

See Also: sendRecvMultipleMessages

Parameters

proc	processor id
tag	id
v	buffer to send

Returns: true if succeed false otherwise

Definition at line 908 of file VCluster_base.hpp.

template<typename T >

void Vcluster_base::sendrecvMultipleMessagesNBX	(	openfpm::vector< size_t > &	prc,
		openfpm::vector< T > &	data,
		openfpm::vector< size_t >	prc_recv,
		openfpm::vector< size_t > &	recv_sz,
		void ()(size_t, size_t, size_t, size_t, size_t, void *)	msg_alloc,
		void *	ptr_arg,
		long int	opt = `NONE`
	)

inline

Send and receive multiple messages.

It send multiple messages to a set of processors the and receive multiple messages from another set of processors, all the processor must call this function. In this particular case the receiver know from which processor is going to receive.

suppose the following situation the calling processor want to communicate

2 messages of size 100 byte to processor 1
1 message of size 50 byte to processor 6
1 message of size 48 byte to processor 7
1 message of size 70 byte to processor 8

Parameters

prc	list of processor with which it should communicate [1,1,6,7,8]
data	data to send for each processors in contain a pointer to some type T this type T must have a method size() that return the size of the data-structure
prc_recv	processor that receive data
recv_sz	for each processor indicate the size of the data received
msg_alloc	This is a call-back with the purpose of allocate space for the incoming message and give back a valid pointer, supposing that this call-back has been triggered by the processor of id 5 that want to communicate with me a message of size 100 byte the call-back will have the following 6 parameters in the call-back are in order: message size required to receive the message [100] total message size to receive from all the processors (NBX does not provide this information) the total number of processor want to communicate with you (NBX does not provide this information) processor id [5] ri request id (it is an id that goes from 0 to total_p, and is incremented every time message_alloc is called) void pointer, parameter for additional data to pass to the call-back
ptr_arg	data passed to the call-back function specified
opt	options, NONE (ignored in this moment)

Definition at line 479 of file VCluster_base.hpp.

template<typename T >

void Vcluster_base::sendrecvMultipleMessagesNBX	(	openfpm::vector< size_t > &	prc,
		openfpm::vector< T > &	data,
		void ()(size_t, size_t, size_t, size_t, size_t, void *)	msg_alloc,
		void *	ptr_arg,
		long int	opt = `NONE`
	)

inline

Send and receive multiple messages.

It send multiple messages to a set of processors the and receive multiple messages from another set of processors, all the processor must call this function

suppose the following situation the calling processor want to communicate

2 vector of 100 integers to processor 1
1 vector of 50 integers to processor 6
1 vector of 48 integers to processor 7
1 vector of 70 integers to processor 8

Parameters

prc	list of processors you should communicate with [1,1,6,7,8]
data	vector containing the data to send [v=vector<vector<int>>, v.size()=4, T=vector<int>], T at the moment is only tested for vectors of 0 or more generic elements (without pointers)
msg_alloc	This is a call-back with the purpose to allocate space for the incoming messages and give back a valid pointer, supposing that this call-back has been triggered by the processor of id 5 that want to communicate with me a message of size 100 byte the call-back will have the following 6 parameters in the call-back in order: message size required to receive the message (100) total message size to receive from all the processors (NBX does not provide this information) the total number of processor want to communicate with you (NBX does not provide this information) processor id (5) ri request id (it is an id that goes from 0 to total_p, and is incremented every time message_alloc is called) void pointer, parameter for additional data to pass to the call-back
ptr_arg	data passed to the call-back function specified
opt	options, only NONE supported

Definition at line 542 of file VCluster_base.hpp.

void Vcluster_base::sendrecvMultipleMessagesNBX	(	size_t	n_send,
		size_t	sz[],
		size_t	prc[],
		void *	ptr[],
		size_t	n_recv,
		size_t	prc_recv[],
		size_t	sz_recv[],
		void ()(size_t, size_t, size_t, size_t, size_t, void *)	msg_alloc,
		void *	ptr_arg,
		long int	opt = `NONE`
	)

inline

Send and receive multiple messages.

It send multiple messages to a set of processors the and receive multiple messages from another set of processors, all the processor must call this function. In this particular case the receiver know from which processor is going to receive.

suppose the following situation the calling processor want to communicate

2 messages of size 100 byte to processor 1
1 message of size 50 byte to processor 6
1 message of size 48 byte to processor 7
1 message of size 70 byte to processor 8

Parameters

n_send	number of send for this processor [4]
prc	list of processor with which it should communicate [1,1,6,7,8]
sz	the array contain the size of the message for each processor (zeros must not be presents) [100,100,50,48,70]
ptr	array that contain the pointers to the message to send
msg_alloc	This is a call-back with the purpose of allocate space for the incoming message and give back a valid pointer, supposing that this call-back has been triggered by the processor of id 5 that want to communicate with me a message of size 100 byte the call-back will have the following 6 parameters in the call-back are in order: message size required to receive the message [100] total message size to receive from all the processors (NBX does not provide this information) the total number of processor want to communicate with you (NBX does not provide this information) processor id [5] ri request id (it is an id that goes from 0 to total_p, and is incremented every time message_alloc is called) void pointer, parameter for additional data to pass to the call-back
ptr_arg	data passed to the call-back function specified
opt	options, NONE (ignored in this moment)

Definition at line 605 of file VCluster_base.hpp.

void Vcluster_base::sendrecvMultipleMessagesNBX	(	size_t	n_send,
		size_t	sz[],
		size_t	prc[],
		void *	ptr[],
		size_t	n_recv,
		size_t	prc_recv[],
		void ()(size_t, size_t, size_t, size_t, size_t, void *)	msg_alloc,
		void *	ptr_arg,
		long int	opt = `NONE`
	)

inline

Send and receive multiple messages.

It send multiple messages to a set of processors the and receive multiple messages from another set of processors, all the processor must call this function. In this particular case the receiver know from which processor is going to receive, but does not know the size.

suppose the following situation the calling processor want to communicate

2 messages of size 100 byte to processor 1
1 message of size 50 byte to processor 6
1 message of size 48 byte to processor 7
1 message of size 70 byte to processor 8

Parameters

n_send	number of send for this processor [4]
prc	list of processor with which it should communicate [1,1,6,7,8]
sz	the array contain the size of the message for each processor (zeros must not be presents) [100,100,50,48,70]
ptr	array that contain the pointers to the message to send
msg_alloc	This is a call-back with the purpose of allocate space for the incoming message and give back a valid pointer, supposing that this call-back has been triggered by the processor of id 5 that want to communicate with me a message of size 100 byte the call-back will have the following 6 parameters in the call-back are in order: message size required to receive the message [100] total message size to receive from all the processors (NBX does not provide this information) the total number of processor want to communicate with you (NBX does not provide this information) processor id [5] ri request id (it is an id that goes from 0 to total_p, and is incremented every time message_alloc is called) void pointer, parameter for additional data to pass to the call-back
ptr_arg	data passed to the call-back function specified
opt	options, NONE (ignored in this moment)

Definition at line 673 of file VCluster_base.hpp.

void Vcluster_base::sendrecvMultipleMessagesNBX	(	size_t	n_send,
		size_t	sz[],
		size_t	prc[],
		void *	ptr[],
		void ()(size_t, size_t, size_t, size_t, size_t, void *)	msg_alloc,
		void *	ptr_arg,
		long int	opt = `NONE`
	)

inline

Send and receive multiple messages.

It send multiple messages to a set of processors the and receive multiple messages from another set of processors, all the processor must call this function

suppose the following situation the calling processor want to communicate

2 messages of size 100 byte to processor 1
1 message of size 50 byte to processor 6
1 message of size 48 byte to processor 7
1 message of size 70 byte to processor 8

Parameters

n_send	number of send for this processor [4]
prc	list of processor with which it should communicate [1,1,6,7,8]
sz	the array contain the size of the message for each processor (zeros must not be presents) [100,100,50,48,70]
ptr	array that contain the pointers to the message to send
msg_alloc	This is a call-back with the purpose of allocate space for the incoming message and give back a valid pointer, supposing that this call-back has been triggered by the processor of id 5 that want to communicate with me a message of size 100 byte the call-back will have the following 6 parameters in the call-back are in order: message size required to receive the message [100] total message size to receive from all the processors (NBX does not provide this information) the total number of processor want to communicate with you (NBX does not provide this information) processor id [5] ri request id (it is an id that goes from 0 to total_p, and is incremented every time message_alloc is called) void pointer, parameter for additional data to pass to the call-back
ptr_arg	data passed to the call-back function specified
opt	options, NONE (ignored in this moment)

Definition at line 751 of file VCluster_base.hpp.

size_t Vcluster_base::size ( )

inline

Get the total number of processors.

It is the same as getProcessingUnits()

See Also: getProcessingUnits()

Returns: the total number of processors

Definition at line 346 of file VCluster_base.hpp.

template<typename T >

void Vcluster_base::sum ( T & num )

inline

Sum the numbers across all processors and get the result.

Parameters

num	to reduce, input and output

Definition at line 382 of file VCluster_base.hpp.

Field Documentation

size_t Vcluster_base::NBX_cnt

private

NBX has a potential pitfall that must be addressed, NBX Send all the messages and probe for incoming messages, if there is an incoming message it receive it producing an acknowledge notification on the sending processor. When all the sends has been acknowledged, the processor call the MPI_Ibarrier when all the processors call MPI_Ibarrier all send has been received. While the processors are waiting for the MPI_Ibarrier to complete, all processors are still probing for incoming message, Unfortunately some processor can quit the MPI_Ibarrier before others and this mean that some processor can exit the probing status before others, these processors can in theory start new communications while the other processor are still in probing status producing a wrong send/recv association to resolve this problem an incremental NBX_cnt is used as message TAG to distinguish that the messages come from other send or subsequent NBX procedures

Definition at line 122 of file VCluster_base.hpp.

openfpm::vector<size_t> Vcluster_base::proc_com

private

temporal vector used for meta-communication ( or meta-data before the real communication )

Definition at line 126 of file VCluster_base.hpp.

std::vector<red> Vcluster_base::r

private

This buffer is a temporal buffer for reductions

MPI_Iallreduce does not accept recv and send buffer to be the same r is used to overcome this problem (is given as second parameter) after the execution the data is copied back

Definition at line 160 of file VCluster_base.hpp.

The documentation for this class was generated from the following file:

openfpm_vcluster/src/VCluster/VCluster_base.hpp

Detailed Description

An example of sending and receive plain buffers

An example of sending vectors of primitives with (T=float,double,lont int,...)

An example of sending vectors of complexes object

An example of gathering numbers from all processors

Public Member Functions

Data Fields

Protected Attributes

Private Member Functions

Private Attributes

Constructor & Destructor Documentation

Member Function Documentation

Field Documentation