OpenFPM_pdata  4.1.0
Project that contain the implementation of distributed structures
 
Loading...
Searching...
No Matches
Vector 4 property serialization

Vector 4 property serialization

This example show how we can use complex properties in a vector when we use a custom structure with a pointer inside. In particular we will show how to construct the serialization and de-serialization methods for the structure my_struct defined here

struct my_struct
{
size_t size;
char * ptr;
std::string str;
Implementation of 1-D std::vector like structure.
openfpm::vector< int > v
vector
Definition main_ser.cpp:38
std::string str
C++ string.
Definition main_ser.cpp:35
char * ptr
C string pointer.
Definition main_ser.cpp:32
size_t size
C string size.
Definition main_ser.cpp:29
};

In order to make my_struct serializable we need 4 methods

  • packRequest This method indicate how many byte are needed to serialize this structure
  • pack This method serialize the data-structure
  • unpack This method de-serialize the data structure
  • noPointers() a static method that inform the OpenFPM sub-system that the structure has no pointers inside

packRequest

template<int ... prp> inline void packRequest(size_t & req) const
{
req += 2*sizeof(size_t) + size + str.size()+1;
v.packRequest(req);
}

This function calculate the size in byte to serialize this structure in this case we serialize 2 numbers so 2*sizeof(size_t). One C string of size "size" and one C++ string of size str.size(). Because std::string has a constructor from null terminating string we add the null terminator to easy construct an std::string. This mean that the C++ string will be serialized in str.size()+1 bytes. The result must be summed to counter req that is used to allocate a buffer big enough for serialization. The function is template and has a variadic argument "int ... prp" this can be ignored

pack

Here we serialize the object. The function give as argument

  • ExtPreAlloc<Memory> mem The memory where we have to serialize the information
  • Pack_stat unused
  • int ... prp unused

The only important parameter is the object mem where we will serialize the information

We first pack the number that indicate the size of the C string. A convenient way is use the function

//Serialize the number that determine the C-string size
static void pack(ExtPreAlloc< Mem >, const T &obj)
Error, no implementation.
Definition Packer.hpp:56

The function Packer<decltype(object),Memory>::pack(object,sts) work if object is a fundamental C++ type, a struct that does not contain pointers, any object that implement the interface pack,unpack,packRequest (like my_struct). Most openfpm data-structure like openfpm::vector and grid implement such interface and can be used directly with Packer<...>::pack(...). After packed the size of the string we have to serialize the content of the pointer string. unfortunately there is no Packer<...>::pack(...) function to do this and must be manually done. This can be done with the following steps

  • Allocate the memory to copy the value of the string
  • Get the pointer to the memory allocated
  • copy the content of the string into the allocated memory
// Allocate and copy the string
mem.allocate(size);
void * t_ptr = mem.getPointer();
memcpy(t_ptr,ptr,size);

For the C++ string the concept is the same, the difference is that we put a null terminator at the end of the serialized string

//Serialize the number that determine the C++-string str.size()+1 given by the null terminator
Packer<size_t, Memory>::pack(mem,str.size()+1,sts);
// Allocate and copy the string
mem.allocate(str.size()+1);
char * t_ptr2 = (char *)mem.getPointer();
memcpy(t_ptr2,str.c_str(),str.size());
// Add null terminator
t_ptr2[str.size()] = 0;
Packer<decltype(v), Memory>::pack(mem,v,sts);
Packing class.
Definition Packer.hpp:50

unpack

Here we de-serialize the object. The function take a reference to object

  • ExtPreAlloc<Memory> mem, contain the serialized information that must be de-serialized
  • Pack_stat contain the offset to the memory to de-serialize
  • int ... prp unused

De-serialization must be in general done in the same order as we serialized We first unpack the number that indicate the size of the C string. A convenient way is to use the function

// unpack the size of the C string
static void unpack(ExtPreAlloc< Mem >, T &obj)
Error, no implementation.
Definition Unpacker.hpp:40

the variable size contain now the size of the packed string we can now

  • create memory with new that store the C string
  • Get the pointer to the serialized information
  • copy the string from mem into the created memory
  • update the offset pointer
// Allocate the C string
ptr = new char[size];
// source pointer
char * ptr_src = (char *)mem.getPointerOffset(ps.getOffset());
// copy from the buffer to the destination
memcpy(ptr,ptr_src,size);
// update the pointer
ps.addOffset(size);

For the C++ string is just the same

  • We get the size of the string
  • we create an std::string out of the null terminating serialized string
  • we assign the created std::string to str
// get the the C++ string size
size_t cpp_size;
// Create the string from the serialized data (de-serialize)
char * ptr_src2 = (char *)mem.getPointerOffset(ps.getOffset());
str = std::string(ptr_src2);
ps.addOffset(cpp_size);
// Unpack the vector
Unpacker<decltype(v), Memory>::unpack(mem,v,ps);
Unpacker class.
Definition Unpacker.hpp:34

noPointers

This method inform the sub-system that the custom structure does not have pointers

static bool noPointers()
{
return false;
}

Constructor and destructor and operator=

Constructor and destructor are not releated to serialization and de-serialization concept. But on how my_struct is constructed and destructed in order to avoid memory leak/corruption. In the constructor we set ptr to NULL in the destructor we destroy the pointer (if different from NULL) to avoid memory leak

{
size = 0;
ptr = NULL;
}
my_struct(const my_struct & my)
{
this->operator=(my);
}
{
this->operator=(my);
}
{
if (ptr != NULL)
delete [] ptr;
}
// we copy pointer and we do double delete
my_struct & operator=(const my_struct & my)
{
size = my.size;
str = my.str;
v = my.v;
ptr = new char[size];
memcpy(ptr,my.ptr,32);
return *this;
}
// we copy pointer and we do double delete
my_struct & operator=(my_struct && my)
{
size = my.size;
my.size = 0;
str.swap(my.str);
v.swap(my.v);
ptr = my.ptr;
my.ptr = 0;
return *this;
}

Initialization and vector creation

After we initialize the library we can create a vector with complex properties with the following line

vector_dist<2,float, aggregate<float,
float[3],
vd(4096,domain,bc,g);
This class implement the point shape in an N-dimensional space.
Definition Point.hpp:28
Distributed vector.
aggregate of properties, from a list of object if create a struct that follow the OPENFPM native stru...

In this this particular case every particle carry two my_struct object

Assign values to properties

In this loop we assign position to particles and we fill the two my_struct that each particle contain. As demostration the first my_struct is filled with the string representation of the particle coordinates. The second my struct is filled with the string representation of the particle position multiplied by 2.0. The the vectors of the two my_struct are filled respectively with the sequence 1,2,3 and 1,2,3,4

auto it = vd.getDomainIterator();
while (it.isNext())
{
auto p = it.get();
// we define x, assign a random position between 0.0 and 1.0
vd.getPos(p)[0] = (float)rand() / RAND_MAX;
// we define y, assign a random position between 0.0 and 1.0
vd.getPos(p)[1] = (float)rand() / RAND_MAX;
// Get the particle position as point
Point<2,float> pt = vd.getPos(p);
// create a C string from the particle coordinates
// and copy into my struct
vd.getProp<my_s1>(p).size = 32;
vd.getProp<my_s1>(p).ptr = new char[32];
strcpy(vd.getProp<my_s1>(p).ptr,pt.toString().c_str());
// create a C++ string from the particle coordinates
vd.getProp<my_s1>(p).str = std::string(pt.toString());
vd.getProp<my_s1>(p).v.add(1);
vd.getProp<my_s1>(p).v.add(2);
vd.getProp<my_s1>(p).v.add(3);
pt = pt * 2.0;
// create a C string from the particle coordinates multiplied by 2.0
// and copy into my struct
vd.getProp<my_s2>(p).size = 32;
vd.getProp<my_s2>(p).ptr = new char[32];
strcpy(vd.getProp<my_s2>(p).ptr,pt.toString().c_str());
// create a C++ string from the particle coordinates
vd.getProp<my_s2>(p).str = std::string(pt.toString());
vd.getProp<my_s2>(p).v.add(1);
vd.getProp<my_s2>(p).v.add(2);
vd.getProp<my_s2>(p).v.add(3);
vd.getProp<my_s2>(p).v.add(4);
// next particle
++it;
}
std::string toString() const
Return the string with the point coordinate.
Definition Point.hpp:398

Mapping and ghost_get

Particles are redistributed across processors and we also synchronize the ghost

See also
Mapping particles
Ghost
// Particles are redistribued across the processors
vd.map();
// Synchronize the ghost
vd.ghost_get<my_s1,my_s2>();

Output and VTK visualization

Vector with complex properties can be still be visualized, because unknown properties are automatically excluded

See also
Visualization, write VTK files
vd.write("particles");

Print 4 particles in the ghost area

Here we print that the first 4 particles to show that the two my_struct contain the right information

size_t fg = vd.size_local();
Vcluster<> & v_cl = create_vcluster();
// Only the master processor print
if (v_cl.getProcessUnitID() == 0)
{
// Print 4 particles
for ( ; fg < vd.size_local()+4 ; fg++)
{
// Print my struct1 information
std::cout << "my_struct1:" << std::endl;
std::cout << "C-string: " << vd.getProp<my_s1>(fg).ptr << std::endl;
std::cout << "Cpp-string: " << vd.getProp<my_s1>(fg).str << std::endl;
for (size_t i = 0 ; i < vd.getProp<my_s1>(fg).v.size() ; i++)
std::cout << "Element: " << i << " " << vd.getProp<my_s1>(fg).v.get(i) << std::endl;
// Print my struct 2 information
std::cout << "my_struct2" << std::endl;
std::cout << "C-string: " << vd.getProp<my_s2>(fg).ptr << std::endl;
std::cout << "Cpp-string: " << vd.getProp<my_s2>(fg).str << std::endl;
for (size_t i = 0 ; i < vd.getProp<my_s2>(fg).v.size() ; i++)
std::cout << "Element: " << i << " " << vd.getProp<my_s2>(fg).v.get(i) << std::endl;
}
}
size_t getProcessUnitID()
Get the process unit id.
Implementation of VCluster class.
Definition VCluster.hpp:59

Finalize

At the very end of the program we have always to de-initialize the library

openfpm_finalize();

Full code

#include "Vector/vector_dist.hpp"
struct my_struct
{
size_t size;
char * ptr;
std::string str;
public:
static bool pack() {return true;}
{
size = 0;
ptr = NULL;
}
my_struct(const my_struct & my)
{
this->operator=(my);
}
{
this->operator=(my);
}
{
if (ptr != NULL)
delete [] ptr;
}
// we copy pointer and we do double delete
{
size = my.size;
str = my.str;
v = my.v;
ptr = new char[size];
memcpy(ptr,my.ptr,32);
return *this;
}
// we copy pointer and we do double delete
{
size = my.size;
my.size = 0;
str.swap(my.str);
v.swap(my.v);
ptr = my.ptr;
my.ptr = 0;
return *this;
}
template<int ... prp> inline void packRequest(size_t & req) const
{
req += 2*sizeof(size_t) + size + str.size()+1;
v.packRequest(req);
}
static bool noPointers()
{
return false;
}
template<typename Memory, int ... prp> inline void pack(ExtPreAlloc<Memory> & mem, Pack_stat & sts) const
{
//Serialize the number that determine the C-string size
// Allocate and copy the string
mem.allocate(size);
void * t_ptr = mem.getPointer();
memcpy(t_ptr,ptr,size);
//Serialize the number that determine the C++-string str.size()+1 given by the null terminator
Packer<size_t, Memory>::pack(mem,str.size()+1,sts);
// Allocate and copy the string
mem.allocate(str.size()+1);
char * t_ptr2 = (char *)mem.getPointer();
memcpy(t_ptr2,str.c_str(),str.size());
// Add null terminator
t_ptr2[str.size()] = 0;
Packer<decltype(v), Memory>::pack(mem,v,sts);
}
template<typename Memory, int ... prp> inline void unpack(ExtPreAlloc<Memory> & mem, Unpack_stat & ps)
{
// unpack the size of the C string
// Allocate the C string
ptr = new char[size];
// source pointer
char * ptr_src = (char *)mem.getPointerOffset(ps.getOffset());
// copy from the buffer to the destination
memcpy(ptr,ptr_src,size);
// update the pointer
// get the the C++ string size
size_t cpp_size;
// Create the string from the serialized data (de-serialize)
char * ptr_src2 = (char *)mem.getPointerOffset(ps.getOffset());
str = std::string(ptr_src2);
ps.addOffset(cpp_size);
// Unpack the vector
Unpacker<decltype(v), Memory>::unpack(mem,v,ps);
}
};
int main(int argc, char* argv[])
{
// initialize the library
openfpm_init(&argc,&argv);
// Here we define our domain a 2D box with internals from 0 to 1.0 for x and y
Box<2,float> domain({0.0,0.0},{1.0,1.0});
// Here we define the boundary conditions of our problem
size_t bc[2]={PERIODIC,PERIODIC};
// extended boundary around the domain, and the processor domain
Ghost<2,float> g(0.01);
// my_struct at position 0 in the aggregate
constexpr int my_s1 = 0;
// my_struct at position 1 in the aggregate
constexpr int my_s2 = 1;
vd(4096,domain,bc,g);
std::cout << "HAS PACK: " << has_pack_agg<aggregate<my_struct,my_struct>>::result::value << std::endl;
auto it = vd.getDomainIterator();
while (it.isNext())
{
auto p = it.get();
// we define x, assign a random position between 0.0 and 1.0
vd.getPos(p)[0] = (float)rand() / RAND_MAX;
// we define y, assign a random position between 0.0 and 1.0
vd.getPos(p)[1] = (float)rand() / RAND_MAX;
// Get the particle position as point
Point<2,float> pt = vd.getPos(p);
// create a C string from the particle coordinates
// and copy into my struct
vd.getProp<my_s1>(p).size = 32;
vd.getProp<my_s1>(p).ptr = new char[32];
strcpy(vd.getProp<my_s1>(p).ptr,pt.toString().c_str());
// create a C++ string from the particle coordinates
vd.getProp<my_s1>(p).str = std::string(pt.toString());
vd.getProp<my_s1>(p).v.add(1);
vd.getProp<my_s1>(p).v.add(2);
vd.getProp<my_s1>(p).v.add(3);
pt = pt * 2.0;
// create a C string from the particle coordinates multiplied by 2.0
// and copy into my struct
vd.getProp<my_s2>(p).size = 32;
vd.getProp<my_s2>(p).ptr = new char[32];
strcpy(vd.getProp<my_s2>(p).ptr,pt.toString().c_str());
// create a C++ string from the particle coordinates
vd.getProp<my_s2>(p).str = std::string(pt.toString());
vd.getProp<my_s2>(p).v.add(1);
vd.getProp<my_s2>(p).v.add(2);
vd.getProp<my_s2>(p).v.add(3);
vd.getProp<my_s2>(p).v.add(4);
// next particle
++it;
}
// Particles are redistribued across the processors
vd.map();
// Synchronize the ghost
vd.ghost_get<my_s1,my_s2>();
vd.write("particles");
size_t fg = vd.size_local();
Vcluster<> & v_cl = create_vcluster();
// Only the master processor print
if (v_cl.getProcessUnitID() == 0)
{
// Print 4 particles
for ( ; fg < vd.size_local()+4 ; fg++)
{
// Print my struct1 information
std::cout << "my_struct1:" << std::endl;
std::cout << "C-string: " << vd.getProp<my_s1>(fg).ptr << std::endl;
std::cout << "Cpp-string: " << vd.getProp<my_s1>(fg).str << std::endl;
for (size_t i = 0 ; i < vd.getProp<my_s1>(fg).v.size() ; i++)
std::cout << "Element: " << i << " " << vd.getProp<my_s1>(fg).v.get(i) << std::endl;
// Print my struct 2 information
std::cout << "my_struct2" << std::endl;
std::cout << "C-string: " << vd.getProp<my_s2>(fg).ptr << std::endl;
std::cout << "Cpp-string: " << vd.getProp<my_s2>(fg).str << std::endl;
for (size_t i = 0 ; i < vd.getProp<my_s2>(fg).v.size() ; i++)
std::cout << "Element: " << i << " " << vd.getProp<my_s2>(fg).v.get(i) << std::endl;
}
}
openfpm_finalize();
}
This class represent an N-dimensional box.
Definition Box.hpp:61
void * getPointerOffset(size_t offset)
Get the base memory pointer increased with an offset.
virtual void * getPointer()
Return the pointer of the last allocation.
virtual bool allocate(size_t sz)
Allocate a chunk of memory.
Packing status object.
Definition Pack_stat.hpp:61
Unpacking status object.
Definition Pack_stat.hpp:16
size_t getOffset()
Return the actual counter.
Definition Pack_stat.hpp:41
void addOffset(size_t off)
Increment the offset pointer by off.
Definition Pack_stat.hpp:31
size_t size()
Stub size.
return if true the aggregate type T has a property that has a complex packing(serialization) method
static bool pack()
Functions to check if the packing object is complex.
Definition main_ser.cpp:45
my_struct & operator=(const my_struct &my)
This is fundamental to avoid crash, otherwise.
Definition main_ser.cpp:76
void packRequest(size_t &req) const
Serialization request.
Definition main_ser.cpp:107
void unpack(ExtPreAlloc< Memory > &mem, Unpack_stat &ps)
De-serialize the data structure.
Definition main_ser.cpp:168