Fastest way to clone a hash_map

[devdude]

Hi,

I have the need to take a snapshot of a hash_map during execution
(actually transform it to a vector). This map is a shared resource
and therefore must be locked prior to any read/write operations thus I
need to minimize the amount of time the map resource is locked.

The map is defined as type <string, boost::shared_p tr<myobject>>. My
algorithm is as such:

void SnapShotToVecto r( vector< pair< string,
boost::shared_p tr<myobject& vec )
{

lockResource(th is->map);
vec.resize( map.size() );
copy(this->map.begin(), this->map.end(),list .begin());
unlockResource( this->map);
}

For about 3M elements w/in the map, I'm noticing that the resize op
takes about 150ms and the copy takes ~850ms. Is there any way to do
better? I suppose the total time doesn' t matter as it's the time the
resource is actually locked is the primary concern.

TIA

[=?ISO-8859-1?Q?Marcel_M=FCller?=]

devdude wrote:

For about 3M elements w/in the map, I'm noticing that the resize op
takes about 150ms and the copy takes ~850ms. Is there any way to do
better? I suppose the total time doesn' t matter as it's the time the
resource is actually locked is the primary concern.

You won't come around to copy the list of elements within the mutex
section. You might do the resize outside the lock, but you risk that a
second more expensive resize is required during the copy.

If you need shorter locks then you have to use a collection with an
iterator that is thread safe in a way that iteration over the mutable
collection gives defined behavior. Directory scans behave like that for
example. But the results are not reproduceable, of course.
Marcel

[=?UTF-8?B?RXJpayBXaWtzdHLDtm0=?=]

On 2008-04-29 17:33, devdude wrote:

Hi,

I have the need to take a snapshot of a hash_map during execution
(actually transform it to a vector). This map is a shared resource
and therefore must be locked prior to any read/write operations thus I
need to minimize the amount of time the map resource is locked.

The map is defined as type <string, boost::shared_p tr<myobject>>. My
algorithm is as such:

void SnapShotToVecto r( vector< pair< string,
boost::shared_p tr<myobject& vec )
{

lockResource(th is->map);
vec.resize( map.size() );
copy(this->map.begin(), this->map.end(),list .begin());
unlockResource( this->map);
}

For about 3M elements w/in the map, I'm noticing that the resize op
takes about 150ms and the copy takes ~850ms. Is there any way to do
better? I suppose the total time doesn' t matter as it's the time the
resource is actually locked is the primary concern.

You can start by moving the vec.resize() outside the lock (actually you
should probably only have to use a reserve() and not resize()). You can
probably also use a readers/writers lock to allow reads to the map while
copying it, which might be enough if you do not perform to many
modifications on it.

Depending on how you have implemented the hash-map you might be able to
add some kind of copy on write functionality, so that if an element is
added/ remove/changed in the map it will not affect the original map but
a copy, which shares the unmodified elements with the original map.

If the readers/writers lock is not enough you need to either make the
copying faster to do something smart with the map, either way you need
some information about the implementation of the map.

--
Erik Wikström

[devdude]

On Apr 29, 12:25 pm, Erik Wikström <Erik-wikst...@telia. comwrote:

On 2008-04-29 17:33, devdude wrote:

Hi,

I have the need to take a snapshot of a hash_map during execution
(actually transform it to a vector). This map is a shared resource
and therefore must be locked prior to any read/write operations thus I
need to minimize the amount of time the map resource is locked.

The map is defined as type <string, boost::shared_p tr<myobject>>. My
algorithm is as such:

void SnapShotToVecto r( vector< pair< string,
boost::shared_p tr<myobject& vec )
{

lockResource(th is->map);
vec.resize( map.size() );
copy(this->map.begin(), this->map.end(),list .begin());
unlockResource( this->map);
}

For about 3M elements w/in the map, I'm noticing that the resize op
takes about 150ms and the copy takes ~850ms. Is there any way to do
better? I suppose the total time doesn' t matter as it's the time the
resource is actually locked is the primary concern.

You can start by moving the vec.resize() outside the lock (actually you
should probably only have to use a reserve() and not resize()). You can
probably also use a readers/writers lock to allow reads to the map while
copying it, which might be enough if you do not perform to many
modifications on it.

Depending on how you have implemented the hash-map you might be able to
add some kind of copy on write functionality, so that if an element is
added/ remove/changed in the map it will not affect the original map but
a copy, which shares the unmodified elements with the original map.

If the readers/writers lock is not enough you need to either make the
copying faster to do something smart with the map, either way you need
some information about the implementation of the map.

--
Erik Wikström

I am actually using a reader/writer lock (are there standard
implementations of r/w locks avail by chance?) but the numbers are
from stress testing using all writes so it doesn't really matter.
hash_map is google's dense_map. moving vec.resize() outside the lock
gives me problems with the copy() -- assertion occurs in the bowels of
stl. Are there strategies to "chunk" the copy through multiple
iterations over the map?

[=?UTF-8?B?RXJpayBXaWtzdHLDtm0=?=]

On 2008-04-29 19:51, devdude wrote:

On Apr 29, 12:25 pm, Erik Wikström <Erik-wikst...@telia. comwrote:

>On 2008-04-29 17:33, devdude wrote:

Hi,

I have the need to take a snapshot of a hash_map during execution
(actually transform it to a vector). This map is a shared resource
and therefore must be locked prior to any read/write operations thus I
need to minimize the amount of time the map resource is locked.

The map is defined as type <string, boost::shared_p tr<myobject>>. My
algorithm is as such:

void SnapShotToVecto r( vector< pair< string,
boost::shared_p tr<myobject& vec )
{

lockResource(th is->map);
vec.resize( map.size() );
copy(this->map.begin(), this->map.end(),list .begin());
unlockResource( this->map);
}

For about 3M elements w/in the map, I'm noticing that the resize op
takes about 150ms and the copy takes ~850ms. Is there any way to do
better? I suppose the total time doesn' t matter as it's the time the
resource is actually locked is the primary concern.

You can start by moving the vec.resize() outside the lock (actually you
should probably only have to use a reserve() and not resize()). You can
probably also use a readers/writers lock to allow reads to the map while
copying it, which might be enough if you do not perform to many
modification s on it.

Depending on how you have implemented the hash-map you might be able to
add some kind of copy on write functionality, so that if an element is
added/ remove/changed in the map it will not affect the original map but
a copy, which shares the unmodified elements with the original map.

If the readers/writers lock is not enough you need to either make the
copying faster to do something smart with the map, either way you need
some information about the implementation of the map.

Please do not quote signatures. And by the way, multi-posting is
generally not a good idea, if you have to post in multiple groups you
should cross-post (that way we will see what others have to say).

I am actually using a reader/writer lock (are there standard
implementations of r/w locks avail by chance?) but the numbers are
from stress testing using all writes so it doesn't really matter.
hash_map is google's dense_map. moving vec.resize() outside the lock
gives me problems with the copy() -- assertion occurs in the bowels of
stl.

Probably because the size of the map changes between the resize and the
copying and you iterate past the end of the vector. You should insert
the elements instead of assigning them, try using reserve()and an insert
iterator.

Notice that if the size of the map increases between the reserve and the
copying the vector might have to reallocate which will cost a lot, but
since reserve() is quite cheap you can probably do it inside the lock.
If that does not work you might try a deque, insertions at the end are
O(1), and not amortised O(1) like the vector, and you never have to
reallocate.

Are there strategies to "chunk" the copy through multiple
iterations over the map?

Do you mean to only copy parts of the map and then wait a bit and copy
the next part? You probably could but the results would be quite
meaningless since some elements will be added after you copy that part,
and others will be deleted.

Have you tried to create a copy of the map inside the lock, and copy to
the vector outside it?

--
Erik Wikström

[devdude]

On Apr 29, 2:42 pm, Erik Wikström <Erik-wikst...@telia. comwrote:

Have you tried to create a copy of the map inside the lock, and copy to
the vector outside it?

If you mean:

hash_map<T1,T2n ewMap (oldMap);

Then I have tried this and the copy const. takes 2+ secs alone to
perform. Is there something faster?