473,581 Members | 3,183 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

TypeError: can't pickle HASH objects?

est
>>import md5
>>a=md5.md5()
import pickle
pickle.dumps( a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python25\li b\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump( obj)
File "C:\Python25\li b\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python25\li b\pickle.py", line 306, in save
rv = reduce(self.pro to)
File "C:\Python25\li b\copy_reg.py", line 69, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle HASH objects

Why can't I pickle a md5 object? Is it because md5 algorithm needs to
read 512-bits at a time?

I need to md5() some stream, pause(python.ex e quits), and resume
later. It seems that the md5 and hashlib in std module could not be
serialized?

Do I have to implement md5 algorithm again for this special occasion?

Or is there anyway to assige a digest when creating md5 objects?
Oct 1 '08 #1
10 7710
On Oct 1, 3:50*pm, est <electronix...@ gmail.comwrote:
>import md5
a=md5.md5()
import pickle
pickle.dumps(a )

Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* File "C:\Python25\li b\pickle.py", line 1366, in dumps
* * Pickler(file, protocol).dump( obj)
* File "C:\Python25\li b\pickle.py", line 224, in dump
* * self.save(obj)
* File "C:\Python25\li b\pickle.py", line 306, in save
* * rv = reduce(self.pro to)
* File "C:\Python25\li b\copy_reg.py", line 69, in _reduce_ex
* * raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle HASH objects

Why can't I pickle a md5 object? Is it because md5 algorithm needs to
read 512-bits at a time?

I need to md5() some stream, pause(python.ex e quits), and resume
later. *It seems that the md5 and hashlib in *std module could not be
serialized?

Do I have to implement md5 algorithm again for this special occasion?

Or is there anyway to assige a digest when creating md5 objects?
I'm sure some of the regulars can correct me if I'm wrong, but looking
at the source code, it seems that this is the error that you'll see if
the object doesn't explicitly support pickling, or possibly isn't
composed of objects that do.

Examining the md5 and hashlib source files, it seems that they rely on
C implementations , and so have internal states opaque to Python. If
you feel confident, you could write your own MD5 class that would have
methods to dump and restore state, but I think you're out of luck when
it comes to the official module.

Mark Sherry
Oct 1 '08 #2
On Oct 1, 2:50*pm, est <electronix...@ gmail.comwrote:
>import md5
a=md5.md5()
import pickle
pickle.dumps(a )

Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* File "C:\Python25\li b\pickle.py", line 1366, in dumps
* * Pickler(file, protocol).dump( obj)
* File "C:\Python25\li b\pickle.py", line 224, in dump
* * self.save(obj)
* File "C:\Python25\li b\pickle.py", line 306, in save
* * rv = reduce(self.pro to)
* File "C:\Python25\li b\copy_reg.py", line 69, in _reduce_ex
* * raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle HASH objects

Why can't I pickle a md5 object? Is it because md5 algorithm needs to
read 512-bits at a time?

I need to md5() some stream, pause(python.ex e quits), and resume
later. *It seems that the md5 and hashlib in *std module could not be
serialized?

Do I have to implement md5 algorithm again for this special occasion?

Or is there anyway to assige a digest when creating md5 objects?
Can you just pickle the stream, the part of it you've read so far?
Oct 2 '08 #3
est
On Oct 2, 11:27*am, "Aaron \"Castironpi \" Brady"
<castiro...@gma il.comwrote:
On Oct 1, 2:50*pm, est <electronix...@ gmail.comwrote:


>>import md5
>>a=md5.md5()
>>import pickle
>>pickle.dumps( a)
Traceback (most recent call last):
* File "<stdin>", line 1, in <module>
* File "C:\Python25\li b\pickle.py", line 1366, in dumps
* * Pickler(file, protocol).dump( obj)
* File "C:\Python25\li b\pickle.py", line 224, in dump
* * self.save(obj)
* File "C:\Python25\li b\pickle.py", line 306, in save
* * rv = reduce(self.pro to)
* File "C:\Python25\li b\copy_reg.py", line 69, in _reduce_ex
* * raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle HASH objects
Why can't I pickle a md5 object? Is it because md5 algorithm needs to
read 512-bits at a time?
I need to md5() some stream, pause(python.ex e quits), and resume
later. *It seems that the md5 and hashlib in *std module could not be
serialized?
Do I have to implement md5 algorithm again for this special occasion?
Or is there anyway to assige a digest when creating md5 objects?

Can you just pickle the stream, the part of it you've read so far?- Hide quoted text -

- Show quoted text -
wow. It's giga-size file. I need stream reading it, md5 it. It may
break for a while.
Oct 2 '08 #4
On Thu, Oct 2, 2008 at 3:34 PM, est <el***********@ gmail.comwrote:
wow. It's giga-size file. I need stream reading it, md5 it. It may
break for a while.
So use generators and consume the stream ?

--JamesMills

--
--
-- "Problems are solved by method"
Oct 2 '08 #5
est
On Oct 2, 1:51*pm, "James Mills" <prolo...@short circuit.net.auw rote:
On Thu, Oct 2, 2008 at 3:34 PM, est <electronix...@ gmail.comwrote:
wow. It's giga-size file. I need stream reading it, md5 it. It may
break for a while.

So use generators and consume the stream ?

--JamesMills

--
--
-- "Problems are solved by method"
no, I need to serialize half-finished digest, not file stream.

Anyone got solution?
Oct 2 '08 #6
On Oct 2, 2:44*am, est <electronix...@ gmail.comwrote:
On Oct 2, 1:51*pm, "James Mills" <prolo...@short circuit.net.auw rote:
On Thu, Oct 2, 2008 at 3:34 PM, est <electronix...@ gmail.comwrote:
wow. It's giga-size file. I need stream reading it, md5 it. It may
break for a while.
So use generators and consume the stream ?
--JamesMills
--
--
-- "Problems are solved by method"

no, I need to serialize half-finished digest, not file stream.

Anyone got solution?
I am looking at '_hashopenssl.c '. If you can find the implementation
of EVP_DigestUpdat e, I'll give it a shot to help you write a ctypes
hack to store and write its state.
Oct 2 '08 #7
est
On Oct 2, 4:22*pm, "Aaron \"Castironpi \" Brady" <castiro...@gma il.com>
wrote:
On Oct 2, 2:44*am, est <electronix...@ gmail.comwrote:


On Oct 2, 1:51*pm, "James Mills" <prolo...@short circuit.net.auw rote:
On Thu, Oct 2, 2008 at 3:34 PM, est <electronix...@ gmail.comwrote:
wow. It's giga-size file. I need stream reading it, md5 it. It may
break for a while.
So use generators and consume the stream ?
--JamesMills
--
--
-- "Problems are solved by method"
no, I need to serialize half-finished digest, not file stream.
Anyone got solution?

I am looking at '_hashopenssl.c '. *If you can find the implementation
of EVP_DigestUpdat e, I'll give it a shot to help you write a ctypes
hack to store and write its state.- Hide quoted text -

- Show quoted text -

http://cvs.openssl.org/fileview?f=op...o/evp/digest.c

int EVP_DigestUpdat e(EVP_MD_CTX *ctx, const void *data,
size_t count)
{
#ifdef OPENSSL_FIPS
FIPS_selftest_c heck();
#endif
return ctx->digest->update(ctx,dat a,count);
}
is this one?
Oct 2 '08 #8
En Wed, 01 Oct 2008 16:50:05 -0300, est <el***********@ gmail.comescrib ió:
>>>import md5
a=md5.md5( )
import pickle
pickle.dumps (a)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Python25\li b\pickle.py", line 1366, in dumps
Pickler(file, protocol).dump( obj)
File "C:\Python25\li b\pickle.py", line 224, in dump
self.save(obj)
File "C:\Python25\li b\pickle.py", line 306, in save
rv = reduce(self.pro to)
File "C:\Python25\li b\copy_reg.py", line 69, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle HASH objects

Why can't I pickle a md5 object? Is it because md5 algorithm needs to
read 512-bits at a time?

I need to md5() some stream, pause(python.ex e quits), and resume
later. It seems that the md5 and hashlib in std module could not be
serialized?
Yep, they're implemented in C and have no provision for serializing.
If you can use the old _md5 module, it is far simpler to serialize; a
md5object just contains a small struct with 6 integers and 64 chars, no
pointers.

With some help from ctypes (and a lot of black magic!) one can extract the
desired state, and restore it afterwards:

--- begin code ---
import _md5
import ctypes

assert _md5.MD5Type.__ basicsize__==96

def get_md5_state(m ):
if type(m) is not _md5.MD5Type:
raise TypeError, 'not a _md5.MD5Type instance'
return ctypes.string_a t(id(m)+8, 88)

def set_md5_state(m , state):
if type(m) is not _md5.MD5Type:
raise TypeError, 'not a _md5.MD5Type instance'
if not isinstance(stat e,str):
raise TypeError, 'state must be str'
if len(state)!=88:
raise ValueError, 'len(state) must be 88'
a88 = ctypes.c_char*8 8
pstate = a88(*list(state ))
ctypes.memmove( id(m)+8, ctypes.byref(ps tate), 88)

--- end code ---

pym1 = _md5.new()
pym1.update("th is is a ")
pys = get_md5_state(m 1)
pydel m1
py>
pym2 = _md5.new()
pyset_md5_state (m2, s)
pym2.update("sh ort test")
pyprint m2.hexdigest()
95ad1986e9a9f19 615cea00b7a44b9 12
pyprint _md5.new("this is a short test").hexdiges t()
95ad1986e9a9f19 615cea00b7a44b9 12

The code above was only tested with Python 2.5.2 on Windows, not more than
you can see. It might or might not work with other versions or platforms.
It may even create a (small) black hole and eat your whole town. Use at
your own risk.

--
Gabriel Genellina

Oct 2 '08 #9
On Oct 2, 4:03*am, est <electronix...@ gmail.comwrote:
On Oct 2, 4:22*pm, "Aaron \"Castironpi \" Brady" <castiro...@gma il.com>
wrote:
On Oct 2, 2:44*am, est <electronix...@ gmail.comwrote:
On Oct 2, 1:51*pm, "James Mills" <prolo...@short circuit.net.auw rote:
On Thu, Oct 2, 2008 at 3:34 PM, est <electronix...@ gmail.comwrote:
wow. It's giga-size file. I need stream reading it, md5 it. It may
break for a while.
So use generators and consume the stream ?
--JamesMills
--
--
-- "Problems are solved by method"
no, I need to serialize half-finished digest, not file stream.
Anyone got solution?
I am looking at '_hashopenssl.c '. *If you can find the implementation
of EVP_DigestUpdat e, I'll give it a shot to help you write a ctypes
hack to store and write its state.- Hide quoted text -
- Show quoted text -

http://cvs.openssl.org/fileview?f=op...o/evp/digest.c

int EVP_DigestUpdat e(EVP_MD_CTX *ctx, const void *data,
* * * * * * *size_t count)
* * * * {
#ifdef OPENSSL_FIPS
* * * * FIPS_selftest_c heck();
#endif
* * * * return ctx->digest->update(ctx,dat a,count);
* * * * }

is this one?
Oops, I needed 'EVP_MD_CTX'. I went Googling and found it.

http://www.google.com/codesearch?hl=...nssl/evp.h#l51

But does Gabriel's work for you?
Oct 2 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
5102
by: Simon Burton | last post by:
Hi, I am pickling big graphs of data and running into this problem: File "/usr/lib/python2.2/pickle.py", line 225, in save f(self, object) File "/usr/lib/python2.2/pickle.py", line 414, in save_list save(element) File "/usr/lib/python2.2/pickle.py", line 219, in save
8
8084
by: Hans Georg Krauthaeuser | last post by:
Dear all, I have a long running application (electromagnetic compatibility measurements in mode-stirred chambers over GPIB) that use pickle (cPickle) to autosave a class instance with all the measured data from time to time. At the beginning, pickling is quite fast but when the data becomes more and more pickling slows down rapidly.
7
3960
by: ‘5ÛHH575-UAZWKVVP-7H2H48V3 | last post by:
(see end of message for example code) When an instance has a dynamically assigned instance method, deepcopy throws a TypeError with the message "TypeError: instancemethod expected at least 2 arguments, got 0". Tested with Python 2.3.4 on OpenBSD and Python 2.4 on Win98; same results. Is this a bug in deepcopy, how I dynamically assign the...
3
2818
by: fdsl ysnh | last post by:
--- python-list-request@python.orgдµÀ: > Send Python-list mailing list submissions to > python-list@python.org > > To subscribe or unsubscribe via the World Wide Web, > visit > http://mail.python.org/mailman/listinfo/python-list > or, via email, send a message with subject or body > 'help' to
12
1525
by: sandravandale | last post by:
It's important that I can read the contents of the dict without flagging it as modified, but I want it to set the flag the moment I add a new element or alter an existing one (the values in the dict are mutable), this is what makes it difficult. Because the values are mutable I don't think you can tell the difference between a read and a...
2
1613
by: DurumDara | last post by:
Hi ! I want to create a database from datas. I want to store my datas in lists/dicts/normal variables. I thinking about that I can use the pickle to serialize/load my datas from the file. But: I remember that in the year of 2004(?) I tried this thing. I store my CD informations in pickled objects (in files).
2
2136
by: Steven Bethard | last post by:
I'd like to be able to pickle instancemethod objects mainly because I want to be able to delay a call like ``foo(spam, badger)`` by dumping ``foo``, ``spam`` and ``badger`` to disk and loading them again later. Sometimes the callable ``foo`` is actually a bound method, e.g. ``bar.baz``, but in such cases, pickle doesn't work by default because...
5
92981
by: Chris | last post by:
Why can pickle serialize references to functions, but not methods? Pickling a function serializes the function name, but pickling a staticmethod, classmethod, or instancemethod generates an error. In these cases, pickle knows the instance or class, and the method, so what's the problem? Pickle doesn't serialize code objects, so why can't it...
1
1585
by: Gabriel Genellina | last post by:
En Sun, 18 May 2008 00:14:19 -0300, Guillaume Bog <guibog@gmail.comescribió: A shelve is just a persistent dictionary that uses pickle to store the objects. If you want to store one or a few objects, using pickle directly may be easier. Any problem you may have with pickle (nonpickleable objects, security risks) will happen with shelve too....
0
7860
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main...
0
8139
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. ...
1
7890
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For...
0
8166
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the...
1
5667
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules. He will explain when you may want to use classes...
0
3799
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in...
0
3813
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2298
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
0
1127
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.