On 30/05/2006 2:57 AM, DurumDara wrote:
Hi !
I need to speedup my MD5/SHA1 calculator app that working on
filesystem's files.
I use the Python standard modules, but I think that it can be faster if
I use C, or other module for it.
I use FSUM before, but I got problems, because I "move" into "DOS area",
and the parameterizing of outer process maked me very angry (not working).
You will see this in this place:
http://mail.python.org/pipermail/pyt...ay/004697.html
So: I must handle unicode filenames. I think that if I find a library
that can working with py's unicode chars, and I can load and use it to
hash files, the code be better, and faster.
Anybody knows about same code ?
Py2.4, Windows, Py2Exe, wxPy... That was the specification.
Hello (again), dd ...
As the effbot has said, the Python md5 and sha modules are written in C.
Hints: (1) the helpfile index says "builtin module" (2) you don't find a
sha.py or md5.py in c:\Python24\Lib\
An md5/sha library will concern itself with strings (which you obtain
from a file's *contents*), just like Python's modules do. Any struggle
with Unicode characters in the *names* of files is a separate concern.
Let's all stop worrying about low-level things like getting the 8.3
filename so that you can pass it to an MS-DOS program, and let's try to
explore why you think there is a problem with your initial approach.
At the end of this posting is a very simple Python function that
calculates the hash of a file (and its length), given the name of the
file (str or unicode, doesn't matter), which hashing module to use, and
a blocksize to use when reading. There is a really flash :-) user
interface that allows you to try it with either a glob pattern "*.txt",
or (as glob doesn't grok Windows mbcs/unicode filenames) a single
utf8-encoded filename.
Please try it out. My expectation is that, with a suitable choice of
blocksize, you will not be able to find anything that is significantly
faster and won't be difficult to interface to (like the FSUM program!).
If you have any problems or more questions, please don't hesitate to ask.
HTH,
John
=== function and driver ===
C:\junk>type hashtestbed.py
def hash_of_file(hash_module, fname, block_size):
f = open(fname, 'rb')
hashobj = hash_module.new()
filesize = 0
while True:
block = f.read(block_size)
if not block: break
filesize += len(block)
hashobj.update(block)
f.close()
return (filesize, hashobj.digest())
def to_hex(s):
return ''.join('%02x' % ord(c) for c in s)
if __name__ == "__main__":
import sha, md5, time, sys, glob
# print sys.argv
mdlname = sys.argv[1]
mdl = {'sha': sha, 'md5': md5}[mdlname]
szs = sys.argv[2].lower()
factor = {'m': 1024*1024, 'k': 1024}.get(szs[-1], 1)
if factor == 1:
bsz = int(szs)
else:
bsz = int(szs[:-1]) * factor
filearg = sys.argv[3]
if filearg.startswith("'"):
# repr(single filename, encoded in utf8)
filenames = [eval(filearg).decode('utf8')]
# print filenames
else:
filenames = glob.glob(sys.argv[3])
# I'm entering the above for the "Best UI of the Year" award :-)
for fn in filenames:
t0 = time.time()
fsz, digest = hash_of_file(mdl, fn, bsz)
seconds = time.time() - t0
print "%s, %r, bksz %d: %d bytes," \
" %.2f secs (%.4f secs/MB)\n\thash = %s" \
% (mdlname, fn, bsz, fsz,
seconds, seconds/fsz*1024*1024, to_hex(digest))
C:\junk>
=== sample usage ===
C:\junk>hashtestbed.py md5 32k '\xe5\xbc\xa0\xe6\x95\x8f.txt'
md5, u'\u5f20\u654f.txt', bksz 32768: 17 bytes, 0.00 secs (0.0000 secs/MB)
hash = 746d0931605368989a20691a906a67f8
C:\junk>hashtestbed.py md5 32k \downloads\python*.msi
md5, '\\downloads\\python-2.4.2.msi', bksz 32768: 9671168 bytes, 0.08
secs (0.00
86 secs/MB)
hash = bfb6fc0704d225c7a86d4ba8c922c7f5
md5, '\\downloads\\python-2.4.3.msi', bksz 32768: 9688576 bytes, 0.06
secs (0.00
67 secs/MB)
hash = ab946459d7cfba4a8500f9ff8d35cc97
md5, '\\downloads\\python-2.5a2.msi', bksz 32768: 10274816 bytes, 0.05
secs (0.0
048 secs/MB)
hash = cedc1e1fed9c4cd137921a80485bf007
=== end ===
