I could use some help making this Python code run faster using only Python code.

[Python Maniac]

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

def scrambleLine(li ne):
s = ''
for c in line:
s += chr(ord(c) | 0x80)
return s

def descrambleLine( line):
s = ''
for c in line:
s += chr(ord(c) & 0x7f)
return s

def scrambleFile(fn ame,action=1):
if (path.exists(fn ame)):
try:
f = open(fname, "r")
toks = fname.split('.' )
while (len(toks) 2):
toks.pop()
fname = '.'.join(toks)
if (action == 1):
_fname = fname + '.scrambled'
elif (action == 0):
_fname = fname + '.descrambled'
if (path.exists(_f name)):
os.remove(_fnam e)
ff = open(_fname, "w+")
if (action == 1):
for l in f:
ff.write(scramb leLine(l))
elif (action == 0):
for l in f:
ff.write(descra mbleLine(l))
except Exception, details:
print 'ERROR :: (%s)' % details
finally:
f.close()
ff.close()
else:
print 'WARNING :: Missing file "%s" - cannot continue.' % fname

[Paul Hankin]

On Sep 20, 10:59 pm, Python Maniac <raych...@hotma il.comwrote:

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

def scrambleLine(li ne):
s = ''
for c in line:
s += chr(ord(c) | 0x80)
return s

def descrambleLine( line):
s = ''
for c in line:
s += chr(ord(c) & 0x7f)
return s
...

Well, scrambleLine will remove line-endings, so when you're
descrambling
you'll be processing the entire file at once. This is particularly bad
because of the way your functions work, adding a character at a time
to
s.

Probably your easiest bet is to iterate over the file using read(N)
for some small N rather than doing a line at a time. Something like:

process_bytes = (descrambleLine , scrambleLine)[action]
while 1:
r = f.read(16)
if not r: break
ff.write(proces s_bytes(r))

In general, rather than building strings by starting with an empty
string and repeatedly adding to it, you should use ''.join(...)

For instance...
def descrambleLine( line):
return ''.join(chr(ord (c) & 0x7f) for c in line)

def scrambleLine(li ne):
return ''.join(chr(ord (c) | 0x80) for c in line)

It's less code, more readable and faster!

--
Paul Hankin

[Diez B. Roggisch]

Python Maniac schrieb:

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

def scrambleLine(li ne):
s = ''
for c in line:
s += chr(ord(c) | 0x80)
return s

def descrambleLine( line):
s = ''
for c in line:
s += chr(ord(c) & 0x7f)
return s

These might benefit from using a lookup-dictionary that maps ord(c) to
the outcome of the operation. Actually, it becomes twice as fast:

import time

lt = {}
for i in xrange(255):
lt[chr(i)] = chr(i | 0x80)

def scrambleLine(li ne):
s = ''
for c in line:
s += chr(ord(c) | 0x80)
return s

def scrambleLineLU( line):
s = ''
for c in line:
s += lt[c]
return s
if __name__ == "__main__":
line = "abcdefghijklmn op" * 1000000
start = time.time()
scrambleLine(li ne)
print time.time() - start

start = time.time()
scrambleLineLU( line)
print time.time() - start

Diez

[Matt McCredie]

On 9/20/07, Python Maniac <ra******@hotma il.comwrote:

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

Well, you could save some time by not applying the scramble one line
at a time (that is if you don't mind losing the line endings in the
scrambled version). For that to be effective though, you probably want
to open in binary mode. Also, your scramble can be written using list
comprehension.

def scramble(s, key=0x80):
return ''.join([chr(ord(c) ^ key) for c in s])
 
output = scramble(f.read())
 

If you use xor (^) as above, you can use the same method for scramble
as descramble (running it again with the same key will descramble) and
you can use an arbitrary key. Though, with 255 combinations, it isn't
very strong encryption.

If you want stronger encryption you can use the following AESish algorithm:

import random
def scramble(s, key):
random.seed(key)
return ''.join([chr(ord(c) ^ random.randint(0,255)) for c in s])
 

This allows you to use much larger keys, but with a similar effect.
Still not strong enough to be unbreakable, but much better than the
origional. It is strong enough that someone knowing how you scrambled
it will have trouble unscrambling it even if they don't know the key.

Matt

[Tim Williams]

On 20/09/2007, Python Maniac <ra******@hotma il.comwrote:

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

def scrambleLine(li ne):
s = ''
for c in line:
s += chr(ord(c) | 0x80)
return s

def descrambleLine( line):
s = ''
for c in line:
s += chr(ord(c) & 0x7f)
return ''.join( [chr(ord(c) & 0x7f) for c in line] )

def scrambleFile(fn ame,action=1):
if (path.exists(fn ame)):
try:
f = open(fname, "r")
toks = fname.split('.' )
while (len(toks) 2):
toks.pop()
fname = '.'.join(toks)
if (action == 1):
_fname = fname + '.scrambled'
elif (action == 0):
_fname = fname + '.descrambled'
if (path.exists(_f name)):
os.remove(_fnam e)
ff = open(_fname, "w+")
if (action == 1):
for l in f:
ff.write(scramb leLine(l))
elif (action == 0):
for l in f:
ff.write(descra mbleLine(l))
except Exception, details:
print 'ERROR :: (%s)' % details
finally:
f.close()
ff.close()
else:
print 'WARNING :: Missing file "%s" - cannot continue.' % fname

--

def scrambleLine(li ne):
return ''.join( [chr(ord(c) | 0x80) for c in line] )

def descrambleLine( line):
return ''.join( [chr(ord(c) & 0x7f) for c in line] )

def scrambleFile(fn ame,action=1):
try:
f = open(fname, "r")
fname = '.'.join(fname. split('.')[:2])
if action:
_fname = fname + '.scrambled'
else:
_fname = fname + '.descrambled'
ff = open(_fname, "w")
if action:
ff.write('\r\n. join([scrambleLine(l) for l in f ]))
else :
ff.write('\r\n. join([descrambleLine( l) for l in f ]))
f.close()
ff.close()
except Exception, details:
print 'ERROR :: (%s)' % details

HTH :)

[Tim Williams]

On 21/09/2007, Tim Williams <td*******@gmai l.comwrote:

On 20/09/2007, Python Maniac <ra******@hotma il.comwrote:

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

def scrambleLine(li ne):
s = ''
for c in line:
s += chr(ord(c) | 0x80)
return s

def descrambleLine( line):
s = ''
for c in line:
s += chr(ord(c) & 0x7f)
return ''.join( [chr(ord(c) & 0x7f) for c in line] )

def scrambleFile(fn ame,action=1):
if (path.exists(fn ame)):
try:
f = open(fname, "r")
toks = fname.split('.' )
while (len(toks) 2):
toks.pop()
fname = '.'.join(toks)
if (action == 1):
_fname = fname + '.scrambled'
elif (action == 0):
_fname = fname + '.descrambled'
if (path.exists(_f name)):
os.remove(_fnam e)
ff = open(_fname, "w+")
if (action == 1):
for l in f:
ff.write(scramb leLine(l))
elif (action == 0):
for l in f:
ff.write(descra mbleLine(l))
except Exception, details:
print 'ERROR :: (%s)' % details
finally:
f.close()
ff.close()
else:
print 'WARNING :: Missing file "%s" - cannot continue.' % fname

--

def scrambleLine(li ne):
return ''.join( [chr(ord(c) | 0x80) for c in line] )

def descrambleLine( line):
return ''.join( [chr(ord(c) & 0x7f) for c in line] )

def scrambleFile(fn ame,action=1):
try:
f = open(fname, "r")
fname = '.'.join(fname. split('.')[:2])
if action:
_fname = fname + '.scrambled'
else:
_fname = fname + '.descrambled'
ff = open(_fname, "w")
if action:
ff.write('\r\n. join([scrambleLine(l) for l in f ]))
else :
ff.write('\r\n. join([descrambleLine( l) for l in f ]))
f.close()
ff.close()
except Exception, details:
print 'ERROR :: (%s)' % details

HTH :)

or maybe even this:

Apologies for the self-reply, its late here !
(a couple of typos fixed too! )

def scrambleFile(fn ame,action=1):
try:
f = open(fname, "r")
fname = '.'.join(fname. split('.')[:2])
if action:
ff = open(fname + '.scrambled', "w")
ff.write('\r\n' .join([scrambleLine(l) for l in f ]))
else :
ff = open(fname + '.descrambled', "w")
ff.write('\r\n' .join([descrambleLine( l) for l in f ]))
f.close()
ff.close()

except Exception, details:
print 'ERROR :: (%s)' % details

:)

[Python Maniac]

On Sep 20, 3:57 pm, "Matt McCredie" <mccre...@gmail .comwrote:

On 9/20/07, Python Maniac <raych...@hotma il.comwrote:

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

Well, you could save some time by not applying the scramble one line
at a time (that is if you don't mind losing the line endings in the
scrambled version). For that to be effective though, you probably want
to open in binary mode. Also, your scramble can be written using list
comprehension.

def scramble(s, key=0x80):
   return ''.join([chr(ord(c) ^ key) for c in s])

output = scramble(f.read())
 

If you use xor (^) as above, you can use the same method for scramble
as descramble (running it again with the same key will descramble) and
you can use an arbitrary key. Though, with 255 combinations, it isn't
very strong encryption.

If you want stronger encryption you can use the following AESish algorithm:

import random
def scramble(s, key):
    random.seed(key)
    return ''.join([chr(ord(c) ^ random.randint(0,255)) for c in s])
 

This allows you to use much larger keys, but with a similar effect.
Still not strong enough to be unbreakable, but much better than the
origional. It is strong enough that someone knowing how you scrambled
it will have trouble unscrambling it even if they don't know the key.

Matt

So far I like what was said in this reply however my ultimate goal is
to allow the scramble method to be more than what is shown above.

I considered using XOR however XOR only works for the case shown above
where the scramble method simply sets or removes the MSB.

What I want to be able to do is set or reset the MSB in addition to
performing a series of additional steps without negatively impacting
performance for the later cases when the MSB is not the only technique
being employed.

Hopefully this sheds more light on the goal I have in mind.

BTW - My original code is able to scramble a 20 MB file in roughly 40
secs using a Dell E6600 2.4 GHz. When I began writing the code for
this problem my best runtime was about 65 secs so I know I was heading
in the right direction.

I was about to begin the process of using the D Language and pyd to
gain better performance but then I thought I might also take this
opportunity to learn something about Python before delving into D.
Obviously I could simply code the whole process using D but that
defeats the purpose for using Python in the first place and so I would
tend to limit my low-level coding to the task of scrammbling each line
of text.

The ironic thing about this exorcise was the fact that the
optimization techniques that worked for Python caused the Ruby version
I coded to decrease performance. It seems Ruby has some definite
ideas about what it feels is optimal that it ain't got much to do with
what I would consider to be traditional optimization techniques where
Ruby is concerned. For instance, Ruby felt the putc method was less
optimal than the write method which makes no sense to me but that's
life with Ruby.

[mensanator]

On Sep 20, 5:46 pm, Paul Hankin <paul.han...@gm ail.comwrote:

On Sep 20, 10:59 pm, Python Maniac <raych...@hotma il.comwrote:

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

def scrambleLine(li ne):
s = ''
for c in line:
s += chr(ord(c) | 0x80)
return s

def descrambleLine( line):
s = ''
for c in line:
s += chr(ord(c) & 0x7f)
return s
...

Well, scrambleLine will remove line-endings, so when you're
descrambling
you'll be processing the entire file at once. This is particularly bad
because of the way your functions work, adding a character at a time
to
s.

Probably your easiest bet is to iterate over the file using read(N)
for some small N rather than doing a line at a time. Something like:

process_bytes = (descrambleLine , scrambleLine)[action]
while 1:
r = f.read(16)
if not r: break
ff.write(proces s_bytes(r))

In general, rather than building strings by starting with an empty
string and repeatedly adding to it, you should use ''.join(...)

For instance...
def descrambleLine( line):
return ''.join(chr(ord (c) & 0x7f) for c in line)

def scrambleLine(li ne):
return ''.join(chr(ord (c) | 0x80) for c in line)

It's less code, more readable and faster!

I would have thought that also from what I've heard here.

def scrambleLine(li ne):
s = ''
for c in line:
s += chr(ord(c) | 0x80)
return s

def scrambleLine1(l ine):
return ''.join([chr(ord(c) | 0x80) for c in line])

if __name__=='__ma in__':
from timeit import Timer
t = Timer("scramble Line('abcdefghi jklmnopqrstuvwx yz')", "from
__main__ import scrambleLine")
print t.timeit()

## scrambleLine
## 13.0013366039
## 12.9461998318
##
## scrambleLine1
## 14.4514098748
## 14.3594400695

How come it's not? Then I noticed you don't have brackets in
the join statement. So I tried without them and got

## 17.6010847978
## 17.6111472418

Am I doing something wrong?

>
--
Paul Hankin

[Michael Spencer]

Python Maniac wrote:

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

def scrambleLine(li ne):
s = ''
for c in line:
s += chr(ord(c) | 0x80)
return s

def descrambleLine( line):
s = ''
for c in line:
s += chr(ord(c) & 0x7f)
return s

Try using str.translate instead - it's usually much faster than a pure python loop:

>>scramble = "".join(chr (c | 0x80) for c in range(256))
"source text".translate (scramble)

'\xf3\xef\xf5\x f2\xe3\xe5\xa0\ xf4\xe5\xf8\xf4 '

>>descramble = "".join(chr (c & 0x7F) for c in range(256))
'\xf3\xef\xf5 \xf2\xe3\xe5\xa 0\xf4\xe5\xf8\x f4'.translate(d escramble)

'source text'

>>>

You might then do the translation inline e.g., untested:

def scrambleFile(fn ame,action=1):

translators = {0: descramble, 1: scramble} # defined as above
try:
translation_act ion = translators[action]
except KeyError:
raise ValueError("act ion must be 0 or 1")

if (path.exists(fn ame)):

....
....
for line in f:
ff.write(line.t ranslate(transl ation_action))
....

HTH
Michael

[Paul Rubin]

Python Maniac <ra******@hotma il.comwrites:

I am new to Python however I would like some feedback from those who
know more about Python than I do at this time.

Use the array module and do 32-bit or 64-bit operations with it.
See http://nightsong.com/phr/crypto/p3.py for a more serious
encryption module written that way.