Hi!
I identified a bottleneck in my programs.
I just want to "encrypt" data by easy xoring. Ok - that's no
encryption at all - I know. But it's hardly readable - and that's
enough :-) Just some quick obscurity.
It turns out not to be quick at all. I really didn't expect this to be
a bottleneck, but it takes quite some time.
Here's the code: $ cat python/EasyCrypt.py #! /usr/bin/env python import operator def xorcrypt(str, salt = 255): if salt > 255: raise "Invalid salt! Must be < 255!" return reduce(lambda x,y: operator.add(x, chr(y)), map(lambda char, _salt = salt: operator.xor(or d(char), _salt), str), "")
xor'ing medium sized-files takes long time. For example a 360
kByte-File takes:
$ time ./just_crypt.py Userdatan/ScanImage01.jpg > bert real 1m52.138s user 0m40.320s sys 1m6.030s
on my 2.66 GHz P4 machine!
Hmmm, do you have some better implementation ideas? Some optimizing
tricks? (Besides coding in C to avoid immutable string problems)
I already took the operator module to speed up a bit - but it seems
that's not enough...
Thanks
Tino 7 2746
Tino Lange wrote: It turns out not to be quick at all. I really didn't expect this to be a bottleneck, but it takes quite some time. return reduce(lambda x,y: operator.add(x, chr(y)), map(lambda char, _salt = salt: operator.xor(or d(char), _salt), str), "")
Running this on a large string builds up a huge list of ints,
that you are converting to chars and then concatenating them
together using +... this creates a HUGE number of temporary
string objects.
The usual pattern of fast string joining is:
''.join(list-of-fragments)
So first try:
return ''.join(map(lam bda char, _salt = salt: chr(operator.xo r(ord(char), _salt)), string))
This runs MUCH faster already.
But the version I'd recommend is:
def xorcrypt(string , salt = 255):
if salt <0 or salt> 255:
raise "Invalid salt! Must be 0<=salt<=255!"
return ''.join( [ chr(ord(c) ^ salt) for c in string ] )
because
1) salt must be 0..255 not only <=255
2) forget about map & lambda, use a list comprehension.
That implementation runs about 20 times faster than your original one;
0.11 seconds for 100 Kb source data. (python 2.3)
HTH,
--Irmen de Jong
On Wed, 30 Jul 2003 00:25:59 +0200, Irmen de Jong <irmen@-NOSPAM-REMOVETHIS-xs4all.nl> wrote: Tino Lange wrote:
It turns out not to be quick at all. I really didn't expect this to be a bottleneck, but it takes quite some time. return reduce(lambda x,y: operator.add(x, chr(y)), map(lambda char, _salt = salt: operator.xor(or d(char), _salt), str), "")
Running this on a large string builds up a huge list of ints, that you are converting to chars and then concatenating them together using +... this creates a HUGE number of temporary string objects. The usual pattern of fast string joining is:
''.join(list-of-fragments)
So first try:
return ''.join(map(lam bda char, _salt = salt: chr(operator.xo r(ord(char), _salt)), string))
This runs MUCH faster already.
But the version I'd recommend is:
def xorcrypt(string , salt = 255):
def xorcrypt(s, salt = 255): # better name choice, even though string module may not be used if salt <0 or salt> 255: raise "Invalid salt! Must be 0<=salt<=255!" return ''.join( [ chr(ord(c) ^ salt) for c in string ] )
return s.translate(''. join([chr(ic^salt) for ic in xrange(256)])) because 1) salt must be 0..255 not only <=255 2) forget about map & lambda, use a list comprehension.
forget about list comprehension, use str.translate ;-) That implementation runs about 20 times faster than your original one; 0.11 seconds for 100 Kb source data. (python 2.3)
s.translate ought to a good deal faster yet ;-)
Regards,
Bengt Richter
Tino Lange <tl*****@nexgo. de> writes: Hmmm, do you have some better implementation ideas? Some optimizing tricks? (Besides coding in C to avoid immutable string problems) I already took the operator module to speed up a bit - but it seems that's not enough...
Use the array module. See <http://www.nightsong.c om/phr/crypto/p2.py>.
On Wed, Jul 30, 2003 at 12:03:06AM +0200, Tino Lange wrote: Hi!
I identified a bottleneck in my programs.
I just want to "encrypt" data by easy xoring. Ok - that's no encryption at all - I know. But it's hardly readable - and that's enough :-) Just some quick obscurity.
It turns out not to be quick at all. I really didn't expect this to be a bottleneck, but it takes quite some time.
If you want higher performance always try to use things that operate
on larger chunks. When you do things byte-by-byte you start to notice the
fact that Python is really an interpreter.
As noted by Bengt Richter xoring with a constant value can be done by
str.translate. It doesn't work for variable values, though.
This code does around 250kb/second on a Pentium 800. XORing is done 32
bits at a time. Conversion to and from character strings is done in even
larger chunks using the array module instead of using ord() and chr().
Oren
from __future__ import generators
import sha
def xor_stream_to_a rrays(fin, seed, hashfunc=sha):
""" fin is a file-like object.
yields arrays that may be written to a stream """
from array import array
h = hashfunc.new(se ed)
maskchunk = h.digest()
chunksize = len(maskchunk)
while True:
datachunk = fin.read(chunks ize)
if len(datachunk) < chunksize:
break
yield array('l', [x^y for (x,y) in zip(
array('l', maskchunk),
array('l', datachunk))])
h.update('x')
maskchunk = h.digest()
maskchunk = maskchunk[:len(datachunk)] # trim to length of remainder
# do the rest by bytes:
yield array('b', [x^y for (x,y) in zip(
array('b', maskchunk),
array('b', datachunk))])
def xor_stream_to_s tream(fin, fout, seed):
""" fin, fout are file-like objects """
for a in xor_stream_to_a rrays(fin, seed):
fout.write(buff er(a))
def xor_string_to_s tring(s, seed):
""" gets a string, returns a string """
from cStringIO import StringIO
fin = StringIO(s)
fout = StringIO()
xor_stream_to_s tream(fin, fout, seed)
return fout.getvalue()
Tino Lange <tl*****@nexgo. de> writes: And it seems that Bengt's reciepe is the fastest. For very small strings (<255 chars) the method irmen2 should be the best choice - it doesn' have to pre-create the translation-table and does everything on-the-fly.
You should be able to use the array module to do the xor's 4 bytes at
a time and get a speedup over the 1-byte version. The
string.translat e version is the fastest, of course, but depends on
using the same translation table for every char in the string.
If you want to encrypt in python, try the p2.py that I posted; it's
been carefully designed with good algorithms and fairly well optimized
and should give much better security than some roll-your-own method.
At 12:25 AM 7/30/2003 +0200, Irmen de Jong wrote: Tino Lange wrote:
It turns out not to be quick at all. I really didn't expect this to be a bottleneck, but it takes quite some time. return reduce(lambda x,y: operator.add(x, chr(y)), map(lambda char, _salt = salt: operator.xor(or d(char), _salt), str), "")
Running this on a large string builds up a huge list of ints, that you are converting to chars and then concatenating them together using +... this creates a HUGE number of temporary string objects. The usual pattern of fast string joining is:
''.join(list-of-fragments)
So first try:
return ''.join(map(lam bda char, _salt = salt: chr(operator.xo r(ord(char), _salt)), string))
This runs MUCH faster already.
But the version I'd recommend is:
def xorcrypt(string , salt = 255): if salt <0 or salt> 255: raise "Invalid salt! Must be 0<=salt<=255!" return ''.join( [ chr(ord(c) ^ salt) for c in string ] )
Great minds think alike? I came up with (independently! ):
return ''.join([chr(ord(char) ^ salt) for char in txt])
I also favor comprehension because it is more readable.
because 1) salt must be 0..255 not only <=255 2) forget about map & lambda, use a list comprehension.
That implementation runs about 20 times faster than your original one; 0.11 seconds for 100 Kb source data. (python 2.3)
HTH, --Irmen de Jong
-- http://mail.python.org/mailman/listinfo/python-list
--- Incoming mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.500 / Virus Database: 298 - Release Date: 7/10/2003
Bob Gailer bg*****@alum.rp i.edu
303 442 2625
---
Outgoing mail is certified Virus Free.
Checked by AVG anti-virus system ( http://www.grisoft.com).
Version: 6.0.500 / Virus Database: 298 - Release Date: 7/10/2003
Tino Lange <tl*****@nexgo. de> writes: Thanks! But BTW your "time-bomb" and your comments in the file tell me that this script must not be used anymore...
Oh yeah. The code is ok, I just want to rename the function and
release it as p3.py. I haven't gotten around to that because nobody
seems to be using it. I keep forgetting. Anyway I'd appreciate it if
you don't distribute p2.py to other people with the time bomb removed,
but feel free to remove it for your own use. This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Peter Clark |
last post by:
Think of something like MyYahoo: a personalized portal with news
aggregator, weather forecast, comics, etc. Now instead of visiting a
web site, think of all of it being sent daily as an email. It does
have a web interface, but mostly for selecting your content: this is
my location for weather, these are the news feeds I'm interested in,
these are the comics I like, save my preferences, and the server takes
care of the rest. Does such a...
|
by: joshsackett |
last post by:
I am starting an encryption project for my database and I'm performing
some tests on decryption speed. A lot of my application queries use a
LIKE parameter in the WHERE clause. To keep from changing my
application I am performing all the work on the back-end; creating
views, triggers and UDFs to encrypt/decrypt the data. A problem has
arisen around the LIKE parameter, though.
Currently:
SELECT SSN, FNAME, LNAME FROM USERS WHERE LNAME...
|
by: pigeon |
last post by:
SSL only works when ms sql server has "force encryption option"
turned on...
If I turn that off, and try to have the client software request
encryption.. I get a ssl security error msg.
The thing is.. I had this working before I upgraded to sp3andsp4and
install my new cert (I did the upgrades at the same time)
|
by: Brian Genisio |
last post by:
Hi all,
So, IE has a code obfuscation method that is specific to IE. I have
read that it has been easily decrypted. Has the decryption for this
method been published anywhere? Is it legal to do so, assuming the
authors of the code give my permission to read the code?
Before anyone flames me, I promise my motives are legitimate. I am
developing the DOM and JS interface for a browser that needs to act like IE.
|
by: Dica |
last post by:
i've used the sample code from msdn to create an encyption/decryption
assembly as found here:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnnetsec/html/SecNetHT10.asp
i'm able to encrypt and then decrypt data okay as in the following code:
// encrypt the data //
Encryptor enc = new Encryptor(EncryptionAlgorithm.TripleDes);
byte key = Encoding.ASCII.GetBytes("0123456789012345");
| |
by: Leon |
last post by:
How can I encrypted data sent across my website from web forms without using
SSL?
Such as on Login the user enter "EmailAddress" & "Password" and Simply
Registration Form
in which the user creates a Password, FirstName, LastName, etc.
I see site like Careerbuilder and Monster allow user to register, login, and
retrieve a lost password
without using a SSL connection "I Know anytime you deal with credit card
info you need a SSL.
Thanks!
|
by: Paul Tillotson |
last post by:
At my company we are looking at deploying clients for our client/server
app outside our firewall, which will then require our postgres box to be
internet-accessible.
Does anyone out there have experience with this or recommended best
practices? We have been looking at either (a) tunnelling everything
over ssh, or (b) just making sure that users have "strong" passwords and
requiring "md5" authentication in pg_hba.conf.
Our client app...
|
by: Ismail Fatih Yýldýrým |
last post by:
I modified the RSACSPSample from MSDN to try out a simple commutative
encryption model using RSA encryption but when i run the progrem the first
encryption command works but during the second encryption command (line :
encryptedData2 = RSAE...) i get a "Key not valid for use in specified
state." exception error even though i provide a valid second key to encrypt
it. How can i overcome this error and get double encryption to work ?
The...
|
by: Dan2kx |
last post by:
Hello to all,
im bored and like to waste time doing silly things that are not really neccessary...
i am however tying to complete a Holiday database for my Boss... looking for sum brownie points tehe
so what i have so far is a logon form, straight forward and boring, does username match the list and the password match that stored, one thing i noticed is that the password match is not case sensitive.... so HeLlO would match hello/hElLo...
|
by: marktang |
last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look !
Part I. Meaning of...
|
by: Hystou |
last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it.
First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
| |
by: Oralloy |
last post by:
Hello folks,
I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>".
The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed.
This is as boiled down as I can make it.
Here is my compilation command:
g++-12 -std=c++20 -Wnarrowing bit_field.cpp
Here is the code in...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: Hystou |
last post by:
Overview:
Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
|
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
| |
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
| |