473,594 Members | 2,870 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

xor: how come so slow?

Hi,
I'm trying to encode a byte data. Let's not focus on the process of
encoding; in fact, I want to emphasize that the method
create_random_b lock takes 0.5s to be executed (even Java it's faster) on
a Dual-Core 3.0Ghz machine:

took 46.746999979s, avg: 0.46746999979s

Thus I suppose that the xor operation between bytes raise the execution
time to 0.5; why I suppose that?
Because in Python there's no support for bytes and even for xoring
bytes, so I used a workaround:
I cycle on the two strings to be xor-red
for every char in the strings
convert one char on integer and then xor them; (ord)
insert one char in the result, transforming the previous integer
in char (chr)

I suppose that ord() and char() are the main problems of my
implementation, but I don't know either way to xor two bytes of data
(bytes are represented as strings).
For more information, see the code attached.

How should I decrease the execution time?

Thank you
from __future__ import division
import random
import time
import sha
import os

class Encoder(object) :
def create_random_b lock(self, data, seed, blocksize):
number_of_block s = int(len(data)/blocksize)
random.seed(see d)
random_block = ['0'] * blocksize
for index in range(number_of _blocks):
if int(random.getr andbits(1)) == 1:
block = data[blocksize*index :blocksize*inde x+blocksize]
for bit in range(len(block )):
random_block[bit] =
chr(ord(random_ block[bit])^ord(block[bit])) # workaround per fare xor
bit a bit di str; xor e' solo supportato per int -ord
return ''.join(random_ block)
x = Encoder()
piece = os.urandom(1024 *1024)
blocksize = 16384
t1 = time.time()
for l in range(100):
seed = random.getrandb its(32)
block = x.create_random _block(piece, seed, blocksize)
t2 = time.time()
print 'took ' + str(t2-t1) + 's, avg: ' + str((t2-t1)/100.0) + 's'
Oct 15 '08 #1
21 2476
My answer is: never do things like this with python.
You will find this module useful: www.pycrypto.org

On Oct 15, 12:19*pm, Michele <mich...@nectar ine.itwrote:
Hi,
I'm trying to encode a byte data. Let's not focus on the process of
encoding; in fact, I want to emphasize that the method
create_random_b lock takes 0.5s to be executed (even Java it's faster) on
a Dual-Core 3.0Ghz machine:

took 46.746999979s, avg: 0.46746999979s

Thus I suppose that the xor operation between bytes raise the execution
time to 0.5; why I suppose that?
Because in Python there's no support for bytes and even for xoring
bytes, so I used a workaround:
I cycle on the two strings to be xor-red
* * for every char in the strings
* * * * convert one char on integer and then xor them; (ord)
* * * * insert one char in the result, transforming the previous integer
in char (chr)

I suppose that ord() and char() are the main problems of my
implementation, but I don't know either way to xor two bytes of data
(bytes are represented as strings).
For more information, see the code attached.

How should I decrease the execution time?

Thank you

from __future__ import division
import random
import time
import sha
import os

class Encoder(object) :
* * def create_random_b lock(self, data, seed, blocksize):
* * * * number_of_block s = int(len(data)/blocksize)
* * * * random.seed(see d)
* * * * random_block = ['0'] * blocksize
* * * * for index in range(number_of _blocks):
* * * * * * if int(random.getr andbits(1)) == 1:
* * * * * * * * block = data[blocksize*index :blocksize*inde x+blocksize]
* * * * * * * * for bit in range(len(block )):
* * * * * * * * * * random_block[bit] =
chr(ord(random_ block[bit])^ord(block[bit])) # workaround per fare xor
bit a bit di str; xor e' solo supportato per int -ord
* * * * return ''.join(random_ block)

x = Encoder()
piece = os.urandom(1024 *1024)
blocksize = 16384
t1 = time.time()
for l in range(100):
* * seed = random.getrandb its(32)
* * block = x.create_random _block(piece, seed, blocksize)
t2 = time.time()
print 'took ' + str(t2-t1) + 's, avg: ' + str((t2-t1)/100.0) + 's'
Oct 15 '08 #2
Few suggestions for your code:
- Use xrange instead of range.
- Loop over lists where you can instead of their indexes.
- array.array("B" , somestring) may help you because it gives a byte
"view" of a string.
- Using psyco helps a lot for such kind of code.
- I think numpy arrays can contain text/chars too, so it may offer you
ways to speed up your code a lot.
- Generally Python is fit to download pages from the net or to act as
glue between different subsystems, or to do bulk string processing,
etc, but for grunt low-level works like this it's often too much slow,
and you can use other lower-level languages.
- You can use a lib already written, or use an extension, for example
you can try ShedSkin, or Pyd.

Bye,
bearophile
Oct 15 '08 #3
On Oct 15, 10:19*pm, Michele <mich...@nectar ine.itwrote:
Hi,
I'm trying to encode a byte data. Let's not focus on the process of
encoding; in fact, I want to emphasize that the method
create_random_b lock takes 0.5s to be executed (even Java it's faster) on
a Dual-Core 3.0Ghz machine:

took 46.746999979s, avg: 0.46746999979s

Thus I suppose that the xor operation between bytes raise the execution
time to 0.5; why I suppose that?
Because in Python there's no support for bytes and even for xoring
bytes, so I used a workaround:
I cycle on the two strings to be xor-red
* * for every char in the strings
* * * * convert one char on integer and then xor them; (ord)
* * * * insert one char in the result, transforming the previous integer
in char (chr)

I suppose that ord() and char() are the main problems of my
implementation, but I don't know either way to xor two bytes of data
(bytes are represented as strings).
For more information, see the code attached.

How should I decrease the execution time?

Thank you

from __future__ import division
import random
import time
import sha
import os

class Encoder(object) :
* * def create_random_b lock(self, data, seed, blocksize):
* * * * number_of_block s = int(len(data)/blocksize)
* * * * random.seed(see d)
* * * * random_block = ['0'] * blocksize
You possibly mean '\0' i.e. the byte all of whose bits are zero.
* * * * for index in range(number_of _blocks):
* * * * * * if int(random.getr andbits(1)) == 1:
getrandbits(1) produces a *long* with one random bit. Any good reason
for preferring this to randrange(2) and randint(0, 1)?

So there's a 50% chance that this block will be XORed into the result;
is that what you intend?
* * * * * * * * block = data[blocksize*index :blocksize*inde x+blocksize]
You don't need to slice out block, certainly not so awkwardly.

* * * * * * * * for bit in range(len(block )):

Perhaps you mean "byte_index ", not "bit".

On my assumption that range(len(block )) is invariant: calculate it
once. That assumption is incorrect, so is your code for calculating
the number of blocks; it ignores a possible short block at the end.

* * * * * * * * * * random_block[bit] =
chr(ord(random_ block[bit])^ord(block[bit]))
The chr() and one ord() are utterly wasteful; leave random_block as a
list of ints and do the chr() thing in the return statement.
# workaround per fare xor
bit a bit di str; xor e' solo supportato per int -ord
* * * * return ''.join(random_ block)
this will become
return ''.join(map(chr , random_block))
or
return ''.join(chr(i) for i in random_block)
as taste or speed dictates :-)

So the whole thing becomes [not tested]:
def create_random_b lock(self, data, seed, blocksize):
datalen = len(data)
assert datalen % blocksize == 0
random.seed(see d)
random_block = [0] * blocksize
block_range = range(blocksize )
for start in xrange(0, datalen, blocksize):
if random.randrang e(2):
for x in block_range:
random_block[x] ^= ord(data[start + x])
return ''.join(map(chr , random_block))

Looks slightly more athletic than before :-)

BTW, +1 misleading subject of the week; it's not XOR that's slow!!

Cheers,
John
Oct 15 '08 #4
Michele wrote:
I'm trying to encode a byte data. Let's not focus on the process of
encoding; in fact, I want to emphasize that the method
create_random_b lock takes 0.5s to be executed (even Java it's faster) on
a Dual-Core 3.0Ghz machine:

took 46.746999979s, avg: 0.46746999979s
How should I decrease the execution time?
Use numpy. You should be able to swap in the following in your script

import numpy
from numpy.core import multiarray as ma
class Encoder(object) :
def create_random_b lock(self, data, seed, blocksize):
number_of_block s = len(data)//blocksize
random.seed(see d)
random_block = ma.fromstring(" 0"*blocksize , numpy.uint8)

for index in range(number_of _blocks):
if random.getrandb its(1):
block =
ma.fromstring(d ata[blocksize*index :blocksize*inde x+blocksize], numpy.uint8)
random_block ^= block
return random_block.to string()

There are absolutely no warranties as I'm just a random tinkerer when it
comes to numpy. But if it works you should get a nice speedup...

Peter
Oct 15 '08 #5
In message <48************ ***********@rea der5.news.tin.i t>, Michele wrote:
class Encoder(object) :
def create_random_b lock(self, data, seed, blocksize):
number_of_block s = int(len(data)/blocksize)
random.seed(see d)
random_block = ['0'] * blocksize
for index in range(number_of _blocks):
if int(random.getr andbits(1)) == 1:
block = data[blocksize*index :blocksize*inde x+blocksize]
for bit in range(len(block )):
random_block[bit] =
chr(ord(random_ block[bit])^ord(block[bit])) # workaround per fare xor
bit a bit di str; xor e' solo supportato per int -ord
return ''.join(random_ block)
OK, this function is randomly choosing blocks from data (of length
number_of_block s * blocksize), and xoring them together to produce a single
block of length blocksize.
piece = os.urandom(1024 *1024)
Is piece really meant to be random? If so, your create_random_b lock function
isn't achieving much--xoring random data together isn't going to produce
anything more exciting than less random data than you started with.
Oct 17 '08 #6
On Fri, 17 Oct 2008 20:51:37 +1300, Lawrence D'Oliveiro wrote:
Is piece really meant to be random? If so, your create_random_b lock
function isn't achieving much--xoring random data together isn't going
to produce anything more exciting than less random data than you started
with.

Hmmm... why do you say that xoring random data with other random data
produces less randomness than you started with?

I'm not saying that you're wrong, and certainly it is pointless since
you're not going to improve on the randomness of /dev/urandom without a
lot of work. But less random?

--
Steven
Oct 17 '08 #7
Michele <mi*****@nectar ine.itwrites:
I suppose that ord() and char() are the main problems
yes
How should I decrease the execution time?
See http://nightsong.com/phr/crypto/p3.py which deals with
the same problem by using the array module to do the xor's
32 bits at a time.
Oct 17 '08 #8
In message <01************ **********@news .astraweb.com>, Steven D'Aprano
wrote:
On Fri, 17 Oct 2008 20:51:37 +1300, Lawrence D'Oliveiro wrote:
>... why do you say that xoring random data with other random data
produces less randomness than you started with?
blocksize <= number_of_block s * blocksize
Oct 17 '08 #9
On Fri, 17 Oct 2008 22:45:19 +1300, Lawrence D'Oliveiro wrote:
In message <01************ **********@news .astraweb.com>, Steven D'Aprano
wrote:
>On Fri, 17 Oct 2008 20:51:37 +1300, Lawrence D'Oliveiro wrote:
>>... why do you say that xoring random data with other random data
produces less randomness than you started with?

blocksize <= number_of_block s * blocksize
I must be thick, because that looks like a non sequitor to me. I don't
see the relevance.

Of course, that's just another way of saying that:

1 <= number_of_block s

and I don't see how this relates to whether xoring random data with other
random data decreases the randomness available.
>>c1 = os.urandom(1)
c2 = os.urandom(1)
c3 = chr( ord(c1)^ord(c2) )
Is it your contention that c3 is more predictable than c1 or c2? If so,
why?

--
Steven
Oct 17 '08 #10

This thread has been closed and replies have been disabled. Please start a new discussion.

Similar topics

1
5244
by: Mike Mc | last post by:
Can anyone tell me an easy way to hide numbers using XOR as mask, i.e. I want to write a program that will get a number, then it will create two random numbers that when they are XOR'ed together will create the original number. e.g. Original number = 65 The program will then come up with two random numbers that fit the bill, i.e. 150 & 215.
50
6123
by: dataangel | last post by:
I wrote a function to compare whether two strings are "similar" because I'm using python to make a small text adventure engine and I want to it to be responsive to slight mispellings, like "inevtory" and the like. To save time the first thing my function does is check if I'm comparing an empty vs. a non-empty string, because they are to be never considered similar. Right now I have to write out the check like this: if str1 and str2: if...
20
400
by: JJK | last post by:
I'd tried to port an application from vb6 to vb.net 2003. While testing is I'd got a error in the XOR operator. This is an example: dim c as Byte .... mask = 190 Fileget(1,c) c = c XOR mask Fileput(2,c) Run 1:
80
35073
by: Christopher Benson-Manica | last post by:
Of course one can get the effect with appropriate use of existing operators, but a ^^ operator would make for nice symmetry (as well as useful to me in something I'm working on). Am I the only one who would find it useful? -- Christopher Benson-Manica | I *should* know what I'm talking about - if I ataru(at)cyberspace.org | don't, I need to know. Flames welcome.
5
6633
by: robert.h.lowe | last post by:
Hi, I'm looking for a computationally fast means of XOR folding the individual bytes of an unsigned 32-bit int, and an unsigned 16-bit int. Can I point a char * at the address of each of the int's (in succession), and safely access each byte? Danger, danger, Will Robinsin!?? Any better ideas?? I'm assuming this is faster than a series of 8-bit right shifts followed by ORs with 255 as input to the XOR. E.g.
6
1358
by: Chris | last post by:
Hi, Can someone explain the use of XOR? Thanks
11
5029
by: John Williams | last post by:
I've written a simple program to do XOR encryption as my first foray into understanding how encryption works. The code compiles fine, however it segmentation faults on every run. using gdb to debug it let me narrow the problem down to the Cipher function I think it faults at line 84 or 85. The program makes it's first read/cipher/write pass without issue but the second pass kills it. Using gdb to print the variables left showed me the...
0
7946
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, weíll explore What is ONU, What Is Router, ONU & Routerís main usage, and What is the difference between ONU and Router. Letís take a closer look ! Part I. Meaning of...
0
7877
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
8374
jinu1996
by: jinu1996 | last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth. The Art of Business Website Design Your website is...
1
8009
by: Hystou | last post by:
Overview: Windows 11 and 10 have less user interface control over operating system update behaviour than previous versions of Windows. In Windows 11 and 10, there is no way to turn off the Windows Update option using the Control Panel or Settings app; it automatically checks for updates and installs any it finds, whether you like it or not. For most users, this new feature is actually very convenient. If you want to control the update process,...
0
8240
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
3867
by: TSSRALBI | last post by:
Hello I'm a network technician in training and I need your help. I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs. The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols. I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
0
3903
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
2389
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
0
1216
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.