471,357 Members | 1,130 Online
Bytes | Software Development & Data Engineering Community
Post +

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 471,357 software developers and data experts.

Convert string to command..

I want to convert a string to command..
For example i have a string:
a="['1']"
I want to do this list..
How can i do ?

Oct 18 '07 #1
35 4612
On Oct 18, 10:23 am, Abandoned <best...@gmail.comwrote:
I want to convert a string to command..
For example i have a string:
a="['1']"
I want to do this list..
How can i do ?
Use the builtin function "eval".

Oct 18 '07 #2
Abandoned wrote:
I want to convert a string to command..
For example i have a string:
a="['1']"
I want to do this list..
How can i do ?
The correct wording here would be expression. To evaluate expressions, there
is the function eval:

a = eval("['1']")

But beware: if the expression contains some potentially harmful code, it
will be executed. So it is generally considered bad style to use eval.

An example that fails would be

a = eval("10**2000**2000")

or such thing.

Another alternative is to use parsers like simplejson to extract the
information. This of course only works, if your expressions are valid json.

Diez
Oct 18 '07 #3
Thanks you all answer..
But "eval" is very slow at very big dictionary {2:3,4:5,6:19....}
(100.000 elements)
Is there any easy alternative ?

Oct 18 '07 #4
Abandoned wrote:
Thanks you all answer..
But "eval" is very slow at very big dictionary {2:3,4:5,6:19....}
(100.000 elements)
Is there any easy alternative ?
How big? How slow? For me, a 10000-element list takes 0.04 seconds to be
parsed. Which I find fast.

Diez
Oct 18 '07 #5
On Oct 18, 6:14 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Abandoned wrote:
Thanks you all answer..
But "eval" is very slow at very big dictionary {2:3,4:5,6:19....}
(100.000 elements)
Is there any easy alternative ?

How big? How slow? For me, a 10000-element list takes 0.04 seconds to be
parsed. Which I find fast.

Diez
173.000 dict elements and it tooks 2.2 seconds this very big time for
my project

Oct 18 '07 #6
Abandoned <be*****@gmail.comwrites:
173.000 dict elements and it tooks 2.2 seconds this very big time
for my project
If you're generating the string from Python, use cPickle instead.
Much faster:
>>import time
d = dict((i, i+1) for i in xrange(170000))
len(d)
170000
>>s=repr(d)
t0 = time.time(); d2 = eval(s); t1 = time.time()
t1-t0
1.5457899570465088
>>import cPickle as pickle
s = pickle.dumps(d, -1)
len(s)
1437693
>>t0 = time.time(); d2 = pickle.loads(s); t1 = time.time()
t1-t0
0.060307979583740234
>>len(d2)
170000

That is 25x speedup. Note that cPickle's format is binary. Using the
textual format makes for more readable pickles, but reduces the
speedup to "only" 9.5x on my machine.
P.S.
Before someone says that using pickle is unsafe, remember that he is
considering *eval* as the alternative. :-)
Oct 18 '07 #7
Abandoned wrote:
On Oct 18, 6:14 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
>Abandoned wrote:
Thanks you all answer..
But "eval" is very slow at very big dictionary {2:3,4:5,6:19....}
(100.000 elements)
Is there any easy alternative ?

How big? How slow? For me, a 10000-element list takes 0.04 seconds to be
parsed. Which I find fast.

Diez

173.000 dict elements and it tooks 2.2 seconds this very big time for
my project
Where does the data come from?

Diez
Oct 18 '07 #8
On Oct 18, 6:26 pm, Hrvoje Niksic <hnik...@xemacs.orgwrote:
Abandoned <best...@gmail.comwrites:
173.000 dict elements and it tooks 2.2 seconds this very big time
for my project

If you're generating the string from Python, use cPickle instead.
Much faster:
>import time
d = dict((i, i+1) for i in xrange(170000))
len(d)
170000
>s=repr(d)
t0 = time.time(); d2 = eval(s); t1 = time.time()
t1-t0
1.5457899570465088
>import cPickle as pickle
s = pickle.dumps(d, -1)
len(s)
1437693
>t0 = time.time(); d2 = pickle.loads(s); t1 = time.time()
t1-t0

0.060307979583740234>>len(d2)

170000

That is 25x speedup. Note that cPickle's format is binary. Using the
textual format makes for more readable pickles, but reduces the
speedup to "only" 9.5x on my machine.

P.S.
Before someone says that using pickle is unsafe, remember that he is
considering *eval* as the alternative. :-)

import cPickle as pickle
a="{2:3,4:6,2:7}"
s=pickle.dumps(a, -1)
g=pickle.loads(s);
print g
'{2:3,4:6,2:7}'

Thank you very much for your answer but result is a string ??

Oct 18 '07 #9
On Oct 18, 6:35 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Abandoned wrote:
On Oct 18, 6:14 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Abandoned wrote:
Thanks you all answer..
But "eval" is very slow at very big dictionary {2:3,4:5,6:19....}
(100.000 elements)
Is there any easy alternative ?
How big? How slow? For me, a 10000-element list takes 0.04 seconds to be
parsed. Which I find fast.
Diez
173.000 dict elements and it tooks 2.2 seconds this very big time for
my project

Where does the data come from?

Diez
Data come from database..
I want to cache to speed up my system and i save the dictionary to
database for speed up but eval is very slow for do this.
Not: 2.2 second only eval operation.

Oct 18 '07 #10
On Thu, 18 Oct 2007 08:41:30 -0700, Abandoned wrote:
import cPickle as pickle
a="{2:3,4:6,2:7}"
s=pickle.dumps(a, -1)
g=pickle.loads(s);
print g
'{2:3,4:6,2:7}'

Thank you very much for your answer but result is a string ??
In Python terms yes, strings in Python can contain any byte value. If you
want to put this into a database you need a BLOB column or encode it as
base64 or something similar more ASCII safe.

Ciao,
Marc 'BlackJack' Rintsch
Oct 18 '07 #11
On Oct 18, 6:51 pm, Marc 'BlackJack' Rintsch <bj_...@gmx.netwrote:
On Thu, 18 Oct 2007 08:41:30 -0700, Abandoned wrote:
import cPickle as pickle
a="{2:3,4:6,2:7}"
s=pickle.dumps(a, -1)
g=pickle.loads(s);
print g
'{2:3,4:6,2:7}'
Thank you very much for your answer but result is a string ??

In Python terms yes, strings in Python can contain any byte value. If you
want to put this into a database you need a BLOB column or encode it as
base64 or something similar more ASCII safe.

Ciao,
Marc 'BlackJack' Rintsch
'{2:3,4:6,2:7}' already in database, i select this and convert to real
dictionary..

Oct 18 '07 #12
Abandoned wrote:
On Oct 18, 6:35 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
>Abandoned wrote:
On Oct 18, 6:14 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Abandoned wrote:
Thanks you all answer..
But "eval" is very slow at very big dictionary {2:3,4:5,6:19....}
(100.000 elements)
Is there any easy alternative ?
>How big? How slow? For me, a 10000-element list takes 0.04 seconds to
be parsed. Which I find fast.
>Diez
173.000 dict elements and it tooks 2.2 seconds this very big time for
my project

Where does the data come from?

Diez

Data come from database..
I want to cache to speed up my system and i save the dictionary to
database for speed up but eval is very slow for do this.
Not: 2.2 second only eval operation.
Does the dictionary change often?

And you should store a pickle to the database then. Besides, making a
database-query of that size (after all, we're talking a few megs here) will
take a while as well - so are you SURE the 2.2 seconds are a problem? Or is
it just that you think they are?

Diez
Oct 18 '07 #13
On Oct 18, 6:57 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Abandoned wrote:
On Oct 18, 6:35 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Abandoned wrote:
On Oct 18, 6:14 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Abandoned wrote:
Thanks you all answer..
But "eval" is very slow at very big dictionary {2:3,4:5,6:19....}
(100.000 elements)
Is there any easy alternative ?
How big? How slow? For me, a 10000-element list takes 0.04 seconds to
be parsed. Which I find fast.
Diez
173.000 dict elements and it tooks 2.2 seconds this very big time for
my project
Where does the data come from?
Diez
Data come from database..
I want to cache to speed up my system and i save the dictionary to
database for speed up but eval is very slow for do this.
Not: 2.2 second only eval operation.

Does the dictionary change often?

And you should store a pickle to the database then. Besides, making a
database-query of that size (after all, we're talking a few megs here) will
take a while as well - so are you SURE the 2.2 seconds are a problem? Or is
it just that you think they are?

Diez- Hide quoted text -

- Show quoted text -
I'm very confused :(
I try to explain main problem...
I have a table like this:
id-1 | id-2 | value
23 24 34
56 68 66
56 98 32455
55 62 655
56 28 123
..... ( 3 millions elements)

I select where id=56 and 100.000 rows are selecting but this took 2
second. (very big for my project)
I try cache to speed up this select operation..
And create a cache table:
id-1 | all
56 {68:66, 98:32455, 62:655}

When i select where id 56 i select 1 row and its took 0.09 second but
i must convert text to dictionary..

Have you got any idea what can i do this conver operation ?
or
Have you got any idea what can i do cache for this table ?

Oct 18 '07 #14
Abandoned <be*****@gmail.comwrites:
import cPickle as pickle
a="{2:3,4:6,2:7}"
s=pickle.dumps(a, -1)
g=pickle.loads(s);
print g
'{2:3,4:6,2:7}'

Thank you very much for your answer but result is a string ??
Because you gave it a string. If you give it a dict, you'll get a
dict:
>>import cPickle as pickle
a = {1:2, 3:4}
s = pickle.dumps(a, -1)
g = pickle.loads(s)
g
{1: 2, 3: 4}

If your existing database already has data in the "{...}" format, then
eval it only the first time. Then you'll get the dict which you can
cache thruogh the use of dumps/loads.
Oct 18 '07 #15
On Oct 18, 7:02 pm, Hrvoje Niksic <hnik...@xemacs.orgwrote:
Abandoned <best...@gmail.comwrites:
import cPickle as pickle
a="{2:3,4:6,2:7}"
s=pickle.dumps(a, -1)
g=pickle.loads(s);
print g
'{2:3,4:6,2:7}'
Thank you very much for your answer but result is a string ??

Because you gave it a string. If you give it a dict, you'll get a
dict:
>import cPickle as pickle
a = {1:2, 3:4}
s = pickle.dumps(a, -1)
g = pickle.loads(s)
g

{1: 2, 3: 4}

If your existing database already has data in the "{...}" format, then
eval it only the first time. Then you'll get the dict which you can
cache thruogh the use of dumps/loads.
Sorry i can't understand :(
Yes my database already has data in the "{..}" format and i select
this and i want to use it for dictionary..
in your example:
first data is a string
finally data is already string

I want to command like eval. (eval is not good because it is slow for
my project)

Oct 18 '07 #16
Abandoned <be*****@gmail.comwrites:
I select where id=56 and 100.000 rows are selecting but this took 2
second. (very big for my project)
I try cache to speed up this select operation..
And create a cache table:
id-1 | all
56 {68:66, 98:32455, 62:655}
If you use Python to create this cache table, then simply don't dump
it as a dictionary, but as a pickle:

id-1 | all
56 <some weird string produced by cPickle.dumps>

When you load it, convert the string to dict with cPickle.loads
instead of with eval.
Oct 18 '07 #17
On Oct 18, 9:09 am, Abandoned <best...@gmail.comwrote:
On Oct 18, 6:57 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
Abandoned wrote:
On Oct 18, 6:35 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
>Abandoned wrote:
On Oct 18, 6:14 pm, "Diez B. Roggisch" <de...@nospam.web.dewrote:
>Abandoned wrote:
Thanks you all answer..
But "eval" is very slow at very big dictionary {2:3,4:5,6:19....}
(100.000 elements)
Is there any easy alternative ?
>How big? How slow? For me, a 10000-element list takes 0.04 seconds to
>be parsed. Which I find fast.
>Diez
173.000 dict elements and it tooks 2.2 seconds this very big time for
my project
>Where does the data come from?
>Diez
Data come from database..
I want to cache to speed up my system and i save the dictionary to
database for speed up but eval is very slow for do this.
Not: 2.2 second only eval operation.
Does the dictionary change often?
And you should store a pickle to the database then. Besides, making a
database-query of that size (after all, we're talking a few megs here) will
take a while as well - so are you SURE the 2.2 seconds are a problem? Or is
it just that you think they are?
Diez- Hide quoted text -
- Show quoted text -

I'm very confused :(
I try to explain main problem...
I have a table like this:
id-1 | id-2 | value
23 24 34
56 68 66
56 98 32455
55 62 655
56 28 123
.... ( 3 millions elements)

I select where id=56 and 100.000 rows are selecting but this took 2
second. (very big for my project)
I try cache to speed up this select operation..
And create a cache table:
id-1 | all
56 {68:66, 98:32455, 62:655}

When i select where id 56 i select 1 row and its took 0.09 second but
i must convert text to dictionary..

Have you got any idea what can i do this conver operation ?
or
Have you got any idea what can i do cache for this table ?
I think several people have given you the correct answer, but for some
reason you aren't getting it. Instead of saving the string
representation of a dictionary to the database, pickle the dictionary
(not the string representation, but the actual dictionary) and save
the pickled object as a BLOB to the database. using pickle to re-
create the dictionary should be much faster than evaluating a string
representation of it.

Matt
Oct 18 '07 #18
Abandoned <be*****@gmail.comwrites:
Sorry i can't understand :(
Yes my database already has data in the "{..}" format and i select
this and i want to use it for dictionary..
But, do you use Python to create that data? If so, simply convert it
to pickle binary format instead of "{...}". As explained in my other
post:
id-1 | all
56 {68:66, 98:32455, 62:655}
If you use Python to create this cache table, then simply don't dump
it as a dictionary, but as a pickle:

id-1 | all
56 <some weird string produced by cPickle.dumps>

When you load it, convert the string to dict with cPickle.loads
instead of with eval.
Oct 18 '07 #19
On Oct 18, 7:40 pm, Hrvoje Niksic <hnik...@xemacs.orgwrote:
Abandoned <best...@gmail.comwrites:
Sorry i can't understand :(
Yes my database already has data in the "{..}" format and i select
this and i want to use it for dictionary..

But, do you use Python to create that data? If so, simply convert it
to pickle binary format instead of "{...}". As explained in my other
post:
id-1 | all
56 {68:66, 98:32455, 62:655}

If you use Python to create this cache table, then simply don't dump
it as a dictionary, but as a pickle:

id-1 | all
56 <some weird string produced by cPickle.dumps>

When you load it, convert the string to dict with cPickle.loads
instead of with eval.
Yes i understand and this very very good ;)
But i have a problem..
a=eval(a)
a=pickle.dumps(a, -1)
cursor.execute("INSERT INTO cache2 VALUES ('%s')" % (a))
conn.commit()

and give me error:

psycopg2.ProgrammingError: invalid byte sequence for encoding "UTF8":
0x80
HINT: This error can also happen if the byte sequence does not match
the encoding expected by the server, which is controlled by
"client_encoding".

Oct 18 '07 #20
On 10/18/07, Adam Atlas <ad**@atlas.stwrote:
>
Use the builtin function "eval".
What is the difference with os.system()?

--
Sebastián Bassi (セバスティアン). Diplomado en Ciencia y Tecnolog*a.
Curso Biologia molecular para programadores: http://tinyurl.com/2vv8w6
GPG Fingerprint: 9470 0980 620D ABFC BE63 A4A4 A3DE C97D 8422 D43D
Oct 18 '07 #21

"Matimus" <mc******@gmail.comwrote in message
news:11**********************@q5g2000prf.googlegro ups.com...
I think several people have given you the correct answer, but for some
reason you aren't getting it. Instead of saving the string
representation of a dictionary to the database...
Mind you, if this were Jeopardy, "Store a binary pickle
of a denormalized table back in the database" would
be a tough one.
Oct 18 '07 #22
Abandoned a crit :
On Oct 18, 6:51 pm, Marc 'BlackJack' Rintsch <bj_...@gmx.netwrote:
>On Thu, 18 Oct 2007 08:41:30 -0700, Abandoned wrote:
>>import cPickle as pickle
a="{2:3,4:6,2:7}"
s=pickle.dumps(a, -1)
g=pickle.loads(s);
print g
'{2:3,4:6,2:7}'
Thank you very much for your answer but result is a string ??
In Python terms yes, strings in Python can contain any byte value. If you
want to put this into a database you need a BLOB column or encode it as
base64 or something similar more ASCII safe.

Ciao,
Marc 'BlackJack' Rintsch

'{2:3,4:6,2:7}' already in database, i select this and convert to real
dictionary..
MVHO is that whoever uses a RDBMS to store language-specific serialized
collections should be shot down without sommation.

Oct 18 '07 #23
Abandoned a crit :
(snip)
import cPickle as pickle
a="{2:3,4:6,2:7}"
s=pickle.dumps(a, -1)
g=pickle.loads(s);
print g
'{2:3,4:6,2:7}'

Thank you very much for your answer but result is a string ??
Of course it's a string. That's what you pickled. What did you hope ? If
you want a dict back, then pickle a dict.
Oct 18 '07 #24
Richard Brodie a crit :
"Matimus" <mc******@gmail.comwrote in message
news:11**********************@q5g2000prf.googlegro ups.com...
>I think several people have given you the correct answer, but for some
reason you aren't getting it. Instead of saving the string
representation of a dictionary to the database...

Mind you, if this were Jeopardy, "Store a binary pickle
of a denormalized table back in the database" would
be a tough one.

Indeed.
Oct 18 '07 #25
Abandoned a crit :
(snip)
I'm very confused :(
I try to explain main problem...
I have a table like this:
id-1 | id-2 | value
23 24 34
56 68 66
56 98 32455
55 62 655
56 28 123
.... ( 3 millions elements)

I select where id=56 and 100.000 rows are selecting but this took 2
second. (very big for my project)
Not to bad in the absolute.
I try cache to speed up this select operation..
And create a cache table:
id-1 | all
56 {68:66, 98:32455, 62:655}
I really doubt this is the right way to go.
When i select where id 56 i select 1 row and its took 0.09 second but
i must convert text to dictionary..

Have you got any idea what can i do this conver operation ?
Other alread answered
Have you got any idea what can i do cache for this table ?
Depends on your RDBMS. And as far as I'm concerned, I'd start by trying
to find out how to optimize this query within the RDBMS - good ones are
usually highly optimized softwares with provision for quite a lot of
performance tuning.
Oct 18 '07 #26
Abandoned <be*****@gmail.comwrites:
When you load it, convert the string to dict with cPickle.loads
instead of with eval.

Yes i understand and this very very good ;)
Good! :-)
psycopg2.ProgrammingError: invalid byte sequence for encoding "UTF8":
0x80
HINT: This error can also happen if the byte sequence does not match
the encoding expected by the server, which is controlled by
"client_encoding".
Use a different column type for cache2's column, one more appropriate
for storing binary characters (perhaps BYTEA for Postgres). Don't
forget to also use a bind variable, something like:

cursor.execute("INSERT INTO cache2 VALUES (?)", a)

Using "INSERT ... ('%s')" % (a) won't work, since the huge binary
string in a can contain arbitrary characters, including the single
quote.
Oct 18 '07 #27
On Thu, 2007-10-18 at 19:53 +0200, Hrvoje Niksic wrote:
Don't
forget to also use a bind variable, something like:

cursor.execute("INSERT INTO cache2 VALUES (?)", a)
I second the advice, but that code won't work. The bind parameters must
be a sequence, and psycopg2 (unfortunately) uses %s for parameter
markers, instead of the SQL standard question mark. So the actual code
would be

cursor.execute("INSERT INTO cache2 VALUES (%s)", (a,) )

HTH,

--
Carsten Haese
http://informixdb.sourceforge.net
Oct 18 '07 #28
On Oct 18, 1:38 pm, Bruno Desthuilliers <bruno.
42.desthuilli...@wtf.websiteburo.oops.comwrote:
Abandoned a crit :
(snip)
I'm very confused :(
I try to explain main problem...
I have a table like this:
id-1 | id-2 | value
23 24 34
56 68 66
56 98 32455
55 62 655
56 28 123
.... ( 3 millions elements)
I select where id=56 and 100.000 rows are selecting but this took 2
second. (very big for my project)

Not to bad in the absolute.
I try cache to speed up this select operation..
And create a cache table:
id-1 | all
56 {68:66, 98:32455, 62:655}

I really doubt this is the right way to go.
When i select where id 56 i select 1 row and its took 0.09 second but
i must convert text to dictionary..
Have you got any idea what can i do this conver operation ?

Other alread answered
Have you got any idea what can i do cache for this table ?

Depends on your RDBMS. And as far as I'm concerned, I'd start by trying
to find out how to optimize this query within the RDBMS - good ones are
usually highly optimized softwares with provision for quite a lot of
performance tuning.
Just the overhead of the query is a killer compared to a dictionary
lookup in Python, even if all you're doing is selecting an integer
from a 1-row, 1-column table.

Usually you can get around that by making a single query to return all
of your results (or a handful of queries), but occasionally it just
doesn't matter how fast the DB can get to the data--the simple act of
asking it is slow enough on its own.

Oct 18 '07 #29
On Thu, 18 Oct 2007 14:05:34 -0300, Sebastian Bassi wrote:
On 10/18/07, Adam Atlas <ad**@atlas.stwrote:
>>
Use the builtin function "eval".

What is the difference with os.system()?
Everything.

eval() evaluates Python expressions like "x.append(2+3)".

os.system() calls your operating system's shell with a command.
--
Steven.

Oct 18 '07 #30
On Oct 18, 8:53 pm, Hrvoje Niksic <hnik...@xemacs.orgwrote:
Abandoned <best...@gmail.comwrites:
When you load it, convert the string to dict with cPickle.loads
instead of with eval.
Yes i understand and this very very good ;)

Good! :-)
psycopg2.ProgrammingError: invalid byte sequence for encoding "UTF8":
0x80
HINT: This error can also happen if the byte sequence does not match
the encoding expected by the server, which is controlled by
"client_encoding".

Use a different column type for cache2's column, one more appropriate
for storing binary characters (perhaps BYTEA for Postgres). Don't
forget to also use a bind variable, something like:

cursor.execute("INSERT INTO cache2 VALUES (?)", a)

Using "INSERT ... ('%s')" % (a) won't work, since the huge binary
string in a can contain arbitrary characters, including the single
quote.
I tryed:
cursor.execute("INSERT INTO cache2 VALUES (?)", a)
and
cursor.execute("INSERT INTO cache2 VALUES (%s)", (a,) )
but the result is same..
psycopg2.ProgrammingError: invalid byte sequence for encoding "UTF8":
0x80
HINT: This error can also happen if the byte sequence does not match
the encoding expected by the server, which is controlled by
"client_encoding".
Oct 19 '07 #31
Abandoned <be*****@gmail.comwrites:
>Use a different column type for cache2's column, one more appropriate
for storing binary characters (perhaps BYTEA for Postgres). Don't
forget to also use a bind variable, something like:

cursor.execute("INSERT INTO cache2 VALUES (?)", a)

Using "INSERT ... ('%s')" % (a) won't work, since the huge binary
string in a can contain arbitrary characters, including the single
quote.

I tryed:
cursor.execute("INSERT INTO cache2 VALUES (?)", a)
Why are you ignoring the first sentence: "Use a different column type
for cache2's column, ..."? The use of bind variables in INSERT will
work only *after* you do the rest of the work.
Oct 19 '07 #32
Abandoned wrote:
I'm very confused :(
I try to explain main problem...
That's always a good first step; try to remember that when you start
your next thread.
I have a table like this:
id-1 | id-2 | value
23 24 34
56 68 66
56 98 32455
55 62 655
56 28 123
.... ( 3 millions elements)

I select where id=56 and 100.000 rows are selecting but this took 2
second. (very big for my project)
I try cache to speed up this select operation..
And create a cache table:
id-1 | all
56 {68:66, 98:32455, 62:655}

When i select where id 56 i select 1 row and its took 0.09 second but
i must convert text to dictionary..
Before you go on with your odd caching schemes -- is the database properly
indexed? Something like

CREATE UNIQUE INDEX mytable_id1_id2 ON mytable (id-1, id-2);

(actual syntax may differ) might speed up the lookup operation
enough that you can do without caching.

Peter
Oct 19 '07 #33
sj*******@yahoo.com a crit :
On Oct 18, 1:38 pm, Bruno Desthuilliers <bruno.
42.desthuilli...@wtf.websiteburo.oops.comwrote:
>Abandoned a crit :
(snip)
>>I'm very confused :(
I try to explain main problem...
I have a table like this:
id-1 | id-2 | value
23 24 34
56 68 66
56 98 32455
55 62 655
56 28 123
.... ( 3 millions elements)
I select where id=56 and 100.000 rows are selecting but this took 2
second. (very big for my project)
Not to bad in the absolute.
>>I try cache to speed up this select operation..
And create a cache table:
id-1 | all
56 {68:66, 98:32455, 62:655}
I really doubt this is the right way to go.
>>When i select where id 56 i select 1 row and its took 0.09 second but
i must convert text to dictionary..
Have you got any idea what can i do this conver operation ?
Other alread answered
>>Have you got any idea what can i do cache for this table ?
Depends on your RDBMS. And as far as I'm concerned, I'd start by trying
to find out how to optimize this query within the RDBMS - good ones are
usually highly optimized softwares with provision for quite a lot of
performance tuning.

Just the overhead of the query is a killer compared to a dictionary
lookup in Python, even if all you're doing is selecting an integer
from a 1-row, 1-column table.
Indeed. But then why use a RDBMS at all ? Please understand that I'm not
saying that a RDBMS will beat a plain dict lookup not that a RDBMS will
solve world's problem, but that storing pickled Python's dicts into a
RDBMS is certainly not the best thing to do. It will *still* have the db
connection overhead anyway, and will be a nightmare to maintain in sync
with the real state of the db. Which is why I suggest *first* looking
for RDBMS-side tuning and optimization - which may include third-part
cache systems FWIW.
Oct 19 '07 #34
Peter Otten a écrit :
(snip)
Before you go on with your odd caching schemes -- is the database properly
indexed? Something like

CREATE UNIQUE INDEX mytable_id1_id2 ON mytable (id-1, id-2);

(actual syntax may differ) might speed up the lookup operation
enough that you can do without caching.
Im my arms(tm) ! At least some sensible advice...

Oct 19 '07 #35
Hrvoje Niksic <hn*****@xemacs.orgwrites:
If you're generating the string from Python, use cPickle instead.
Much faster:
[...]
>>>t0 = time.time(); d2 = eval(s); t1 = time.time(); t1-t0
1.5457899570465088
>>>t0 = time.time(); d2 = pickle.loads(s); t1 = time.time(); t1-t0
0.060307979583740234
It just occurred to me, for simple data structures like the ones we're
discussing here (dicts of ints), marshal should also be considered.
marshal is the module used for generating and loading .pyc files and,
while it doesn't support all the bells and whistles of pickle, it's
very fast:
>>t0 = time.time(); d2 = marshal.loads(s); t1 = time.time(); t1-t0
0.029728889465332031

Benchmarks made with the timeit module confirm this difference.

Marshal has the added advantage of using much less space than the
(binary) pickle -- the example dictionary provided above pickled to a
string of 2667791 bytes, while marshal produced a string of 1700002
bytes.
Oct 19 '07 #36

This discussion thread is closed

Replies have been disabled for this discussion.

Similar topics

4 posts views Thread by Mike Collins | last post: by
3 posts views Thread by GM | last post: by
4 posts views Thread by Franky | last post: by
reply views Thread by =?Utf-8?B?cm9uZSBtYXRpYXM=?= | last post: by
3 posts views Thread by mamul | last post: by

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.