I am trying to convert raw binary data to data with escaped octets in
order to store it in a bytea field on postgresql server. I could do this
easily in c/c++ but I need to do it in python. I am not sure how to read
and evaluate the binary value of a byte in a long string when it is a non
printable ascii value in python. I read some ways to use unpack from the
struct module, but i really couldn't understand where that would help. I
looked at the MIMIEncode module but I don't know how to convert the object
to a string. Is there a module that will convert the data? It seems to me
that this question must have been answered a million times before but I
can't find anything.
See http://www.postgresql.org/docs/8.1/i...pe-binary.html
for a description of the problem domain. 10 1882
Chason Hayes <ch*****@hotmai l.com> wrote:
... easily in c/c++ but I need to do it in python. I am not sure how to read and evaluate the binary value of a byte in a long string when it is a non printable ascii value in python.
If you have a bytestring (AKA plain string) s, the binary value of its
k-th byte is ord(s[k]).
Alex
Chason Hayes wrote: I am trying to convert raw binary data to data with escaped octets in order to store it in a bytea field on postgresql server. I could do this easily in c/c++ but I need to do it in python. I am not sure how to read and evaluate the binary value of a byte in a long string when it is a non printable ascii value in python. I read some ways to use unpack from the struct module, but i really couldn't understand where that would help. I looked at the MIMIEncode module but I don't know how to convert the object to a string. Is there a module that will convert the data? It seems to me that this question must have been answered a million times before but I can't find anything. See http://www.postgresql.org/docs/8.1/i...pe-binary.html for a description of the problem domain.
The URL you reference is discussing how you represent arbitrary values
in string literals. If you already have the data in a Python string the
best advise is to use a parameterized query - that way your Python DB
API module will do the escaping for you!
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/
On Mon, 06 Feb 2006 13:39:17 +0000, Steve Holden wrote: Chason Hayes wrote: I am trying to convert raw binary data to data with escaped octets in order to store it in a bytea field on postgresql server. I could do this easily in c/c++ but I need to do it in python. I am not sure how to read and evaluate the binary value of a byte in a long string when it is a non printable ascii value in python. I read some ways to use unpack from the struct module, but i really couldn't understand where that would help. I looked at the MIMIEncode module but I don't know how to convert the object to a string. Is there a module that will convert the data? It seems to me that this question must have been answered a million times before but I can't find anything. See http://www.postgresql.org/docs/8.1/i...pe-binary.html for a description of the problem domain.
The URL you reference is discussing how you represent arbitrary values in string literals. If you already have the data in a Python string the best advise is to use a parameterized query - that way your Python DB API module will do the escaping for you!
regards Steve
Thanks for the input. I tried that with a format string and a
dictionary, but I still received a database error indicating illegal
string values. This error went away completely when I used a test file
consisting only of text, but reproduced everytime with a true binary file.
If you can let me know where I am wrong or show me a code snippet with a
sql insert that contains a variable with raw binary data that works,
I would greatly appreciate it.
Chason
On Sun, 05 Feb 2006 21:07:23 -0800, Alex Martelli wrote: Chason Hayes <ch*****@hotmai l.com> wrote: ... easily in c/c++ but I need to do it in python. I am not sure how to read and evaluate the binary value of a byte in a long string when it is a non printable ascii value in python.
If you have a bytestring (AKA plain string) s, the binary value of its k-th byte is ord(s[k]).
Alex
Thank you very much, That is the function that I was looking for to write
a filter.
Chason
Chason Hayes wrote: On Mon, 06 Feb 2006 13:39:17 +0000, Steve Holden wrote:
[...] The URL you reference is discussing how you represent arbitrary values in string literals. If you already have the data in a Python string the best advise is to use a parameterized query - that way your Python DB API module will do the escaping for you!
regards Steve
Thanks for the input. I tried that with a format string and a dictionary, but I still received a database error indicating illegal string values. This error went away completely when I used a test file consisting only of text, but reproduced everytime with a true binary file. If you can let me know where I am wrong or show me a code snippet with a sql insert that contains a variable with raw binary data that works, I would greatly appreciate it.
I tried and my experience was exactly the same, which made me think less
of PostgreSQL.
They don't seem to implement the SQL BLOB type properly, so it looks as
though that rebarbative syntax with all the backslashes is necessary. Sorry.
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/
On Mon, 06 Feb 2006 04:40:31 GMT, Chason Hayes <ch*****@hotmai l.com> wrote: I am trying to convert raw binary data to data with escaped octets in order to store it in a bytea field on postgresql server. I could do this easily in c/c++ but I need to do it in python. I am not sure how to read and evaluate the binary value of a byte in a long string when it is a non printable ascii value in python. I read some ways to use unpack from the struct module, but i really couldn't understand where that would help. I looked at the MIMIEncode module but I don't know how to convert the object to a string. Is there a module that will convert the data? It seems to me that this question must have been answered a million times before but I can't find anything.
Have you considered just encoding the data as text in hex or base64, e.g., import binascii s = '\x00\x01\x02\x 03ABCD0123' binascii.hexlif y(s)
'00010203414243 4430313233' binascii.b2a_ba se64(s)
'AAECA0FCQ0QwMT Iz\n'
which is also reversible later of course: h = binascii.hexlif y(s) binascii.unhexl ify(h)
'\x00\x01\x02\x 03ABCD0123' b64 = binascii.b2a_ba se64(s) binascii.a2b_ba se64(b64)
'\x00\x01\x02\x 03ABCD0123'
Regards,
Bengt Richter
On Tue, 07 Feb 2006 15:06:49 +0000, Bengt Richter wrote: On Mon, 06 Feb 2006 04:40:31 GMT, Chason Hayes <ch*****@hotmai l.com> wrote:
I am trying to convert raw binary data to data with escaped octets in order to store it in a bytea field on postgresql server. I could do this easily in c/c++ but I need to do it in python. I am not sure how to read and evaluate the binary value of a byte in a long string when it is a non printable ascii value in python. I read some ways to use unpack from the struct module, but i really couldn't understand where that would help. I looked at the MIMIEncode module but I don't know how to convert the object to a string. Is there a module that will convert the data? It seems to me that this question must have been answered a million times before but I can't find anything. Have you considered just encoding the data as text in hex or base64, e.g.,
>>> import binascii >>> s = '\x00\x01\x02\x 03ABCD0123' >>> binascii.hexlif y(s) '00010203414243 4430313233' >>> binascii.b2a_ba se64(s) 'AAECA0FCQ0QwMT Iz\n'
which is also reversible later of course: >>> h = binascii.hexlif y(s) >>> binascii.unhexl ify(h) '\x00\x01\x02\x 03ABCD0123' >>> b64 = binascii.b2a_ba se64(s) >>> binascii.a2b_ba se64(b64)
'\x00\x01\x02\x 03ABCD0123'
Regards, Bengt Richter
I had just about come to that conclusion last night while I was working on
it. I was going to use
import base64
base64.stringen code(binarydata )
and
base64.stringde code(stringdata )
I then wasn't sure if I should still use the bytea field or just use a
text field.
Do you have a suggestion?
On Tue, 07 Feb 2006 01:58:00 +0000, Steve Holden wrote: Chason Hayes wrote: On Mon, 06 Feb 2006 13:39:17 +0000, Steve Holden wrote: [...] The URL you reference is discussing how you represent arbitrary values in string literals. If you already have the data in a Python string the best advise is to use a parameterized query - that way your Python DB API module will do the escaping for you!
regards Steve
Thanks for the input. I tried that with a format string and a dictionary, but I still received a database error indicating illegal string values. This error went away completely when I used a test file consisting only of text, but reproduced everytime with a true binary file. If you can let me know where I am wrong or show me a code snippet with a sql insert that contains a variable with raw binary data that works, I would greatly appreciate it.
I tried and my experience was exactly the same, which made me think less of PostgreSQL.
They don't seem to implement the SQL BLOB type properly, so it looks as though that rebarbative syntax with all the backslashes is necessary. Sorry.
regards Steve
with regards to escaping data parameters I have found that I have to
specifically add quotes to my strings for them to be understood by
pstgresql. For example
ifs=open("binar ydatafile","r")
binarydata=ifs. read()
stringdata=base 64.encodestring (binarydata)
#does not work
cursor.execute( "insert into binarytable values(%s)" % stringdata)
#need to do this first
newstringdata = "'" + stringdata + "'"
then the select statment works.
Is this expected behavior? Is there a better way of doing this?
thanks for any insight
Chason
Chason Hayes wrote: On Tue, 07 Feb 2006 01:58:00 +0000, Steve Holden wrote:
Chason Hayes wrote:
On Mon, 06 Feb 2006 13:39:17 +0000, Steve Holden wrote:
[...]
The URL you reference is discussing how you represent arbitrary values in string literals. If you already have the data in a Python string the best advise is to use a parameterized query - that way your Python DB API module will do the escaping for you!
regards Steve
Thanks for the input. I tried that with a format string and a dictionary , but I still received a database error indicating illegal string values. This error went away completely when I used a test file consisting only of text, but reproduced everytime with a true binary file. If you can let me know where I am wrong or show me a code snippet with a sql insert that contains a variable with raw binary data that works, I would greatly appreciate it.
I tried and my experience was exactly the same, which made me think less of PostgreSQL.
They don't seem to implement the SQL BLOB type properly, so it looks as though that rebarbative syntax with all the backslashes is necessary. Sorry.
regards Steve
with regards to escaping data parameters I have found that I have to specifically add quotes to my strings for them to be understood by pstgresql. For example
ifs=open("binar ydatafile","r") binarydata=ifs. read() stringdata=base 64.encodestring (binarydata)
#does not work cursor.execute( "insert into binarytable values(%s)" % stringdata)
#need to do this first newstringdata = "'" + stringdata + "'"
then the select statment works. Is this expected behavior? Is there a better way of doing this?
thanks for any insight
Yes, parameterize your queries. I assume you are using psycopg or
something similar to create the database connection (i.e. I something
that expects the "%s" parameter style - there are other options, but we
needn't discuss them here).
The magic incantation you seek is:
cursor.execute( "insert into binarytable values(%s)", (stringdata, ))
Note that here there are TWO arguments to the .execute() method. The
first is a parameterized SQL statement, and the second is a tuple of
data items, one for each parameter mark in the SQL.
Using this technique all necessary quoting (and even data conversion
with a good database module) is performed inside the database driver,
meaning (among other things) that your program is no longer vulnerable
to the dreaded SQL injection errors.
This is the technique I was hoping would work with the bytea datatype,
but alas it doesn't. ISTM that PostgreSQL needs a bit of work there,
even though it is otherwise a very polished product.
regards
Steve
--
Steve Holden +44 150 684 7255 +1 800 494 3119
Holden Web LLC www.holdenweb.com
PyCon TX 2006 www.python.org/pycon/ This thread has been closed and replies have been disabled. Please start a new discussion. Similar topics |
by: Justin Koivisto |
last post by:
I am replacing a string in a text block that has a literal $ in it, and
preg_replace is seeing it as a backreference. Here is what I am using:
foreach($price_lists as $list)
$x=preg_replace('/--PRICE-LIST--/',$list,$x,1);
OK, so what this does it is takes each array element and replaces only
the first occurrance of "--PRICE-LIST--" with it. I would have used
str_replace, but I didn't think it should be necessary to create an
array...
|
by: Walter L. Preuninger II |
last post by:
I need to convert escape sequences entered into my program to the actual
code.
For example, \r becomes 0x0d
I have looked over the FAQ, and searched the web, with no results.
Is there a function that can do this, or do I need to use predefined
constants or a table of the values?
Below is a sample program.
When run with the input w\rx, I want to see the output::
|
by: gary |
last post by:
How would one make the ECMA-262 String.replace method work with a
string literal?
For example, if my string was "HELLO" how would I make it work
in this instance.
Please note my square brackets are not regular expression syntax.
Thanks,
|
by: emailscotta |
last post by:
Below I declared a basic object literal with 2 methods. The
"doSomething" method is call from the "useDoSomething" method but the
call is only sucessful if I use the "this" keyword or qualify the call
with "SomeObj".
Can someone describe why this is happening?
var SomeObj = {
doSomething : function()
{
|
by: Trev |
last post by:
Hi everyone,
Thanks to all who have helped with various issues in the past. I've
come up with a new one though:
I've run some html through a javascript converter; basically it takes
the html and outputs the
following:
var str='';
str+= // whatever text I want
document.write(str);
| |
by: terence.parker |
last post by:
This should be simple, but i've looked and looked and it seems all
anyone wants to do is get the percent-encoding or convert FROM utf8
octets.
But I want the octets themselves.
As in, I want to input Chinese (or anything, for that matter) and get
out something along the lines of \x05\x50\x3a etc...
It seems neither utf8_encode nor decide does this job, and neither
|
by: -Lost |
last post by:
For example:
var newlines = 'a\n\nb\n\nc';
alert(newlines);
Yet, if I get that *exact* same line from an XMLHttpRequest's
responseText, it is always alerted as:
a\n\nb\n\nc
|
by: Torsten Bronger |
last post by:
Hallöchen!
I need some help with finding matches in a string that has some
characters which are marked as escaped (in a separate list of
indices). Escaped means that they must not be part of any match.
My current approach is to look for matches in substrings with the
escaped characters as boundaries between the substrings. However,
then ^ and $ in the patterns are treated wrongly. (Although I use
startpos and endpos parameters for...
|
by: =?Utf-8?B?Qm9iQWNoZ2lsbA==?= |
last post by:
How can I use a quote as a literal so it does get confused as not a literal?
Thanks!
Bob
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
| |
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: isladogs |
last post by:
The next Access Europe User Group meeting will be on Wednesday 1 May 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM).
In this session, we are pleased to welcome a new presenter, Adolph Dupré who will be discussing some powerful techniques for using class modules.
He will explain when you may want to use classes instead of User Defined Types (UDT). For example, to manage the data in unbound forms.
Adolph will...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |