Filtering out non-readable characters

[MKoool]

I have a file with binary and ascii characters in it. I massage the
data and convert it to a more readable format, however it still comes
up with some binary characters mixed in. I'd like to write something
to just replace all non-printable characters with '' (I want to delete
non-printable characters).

I am having trouble figuring out an easy python way to do this... is
the easiest way to just write some regular expression that does
something like replace [^\p] with ''?

Or is it better to go through every character and do ord(character),
check the ascii values?

What's the easiest way to do something like this?

thanks

[Steven D'Aprano]

On Sat, 16 Jul 2005 16:42:58 -0400, Peter Hansen wrote:

Steven D'Aprano wrote:

On Sat, 16 Jul 2005 10:25:29 -0400, Peter Hansen wrote:

Bengt Richter wrote:

>>> identity = ''.join([chr(i) for i in xrange(256)])

And note that with Python 2.4, in each case the above square brackets
are unnecessary (though harmless), because of the arrival of "generator
expression s" in the language.

But to use generator expressions, wouldn't you need an extra pair of round
brackets?

eg identity = ''.join( ( chr(i) for i in xrange(256) ) )

Come on, Steven. Don't tell us you didn't have access to a Python
interpreter to check before you posted:

Er, as I wrote in my post:

"Steven
who is still using Python 2.3, and probably will be for quite some time"

So, no, I didn't have access to a Python interpreter running version 2.4.

I take it then that generator expressions work quite differently
than list comprehensions? The equivalent "implied delimiters" for a list
comprehension would be something like this:

L = [1, 2, 3]
L[ i for i in range(2) ] File "<stdin>", line 1
L[ i for i in range(2) ]
^
SyntaxError: invalid syntax

which is a very different result from:
L[ [i for i in range(2)] ]

Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: list indices must be integers

In other words, a list comprehension must have the [ ] delimiters to be
recognised as a list comprehension, EVEN IF the square brackets are there
from some other element. But a generator expression doesn't care where the
round brackets come from, so long as they are there: they can be part of
the function call.

I hope that makes sense to you.
--
Steven

[Steven D'Aprano]

On Sat, 16 Jul 2005 19:01:50 -0400, Peter Hansen wrote:

George Sakkis wrote:

"Bengt Richter" <bo**@oz.net> wrote:

>>> identity = ''.join([chr(i) for i in xrange(256)])

Or equivalently:

>identity = string.maketran s('','')

Wow! That's handy, not to mention undocumented. (At least in the
string module docs.) Where did you learn that, George?

I can't answer for George, but I also noticed that behaviour. I discovered
it by trial and error. I thought, oh what a nuisance that the arguments
for maketrans had to include all 256 characters, then I wondered what
error you would get if you left some out, and discovered that you didn't
get an error at all.

That actually disappointed me at the time, because I was looking for
behaviour where the missing characters weren't filled in, but I've come to
appreciate it since.
--
Steven

[Peter Hansen]

Steven D'Aprano wrote:

On Sat, 16 Jul 2005 16:42:58 -0400, Peter Hansen wrote:

Come on, Steven. Don't tell us you didn't have access to a Python
interpreter to check before you posted:

Er, as I wrote in my post:

"Steven
who is still using Python 2.3, and probably will be for quite some time"

Sorry, missed that! I don't generally notice signatures much, partly
because Thunderbird is smart enough to "grey them out" (the main text is
displayed as black, quoted material in blue, and signatures in a light
gray.)

I don't have a firm answer (though I suspect the language reference
does) about when "dedicated" parentheses are required around a generator
expression. I just know that, so far, they just work when I want them
to. Like most of Python. :-)

-Peter

[Steven Bethard]

Bengt Richter wrote:

Thanks for the nudge. Actually, I know about generator expressions, but
at some point I must have misinterpreted some bug in my code to mean
that join in particular didn't like generator expression arguments,
and wanted lists.

I suspect this is bug 905389 [1]:

def gen(): .... yield 1
.... raise TypeError('from gen()')
.... ''.join([x for x in gen()]) Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
File "<interacti ve input>", line 3, in gen
TypeError: from gen() ''.join(x for x in gen())

Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
TypeError: sequence expected, generator found

I run into this every month or so, and have to remind myself that it
means that my generator is raising a TypeError, not that join doesn't
accept generator expressions...

STeVe

[1] http://www.python.org/sf/905389

[Michael Ströder]

Peter Hansen wrote:

''.join(chr(c) for c in range(65, 91))

'ABCDEFGHIJKLMN OPQRSTUVWXYZ'

Wouldn't this be a candidate for making the Python language stricter?

Do you remember old Python versions treating l.append(n1,n2) the same
way like l.append((n1,n2 )). I'm glad this is forbidden now.

Ciao, Michael.

[Peter Hansen]

Michael Ströder wrote:

Peter Hansen wrote:

>''.join(ch r(c) for c in range(65, 91))

'ABCDEFGHIJKL MNOPQRSTUVWXYZ'

Wouldn't this be a candidate for making the Python language stricter?

Why would that be true? I believe str.join() takes any iterable, and a
generator (as returned by a generator expression) certainly qualifies.

-Peter

[Robert Kern]

Michael Ströder wrote:

Peter Hansen wrote:

>''.join(ch r(c) for c in range(65, 91))

'ABCDEFGHIJKL MNOPQRSTUVWXYZ'

Wouldn't this be a candidate for making the Python language stricter?

Do you remember old Python versions treating l.append(n1,n2) the same
way like l.append((n1,n2 )). I'm glad this is forbidden now.

That wasn't a syntax issue; it was an API issue. list.append() allowed
multiple arguments and interpreted them as if they were a single tuple.
That was confusing and unnecessary.

Allowing generator expressions to forgo extra parentheses where they
aren't required is something different, and in my opinion, a good thing.

--
Robert Kern
rk***@ucsd.edu

"In the fields of hell where the grass grows high
Are the graves of dreams allowed to die."
-- Richard Harter

[Ross]

On 15 Jul 2005 17:33:39 -0700, "MKoool" <mo***@teraboli c.com> wrote:

I have a file with binary and ascii characters in it. I massage the
data and convert it to a more readable format, however it still comes
up with some binary characters mixed in. I'd like to write something
to just replace all non-printable characters with '' (I want to delete
non-printable characters).

I am having trouble figuring out an easy python way to do this... is
the easiest way to just write some regular expression that does
something like replace [^\p] with ''?

Or is it better to go through every character and do ord(character),
check the ascii values?

What's the easiest way to do something like this?

thanks

Easiest way is open the file with EdXor (freeware editor), select all,
Format > Wipe Non-Ascii.

Ok it's not python, but it's the easiest.

[Steven D'Aprano]

On Tue, 19 Jul 2005 20:28:31 +1200, Ross wrote:

On 15 Jul 2005 17:33:39 -0700, "MKoool" <mo***@teraboli c.com> wrote:

I have a file with binary and ascii characters in it. I massage the
data and convert it to a more readable format, however it still comes
up with some binary characters mixed in. I'd like to write something
to just replace all non-printable characters with '' (I want to delete
non-printable characters).

I am having trouble figuring out an easy python way to do this... is
the easiest way to just write some regular expression that does
something like replace [^\p] with ''?

Or is it better to go through every character and do ord(character),
check the ascii values?

What's the easiest way to do something like this?

thanks

Easiest way is open the file with EdXor (freeware editor), select all,
Format > Wipe Non-Ascii.

Ok it's not python, but it's the easiest.

1 Open Internet Explorer
2 Go to Google
3 Search for EdXor
4 Browser locks up
5 Force quit with ctrl-alt-del
6 Run anti-virus program
7 Download new virus definitions
8 Remove viruses
9 Run anti-spyware program
10 Download new definitions
11 Remove spyware
12 Open Internet Explorer
13 Download Firefox
14 Install Firefox
15 Open Firefox
16 Go to Google
17 Search for EdXor
18 Download application
19 Run installer
20 Reboot
21 Run EdXor
22 Open file
23 Select all
24 Select Format>Wipe Non-ASCII
25 Select Save
26 Quit EdXor

Hmmm. Perhaps not *quite* the easiest way :-)

--
Steven.

[Bengt Richter]

On Sun, 17 Jul 2005 15:42:08 -0600, Steven Bethard <st************ @gmail.com> wrote:

Bengt Richter wrote:

Thanks for the nudge. Actually, I know about generator expressions, but
at some point I must have misinterpreted some bug in my code to mean
that join in particular didn't like generator expression arguments,
and wanted lists.

I suspect this is bug 905389 [1]:

def gen():... yield 1
... raise TypeError('from gen()')
... ''.join([x for x in gen()])Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
File "<interacti ve input>", line 3, in gen
TypeError: from gen() ''.join(x for x in gen())

Traceback (most recent call last):
File "<interacti ve input>", line 1, in ?
TypeError: sequence expected, generator found

I run into this every month or so, and have to remind myself that it
means that my generator is raising a TypeError, not that join doesn't
accept generator expressions...

STeVe

[1] http://www.python.org/sf/905389

That must have been it, thanks.

Regards,
Bengt Richter