Filtering out non-readable characters

[MKoool]

I have a file with binary and ascii characters in it. I massage the
data and convert it to a more readable format, however it still comes
up with some binary characters mixed in. I'd like to write something
to just replace all non-printable characters with '' (I want to delete
non-printable characters).

I am having trouble figuring out an easy python way to do this... is
the easiest way to just write some regular expression that does
something like replace [^\p] with ''?

Or is it better to go through every character and do ord(character),
check the ascii values?

What's the easiest way to do something like this?

thanks

[Bengt Richter]

On 15 Jul 2005 17:33:39 -0700, "MKoool" <mo***@teraboli c.com> wrote:

I have a file with binary and ascii characters in it. I massage the
data and convert it to a more readable format, however it still comes
up with some binary characters mixed in. I'd like to write something
to just replace all non-printable characters with '' (I want to delete
non-printable characters).

I am having trouble figuring out an easy python way to do this... is
the easiest way to just write some regular expression that does
something like replace [^\p] with ''?

Or is it better to go through every character and do ord(character),
check the ascii values?

What's the easiest way to do something like this?

import string
string.printabl e '0123456789abcd efghijklmnopqrs tuvwxyzABCDEFGH IJKLMNOPQRSTUVW XYZ!"#$%&\'()*+ ,-./:;<=>?@[\\]^_`{|}~ \t\n\r\x0b\x0c' identity = ''.join([chr(i) for i in xrange(256)])
unprintable = ''.join([c for c in identity if c not in string.printabl e])

def remove_unprinta ble(s): ... return s.translate(ide ntity, unprintable)
... set(remove_unpr intable(identit y)) - set(string.prin table) set([]) set(remove_unpr intable(identit y)) set(['\x0c', ' ', '$', '(', ',', '0', '4', '8', '<', '@', 'D', 'H', 'L', 'P', 'T', 'X', '\\', '`
', 'd', 'h', 'l', 'p', 't', 'x', '|', '\x0b', '#', "'", '+', '/', '3', '7', ';', '?', 'C', 'G',
'K', 'O', 'S', 'W', '[', '_', 'c', 'g', 'k', 'o', 's', 'w', '{', '\n', '"', '&', '*', '.', '2',
'6', ':', '>', 'B', 'F', 'J', 'N', 'R', 'V', 'Z', '^', 'b', 'f', 'j', 'n', 'r', 'v', 'z', '~', '
\t', '\r', '!', '%', ')', '-', '1', '5', '9', '=', 'A', 'E', 'I', 'M', 'Q', 'U', 'Y', ']', 'a',
'e', 'i', 'm', 'q', 'u', 'y', '}']) sorted(set(remo ve_unprintable( identity))) == sorted(set(stri ng.printable)) True sorted((remove_ unprintable(ide ntity))) == sorted((string. printable)) True

After that, to get clean file text, something like

cleantext = remove_unprinta ble(file('uncle an.txt').read() )

should do it. Or you should be able to iterate by lines something like (untested)

for uncleanline in file('unclean.t xt'):
cleanline = remove_unprinta ble(uncleanline )
# ... do whatever with clean line

If there is something in string.printabl e that you don't want included, just use your own
string of printables. BTW,
help(str.transl ate)

Help on method_descript or:

translate(...)
S.translate(tab le [,deletechars]) -> string

Return a copy of the string S, where all characters occurring
in the optional argument deletechars are removed, and the
remaining characters have been mapped through the given
translation table, which must be a string of length 256.

Regards,
Bengt Richter

[Raymond Hettinger]

Wow, that was the most thorough answer to a comp.lang.pytho n question
since the Martellibot got busy in the search business.

[Peter Hansen]

Bengt Richter wrote:

>>> identity = ''.join([chr(i) for i in xrange(256)])
>>> unprintable = ''.join([c for c in identity if c not in string.printabl e])

And note that with Python 2.4, in each case the above square brackets
are unnecessary (though harmless), because of the arrival of "generator
expressions" in the language. (Bengt knows this already, of course, but
his brain is probably resisting the reprogramming. :-) )

-Peter

[Steven D'Aprano]

On Sat, 16 Jul 2005 10:25:29 -0400, Peter Hansen wrote:

Bengt Richter wrote:

>>> identity = ''.join([chr(i) for i in xrange(256)])
>>> unprintable = ''.join([c for c in identity if c not in string.printabl e])

And note that with Python 2.4, in each case the above square brackets
are unnecessary (though harmless), because of the arrival of "generator
expressions" in the language.

But to use generator expressions, wouldn't you need an extra pair of round
brackets?

eg identity = ''.join( ( chr(i) for i in xrange(256) ) )

with the extra spaces added for clarity.

That is, the brackets after join make the function call, and the nested
brackets make the generator. That, at least, is my understanding.

--
Steven
who is still using Python 2.3, and probably will be for quite some time

[Peter Hansen]

Steven D'Aprano wrote:

On Sat, 16 Jul 2005 10:25:29 -0400, Peter Hansen wrote:

Bengt Richter wrote:

>>> identity = ''.join([chr(i) for i in xrange(256)])

And note that with Python 2.4, in each case the above square brackets
are unnecessary (though harmless), because of the arrival of "generator
expressions " in the language.

But to use generator expressions, wouldn't you need an extra pair of round
brackets?

eg identity = ''.join( ( chr(i) for i in xrange(256) ) )

Come on, Steven. Don't tell us you didn't have access to a Python
interpreter to check before you posted:

c:\>python
Python 2.4 (#60, Nov 30 2004, 11:49:19) [MSC v.1310 32 bit (Intel)] on win32

''.join(chr(c) for c in range(65, 91))

'ABCDEFGHIJKLMN OPQRSTUVWXYZ'

-Peter

[Bengt Richter]

On Sat, 16 Jul 2005 10:25:29 -0400, Peter Hansen <pe***@engcorp. com> wrote:

Bengt Richter wrote:

>>> identity = ''.join([chr(i) for i in xrange(256)])
>>> unprintable = ''.join([c for c in identity if c not in string.printabl e])

And note that with Python 2.4, in each case the above square brackets
are unnecessary (though harmless), because of the arrival of "generator
expressions" in the language. (Bengt knows this already, of course, but
his brain is probably resisting the reprogramming. :-) )

Thanks for the nudge. Actually, I know about generator expressions, but
at some point I must have misinterpreted some bug in my code to mean
that join in particular didn't like generator expression arguments,
and wanted lists. Actually it seems to like anything at all that can
be iterated produce a sequence of strings. So I'm glad to find that
join is fine after all, and to get that misap[com?:-)]prehension
out of my mind ;-)

Regards,
Bengt Richter

[George Sakkis]

"Bengt Richter" <bo**@oz.net> wrote:

>>> identity = ''.join([chr(i) for i in xrange(256)])
>>> unprintable = ''.join([c for c in identity if c not in string.printabl e])
Or equivalently:
identity = string.maketran s('','')
unprintable = identity.transl ate(identity, string.printabl e)

George

[Peter Hansen]

George Sakkis wrote:

"Bengt Richter" <bo**@oz.net> wrote:

>>> identity = ''.join([chr(i) for i in xrange(256)])
Or equivalently:identity = string.maketran s('','')

Wow! That's handy, not to mention undocumented. (At least in the
string module docs.) Where did you learn that, George?

-Peter

[Steven D'Aprano]

Replying to myself... this is getting to be a habit.

On Sun, 17 Jul 2005 15:08:12 +1000, Steven D'Aprano wrote:

I hope that makes sense to you.

That wasn't meant as a snide little dig at Peter, and I'm sorry if anyone
reads it that way. I found myself struggling to explain simply the
different behaviour between list comps and generator expressions, and
couldn't be sure I was explaining myself as clearly as I wanted. It might
have been better if I had left off the "to you".

--
Steven