_csv.Error: string with NUL bytes

[fscked]

Anyone have an idea of what I might do to fix this? I have googled adn
can only find some random conversations about it that doesn't make
sense to me.

I am basically reading in a csv file to create an xml and get this
error.

I don't see any empty values in any fields or anything...

[Larry Bates]

fscked wrote:

Anyone have an idea of what I might do to fix this? I have googled adn
can only find some random conversations about it that doesn't make
sense to me.

I am basically reading in a csv file to create an xml and get this
error.

I don't see any empty values in any fields or anything...

You really should post some code and the actual traceback error your
get for us to help. I suspect that you have an ill-formed record in
your CSV file. If you can't control that, you may have to write your
own CSV dialect parser.

-Larry

[fscked]

On May 3, 9:11 am, Larry Bates <larry.ba...@we bsafe.comwrote:

fscked wrote:

Anyone have an idea of what I might do to fix this? I have googled adn
can only find some random conversations about it that doesn't make
sense to me.

I am basically reading in a csv file to create an xml and get this
error.

I don't see any empty values in any fields or anything...

You really should post some code and the actual traceback error your
get for us to help. I suspect that you have an ill-formed record in
your CSV file. If you can't control that, you may have to write your
own CSV dialect parser.

-Larry

Certainly, here is the code:

import os,sys
import csv
from elementtree.Ele mentTree import Element, SubElement, ElementTree

def indent(elem, level=0):
i = "\n" + level*" "
if len(elem):
if not elem.text or not elem.text.strip ():
elem.text = i + " "
for elem in elem:
indent(elem, level+1)
if not elem.tail or not elem.tail.strip ():
elem.tail = i
else:
if level and (not elem.tail or not elem.tail.strip ()):
elem.tail = i

root = Element("{Boxes }boxes")
myfile = open('test.csv' , 'rb')
csvreader = csv.reader(myfi le)

for boxid, mac, activated, hw_ver, sw_ver, heartbeat, name, address,
phone, country, city, in csvreader:
mainbox = SubElement(root , "{Boxes}box ")
mainbox.attrib["city"] = city
mainbox.attrib["country"] = country
mainbox.attrib["phone"] = phone
mainbox.attrib["address"] = address
mainbox.attrib["name"] = name
mainbox.attrib["pl_heartbe at"] = heartbeat
mainbox.attrib["sw_ver"] = sw_ver
mainbox.attrib["hw_ver"] = hw_ver
mainbox.attrib["date_activated "] = activated
mainbox.attrib["mac_addres s"] = mac
mainbox.attrib["boxid"] = boxid

indent(root)

ElementTree(roo t).write('test. xml', encoding='UTF-8')

The traceback is as follows:

Traceback (most recent call last):
File "createXMLPacka ge.py", line 35, in ?
for boxid, mac, activated, hw_ver, sw_ver, heartbeat, name,
address, phone, country, city, in csvreader:
_csv.Error: string with NUL bytes
Exit code: 1 , 0001h

[Marc 'BlackJack' Rintsch]

In <11************ **********@o5g2 000hsb.googlegr oups.com>, fscked wrote:

The traceback is as follows:

Traceback (most recent call last):
File "createXMLPacka ge.py", line 35, in ?
for boxid, mac, activated, hw_ver, sw_ver, heartbeat, name,
address, phone, country, city, in csvreader:
_csv.Error: string with NUL bytes
Exit code: 1 , 0001h

As Larry said, this most likely means there are null bytes in the CSV file.

Ciao,
Marc 'BlackJack' Rintsch

[fscked]

On May 3, 9:29 am, Marc 'BlackJack' Rintsch <[e-mail] wrote:

In <1178209090.674 787.202...@o5g2 000hsb.googlegr oups.com>, fscked wrote:

The traceback is as follows:

Traceback (most recent call last):
File "createXMLPacka ge.py", line 35, in ?
for boxid, mac, activated, hw_ver, sw_ver, heartbeat, name,
address, phone, country, city, in csvreader:
_csv.Error: string with NUL bytes
Exit code: 1 , 0001h

As Larry said, this most likely means there are null bytes in the CSV file.

Ciao,
Marc 'BlackJack' Rintsch

How would I go about identifying where it is?

[dustin]

On Thu, May 03, 2007 at 09:57:38AM -0700, fscked wrote:

As Larry said, this most likely means there are null bytes in the CSV file.

Ciao,
Marc 'BlackJack' Rintsch

How would I go about identifying where it is?

A hex editor might be easiest.

You could also use Python:

print open("filewithn uls").read().re place("\0", ">>>NUL<<<" )

Dustin

[IAmStarsky]

On May 3, 10:12 am, [e-mail]. us wrote:

On Thu, May 03, 2007 at 09:57:38AM -0700, fscked wrote:

As Larry said, this most likely means there are null bytes in the CSV file.

Ciao,
Marc 'BlackJack' Rintsch

How would I go about identifying where it is?

A hex editor might be easiest.

You could also use Python:

print open("filewithn uls").read().re place("\0", ">>>NUL<<<" )

Dustin

Hmm, interesting if I run:

print open("test.csv" ).read().replac e("\0", ">>>NUL<<<" )

every single character gets a >>>NUL<<< between them...

What the heck does that mean?

Example, here is the first field in the csv

89114608511,

the above code produces:

>>>NUL<<<8>>>NU L<<<9>>>NUL<<<1 >>>NUL<<<1>>>NU L<<<4>>>NUL<<<6 >>>NUL<<<0>>>NU L<<<8>>>NUL<<<5 >>>NUL<<<1>>>NU L<<<1>>>NUL<<< ,

[dustin]

On Thu, May 03, 2007 at 10:28:34AM -0700, IA********@gmai l.com wrote:

On May 3, 10:12 am, [e-mail]. us wrote:

On Thu, May 03, 2007 at 09:57:38AM -0700, fscked wrote:

As Larry said, this most likely means there are null bytes in the CSV file.

Ciao,
Marc 'BlackJack' Rintsch

How would I go about identifying where it is?

A hex editor might be easiest.

You could also use Python:

print open("filewithn uls").read().re place("\0", ">>>NUL<<<" )

Dustin

Hmm, interesting if I run:

print open("test.csv" ).read().replac e("\0", ">>>NUL<<<" )

every single character gets a >>>NUL<<< between them...

What the heck does that mean?

Example, here is the first field in the csv

89114608511,

the above code produces:

>>NUL<<<8>>>NUL <<<9>>>NUL<<<1> >>NUL<<<1>>>NUL <<<4>>>NUL<<<6> >>NUL<<<0>>>NUL <<<8>>>NUL<<<5> >>NUL<<<1>>>NUL <<<1>>>NUL<<< ,

I'm guessing that your file is in UTF-16, then -- Windows seems to do
that a lot. It kind of makes it *not* a CSV file, but oh well. Try

print open("test.csv" ).decode('utf-16').read().rep lace("\0", ">>>NUL<<<" )

I'm not terribly unicode-savvy, so I'll leave it to others to suggest a
way to get the CSV reader to handle such encoding without reading in the
whole file, decoding it, and setting up a StringIO file.

Dustin

[Peter Otten]

du****@v.igoro. us wrote:

I'm guessing that your file is in UTF-16, then -- Windows seems to do
that a lot. It kind of makes it *not* a CSV file, but oh well. Try

print open("test.csv" ).decode('utf-16').read().rep lace("\0",
">>>NUL<<<" )

I'm not terribly unicode-savvy, so I'll leave it to others to suggest a
way to get the CSV reader to handle such encoding without reading in the
whole file, decoding it, and setting up a StringIO file.

Not pretty, but seems to work:

from __future__ import with_statement

import csv
import codecs

def recoding_reader (stream, from_encoding, args=(), kw={}):
intermediate_en coding = "utf8"
efrom = codecs.lookup(f rom_encoding)
einter = codecs.lookup(i ntermediate_enc oding)
rstream = codecs.StreamRe coder(stream, einter.encode, efrom.decode,
efrom.streamrea der, einter.streamwr iter)

for row in csv.reader(rstr eam, *args, **kw):
yield [unicode(column, intermediate_en coding) for column in row]

def main():
file_encoding = "utf16"

# generate sample data:
data = u"\xe4hnlich,\x fcblich\r\nalph a,beta\r\ngamma ,delta\r\n"
with open("tmp.txt", "wb") as f:
f.write(data.en code(file_encod ing))

# read it
with open("tmp.txt", "rb") as f:
for row in recoding_reader (f, file_encoding):
print u" | ".join(row)

if __name__ == "__main__":
main()

Data from the file is recoded to UTF-8, then passed to a csv.reader() whose
output is decoded to unicode.

Peter

[John Machin]

On May 4, 3:40 am, [e-mail]. us wrote:

On Thu, May 03, 2007 at 10:28:34AM -0700, IAmStar...@gmai l.com wrote:

On May 3, 10:12 am, [e-mail]. us wrote:

On Thu, May 03, 2007 at 09:57:38AM -0700, fscked wrote:
As Larry said, this most likely means there are null bytes in the CSV file.

Ciao,
Marc 'BlackJack' Rintsch

How would I go about identifying where it is?

A hex editor might be easiest.

You could also use Python:

print open("filewithn uls").read().re place("\0", ">>>NUL<<<" )

Dustin

Hmm, interesting if I run:

print open("test.csv" ).read().replac e("\0", ">>>NUL<<<" )

every single character gets a >>>NUL<<< between them...

What the heck does that mean?

Example, here is the first field in the csv

89114608511,

the above code produces:

>>>NUL<<<8>>>NU L<<<9>>>NUL<<<1 >>>NUL<<<1>>>NU L<<<4>>>NUL<<<6 >>>NUL<<<0>>>NU L<<<8>>>NUL<<<5 >>>NUL<<<1>>>NU L<<<1>>>NUL<<< ,

I'm guessing that your file is in UTF-16, then -- Windows seems to do
that a lot.

Do what a lot? Encode data in UTF-16xE without putting in a BOM or
telling the world in some other fashion what x is? Humans seem to do
that occasionally. When they use Windows software, the result is
highly likely to be encoded in UTF-16LE -- unless of course the human
deliberately chooses otherwise (e.g. the "Unicode bigendian" option in
NotePad's "Save As" dialogue). Further, the data is likely to have a
BOM prepended.

The above is consistent with BOM-free UTF-16BE.