Import .csv to numpy instead of list to numpy

Benjamin Gross

Currently when I import a data file, I create a list, and then use the

Expand|Select|Wrap|Line Numbers

numpy.array(lst)

method to convert the list into a numpy array.

However, ideally I would like to not have to use a transition data type before converting it to a numpy array (and import the .csv file directly into a numpy array).

I have tried the

Expand|Select|Wrap|Line Numbers

numpy.genfromtext()

method, however, that does not keep the inherent matrix structure I am attempting to preserve (it creates a single vector instead of maintaining the distinct columns and rows from the .csv).

My import method basically looks like this:

Expand|Select|Wrap|Line Numbers

 
import csv

f = open('fileName.csv','rb')

rdr = csv.reader(f,delimeter=',')

lst = []

for row in rdr:

lst.append(row)

When I try to add something like:

Expand|Select|Wrap|Line Numbers

 
numArray = numpy.array([])

for row in rdr:

numpy.append(numArray,row)

However, I am finding that the result is just an empty numpy array at the completion of the loop.

I have also tried

Expand|Select|Wrap|Line Numbers

numpy.row_stack

, however it seems to me like the number of columns must be known apriori to use this (and I don't know how to use the

Expand|Select|Wrap|Line Numbers

csv.reader()

to determine the number of columns.

Any ideas / assistance you could provide would be most greatly appreciated.!!!

bvdet

The following bypasses the intermediate assignment to lst and typecasts each str element to float.

Expand|Select|Wrap|Line Numbers

 import csv

import numpy
 
f = open("array.txt")
 
numArray = numpy.array([map(float, item) for item in csv.reader(f)])
 
f.close()

Benjamin Gross

Thanks bvdet!

If have different types in each column, (say of string() and float()) is there any easy way for me to use the map functionality in the list comprehension you provided to account for the different data types?

Thanks for your help!

bvdet

Encapsulate the creation of the array in a function with the data type as an argument.

Expand|Select|Wrap|Line Numbers

 def create_array(fileObj, dataType):

    return numpy.array([map(dataType, item) for item in csv.reader(fileObj)])

bvdet

Oops, I misread your previous message. I am no expert on Numpy, but a Numpy array can only have one data type. Why not create a list of lists or your own container object?

Benjamin Gross

I starting off using a list of lists. However, I need to pass pairs of vectors (a date column along with a float column) into another function for processing.

In a numpy array, this consist of:

Expand|Select|Wrap|Line Numbers

numpyArray[:,0:2]

It doesn't seem like there's as clean a way to do this with lists when I'm reading in an entire matrix of data. That's why i was thinking that the numpy.array() was probably the correct data type for me to use.

bvdet

You can create a class to contain your list of lists. If you have __getitem__ and __iter__ overloads, the information you want to pass could be accessed like this:

Expand|Select|Wrap|Line Numbers

[item[0:2] for item in arrayObj]

It should work the same for a list of lists.

Benjamin Gross

I'm fairly new to Python (programming) so I don't feel all that comfortable creating classes right now. Thanks so much for your input and suggestions.

Import .csv to numpy instead of list to numpy

Post your reply

Similar topics

Question stats

	Ask Question
Home Questions Articles Browse Topics Latest Top Members FAQ