Import .csv to numpy instead of list to numpy

[Benjamin Gross]

Currently when I import a data file, I create a list, and then use the

numpy.array(lst)

method to convert the list into a numpy array.

However, ideally I would like to not have to use a transition data type before converting it to a numpy array (and import the .csv file directly into a numpy array).

I have tried the

numpy.genfromtext()

method, however, that does not keep the inherent matrix structure I am attempting to preserve (it creates a single vector instead of maintaining the distinct columns and rows from the .csv).

My import method basically looks like this:

import csv
f = open('fileName.csv','rb')
rdr = csv.reader(f,delimeter=',')
lst = []
for row in rdr:
lst.append(row)

When I try to add something like:

numArray = numpy.array([])
for row in rdr:
numpy.append(numArray,row)

However, I am finding that the result is just an empty numpy array at the completion of the loop.

I have also tried

numpy.row_stack

, however it seems to me like the number of columns must be known apriori to use this (and I don't know how to use the

csv.reader()

to determine the number of columns.

Any ideas / assistance you could provide would be most greatly appreciated.!!!

[bvdet]

The following bypasses the intermediate assignment to lst and typecasts each str element to float.

import csv
import numpy
 
f = open("array.txt")
 
numArray = numpy.array([map(float, item) for item in csv.reader(f)])
 
f.close()

[Benjamin Gross]

Thanks bvdet!

If have different types in each column, (say of string() and float()) is there any easy way for me to use the map functionality in the list comprehension you provided to account for the different data types?

Thanks for your help!

[bvdet]

Encapsulate the creation of the array in a function with the data type as an argument.

def create_array(fileObj, dataType):
    return numpy.array([map(dataType, item) for item in csv.reader(fileObj)])

[bvdet]

Oops, I misread your previous message. I am no expert on Numpy, but a Numpy array can only have one data type. Why not create a list of lists or your own container object?

[Benjamin Gross]

I starting off using a list of lists. However, I need to pass pairs of vectors (a date column along with a float column) into another function for processing.

In a numpy array, this consist of:

numpyArray[:,0:2]

It doesn't seem like there's as clean a way to do this with lists when I'm reading in an entire matrix of data. That's why i was thinking that the numpy.array() was probably the correct data type for me to use.

[bvdet]

You can create a class to contain your list of lists. If you have __getitem__ and __iter__ overloads, the information you want to pass could be accessed like this:

[item[0:2] for item in arrayObj]

It should work the same for a list of lists.

[Benjamin Gross]

I'm fairly new to Python (programming) so I don't feel all that comfortable creating classes right now. Thanks so much for your input and suggestions.