How can I read a data file into a 2xN matrix using NumPy arrays?

[csgirlie]

I'm trying to store or arrange three sets of two-dimensional data into three 2xN matrices that are stored as NumPy arrays.

import os                # for file handling functions
import numpy as np            # for array/matrix processing
import matplotlib.pyplot as plt        # for general plotting
from mpl_toolkits.mplot3d import axes3d    # for 3d plotting
 
wordList=[]
numFiles = 0
numDataSets = numVectors = sizeVector = 0
 
print"\n\nBeginning data processing."
# traverse filePath folder
filePath = "Train"
for file in os.listdir(filePath):
    numDataSets +=1                # one data set per file
    print "Loading the file: " + file + "."
    tempFile = open(filePath+"/"+file, 'rU')
    for line in tempFile:
        numVectors +=1            # one vector per line
        for word in line.split():
            sizeVector +=1        # components in vector
            wordList.append(word)
 
sizeVector = sizeVector / numVectors
numVectors = numVectors / numDataSets
 
# print"\nHere's all the data:\n", wordList, "\n\n"
print "numDataSets: ",numDataSets
print "numVectors: ",numVectors
print "sizeVector: ",sizeVector
x = eval(wordList[5])
 
# Structure our arrays to hold our data.
# We'll use the three sample class data sets from the text.
# sizeVector of rows and numVectors of columns for column vector data.
# In other words, each feature vector will be a column vector for set array.
w1 = np.zeros((sizeVector, numVectors))
w2 = np.zeros((sizeVector, numVectors))
w3 = np.zeros((sizeVector, numVectors))
 
#   Load matrix array with class w1 numeric values,
# replacing elements indexed as in matrices row, col starting from 0.
#   Pull elements from wordList based on dataSet, sizeVector, and
# which vector we're on during the column, row iteration.
 
skip = numVectors*sizeVector
 
dataSet = 0    
for j in range(sizeVector):
    for i in range(numVectors):
        (do something - nothing I've tried is working)
print "\nClass w1 is as follows: \n",w1
 
dataSet = 1    
for j in range(sizeVector):
    for i in range(numVectors):
        (do something - nothing I've tried is working)
print "\nClass w2 is as follows: \n",w2
 
dataSet = 2    
for j in range(sizeVector):
    for i in range(numVectors):
        (do something - nothing I've tried is working)
print "\nClass w3 is as follows: \n",w3

I need advice on how to get my dataSet(s) set up in this 2xN format.

[Glenton]

Hi

This is a bit difficult to engage on, because of the level of detail. You might also get better responses if you say more generally what you're trying to achieve and why.

But, if I understand it, you have a list, called wordList which has (3 x numVectors x sizeVector) elements, and you want to change it into 3 separate numpy arrays each of which has 2 columns and (numVectors x sizeVector)/2 rows. Or, perhaps into arrays which have sizeVector columns and numVectors rows.

So the key lines might be something like
w1[i,j]=wordList[i+numVectors*j]
w2[i,j]=wordList[i+numVectors*j+ size]
w2[i,j]=wordList[i+numVectors*j+ 2*size]

or maybe w1[i][j]

I haven't had a chance to check it yet. Is something like this what you've tried?