creating huge transition matrix

[suppoor]

Hello every one ,

I am writing a code to generate huge transition matrix (2 order markov and one order markov)

here are my question :

1) is there any available source for efficiently generating mrrkov transition matrix ONLY ?
2) I have around 800000 nodes so what could be a better way to create such a huge structure ---> i am looking for access speed into an element in the transition matrix

Any suggestion about tools which might be more efficient .

Thanks in advance .

Regards

[Glenton]

You definitely want to be using numpy and/or scipy for the matrix structures. Then the heavy lifting will be done in the background with C, and you can do array operations rather than loop operations.

Here's a link to someone who tackled this problem:
Large Markov Chains with Python

[suppoor]

hi glen ton thanks for your suggestion,

Well i had a look into to NUMpy and SCIpy . I believe my requirement is more complex as i need 3 dimensional where the element in the 3rd dimension element might have to point 2 dimensional element in some cases.

I didn't understand by the " heavy lifting will be done by C" .. you r referring to numpy implementation ? or i need to implement it explicitly ?

[Glenton]

Hi. Numpy and Scipy were created to handle n-dimensional arrays so 3 is certainly not a problem. It's very similar to matlab (except better IMHO).

When you program in compiled languages like C or Java you get a huge performance benefit over higher level languages like python or perl. The flip side is that the actual coding is generally far simpler and more terse in python. Eg C may take 100th time to run but have triple the lines of code.

Also handling things like graphing, files, user interface and other scripting functions is far simpler in a high level language. Thus they often work quite well together. However numpy captures a great deal of this benefit *invisibly* to the coder. If you know how to use it you can capture even more benefit, Eg by using array functions instead of looping through the array.

However if that's not useful you could just use lists of lists of lists.