473,883 Members | 1,712 Online
Bytes | Software Development & Data Engineering Community
+ Post

Home Posts Topics Members FAQ

looping through a big file containing a set of files.

111 New Member
hey!
I have a program that takes two input files(one in the matrix form) and one in the sequence form.Now my problem is that i have to give the matrix file(containing many matrices) and sequence file containing many sequences and calculate the same log score as I did for one matrix file and one sequence file.
how it should exactly work is that. for every sequence it should calculate log values for all the weight matrices,then go to the second sequence and calculate all the log values using the matrices.
my matrix file is huge containing so many matrices. a part of it is here.

//
NA Abd-B
PO A C G T
01 10.19 0.00 10.65 6.24
02 5.79 0.67 10.50 10.11
03 4.50 0.00 0.00 22.57
04 0.00 0.00 0.00 27.08
05 0.00 0.00 0.00 27.08
06 0.00 0.00 0.00 27.08
07 27.08 0.00 0.00 0.00
08 0.00 2.83 0.00 24.25
09 0.00 0.00 24.45 2.62
10 19.33 0.00 4.34 3.41
11 0.31 12.28 3.39 11.09
//
//
NA Adf1
PO A C G T
01 0.71 0.08 26.02 1.55
02 3.03 23.00 1.24 1.09
03 0.26 10.50 3.29 14.31
04 0.00 0.06 28.23 0.07
05 0.12 27.27 0.06 0.91
06 1.44 20.36 0.37 6.19
07 5.35 0.28 21.49 1.24
08 7.81 16.10 3.81 0.63
09 0.51 17.77 0.45 9.63
10 0.00 0.14 28.21 0.00
11 0.00 25.69 0.20 2.46
12 0.48 9.98 0.07 17.82
13 1.27 0.00 27.01 0.07
14 15.59 7.98 2.92 1.87
15 4.28 22.37 0.00 1.70
16 0.18 0.77 22.70 4.70
//
//
NA Aef1
PO A C G T
01 0.00 0.06 12.49 0.00
02 3.80 0.17 0.00 8.57
03 0.87 0.06 0.00 11.62
04 0.06 9.76 2.32 0.41
05 9.82 0.00 2.73 0.00
06 9.76 0.00 0.00 2.78
07 3.80 0.31 0.00 8.43
08 0.00 0.00 0.00 12.54
09 0.00 6.53 5.85 0.17
10 0.00 12.38 0.17 0.00
11 2.73 1.02 8.80 0.00
12 5.85 0.00 6.70 0.00
13 1.02 5.96 0.00 5.57
14 0.00 5.16 4.66 2.73
15 1.03 7.55 3.97 0.00
16 4.82 5.00 2.73 0.00
//
//
NA Antp
PO A C G T
01 5.52 14.49 27.56 0.49
02 8.17 14.02 11.42 14.47
03 18.18 27.29 1.31 1.29
04 40.26 5.66 1.83 0.32
05 19.05 12.67 0.43 15.91
06 9.94 0.07 0.20 37.86
07 26.63 15.17 0.00 6.27
08 47.45 0.06 0.00 0.56
09 0.81 0.48 0.00 46.79
10 26.46 19.05 1.81 0.75
11 48.07 0.00 0.00 0.00
12 30.51 0.00 0.00 17.56
13 43.45 0.00 0.00 4.62
14 30.06 5.98 0.00 12.03
15 0.38 0.64 0.00 47.05
16 22.14 0.29 7.15 18.49
//
//

the sequence file is here( I mean this is also a part of my file)the actual file starts from "CC" the line before is just heading which we omit and this file is containg two sequences.
>CG9571_O-E|Drosophila melanogaster|CG 9571|FBgn003108 6|X:19926374..1 9927133
CCAGTCCACCGGCCG CCGATCTATTTATAC GAGAGGAAGAGGCTG AACTCGAGGATTACC CGTGTATCCTGGGAC GCG
GATTAGCGATCCATT CCCCTTTTAATCGCC GCGCAAACAGATTCA TGAAAGCCTTCGGAT TCATTCATTGATCCA CAT
CTACGGGAACGGGAG TCGCAAACGTTTTCG GATTAGCGCTGGACT AGCGGTTTCTAAATT GGATTATTTCTACCT GAC
CCTGGAGCCATCGTC CTCGTCCTCC
>Cp36_DRR|Droso phila melanogaster|Cp 36|FBgn0000359| X:8323349..8324 136
AGTCGACCAGCACGA GATCTCACCTACCTT CTTTATAAGCGGGGT CTCTAGAAGCTAAAT CCATGTCCACGTCAA ACC
AAAGACTTGCGGTCT CCAGACCATTGAGTT CTATAAATGGGACTG AGCCACACCATACAC CACACACCACACATA CAC
ACACGCCAACACATT ACACACAACACGAAC TACACAAACACTGAG ATTAAGGAAATTATT AAAAAAAATAATAAA ATT
AATACAAAAAAAATA TATATATATA
this is my code which works(prints the log value for one sequence and one matrix)
Expand|Select|Wrap|Line Numbers
  1. from math import *
  2. import random
  3. f=open("deeps1.txt","r")
  4. line=f.next()
  5. while not line.startswith('PO'):
  6.     line=f.next()
  7.  
  8. headerlist=line.strip().split()[1:]
  9. linelist=[]
  10.  
  11.  
  12. line=f.next().strip()
  13. while not line.startswith('/'):
  14.     if line != '':
  15.         linelist.append(line.strip().split())
  16.     line=f.next().strip()
  17.  
  18. keys=[i[0] for i in linelist]
  19. values=[[float(s) for s in item] for item in [j[1:] for j in linelist]]
  20.  
  21. array={}
  22. linedict=dict(zip(keys,values))
  23. keys = linedict.keys()
  24. keys.sort()
  25. for key in keys:
  26.     array=[key,linedict[key]]
  27.  
  28. datadict={}
  29. datadict1={}
  30. for i,item in enumerate(headerlist):
  31.     datadict[item]={}
  32.     for key_ in linedict:
  33.         datadict[item][key_]=linedict[key_][i]
  34.  
  35.  
  36. for keymain in datadict:
  37.     for keysub in datadict[keymain]:
  38.         datadict[keymain][keysub]+=1.0
  39.  
  40. datadict1=datadict.copy()
  41. for keysub in datadict:
  42.     for keysub in datadict[keymain]:
  43.         datadict1[keymain][keysub]=datadict[keymain][keysub]/(sum(values[int(keysub)-1])+4)
  44.  
  45.  
  46.  
  47. def readfasta():
  48.     file1= open("chr011.py",'r')
  49.     file_content=file1.readlines()
  50.     first=1
  51.     list1=""    
  52.     for line in file_content:
  53.         if line[0]==">":
  54.             if first==0:
  55.                 print "***********"
  56.                 list1+=sequence
  57.                 print "***********"
  58.             else:
  59.                 first=0
  60.                 sequence=""
  61.                 seq=""
  62.                 for i in range(0,len(line)-1):
  63.                     seq+=line[i]
  64.         else:
  65.                 for i in range(0,len(line)-1):
  66.             sequence+=line[i]  
  67.     list1+=sequence
  68.     return list1
  69.  
  70.  
  71.  
  72. p=readfasta()
  73.  
  74.  
  75.  
  76.  
  77.  
  78. res=1
  79. part=""
  80. q=len(p)
  81. seqq=""
  82.  
  83. value={"A":0.3,"T":0.3,"C":0.2,"G":0.2}
  84. for i in range(q-16):
  85.     part=p[i:i+16]
  86.     seqq=part
  87.     res=1
  88.     score=1
  89.     for j in range(16):
  90.         key=seqq[j]
  91.         res=res*datadict1[key]["%02d"%(j+1)]
  92.         #print res
  93.     for key in seqq:
  94.         score=score * value[key]
  95.     #print score,"*******************",res
  96.     log_ratio=log10(res/score)
  97.     print i,log_ratio
  98.  
what changes should i make and how?/
waiting for your reply,
cheers!
Jul 13 '07
103 5934
aboxylica
111 New Member
and please do send the full code. i tried executing mine.looks like there are some errors.so il check up with urs and try executing.
waiting,
cheers!!
Jul 16 '07 #51
bvdet
2,851 Recognized Expert Moderator Specialist
and what does this line mean??
Expand|Select|Wrap|Line Numbers
  1. if __name__ == '__main__':
  2.  
A module type is a container that holds objects loaded with the import statement. The identifier '__name__' is assigned to the module name. If the module is run as a top level program, '__name__' is assigned to the string '__main__'.

Expand|Select|Wrap|Line Numbers
  1. if __name__ == '__main__':
  2.     .... execute code ....
  3.  
The code under the 'if' statement only executes if the module is run as a program. That way the functions defined above can be imported by other programs without executing the conditional code.
Jul 16 '07 #52
bvdet
2,851 Recognized Expert Moderator Specialist
and please do send the full code. i tried executing mine.looks like there are some errors.so il check up with urs and try executing.
waiting,
cheers!!
The whole thing:
Expand|Select|Wrap|Line Numbers
  1. def parseArray(fn, dataset=1, key='PO', term='/'):
  2.     '''
  3.     Read a formatted data file in matrix format and
  4.     compile data into a dictionary
  5.     '''
  6.     f = open(fn)
  7.  
  8.     # skip to required data set
  9.     for _ in range(dataset):
  10.         try:
  11.             line = f.next()
  12.             while not line.startswith(key):
  13.                 line = f.next()
  14.         except StopIteration, e:
  15.             print 'We have reached the end of the file!'
  16.             f.close()
  17.             return False
  18.  
  19.     headerList = line.strip().split()[1:]
  20.     lineList = []
  21.  
  22.     line = f.next().strip()
  23.     while not line.startswith(term):
  24.         if line != '':
  25.             lineList.append(line.strip().split())
  26.         line = f.next().strip()
  27.  
  28.     f.close()
  29.  
  30.     # Key list
  31.     keys = [i[0] for i in lineList]
  32.     # Values list
  33.     values = [[float(s) for s in item] for item in [j[1:] for j in lineList]]
  34.  
  35.     # Create a dictionary from keys and values
  36.     lineDict = dict(zip(keys, values))
  37.  
  38.     dataDict = {}
  39.  
  40.     for i, item in enumerate(headerList):
  41.         dataDict[item] = {}
  42.         for key in lineDict:
  43.             dataDict[item][key] = lineDict[key][i]
  44.  
  45.     # Add 1.0 to every element in dataDict subdictionaries
  46.     for keyMain in dataDict:
  47.         for keySub in dataDict[keyMain]:
  48.             dataDict[keyMain][keySub] += 1.0
  49.  
  50.     # Normalize original data (with 1 added) and update data
  51.     valueSums = [sum(item)+4 for item in values]
  52.     # print valueSums
  53.  
  54.     for keyMain in dataDict:
  55.         for keySub in dataDict[keyMain]:
  56.             dataDict[keyMain][keySub] /= valueSums[int(keySub)-1]
  57.  
  58.     return dataDict
  59.  
  60.  
  61. def parseData(fn, dataset=1, key='>'):
  62.     '''
  63.     Read a formatted data file of alpha sequences
  64.     Return a list of sequences
  65.     The first element in the list is the header
  66.     '''    
  67.     # initialize output list
  68.     dataList = []
  69.  
  70.     # open file for reading
  71.     f = open(fn)
  72.  
  73.     # skip to required data set
  74.     for _ in range(dataset):
  75.         try:
  76.             s = f.next()
  77.             while not s.startswith(key):
  78.                 s = f.next()
  79.         except StopIteration, e:
  80.             print 'We have reached the end of the file!'
  81.             f.close()
  82.             return False
  83.  
  84.     # initialize output list
  85.     dataList = [s,]
  86.  
  87.     for line in f:
  88.         if not line.startswith(key):
  89.             dataList.append(line.strip())
  90.         else:
  91.             break
  92.  
  93.     f.close()
  94.     return dataList
  95.  
  96. if __name__ == '__main__':
  97.  
  98.     arraySet = 1
  99.     seqSet = 3
  100.  
  101.     value={"A":0.3,"T":0.3,"C":0.2,"G":0.2}
  102.  
  103.     fnArray = r'H:\TEMP\temsys\data9.txt'
  104.     fnSeq = r'H:\TEMP\temsys\data12.txt'
  105.  
  106.     dataArray = parseArray(fnArray, arraySet)
  107.     dataSeq = parseData(fnSeq, seqSet)
  108.  
  109.     # This is the complete sequence  
  110.     seq = ''.join(dataSeq[1:])
  111.     # These are the subkeys of dataArray - '01', '02', '03',.............
  112.     subKeys = dataArray['A'].keys()
  113.     subKeys.sort()
  114.  
  115.     # Calculate num/den for each slice of sequence
  116.     # Each sequence slice length = length of subKeys
  117.     # Example:
  118.     # seq = 'ATCGATA'
  119.     # subKeys length = 3
  120.     # 'ATC', 'TCG', 'CGA', 'GAT', 'ATA'
  121.     numList = []
  122.     denList = []
  123.     seqList = []
  124.     for i in xrange(len(seq) - len(subKeys) + 1):
  125.         subseq = seq[0:len(subKeys)]
  126.         seqList.append(subseq)
  127.         num, den = 1, 1
  128.         for j, s in enumerate(subseq):
  129.             num *= dataArray[s][subKeys[j]]
  130.             den *= value[s]
  131.         numList.append(num)
  132.         denList.append(den)
  133.         seq = seq[1:]
  134.  
  135.     resultList = []
  136.     for i, num in enumerate(numList):
  137.         resultList.append(num/denList[i])
  138.  
  139.     outStr = '\n'.join(['Sequence = %s Calculation = %0.12f' % (seqList[i], res) for i, res in enumerate(resultList)])
  140.     print 'Array set # = %d\nSequence set # = %d' % (arraySet, seqSet)
  141.     print 'Sequence Header: %s' % dataSeq[0]
  142.     print outStr
Check the values in variables dataArray and dataSeq for appropriateness .
Jul 16 '07 #53
aboxylica
111 New Member
has log(num/deno) been done??
Jul 17 '07 #54
aboxylica
111 New Member
looks like it is correct, I inserted a known sequence at a particular position which should yield a high value,it did give a hike at that position:) ..i did the log calculation..
Expand|Select|Wrap|Line Numbers
  1.  for i, num in enumerate(numList):
  2.  
  3.         resultList.append(log10(num/denList[i]))
  4.         print resultList
  5.  
looks right to me.!only generalising this is left!!
cheers!!
Jul 17 '07 #55
aboxylica
111 New Member
hey,
I am waiting for ur reply bvdet.
I know am disturbing you too much but am really new to programming and python.so thats why I couldn find a way out.
waiting,
cheers!!
Jul 17 '07 #56
bvdet
2,851 Recognized Expert Moderator Specialist
hey,
I am waiting for ur reply bvdet.
I know am disturbing you too much but am really new to programming and python.so thats why I couldn find a way out.
waiting,
cheers!!
You have taken on an ambitious project if you are new to programming. I have given you all that you need to complete the project. All you need to do is:
Iterate on each data file until you reach the end of each.
Compile the data for each iteration.
Something like this (untested):
Expand|Select|Wrap|Line Numbers
  1. indxSeq = 1
  2. while True:
  3.     dataSeq = parseData(fnSeq, indxSeq)
  4.     if not dataSeq:
  5.         break
  6.     indxArray = 1
  7.         while True:
  8.             dataArray = parseArray(fnArray, indxArray)
  9.             if not dataArray:
  10.                 break
  11.             ............. calculations ................ 
  12.             indxArray += 1
  13.         indxSeq += 1
It is not fair to me or others on this list for me to write the whole program for you. Study some tutorials, read Python documentation, practice with some simple exercises, and give it a shot. Come back with some code, and we will try to help if you need it. This project is interesting. Good luck and have fun!
Jul 17 '07 #57
aboxylica
111 New Member
hey,
Thanks a lot. I will try and come back with queries.sorry for bugging you.
Jul 17 '07 #58
bvdet
2,851 Recognized Expert Moderator Specialist
hey,
Thanks a lot. I will try and come back with queries.sorry for bugging you.
You don't have to apologize. I only did what I wanted to. :)
Jul 17 '07 #59
aboxylica
111 New Member
hey,
Thats very sweet of you:)
I dont really know how to improve.as in when am reading the tutorial.I just gain theoritical knowledge.How do I put it to practical knowledge.
I have gone through the python documentation.B ut when it comes to programming I cant really find a way out.like this case.. I had the code to work for a single sequence and a single matrix.I could have improvised that..but i was not able to think.how do I crack it and become better?
Jul 17 '07 #60

Sign in to post your reply or Sign up for a free account.

Similar topics

8
4066
by: kaptain kernel | last post by:
i've got a while loop thats iterating through a text file and pumping the contents into a database. the file is quite large (over 150mb). the looping causes my CPU load to race up to 100 per cent. Even if i remove the mysql insert query and just loop through the file , it still hits 100 per cent CPU. This has the knock on effect of slowing my script down so that mysql inserts are occuring every 1/2 second or so.
5
9444
by: B-Dog | last post by:
I have an old dos program that uses dat files to store the data and I'm trying to convert to dotnet. I'd like to be able to import the data into an access database but I don't know which format the dat files are in. Here is the first few lines of the dat file. If anyone could help me figure out which type this is it would be greatly appreciated. ˙˙˙˙ ®   öf¨"ó Mr John Smithh Parkside Jewelers 1776...
2
24233
by: deko | last post by:
I have a table that contains a bunch of pictures. When the user selects a particular image in a form, I need a way to extract the selected bitmap image (stored in an OLE Object table field) to the file system so the user can do stuff with "somePicture.bmp", for example. Is there an easy way to do this? Thanks in advance.
10
24098
by: bienwell | last post by:
Hi, I have a question about file included in ASP.NET. I have a file that includes all the Sub functions (e.g FileFunct.vb). One of the functions in this file is : Sub TestFunct(ByVal strInput As String) return (strInput & " test") End Sub
3
9565
by: Chung Leong | last post by:
Here's the rest of the tutorial I started earlier: Aside from text within a document, Indexing Service let you search on meta information stored in the files. For example, MusicArtist and MusicAlbum let you find MP3 and other music files based on the singer and album name; DocAuthor let you find Office documents created by a certain user; DocAppName let you find files of a particular program, and so on. Indexing Service uses plug-ins...
1
2961
by: Alex | last post by:
Hello, I have a stored procedure that processes an individual file from a directory and archives it in a subdirectory.Now, the problem is, when i execute it , it will only process one file. What i want to do is to check to see if there are any files in the folder, and if there are , process them all, and once done, go to the next part in a DTS package, if there are no files, simply go to the next part in the DTS package. I tried an...
1
6525
by: laredotornado | last post by:
Hi, I'm using PHP 4.4.4 on Apache 2 on Fedora Core 5. PHP was installed using Apache's apxs and the php library was installed to /usr/local/php. However, when I set my "error_reporting" setting to be "E_ALL", notices are still not getting reported. The perms on my file are 664, with owner root and group root. The php.ini file is located at /usr/local/lib/php/php.ini. Any ideas why the setting does not seem to be having an effect? ...
5
2699
by: Mark | last post by:
Hi I have an application (in vb.NET 2005) which holds data in SQL Server and some of the SQL records are simply paths to related files. I would like to be able to do a text search on both the SQL data and the contents of any referenced files. The returned list being a listing which includes both records containing the text and files containing the text. Does anyone have a simple example of programatically searching for files
0
1971
by: anthon | last post by:
Hi all - first post! anywho; I need to create a function for speeding up and down a looping clip. imagine a rotating object, triggered by an action, and slowly decreasing in speed, till it reaches a point 0 (compare a hand spinning a fortune wheel). now, this is quite an easy this to achieve, since you just have to set an interval to increase rotation, with a value that decreases over time (on every call), till it finally reaches a...
0
9933
marktang
by: marktang | last post by:
ONU (Optical Network Unit) is one of the key components for providing high-speed Internet services. Its primary function is to act as an endpoint device located at the user's premises. However, people are often confused as to whether an ONU can Work As a Router. In this blog post, we’ll explore What is ONU, What Is Router, ONU & Router’s main usage, and What is the difference between ONU and Router. Let’s take a closer look ! Part I. Meaning of...
0
9781
by: Hystou | last post by:
Most computers default to English, but sometimes we require a different language, especially when relocating. Forgot to request a specific language before your computer shipped? No problem! You can effortlessly switch the default language on Windows 10 without reinstalling. I'll walk you through it. First, let's disable language synchronization. With a Microsoft account, language settings sync across devices. To prevent any complications,...
0
11123
Oralloy
by: Oralloy | last post by:
Hello folks, I am unable to find appropriate documentation on the type promotion of bit-fields when using the generalised comparison operator "<=>". The problem is that using the GNU compilers, it seems that the internal comparison operator "<=>" tries to promote arguments from unsigned to signed. This is as boiled down as I can make it. Here is my compilation command: g++-12 -std=c++20 -Wnarrowing bit_field.cpp Here is the code in...
0
10407
tracyyun
by: tracyyun | last post by:
Dear forum friends, With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
0
9567
agi2029
by: agi2029 | last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own.... Now, this would greatly impact the work of software developers. The idea...
0
7114
by: conductexam | last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one. At the time of converting from word file to html my equations which are in the word document file was convert into image. Globals.ThisAddIn.Application.ActiveDocument.Select();...
0
5982
by: adsilva | last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
1
4606
by: 6302768590 | last post by:
Hai team i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
3
3230
bsmnconsultancy
by: bsmnconsultancy | last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.