kdt 50
New Member
Hi,
I'm checking to see if you guys may be able to help me with an algorithm for finding patterns. I have around 2000 short sequences (of length 9) that are aligned. I want to be able to extract all common patterns on the same positions and report the number of occurrences.
For example in the following:
ACGCATTCA
ACTGGATAC
TCAGCCATC
I would like the following output (where a full stop represents any character)
(AC....T..) 2 occurrences (pattern between sequence 1 and 2)
(.C.G...C) 2 occurrences (pattern between sequence 2 and 3)
(.C.......) 2 occurrences (pattern between sequence 1 and 3)
As you can see, the way that I am planning on doing this now requires sum(n-1...1) comparisons. Is there a more efficient way of doing this with less comparisons?
Thanks
13 2541 bartonc 6,596
Recognized Expert Expert
Hi,
I'm checking to see if you guys may be able to help me with an algorithm for finding patterns. I have around 2000 short sequences (of length 9) that are aligned. I want to be able to extract all common patterns on the same positions and report the number of occurrences.
For example in the following:
ACGCATTCA
ACTGGATAC
TCAGCCATC
I would like the following output (where a full stop represents any character)
(AC....T..) 2 occurrences (pattern between sequence 1 and 2)
(.C.G...C) 2 occurrences (pattern between sequence 2 and 3)
(.C.......) 2 occurrences (pattern between sequence 1 and 3)
As you can see, the way that I am planning on doing this now requires sum(n-1...1) comparisons. Is there a more efficient way of doing this with less comparisons?
Thanks
Coincidentally enough, regular expressions use the "full stop" as you have specified (but in a regex, it's called a "dot"). I can give more detail, but am in a rush at the moment: -
>>> s = 'ACGCATTCA\nACTGGATAC\nTCAGCCATC\n'
-
>>> s = s * 100
-
>>> import re
-
>>> patternA = re.compile('AC....T..', re.MULTILINE)
-
>>> patternA.match(s)
-
-
>>> res = patternA.findall(s)
-
>>> len(res)
-
200
-
>>>
[EDIT] I've re-read the problem and have an idea. Back soon.[/EDIT]
bartonc 6,596
Recognized Expert Expert
Coincidentally enough, regular expressions use the "full stop" as you have specified (but in a regex, it's called a "dot"). I can give more detail, but am in a rush at the moment: -
>>> s = 'ACGCATTCA\nACTGGATAC\nTCAGCCATC\n'
-
>>> s = s * 100
-
>>> import re
-
>>> patternA = re.compile('AC....T..', re.MULTILINE)
-
>>> patternA.match(s)
-
-
>>> res = patternA.findall(s)
-
>>> len(res)
-
200
-
>>>
[EDIT] I've re-read the problem and have an idea. Back soon.[/EDIT]
I think that I've got it: -
>>> patterns = ['AC....T..', '.C.G....C', '.C.......']
-
>>> reObjs = [re.compile(pat) for pat in patterns]
-
>>> s = 'ACGCATTCA\nACTGGATAC\nTCAGCCATC'
-
>>> for i, reObj in enumerate(reObjs):
-
... print "pattern: %s matches at the following positions:" %patterns[i]
-
... result = reObj.finditer(s)
-
... for match in result:
-
... print (match.start() / 10) + 1
-
...
-
pattern: AC....T.. matches at the following positions:
-
1
-
2
-
pattern: .C.G....C matches at the following positions:
-
2
-
3
-
pattern: .C....... matches at the following positions:
-
1
-
2
-
3
-
>>>
And it points out an error in your example.
bartonc 6,596
Recognized Expert Expert
I think that I've got it: -
>>> patterns = ['AC....T..', '.C.G....C', '.C.......']
-
>>> reObjs = [re.compile(pat) for pat in patterns]
-
>>> s = 'ACGCATTCA\nACTGGATAC\nTCAGCCATC'
-
>>> for i, reObj in enumerate(reObjs):
-
... print "pattern: %s matches at the following positions:" %patterns[i]
-
... result = reObj.finditer(s)
-
... for match in result:
-
... print (match.start() / 10) + 1
-
...
-
pattern: AC....T.. matches at the following positions:
-
1
-
2
-
pattern: .C.G....C matches at the following positions:
-
2
-
3
-
pattern: .C....... matches at the following positions:
-
1
-
2
-
3
-
>>>
And it points out an error in your example.
Putting it all together, you get: -
>>> for i, reObj in enumerate(reObjs):
-
... positions = [((match.start() / 10) + 1) for match in reObj.finditer(s)]
-
... nTimes = len(positions)
-
... print "pattern: %s matches %d times, at the following positions: %s" %(patterns[i], nTimes, str(positions))
-
-
-
pattern: AC....T.. matches 2 times, at the following positions: [1, 2]
-
pattern: .C.G....C matches 2 times, at the following positions: [2, 3]
-
pattern: .C....... matches 3 times, at the following positions: [1, 2, 3]
Hi,
I'm checking to see if you guys may be able to help me with an algorithm for finding patterns. I have around 2000 short sequences (of length 9) that are aligned. I want to be able to extract all common patterns on the same positions and report the number of occurrences.
For example in the following:
ACGCATTCA
ACTGGATAC
TCAGCCATC
I would like the following output (where a full stop represents any character)
(AC....T..) 2 occurrences (pattern between sequence 1 and 2)
(.C.G...C) 2 occurrences (pattern between sequence 2 and 3)
(.C.......) 2 occurrences (pattern between sequence 1 and 3)
As you can see, the way that I am planning on doing this now requires sum(n-1...1) comparisons. Is there a more efficient way of doing this with less comparisons?
Thanks
i don't see why you can't use string slicing and indexing.
eg -
...
-
if line[0:2] == "AC" and line[6] == "T":
-
#do something and so on
-
....
-
kdt 50
New Member
hi
thanks for all your replies, but I dont think I made my requirements clear.
I am looking to find all existing patterns in a bunch of sequences, I will not know the patterns beforehand so I want to find all patterns that exist in more than two sequences
thanks
bvdet 2,851
Recognized Expert Moderator Specialist
hi
thanks for all your replies, but I dont think I made my requirements clear.
I am looking to find all existing patterns in a bunch of sequences, I will not know the patterns beforehand so I want to find all patterns that exist in more than two sequences
thanks
This took about a minute to run on my machine on 2000 strings, and produced a data file with 83,000,000 bytes. ' sList' is the data sequence. - from sets import Set as set
-
import re
-
import time
-
-
patt = re.compile('[ACGT]')
-
-
outList = ['Opening output file: %s' % time.ctime(), ]
-
-
dd = {}
-
indx = 0
-
-
while len(sList) > 0:
-
s1 = sList[0]
-
for j, item in enumerate(sList[1:]):
-
res = ''
-
for i, s in enumerate(s1):
-
if s == item[i]:
-
res += s
-
else:
-
res += '.'
-
if patt.search(res):
-
if dd.has_key(res):
-
dd[res].append([indx, j+1+indx])
-
else:
-
dd[res] = [[indx, j+1+indx], ]
-
indx += 1
-
-
sList.pop(0)
-
-
keys = dd.keys()
-
keys.sort()
-
-
for key in keys:
-
quan = len(set([item[j] for j in range(2) for item in dd[key]]))
-
outList.append('(%s) %d occurrences:' % (key, quan))
-
for v in dd[key]:
-
outList.append(' Pattern between sequence %d and %d' % (v[0], v[1]))
-
-
outList.append('Writing data to output file: %s' % time.ctime())
-
fn = r'H:\TEMP\temsys\string_patterns.txt'
-
f = open(fn, 'w')
-
f.write('\n'.join(outList))
-
f.close()
This is probably similar to the algorithm you were going to use. Here's a sample from the output: - Opening output file: Wed Aug 01 08:50:53 2007
-
(........A) 676 occurrences:
-
Pattern between sequence 0 and 72
-
Pattern between sequence 0 and 89
-
.......................
-
(....CA.G.) 31 occurrences:
-
Pattern between sequence 30 and 60
-
Pattern between sequence 30 and 76
-
Pattern between sequence 30 and 242
-
Pattern between sequence 30 and 388
-
.......................
-
(TTGCCA.A.) 2 occurrences:
-
Pattern between sequence 612 and 1016
-
(TTGCCAA..) 2 occurrences:
-
Pattern between sequence 33 and 1016
-
Writing data to output file: Wed Aug 01 08:51:53 2007
kdt 50
New Member
Very nice indeed bvdet. I need to make some slight changes, but this is a great step in the right direction. Thank you soo much.
Would anyone happen to know some tips to speed this up. I think you can use psycho to compile parts in C, but is there a way of converting to binary and then performing manipulations - would this be faster?
Thanks
bartonc 6,596
Recognized Expert Expert
Very nice indeed bvdet. I need to make some slight changes, but this is a great step in the right direction. Thank you soo much.
Would anyone happen to know some tips to speed this up. I think you can use psycho to compile parts in C, but is there a way of converting to binary and then performing manipulations - would this be faster?
Thanks
Tests that I've run on other code suggest that using tuples instead of lists gives a significant performance boost: - # line 21 here
-
dd[res].append((indx, j+1+indx))
-
else:
-
dd[res] = [(indx, j+1+indx), ]
If that list is appended to elsewhere, the tuples can be added together more quickly than the append, also.
kdt 50
New Member
Hi,
A quick question on the above algorithm, it seems to performing more comparisons than necessary, though I can't seem to find where.
I decided to experiment to get the number of comparisons required to compare each line with every other line - and not against itself (line 1 to 1) or lines already compared ( as comparing line 1 to 2 is the same as comparing line 2 to 1). -
line_num = 0
-
counter = 0
-
-
for line in myList:
-
elmt_num = 0
-
-
for elmt in myList:
-
i = 0
-
if line_num < elmt_num # As I don't want to compare twice and against the same seq
-
while i<9:
-
counter += 1
-
i += 1
-
-
elmt_num += 1
-
-
line_num += 1
-
-
print "%d comparisons were made between %d lines" % (counter, line_num)
-
I get the following output for my dataset 7587459 comparisons were made between 1299 lines
Which is exactly what I would expect from comparing 9 times (n**2 -n)/2 where n is the number of sequences.
The output I get from bvdet's algorithm is however 8430510 comparisons (10 time (n**2-n)/2) I fail to see where the extra 1 comparison each time is coming from.
If someone can let me know, I would be thankful.
Cheers
Sign in to post your reply or Sign up for a free account.
Similar topics |
by: kittykat |
last post by:
Hi,
I was wondering if you could help me. I am writing a program in C++, and
the problem is, i have very limited experience in this language.
I would like my user to enter a specific pattern, and I want my program to
search a text file for this pattern, and let the user know if this pattern
exists or not.
So far, i have figured out how to make my prgram read the text file, but
i'm not sure how to take the information the user inserts...
|
by: Séb |
last post by:
Hi everyone,
I'm relatively new to python and I want to write a piece of code who do
the following work for data mining purpose :
1) I have a list of connexion between some computers. This list has
this format :
Ip A Date Ip B
.... ... ...
|
by: bpontius |
last post by:
The GES Algorithm
A Surprisingly Simple Algorithm for Parallel Pattern Matching
"Partially because the best algorithms presented in the literature
are difficult to understand and to implement, knowledge of fast and
practical algorithms is not commonplace."
Hume and Sunday, "Fast String Searching", Software - Practice
and Experience, Vol. 21 # 11, pp 1221-48
|
by: aevans1108 |
last post by:
expanding this message to microsoft.public.dotnet.xml
Greetings
Please direct me to the right group if this is an inappropriate place
to post this question. Thanks.
I want to format a numeric value according to an arbitrary regular
expression.
|
by: Daniel Santa Cruz |
last post by:
Hello all,
I've been trying to go over my OO Patterns book, and I decided to try
to implement them in Python this time around. I figured this would
help me learn the language better.
Well, I've gotten stuck with my first go at OO patterns with Python. I
guess it goes without say that some of the stuff that are taken for
granted in most of the books (ie. Interfaces, Abstract classes) don't
really apply to Python per say, but the idea...
| |
by: Krivenok Dmitry |
last post by:
Hello All!
I am trying to implement my own Design Patterns Library.
I have read the following documentation about Observer Pattern:
1) Design Patterns by GoF
Classic description of Observer. Also describes implementation
via ChangeManager (Mediator + Singleton)
2) Pattern hatching by John Vlissides
Describes Observer's implementation via Visitor Design Pattern.
3) Design Patterns Explained by Alan Shalloway and James Trott
|
by: ltruett |
last post by:
....and I've finally completed my series of GOF design patterns using
PHP 5 with the Template Pattern.
http://www.fluffycat.com/PHP-Design-Patterns/Template/
This is a pretty useful pattern, and one that you could easily use
without even realizing it is a pattern.
Essentially you have an abstract template class that defines a
non-abstract method with an algorithm. Also in the abstract template
|
by: dhinakar_ve |
last post by:
Hi All,
I am writing a function to generate the strings based on a
pattern. For example A will generate A1, A2 and A3. If the
pattern is A then it will generate the strings A11, A12,
A21, A22, A31, A32. What is the best algorithm to accomplish this?
Thanks for your time.
ananihdv
|
by: krishnai888 |
last post by:
I had already asked this question long back but no one has replied to me..I hope someone replies to me because its very important for me as I am doing my internship.
I am currently writing a code involving lot of matrices. At one point I need to calculate the square root of a matrix e.g. A which contains non-zero off-diagonal elements. I searched for a lot of info on net but no algorithm worked. My best bet for finding square root was to find...
|
by: jinu1996 |
last post by:
In today's digital age, having a compelling online presence is paramount for businesses aiming to thrive in a competitive landscape. At the heart of this digital strategy lies an intricately woven tapestry of website design and digital marketing. It's not merely about having a website; it's about crafting an immersive digital experience that captivates audiences and drives business growth.
The Art of Business Website Design
Your website is...
|
by: tracyyun |
last post by:
Dear forum friends,
With the development of smart home technology, a variety of wireless communication protocols have appeared on the market, such as Zigbee, Z-Wave, Wi-Fi, Bluetooth, etc. Each protocol has its own unique characteristics and advantages, but as a user who is planning to build a smart home system, I am a bit confused by the choice of these technologies. I'm particularly interested in Zigbee because I've heard it does some...
| |
by: agi2029 |
last post by:
Let's talk about the concept of autonomous AI software engineers and no-code agents. These AIs are designed to manage the entire lifecycle of a software development project—planning, coding, testing, and deployment—without human intervention. Imagine an AI that can take a project description, break it down, write the code, debug it, and then launch it, all on its own....
Now, this would greatly impact the work of software developers. The idea...
|
by: conductexam |
last post by:
I have .net C# application in which I am extracting data from word file and save it in database particularly. To store word all data as it is I am converting the whole word file firstly in HTML and then checking html paragraph one by one.
At the time of converting from word file to html my equations which are in the word document file was convert into image.
Globals.ThisAddIn.Application.ActiveDocument.Select();...
|
by: TSSRALBI |
last post by:
Hello
I'm a network technician in training and I need your help.
I am currently learning how to create and manage the different types of VPNs and I have a question about LAN-to-LAN VPNs.
The last exercise I practiced was to create a LAN-to-LAN VPN between two Pfsense firewalls, by using IPSEC protocols.
I succeeded, with both firewalls in the same network. But I'm wondering if it's possible to do the same thing, with 2 Pfsense firewalls...
|
by: adsilva |
last post by:
A Windows Forms form does not have the event Unload, like VB6. What one acts like?
|
by: 6302768590 |
last post by:
Hai team
i want code for transfer the data from one system to another through IP address by using C# our system has to for every 5mins then we have to update the data what the data is updated we have to send another system
|
by: muto222 |
last post by:
How can i add a mobile payment intergratation into php mysql website.
| |
by: bsmnconsultancy |
last post by:
In today's digital era, a well-designed website is crucial for businesses looking to succeed. Whether you're a small business owner or a large corporation in Toronto, having a strong online presence can significantly impact your brand's success. BSMN Consultancy, a leader in Website Development in Toronto offers valuable insights into creating effective websites that not only look great but also perform exceptionally well. In this comprehensive...
| |