473,327 Members | 2,007 Online
Bytes | Software Development & Data Engineering Community
Post Job

Home Posts Topics Members FAQ

Join Bytes to post your question to a community of 473,327 software developers and data experts.

List confusion

Hello,

I am a newbie at python, as will shortly become painfully apparent. I wrote some code that goes through a list of file names that contain names of stars. I put the star names in 'tgtnamelist'. The corresponding file names are in 'imagelist'. 'imagelist' and 'tgtnamelist' have the same length. 'tgtnamelist' does not contain len(tgtnamelist) different star names, so I want to group all the filenames that correspond to a unique star name in one list. Basically I want to have subset lists of filenames and which star they contain.


Now, the code I've written works just fine and does exactly what I want, but it's a part of a much larger code where I haev to go through tgtnamelist and iamgelist repeatedly and a whole lot of nested 'for' and 'if' loops so at the end everything becomes very confusing!! Which is why I want to trim down the code a little, so I figure I'll start by getting rid of some unnecessary lists.

Is there a way to do this? To match two lists, one with entries that repeat themselves in subsets of unknown length, to another list with unique entires. The only thing connecting the lists is that target 0 is in file 0, target 1 is in file 1, etc. Target names are unordered, so the same star name can be spread out throughout the list.

Any help, tips and hints would be greatly appreciated!!


#first target
currtgtname=tgtnamelist[0]
#located in first file
currimname=imagelist[0]
del imagelist[0]
del tgtnamelist[0]
#all files containing one star
subsets=[]
#all subsets of stars
alltargets=[]

for curr in range(len(imagelist)):
subsets.append(currimname)
if tgtnamelist[curr]==currtgtname:
currimname=imagelist[curr]
else:
currtgtname=tgtnamelist[curr]
currimname=imagelist[curr]
alltargets.append(list(subsets))
del subsets[:]
Mar 19 '07 #1
6 1582
bvdet
2,851 Expert Mod 2GB
Hello,

I am a newbie at python, as will shortly become painfully apparent. I wrote some code that goes through a list of file names that contain names of stars. I put the star names in 'tgtnamelist'. The corresponding file names are in 'imagelist'. 'imagelist' and 'tgtnamelist' have the same length. 'tgtnamelist' does not contain len(tgtnamelist) different star names, so I want to group all the filenames that correspond to a unique star name in one list. Basically I want to have subset lists of filenames and which star they contain.


Now, the code I've written works just fine and does exactly what I want, but it's a part of a much larger code where I haev to go through tgtnamelist and iamgelist repeatedly and a whole lot of nested 'for' and 'if' loops so at the end everything becomes very confusing!! Which is why I want to trim down the code a little, so I figure I'll start by getting rid of some unnecessary lists.

Is there a way to do this? To match two lists, one with entries that repeat themselves in subsets of unknown length, to another list with unique entires. The only thing connecting the lists is that target 0 is in file 0, target 1 is in file 1, etc. Target names are unordered, so the same star name can be spread out throughout the list.

Any help, tips and hints would be greatly appreciated!!


#first target
currtgtname=tgtnamelist[0]
#located in first file
currimname=imagelist[0]
del imagelist[0]
del tgtnamelist[0]
#all files containing one star
subsets=[]
#all subsets of stars
alltargets=[]

for curr in range(len(imagelist)):
subsets.append(currimname)
if tgtnamelist[curr]==currtgtname:
currimname=imagelist[curr]
else:
currtgtname=tgtnamelist[curr]
currimname=imagelist[curr]
alltargets.append(list(subsets))
del subsets[:]
We can help you, but let me ask you a couple of things first. Please repost your code with CODE TAGS (see reply guidelines). That would make your code much easier to follow. Also, would you post samples of your lists that you need to process, and a sample file. Dictionaries may be better suited for your project.
Mar 19 '07 #2
Ok, pardon my ignorance, I'll repost the code.

this is the way it looks now:

Expand|Select|Wrap|Line Numbers
  1. #first target
  2. currtgtname=tgtnamelist[0]
  3. #located in first file
  4. currimname=imagelist[0]
  5. del imagelist[0]
  6. del tgtnamelist[0]
  7. #all files containing a certain star
  8. subsets=[]
  9. #all subsets of all stars
  10. alltargets=[]
  11.  
  12. for curr in range(len(imagelist)):
    subsets.append(currimname)
  13. if tgtnamelist[curr]==currtgtname:
    currimname=imagelist[curr]
    else:
    currtgtname=tgtnamelist[curr]
  14. currimname=imagelist[curr]
  15. alltargets.append(list(subsets))
  16. del subsets[:]
The basic problem is this: I have images with header information which contains fields such as target name, etc. I have extracted the values of target names from those fields in tgtnamelist with PyFits. tgtnamelist can look like this:
Expand|Select|Wrap|Line Numbers
  1. tgtnamelist = ['pg2203+051', 'MarkA', 'pg2203+051', 'pg2203+051', 'sa95', 'MarkA', 'MarkA']
imagelist simply contains the names of the corresponding image files, e.g.:

Expand|Select|Wrap|Line Numbers
  1. imagelist = ['image1.fits', 'image2.fits', 'image3.fits', 'image4.fits', 'image5.fits', 'image6.fits', 'image7.fits']

where
image1.fits contains pg2203+051
image2.fits contains MarkA
image3.fits contains pg2203+051
image4.fits contains pg2203+051
image5.fits contains sa95
image6.fits contains MarkA
image7.fits contains MarkA

my code groups the filenames like this:

Expand|Select|Wrap|Line Numbers
  1. subset1=['image1.fits', 'image3.fits', 'image4.fits']
  2. subset2=['image2.fits', 'image6.fits', 'image7.fits']
  3. subset3=['image5.fits']
and then returns alltargets as

Expand|Select|Wrap|Line Numbers
  1. alltargets=[subset1, subset2, subset3]
which is what I want, but combined with the rest of the code there's just too many lists to keep track of and I'm finding it very difficult to continue expanding the code which is why I'm here. I thought of using dictionaries but don't they take keys with unique values only?


anyway, just to show you the list confusion I'm talking about , here the entire procedure. You don't have to look at it in detail at all, the excerpt above works as a stand-alone. But as you can see I loop through the list repeatedly just because when I tried to do everything in one loop I completely lost track of what happens, when and why.





Expand|Select|Wrap|Line Numbers
  1. #####-----------------------############## 
  2. ##   group images into bias,flat   #####
  3. ##   object and std                    #####
  4. #####-----------------------############## 
  5. def groupimages(headers,imagenames,stdnames): 
  6.     exptimelist=[] 
  7.     tgtnamelist=[] 
  8.     for curr in headers:
  9.         if curr.has_key("EXPTIME"): 
  10.             exptime=curr.get("EXPTIME")
  11.             exptimelist.append(exptime)
  12.         if curr.has_key("TCSTGT"): 
  13.             tgtname=curr.get("TCSTGT")
  14.             tgtnamelist.append(tgtname) 
  15.         else: 
  16.             print "FIX TCSTGT keywords..."         
  17.             system.exit(0) 
  18.     biaslist=[] 
  19.     flatlist=[] 
  20.     imagelist=list(imagenames) 
  21.     objlist=[] 
  22.     delindex=[]
  23.  
  24.     for curr in range(len(headers)):
  25.         if exptimelist[curr]==0.0: 
  26.             biaslist.append(imagenames[curr])
  27.             delindex.append(curr)
  28.         elif tgtnamelist[curr].find("Blank")!=-1: 
  29.             flatlist.append(imagenames[curr])
  30.             delindex.append(curr)
  31.     for i in reversed(delindex):
  32.         del imagelistcopy[i]
  33.         del exptimelist[i] 
  34.         del tgtnamelist[i]
  35.     del delindex[:]
  36.  
  37.     #now have only object files remaining
  38.     currtgtname=tgtnamelist[0]
  39.     currimname=imagelistcopy[0]
  40.     del imagelistcopy[0]
  41.     del tgtnamelist[0]
  42.     subsets=[]
  43.     alltargets=[]
  44.  
  45.     for curr in range(len(imagelistcopy)):
  46.         subsets.append(currimname)
  47.         if tgtnamelist[curr]==currtgtname:
  48.             currimname=imagelistcopy[curr]
  49.         else:
  50.             currtgtname=tgtnamelist[curr]
  51.             currimname=imagelistcopy[curr]
  52.             alltargets.append(list(subsets))
  53.             del subsets[:]
  54.     #now all objects are groupped.
  55.     #separate stds from science frames
  56.     stdlist=[]
  57.     delindex=[]
  58.     for i in range(len(alltargets)):
  59.         objimage=alltargets[i][0]
  60.         hdr=PF.getheader(objimage)
  61.         if hdr.get("TCSTGT").upper() in stdnames:
  62.             stdlist.append(alltargets[i])
  63.             delindex.append(i)
  64.             print "found an STD, ", hdr.get("TCSTGT")
  65.     #remove stds from object list         
  66.     for i in reversed(delindex):
  67.         del alltargets[i]
  68.      return biaslist,flatlist,alltargets,stdlist
  69. #pfew!
  70.  
Mar 19 '07 #3
I forgot to mention,

imagelist

does not have to contain filenames which can be ordered in some logical fashion, eg. it doesn't have to be
Expand|Select|Wrap|Line Numbers
  1. imagelist=['image1.fits','image2.fits','image3.fits',....]
  2.  
it could be

Expand|Select|Wrap|Line Numbers
  1. imagelist=['std1.fits','li100043.fits','ONT-12:34:54.fits',....]
Mar 19 '07 #4
bvdet
2,851 Expert Mod 2GB
I forgot to mention,

imagelist

does not have to contain filenames which can be ordered in some logical fashion, eg. it doesn't have to be
Expand|Select|Wrap|Line Numbers
  1. imagelist=['image1.fits','image2.fits','image3.fits',....]
  2.  
it could be

Expand|Select|Wrap|Line Numbers
  1. imagelist=['std1.fits','li100043.fits','ONT-12:34:54.fits',....]
I can see how you are confused. I may not understand what you are trying to accomplish, but I believe a dictionary of dictionaries would be a better choice. The image file names could be your keys for your primary dictionary. Iterate once for each file - each iteration adds a secondary dictionary. Your required lists can be extracted from this dictionary. Here's a simple example:
Expand|Select|Wrap|Line Numbers
  1. imagelist = ['image1.fits', 'image2.fits', 'image3.fits', 'image4.fits', 'image5.fits', 'image6.fits', 'image7.fits']
  2.  
  3. datalist = [['pg2203+051', 'Z', '275'],
  4.             ['MarkA', 'A', '150'],
  5.             ['pg2203+051', 'Z', '275'],
  6.             ['pg2203+051', 'A', '275'],
  7.             ['sa95', 'T', '500'],
  8.             ['MarkA', 'B', '150'],
  9.             ['MarkA', 'C', '150']
  10.             ]
  11.  
  12. #  imagelist and datalist correspond
  13.  
  14. dd = {}
  15. for i, item in enumerate(imagelist):
  16.     dd[item] = dict(zip(['star', 'type', 'dist'], datalist[i]))
  17.  
  18. for key in dd:
  19.     print '%s = %s' % (key, dd[key])
  20.  
  21. '''
  22. >>> image2.fits = {'type': 'A', 'star': 'MarkA', 'dist': '150'}
  23. image6.fits = {'type': 'B', 'star': 'MarkA', 'dist': '150'}
  24. image1.fits = {'type': 'Z', 'star': 'pg2203+051', 'dist': '275'}
  25. image7.fits = {'type': 'C', 'star': 'MarkA', 'dist': '150'}
  26. image5.fits = {'type': 'T', 'star': 'sa95', 'dist': '500'}
  27. image3.fits = {'type': 'Z', 'star': 'pg2203+051', 'dist': '275'}
  28. image4.fits = {'type': 'A', 'star': 'pg2203+051', 'dist': '275'}
  29. '''
  30.  
  31. # Python 2.3 - learned this from ghostdog74
  32. starList = [x['star'] for x in dd.values() if x['star'] not in locals()['_[1]'].__self__]
  33.  
  34. # Python 2.4
  35. # starList = list(set([x['star'] for x in dd.values()]))
  36.  
  37. print starList
  38.  
  39. '''
  40. >>> ['MarkA', 'pg2203+051', 'sa95']
  41. '''
  42.  
  43. # Star, image files
  44. starImageDict = {}
  45.  
  46. for key in dd:
  47.     try:
  48.         starImageDict[dd[key]['star']].append(key)
  49.     except:
  50.         starImageDict[dd[key]['star']] = [key, ]
  51.  
  52. print starImageDict
  53.  
  54. '''
  55. >>> {'sa95': ['image5.fits'], 'MarkA': ['image2.fits', 'image6.fits', 'image7.fits'], 'pg2203+051': ['image1.fits', 'image3.fits', 'image4.fits']}
  56. >>>
  57. '''
I don't know PyFits, but I am assume the attributes for the secondary dictionaries are readily available to create datalist.
Mar 19 '07 #5
I apologize for bumping this thread, I just wanted to thank you!! Thank you very much, this will seriously reduce the amount of code!!
Mar 20 '07 #6
bvdet
2,851 Expert Mod 2GB
I apologize for bumping this thread, I just wanted to thank you!! Thank you very much, this will seriously reduce the amount of code!!
You are welcome. The positive comments are much appreciated. Let us know if we can help again!
Mar 20 '07 #7

Sign in to post your reply or Sign up for a free account.

Similar topics

1
by: Mark Elston | last post by:
I recently stumbled over List Comprehension while reading the Python Cookbook. I have not kept up with the What's New sections in the online docs. :) Anyway, I thought I was following the...
5
by: flupke | last post by:
Hi, i'm having trouble with deleting elements from a list in a for loop ============== test program ============== el = print "**** Start ****" print "List = %s " % el index = 0 for line...
0
by: Boris Ammerlaan | last post by:
This notice is posted about every week. I'll endeavor to use the same subject line so that those of you who have seen it can kill-file the subject; additionally, Supersedes: headers are used to...
2
by: mervyntracy | last post by:
Hi There, I have recently started coding in asp.net (just 2 and a half days now). I am writing a simple test app that gets data from a data base and displays the value perfectly in the drop down...
12
by: kalyan | last post by:
Hi, I am using Linux + SysV Shared memory (sorry, but my question is all about offset + pointers and not about linux/IPC) and hence use offset's instead on pointers to store the linked list in...
0
by: DolphinDB | last post by:
Tired of spending countless mintues downsampling your data? Look no further! In this article, you’ll learn how to efficiently downsample 6.48 billion high-frequency records to 61 million...
0
by: ryjfgjl | last post by:
ExcelToDatabase: batch import excel into database automatically...
0
by: jfyes | last post by:
As a hardware engineer, after seeing that CEIWEI recently released a new tool for Modbus RTU Over TCP/UDP filtering and monitoring, I actively went to its official website to take a look. It turned...
0
by: ArrayDB | last post by:
The error message I've encountered is; ERROR:root:Error generating model response: exception: access violation writing 0x0000000000005140, which seems to be indicative of an access violation...
1
by: CloudSolutions | last post by:
Introduction: For many beginners and individual users, requiring a credit card and email registration may pose a barrier when starting to use cloud servers. However, some cloud server providers now...
1
by: Defcon1945 | last post by:
I'm trying to learn Python using Pycharm but import shutil doesn't work
1
by: Shællîpôpï 09 | last post by:
If u are using a keypad phone, how do u turn on JavaScript, to access features like WhatsApp, Facebook, Instagram....
0
by: af34tf | last post by:
Hi Guys, I have a domain whose name is BytesLimited.com, and I want to sell it. Does anyone know about platforms that allow me to list my domain in auction for free. Thank you
0
isladogs
by: isladogs | last post by:
The next Access Europe User Group meeting will be on Wednesday 3 Apr 2024 starting at 18:00 UK time (6PM UTC+1) and finishing by 19:30 (7.30PM). In this session, we are pleased to welcome former...

By using Bytes.com and it's services, you agree to our Privacy Policy and Terms of Use.

To disable or enable advertisements and analytics tracking please visit the manage ads & tracking page.