Removing the duplicate data from List,Tuples and Dictionary

[psbasha]

Hi,

How to remove the duplicate data from List,Tuples and Dictionary?.

Thanks in advance
PSB

[psbasha]

Hi ,

For example :

list1 = [ [1,0,0],[0,1,0],[1,1,0],[0,0,0],[1,1,0]]

I dont want to have the duplicate values in the list

the output has to be as mentioned below

list1 = [ [1,0,0],[0,1,0],[1,1,0],[0,0,0]]

Thanks & Regards
PSB

[bvdet]

Hi ,

For example :

list1 = [ [1,0,0],[0,1,0],[1,1,0],[0,0,0],[1,1,0]]

I dont want to have the duplicate values in the list

the output has to be as mentioned below

list1 = [ [1,0,0],[0,1,0],[1,1,0],[0,0,0]]

Thanks & Regards
PSB

Since the items in the list are unhashable, it's a bit more complicated:

>>> list1
[[1, 0, 0], [0, 1, 0], [1, 1, 0], [0, 0, 0], [1, 1, 0]]
>>> set([x for x in list1])
 
Traceback (most recent call last):
  File "<pyshell#12>", line 1, in -toplevel-
    set([x for x in list1])
TypeError: list objects are unhashable
>>> set([str(x) for x in list1])
set(['[1, 0, 0]', '[1, 1, 0]', '[0, 1, 0]', '[0, 0, 0]'])
>>> list(eval(x) for x in set([str(x) for x in list1]))
[[1, 0, 0], [1, 1, 0], [0, 1, 0], [0, 0, 0]]
>>> 

[bartonc]

Since the items in the list are unhashable, it's a bit more complicated:

>>> list1
[[1, 0, 0], [0, 1, 0], [1, 1, 0], [0, 0, 0], [1, 1, 0]]
>>> set([x for x in list1])
 
Traceback (most recent call last):
  File "<pyshell#12>", line 1, in -toplevel-
    set([x for x in list1])
TypeError: list objects are unhashable
>>> set([str(x) for x in list1])
set(['[1, 0, 0]', '[1, 1, 0]', '[0, 1, 0]', '[0, 0, 0]'])
>>> list(eval(x) for x in set([str(x) for x in list1]))
[[1, 0, 0], [1, 1, 0], [0, 1, 0], [0, 0, 0]]
>>> 

Pretty darn clever, there, BV.

[ghostdog74]

another way

>>> list1
[[1, 0, 0], [0, 1, 0], [1, 1, 0], [0, 0, 0], [1, 1, 0]]
>>> [ item for item in list1 if item not in locals()['_[1]'] ]
[[1, 0, 0], [0, 1, 0], [1, 1, 0], [0, 0, 0]]
 

[bartonc]

another way

>>> list1
[[1, 0, 0], [0, 1, 0], [1, 1, 0], [0, 0, 0], [1, 1, 0]]
>>> [ item for item in list1 if item not in locals()['_[1]'] ]
[[1, 0, 0], [0, 1, 0], [1, 1, 0], [0, 0, 0]]
 

You'll have to explain that one:
>>> list1 = [[1, 0, 0], [0, 1, 0], [1, 1, 0], [0, 0, 0], [1, 1, 0]]
>>> locals()['_[1]']
File "<console>", line 1, in ?
''' exceptions.KeyError : '_[1]' '''
>>>

[bvdet]

You'll have to explain that one:
>>> list1 = [[1, 0, 0], [0, 1, 0], [1, 1, 0], [0, 0, 0], [1, 1, 0]]
>>> locals()['_[1]']
File "<console>", line 1, in ?
''' exceptions.KeyError : '_[1]' '''
>>>

I looked it up. From Python 2.3 docs:

In Python 2.3, a list comprehension "leaks" the control variables of each "for" it contains into the containing scope. However, this behavior is deprecated, and relying on it will not work once this bug is fixed in a future release.

'_[1]' is a temporary name used while the list is being constructed. Additional names '_[2]', '_[3]' are used for nested comprehensions. Totally undocumented, but real cool!

[ghostdog74]

You'll have to explain that one:
>>> list1 = [[1, 0, 0], [0, 1, 0], [1, 1, 0], [0, 0, 0], [1, 1, 0]]
>>> locals()['_[1]']
File "<console>", line 1, in ?
''' exceptions.KeyError : '_[1]' '''
>>>

locals()['_[1]'] is used in list comprehension to denote a "temporary" name, as bv has pointed out in the docs.

[bartonc]

locals()['_[1]'] is used in list comprehension to denote a "temporary" name, as bv has pointed out in the docs.

Very, very cool. You guys make me feel inadequate.

[ilikepython]

Would this work:

count = -1
list1 = [[0,1,1], [0,1,1], [1,0,1], [1,0,0], [0,0,0]]
list2 = list1[:]
for item in list1:
    count+= 1
    if item in list2:
        del list1[count]
 

[bvdet]

Would this work:

count = -1
list1 = [[0,1,1], [0,1,1], [1,0,1], [1,0,0], [0,0,0]]
list2 = list1[:]
for item in list1:
    count+= 1
    if item in list2:
        del list1[count]
 

No. When you delete an item from the object you are iterating on, it gets messed up. This works basically the same as the list comprehension above:

>>> list1 = [[0,1,1], [0,1,1], [1,0,1], [1,0,0], [0,0,0], [0,1,0], [1,0,0], [0,0,1], [1,0,1]]
>>> list2 = []
>>> for item in list1:
...     if item not in list2:
...         list2.append(item)
...         
>>> list1
[[0, 1, 1], [0, 1, 1], [1, 0, 1], [1, 0, 0], [0, 0, 0], [0, 1, 0], [1, 0, 0], [0, 0, 1], [1, 0, 1]]
>>> list2
[[0, 1, 1], [1, 0, 1], [1, 0, 0], [0, 0, 0], [0, 1, 0], [0, 0, 1]]
>>> 

[ilikepython]

No. When you delete an item from the object you are iterating on, it gets messed up. This works basically the same as the list comprehension above:

>>> list1 = [[0,1,1], [0,1,1], [1,0,1], [1,0,0], [0,0,0], [0,1,0], [1,0,0], [0,0,1], [1,0,1]]
>>> list2 = []
>>> for item in list1:
...     if item not in list2:
...         list2.append(item)
...         
>>> list1
[[0, 1, 1], [0, 1, 1], [1, 0, 1], [1, 0, 0], [0, 0, 0], [0, 1, 0], [1, 0, 0], [0, 0, 1], [1, 0, 1]]
>>> list2
[[0, 1, 1], [1, 0, 1], [1, 0, 0], [0, 0, 0], [0, 1, 0], [0, 0, 1]]
>>> 

Oh ok, thank you

[pes456]

@ilikepython
hey i managed to do this for dictionary:

d = {1:'a', 2:'a', 'k':1, 'e':1, 'p':'z',3:'a',4:'a',5:'a',6:'a',7:'a',8:'a',9:'a',10:'b'}
 
values = {}
for key in d:
    if d[key] in values:
        values[d[key]] += 1
    else:
        values[d[key]] = 1
 
dups = [key for key in values if values[key]>1]
if len(dups):
    print "Duplicate values:"
    for dup in dups:
        print dup

[Dev Player]

If original order of your data matters the above solutions are the way to go. BUT, some assumptions are made in the answers to your question.

You say "remove duplicate data". It's assumed that you mean to remove the -extra- duplicate items from your data, but to also retain only one copy of that item, as opposed to removing all items that are duplicated including the first occurance.

It also assumes that the duplciates to remove are after the first occurance of an item, but leave only the first one.

This therefore presumes order is important. It wasn't clear that you wanted to remove "left" hand duplicates or "right" hand duplicates.

There are times when you want to remove the "old" duplicate data and have the new data that's just appended (to the proverbial end) to be kept.

It's also presumed the original type() of the data is important; ie a list remain a list.

But because you are removing duplicates at arbitrary locations in your data (as you don't know when the duplicates are going to occur) the -need- for order becomes a little more vague.

Additionally you mention removing items with duplicate values in a dict(). Because the order of dict items is not (historically) gaurenteed, this also raises the question for the need for "ordered" items; as removing duplicate-valued items in dicts are considered arbitray; which key to remove?

The question on removing dups for dicts is: Which key is more important to keep then the other keys that have the same values? You can't say the left most or right most, as they are not reliably ordered.

So if order doesn't matter:

list1 = [[0,1,1], [0,1,1], [1,0,1], [1,0,0], [0,0,0]] 
list2 =[list(y) for y in set([tuple(x) for x in list1])]

If the original order is not important but you want it ordered,
use the list3 = sorted(list2) or list2.sort() builtin function.

DICTs:
You don't have to use "for" loops or list comprehensions,
you can use dict comprehensions; they exsist.

dict1 = {1:'a', 2:'a', 'k':1, 'e':1, 'p':'z',3:'a',4:'a',5:'a',6:'a',7:'a',8:'a',9:'a',10:'b'}
nondups = {k:v for k,v in dict1.items() if dict1.values().count(v)==1}
print nondups
{'p': 'z', 10: 'b'}
 
ddups = {k:v for k,v in dict1.items() if dict1.values().count(v)>1}
print ddups
{1: 'a', 2: 'a', 3: 'a', 4: 'a', 5: 'a', 6: 'a', 7: 'a', 8: 'a', 9: 'a', 'k': 1, 'e': 1}

Remember the print out of a dict uses the dict.__str__() method which orders the printout nicely. BUT items retrieved by iterating over a dict may not be the same order. Test your real data to see.

for k in dict1: print k

I don't believe there are such things as tuple comprehensions as a tuple is like a string, nonmutable. Using the format:

(x for x in seq) 

returns a generator, not a tuple.

If the original order for lists matter: I've seen the data get transformed into something with count and or order data inserted per item.

For the dict1 above you might see an interm data format like this:

dict2 = {(1,2,3,4,5,6,7,8,9):'a', (10,):'b', ('e','k'):1, ('p',):'z')}

The above will work because the original keys are already hashable and now the keys are in the hashable tuple sequence. But with this approach you would have to delete the item from the dict and readd it with a new key every time a new duplicate is found.

or a more dynamic version where new duplicates can easily be added and accounted for. You make a new dict where you switch the data with the key and the key with the data if the data is hashable.

dict2 = {'a':[1,2,3,4,5,6,7,8,9], 'b':[10,], 1:['e','k'], 'z':['p',]}

There are other fancier methods, like elementtrees and linked lists. for dealing with duplicate data; which often depends on what your data is, and why you are you even getting duplicate data.

Perhaps some of these methods will illuminate.

Here the values are lists, not tuples, as you can just append new duplicates to the values list. However this approach requires the data can be a key in a dict, which often it can not.