Skip to content
Related Articles

Related Articles

Python – Ways to remove duplicates from list

Improve Article
Save Article
  • Difficulty Level : Easy
  • Last Updated : 14 Nov, 2022
Improve Article
Save Article

This article focuses on one of the operations of getting a unique list from a list that contains a possible duplicate. Removing duplicates from list operation has a large number of applications and hence, its knowledge is good to have. 

Method 1: Using *set() 

This is the fastest and smallest method to achieve a particular task. It first removes the duplicates and returns a dictionary which has to be converted to list. 

Python3




# Python 3 code to demonstrate
# removing duplicate elements from the list
l = [1, 2, 4, 2, 1, 4, 5]
print("Original List: ", l)
res = [*set(l)]
print("List after removing duplicate elements: ", res)


Output

Original List:  [1, 2, 4, 2, 1, 4, 5]
List after removing duplicate elements:  [1, 2, 4, 5]

Method 2: Using list comprehension 

This method has working similarly to the above method, but this is just a one-liner shorthand of a longer method done with the help of list comprehension.  

Python3




# Python 3 code to demonstrate
# removing duplicated from list
# using list comprehension
 
# initializing list
test_list = [1, 3, 5, 6, 3, 5, 6, 1]
print("The original list is : "
      + str(test_list))
 
# using list comprehension
# to remove duplicated
# from list
res = []
[res.append(x) for x in test_list if x not in res]
 
# printing list after removal
print ("The list after removing duplicates : "
       + str(res))


Output

The original list is : [1, 3, 5, 6, 3, 5, 6, 1]
The list after removing duplicates : [1, 3, 5, 6]

Method 3: Using set()

This is the most popular way by which the duplicates are removed from the list. But the main and notable drawback of this approach is that the ordering of the element is lost in this particular method.  

Python3




# Python 3 code to demonstrate
# removing duplicated from list
# using set()
 
# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("The original list is : "
        + str(test_list))
 
# using set()
# to remove duplicated
# from list
test_list = list(set(test_list))
 
# printing list after removal
# distorted ordering
print ("The list after removing duplicates : "
        + str(test_list))


Output

The original list is : [1, 5, 3, 6, 3, 5, 6, 1]
The list after removing duplicates : [1, 3, 5, 6]

Method 4: Using list comprehension + enumerate() 

list comprehension coupled with enumerate function can also achieve this task. It basically looks for already occurred elements and skips adding them. It preserves the list ordering.  

Python3




# Python 3 code to demonstrate
# removing duplicated from list
# using list comprehension + enumerate()
 
# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("The original list is : "
        + str(test_list))
 
# using list comprehension + enumerate()
# to remove duplicated
# from list
res = [i for n, i in enumerate(test_list) if i not in test_list[:n]]
 
# printing list after removal
print ("The list after removing duplicates : "
        + str(res))


Output

The original list is : [1, 5, 3, 6, 3, 5, 6, 1]
The list after removing duplicates : [1, 5, 3, 6]

Method 5: Using collections.OrderedDict.fromkeys()

This is the fastest method to achieve a particular task. It first removes the duplicates and returns a dictionary which has to be converted to a list. This works well in the case of strings also. 

Python3




# Python 3 code to demonstrate
# removing duplicated from list
# using collections.OrderedDict.fromkeys()
from collections import OrderedDict
 
# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print ("The original list is : "
       + str(test_list))
 
# using collections.OrderedDict.fromkeys()
# to remove duplicated
# from list
res = list(OrderedDict.fromkeys(test_list))
 
# printing list after removal
print ("The list after removing duplicates : "
       + str(res))


Output

The original list is : [1, 5, 3, 6, 3, 5, 6, 1]
The list after removing duplicates : [1, 5, 3, 6]

Method 6: Using in, not in operators

Python3




# Python 3 code to demonstrate
# removing duplicates from list
 
# initializing list
test_list = [1, 5, 3, 6, 3, 5, 6, 1]
print("The original list is : " + str(test_list))
 
res = []
for i in test_list:
    if i not in res:
        res.append(i)
 
# printing list after removal
print("The list after removing duplicates : " + str(res))


Output

The original list is : [1, 5, 3, 6, 3, 5, 6, 1]
The list after removing duplicates : [1, 5, 3, 6]

Method 7: Using list comprehension and Array.index() method. 

In this method, we use list comprehension to iterate over the list and array indexing to get the item from array. We add the items to array only if first index of element in array match with current index of element else neglects the element.

Python




# Python 3 code to demonstrate
# removing duplicated from list
# using list comprehension and arr.index
 
# initializing list
arr = [1, 5, 3, 6, 3, 5, 6, 1]
print ('The original list is : '+ str(arr))
 
# using list comprehension + arr.index()
# to remove duplicated
# from list
res = [arr[i] for i in range(len(arr)) if i == arr.index(arr[i]) ]
 
# printing list after removal
# of duplicate
print('The list after removing duplicates :'
        ,res)


Output

The original list is : [1, 5, 3, 6, 3, 5, 6, 1]
('The list after removing duplicates :', [1, 5, 3, 6])

Method 8: Using Counter() method.

In this method, we use Counter() method to make a dictionary from a given array. Now retrieve all the keys using keys() method which gives only unique values from the previous list. 

Python3




from collections import Counter
# Python 3 code to demonstrate
# removing duplicated from list
# using Counter() method
 
# initializing list
arr = [1, 5, 3, 6, 3, 5, 6, 1]
print ('The original list is : '+ str(arr))
 
# using Counter() + keys()
# to remove duplicated
# from list
temp = Counter(arr)
res = [*temp]
 
# printing list after removal
# of duplicate
print('The list after removing duplicates :'
        ,res)


Output

The original list is : [1, 5, 3, 6, 3, 5, 6, 1]
The list after removing duplicates : [1, 5, 3, 6]

My Personal Notes arrow_drop_up
Related Articles

Start Your Coding Journey Now!