How to de-duplicate python lists

created at 08-05-2021 views: 2

1 set() method

The processing is relatively simple, and the set method is used for processing, but the result will not retain the previous order.


lis = [1, 2, 2, 3, 7, 7, 9, 9, 10, 10]
lis = set(lis)
print(lis)

output:

{1, 2, 3, 7, 9, 10}

2 keys() method

As a result, the original order cannot be maintained

li=[1,2,3,4,5,1,2,3]
li_new = {}.fromkeys(li).keys()
print(li_new)

output:

dict_keys([1, 2, 3, 4, 5])

3 itertools module method

import itertools
list = [1,4,3,3,4,2,3,4,5,6,1]
list.sort()
it = itertools.groupby(list)
for k, g in it:
    print (k)

output:

1
2
3
4
5
6

4 set + index (index) method

Ensure that the order after deduplication remains unchanged

li=[1,2,3,4,5,1,2,3]
new_li=list(set(li))
new_li.sort(key=li.index)
print(new_li)

output:

[1, 2, 3, 4, 5]
created at:08-05-2021
edited at: 08-05-2021: