How to de-duplicate python lists

1 set() method¶

The processing is relatively simple, and the set method is used for processing, but the result will not retain the previous order.

lis = [1, 2, 2, 3, 7, 7, 9, 9, 10, 10]
lis = set(lis)
print(lis)

output:

{1, 2, 3, 7, 9, 10}

2 keys() method¶

As a result, the original order cannot be maintained

li=[1,2,3,4,5,1,2,3]
li_new = {}.fromkeys(li).keys()
print(li_new)

output:

dict_keys([1, 2, 3, 4, 5])

3 itertools module method¶

import itertools
list = [1,4,3,3,4,2,3,4,5,6,1]
list.sort()
it = itertools.groupby(list)
for k, g in it:
    print (k)

output:

1
2
3
4
5
6

4 set + index (index) method¶

Ensure that the order after deduplication remains unchanged

li=[1,2,3,4,5,1,2,3]
new_li=list(set(li))
new_li.sort(key=li.index)
print(new_li)

output:

[1, 2, 3, 4, 5]