Remove Duplicates From List – Quick and Easy
I needed a quick way to take my several thousand plus list of numbers and remove all duplicates. Just a one off task where I didn’t care about the original order of the list, only that there should be no duplicates. Enter python! Here’s how I did it.
- Open Terminal on Mac
- Type “python” and execute
- Format my numbers in an array like [39213123667, 532092995671, 659203651894,… in a text editor for easy copy and paste
- Use python’s set to remove duplicates and then convert back to a list and print out:
listWithDuplicates = [39213123667, 532092995671, 659203651894,... # sets are unordered collections of distinct objects deduplicated = list(set(listWithDuplicates)) # used this to see how many duplicates were removed len(listWithDuplicates) len(deduplicated) # print out list to start doing real work deduplicated