Remove Duplicates From List – Quick and Easy

I needed a quick way to take my several thousand plus list of numbers and remove all duplicates. Just a one off task where I didn’t care about the original order of the list, only that there should be no duplicates. Enter python! Here’s how I did it.

  1. Open Terminal on Mac
  2. Type “python” and execute
  3. Format my numbers in an array likeĀ [39213123667, 532092995671, 659203651894,… in a text editor for easy copy and paste
  4. Use python’s set to remove duplicates and then convert back to a list and print out:
listWithDuplicates = [39213123667, 532092995671, 659203651894,...

# sets are unordered collections of distinct objects
deduplicated = list(set(listWithDuplicates))

# used this to see how many duplicates were removed
len(listWithDuplicates)
len(deduplicated)

# print out list to start doing real work
deduplicated