A command line tool to dedup lines in file or input stream. In contrast to uniq, dedup does not only compare adjacent lines, but will dedup lines globally.
uniq
dedup
Use --help option to get more information.
--help
dedup --help