Pandas remove duplicates efficiently (using only a subset of columns)

d.drop_duplicates(subset=['text'])