[PHP] Removing duplicate in an array… really fast

Let’s imagine you have a huge array, with say couple ten thousands of elements and you want to remove any duplicates entries in this array. As PHP programmer you’ll be thinking of array_unique. For a “small” sized array and one dimensional array this one does the trick.

I have some nice snippets which cover multi dimensional and/or huge sized array.

First: you have a huge, one dimensional array.

array_flip(array_flip(array_reverse($input,true)))

This one works only on one dimensional array. The key of the last element will be preserved and the keys are not sorted

Second: for a huge, multi dimensional array

function super_unique($array) {
  $result = array_map("unserialize", array_unique(array_map("serialize", $array)));
  foreach ($result as $key => $value) {
    if ( is_array($value) ) {
      $result[$key] = super_unique($value);
    }
  }
  return $result;
}

sources:

  1. http://dk.php.net/manual/en/function.array-unique.php#97285
  2. http://www.puremango.co.uk/2010/06/fast-php-array_unique-for-removing-duplicates/

Leave a Reply

Your email address will not be published. Required fields are marked *