You can find a list of our scientific publications here.
The current version 3.9.1 of ARX was released on 27/11/2022.
Please report bugs and feature requests on our GitHub page.
Follow us on Twitter!
Home › Development › Benchmark

Benchmark

This page provides background information and data for the benchmark presented in our paper A Benchmark of Globally-Optimal Anonymization Methods for Biomedical Data at the 27th IEEE International Symposium on Computer-Based Medical Systems (CBMS 2014).

Privacy criteria

All 11 reasonable combinations of the following privacy criteria are evaluated in our benchmark:

  1. 5-anonymity
  2. recursive-(4,3)-diversity
  3. 0.2-closeness (EMD with hierarchical distance)
  4. (0.05, 0.15)-presence

Datasets

Below is a list of links for downloading ARX project-files for each dataset. Each project-file contains the following data:

  • Dataset
  • Generalization hierarchies
  • Research subsets
  • Specification of attribute types

An ARX project-file is a compressed folder that can either be opened with the ARX GUI or with common file managers. Within the folder, the data is available in CSV format and the configuration is available as a human readable XML file.

As the licenses of the benchmarking datasets do not permit distribution, the downloads are password protected. Prior to downloading, please request the password.

Implementation

The implementation of our benchmark, including all algorithms, is available on Github. The benchmarking environment is based upon ARX and SUBFRAME. Currently, the following globally-optimal anonymization algorithms are implemented:

  1. Depth-First-Search: the implementation can be found here.
  2. Breadth-First-Search: the implementation can be found here.
  3. Incognito: the implementation can be found here.
  4. Optimal Lattice Anonymization: implementation details are presented in this paper and the implementation can be found here.
  5. Flash: the implementation can be found here.

Results

The following figures show key parameters averaged over either the datasets or the privacy criteria. The number of checks gives an indication of an algorithm’s pruning power, the number of roll-ups gives an indication of an algorithm’s optimizability and, finally, the execution times give an indication of an algorithm’s overall performance within the ARX runtime environment.

On a Desktop PC with a quad-core 3.1 GHz Intel Core i5 CPU running a 64-bit Linux 3.0.14 kernel and a 64-bit Sun JVM (1.7.0 21) the following results are produced (java -Xmx4G -XX:+UseConcMarkSweepGC -jar anonbench-0.2.jar):

Geometric mean of key parameters over all five benchmark datasets:

Image

Image

Image

Geometric mean of key parameters over all eleven combinations of privacy criteria:

Image

Image

Image

Changelog

Since the publication of the paper, we have updated anonbench 0.1, which was based on ARX 2.0.0, to anonbench 0.2, which is based on ARX 2.3.0. Due to bugfixes and various performance-related changes in ARX the results of the benchmark have changed slightly in this process. We note, however, that all conclusions drawn from our original results are still valid and strongly recommend using the latest version of anonbench.