KSM Preload

Introduction

Get it

Download the stable version as tarball .

Check it on Github.

Read the Changelog for the stable version.

Enables legacy applications to leverage Linux’s memory deduplication. Only works on Linux ≥ 2.6.32.

If you have multiple processes generating similar data (like multiple applications serving different vhosts), this easy to use tool may allow you to spare memory by deduplicating it. If two pages (that is on, most systems, consecutive blocks of 4k) are identical, they will be merged into a single page to reduce memory usage.

Explanation

Linux ≥ 2.6.32 features a memory-saving mechanism that works by deduplicating areas of memory that are identical in different processes (even if they were generated at runtime and after the fork() of their common ancestors).

This mechanism requires the application to opt-in using the madvise() syscall. KSM Preload enables legacy applications (about any current application) to leverage this system by calling madvise(…, MADV_MERGEABLE) on every heap-allocated pages.

In contrast to similar hacks, special care has been taken to make it safe to use, including with other preloaded libraries.

Numbers

I did a very few experiments to evaluate the efficiency of the method. They’re not worth much, but still…

Those data where gathered by running 50 instances of Apache on a single machine, each serving a vhost (for a mutualised hosting provider).
Each instance of Apache served different files and forums, but used the same versions of the softwares.
  • Deduplicated: Pages which were effectively deduplicated, sparing memory.
  • Unique: Pages that could not be deduplicated because there were no other copies of them on the system. The system lost CPU time trying to deduplicate them.
  • Volatile: Pages that could not be deduplicated because they’re changing to often. The system lost CPU time trying to deduplicate them.
Situation Deduplicated Uniques Volatiles Spared memory
No load 1351 2638 10500 ~5MB
Static pages under Siege 15289 3239 1116 ~60MB
PhpBB under siege 25534 36687 775 ~100MB

Instructions

To run your programs using ksm_preload, first download the stable version. then compile it with “make”, and use the provided ksm-wrapper :

make
sudo ./ksm-wrapper echo "I am being run via ksm-preload"

In order to see if it worked, you can run the following snipet :

cd /sys/kernel/mm/ksm
watch 'for f in *; do echo -n $f: ; cat $f; done'

The interpretation is the same as for the “Numbers” section above.