Setting ElasticSearch Xmx
A common question about ElasticSearch (ES) setup is : “How much memory should be allocated to ElasticSearch Java Virtual Machine ?”.
- Short answer: just launch this script.
- Long story:
You should never allocate all the available server memory to ES !
In most cases, ES Xmx, maximum Java Virtual Machine (JVM) heap size, should be set to half the memory of the server.
Why half ? Because ES uses the other half to cache several files. Those files are all the Lucene data structures that are stored on disk. In fact it takes advantage of Linux’s automatic tendency to cache files, that often need to be read or written, in memory. These files are read when ES computes requests and written when it indexes documents.
And now you’re wondering: “Ok, the explanation is quite simple. Why would I need a script to do that ?”.
The answer lies in the “in most cases” part of my first piece of advice.
There are two cases where you want to allocate even less memory to ES.
First, the server runs other processes that consume a significant memory amount. In that case, only half of the available memory should be allocated to ES (server memory minus other processes memory).
Second, the server has 64GB memory or more, and here comes the tricky part. More is not always better. Giving more than ~32GB memory to a JVM is counterproductive because of a Java tweak. This tweak, called compressed Ordinary Object Pointers (OOPs), allows the JVM to use 32-bit memory pointers even on 64-bit operating systems when under ~32GB of memory. This means shorter pointers and better overall performance. This point is better covered in ES documentation.
In fact, it takes up to around 40–50 GB of allocated heap before you have the same effective memory of a heap just under 32 GB using compressed oops.
- ES documentation
To check if the tweak is enabled, you can run the following command :
java -Xmx32766m -XX:+PrintFlagsFinal 2> /dev/null | grep UseCompressedOops
bool UseCompressedOops := true <- it is enabled
Unluckily for us the shift point for this flag activation is not a constant. It depends on your installation.
The script cuts the available memory in half. It checks if the tweak is still enabled for that Xmx. If not, a simple recursive loop outputs the max memory value before disabling the tweak.
This is not much but, hopefully, it can save you some time. :)
As always, there is place for improvement, please file issues on the github repository.