I have not yet been able to find any correct example of setting Java memory settings when running in Docker containers. There are numerous examples, yet they are all wrong.
Wrong Example 1: Setting Nothing
By default, Java doesn’t look at the cgroups restrictions that Docker uses for resource isolation. So, if max heap is unset, Java will default to 1/4 the RAM on the host machine, regardless of what the memory restrictions of the Docker container are. So, running on a host with 8Gb RAM, and a container with “-m 200m”, Java will default max heap to 25% of 8Gb, or 2Gb. The container will likely start up fine, and run for some period of time, but Java will at some point try to expand past that 200 megabyte limit (because it thinks it has 2 gigs of heap space to use), and Docker will kill the container.
Wrong Example 2: Java Knows Better Now
Recently (Dec. 2016) added to JDK8 and JDK9 was a setting that told Java to look at the cgroup restrictions when setting the max heap, rather than the physical RAM on the host. Now, by setting the dual properties
-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap
Java will correctly know it is running in an environment that has extra memory limitations. The documentation for this feature says that Java will look at the limit defined by the cgroup settings, and set max heap to that limit. In my experimentation (with the Oracle JDK), Java will set max heap to 50% of that limit. So, using the same example as above, Java will default max heap to 50% of 200mb, or 100mb. With this setup, Java will likely run for a good long time without issue. However, this does not take MetaSpace into consideration, and this can still get you into trouble.
Wrong Example 3: MaxMetaSpaceSize
Even if you allow Java to “correctly” default the max heap size, or if you set a good value yourself, your container still may get OOM killed by Docker. MetaSpace is effectively unlimited by default. So, if you are running a couple of Java agents in your application (such as APM monitoring tools, or security scanning tools), your application could easily eat up 70-100mb of MetaSpace. In the above scenario, with a 200mb container limit, and a 100mb max heap, your container could get an OOM death if it tries to use up 100mb of MetaSpace (because your container OS still needs some resources).
Can’t Do Anything Right
Yet even if you set all the proper max sizes, OOM deaths could still plague your system. It turns out Java doesn’t just deal with memory in terms of Heap and MetaSpace (or PermGen if you’re still old-school). Java also has a code cache (default max looks like 240mb), and a compressed class space (default max looks like 1Gb). On top of those mysterious memory-taker-uppers, Java also now can use some amount of direct memory (basically meaning memory the JVM doesn’t manage directly). I haven’t been able to find any way to limit how much direct memory Java is allowed to use. Even if such a thing did exist, we are talking about settings so low-level in the JVM that I am very reluctant to make any changes to them (what happens when compressed class space is set too small?).
These are just some of the reasons why I don’t really understand why it looks like people all over the place are using Java in microservices running on Docker, but when I dig in to try and really find out how it is working, and why I can’t stand up a Java based microservice in a 60mb container, I get worried that nobody really knows what they are doing.
Originally Posted on my Blogger site July of 2017