“Pile of Laptop & Desktop RAM Memory Modules” by Wilbysuffolk, CC BY-SA 4.0 A Java pod that restarts with `OOMKilled` is a frustrating problem, because the JVM often leaves no stack trace and no `OutOfMemoryError`. The kernel simply ends the process. This guide is for engineers who run Java services on Kubernetes and need to answer one question: **which memory pool grew**? We will use Native Memory Tracking (NMT) and `jcmd` to find out, then map the result to a fix. ## Why Java OOMKilled is not a Java OutOfMemoryError Two different things get confused. A Java `OutOfMemoryError` is thrown by the JVM when it cannot satisfy an allocation inside its own limits, for example when the heap is full. `OOMKilled` means the container as a whole used more memory than its Kubernetes limit. The Kubernetes documentation explains that memory limits are enforced by the kernel with out-of-memory kills, and that this enforcement is reactive: a container can briefly exceed the limit, and it is killed when the kernel detects memory pressure. The container’s memory is more than the Java heap. It also includes class metadata, thread stacks, JIT-compiled code, GC data structures, direct buffers and memory from native libraries. A pod can be killed while the heap graph looks healthy. That is why heap dumps alone often do not explain it. ## Confirm that it really was an OOM kill Start by checking the termination reason. The Kubernetes memory tutorial shows that a killed container reports `reason: OOMKilled` and `exitCode: 137`, which is 128 + 9 (SIGKILL). kubectl get pod my-app-7d9f -o jsonpath='{.status.containerStatuses[0].lastState.terminated}' kubectl describe pod my-app-7d9f | grep -A6 "Last State" Replace `my-app-7d9f` with your pod name. If the reason is something else (for example an error exit from your own code), this guide does not apply. ## Do the budget arithmetic first Before measuring, add up what the JVM is allowed to use. The java command reference gives these facts: * `-XX:MaxRAMPercentage` defaults to 25 percent, and the RAM it is a percentage of is limited by container constraints. * The default maximum code cache is 240 MB (48 MB without tiered compilation). * The default thread stack size on Linux/x64 is 1024 KB. * Metaspace is not limited by default (`-XX:MaxMetaspaceSize`), and direct buffer size is chosen automatically unless you set `-XX:MaxDirectMemorySize`. Take a hypothetical pod with a 1024Mi limit whose JVM runs with `-XX:MaxRAMPercentage=75`, so the maximum heap is 768Mi. If the service has 200 threads on Linux/x64, their stacks can reserve up to 200 x 1 MB = 200 MB. Only the stack pages a thread actually touches count toward the container, so real use is usually lower, but in the worst case heap plus stacks is 968 MB. That leaves 56 MB of the 1024Mi limit for metaspace, code cache, GC structures, direct buffers and everything else. A budget this tight can explain a kill without any leak at all. The JVM Container Memory Calculator does this addition for you with your own heap, metaspace, thread and direct memory inputs. ## Enable Native Memory Tracking NMT is off by default. According to the java command reference, `-XX:NativeMemoryTracking` accepts `off`, `summary` and `detail`. The Java troubleshooting guide states that enabling NMT costs a 5-10 percent performance drop, so turn it on in a test or canary environment first. It cannot be switched on in a running JVM; it needs a restart. env: - name: JAVA_TOOL_OPTIONS value: "-XX:NativeMemoryTracking=summary" This sets the flag through `JAVA_TOOL_OPTIONS`, which the JVM reads at startup, so you do not have to edit the image. Use `detail` only when `summary` points to a category and you need call sites. ## Read the NMT summary with jcmd Run `jcmd` inside the container. In many images the Java process is PID 1. `jcmd` ships with the JDK, so JRE-only or distroless images may not include it; in that case use an image variant that has the JDK tools, or an ephemeral debug container with a JDK that shares the pod’s process namespace. Run `jcmd` as the same user as the Java process, or it cannot attach. kubectl exec my-app-7d9f -- jcmd 1 VM.native_memory The output lists the total reserved and committed memory, then one block per category. The categories documented by Oracle include Java Heap, Class, Thread, Code, GC, Compiler, Internal, Symbol and Native Memory Tracking itself. The page notes that the list may change between releases. Focus on **committed** values, since committed memory is what counts against the container. One limit matters here: NMT does not track allocations made by non-JVM native code, such as a JNI library. If the NMT total is far below what the container reports, suspect native libraries rather than the JVM. ### Compare against a baseline A single snapshot rarely shows growth. Take a baseline after warm-up, wait while the problem develops, then diff. kubectl exec my-app-7d9f -- jcmd 1 VM.native_memory baseline # ...wait until memory has grown... kubectl exec my-app-7d9f -- jcmd 1 VM.native_memory summary.diff The diff prints each category with a plus or minus delta. The category with the largest positive delta is your suspect. If you need allocation sites, restart with `-XX:NativeMemoryTracking=detail` and use `detail.diff` the same way. ## Match the growing pool to a fix Growing category | Typical cause | What to try ---|---|--- Java Heap | Heap max is too large for the limit, or real object growth | Lower `-XX:MaxRAMPercentage`, or take a heap dump with `-XX:+HeapDumpOnOutOfMemoryError` (disabled by default) and `-XX:HeapDumpPath` Thread | Too many threads, each with a stack | Cap thread pools; reduce `-Xss` only if the code allows it Class | Metaspace growth from class loading, for example repeated redeploys or dynamic proxies | Find the class loader leak; set `-XX:MaxMetaspaceSize` to fail early with a clear error Code | Code cache filling toward its maximum | Check `-XX:ReservedCodeCacheSize` against your budget GC | Collector structures that scale with heap size | Reduce heap, or leave more headroom Nothing in NMT, but container grows | Direct buffers or native code outside NMT | Set `-XX:MaxDirectMemorySize`; review JNI or native libraries Setting `-XX:MaxMetaspaceSize` or `-XX:MaxDirectMemorySize` does not remove a leak, but it turns a silent kernel kill into a Java `OutOfMemoryError` with a message, which is much easier to diagnose. ## Set limits that leave headroom Once you know the pools, size the container with a margin. A reasonable pattern is to set the heap as a percentage and leave the rest explicitly for non-heap memory: resources: requests: memory: "1536Mi" limits: memory: "1536Mi" env: - name: JAVA_TOOL_OPTIONS value: "-XX:MaxRAMPercentage=60 -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=128m" Line by line: the request equals the limit so the scheduler reserves what the pod can use; `MaxRAMPercentage=60` gives a heap cap of about 921Mi (0.60 x 1536Mi); the metaspace and direct memory caps are bounded explicitly. These values are an illustration, not a recommendation. Replace them with the committed sizes you measured with NMT. Note that `Mi` is a binary unit; `1536M` would be a different, smaller amount. ## Key takeaways * Java `OOMKilled` means the whole container exceeded its limit; it is not the same as a Java `OutOfMemoryError`. * Confirm with `reason: OOMKilled` and exit code 137 before investigating. * Add up heap, stacks, code cache, metaspace and direct memory first; many kills are budget mistakes. * Enable `-XX:NativeMemoryTracking=summary` (5-10 percent overhead, restart needed), take a `baseline`, then run `summary.diff`. * NMT does not see memory allocated by non-JVM native code. * Cap metaspace and direct memory so failures are explicit and diagnosable. ## Conclusion Treat a Java `OOMKilled` as an accounting problem. Work out the budget, measure which pool grows with NMT diffs, and then either fix the growth or give the container honest headroom. Repeat the diff after each change to confirm that the growing category is the one you fixed. ## Related reading * JVM Container Memory Calculator * OpsHubs ops calculators and tools _Originally published at OpsHubs, where you can also find free ops calculators. This article was drafted with AI assistance and reviewed before publishing._

Posted by Manikumar on Oct 28 Severity: important Affected versions: - Apache Kafka 3.3.0 through 4.1.2 - Apache Kafka 4.2.0 through 4.2.1 - Apache Kafka 4.3.0 through 4.3.1 Description: Apache Kafka's code-generated protocol readers pre-size an array's collection from the element count declared by the sender. That count is checked only against the bytes remaining in the request buffer, which assumes about one byte per element; because most elements need several...

ProtobufParser parses length-prefixed protobuf string and bytes fields by checking (_inputPtr + len)

ProtobufParser._readStringKey() reads the length prefix of a protobuf map entry's string key into an attacker-controlled int and then checks (_inputPtr + len)

Posted by Manikumar on Oct 28 Severity: important Affected versions: - Apache Kafka 3.3.0 through 4.1.2 - Apache Kafka 4.2.0 through 4.2.1 - Apache Kafka 4.3.0 through 4.3.1 Description: Apache Kafka's code-generated protocol readers pre-size an array's collection from the element count declared by the sender. That count is checked only against the bytes remaining in the request buffer, which assumes about one byte per element; because most elements need several...

The asynchronous Smile parser (NonBlockingByteArrayParser) uses a raw-binary length prefix taken from untrusted input to allocate a byte[] before the body is read, with no upper bound and no overflow check. On the raw-binary marker 0xFD, _decodeVInt() computes (value >> 25) …

Posted by Emmanuel Lécharny on Oct 02 Severity: critical Affected versions: - Apache Directory LDAP API 1.2.0 before 1.2.9 Description: Memory allocation with excessive size value vulnerability in Apache Directory LDAP API. A malicious peer (or a MITM) can send a small BER-encoded response causing a large memory allocation before any data is received. This can lead to an OutOfMemoryError and denial of service. The client JVM OOMs (OutOfMemoryError bypasses the...

Running into OutOfMemoryError or No space left on device during a Delta write in AWS Glue? This article walks through the issue, the role of Glue workers, and a few simple checks that can help identify the real bottleneck.

Where Does Your Data Actually Live?

A practical guide to Java virtual threads: pinning and how to detect it, what happens to blocking calls against a degraded resource, the semaphore pattern, the Java 21 to 25 timeline, and the 2023 adv...

Looking for a true Java heap space error solution? Learn the root causes of java.lang.OutOfMemoryError and step-by-step methods to fix memory leaks for good.

We’ve all been there. Your web application needs to do something time-consuming; send an email, process an uploaded file, generate a report,.. but making the user wait is simply not an option.

When it comes to tuning and troubleshooting memory issues in Java, many people think only of the Java...

The java.lang.OutOfMemoryError: GC overhead limit exceeded is another type of OutOfMemoryError in Java which comes when JVM spent too much time doing garbage collection without any success. For example, if almost 98% of CPU for a Java process is busy doing GC and reclaims very less amount of Java heap space around 2%, then JVM throws **"java.lang.OutOfMemoryError: GC overhead limit exceeded"** error. Though, the definition of 98% CPU time may vary between different Garbage collectors and different JVM versions. Unlike java.lang.OutOfMemoryError: Java heap space and java.langOutOfMemory: Permgen space, is not so frequent and you will rarely encounter this error in your Java application, but this also denotes resource exhaustion. Btw, if you are thinking that there is only one type of OutOfMemoryError in Java which comes when there is not enough memory in heap to create objects in heap, you are wrong. There are different types of OutOfMemoryError in Java and each denotes a different type of resource problems e.g. **java.lang.OutOfMemoryError: Java heap space** This error means not enough heap memory to create new objects. **java.lang.OutOfMemoryError: Permgen space** This error means not enough memory in the permanent generation of the heap to load class metadata. **java.lang.OutOfMemoryError: Direct buffer memory** means not enough heap memory to allocate to direct byte buffer. **java.lang.OutOfMemoryError: unable to create new native thread** means your application has exceeded the limit to create threads. **java.lang.OutOfMemoryError: Metaspace** this error is Java 8 equivalent of Permgen error because the Permanent generation has been removed in Java 8 and a new Metaspace is introduced. Now, let's try to find out some of the possible causes of the "GC overhead limit exceeded" error in JVM. ## _Possible cause of GC overhead limit exceeded OutOfMemoryError in Java_ If you suddenly start getting java.lang.OutOfMemoryError: GC overhead limit exceeded in your Java application then there are two possibilities either your application have developed a memory leak or you have introduced code changes which demand more memory and tend to store more objects in memory which is preventing Garbage collector to free memory from the heap. To verify this you can do the following things 1) find out if you have increased the size of your application cache(if any) but not increased -Xmx (max heap size setting) accordingly. 2) find out if you have introduced a new cache or caching data structure e.g. Collection classes like HashMap which stores object in memory. As a general best practice to avoid resource exhaustion you should always use the bounded cache as in the case of unbounded cache you may run out of memory but with the bounded cache, you don't grow without your limit. And, if you are serious about improving your advanced JVM skill and learn things like taking and analyzing heap dumps then highly recommend you to join **Java Application Performance and Memory Management** course on Udemy. It's one of the advanced courses for Java programmers to learn more about Performance and Memory management including troubleshooting memory leaks in Java. ## _Suggestions to fix java.lang.OutOfMemoryError: GC overhead limit exceeded_ One of the first things you can do to solve java.lang.OutOfMemoryError: GC overhead limit exceeded is to find out whether you have a memory leak or not because if you have memory leak changing heap sizes will just delay the occurrence of java.lang.OutOfMemoryError: GC overhead limit exceeded, it won't eliminate it. But if your application genuinely needs more memory may be because of increased cache size or the introduction of new caches then you can do the following things to fix java.lang.OutOfMemoryError: GC overhead limit exceeded in Java: 1) Increase the maximum heap size to a number that is suitable for your application e.g. -Xmx=4G. Remember, if you are running on 32-bit JVM you cannot set a maximum heap size of more than 2G on different operating systems e.g. Windows, Linux, or Solaris. If your application needs more memory, it's better to switch to a 64-bit Java Virtual Machine. If you try to provide more than 2G to a 32-bit JVM, it will not start and throw invalid heap size error. 2) If you are not using already then try using -XX:+UseConcMarkSweepGC Garbage collector in your Java application. This is one of the best garbage collectors available in JVM prior to Java 7. From Java 7 onward you can also use the G1 Garbage collector, which will also become the default garbage collector from Java 9. If you want to learn more about different types of Garbage collectors and how they work, I suggest you read a good book on JVM internals and Java performance tuning e.g. you can choose from definitive guide to Java performance by Scott Oaks or a more recent Java performance companion from Charlie hunt, both are great book to learn JVM internals and GC. 3) Avoid creating too much garbage and reuse temporary objects if you can. This should be the last thing you should do because reusing temporary objects may create subtle issues. make sure anything you reuse is safe and can be reusable in terms of functionality especially in a concurrent and multi-threading environment. Don't try to reuse static variables without proper synchronization to avoid any surprise race conditions in your Java application. That's all about **how to fix this OutOfMemoryError in your Java application**. The " java.lang.OutOfMemoryError: GC overhead limit exceeded" is one of the rare errors in Java application, but when it comes it takes some time to go away because finding the actual cause is not very straight forward. You need a combination of code analysis as well as load analysis for your application. It may also be the time for capacity planning for your application. Other **Java Troubleshooting Guides** you may like to explore * How to solve java.sql.BatchUpdateException: String or binary data would be truncated (guide) * How to solve java.lang.ClassNotFoundException: com.mysql.jdbc.Driver error? (hint) * 5 Reasons of NoClassDefFoundError in Java? (tutorial) * How to fix Caused By: java.lang.NoClassDefFoundError: org/apache/log4j/Logger (solution) * Cause of "java.lang.SecurityException: Missing required Permissions manifest attribute in main jar" [solution] * How to fix "illegal start of expression" compile time error in Java? (tutorial) * How to fix "Error: Could not find or load main class" in Eclipse? (guide) * How to fis unable to find certification path error in Java SSL? (guide) * 10 common reasons of java.lang.NumberFormatException in Java? (tutorial) * java.sql.SQLException: No suitable driver found for 'jdbc:mysql://localhost:3306/mysql [Solution] * How to solve "variable might not have initialized" compile time error in Java? (answer) * How to deal with "No JVM installation found, please install 64-bit JDK" error? (solution) * How to fix 'javac' is not recognized as an internal or external command (solution) * How to avoid ConcurrentModificationException in Java? (tutorial) * How to solve "could not create the Java virtual machine" error in Java? (solution) * Common reasons of java.lang.ArrayIndexOutOfBoundsException in Java? (solution) * java.lang.ClassNotFoundException : org.Springframework.Web.Context.ContextLoaderListener (solution) * java.sql.SQLServerException: The index 58 is out of range - JDBC (solution) * Fixing java.lang.unsupportedclassversionerror unsupported major.minor version 50.0 (solution) * java.lang.ClassNotFoundException: org.apache.commons.logging.LogFactory error (solution) * Cause and solution of "class, interface, or enum expected" compiler error in Java? (fix) * How to connect to MySQL database in Java? (tutorial) * How to solve java.lang.classnotfoundexception oracle.jdbc.driver.oracledriver? (solution) Thanks for reading this article so far. If you like this tutorial then please share it with your friends and colleagues. If you have any questions or suggest then please drop a comment.

**Tomcat web serve** r often suffers from **java.lang.OutOfMemoryError: PermGen space** whenever you deploy and un-deploy your web application a couple of times. No matter you are using **tomcat6** , **tomcat7,** or using bundled tomcat in Netbeans or Eclipse you will face this error now and then while developing web application on tomcat server. I thought about this article after writing 2 Solution of OutOfMemoryError in Java. I have touched this issue there but then I thought to write a separate tutorial for tomcat outofmemoryerror because I am getting this error too frequently. In this article, we will see **what causes java.lang.OutOfMemoryError: PermGen Space in tomcat** and _how to fix java.lang.OutOfMemoryError: PermGen Space in the tomcat server_. ## _Tomcat – java.lang.OutOfMemoryError: PermGen space Cause and Solution_ ### 1. Cause of OutOfMemoryError in PermGen space in Tomcat: PermGen Space of the heap is used to store classes and Metadata about classes in Java. When a class is loaded by a classloader it got stored in PermGen space, it gets unloaded only when the classloader which loaded this class got garbage collected. If any object retains reference of classloader than it's not garbage collected and Perm Gen Space is not freed up. This causes a memory leak in PermGen Space and eventually causes **java.lang.OutOfMemoryError****: PermGen space.** Another important point is that when you deploy your web application a new Classloader gets created and it loads the classes used by the web application. So if Classloader doesn't get garbage collected when your web application stops you will have a memory leak in tomcat. ## _Solution of Tomcat: OutOfMemroyError:_ 1) Find the offending classes which are retaining reference of Classloader and preventing it from being garbage collected. Tomcat provides memory leak detection functionality _after tomcat 6_ onwards which can help you to find when the particular library, framework or class is causing a memory leak in tomcat. Here are some of the common causes of **java.lang.OutOfMemoryError: PermGen space in tomcat server** : 1) **JDBC Drivers:** JDBC drivers are the most common cause of java.lang.OutOfMemoryError: PermGen space in tomcat if the web app doesn't unregister during the stop. One hack to get around this problem is that JDBC driver to be loaded by a common classloader than application classloader and you can do this by transferring driver's jar into tomcat lib instead of bundling it on the web application's war file. 2) **Logging framework:** A similar solution can be applied to prevent logging libraries like Log4j causing java.lang.OutOfMemoryError: PermGen space in tomcat. 3) **Application Threads that have not stopped.** Check your code carefully if you are leaving your thread unattended and running in a while loop that can retain the classloader's reference and cause java.lang.OutOfMemoryError: PermGen space in the tomcat web server. Another common culprit is ThreadLocal, avoid using it until you need it absolutely, if do you make sure to set them null or free any object ThreadLocal variables are holding. Another Simple Solution is to increase PermGen heap size in catalina.bat or Catalina.sh of tomcat server; this can give you some breathing space but eventually, this will also return in _java.lang.OutOfMemoryError: PermGen space_ after some time. ## Steps to increase PermGen Heap Space in Tomcat: 1) Go to Tomcat installation directory i.e C:\Program Files\Apache Software Foundation\Apache Tomcat 7.0.14\bin in Windows and something similar in Linux. 2) Add JAVA_OPTS in your catalina.bat or Catalina.sh In Windows: set JAVA_OPTS="-Xms1024m -Xmx10246m -XX:NewSize=256m -XX:MaxNewSize=356m -XX:PermSize=256m -XX:MaxPermSize=356m" In linux: export JAVA_OPTS="-Xms1024m -Xmx10246m -XX:NewSize=256m -XX:MaxNewSize=356m -XX:PermSize=256m -XX:MaxPermSize=356m" You can change the actual heap size and PermGen Space as per your requirement. 3) Restart Tomcat. As I said earlier **increasing PermGen space** can prevent **java.lang.OutOfMemoryError: PermGen** in tomcat only for some time and it will eventually occur based on how many times you redeploy your web application, its best to find the offending class which is causing a memory leak in tomcat and fix it. _**Related Java Tutorials**_ How to Solve Java.lang.ClassNotFoundException in Java Difference between NoClassDefFoundError and ClassNotFoundException in Java How to fix Bad version number in .class file Difference between ArrayList and Vector in Java How SubString works in Java Difference between SynchronizedCollection and ConcurrentCollection in Java Difference between Comparator and Comparable in Java

Posted by Gary D. Gregory on Feb 19 Severity: moderate Affected versions: - Apache Commons Compress 1.21 before 1.26.0 Description: Allocation of Resources Without Limits or Throttling vulnerability in Apache Commons Compress.This issue affects Apache Commons Compress: from 1.21 before 1.26. Users are recommended to upgrade to version 1.26, which fixes the issue. Credit: Yakov Shafranovich, Amazon Web Services (reporter) References: https://commons.apache.org/ ...