Java lang outofmemoryerror gc overhead limit exceeded как исправить

jUnit тесты работают с большими объемами данных, в частности, данная ошибка возникает, когда считываем данные из базы в объект.

ОЗУ — 16 Гб, 4 виртуальных процессора

когда возникла ошибка

java.lang.OutOfMemoryError: GC overhead limit exceeded
at sun.reflect.GeneratedConstructorAccessor47.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at java.lang.Class.newInstance(Class.java:442)

то я добавил параметр -XX:-UseGCOverheadLimit
но получил следующую ошибку:

java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.reflect.Field.copy(Field.java:150)
    at java.lang.reflect.ReflectAccess.copyField(ReflectAccess.java:144)
    at sun.reflect.ReflectionFactory.copyField(ReflectionFactory.java:309)
    at java.lang.Class.copyFields(Class.java:3115)
    at java.lang.Class.getFields(Class.java:1557)
    at main.java.DataFormat.DbTable.loadObjectFromResultSet(DbTable.java:101)
    at main.java.DataFormat.DbTable.getEntities(DbTable.java:57)

все параметры JVM -Xmx8192M -XX:-UseGCOverheadLimit

в момент появления ошибки были следующие параметры системы:
ОЗУ стабильно на 46% (7400Мб), ЦПУ 77%-82% (82% в момент появления ошибки)

подскажите, как исправить данную проблему?

Спарвка

1. java.lang.OutOfMemoryError: GC overhead limit exceeded

Данная ошибка может возникнуть как при переполнении первой, так и
второй областей. Связана она с тем, что памяти осталось мало и GC
постоянно работает, пытаясь высвободить немного места. Данную ошибку
можно отключить с помощью параметра -XX:-UseGCOverheadLimit, но,
конечно же, её надо не отключать, а либо решать проблему утечки
памяти, либо выделять больше объема, либо менять настройки GC.

Ответ
String объект слишком много весит. а у меня там очень много стрингов. Для примера, посчитали файл, который весит 250 МБ, имеет 1600 символов в строке и 220 000 строк. Заполняю в ArrayList<String> lines, где каждая строка записана как объект из 250 полей, которые принимают значение из 1600 символов. Такой объект весил чуть больше 6Гб.

Решение: увеличил память до 14Гб. теперь проблема в том, что 6Гбайтный объект не помещается в куче, надо думать что-то другое и пожертвовать временем выполнения теста.

java.lang.OutOfMemoryError:
GC overhead limit exceeded

Java runtime environment contains a built-in Garbage Collection (GC) process. In many other programming languages, the developers need to manually allocate and free memory regions so that the freed memory can be reused.

Java applications on the other hand only need to allocate memory. Whenever a particular space in memory is no longer used, a separate process called Garbage Collection clears the memory for them. How the GC detects that a particular part of memory is explained in more detail in the Garbage Collection Handbook, but you can trust the GC to do its job well.

The java.lang.OutOfMemoryError: GC overhead limit exceeded error is displayed when your application has exhausted pretty much all the available memory and GC has repeatedly failed to clean it.

What is causing it?

The java.lang.OutOfMemoryError: GC overhead limit exceeded error is the JVM’s way of signalling that your application spends too much time doing garbage collection with too little result. By default the JVM is configured to throw this error if it spends more than 98% of the total time doing GC and when after the GC only less than 2% of the heap is recovered.

What would happen if this GC overhead limit would not exist? Note that the java.lang.OutOfMemoryError: GC overhead limit exceeded error is only thrown when 2% of the memory is freed after several GC cycles. This means that the small amount of heap the GC is able to clean will likely be quickly filled again, forcing the GC to restart the cleaning process again. This forms a vicious cycle where the CPU is 100% busy with GC and no actual work can be done. End users of the application face extreme slowdowns – operations which normally complete in milliseconds take minutes to finish.

So the “java.lang.OutOfMemoryError: GC overhead limit exceeded” message is a pretty nice example of a fail fast principle in action.

Give me an example

In the following example we create a “GC overhead limit exceeded” error by initializing a Map and adding key-value pairs into the map in an unterminated loop:

class Wrapper {
  public static void main(String args[]) throws Exception {
    Map map = System.getProperties();
    Random r = new Random();
    while (true) {
      map.put(r.nextInt(), "value");
    }
  }
}

As you might guess this cannot end well. And, indeed, when we launch the above program with:

java -Xmx100m -XX:+UseParallelGC Wrapper

we soon face the java.lang.OutOfMemoryError: GC overhead limit exceeded message. But the above example is tricky. When launched with different Java heap size or a different GC algorithm, my Mac OS X 10.9.2 with Hotspot 1.7.0_45 will choose to die differently. For example, when I run the program with smaller Java heap size like this:

java -Xmx10m -XX:+UseParallelGC Wrapper

the application will die with a more common java.lang.OutOfMemoryError: Java heap space message that is thrown on Map resize. And when I run it with other garbage collection algorithms besides ParallelGC, such as -XX:+UseConcMarkSweepGC or -XX:+UseG1GC, the error is caught by the default exception handler and is without stacktrace as the heap is exhausted to the extent where the stacktrace cannot even be filled on Exception creation.

These variations are truly good examples that demonstrate that in resource-constrained situations you cannot predict the way your application is going to die so do not base your expectations on a specific sequence of actions to be completed.

What is the solution?

As a tongue-in-cheek solution, if you just wished to get rid of the “java.lang.OutOfMemoryError: GC overhead limit exceeded” message, adding the following to your startup scripts would achieve just that:

-XX:-UseGCOverheadLimit

I would strongly suggest NOT to use this option though – instead of fixing the problem you just postpone the inevitable: the application running out of memory and needing to be fixed. Specifying this option will just mask the original java.lang.OutOfMemoryError: GC overhead limit exceeded error with a more familiar message java.lang.OutOfMemoryError: Java heap space.

On a more serious note – sometimes the GC overhead limit error is triggered because the amount of heap you have allocated to your JVM is just not enough to accommodate the needs of your applications running on that JVM. In that case, you should just allocate more heap – see at the end of this chapter for how to achieve that.

In many cases however, providing more Java heap space will not solve the problem. For example, if your application contains a memory leak, adding more heap will just postpone the java.lang.OutOfMemoryError: Java heap space error. Additionally, increasing the amount of Java heap space also tends to increase the length of GC pauses affecting your application’s throughput or latency.

If you wish to solve the underlying problem with the Java heap space instead of masking the symptoms, you need to figure out which part of your code is responsible for allocating the most memory. In other words, you need to answer these questions:

1. Which objects occupy large portions of heap
2. where these objects are being allocated in source code

At this point, make sure to clear a couple of days in your calendar (or – see an automated way below the bullet list). Here is a rough process outline that will help you answer the above questions:

• Get clearance for acquiring a heap dump from your JVM-to-troubleshoot. “Dumps” are basically snapshots of heap contents that you can analyze, and contain everything that the application kept in memory at the time of the dump. Including passwords, credit card numbers etc.
• Instruct your JVM to dump the contents of its heap memory into a file. Be prepared to get a few dumps, as when taken at a wrong time, heap dumps contain a significant amount of noise and can be practically useless. On the other hand, every heap dump “freezes” the JVM entirely, so don’t take too many of them or your end users start swearing.
• Find a machine that can load the dump. When your JVM-to-troubleshoot uses for example 8GB of heap, you need a machine with more than 8GB to be able to analyze heap contents. Fire up dump analysis software (we recommend Eclipse MAT, but there are also equally good alternatives available).
• Detect the paths to GC roots of the biggest consumers of heap. We have covered this activity in a separate post here. Don’t worry, it will feel cumbersome at first, but you’ll get better after spending a few days digging.
• Next, you need to figure out where in your source code the potentially hazardous large amount of objects is being allocated. If you have good knowledge of your application’s source code you’ll hopefully be able to do this in a couple searches. When you have less luck, you will need some energy drinks to assist.

Alternatively, we suggest Plumbr, the only Java monitoring solution with automatic root cause detection. Among other performance problems it catches all java.lang.OutOfMemoryErrors and automatically hands you the information about the most memory-hungry data structres. It takes care of gathering the necessary data behind the scenes – this includes the relevant data about heap usage (only the object layout graph, no actual data), and also some data that you can’t even find in a heap dump. It also does the necessary data processing for you – on the fly, as soon as the JVM encounters an java.lang.OutOfMemoryError. Here is an example java.lang.OutOfMemoryError incident alert from Plumbr:

Without any additional tooling or analysis you can see:

• Which objects are consuming the most memory (271 com.example.map.impl.PartitionContainer instances consume 173MB out of 248MB total heap)
• Where these objects were allocated (most of them allocated in the MetricManagerImpl class, line 304)
• What is currently referencing these objects (the full reference chain up to GC root)

Equipped with this information you can zoom in to the underlying root cause and make sure the data structures are trimmed down to the levels where they would fit nicely into your memory pools.

However, when your conclusion from memory analysis or from reading the Plumbr report are that memory use is legal and there is nothing to change in the source code, you need to allow your JVM more Java heap space to run properly. In this case, alter your JVM launch configuration and add (or increase the value if present) just one parameter in your startup scripts:

java -Xmx1024m com.yourcompany.YourClass

In the above example the Java process is given 1GB of heap. Modify the value as best fits to your JVM. However, if the result is that your JVM still dies with OutOfMemoryError, you might still not be able to avoid the manual or Plumbr-assisted analysis described above.

In Java, all objects are stored in a heap. They are allocated using a new operator. The OutOfMemoryError Exception in Java looks like this: 

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space

Usually, this error is thrown when the Java Virtual Machine cannot allocate an object because it is out of memory. No more memory could be made available by the garbage collector.

OutOfMemoryError usually means that you’re doing something wrong, either holding onto objects too long or trying to process too much data at a time. Sometimes, it indicates a problem that’s out of your control, such as a third-party library that caches strings or an application server that doesn’t clean up after deploys. And sometimes, it has nothing to do with objects on the heap.

The java.lang.OutOfMemoryError exception can also be thrown by native library code when a native allocation cannot be satisfied (for example, if swap space is low). Let us understand various cases when the OutOfMemory error might occur.

Symptom or Root cause?

To find the cause, the text of the exception includes a detailed message at the end. Let us examine all the errors. 

Error 1 – Java heap space: 

This error arises due to the applications that make excessive use of finalizers. If a class has a finalize method, objects of that type do not have their space reclaimed at garbage collection time. Instead, after garbage collection, the objects are queued for finalization, which occurs later. 

Implementation: 

• finalizers are executed by a daemon thread that services the finalization queue.
• If the finalizer thread cannot keep up with the finalization queue, the Java heap could fill up, and this type of OutOfMemoryError exception would be thrown.
• The problem can also be as simple as a configuration issue, where the specified heap size (or the default size, if it is not specified) is insufficient for the application.

Output:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at Heap.main(Heap.java:11)

When you execute the above code above you might expect it to run forever without any problems. As a result, over time, with the leaking code constantly used, the “cached” results end up consuming a lot of Java heap space, and when the leaked memory fills all of the available memory in the heap region and Garbage Collection is not able to clean it, the java.lang.OutOfMemoryError:Java heap space is thrown.

Prevention: Check how to monitor objects for which finalization is pending in Monitor the Objects Pending Finalization.

Error 2 – GC Overhead limit exceeded: 

This error indicates that the garbage collector is running all the time and Java program is making very slow progress. After a garbage collection, if the Java process is spending more than approximately 98% of its time doing garbage collection and if it is recovering less than 2% of the heap and has been doing so far the last 5 (compile-time constant) consecutive garbage collections, then a java.lang.OutOfMemoryError is thrown. 

This exception is typically thrown because the amount of live data barely fits into the Java heap having little free space for new allocations. 

If you run this program with java -Xmx100m -XX:+UseParallelGC Wrapper, then the output will be something like this : 

Exception in thread "main" java.lang.OutOfMemoryError: GC overhead limit exceeded
    at java.lang.Integer.valueOf(Integer.java:832)
    at Wrapper.main(error.java:9)

Prevention: Increase the heap size and turn off it with the command line flag -XX:-UseGCOverheadLimit. 

Error 3 – Permgen space is thrown: 

Java memory is separated into different regions. The size of all those regions, including the permgen area, is set during the JVM launch. If you do not set the sizes yourself, platform-specific defaults will be used. 

The java.lang.OutOfMemoryError: PermGen space error indicates that the Permanent Generation’s area in memory is exhausted. 

In the above sample code, code iterates over a loop and generates classes at run time. Class generation complexity is being taken care of by the Javassist library. 

Running the above code will keep generating new classes and loading their definitions into Permgen space until the space is fully utilized and the java.lang.OutOfMemoryError: Permgen space is thrown. 

Prevention : When the OutOfMemoryError due to PermGen exhaustion is caused during the application launch, the solution is simple. The application just needs more room to load all the classes to the PermGen area, so we need to increase its size. To do so, alter your application launch configuration and add (or increase if present) the -XX:MaxPermSize parameter similar to the following example: 

java -XX:MaxPermSize=512m com.saket.demo.Permgen

Error 4 – Metaspace: 

Java class metadata is allocated in native memory. Suppose metaspace for class metadata is exhausted, a java.lang.OutOfMemoryError exception with a detail MetaSpace is thrown. 

The amount of metaspace used for class metadata is limited by the parameter MaxMetaSpaceSize, which is specified on the command line. When the amount of native memory needed for a class metadata exceeds MaxMetaSpaceSize, a java.lang.OutOfMemoryError exception with a detail MetaSpace is thrown.

This code will keep generating new classes and loading their definitions to Metaspace until the space is fully utilized and the java.lang.OutOfMemoryError: Metaspace is thrown. When launched with -XX:MaxMetaspaceSize=64m then on Mac OS X my Java 1.8.0_05 dies at around 70, 000 classes loaded.

Prevention: If MaxMetaSpaceSize, has been set on the command line, increase its value. MetaSpace is allocated from the same address spaces as the Java heap. Reducing the size of the Java heap will make more space available for MetaSpace. This is only a correct trade-off if there is an excess of free space in the Java heap. 

Error 5 – Requested array size exceeds VM limit: 

This error indicates that the application attempted to allocate an array that is larger than the heap size. For example, if an application attempts to allocate an array of 1024 MB but the maximum heap size is 512 MB then OutOfMemoryError will be thrown with “Requested array size exceeds VM limit”. 

Output:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at GFG.main(GFG.java:12)

The java.lang.OutOfMemoryError: Requested array size exceeds VM limit can appear as a result of either of the following situations: 

• Your arrays grow too big and end up having a size between the platform limit and the Integer.MAX_INT
• You deliberately try to allocate arrays larger than 2^31-1 elements to experiment with the limits.

Error 6 – Request size bytes for a reason. Out of swap space?: 

This apparent exception occurred when an allocation from the native heap failed and the native heap might be close to exhaustion. The error indicates the size (in bytes) of the request that failed and the reason for the memory request. Usually, the reason is the name of the source module reporting the allocation failure, although sometimes it is the actual reason. 

The java.lang.OutOfMemoryError: Out of swap space error is often caused by operating-system-level issues, such as: 

• The operating system is configured with insufficient swap space.
• Another process on the system is consuming all memory resources.

Prevention: When this error message is thrown, the VM invokes the fatal error handling mechanism (that is, it generates a deadly error log file, which contains helpful information about the thread, process, and system at the time of the crash). In the case of native heap exhaustion, the heap memory and memory map information in the log can be useful.

Error 7 – reason stack_trace_with_native_method: 

Whenever this error message(reason stack_trace_with_native_method) is thrown then a stack trace is printed in which the top frame is a native method, then this is an indication that a native method has encountered an allocation failure. The difference between this and the previous message is that the allocation failure was detected in a Java Native Interface (JNI) or native method rather than the JVM code. 

The exact native thread limit is platform-dependent. For example, tests Mac OS X reveals that: 64-bit Mac OS X 10.9, Java 1.7.0_45 – JVM dies after #2031 threads have been created

Prevention: Use native utilities of the OS to diagnose the issue further. For more information about tools available for various operating systems, see Native Operating System tools. 

This article is contributed by Saket Kumar. If you like GeeksforGeeks and would like to contribute, you can also write an article using write.geeksforgeeks.org or mail your article to review-team@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks. Please write comments if you find anything incorrect, or you want to share more information about the topic discussed above.