java.io.IOException: Map failed and java.lang.OutOfMemoryError: Map failed

While working with memory mapped file, you may get java.io.IOException: Map failed error, which is mainly caused by Caused by: java.lang.OutOfMemoryError: Map failed error as shown below. This error usually comes while mapping a big file in memory e.g. trying to map a file greater than 1 or 2GB

java.io.IOException: Map failed
 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
Caused by: java.lang.OutOfMemoryError: Map failed
 at sun.nio.ch.FileChannelImpl.map0(Native Method)
 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885)
 ... 6 more


How to Solve java.io.IOException: Map failed

Depending upon which operating system you are working, e.g. Windows or UNIX, you can try different things to solve this error. One of the quick way to solve this by reducing size, which you want to map into memory. For example if you are trying to map 1GB into memory, you may want to only map 512MB or lower, if your code uses region based approach, this may work file. Alternatively you can also use -d64 and -XX:MaxDirectMemorySize JVM option to enable large direct buffers :


-d64
This JVM flag instruct virtual machine to run on 64-bit mode. Since 64-bit installation on Solaris and may be on other UNIX systems comes with 32-bit JVM as well, which may be efficient for small programs, due to small size of OOPS pointer as discusses on why use -XX:UseCompressedOOPS flag in 64-bit JVM. By providing this option to your JVM you make sure that it's running on 64-bit mode, allowing it to map more memory.


-XX:MaxDirectMemorySize
This option is not on official list of JVM flags but still a good option to try. It allows you configure memory used by direct buffers.

 One more option, you can try is -XX:+UseLargePages.

 For example, if your program is using 6GB of memory mapped file, you may want to pass

 java -d64 -XX:MaxDirectMemorySize=6g -XX:+UseLargePages BigApp



That's all on how to solve java.io.IOException: Map failed and java.lang.OutOfMemoryError: Map failed. You can use any of these options or combination of them to see what works for you. Make sure you use 64-bit JVM if you are mapping large files in memory. Remember though both error are different, one is IOException and other is OutOfMemoryError, cause is more or less same, difference is IOException comes while using that file and OutOfMemoryError comes while mapping that file in memory.


4 comments :

SARAL SAXENA said...

well this error mostly comes for example ..
Consider application, which create 5-6 threads, each thread in cycle allocate MappedByteBuffer for 5mb page size.

MappedByteBuffer b = ch.map(FileChannel.MapMode.READ_ONLY, r, 1024*1024*5);

Sooner or later, when application works with big files, exception is thrown

java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)



I have avoid this exception by trigger a GC by cleaning up the mapped byte buffers directly.

public static void clean(ByteBuffer bb) {
if(bb == null) return;
Cleaner cleaner = ((DirectBuffer) bb).cleaner();
if(cleaner != null) cleaner.clean();
}
Provided you call this before discarding, you won't run out of virtual memory.

Anonymous said...

Hello, I am getting following exception while starting process in JVM "java.io.IOException: Failed to write to file: No space left on device" do you know what might be the cause here? what does it mean by device here?

Peter Lawrey said...

The -d64 flag is the default on solaris systems. It cannot be used on windows or linux.
The -XX:+UseCompressedOops is the default for up to 32 GB on java 6/7 and up to 64 GB on java 8.
The -XX:MaxDirectMemorySize controls the maximum direct memory ie ByteBuffer.allocateDirect() not memory mapping. You can create mappings of 128 TB on linux.
Using large pages might make a difference but generally it avoided as it increases the cost of paging.

Anonymous said...

@Peter, than what is real cause of java.io.IOException: Map failed and how do you fix it?

Post a Comment