While working with memory mapped file, you may get java.io.IOException: Map failed error, which is mainly caused by Caused by: java.lang.OutOfMemoryError: Map failed error as shown below. This error usually comes while mapping a big file in memory e.g. trying to map a file greater than 1 or 2GB
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885)
... 6 more
For example, if you are trying to map 1GB into memory, you may want to only map 512MB or lower, if your code uses a region-based approach, this may work file.
java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:888)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885)
... 6 more
How to Solve java.io.IOException: Map failed
Depending upon which operating system you are working, like Windows or UNIX, you can try different things to solve this error. One of the quick ways to solve this is by reducing the size, which you want to map into memory.For example, if you are trying to map 1GB into memory, you may want to only map 512MB or lower, if your code uses a region-based approach, this may work file.
By the way, if you don't know or have confusion how exactly memory mapped file work and how it can access memory outside the JVM then here is a nice diagram which explains that to you:
Alternatively, you can also use -d64 and -XX:MaxDirectMemorySize JVM option to enable large direct buffers :
-d64
This JVM flag instructs the virtual machine to run on 64-bit mode. Since 64-bit installation on Solaris and maybe on other UNIX systems comes with 32-bit JVM as well, which may be efficient for small programs, due to small size of OOPS pointer as discusses on why use -XX:UseCompressedOOPS flag in 64-bit JVM. By providing this option to your JVM you make sure that it's running on 64-bit mode, allowing it to map more memory.-XX:MaxDirectMemorySize
This option is not on the official list of JVM flags but is still a good option to try. It allows you to configure memory used by direct buffers.One more option, you can try is -XX:+UseLargePages.
For example, if your program is using 6GB of memory-mapped file, you may want to pass
$ java -d64 -XX:MaxDirectMemorySize=6g -XX:+UseLargePages BigAppThat's all on how to solve java.io.IOException: Map failed and java.lang.OutOfMemoryError: Map failed. You can use any of these options or a combination of them to see what works for you. Make sure you use 64-bit JVM if you are mapping large files in memory.
Remember though both errors are different, one is IOException and the other is OutOfMemoryError, the cause is more or less the same, the difference is IOException comes while using that file and OutOfMemoryError comes while mapping that file in memory.
well this error mostly comes for example ..
ReplyDeleteConsider application, which create 5-6 threads, each thread in cycle allocate MappedByteBuffer for 5mb page size.
MappedByteBuffer b = ch.map(FileChannel.MapMode.READ_ONLY, r, 1024*1024*5);
Sooner or later, when application works with big files, exception is thrown
java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:755)
I have avoid this exception by trigger a GC by cleaning up the mapped byte buffers directly.
public static void clean(ByteBuffer bb) {
if(bb == null) return;
Cleaner cleaner = ((DirectBuffer) bb).cleaner();
if(cleaner != null) cleaner.clean();
}
Provided you call this before discarding, you won't run out of virtual memory.
Hello, I am getting following exception while starting process in JVM "java.io.IOException: Failed to write to file: No space left on device" do you know what might be the cause here? what does it mean by device here?
ReplyDeleteThe -d64 flag is the default on solaris systems. It cannot be used on windows or linux.
DeleteThe -XX:+UseCompressedOops is the default for up to 32 GB on java 6/7 and up to 64 GB on java 8.
The -XX:MaxDirectMemorySize controls the maximum direct memory ie ByteBuffer.allocateDirect() not memory mapping. You can create mappings of 128 TB on linux.
Using large pages might make a difference but generally it avoided as it increases the cost of paging.
@Peter, than what is real cause of java.io.IOException: Map failed and how do you fix it?
ReplyDelete