GC overhead limit exceeded

[vai]

I had a bot pinging my server and I saw this error in the server logs - do you have any insights on what might be causing this error in es5 ?


2011-Oct-10 19:51:55:738 [worker-5 BinaryTCP-0] ERROR com.electrotank.electroserver5.servers.registry.Re gistryEndpoint - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2011-Oct-10 20:01:58:743 [pool-1-thread-3] WARN com.electrotank.electroserver5.protocol.IOErrorHan dler - IOErrorHandler.exceptionCaught
java.nio.BufferOverflowException
at java.nio.Buffer.nextPutIndex(Buffer.java:501)
at java.nio.HeapByteBuffer.putLong(HeapByteBuffer.jav a:398)
at com.electrotank.electroserver5.protocol.etio.Abstr actByteBufferMessageWriter.writeLong(AbstractByteB ufferMessageWriter.java:116)
at com.electrotank.electroserver5.protocol.AbstractMe ssageWriter.writeLong(AbstractMessageWriter.java:3 6)
at com.electrotank.electroserver5.protocol.codecs.int ernal.GatewayKickUserRequestCodec.encode(GatewayKi ckUserRequestCodec.java:24)
at com.electrotank.electroserver5.protocol.codecs.ext ernal.PrefixingMessageCodecDecorator.encode(Prefix ingMessageCodecDecorator.java:55)
at com.electrotank.electroserver5.protocol.Codec.enco de(Codec.java:37)
at com.electrotank.electroserver5.protocol.etio.tcp.T CPEncoder.onMessage(TCPEncoder.java:64)
at com.electrotank.electroserver5.protocol.etio.tcp.T CPEncoder.onMessage(TCPEncoder.java:45)
at org.jetlang.channels.MemoryChannel.publish(MemoryC hannel.java:27)
at com.electrotank.electroserver5.protocol.etio.EtIOC lientSession.writeAndCloseConnection(EtIOClientSes sion.java:116)
at com.electrotank.electroserver5.entities.DefaultGat ewayClient.sendAndClose(DefaultGatewayClient.java: 30)
at com.electrotank.electroserver5.servers.gateway.tra nsactions.GatewayKickUserRequestTransaction.execut e(GatewayKickUserRequestTransaction.java:35)
at com.electrotank.electroserver5.servers.gateway.tra nsactions.GatewayKickUserRequestTransaction.execut e(GatewayKickUserRequestTransaction.java:13)
at com.electrotank.electroserver5.servers.gateway.tra nsactions.TransactionHandler.processMessage(Transa ctionHandler.java:33)
at com.electrotank.electroserver5.servers.gateway.Gat ewayEndpoint.receive(GatewayEndpoint.java:63)
at com.electrotank.electroserver5.servers.registry.Re gistryEndpoint.sendMessageToGateway(RegistryEndpoi nt.java:241)
at com.electrotank.electroserver5.servers.registry.Re gistryEndpoint.sendMessageToClient(RegistryEndpoin t.java:249)
at com.electrotank.electroserver5.entities.ConnectedU ser.kick(ConnectedUser.java:160)
at com.electrotank.electroserver5.entities.ConnectedU ser.kick(ConnectedUser.java:166)
at com.electrotank.electroserver5.servers.registry.Id leMonitor.run(IdleMonitor.java:47)
at java.util.concurrent.Executors$RunnableAdapter.cal l(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndRe set(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(Future Task.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.access$101(ScheduledThreadPoolE xecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.runPeriodic(ScheduledThreadPool Executor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$S cheduledFutureTask.run(ScheduledThreadPoolExecutor .java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.run Task(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
2011-Oct-10 20:02:11:218 [openclose] ERROR com.electrotank.electroserver5.servers.registry.Re gistryEndpoint - GC overhead limit exceeded
2011-Oct-10 20:02:12:892 [openclose] ERROR com.electrotank.electroserver5.DefaultExceptionHan dler - Uncaught exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
2011-Oct-10 20:02:17:853 [btpool0-1 - Acceptor0 SelectChannelConnector @ <ip>:8080] ERROR com.electrotank.electroserver5.DefaultExceptionHan dler - Uncaught exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
2011-Oct-10 21:35:11:359 [worker-14 BinaryTCP-0] ERROR com.electrotank.electroserver5.DefaultExceptionHan dler - Uncaught exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
2011-Oct-10 21:35:15:088 [pool-1-thread-1] ERROR com.electrotank.electroserver5.DefaultExceptionHan dler - Uncaught exception

[vai]

There is one more thing which I've noticed - every time I restart the server, in console.log, I see a message like "Low memory!! -
90.99
%" ..

[tcarr]

"GC overhead limit exceeded" means that your Garbage Collector is complaining that your application is taking up more memory than it is comfortable with. You should increase your ES5's heap space, particularly since this is happening right after a restart (so it's not just a case of a memory leak that needs to be tracked down). See Adjusting ElectroServer's Memory (http://www.electrotank.com/docs/es5/manual/server_configuration_tweaks.htm). Please note that if you run ES5 using SAFEMODE then the memory settings from the ES Admin are ignored - if that is the case, let me know and I'll tell you how to set them in your start script.

[vai]

Just to make it clear, I get the low memory notification in console.log every time I restart the server. The GC overhead limit exceeded happened just once yesterday and I must have hit the server 200,000 times yesterday by the time I got the gc overhead error.

[vai]

So, I changed the max heap size to 264MB and es5 would not start. Then I used java -jar lib/ElectroServer5-bootstrap.jar -safemode -mode StandAlone -config config/Configuration.xml to start the server in safe mode but still it does not start the server. In the log file, I can see that my plugins were loaded but nothing after that. Normally, I would see something like -


2011-Oct-11 06:07:55:703 [main] INFO com.electrotank.electroserver5.servers.gateway.Gat ewayServer - GatewayServer.doStart
2011-Oct-11 06:07:55:760 [main] INFO com.electrotank.electroserver5.protocol.IOProvider - Opened BinaryTCP listener on

[tcarr]

When ES5 doesn't restart correctly, sometimes it's because it didn't shut down correctly. Try killing ES5 (for Windows, check on all java processes in Task Manager; for Linux check "ps -ef | grep java"), then starting. You should also check your hard drive space.

Hmmm... another thing to check is "all your plugins". Do you have any server level components that don't have a logging line at the end of their init method? ES5 might be hung up initializing one of those server level components, particularly if there isn't enough memory for everything and the OS starts thrashing.

[vai]

there is no es5 process running .. I will add a debug statement at the end of init() of all of my plugins.. do you know how much memory would be availalbe in safe mode ?

[tcarr]

Safe mode will give you whatever your JVM default memory is. This varies with version of JVM, and if you have customized the default JVM heap size. If you look at your old logs from when ES5 did start successfully, you will see two sets of the startup info in the logs, before it starts mentioning your various plugins and gateway listeners. The first one will give you the JVM default heap size, the second will give you the heap size set in ES Admin.

It's entirely possible that 264MB is still too small for all your extensions. Many ES5 applications would have no problem fitting in that size, but assuming your server has plenty of memory, you might need to try 512 MB or 1024, then check the ES Admin home screen to see how much you actually use.

[vai]

I was using 64MB previously and all of my extensions would load properly. Today I changed it to 264MB and after that i could not start the server. can i specify the heap size in an startup script like you mentioned before?

[tcarr]

Your problem is likely that your OS doesn't *have* 264MB available for the JVM, so that the bootstrapper can't get the ES5 main process started.

What happens when you start using SAFEMODE? Do you get any error messages? SAFEMODE is supposed to allow you to start ES5 enough that you can get to the ES Admin to undo changes to the heap size or command line parameters that are preventing ES5 from starting.

If you tell me your OS I can give you a startup script or batch file. If you are running ES5 as a service I need to know that.

[vai]

when I start the server using safemode, I think it does not give me enough memory to load up all the plugins. I am using a linux box with centos and I believe it is not a service on the remote box.

[tcarr]

There are several ways to run ES5 from a Linux script. See Running ElectroServer (http://www.electrotank.com/docs/es5/manual/running_electroserver.htm). If you use one of the "java jar" scripts, you can specify the JVM's heap size by adding -Xmx64M (or whatever amount of Mb you want to give it) to the other options on the java jar line.

If you want to revert your ES Admin's heap size instead of fooling with the command line and safemode, I can tell you how to use DbUtility to do that.

[vai]

if i use java -jar -Xms128M -Xmx128M lib/ElectroServer5-bootstrap.jar -mode StandAlone -config config/Configuration.xml, the server starts properly .. but if I set the heap size to 128M through es5 admin, the server does not start.

[tcarr]

It sounds like your default JVM might be grabbing too much memory for the bootstrap process, or else that the ES Admin's settings were not made correctly. Can you give me a screen shot of the ES Admin's general settings, showing both the heap and command line boxes? and the ElectroServer5.log file for when you try to start ES5 without the java jar script, so that I can see the amount of memory that JVM is grabbing for the bootstrapper.

Or you can just shrug and decide to use the java jar script. I use that kind of script all the time myself. If you use that and then use the ES Admin's Restart button, does it start correctly?

[vai]

227


i had started my server the last time using java jar script .. so i just restarted it from es5 admin and it did come up fine.

[vai]

227


i had started my server the last time using java jar script .. so i just restarted it from es5 admin and it did come up fine.

[tcarr]

It sounds like the problem is that 128M is close to the limit of the available memory on the box. The bootstrap process takes some memory, then there isn't 128M left for the main ES5 process. You can experiment with smaller amounts until you find one that works for you, or you can just continue to use the java jar start script.

(or find a way to get more memory on your host - either killing other processes or moving to another server that has more RAM, etc)

[vai]

If i start the server with "java -jar -Xms128M -Xmx128M lib/ElectroServer5-bootstrap.jar -mode StandAlone -config config/Configuration.xml" isn't it 128M minimum that it would get?

[tcarr]

If you don't use the "-safemode" option on your java jar line, this reserves 128M for the bootstrap process, which then tries to create the main process but there isn't enough memory for it. If you add "-safemode" then it's just a single process and it ignores the ES Admin's command line parameters. When you use the Restart button on the ES Admin, it stays in the main process thread, shutting down and then starting again without making another process.

[vai]

two more questions :)
1) do the bootstrap and main process share 128M ?
2) what does the max heap size set (in es5 admin tool) .. right now in my startup script I have java -jar -Xms128M -Xmx128M lib/ElectroServer5-bootstrap.jar -mode StandAlone -config config/Configuration.xml and in the max heap size in es5 admin tool it says 64MB.

Many Thanks!

[tcarr]

No they don't share the 128M. Try taking the -Xms128M off, because that is telling the bootstrap process to take all 128M. If you are not using safemode, and the ES Admin says your max heap size is 64, then that's how much memory the main ES5 process is given. You want to use less memory for bootstrap and more for main process.