GC overhead limit exceeded

[vai]

I had a bot pinging my server and I saw this error in the server logs - do you have any insights on what might be causing this error in es5 ?

2011-Oct-10 19:51:55:738 [worker-5 BinaryTCP-0] ERROR com.electrotank.electroserver5.servers.registry.RegistryEndpoint  - GC overhead limit exceeded
java.lang.OutOfMemoryError: GC overhead limit exceeded
2011-Oct-10 20:01:58:743 [pool-1-thread-3] WARN  com.electrotank.electroserver5.protocol.IOErrorHandler  - IOErrorHandler.exceptionCaught
java.nio.BufferOverflowException
	at java.nio.Buffer.nextPutIndex(Buffer.java:501)
	at java.nio.HeapByteBuffer.putLong(HeapByteBuffer.java:398)
	at com.electrotank.electroserver5.protocol.etio.AbstractByteBufferMessageWriter.writeLong(AbstractByteBufferMessageWriter.java:116)
	at com.electrotank.electroserver5.protocol.AbstractMessageWriter.writeLong(AbstractMessageWriter.java:36)
	at com.electrotank.electroserver5.protocol.codecs.internal.GatewayKickUserRequestCodec.encode(GatewayKickUserRequestCodec.java:24)
	at com.electrotank.electroserver5.protocol.codecs.external.PrefixingMessageCodecDecorator.encode(PrefixingMessageCodecDecorator.java:55)
	at com.electrotank.electroserver5.protocol.Codec.encode(Codec.java:37)
	at com.electrotank.electroserver5.protocol.etio.tcp.TCPEncoder.onMessage(TCPEncoder.java:64)
	at com.electrotank.electroserver5.protocol.etio.tcp.TCPEncoder.onMessage(TCPEncoder.java:45)
	at org.jetlang.channels.MemoryChannel.publish(MemoryChannel.java:27)
	at com.electrotank.electroserver5.protocol.etio.EtIOClientSession.writeAndCloseConnection(EtIOClientSession.java:116)
	at com.electrotank.electroserver5.entities.DefaultGatewayClient.sendAndClose(DefaultGatewayClient.java:30)
	at com.electrotank.electroserver5.servers.gateway.transactions.GatewayKickUserRequestTransaction.execute(GatewayKickUserRequestTransaction.java:35)
	at com.electrotank.electroserver5.servers.gateway.transactions.GatewayKickUserRequestTransaction.execute(GatewayKickUserRequestTransaction.java:13)
	at com.electrotank.electroserver5.servers.gateway.transactions.TransactionHandler.processMessage(TransactionHandler.java:33)
	at com.electrotank.electroserver5.servers.gateway.GatewayEndpoint.receive(GatewayEndpoint.java:63)
	at com.electrotank.electroserver5.servers.registry.RegistryEndpoint.sendMessageToGateway(RegistryEndpoint.java:241)
	at com.electrotank.electroserver5.servers.registry.RegistryEndpoint.sendMessageToClient(RegistryEndpoint.java:249)
	at com.electrotank.electroserver5.entities.ConnectedUser.kick(ConnectedUser.java:160)
	at com.electrotank.electroserver5.entities.ConnectedUser.kick(ConnectedUser.java:166)
	at com.electrotank.electroserver5.servers.registry.IdleMonitor.run(IdleMonitor.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)
2011-Oct-10 20:02:11:218 [openclose] ERROR com.electrotank.electroserver5.servers.registry.RegistryEndpoint  - GC overhead limit exceeded
2011-Oct-10 20:02:12:892 [openclose] ERROR com.electrotank.electroserver5.DefaultExceptionHandler  - Uncaught exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
2011-Oct-10 20:02:17:853 [btpool0-1 - Acceptor0 SelectChannelConnector @ <ip>:8080] ERROR com.electrotank.electroserver5.DefaultExceptionHandler  - Uncaught exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
2011-Oct-10 21:35:11:359 [worker-14 BinaryTCP-0] ERROR com.electrotank.electroserver5.DefaultExceptionHandler  - Uncaught exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
2011-Oct-10 21:35:15:088 [pool-1-thread-1] ERROR com.electrotank.electroserver5.DefaultExceptionHandler  - Uncaught exception

[vai]

There is one more thing which I've noticed - every time I restart the server, in console.log, I see a message like "Low memory!! -
90.99
%" ..

[tcarr]

"GC overhead limit exceeded" means that your Garbage Collector is complaining that your application is taking up more memory than it is comfortable with. You should increase your ES5's heap space, particularly since this is happening right after a restart (so it's not just a case of a memory leak that needs to be tracked down). See Adjusting ElectroServer's Memory. Please note that if you run ES5 using SAFEMODE then the memory settings from the ES Admin are ignored - if that is the case, let me know and I'll tell you how to set them in your start script.

[vai]

Just to make it clear, I get the low memory notification in console.log every time I restart the server. The GC overhead limit exceeded happened just once yesterday and I must have hit the server 200,000 times yesterday by the time I got the gc overhead error.

[vai]

So, I changed the max heap size to 264MB and es5 would not start. Then I used java -jar lib/ElectroServer5-bootstrap.jar -safemode -mode StandAlone -config config/Configuration.xml to start the server in safe mode but still it does not start the server. In the log file, I can see that my plugins were loaded but nothing after that. Normally, I would see something like -

2011-Oct-11 06:07:55:703 [main] INFO  com.electrotank.electroserver5.servers.gateway.GatewayServer  - GatewayServer.doStart
2011-Oct-11 06:07:55:760 [main] INFO  com.electrotank.electroserver5.protocol.IOProvider  - Opened BinaryTCP listener on

[tcarr]

When ES5 doesn't restart correctly, sometimes it's because it didn't shut down correctly. Try killing ES5 (for Windows, check on all java processes in Task Manager; for Linux check "ps -ef | grep java"), then starting. You should also check your hard drive space.

Hmmm... another thing to check is "all your plugins". Do you have any server level components that don't have a logging line at the end of their init method? ES5 might be hung up initializing one of those server level components, particularly if there isn't enough memory for everything and the OS starts thrashing.

[vai]

there is no es5 process running .. I will add a debug statement at the end of init() of all of my plugins.. do you know how much memory would be availalbe in safe mode ?

[tcarr]

Safe mode will give you whatever your JVM default memory is. This varies with version of JVM, and if you have customized the default JVM heap size. If you look at your old logs from when ES5 did start successfully, you will see two sets of the startup info in the logs, before it starts mentioning your various plugins and gateway listeners. The first one will give you the JVM default heap size, the second will give you the heap size set in ES Admin.

It's entirely possible that 264MB is still too small for all your extensions. Many ES5 applications would have no problem fitting in that size, but assuming your server has plenty of memory, you might need to try 512 MB or 1024, then check the ES Admin home screen to see how much you actually use.

[vai]

I was using 64MB previously and all of my extensions would load properly. Today I changed it to 264MB and after that i could not start the server. can i specify the heap size in an startup script like you mentioned before?

[tcarr]

Your problem is likely that your OS doesn't *have* 264MB available for the JVM, so that the bootstrapper can't get the ES5 main process started.

What happens when you start using SAFEMODE? Do you get any error messages? SAFEMODE is supposed to allow you to start ES5 enough that you can get to the ES Admin to undo changes to the heap size or command line parameters that are preventing ES5 from starting.

If you tell me your OS I can give you a startup script or batch file. If you are running ES5 as a service I need to know that.