You can plot a graph during these times to see if it's spending a lot of time in garbage collection.  Other options are to take jstack thread dumps and see what the app is up to at that time.

My favorite way to fiddle is to plug VisualVM (replacement for jconsole) into the jvm and watch when issues are occuring.  That uses JMX and you can monitor threads and mbeans.


On Wed, Oct 9, 2013 at 4:13 PM, Vimal Shah <vimals@sokikom.com> wrote:
jstat works nicely now James. Thank you. I will try the various options as well.

A little more about the problem. Every 4-6 weeks the CPU util looks like:

Inline image 1


The application slows and Alerts are emailed. The application is restarted (in this case on 10/2 at 1:00). This allows the application to return to normal responsiveness and allows the box itself to go back to 3% CPU util. The ideal tool would be one that would help us understand this. Can jstat be used to investigate this? Would a call trace and heap dump be more useful as identification to the function that was called when this error occurred?



On Wed, Oct 9, 2013 at 3:25 PM, James Mcphee <jmcphe@gmail.com> wrote:
For jstat, you have to be a user with permission to the process.  In this case, either the user the java process is running as, or root.  I use jstat when I want to script things to watch them on the console.

For JMX, there are jvm arguments you can use to open it up to external hosts, provide authentication, etc.  That will depend on the server you're running and you should consult the documentation for that server.


On Wed, Oct 9, 2013 at 3:19 PM, Vimal Shah <vimals@sokikom.com> wrote:
James,

What other details can I provide?

It does sound like JMX is the standard. When running jstat with my JAVA PID 

$ jstat -gc 19326
Could not attach to 19326

I don't have JMX configured. After some reading it seems that I need to get a Mbeans server running. Is this what I have to do?



On Wed, Oct 9, 2013 at 1:15 PM, James Mcphee <jmcphe@gmail.com> wrote:
We'd need more details to give you much more than "Using JMX or jstat would be easiest"

Example:
./jstat -gc 27104
 S0C    S1C    S0U    S1U      EC       EU        OC         OU       PC     PU    YGC     YGCT    FGC    FGCT     GCT   
20928.0 21312.0  0.0   4864.0 305728.0 128383.7  262656.0   223537.3  170240.0 130738.6    295    6.912  10      8.268   15.180



On Wed, Oct 9, 2013 at 1:05 PM, Vimal Shah <vimals@sokikom.com> wrote:
Hello,

Does anyone have a step by step guide to getting monitoring (heap stack) set up of a Java app on an Amazon VPC box? The application is smartfoxserver and it's on Ubuntu 12.04.

I've scratched the surface on the following:
Application code changes aren't within my reach at the moment. An apt-get install of a monitoring tool on the server under question would be ideal. If this tool can have low overhead even better.

Thank you all in advance.

--
Vimal (rhymes with Kimmel) Shah
Front-End / Infrastructure Engineer
Sokikom
Mobile: (480) 752-9269
Email:   vimals@sokikom.com
Web:    www.sokikom.com

---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
http://lists.phxlinux.org/mailman/listinfo/plug-discuss



--
James McPhee
jmcphe@gmail.com

---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
http://lists.phxlinux.org/mailman/listinfo/plug-discuss



--
Vimal (rhymes with Kimmel) Shah
Front-End / Infrastructure Engineer
Sokikom
Mobile: (480) 752-9269
Email:   vimals@sokikom.com
Web:    www.sokikom.com

---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
http://lists.phxlinux.org/mailman/listinfo/plug-discuss



--
James McPhee
jmcphe@gmail.com

---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
http://lists.phxlinux.org/mailman/listinfo/plug-discuss



--
Vimal (rhymes with Kimmel) Shah
Front-End / Infrastructure Engineer
Sokikom
Mobile: (480) 752-9269
Email:   vimals@sokikom.com
Web:    www.sokikom.com

---------------------------------------------------
PLUG-discuss mailing list - PLUG-discuss@lists.phxlinux.org
To subscribe, unsubscribe, or to change your mail settings:
http://lists.phxlinux.org/mailman/listinfo/plug-discuss



--
James McPhee
jmcphe@gmail.com