Java Monitoring on a VPC

Joseph Sinclair plug-discussion at stcaz.net
Wed Oct 9 18:04:40 MST 2013 

No need for gnome, ssh in with forwarding for a reasonable port (e.g. 27635) and use the JMX remote support (built into the JVM, nothing for you to write), you probably need to restart the server with the appropriate flags for this to work[1].
The run jVisualVM on your local box connecting to the local port you forwarded to the remote machine's port.

[1] A possible set of options (note, this is NOT secure, reenable the security options before using in production!) is:
-Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=27635
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Djava.rmi.server.hostname=the.public.ip
The last option might be better set to the loopback address since you're using port forwarding to pull that via ssh back to your local box, but you'll have to experiment to see what works...

Note also that JMX allows you to modify the running JVM as well as monitor, so make CERTAIN you secure the port and interfaces before using this on a public server, as any port knocking attack would open your entire process to arbitrary malicious interference.

On 10/09/2013 05:47 PM, Vimal Shah wrote:
> That makes sense and the script is writing itself! It seems that jmap and
> jhat would be good to:
> http://www.marcsturlese.com/2009/05/09/analyzing-java-heaps-with-jmap-and-jhat/
> 
> To get jvisualvm working, I'm assuming I need to put Gnome on the server
> and then use putty with X-windows and forwarding enabled? If this is
> correct, are there performance ramifications to adding Gnome? If not, any
> references to how to do this?
> 
> 
> On Wed, Oct 9, 2013 at 4:19 PM, James Mcphee <jmcphe at gmail.com> wrote:
> 
>> You can plot a graph during these times to see if it's spending a lot of
>> time in garbage collection.  Other options are to take jstack thread dumps
>> and see what the app is up to at that time.
>>
>> My favorite way to fiddle is to plug VisualVM (replacement for jconsole)
>> into the jvm and watch when issues are occuring.  That uses JMX and you can
>> monitor threads and mbeans.
>>
>>
>> On Wed, Oct 9, 2013 at 4:13 PM, Vimal Shah <vimals at sokikom.com> wrote:
>>
>>> jstat works nicely now James. Thank you. I will try the various options
>>> as well.
>>>
>>> A little more about the problem. Every 4-6 weeks the CPU util looks like:
>>>
>>> [image: Inline image 1]
>>>
>>>
>>> The application slows and Alerts are emailed. The application is
>>> restarted (in this case on 10/2 at 1:00). This allows the application to
>>> return to normal responsiveness and allows the box itself to go back to 3%
>>> CPU util. The ideal tool would be one that would help us understand this.
>>> Can jstat be used to investigate this? Would a call trace and heap dump be
>>> more useful as identification to the function that was called when this
>>> error occurred?
>>>
>>>
>>>
>>> On Wed, Oct 9, 2013 at 3:25 PM, James Mcphee <jmcphe at gmail.com> wrote:
>>>
>>>> For jstat, you have to be a user with permission to the process.  In
>>>> this case, either the user the java process is running as, or root.  I use
>>>> jstat when I want to script things to watch them on the console.
>>>>
>>>> For JMX, there are jvm arguments you can use to open it up to external
>>>> hosts, provide authentication, etc.  That will depend on the server you're
>>>> running and you should consult the documentation for that server.
>>>>
>>>>
>>>> On Wed, Oct 9, 2013 at 3:19 PM, Vimal Shah <vimals at sokikom.com> wrote:
>>>>
>>>>> James,
>>>>>
>>>>> What other details can I provide?
>>>>>
>>>>> It does sound like JMX is the standard. When running jstat with my JAVA
>>>>> PID
>>>>>
>>>>> $ jstat -gc 19326
>>>>> Could not attach to 19326
>>>>>
>>>>> I don't have JMX configured. After some reading it seems that I need to
>>>>> get a Mbeans server running. Is this what I have to do?
>>>>>
>>>>> http://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html
>>>>>
>>>>>
>>>>> On Wed, Oct 9, 2013 at 1:15 PM, James Mcphee <jmcphe at gmail.com> wrote:
>>>>>
>>>>>> We'd need more details to give you much more than "Using JMX or jstat
>>>>>> would be easiest"
>>>>>>
>>>>>> Example:
>>>>>> ./jstat -gc 27104
>>>>>>  S0C    S1C    S0U    S1U      EC       EU        OC         OU
>>>>>> PC     PU    YGC     YGCT    FGC    FGCT     GCT
>>>>>> 20928.0 21312.0  0.0   4864.0 305728.0 128383.7  262656.0   223537.3
>>>>>>  170240.0 130738.6    295    6.912  10      8.268   15.180
>>>>>>
>>>>>> http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstat.html
>>>>>>
>>>>>>
>>>>>> On Wed, Oct 9, 2013 at 1:05 PM, Vimal Shah <vimals at sokikom.com> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> Does anyone have a step by step guide to getting monitoring (heap
>>>>>>> stack) set up of a Java app on an Amazon VPC box? The application is
>>>>>>> smartfoxserver and it's on Ubuntu 12.04.
>>>>>>>
>>>>>>> I've scratched the surface on the following:
>>>>>>>
>>>>>>>    - https://code.google.com/p/javamelody/
>>>>>>>    -
>>>>>>>    http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html
>>>>>>>    - http://jamonapi.sourceforge.net/
>>>>>>>
>>>>>>>
>>>>>>> Application code changes aren't within my reach at the moment. An
>>>>>>> apt-get install of a monitoring tool on the server under question would be
>>>>>>> ideal. If this tool can have low overhead even better.
>>>>>>>
>>>>>>> Thank you all in advance.
>>>>>>>
>>>>>>> --
>>>>>>> Vimal (rhymes with Kimmel) Shah
>>>>>>> Front-End / Infrastructure Engineer
>>>>>>> Sokikom
>>>>>>> Mobile: (480) 752-9269
>>>>>>> Email:   vimals at sokikom.com
>>>>>>> Web:    www.sokikom.com
>>>>>>>
>>>>>>> Follow us: twitter.com/sokikom <http://www.twitter.com/sokikom>
>>>>>>> Like us: facebook.com/sokikom <http://www.facebook.com/sokikom>
>>>>>>>
>>>>>>> ---------------------------------------------------
>>>>>>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
>>>>>>> To subscribe, unsubscribe, or to change your mail settings:
>>>>>>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> James McPhee
>>>>>> jmcphe at gmail.com
>>>>>>
>>>>>> ---------------------------------------------------
>>>>>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
>>>>>> To subscribe, unsubscribe, or to change your mail settings:
>>>>>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Vimal (rhymes with Kimmel) Shah
>>>>> Front-End / Infrastructure Engineer
>>>>> Sokikom
>>>>> Mobile: (480) 752-9269
>>>>> Email:   vimals at sokikom.com
>>>>> Web:    www.sokikom.com
>>>>>
>>>>> Follow us: twitter.com/sokikom <http://www.twitter.com/sokikom>
>>>>> Like us: facebook.com/sokikom <http://www.facebook.com/sokikom>
>>>>>
>>>>> ---------------------------------------------------
>>>>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
>>>>> To subscribe, unsubscribe, or to change your mail settings:
>>>>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> James McPhee
>>>> jmcphe at gmail.com
>>>>
>>>> ---------------------------------------------------
>>>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
>>>> To subscribe, unsubscribe, or to change your mail settings:
>>>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>>>
>>>
>>>
>>>
>>> --
>>> Vimal (rhymes with Kimmel) Shah
>>> Front-End / Infrastructure Engineer
>>> Sokikom
>>> Mobile: (480) 752-9269
>>> Email:   vimals at sokikom.com
>>> Web:    www.sokikom.com
>>>
>>> Follow us: twitter.com/sokikom <http://www.twitter.com/sokikom>
>>> Like us: facebook.com/sokikom <http://www.facebook.com/sokikom>
>>>
>>> ---------------------------------------------------
>>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
>>> To subscribe, unsubscribe, or to change your mail settings:
>>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>>
>>
>>
>>
>> --
>> James McPhee
>> jmcphe at gmail.com
>>
>> ---------------------------------------------------
>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
>> To subscribe, unsubscribe, or to change your mail settings:
>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
>>
> 
> 
> 
> 
> 
> ---------------------------------------------------
> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> To subscribe, unsubscribe, or to change your mail settings:
> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
> 

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 490 bytes
Desc: OpenPGP digital signature
URL: <http://lists.phxlinux.org/pipermail/plug-discuss/attachments/20131009/33e15224/attachment.pgp>

More information about the PLUG-discuss mailing list