Java Monitoring on a VPC
Vimal Shah
vimals at sokikom.com
Thu Oct 10 09:32:44 MST 2013
Thank you Joseph. I will be attempting that very soon.
I'd like to get jmap and jhat working also. I seem to be close to getting
jmap working but get a VMVersionMismatchException:
$ sudo ./jmap 19326
Attaching to process ID 19326, please wait...
Warning: the type "const char*" (declared in the remote VM in
VMStructs::localHotSpotVMTypes) had its size declared as 8 twice.
Continuing.
Warning: the type "const size_t" (declared in the remote VM in
VMStructs::localHotSpotVMTypes) had its size declared as 8 twice.
Continuing.
Warning: the type "const int" (declared in the remote VM in
VMStructs::localHotSpotVMTypes) had its size declared as 4 twice.
Continuing.
Warning: the type "const MemRegion" (declared in the remote VM in
VMStructs::localHotSpotVMTypes) had its size declared as 16 twice.
Continuing.
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at sun.tools.jmap.JMap.runTool(JMap.java:197)
at sun.tools.jmap.JMap.main(JMap.java:128)
Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported
versions are 23.21-b01. Target VM is 11.0-b16
at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:234)
at sun.jvm.hotspot.runtime.VM.<init>(VM.java:297)
at sun.jvm.hotspot.runtime.VM.initialize(VM.java:367)
at
sun.jvm.hotspot.bugspot.BugSpotAgent.setupVM(BugSpotAgent.java:598)
at sun.jvm.hotspot.bugspot.BugSpotAgent.go(BugSpotAgent.java:493)
at
sun.jvm.hotspot.bugspot.BugSpotAgent.attach(BugSpotAgent.java:331)
at sun.jvm.hotspot.tools.Tool.start(Tool.java:163)
at sun.jvm.hotspot.tools.PMap.main(PMap.java:67)
... 6 more
I've also tried:
$ sudo jmap -J-d64 -heap 19326
Same error.
$ java -version
java version "1.7.0_21"
Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
Seems that the hot-spot is the right build for the supported version. Am I
not executing it from the correct location? jmap is being executed from:
/usr/lib/jvm/java-7-oracle/bin
On Wed, Oct 9, 2013 at 6:04 PM, Joseph Sinclair
<plug-discussion at stcaz.net>wrote:
> No need for gnome, ssh in with forwarding for a reasonable port (e.g.
> 27635) and use the JMX remote support (built into the JVM, nothing for you
> to write), you probably need to restart the server with the appropriate
> flags for this to work[1].
> The run jVisualVM on your local box connecting to the local port you
> forwarded to the remote machine's port.
>
> [1] A possible set of options (note, this is NOT secure, reenable the
> security options before using in production!) is:
> -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote.port=27635
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Djava.rmi.server.hostname=the.public.ip
> The last option might be better set to the loopback address since you're
> using port forwarding to pull that via ssh back to your local box, but
> you'll have to experiment to see what works...
>
> Note also that JMX allows you to modify the running JVM as well as
> monitor, so make CERTAIN you secure the port and interfaces before using
> this on a public server, as any port knocking attack would open your entire
> process to arbitrary malicious interference.
>
> On 10/09/2013 05:47 PM, Vimal Shah wrote:
> > That makes sense and the script is writing itself! It seems that jmap and
> > jhat would be good to:
> >
> http://www.marcsturlese.com/2009/05/09/analyzing-java-heaps-with-jmap-and-jhat/
> >
> > To get jvisualvm working, I'm assuming I need to put Gnome on the server
> > and then use putty with X-windows and forwarding enabled? If this is
> > correct, are there performance ramifications to adding Gnome? If not, any
> > references to how to do this?
> >
> >
> > On Wed, Oct 9, 2013 at 4:19 PM, James Mcphee <jmcphe at gmail.com> wrote:
> >
> >> You can plot a graph during these times to see if it's spending a lot of
> >> time in garbage collection. Other options are to take jstack thread
> dumps
> >> and see what the app is up to at that time.
> >>
> >> My favorite way to fiddle is to plug VisualVM (replacement for jconsole)
> >> into the jvm and watch when issues are occuring. That uses JMX and you
> can
> >> monitor threads and mbeans.
> >>
> >>
> >> On Wed, Oct 9, 2013 at 4:13 PM, Vimal Shah <vimals at sokikom.com> wrote:
> >>
> >>> jstat works nicely now James. Thank you. I will try the various options
> >>> as well.
> >>>
> >>> A little more about the problem. Every 4-6 weeks the CPU util looks
> like:
> >>>
> >>> [image: Inline image 1]
> >>>
> >>>
> >>> The application slows and Alerts are emailed. The application is
> >>> restarted (in this case on 10/2 at 1:00). This allows the application
> to
> >>> return to normal responsiveness and allows the box itself to go back
> to 3%
> >>> CPU util. The ideal tool would be one that would help us understand
> this.
> >>> Can jstat be used to investigate this? Would a call trace and heap
> dump be
> >>> more useful as identification to the function that was called when this
> >>> error occurred?
> >>>
> >>>
> >>>
> >>> On Wed, Oct 9, 2013 at 3:25 PM, James Mcphee <jmcphe at gmail.com> wrote:
> >>>
> >>>> For jstat, you have to be a user with permission to the process. In
> >>>> this case, either the user the java process is running as, or root.
> I use
> >>>> jstat when I want to script things to watch them on the console.
> >>>>
> >>>> For JMX, there are jvm arguments you can use to open it up to external
> >>>> hosts, provide authentication, etc. That will depend on the server
> you're
> >>>> running and you should consult the documentation for that server.
> >>>>
> >>>>
> >>>> On Wed, Oct 9, 2013 at 3:19 PM, Vimal Shah <vimals at sokikom.com>
> wrote:
> >>>>
> >>>>> James,
> >>>>>
> >>>>> What other details can I provide?
> >>>>>
> >>>>> It does sound like JMX is the standard. When running jstat with my
> JAVA
> >>>>> PID
> >>>>>
> >>>>> $ jstat -gc 19326
> >>>>> Could not attach to 19326
> >>>>>
> >>>>> I don't have JMX configured. After some reading it seems that I need
> to
> >>>>> get a Mbeans server running. Is this what I have to do?
> >>>>>
> >>>>> http://docs.oracle.com/javase/tutorial/jmx/mbeans/standard.html
> >>>>>
> >>>>>
> >>>>> On Wed, Oct 9, 2013 at 1:15 PM, James Mcphee <jmcphe at gmail.com>
> wrote:
> >>>>>
> >>>>>> We'd need more details to give you much more than "Using JMX or
> jstat
> >>>>>> would be easiest"
> >>>>>>
> >>>>>> Example:
> >>>>>> ./jstat -gc 27104
> >>>>>> S0C S1C S0U S1U EC EU OC OU
> >>>>>> PC PU YGC YGCT FGC FGCT GCT
> >>>>>> 20928.0 21312.0 0.0 4864.0 305728.0 128383.7 262656.0 223537.3
> >>>>>> 170240.0 130738.6 295 6.912 10 8.268 15.180
> >>>>>>
> >>>>>> http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstat.html
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Oct 9, 2013 at 1:05 PM, Vimal Shah <vimals at sokikom.com>
> wrote:
> >>>>>>
> >>>>>>> Hello,
> >>>>>>>
> >>>>>>> Does anyone have a step by step guide to getting monitoring (heap
> >>>>>>> stack) set up of a Java app on an Amazon VPC box? The application
> is
> >>>>>>> smartfoxserver and it's on Ubuntu 12.04.
> >>>>>>>
> >>>>>>> I've scratched the surface on the following:
> >>>>>>>
> >>>>>>> - https://code.google.com/p/javamelody/
> >>>>>>> -
> >>>>>>>
> http://docs.oracle.com/javase/6/docs/technotes/guides/management/jconsole.html
> >>>>>>> - http://jamonapi.sourceforge.net/
> >>>>>>>
> >>>>>>>
> >>>>>>> Application code changes aren't within my reach at the moment. An
> >>>>>>> apt-get install of a monitoring tool on the server under question
> would be
> >>>>>>> ideal. If this tool can have low overhead even better.
> >>>>>>>
> >>>>>>> Thank you all in advance.
> >>>>>>>
> >>>>>>> --
> >>>>>>> Vimal (rhymes with Kimmel) Shah
> >>>>>>> Front-End / Infrastructure Engineer
> >>>>>>> Sokikom
> >>>>>>> Mobile: (480) 752-9269
> >>>>>>> Email: vimals at sokikom.com
> >>>>>>> Web: www.sokikom.com
> >>>>>>>
> >>>>>>> Follow us: twitter.com/sokikom <http://www.twitter.com/sokikom>
> >>>>>>> Like us: facebook.com/sokikom <http://www.facebook.com/sokikom>
> >>>>>>>
> >>>>>>> ---------------------------------------------------
> >>>>>>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> >>>>>>> To subscribe, unsubscribe, or to change your mail settings:
> >>>>>>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> James McPhee
> >>>>>> jmcphe at gmail.com
> >>>>>>
> >>>>>> ---------------------------------------------------
> >>>>>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> >>>>>> To subscribe, unsubscribe, or to change your mail settings:
> >>>>>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
> >>>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Vimal (rhymes with Kimmel) Shah
> >>>>> Front-End / Infrastructure Engineer
> >>>>> Sokikom
> >>>>> Mobile: (480) 752-9269
> >>>>> Email: vimals at sokikom.com
> >>>>> Web: www.sokikom.com
> >>>>>
> >>>>> Follow us: twitter.com/sokikom <http://www.twitter.com/sokikom>
> >>>>> Like us: facebook.com/sokikom <http://www.facebook.com/sokikom>
> >>>>>
> >>>>> ---------------------------------------------------
> >>>>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> >>>>> To subscribe, unsubscribe, or to change your mail settings:
> >>>>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> James McPhee
> >>>> jmcphe at gmail.com
> >>>>
> >>>> ---------------------------------------------------
> >>>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> >>>> To subscribe, unsubscribe, or to change your mail settings:
> >>>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Vimal (rhymes with Kimmel) Shah
> >>> Front-End / Infrastructure Engineer
> >>> Sokikom
> >>> Mobile: (480) 752-9269
> >>> Email: vimals at sokikom.com
> >>> Web: www.sokikom.com
> >>>
> >>> Follow us: twitter.com/sokikom <http://www.twitter.com/sokikom>
> >>> Like us: facebook.com/sokikom <http://www.facebook.com/sokikom>
> >>>
> >>> ---------------------------------------------------
> >>> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> >>> To subscribe, unsubscribe, or to change your mail settings:
> >>> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
> >>>
> >>
> >>
> >>
> >> --
> >> James McPhee
> >> jmcphe at gmail.com
> >>
> >> ---------------------------------------------------
> >> PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> >> To subscribe, unsubscribe, or to change your mail settings:
> >> http://lists.phxlinux.org/mailman/listinfo/plug-discuss
> >>
> >
> >
> >
> >
> >
> > ---------------------------------------------------
> > PLUG-discuss mailing list - PLUG-discuss at lists.phxlinux.org
> > To subscribe, unsubscribe, or to change your mail settings:
> > http://lists.phxlinux.org/mailman/listinfo/plug-discuss
> >
>
>
--
Vimal (rhymes with Kimmel) Shah
Front-End / Infrastructure Engineer
Sokikom
Mobile: (480) 752-9269
Email: vimals at sokikom.com
Web: www.sokikom.com
Follow us: twitter.com/sokikom <http://www.twitter.com/sokikom>
Like us: facebook.com/sokikom <http://www.facebook.com/sokikom>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.phxlinux.org/pipermail/plug-discuss/attachments/20131010/60a97e22/attachment.html>
More information about the PLUG-discuss
mailing list