Thursday, April 11, 2019

Java Memory Dump

1. Using JVM option, Automatically when OutOfMemory happens.
    -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/esDump.hprof
    if dump path is not specified, usually it will be in java.io.tmpdir. /tmp as a hint.

2. Using jmap  tools shipped with JDK. If you did not run out of file descriptors and still be able to connect to jvm.
    jmap -dump:live,format=b,file=/tmp/heap0409.bin 13419
    if live is not specified, entire memory will be dumped and that can be big and slow.

3. Using Linux gdb
    3.1 view memory map of a process
        cat /proc/{$pid}/maps
    3.2 gdb attach {$pid}
    3.3 issue dump memory command in gdb. need to removed - and add 0x before the memory ranged displayed in 3.1.
        dump memory /tmp/memdump 0x02079000 0x020b6000

        dump memory /tmp/memdump2 0x020b6000 0x02171000
4. Using JDK tools such JVisualVM, jconsole, java mission control etc. Server side accessibility need to be enabled
    4.1 enable server to be accessible
        4.1.1 jmx
        -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname={$ip to provide service} -Dcom.sun.management.jmxremote.rmi.port=9091
        4.1.2 jstatd
                #enable remote debug via jstatd 

                #Create jstatd.policy. put it in /root
grant codebase "file:/usr/lib/jvm/jdk1.8.0_191/lib/tools.jar" {
    permission java.security.AllPermission;
};
               #Start stated
              jstatd -p 9090 -J-Djava.security.policy=~jstatd.policy
              jstatd -p 9090 -J-Djava.security.policy=/root/jstat.policy -J-Djava.rmi.server.hostname={$ip as remote host}
    4.2 dump memory or thread in the tool

    4.3 view and analyze with same tool

Other useful tools usually used together

1. view tcp memory
#tcp_mem etc
cat /proc/sys/net/ipv4/tcp_mem
cat /proc/net/sockstat

2. lsof in a process
#found evidence of opening too many files.

#list open files
lsof -p $pid
lsof -a -i4 -i6 -itcp -p $pid

3. netstat for network and connections
netstat -a -n | grep tcp | grep 9200 
#view connection count from hosts
#check ip addresses 
netstat -nat | awk '{print $5}'| awk -F":" '{print $1}' | sort | uniq -c
#further on specific port
netstat -nat | grep 9200 |awk '{print $5}'| awk -F":" '{print $1}' | sort | uniq -c

4. network tuning under /proc/sys/net

#putting them all together. To make socket close quicker and fail quicker
echo 20 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
cat /proc/sys/net/ipv4/tcp_fin_timeout
cat /proc/sys/net/ipv4/tcp_keepalive_intvl
cat /proc/sys/net/ipv4/tcp_keepalive_probes

5. turn on and off the swap file on linux
#fallocate -l 2G /swapfile
dd if=/dev/zero of=/swapfile bs=1024 count=2097152
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
##enlarge to 8 gb from 2gb
swapoff /swapfile
#dd always works
dd if=/dev/zero of=/swapfile bs=1M count=6144 oflag=append conv=notrunc
mkswap /swapfile
swapon /swapfile

6. Enable core dump: 

ulimit -c unlimited

7. use crontab to collect information periodicaly.
#run every 10 minutes
*/10 * * * * /home/ama/indexerCheck.sh

No comments: