Java Memory Dump

1. Using JVM option, Automatically when OutOfMemory happens.
    -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/esDump.hprof
    if dump path is not specified, usually it will be in java.io.tmpdir. /tmp as a hint.

2. Using jmap  tools shipped with JDK. If you did not run out of file descriptors and still be able to connect to jvm.
    jmap -dump:live,format=b,file=/tmp/heap0409.bin 13419
    if live is not specified, entire memory will be dumped and that can be big and slow.

3. Using Linux gdb
    3.1 view memory map of a process
        cat /proc/{$pid}/maps
    3.2 gdb attach {$pid}
    3.3 issue dump memory command in gdb. need to removed - and add 0x before the memory ranged displayed in 3.1.
        dump memory /tmp/memdump 0x02079000 0x020b6000

        dump memory /tmp/memdump2 0x020b6000 0x02171000

4. Using JDK tools such JVisualVM, jconsole, java mission control etc. Server side accessibility need to be enabled

    4.1 enable server to be accessible

        4.1.1 jmx

        -Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname={$ip to provide service} -Dcom.sun.management.jmxremote.rmi.port=9091

        4.1.2 jstatd

                #enable remote debug via jstatd 

                #Create jstatd.policy. put it in /root

grant codebase "file:/usr/lib/jvm/jdk1.8.0_191/lib/tools.jar" {

    permission java.security.AllPermission;

};

               #Start stated

              jstatd -p 9090 -J-Djava.security.policy=~jstatd.policy

              jstatd -p 9090 -J-Djava.security.policy=/root/jstat.policy -J-Djava.rmi.server.hostname={$ip as remote host}

    4.2 dump memory or thread in the tool

    4.3 view and analyze with same tool

Other useful tools usually used together

1. view tcp memory

#tcp_mem etc

cat /proc/sys/net/ipv4/tcp_mem

cat /proc/net/sockstat

2. lsof in a process

#found evidence of opening too many files.

#list open files

lsof -p $pid

lsof -a -i4 -i6 -itcp -p $pid

3. netstat for network and connections

netstat -a -n | grep tcp | grep 9200 

#view connection count from hosts

#check ip addresses 

netstat -nat | awk '{print $5}'| awk -F":" '{print $1}' | sort | uniq -c

#further on specific port

netstat -nat | grep 9200 |awk '{print $5}'| awk -F":" '{print $1}' | sort | uniq -c

4. network tuning under /proc/sys/net

#putting them all together. To make socket close quicker and fail quicker

echo 20 > /proc/sys/net/ipv4/tcp_fin_timeout

echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl

echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes

cat /proc/sys/net/ipv4/tcp_fin_timeout

cat /proc/sys/net/ipv4/tcp_keepalive_intvl

cat /proc/sys/net/ipv4/tcp_keepalive_probes

5. turn on and off the swap file on linux

#fallocate -l 2G /swapfile

dd if=/dev/zero of=/swapfile bs=1024 count=2097152

chmod 600 /swapfile

mkswap /swapfile

swapon /swapfile

##enlarge to 8 gb from 2gb

swapoff /swapfile

#dd always works

dd if=/dev/zero of=/swapfile bs=1M count=6144 oflag=append conv=notrunc

mkswap /swapfile

swapon /swapfile

6. Enable core dump: 

ulimit -c unlimited

7. use crontab to collect information periodicaly.

#run every 10 minutes

*/10 * * * * /home/ama/indexerCheck.sh