-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/esDump.hprof
if dump path is not specified, usually it will be in java.io.tmpdir. /tmp as a hint.
2. Using jmap tools shipped with JDK. If you did not run out of file descriptors and still be able to connect to jvm.
jmap -dump:live,format=b,file=/tmp/heap0409.bin 13419
if live is not specified, entire memory will be dumped and that can be big and slow.
3. Using Linux gdb
3.1 view memory map of a process
cat /proc/{$pid}/maps
3.2 gdb attach {$pid}
3.3 issue dump memory command in gdb. need to removed - and add 0x before the memory ranged displayed in 3.1.
dump memory /tmp/memdump 0x02079000 0x020b6000
dump memory /tmp/memdump2 0x020b6000 0x02171000
4. Using JDK tools such JVisualVM, jconsole, java mission control etc. Server side accessibility need to be enabled
4.1 enable server to be accessible
4.1.1 jmx
-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=9090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname={$ip to provide service} -Dcom.sun.management.jmxremote.rmi.port=9091
4.1.2 jstatd
#enable remote debug via jstatd
#Create jstatd.policy. put it in /root
grant codebase "file:/usr/lib/jvm/jdk1.8.0_191/lib/tools.jar" {
permission java.security.AllPermission;
};
#Start stated
jstatd -p 9090 -J-Djava.security.policy=~jstatd.policy
jstatd -p 9090 -J-Djava.security.policy=/root/jstat.policy -J-Djava.rmi.server.hostname={$ip as remote host}
4.2 dump memory or thread in the tool
4.3 view and analyze with same tool
Other useful tools usually used together
1. view tcp memory
#tcp_mem etc
cat /proc/sys/net/ipv4/tcp_mem
cat /proc/net/sockstat
2. lsof in a process
#found evidence of opening too many files.
#list open files
lsof -p $pid
lsof -a -i4 -i6 -itcp -p $pid
3. netstat for network and connections
netstat -a -n | grep tcp | grep 9200
#view connection count from hosts
#check ip addresses
netstat -nat | awk '{print $5}'| awk -F":" '{print $1}' | sort | uniq -c
#further on specific port
netstat -nat | grep 9200 |awk '{print $5}'| awk -F":" '{print $1}' | sort | uniq -c
4. network tuning under /proc/sys/net
#putting them all together. To make socket close quicker and fail quicker
echo 20 > /proc/sys/net/ipv4/tcp_fin_timeout
echo 30 > /proc/sys/net/ipv4/tcp_keepalive_intvl
echo 5 > /proc/sys/net/ipv4/tcp_keepalive_probes
cat /proc/sys/net/ipv4/tcp_fin_timeout
cat /proc/sys/net/ipv4/tcp_keepalive_intvl
cat /proc/sys/net/ipv4/tcp_keepalive_probes
5. turn on and off the swap file on linux
#fallocate -l 2G /swapfile
dd if=/dev/zero of=/swapfile bs=1024 count=2097152
chmod 600 /swapfile
mkswap /swapfile
swapon /swapfile
##enlarge to 8 gb from 2gb
swapoff /swapfile
#dd always works
dd if=/dev/zero of=/swapfile bs=1M count=6144 oflag=append conv=notrunc
mkswap /swapfile
swapon /swapfile
6. Enable core dump:
ulimit -c unlimited
7. use crontab to collect information periodicaly.
#run every 10 minutes
*/10 * * * * /home/ama/indexerCheck.sh
