My solairs mchine is pretty slow after using it for 4-5 days

anjan_bhushan · 07-28-2003, 11:19 AM

We are testing an enterprise application based on java. I left the system running for 4 days. When I can back, and connected to the machine using telnet, all the response is pretty slow. It is taking 4-8 mintute to process even ps or ls---

1. What parametrs I should check
2. How to check if a PARTICULAR jvm has how many file handles
3. How to check the other java parametrs which may be causing it.

Some info
-----------
1. network is find and checked at switch everything is okay

following is output of df -k

Code:

[root-olive][root-olive]/opt/nc421/bmss/logs> df -k
Filesystem           1024-blocks        Used   Available Capacity  Mounted on
/dev/dsk/c0t0d0s0         770943      396000      320977    56%    /
/dev/dsk/c0t0d0s3        1590199      699357      843137    46%    /usr
/proc                          0           0           0     0%    /proc
fd                             0           0           0     0%    /dev/fd
mnttab                         0           0           0     0%    /etc/mnttab
/dev/dsk/c0t0d0s1         770943      146284      570693    21%    /var
swap                      460848          32      460816     1%    /var/run
/dev/dsk/c0t1d0s6       17648282    14579162     2892638    84%    /dsk2
/dev/dsk/c0t0d0s5       10325760     3460041     6762462    34%    /opt
/dev/dsk/c0t1d0s7       17635562    12989501     4469706    75%    /dsk3
/dev/dsk/c0t0d0s6        1018191      824456      132644    87%    /tmp
/dev/dsk/c0t0d0s7        1761235      796338      912060    47%    /export/home

other output

Quote:

[root-olive]/opt/nc421/bmss/logs> ps -eaf
UID PID PPID C STIME TTY TIME CMD
root 0 0 0 Jul 17 ? 0:00 sched
root 1 0 0 Jul 17 ? 0:00 /etc/init -
root 2 0 0 Jul 17 ? 0:00 pageout
root 3 0 0 Jul 17 ? 23:13 fsflush
root 443 1 0 Jul 17 ? 0:00 /usr/lib/saf/sac -t 300
root 225 1 0 Jul 17 ? 0:00 /usr/lib/power/powerd
root 451 1 0 Jul 17 ? 0:00 /usr/lib/sendmail -bd -q15m
root 292 1 0 Jul 17 ? 0:00 /usr/lib/snmp/snmpdx -y -c /etc/snmp/conf
root 453 161 0 Jul 17 ? 0:00 in.telnetd
root 53 1 0 Jul 17 ? 0:00 /usr/lib/sysevent/syseventd
root 173 1 0 Jul 17 ? 0:00 /usr/lib/nfs/lockd
root 212 1 0 Jul 17 ? 0:00 /usr/lib/lpsched
root 178 1 0 Jul 17 ? 0:00 /usr/lib/autofs/automountd
root 137 1 0 Jul 17 ? 0:00 /usr/sbin/rpcbind
root 161 1 0 Jul 17 ? 0:00 /usr/sbin/inetd -s
daemon 177 1 0 Jul 17 ? 0:00 /usr/lib/nfs/statd
root 193 1 0 Jul 17 ? 0:03 /usr/sbin/syslogd
root 455 453 0 Jul 17 pts/1 0:00 -ksh
root 196 1 0 Jul 17 ? 0:00 /usr/sbin/cron
root 205 1 0 Jul 17 ? 0:01 /usr/sbin/nscd
root 241 1 0 Jul 17 ? 0:00 /usr/sbin/vold
root 239 1 0 Jul 17 ? 0:00 /usr/sadm/lib/wbem/cimomboot start
root 237 1 0 Jul 17 ? 0:00 /usr/lib/utmpd
root 444 1 0 Jul 17 console 0:00 /usr/lib/saf/ttymon -g -h -p olive console login: -T sun -d /dev/console -l co
root 338 1 0 Jul 17 ? 0:00 /opt/CSCOnfc/bin/NFCD
root 282 1 0 Jul 17 ? 0:00 /usr/lib/nfs/mountd
root 283 1 0 Jul 17 ? 0:00 /usr/dt/bin/dtlogin -daemon
root 298 1 0 Jul 17 ? 0:00 /usr/lib/dmi/dmispd
root 286 1 0 Jul 17 ? 0:00 /usr/lib/nfs/nfsd -a 16
root 299 1 0 Jul 17 ? 0:00 /usr/lib/dmi/snmpXdmid -s olive
root 446 443 0 Jul 17 ? 0:00 /usr/lib/saf/ttymon
root 341 338 0 Jul 17 ? 0:00 NFCollector
root 440 1 0 Jul 17 ? 0:00 /usr/bin/perl /export/home/ftpuser/webmin-1.090/miniserv.pl /etc/webmin/miniser
root 342 338 0 Jul 17 ? 0:00 NFCGW
root 343 341 0 Jul 17 ? 0:00 NFCollector
root 344 292 0 Jul 17 ? 1:32 mibiisa -r -p 32786
aponetnc 846 843 0 Jul 17 ? 31:44 /opt/nc421/customer/jre/bin/java -Dapogee.reportserver=true -Xmx512M -Djava.lib
aponetnc 744 1 0 Jul 17 pts/1 1:39 /opt/nc421/objy/bin/ools
aponetnc 845 843 0 Jul 17 ? 28:43 /opt/nc421/customer/jre/bin/java -Dapogee.housekeepserver=true -Xmx256M -Djava.
aponetnc 843 842 0 Jul 17 ? 0:20 /opt/nc421/customer/jre/bin/java -DJMS_BROKER_URL=localhost:2506 -Xms64m -Xmx12
aponetnc 743 1 0 Jul 17 pts/1 0:18 /opt/nc421/objy/bin/ooams xyzzy 8
aponetnc 828 827 0 Jul 17 ? 0:57 /opt/nc421/customer/jre/bin/java -Dnsm.server.prop=/opt/nc421/nsm/conf/nsm_serv
root 2759 901 0 13:12:04 pts/1 0:00 ps -eaf
aponetnc 827 1 0 Jul 17 ? 0:00 /opt/nc421/customer/bin/aposvc /opt/nc421/customer/conf/nsms.conf wrapper.pidfi
aponetnc 844 843 0 Jul 17 ? 43:46 /opt/nc421/customer/jre/bin/java -Dapogee.bmsscore=true -Xmx256M -Djava.library
aponetnc 842 1 0 Jul 17 ? 0:00 /opt/nc421/customer/bin/aposvc /opt/nc421/customer/conf/nsmd.conf wrapper.pidfi
aponetnc 847 843 0 Jul 17 ? 20:53 /opt/nc421/customer/jre/bin/java -Dapogee.csscore=true -Xmx256M -Djava.library.
root 2760 828 0 13:12:27 ? 0:00 /usr/sbin/ping 10.11.12.111
root 901 455 0 Jul 17 pts/1 0:00 ksh
------------------------------------------------------------------------------------
[root-olive]/opt/nc421/bmss/logs> netstat

TCP: IPv4
Local Address Remote Address Swind Send-Q Rwind Recv-Q State
-------------------- -------------------- ----- ------ ----- ------ -------
olive.telnet 10.30.50.214.32882 63712 0 24616 0 ESTABLISHED
localhost.32853 localhost.1778 32768 0 32768 0 ESTABLISHED
localhost.1778 localhost.32853 32768 0 32768 0 ESTABLISHED
localhost.32860 localhost.1779 32768 0 32768 0 ESTABLISHED
localhost.1779 localhost.32860 32768 0 32768 0 ESTABLISHED
olive.32881 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.32881 40960 0 32768 0 ESTABLISHED
olive.32882 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.32882 40960 0 32768 0 ESTABLISHED
olive.32883 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.32883 40960 0 32768 0 ESTABLISHED
olive.32901 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.32901 40960 0 32768 0 ESTABLISHED
olive.32903 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.32903 40960 0 32768 0 ESTABLISHED
olive.33055 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.33055 40960 0 32768 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.2616 31440 0 33580 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.2760 31616 0 33580 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.3836 53836 0 33580 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.1233 62032 0 33580 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.1135 31440 0 33580 0 ESTABLISHED
olive.6779 chirag-test-7500w2k.1136 53248 0 33580 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.1279 32016 0 33580 0 ESTABLISHED
olive.6779 chirag-test-7500w2k.3455 61955 0 33580 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.4800 32768 0 33580 0 ESTABLISHED
olive.6779 chirag-test-7500w2k.3228 62825 0 33580 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.3627 31480 0 33580 0 ESTABLISHED
olive.6779 chirag-test-7500w2k.3628 53248 0 33580 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.3771 55346 0 33580 0 ESTABLISHED
olive.6779 chirag-test-7500w2k.3772 64090 0 33580 0 ESTABLISHED
olive.39917 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.39917 40960 0 32768 0 ESTABLISHED
olive.39918 olive.6779 32768 0 40960 0 ESTABLISHED
olive.6779 olive.39918 40960 0 32768 0 ESTABLISHED
olive.39919 olive.6779 32768 0 40960 0 ESTABLISHED
olive.6779 olive.39919 40960 0 32768 0 ESTABLISHED
olive.39954 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.39954 40960 0 32768 0 ESTABLISHED
olive.45717 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.45717 40960 0 32768 0 ESTABLISHED
olive.54031 olive.6780 32768 0 40960 0 ESTABLISHED
olive.6780 olive.54031 40960 0 32768 0 ESTABLISHED
olive.6780 chirag-test-7500w2k.1339 55248 0 33580 0 ESTABLISHED
olive.59321 olive.32888 32768 0 32768 0 TIME_WAIT
olive.59322 olive.32909 32768 0 32768 0 TIME_WAIT
olive.6779 chirag-test-7500w2k.1707 64172 0 33580 0 CLOSE_WAIT
olive.59326 chirag-test-7500w2k.1312 63810 0 24820 0 TIME_WAIT
olive.59327 dominator.32796 24820 0 24820 0 TIME_WAIT
olive.59328 cheese.32793 24820 0 24820 0 TIME_WAIT
olive.59329 olive.32865 32768 0 32768 0 TIME_WAIT
olive.6779 chirag-test-7500w2k.1721 64172 0 33580 0 CLOSE_WAIT
olive.6779 chirag-test-7500w2k.1722 64172 0 33580 0 CLOSE_WAIT
olive.6779 chirag-test-7500w2k.1726 64172 0 33580 0 CLOSE_WAIT
olive.6779 chirag-test-7500w2k.1732 64172 0 33580 0 CLOSE_WAIT
olive.6779 chirag-test-7500w2k.1733 64172 0 33580 0 CLOSE_WAIT
olive.6779 chirag-test-7500w2k.1734 64172 0 33580 0 CLOSE_WAIT
olive.6779 chirag-test-7500w2k.1735 64172 0 33580 0 CLOSE_WAIT
olive.6779 chirag-test-7500w2k.1736 64172 0 33580 0 ESTABLISHED

Active UNIX domain sockets
Address Type Vnode Conn Local Addr Remote Addr
300017bf7a8 stream-ord 300018516f0 00000000 /tmp/nfcunix.st
300017bf958 dgram 300017a2650 00000000 /tmp/nfc.events
300017bfb08 dgram 300017a2828 00000000 /tmp/nfcunix.dg
300017bfcb8 dgram 00000000 00000000
----------------------------------------------------------------
[root-olive]/opt/nc421/bmss/logs> ps -eaf|grep nc42
aponetnc 846 843 0 Jul 17 ? 31:53 /opt/nc421/customer/jre/bin/java -Dapogee.reportserver=true -Xmx512M -Djava.lib
aponetnc 744 1 0 Jul 17 pts/1 1:39 /opt/nc421/objy/bin/ools
root 2778 901 0 13:39:01 pts/1 0:00 grep nc42
aponetnc 845 843 0 Jul 17 ? 28:51 /opt/nc421/customer/jre/bin/java -Dapogee.housekeepserver=true -Xmx256M -Djava.
aponetnc 843 842 0 Jul 17 ? 0:20 /opt/nc421/customer/jre/bin/java -DJMS_BROKER_URL=localhost:2506 -Xms64m -Xmx12
aponetnc 743 1 0 Jul 17 pts/1 0:19 /opt/nc421/objy/bin/ooams xyzzy 8
aponetnc 828 827 0 Jul 17 ? 0:57 /opt/nc421/customer/jre/bin/java -Dnsm.server.prop=/opt/nc421/nsm/conf/nsm_serv
aponetnc 827 1 0 Jul 17 ? 0:00 /opt/nc421/customer/bin/aposvc /opt/nc421/customer/conf/nsms.conf wrapper.pidfi
aponetnc 844 843 0 Jul 17 ? 43:54 /opt/nc421/customer/jre/bin/java -Dapogee.bmsscore=true -Xmx256M -Djava.library
aponetnc 842 1 0 Jul 17 ? 0:00 /opt/nc421/customer/bin/aposvc /opt/nc421/customer/conf/nsmd.conf wrapper.pidfi
aponetnc 847 843 0 Jul 17 ? 20:59 /opt/nc421/customer/jre/bin/java -Dapogee.csscore=true -Xmx256M -Djava.library.
----------------------------------------------------------------------------------------------------------------

Any guidance will be appreciated

stickman · 07-28-2003, 12:44 PM

For starters, I'd go back and turn off any of those extra processes that you don't require. Good candidates are CDE, autofs, nfs, inetd, vold, powerd, and anything SNMP related. Restart you app so that you have a good starting point. If other services outside your app are running slow, you might want to start watching memory and swap utilizations on the system. Have you done any kernel or system tuning for this app? The problem could be any number of things, and you probably need to watch the the system with iostat, vmstat, etc to see what is going on. You might even want to truss any suspect processes to see what they are doing.