RHEL eating RAM space in production environment (top command)

fawaz25 · 12-02-2018, 05:29 AM

Dear Members,

I have attached the top command output and free command output with the message. I see all RAM is being consumed. What can be the issue for this?

would be glad for your help.

smallpond · 12-02-2018, 06:11 AM

https://www.linuxatemyram.com

berndbausch · 12-02-2018, 08:33 AM

Quote:

Originally Posted by fawaz25

the server becomes slow and goes down

What do you mean by “goes down”?

Quote:

I have attached the top command output and free command output with the message

Try again, nothing is attached.

scasey · 12-02-2018, 10:00 AM

I believe one has to have more than 1 post before they can attach files...
That said, if this a RHEL server, have you contacted RedHat Support?

berndbausch · 12-02-2018, 10:09 AM

Quote:

Originally Posted by scasey

I believe one has to have more than 1 post before they can attach files...

The output of top and free can easily be added to the body of a post.

scasey · 12-02-2018, 10:14 AM

Quote:

Originally Posted by berndbausch

The output of top and free can easily be added to the body of a post.

Absolutely! Adding the output in the body in [code] tags would be preferred to an attachment, too.

fawaz25 · 12-02-2018, 11:49 PM

Using RHEL 5.7.
The output of top command (SHIFT+M) is as follows:

Code:

top - 08:40:17 up 3 days, 20:26,  1 user,  load average: 18.22, 18.67, 18.38
Tasks: 627 total,   1 running, 626 sleeping,   0 stopped,   0 zombie
Cpu(s): 19.4%us,  8.0%sy,  0.0%ni, 68.0%id,  0.0%wa,  0.0%hi,  4.6%si,  0.0%st
Mem:  128994532k total, 128674952k used,   319580k free,   556464k buffers
Swap:        0k total,        0k used,        0k free, 79488444k cached

# free -m command gives the following output:

Code:

            total       used       free     shared    buffers     cached
Mem:        125971     125661        309          0        543      77611
-/+ buffers/cache:      47506      78464
Swap:            0          0          0

berndbausch · 12-03-2018, 12:25 AM

Please use code tags. See my signature if you don't know how.

Code:

08:40:17 up 3 days, 20:26,  1 user,  load average: 18.22, 18.67, 18.38
Tasks: 627 total,   1 running, 626 sleeping,   0 stopped,   0 zombie
Cpu(s): 19.4%us,  8.0%sy,  0.0%ni, 68.0%id,  0.0%wa,  0.0%hi,  4.6%si,  0.0%st
Mem:  128994532k total, 128674952k used,   319580k free,   556464k buffers
Swap:        0k total,        0k used,        0k free, 79488444k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU   %MEM    TIME+  COMMAND
11936 hcmpsoft  24   0 1312m 1.2g 1780 S 316.8  1.0   4399:28 java
12126 hcmpsoft  22   0 1271m 1.2g 1764 S 290.2  1.0   4124:44 java
11974 hcmpsoft  25   0 1269m 1.2g 1764 S 298.5  1.0   4219:08 java
12050 hcmpsoft  24   0 1265m 1.2g 1764 S 228.8  1.0   3343:02 java
10684 hcmpsoft  15   0 31.9g 1.2g  47m S  0.0   0.9   2:57.62 PSAPPSRV
10242 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   2:58.41 PSAPPSRV
10394 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   3:10.13 PSAPPSRV
10722 hcmpsoft  15   0 31.9g 1.1g  47m S  0.0   0.9   3:16.06 PSAPPSRV
10646 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   3:17.67 PSAPPSRV
10166 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   3:09.95 PSAPPSRV
10798 hcmpsoft  15   0 31.8g 1.1g  48m S  0.0   0.9   3:00.12 PSAPPSRV
11684 hcmpsoft  15   0 31.9g 1.1g  47m S  0.0   0.9   3:58.00 PSAPPSRV
10836 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   3:09.22 PSAPPSRV
10356 hcmpsoft  15   0 31.8g 1.1g  47m S  0.0   0.9   3:03.21 PSAPPSRV
10912 hcmpsoft  15   0 32.0g 1.1g  48m S  0.0   0.9   2:58.89 PSAPPSRV
11393 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   2:47.18 PSAPPSRV
10128 hcmpsoft  15   0 31.8g 1.1g  48m S  0.0   0.9   3:06.08 PSAPPSRV
 9399 hcmpsoft  15   0 31.8g 1.1g  48m S  1.7   0.9   3:06.92 PSAPPSRV
10432 hcmpsoft  15   0 31.8g 1.1g  48m S  0.0   0.9   3:03.47 PSAPPSRV
10950 hcmpsoft  16   0 31.9g 1.1g  48m S  0.0   0.9   3:03.04 PSAPPSRV
10280 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   2:52.59 PSAPPSRV
10470 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   3:04.24 PSAPPSRV
10988 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   3:02.70 PSAPPSRV
 9195 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   3:00.40 PSAPPSRV
10318 hcmpsoft  15   0 31.8g 1.1g  48m S  0.0   0.9   3:15.51 PSAPPSRV
10760 hcmpsoft  15   0 31.9g 1.1g  48m S  0.0   0.9   3:03.67 PSAPPSRV
11203 hcmpsoft  16   0 31.9g 1.1g  47m S  0.0   0.9   2:21.44 PSAPPSRV
11469 hcmpsoft  15   0 31.8g 1.1g  48m S  0.0   0.9   2:38.83 PSAPPSRV
10204 hcmpsoft  15   0 31.9g 1.0g  47m S  0.0   0.8   2:55.65 PSAPPSRV
 8424 hcmpsoft  15   0 31.8g 1.0g  48m S  0.0   0.8   3:06.12 PSAPPSRV
10874 hcmpsoft  15   0 31.8g 1.0g  48m S  0.0   0.8   2:53.60 PSAPPSRV
11241 hcmpsoft  15   0 31.8g 1.0g  47m S  0.0   0.8   2:31.95 PSAPPSRV
11355 hcmpsoft  16   0 31.8g 1.0g  48m S  0.0   0.8   2:37.33 PSAPPSRV
11507 hcmpsoft  15   0 31.8g 1.0g  48m S  0.0   0.8   2:30.55 PSAPPSRV
11317 hcmpsoft  16   0 31.8g 1.0g  48m S  0.0   0.8   2:31.78 PSAPPSRV
11279 hcmpsoft  15   0 31.8g 1.0g  48m S  0.0   0.8   2:39.42 PSAPPSRV
11431 hcmpsoft  15   0 31.8g 1.0g  48m S  0.0   0.8   2:36.71 PSAPPSRV
 9258 hcmpsoft  15   0 31.8g 1.0g  47m S  0.0   0.8   3:00.52 PSAPPSRV
11165 hcmpsoft  15   0 31.8g 1.0g  48m S  0.0   0.8   2:32.34 PSAPPSRV
 9338 hcmpsoft  15   0 31.8g 1.0g  48m S  0.0   0.8   3:16.69 PSAPPSRV
 9453 hcmpsoft  15   0 31.7g 1.0g  47m S  0.0   0.8   2:56.46 PSAPPSRV
 9201 hcmpsoft  15   0 31.7g 1.0g  47m S  0.0   0.8   3:01.95 PSAPPSRV

# The output of cat proc/meminfo is as follows:

MemTotal:     128994532 kB
MemFree:        318376 kB
Buffers:        556384 kB
Cached:       79474344 kB
SwapCached:          0 kB
Active:       99364828 kB
Inactive:     27979524 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:     128994532 kB
LowFree:        318376 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Dirty:            1472 kB
Writeback:           0 kB
AnonPages:    47313640 kB
Mapped:         112360 kB
Slab:          1078740 kB
PageTables:     153380 kB
NFS_Unstable:        0 kB
Bounce:              0 kB
CommitLimit:  64497264 kB
Committed_AS: 132027656 kB
VmallocTotal: 34359738367 kB
VmallocUsed:    262880 kB
VmallocChunk: 34359474939 kB
HugePages_Total:     0
HugePages_Free:      0
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

# free -m command gives the following output:
            total       used       free     shared    buffers     cached
Mem:        125971     125661        309          0        543      77611
-/+ buffers/cache:      47506      78464
Swap:            0          0          0

berndbausch · 12-03-2018, 12:38 AM

Your CPU is currently 68% idle. Memory is practically depleted; almost nothing is available or used for the buffer cache. I conclude that your application uses up memory. You say

Quote:

I am sure the application won't use this much RAM as it is serving few users only

How did you determine this?

I see 42 processes running, each of them using 1GB or a bit more RAM, totaling 45GB. I wonder how many PSAPPSRV are running altogether? Couldn't they add up to 128GB? Hint: you can scroll top with pgup/pgdown, or run ps -el | grep PSAPPSRV.

syg00 · 12-03-2018, 12:58 AM

What about "uname -a".

And yes, why all the (PeopleSoft ?) instances if only for a few users ?.

fawaz25 · 12-03-2018, 01:15 AM

Quote:

Originally Posted by berndbausch

Your CPU is currently 68% idle. Memory is practically depleted; almost nothing is available or used for the buffer cache. I conclude that your application uses up memory. You say

How did you determine this?

I see 42 processes running, each of them using 1GB or a bit more RAM, totaling 45GB. I wonder how many PSAPPSRV are running altogether? Couldn't they add up to 128GB? Hint: you can scroll top with pgup/pgdown, or run ps -el | grep PSAPPSRV.

Yes I do see many instances running. The total used RAM computed from top command (37.9%) is same as that shown by the free command.

berndbausch · 12-03-2018, 03:10 AM

Quote:

Originally Posted by fawaz25

I wonder why with just 1 user logged in, PSAPPSRV is consuming around 35% of Memory with JAVA just consuming around 4% of memory. Can this be an application related issue??

It is still far from the 128GB you have. It's puzzling that the difference is so big.
EDIT: I misread the free output. It makes sense now.

Regarding your question, the kernel doesn't just launch PSAPPSRV processes because it feels like it. This is most definitely an application issue. It could be caused by a condition outside of the application, but still, the application should be able to handle external conditions correctly and not start tens of processes if they are not needed.

EDIT: I just saw that PSAPPSRV is probably a Peoplesoft process. I would ask the question in an Oracle or Peoplesoft forum.

fawaz25 · 12-03-2018, 03:25 AM

Quote:

Originally Posted by syg00

What about "uname -a".

And yes, why all the (PeopleSoft ?) instances if only for a few users ?.

I too am trying to figure that out.

fawaz25 · 12-03-2018, 03:43 AM

Quote:

Originally Posted by berndbausch

It is still far from the 128GB you have. It's puzzling that the difference is so big.
EDIT: I misread the free output. It makes sense now.

Regarding your question, the kernel doesn't just launch PSAPPSRV processes because it feels like it. This is most definitely an application issue. It could be caused by a condition outside of the application, but still, the application should be able to handle external conditions correctly and not start tens of processes if they are not needed.

EDIT: I just saw that PSAPPSRV is probably a Peoplesoft process. I would ask the question in an Oracle or Peoplesoft forum.

Thank you for your comments. I really appreciate.

pan64 · 12-03-2018, 05:44 AM

just a question:
is this only one app with several threads or a lot of apps? (see -H in top)