Hi All,

Iam using CentOS release 6.7 (Final) and iam running Apache web server in this server. Most frequently iam receiving Out Of memory alerts from this server. So when i checked /var/log/messages i saw oom killer has involved and killed the highest consuming process to make the memory space available.

In the below logs httpd "21696" process id has consumed huge space and the process has been killed by oom killer. Iam unable to find what caused this process to consume more memory. I checked application logs and iam unable to find the root cause of this oom killer. Could you please assist me on this issue.


Apr 19 12:11:15 ram1 kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 19 12:11:15 ram1 kernel: java cpuset=/ mems_allowed=0
Apr 19 12:11:15 ram1 kernel: Pid: 21137, comm: java Not tainted 2.6.32-504.12.2.el6.x86_64 #1
Apr 19 12:11:15 ram1 kernel: Call Trace:
Apr 19 12:11:15 ram1 kernel: [<ffffffff810d40c1>] ? cpuset_print_task_mems_allowed+0x91/0xb0
Apr 19 12:11:15 ram1 kernel: [<ffffffff81127300>] ? dump_header+0x90/0x1b0
Apr 19 12:11:15 ram1 kernel: [<ffffffff8122eb5c>] ? security_real_capable_noaudit+0x3c/0x70
Apr 19 12:11:15 ram1 kernel: [<ffffffff81127782>] ? oom_kill_process+0x82/0x2a0
Apr 19 12:11:15 ram1 kernel: [<ffffffff811276c1>] ? select_bad_process+0xe1/0x120
Apr 19 12:11:15 ram1 kernel: [<ffffffff81127bc0>] ? out_of_memory+0x220/0x3c0
Apr 19 12:11:15 ram1 kernel: [<ffffffff811344df>] ? __alloc_pages_nodemask+0x89f/0x8d0
Apr 19 12:11:15 ram1 kernel: [<ffffffff8116c69a>] ? alloc_pages_current+0xaa/0x110
Apr 19 12:11:15 ram1 kernel: [<ffffffff811246f7>] ? __page_cache_alloc+0x87/0x90
Apr 19 12:11:15 ram1 kernel: [<ffffffff811240de>] ? find_get_page+0x1e/0xa0
Apr 19 12:11:15 ram1 kernel: [<ffffffff81125697>] ? filemap_fault+0x1a7/0x500
Apr 19 12:11:15 ram1 kernel: [<ffffffff8114eae4>] ? __do_fault+0x54/0x530
Apr 19 12:11:15 ram1 kernel: [<ffffffff8114f0b7>] ? handle_pte_fault+0xf7/0xb00
Apr 19 12:11:15 ram1 kernel: [<ffffffff810a3da4>] ? hrtimer_start_range_ns+0x14/0x20
Apr 19 12:11:15 ram1 kernel: [<ffffffff8114fcea>] ? handle_mm_fault+0x22a/0x300
Apr 19 12:11:15 ram1 kernel: [<ffffffff81063bf3>] ? perf_event_task_sched_out+0x33/0x70
Apr 19 12:11:15 ram1 kernel: [<ffffffff8104d0d8>] ? __do_page_fault+0x138/0x480
Apr 19 12:11:15 ram1 kernel: [<ffffffff810097cc>] ? __switch_to+0x1ac/0x320
Apr 19 12:11:15 ram1 kernel: [<ffffffff81529a3e>] ? thread_return+0x4e/0x7d0
Apr 19 12:11:15 ram1 kernel: [<ffffffff8100c42e>] ? xen_hvm_callback_vector+0xe/0x20
Apr 19 12:11:15 ram1 kernel: [<ffffffff8153003e>] ? do_page_fault+0x3e/0xa0
Apr 19 12:11:15 ram1 kernel: [<ffffffff8152d3f5>] ? page_fault+0x25/0x30
Apr 19 12:11:15 ram1 kernel: Mem-Info:
Apr 19 12:11:15 ram1 kernel: Node 0 DMA per-cpu:
Apr 19 12:11:15 ram1 kernel: CPU 0: hi: 0, btch: 1 usd: 0
Apr 19 12:11:15 ram1 kernel: CPU 1: hi: 0, btch: 1 usd: 0
Apr 19 12:11:15 ram1 kernel: CPU 2: hi: 0, btch: 1 usd: 0
Apr 19 12:11:15 ram1 kernel: CPU 3: hi: 0, btch: 1 usd: 0
Apr 19 12:11:15 ram1 kernel: Node 0 DMA32 per-cpu:
Apr 19 12:11:15 ram1 kernel: CPU 0: hi: 186, btch: 31 usd: 30
Apr 19 12:11:15 ram1 kernel: CPU 1: hi: 186, btch: 31 usd: 0
Apr 19 12:11:15 ram1 kernel: CPU 2: hi: 186, btch: 31 usd: 0
Apr 19 12:11:15 ram1 kernel: CPU 3: hi: 186, btch: 31 usd: 0
Apr 19 12:11:15 ram1 kernel: Node 0 Normal per-cpu:
Apr 19 12:11:15 ram1 kernel: CPU 0: hi: 90, btch: 15 usd: 84
Apr 19 12:11:15 ram1 kernel: CPU 1: hi: 90, btch: 15 usd: 0
Apr 19 12:11:15 ram1 kernel: CPU 2: hi: 90, btch: 15 usd: 0
Apr 19 12:11:15 ram1 kernel: CPU 3: hi: 90, btch: 15 usd: 0
Apr 19 12:11:15 ram1 kernel: active_anon:887396 inactive_anon:11694 isolated_anon:0
Apr 19 12:11:15 ram1 kernel: active_file:45 inactive_file:1674 isolated_file:0
Apr 19 12:11:15 ram1 kernel: unevictable:0 dirty:0 writeback:0 unstable:0
Apr 19 12:11:15 ram1 kernel: free:20923 slab_reclaimable:3221 slab_unreclaimable:35849
Apr 19 12:11:15 ram1 kernel: mapped:28840 shmem:30475 pagetables:7522 bounce:0
Apr 19 12:11:15 ram1 kernel: Node 0 DMA free:15740kB min:248kB low:308kB high:372kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15344kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
Apr 19 12:11:15 ram1 kernel: lowmem_reserve[]: 0 3768 4016 4016
Apr 19 12:11:15 ram1 kernel: Node 0 DMA32 free:63948kB min:63164kB low:78952kB high:94744kB active_anon:3395108kB inactive_anon:26656kB active_file:88kB inactive_file:6908kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3858656kB mlocked:0kB dirty:0kB writeback:0kB mapped:85816kB shmem:92016kB slab_reclaimable:7324kB slab_unreclaimable:113284kB kernel_stack:672kB pagetables:25044kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:544 all_unreclaimable? yes
Apr 19 12:11:15 ram1 kernel: lowmem_reserve[]: 0 0 248 248
Apr 19 12:11:15 ram1 kernel: Node 0 Normal free:4004kB min:4164kB low:5204kB high:6244kB active_anon:154476kB inactive_anon:20120kB active_file:92kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:254520kB mlocked:0kB dirty:0kB writeback:0kB mapped:29544kB shmem:29884kB slab_reclaimable:5560kB slab_unreclaimable:30112kB kernel_stack:1056kB pagetables:5044kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:138 all_unreclaimable? yes
Apr 19 12:11:15 ram1 kernel: lowmem_reserve[]: 0 0 0 0
Apr 19 12:11:15 ram1 kernel: Node 0 DMA: 1*4kB 1*8kB 1*16kB 1*32kB 1*64kB 0*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15740kB
Apr 19 12:11:15 ram1 kernel: Node 0 DMA32: 845*4kB 459*8kB 301*16kB 215*32kB 146*64kB 102*128kB 39*256kB 15*512kB 5*1024kB 0*2048kB 0*4096kB = 63932kB
Apr 19 12:11:15 ram1 kernel: Node 0 Normal: 255*4kB 128*8kB 60*16kB 13*32kB 10*64kB 2*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 4316kB
Apr 19 12:11:15 ram1 kernel: 32256 total pagecache pages
Apr 19 12:11:15 ram1 kernel: 0 pages in swap cache
Apr 19 12:11:15 ram1 kernel: Swap cache stats: add 0, delete 0, find 0/0
Apr 19 12:11:15 ram1 kernel: Free swap = 0kB
Apr 19 12:11:15 ram1 kernel: Total swap = 0kB
Apr 19 12:11:15 ram1 kernel: Total swap = 0kB
Apr 19 12:11:15 ram1 kernel: 1047551 pages RAM
Apr 19 12:11:15 ram1 kernel: 68290 pages reserved
Apr 19 12:11:15 ram1 kernel: 454939 pages shared
Apr 19 12:11:15 ram1 kernel: 923407 pages non-shared
Apr 19 12:11:15 ram1 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 19 12:11:15 ram1 kernel: [ 847] 0 847 23283 71 0 -17 -1000 auditd
Apr 19 12:11:15 ram1 kernel: [ 888] 0 888 3057 289 1 0 0 xe-daemon
Apr 19 12:11:15 ram1 kernel: [ 977] 0 977 69991 1519 1 0 0 nova-agent
Apr 19 12:11:15 ram1 kernel: [ 1750] 0 1750 1016 22 0 0 0 mingetty
Apr 19 12:11:15 ram1 kernel: [ 1752] 0 1752 1016 22 3 0 0 mingetty
Apr 19 12:11:15 ram1 kernel: [ 1754] 0 1754 1016 21 3 0 0 mingetty
Apr 19 12:11:15 ram1 kernel: [ 1756] 0 1756 1016 21 3 0 0 mingetty
Apr 19 12:11:15 ram1 kernel: [ 1758] 0 1758 1016 22 3 0 0 mingetty
Apr 19 12:11:15 ram1 kernel: [ 1763] 0 1763 1016 21 0 0 0 mingetty
Apr 19 12:11:15 ram1 kernel: [ 3118] 0 3118 106839 1706 0 0 0 fail2ban-server
Apr 19 12:11:15 ram1 kernel: [ 4213] 0 4213 5773 62 0 0 0 rpc.idmapd
Apr 19 12:11:15 ram1 kernel: [ 8844] 0 8844 35192 1660 3 0 0 munin-node
Apr 19 12:11:15 ram1 kernel: [ 3132] 29 3132 5837 113 0 0 0 rpc.statd
Apr 19 12:11:15 ram1 kernel: [ 3139] 0 3139 77231 2890 0 0 0 python
Apr 19 12:11:15 ram1 kernel: [ 3301] 0 3301 29663 167 0 0 0 screen
Apr 19 12:11:15 ram1 kernel: [ 3302] 0 3302 27103 114 1 0 0 bash
Apr 19 12:11:15 ram1 kernel: [ 6428] 0 6428 20217 227 0 0 0 master
Apr 19 12:11:15 ram1 kernel: [ 6434] 89 6434 20280 236 0 0 0 qmgr
Apr 19 12:11:15 ram1 kernel: [15078] 0 15078 29223 165 0 0 0 crond
Apr 19 12:11:15 ram1 kernel: [11300] 497 11300 1558 30 0 0 0 tracelyzer-conn
Apr 19 12:11:15 ram1 kernel: [11307] 497 11307 48337 4895 0 0 0 tracelyzer
Apr 19 12:11:15 ram1 kernel: [ 6869] 0 6869 2661 94 1 -17 -1000 udevd
Apr 19 12:11:15 ram1 kernel: [ 4587] 32 4587 4744 59 0 0 0 rpcbind
Apr 19 12:11:15 ram1 kernel: [22976] 38 22976 7684 157 1 0 0 ntpd
Apr 19 12:11:15 ram1 kernel: [29374] 0 29374 16557 177 0 -17 -1000 sshd
Apr 19 12:11:15 ram1 kernel: [18814] 0 18814 112578 6396 0 0 0 rackspace-monit
Apr 19 12:11:15 ram1 kernel: [ 1981] 0 1981 52731 182 3 0 0 sssd
Apr 19 12:11:15 ram1 kernel: [ 1983] 0 1983 69796 201 0 0 0 sssd_nss
Apr 19 12:11:15 ram1 kernel: [ 1984] 0 1984 66180 199 3 0 0 sssd_pam
Apr 19 12:11:15 ram1 kernel: [ 6150] 497 6150 15619 328 0 0 0 ssh
Apr 19 12:11:15 ram1 kernel: [15819] 0 15819 62817 313 0 0 0 rsyslogd
Apr 19 12:11:15 ram1 kernel: [21105] 0 21105 956681 85918 1 0 0 java
Apr 19 12:11:15 ram1 kernel: [ 5812] 0 5812 125516 9161 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [19159] 0 19159 2660 93 2 -17 -1000 udevd
Apr 19 12:11:15 ram1 kernel: [19160] 0 19160 2660 93 3 -17 -1000 udevd
Apr 19 12:11:15 ram1 kernel: [24114] 0 24114 56648 293 0 0 0 smbd
Apr 19 12:11:15 ram1 kernel: [24153] 0 24153 56777 293 0 0 0 smbd
Apr 19 12:11:15 ram1 kernel: [ 5467] 48 5467 170267 41594 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [ 5473] 48 5473 169706 40712 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [ 5476] 48 5476 170864 41844 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [ 9042] 48 9042 168430 39095 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [14502] 48 14502 196046 66480 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [21696] 48 21696 290025 160129 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [21702] 48 21702 257561 127878 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [21703] 48 21703 167920 38426 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [21707] 48 21707 168685 39713 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [ 3256] 0 3256 116004 27569 0 0 0 sssd_be
Apr 19 12:11:15 ram1 kernel: [24708] 48 24708 170097 40859 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [ 7136] 48 7136 240875 111381 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [14782] 48 14782 168920 38446 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [21641] 48 21641 169198 38489 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [27772] 48 27772 168590 37915 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [28566] 48 28566 167451 36974 2 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [ 6181] 0 6181 26552 244 3 0 0 sshd
Apr 19 12:11:15 ram1 kernel: [ 6183] 526 6183 26552 250 0 0 0 sshd
Apr 19 12:11:15 ram1 kernel: [ 6184] 526 6184 27075 89 0 0 0 bash
Apr 19 12:11:15 ram1 kernel: [ 6725] 0 6725 44821 159 1 0 0 sudo
Apr 19 12:11:15 ram1 kernel: [ 6726] 0 6726 37826 100 0 0 0 su
Apr 19 12:11:15 ram1 kernel: [ 6727] 0 6727 27108 120 0 0 0 bash
Apr 19 12:11:15 ram1 kernel: [ 6756] 48 6756 166706 34608 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [10220] 48 10220 230896 116566 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [10289] 48 10289 192040 56270 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [13156] 89 13156 20762 225 0 0 0 pickup
Apr 19 12:11:15 ram1 kernel: [13194] 48 13194 135719 23575 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [13340] 48 13340 137634 18561 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [13402] 48 13402 164018 26829 0 0 0 httpd
Apr 19 12:11:15 ram1 kernel: [13409] 0 13409 26386 75 0 0 0 less
Apr 19 12:11:15 ram1 kernel: [14609] 0 14609 1018 21 0 0 0 sleep
Apr 19 12:11:15 ram1 kernel: Out of memory: Kill process 21696 (httpd) score 163 or sacrifice child
Apr 19 12:11:15 ram1 kernel: Killed process 21696, UID 48, (httpd) total-vm:1160100kB, anon-rss:550868kB, file-rss:89648kB

Thanks in advance

Ram Kannan.

Please see the below physical memory available in this server.

[root@ram1 ~]# free -m
total used free shared buffers cached
Mem: 3825 2379 1445 119 26 371
-/+ buffers/cache: 1982 1843
Swap: 0 0 0
[root@ram1 ~]#

[root@ram1 ~]# cat /var/log/messages | grep "oom"
Apr 17 13:24:44 ram1 kernel: httpd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Apr 17 13:24:44 ram1 kernel: [<ffffffff81127782>] ? oom_kill_process+0x82/0x2a0
Apr 17 13:24:44 ram1 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 17 13:25:24 ram1 kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 17 13:25:24 ram1 kernel: [<ffffffff81127782>] ? oom_kill_process+0x82/0x2a0
Apr 17 13:25:24 ram1 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 19 11:42:52 ram1 kernel: httpd invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0, oom_score_adj=0
Apr 19 11:42:52 ram1 kernel: [<ffffffff81127782>] ? oom_kill_process+0x82/0x2a0
Apr 19 11:42:52 ram1 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 19 12:06:08 ram1 kernel: httpd invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 19 12:06:08 ram1 kernel: [<ffffffff81127782>] ? oom_kill_process+0x82/0x2a0
Apr 19 12:06:08 ram1 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
Apr 19 12:11:15 ram1 kernel: java invoked oom-killer: gfp_mask=0x201da, order=0, oom_adj=0, oom_score_adj=0
Apr 19 12:11:15 ram1 kernel: [<ffffffff81127782>] ? oom_kill_process+0x82/0x2a0
Apr 19 12:11:15 ram1 kernel: [ pid ] uid tgid total_vm rss cpu oom_adj oom_score_adj name
[root@ram1 ~]#

Sorry, there's nothing we can assist WITH, based on what you've provided. Saying "apache web server" tells us nothing...you don't say what kind pages it's serving up, what they're written in, if this server houses a database, if anything has changed on the web code, or even if this has EVER worked correctly.

First thing to do is to check your web page code....you could have a bad program/memory leak, PHP could be using a ton of memory, bad javascript...who knows???

Thanks a lot.

I will check with application team to check the code.

That should have been the FIRST THING you did, before posting a fairly vague question. Basic troubleshooting should always be done first.

you can monitor your resources with atop and htop commands and see what process is consuming the memory of your server.
BTW. I personally suggest you to delete the apache and switch to litespeed.