/proc/kcore problems

The_Watcher · 02-22-2008, 12:22 PM

Hi all,

I have a big /proc/kcore file of 11GB:

-r-------- 1 root root 11G Feb 22 18:08 kcore

I have read the past mails on kcore, and I understand that it is a virtual file, not really there. The problem is that my file system is full. Here is my df -H :

Filesystem Size Used Avail Use% Mounted on
/dev/sda3 26G 26G 0 100% /
/dev/sda2 104M 83M 16M 85% /boot
none 4.2G 0 4.2G 0% /dev/shm
/dev/mapper/vastvolume-vastdata
549G 122G 399G 24% /vastdata

As you can see sda3 is full. Unfortunately the past mails never really explained how to get rid of this problem.

I am a novice as a manager -- I am terrified of anything that goes wrong even on my simple workstation. I use RHEL.

I shut the machine down and restarted thinkign that might solve the problem, but instead it would not let me logon as my username, and so I am having to logon as root -- not a healthy option.

The question is how can (1) I trace how the sda3 disk became full? I was actually downloading some free software (Portland fortran compiler, pgf90, as it happens) which I had done before only 2 weesk ago. I do not know what I did but now I cannot do anything as my username.

(2) How can I find out where and what files are taking up the extra space?
(3) How can I delete these and restore to my normal functionality?
(4) While I am at it, what commands are there that can give a finer breakdown of the files kept on sda3,sda2 etc -- at the moment I do not know where to search?
(5) Any other advice.

Thank you.
Desparately yours,

rayfordj · 02-22-2008, 12:32 PM

(1) 'find' can be used to find files of X size or larger
'du -ckx' (or any variant you choose) can be used to walk the root ( / ) filesystem to find the directories. 'du' method is more of a manual process that you walk it but can be "automated"

(2) 'find'
additional info http://www.linuxquestions.org/linux/...uick_find_Tips

Code:

find / -mount -size +500M -print

for example will find all files larger than 500MB and print their location.

(3) after identifying them and deciding if they are something you can do without just use 'rm' to remove/delete the file(s)

(4) 'find' ?

(5) if you are unable to login to usual runlevel boot then you should try booting to Single-User (or runlevel 1) by passing the kernel option via your bootloader (?grub?) by editing the kernel line and adding ' S'.

example
before:

Code:

        root (hd0,0)
        kernel /vmlinuz-2.6.23.9-85.fc8 ro root=/dev/sda3 
        initrd /initrd-2.6.23.9-85.fc8.img

after:

Code:

        root (hd0,0)
        kernel /vmlinuz-2.6.23.9-85.fc8 ro root=/dev/sda3 S
        initrd /initrd-2.6.23.9-85.fc8.img

and then selecting the option to boot 'b'

The_Watcher · 02-22-2008, 02:44 PM

Thanks rayford for the tips.

In fact, I have already made space by deleting unwanted files and also archiving some others to an external hard drive. So now I can logon as username at least.

Code:

find / -mount -size +500M -print

This gives:

/iso/RHEL4-US-x86_64-source-disc4-ftp.iso
/iso/RHEL4-US-x86_64-source-disc1-ftp.iso
/iso/RHEL4-US-RC4-x86_64-WS-disc4-ftp.iso
/iso/RHEL4-US-RC4-x86_64-WS-disc5-ftp.iso
/iso/RHEL4-US-RC4-x86_64-WS-disc2-ftp.iso
/iso/RHEL4-US-x86_64-source-disc2-ftp.iso
/iso/RHEL4-US-RC4-x86_64-WS-disc3-ftp.iso
/iso/HP_DriverCD_RHEL4.iso
/iso/RHEL4-US-x86_64-source-disc3-ftp.iso

which when I look at the sizes gives:

-rwxr-xr-x 1 root root 176M Mar 24 2006 /iso/RHEL4-US-RC4-x86_64-WS-disc1-ftp.iso
-rwxr-xr-x 1 root root 597M Sep 26 2006 /iso/RHEL4-US-RC4-x86_64-WS-disc2-ftp.iso
-rwxr-xr-x 1 root root 639M Sep 26 2006 /iso/RHEL4-US-RC4-x86_64-WS-disc3-ftp.iso
-rwxr-xr-x 1 root root 638M Sep 26 2006 /iso/RHEL4-US-RC4-x86_64-WS-disc4-ftp.iso
-rwxr-xr-x 1 root root 267M Sep 26 2006 /iso/RHEL4-US-RC4-x86_64-WS-disc5-ftp.iso
-rwxr-xr-x 1 root root 523M Sep 26 2006 /iso/RHEL4-US-x86_64-source-disc1-ftp.iso
-rwxr-xr-x 1 root root 523M Sep 26 2006 /iso/RHEL4-US-x86_64-source-disc2-ftp.iso
-rwxr-xr-x 1 root root 523M Sep 26 2006 /iso/RHEL4-US-x86_64-source-disc3-ftp.iso
-rwxr-xr-x 1 root root 523M Sep 26 2006 /iso/RHEL4-US-x86_64-source-disc4-ftp.iso

Are these *.iso important files, or can they be deleted?

But I am still puzzled as to what is happening -- it is as if some massive file somewhere, either in my hard disc, or RAM or somewhere else is taking up 11GB. Why can't I see it and simply delete it?

Or, as I suspect, I am missing something?

Cheers.

rayfordj · 02-22-2008, 06:03 PM

if you are referring to /proc/kcore it is not an issue. nothing in /proc consumes on-disk storage, it is just your window into the kernel, so to speak.

my desktop:

Code:

$ ls -hl /proc/kcore
-r-------- 1 root root 897M 2008-02-22 17:53 /proc/kcore

one of my servers:

Code:

$ ls -hl /proc/kcore
-r-------- 1 root root 4.8G Feb 22 17:55 /proc/kcore

kcore is the kernel core image.

you can find more info on items in /proc by reading the kernel-docs (may need to install kernel-doc RPM).

/usr/share/doc/kernel-doc-*/Documentation/filesystems/proc.txt

if it is on-disk storage you are lacking then looking under root [sda3] (since that is your problem mount related to this post) is where you should focus.

as far as the ISOs, it really is your call if they are something you should keep around or not. Even if you keep the installation media around in ISO form, it has been my experience that most individuals do not need the source disks/ISOs. those ISOs (*-source-*) contain the .src.rpm packages that RH used to build the RPMs used on the installation media.

The_Watcher · 02-23-2008, 11:09 AM

Thanks again rayford,

Quote:

Originally Posted by rayfordj

if you are referring to /proc/kcore it is not an issue. nothing in /proc consumes on-disk storage, it is just your window into the kernel, so to speak.

if it is on-disk storage you are lacking then looking under root [sda3] (since that is your problem mount related to this post) is where you should focus.

as far as the ISOs, it really is your call if they are something you should keep around or not. Even if you keep the installation media around in ISO form, it has been my experience that most individuals do not need the source disks/ISOs. those ISOs (*-source-*) contain the .src.rpm packages that RH used to build the RPMs used on the installation media.

So would I be correct in summarising that there isn't actually a problem? I have just come to the natural size of by disk space on sda3? I had thought that it was a system crash or I had done something inadvertently.

On the *.iso files: I do not want to keep them unecessarily if they are not actively used in any way -- so what are they ever useful for, or are they just for record keeping. What about when you upgrade, like I am planning an upgrade to RHEL5.0 soon -- would I need these files then?

What about updates, are these files used then?

Instead of deleting them, can I tar them, or move them somewhere else, say on to an external hard disc? I presume I could bring them back and use them if I ever needed them?

Cheers.

rayfordj · 02-23-2008, 11:28 AM

yeah, i've seen several posts previously about cause for alarm regarding a crash by the presence of /proc/kcore; it's a "naturally" present file and no cause for alarm.

they (the source ISOs) are useful if you want to compile your own RPMs using the templates (spec file) that RH used.

the ISOs are available from RHN.RedHat.com if you have an account (or you can create an eval account for a 30-day eval). You shouldn't need them for an update/upgrade either.

I would say they are your's to do with as you please. If you would like the security of knowing you have them somewhere in the event you ever have a need to obtain something from them then absolutely you are welcome to copy them to an external drive (cp, tar, whatever).

The_Watcher · 02-23-2008, 06:11 PM

Quote:

Originally Posted by rayfordj

yeah, i've seen several posts previously about cause for alarm regarding a crash by the presence of /proc/kcore; it's a "naturally" present file and no cause for alarm.

they (the source ISOs) are useful if you want to compile your own RPMs using the templates (spec file) that RH used.

the ISOs are available from RHN.RedHat.com if you have an account (or you can create an eval account for a 30-day eval). You shouldn't need them for an update/upgrade either.

I would say they are your's to do with as you please. If you would like the security of knowing you have them somewhere in the event you ever have a need to obtain something from them then absolutely you are welcome to copy them to an external drive (cp, tar, whatever).

Cheers rayford -- you've been a great help

Best

syg00 · 02-23-2008, 07:05 PM

It's not just big files that consume space - lots of little ones can too. Try this to see where (which directories) space is going

Code:

du / -x --max-depth=1 | sort -nr | less

Run it as root/sudo to avoid errors.

igadoter · 02-21-2009, 04:40 AM

Quote:

Originally Posted by The_Watcher

Hi all,

I have a big /proc/kcore file of 11GB:

-r-------- 1 root root 11G Feb 22 18:08 kcore

I have read the past mails on kcore, and I understand that it is a virtual file, not really there. The problem is that my file system is full. Here is my df -H :

Filesystem Size Used Avail Use% Mounted on
/dev/sda3 26G 26G 0 100% /
/dev/sda2 104M 83M 16M 85% /boot
none 4.2G 0 4.2G 0% /dev/shm
/dev/mapper/vastvolume-vastdata
549G 122G 399G 24% /vastdata

As you can see sda3 is full. Unfortunately the past mails never really explained how to get rid of this problem.

I am a novice as a manager -- I am terrified of anything that goes wrong even on my simple workstation. I use RHEL.

I shut the machine down and restarted thinkign that might solve the problem, but instead it would not let me logon as my username, and so I am having to logon as root -- not a healthy option.

The question is how can (1) I trace how the sda3 disk became full? I was actually downloading some free software (Portland fortran compiler, pgf90, as it happens) which I had done before only 2 weesk ago. I do not know what I did but now I cannot do anything as my username.

(2) How can I find out where and what files are taking up the extra space?
(3) How can I delete these and restore to my normal functionality?
(4) While I am at it, what commands are there that can give a finer breakdown of the files kept on sda3,sda2 etc -- at the moment I do not know where to search?
(5) Any other advice.

Thank you.
Desparately yours,

On my system I'm using jfs-file system not native ext*. When disk was completely full and I get the message 'no disk space'.
I tried to remove 'rm' very big file but it does'nt work. I have to
restart and after that 'rm' was OK.

When I see 91% used I remove something to 90% or 89%.