Hello,
I have a server running CentOS 5.3 that I use to host some websites.

There are many bad things happening on my server, and I hope that someone can suggest me a solution.

1) I have noticed many strange things in the Apache error log, like someone downloading strange tgz files like "MechBot.tgz", "Stealth.tgz", "cmd.tgz", or attempts to do commands like "cat" "chmod" "mkdir" or "rm".

2) I also noticed many strange files and directories inside the "tmp" and "var/tmp" directories owned by the apache user, with names like "." or ".img" or "config93DKDJ", and also some scripts with names like "mech" or "juno".

3) There was a cron job created by the apache user pointing to /tmp/.img/update that ran every minute, and I've deleted it.

4) Furthermore I've seen that there are continuous outgoing connections on ports like 6667 6669 or high ports like 54377.

So I really think that someone is trying to hack my system.

I think it all began after I have deleted the access and error log files of a website because they were becoming very big, even if I don't know how this could have caused the attack. I use Awstats, I don't know if it could be related.

How can I safely delete all those bad files in the tmp directory and stop the attack?
I know about tools like Chkrootkit and RKhunter. Are they useful? Is it better to run them on a live cd? And how can I do it?

Thank you.

Roberto

Circumstances seem rather unfortunate but welcome, hope you like it here.

You don't need a quick solution. What you need is to shape up Linux and esp. admin-wise, mitigation, thorough analysis and a plan to avoid this in the future.
To help us help you you should answer all questions, answer them quickly, as verbose as possible and stay with the thread until completion.
* Should you at any stage want to share (bulk) data (privately) then feel free to email Hangdog42 or me.

I'll go over what "evidence" you posted in a later reply but for now let's start with the most important items.


Hardening the OS should be one of the first things to do before exposing it to the 'net as it helps audit the server, limit attack risks and fallout.
- Did you harden the machine?
- Are you new to Linux or do you consider yourself knowledgeable?
- Is this your own physical machine, a VPS or cloud instance or what else?
- Do you SSH into the machine for maintenance purposes or do you use a web-based management panel?
Centos is at 5.8 right now. Keeping things up to date, meaning the OS and any software you run in your web stack including any 3rd party plugins or homebrewn scripts, means less risk due to vulnerabilities.
- Did you keep any software up to date?
- Do you keep backups?
- Have there been breaches or anomalies earlier on?
- What software do you run on top of your web server? Please list as verbose as possible: services the machine provides like FTP, SSH, etc, etc, and including any web-based software like management panel, statistics, web log, forum, shopping cart software, plugins and others,


Deleting "evidence" is the best way to help crackers because it thwarts the investigation. Don't do so until asked by me or Hangdog42.


Before you do anything please read the CERT Intruder Detection Checklist as it contains commands we would like you to run. Let's gather data and mitigate things after that:

Please run the following commands as root and attach as plain text:
- A full process listing: - Open files: - Temporary files: - Known crontabs: - Network connections: - User login data: - RPM database verification: Unless there is anything else to add or consider now shut down your web server and database and any services like cron, at, CUPS, FTP, NFS, etc, etc that are exposed and unnecessary during the investigation. Again, please answer all questions, as quickly and as verbose as possible. If there's anything you need to ask before commencing please feel free to do so.

1 members found this post helpful.

First STOP all Apache (httpd) activities and block/disconnect the connections going through
check
and take a look at the services/IP addresses and ports opened and killing the processes using it to stop it.
Then don't run it again until you get cleaned it out.

use the forum's search option and search for "server hacked" and you will find many threads about the topic.


PS: take the necessary steps unSpawn gave you, he knows what is needed!

1 members found this post helpful.

@OP: please disregard lithos' reply. He may have 1K+ posts but he sure doesn't handle incidents according to our standards. Let this serve as a reminder to all who think of posting in threads concerning (perceived) incidents that we need members to stay with the thread, show knowledge of incident handling including proper triage, and definitely not redirecting new members in need to our search facility, no matter how good it is.

I am 100% confident that your server has been thoroughly compromised.

If you have outgoing connections from your box, then it's almost a certainty that malware is running on it already.

In my experience, too-many people simply set up the 'easy' Plesk-based servers that their hosting companies push off onto them. Those server configurations are already running "everything but the kitchen sink" and they have well-known vulnerabilities.

If you are running on a shared host, then in too-many cases any other user on the same machine can effortlessly examine and/or manipulate your files. (They can find out what your database password is, literally by reading your source code with less.)

You will need to set up this site on a dedicated (virtual...) server that you build from a minimal configuration from the ground up, installing upon it only what you need and configuring security from the very start.

I recommend that the existing server must be replaced from the ground up in this way. I have no confidence that a server can be rescued. (It could have been root-kitted and you would not know.) You might have to set up a parallel configuration and then cause the DNS entries to refer to it as a cut-over strategy.

There are very well-respected "cookbook" guides for doing that, and of course, this forum on this web site is well-staffed by qualified experts who can assist you. You have found a well-respected security resource, right here where you are. What you read from folks like unSpawn, for example, should be considered implicitly knowledgeable and trustworthy.

my 2 cents...

IF you are using WHM change your root password.

Apparently you haven't read or understood post #3. Until you do, could you please stay out of any incident threads?
TIA


Given the lack of data wouldn't you say that's a bit early? And malware running doesn't necessarily mean root compromise as threads the past five or so years in this forum have shown...


Could you please wait with that until we've gathered nfo and assessed the situation as we usually do?


Thanks for the vote of confidence but I am not without mistakes. BTW there's a few other members I consider well-versed in the Art of Incident handling: Noway2 and Hangdog42 (unixfool and slimm609 too but they haven't been around lately) and their approach shows it: dealing with problems in a structured way, providing proper guidance, emphasis on facts and not fiction, avoiding speculation, staying with the OP until the problem is resolved, etc, etc.
Since most members facing incidents are new and need a helping hand badly it should be clear drive-by one-liner posts only serve to show their owners ineptitude and failure to understand its consequences.


// Please note the text below was in response to the OP and not other LQ members.
// Unfortunately the vBB post collapsing feature worked against me as this time stamp shows:
// ---------- Post added 07-08-12 at 10:32 ----------
//


Please note some of the following may be interpreted as me being harsh. Before you shoot the messenger do realize my only intention is to show what went wrong and how to improve. If you find you may have been negligent, lacking knowledge, not providing adequate services and endangering the rest of the 'net just learn from it.

Before I start, if you have a different machine(*) at your disposal, copy over all the system and daemon logs and run them through Logwatch(**) with the "--detail High --service All --range All --archives --numeric --save /safe/path/logwatch.log" switches (replace "/safe/path/" with a real path). Quickest, most efficient way to generate leads IMHO.


Applications running on top of your web stack (comprising of a web and database server and language interpreters like Perl, PHP, Python, Ruby and others) are often riddled with holes. Especially PHP (which stands for "Pretty Horrific Programming") applications suffer a lot from that. One of the most common items in logs you see are probes for parameters which can be abused for executing commands. Noticing those or 'curl' or 'wget' output in your web servers log files you should check if they succeed (return code in the 200 range) or not. Updating applications should then be your first reflex, controlling requests (see for instance mod_security), ensuring you are informed of any potential abuse (audit, Logwatch?) and checking your software documentation for information on security. For instance software like PHP, Joomla or WordPress provide security documentation on their web sites, it just seems not many people read and implement measures. The longer you put off responding to potential abuse the harder it becomes to assess and recover (if possible) from damage done, up to the point where things just aren't considered salvageable anymore.


Unfortunately "strange files and directories" may mean anything ranging from temporary files for benign local processes (manual pages, locate, PHP, etc, etc) to hostiles having succeeded in downloading something. Unless modified later on, listing file details using 'stat' shows the account the file was downloaded with, listing open files ('fuser -v', 'lsof -Pwlnp [PID]') will do just that: check which process has which files in use and running 'file' on files will show you what the purpose is regardless of easy obfuscation like extensions wrt face value. BTW Mech is EnergyMech, a commonly encountered IRC bot, and Juno is a TCP flooder.


This points to a few things both unfortunate and potentially fortunate. First of all, as I said before, deleting files only aids the perp. Unfortunately that's a reflex even knowledgeable server admins suffer from. Secondly while files may be executed in different ways this could point to /tmp not being a separate partition and not mounted with the "noexec" option. As for the ports themselves I don't know what process uses 54377 but 6667 and 6669 are commonly used for IRC. Finally the Apache cronjob means any process started from this cronjob (and crontabs are easily modified by just reading in another file) in the limited context of the web server account. That's common behaviour seen on compromised web servers but do note this does not rule out the perp having managed to escalate privileges later on.


Unless you don't care for statistics and auditing log files shouldn't be deleted but properly rotated. As for software and its problems you should check out your distributions repositories for updates, the vendors web site or CVE to find out.


Rootkit Hunter is but AFAIK Chkrootkit hasn't been updated in ages. What's more both are post-incident auditing tools, meaning you run them after the fact. The emphasis should be on preventing incidents instead, meaning a well-chosen set of tools, configured for limited access, properly hardened and audited regularly.


While this is a typical view held by any owner I have to point out that by not properly taking care you potentially cause trouble for others. And since Linux is a multi-user, networked OS used predominantly in the server role this spells trouble for all of us. I don't know your reasons for choosing Linux but often people seem to choose it because one doesn't need to pay any licensing fees. But not having to part with money doesn't mean you don't need to invest anything else.
Linux may be free to use but using it is not free of responsibilities.

2 members found this post helpful.

My bad.

A particular thing caught my attention above all: that there are continuous outgoing connections on ports like 6667 6669 or high ports like 54377.

... and the owner of the site didn't expect it and didn't know why. Yes, "54377" could be anything, and the other two are well-known port numbers. But an outgoing connection that the owner doesn't expect to see, on a server "that is used to host some websites" (note no mention of IRC), is sufficient reason for me at least to believe that the owner has lost control of the system.

Your comments in post #7 et al are good advice, and I agree with your approach. I will nevertheless hold to my intuition, at least, that control of this server has been very badly usurped and might have been so for some time.

No I don't handle it the right way, but I hope to in some future time (if needed). My posts number is no measure of whatsoever knowledge.
I certainly agree that proper handling of incidents can lead to the solution (I was just trying to ... well ... help ... my way to stop and then investigate)

Good luck roby84
I hope you get out of the troubles.


Regards to all.

@sundialsvcs: thanks for not taking no offense but that text wasn't meant in reply to you! I intended to separate replies over two posts but made the 2nd one too soon after the first one. vBB has this nifty feature that tries to stop people from making posts in rapid succession. Unfortunately it worked too well this time.
If you don't mind I have edit out that part of your reply to reflect this.


...let's find out I'd say.

At sundialsvcs and lithos, in my opinion you are both very helpful and make major, valuable, contributions to the security forum. Pleaase do not take unSpawns post as a personal criticism. I think that you both have the right idea, that there is a suspicious and unexpected process and that it should be investigated with netstat. In addition, blocking connections is the right thing to do in a suspected compromise, but instead of killing the process it is better to either unplug the network cable or raise a firewall. The reason being is that you want to try to use lsof, ps, and netstat to try to find the entire process tree and correlate the output and killing the process will prevent this. With respect to the fact that too many people plopping plesk on a vhost and running a public server, without understanding their role or responsibilities even if they are willing to perform them, you are absolutely 100% correct. You are also correct that they may have to create a new server and when they do should learn to harden it from the start. We generally like to avoid mentioning this upfront because too many people and places simply wipe and reinstall rather than investigate how they became compromised (and in my opinion this is akin to having unprotected sex in that you hope nothing happens but are taking a risk).

edit: it looks like unspawn beat me to replying

2 members found this post helpful.

The OP finally replied to say they hired somebody to fix things for them. That's a week of wasted time and more importantly customers remaining unprotected (and probably unaware of what happened).

Blast !

I was looking forward to a good read

I wish he saved it and I hope he could possibly post some of the solutions or maybe what/ where was the cause.

Good luck