Here lately, I have seen tons of web applications get exploited. Just by individuals not updating 3rd party extensions or applying patches.If you take security seriously you should do these things. Most people are afraid of other things breaking when they update, so theyjust leave it alone. It introduces attack vectors for that particular server. My philosophy behind it all is, update your servers. If something breaks, it can be fixed. Would you rather take a little more time and do things right, or let your server get compromised?
Here are some ways to check for signs of your server being hacked:
A favorite place for bots to hide are in /tmp and in /var/tmp/ or /dev/shm/ or in a users home directory. Sometimes it may be hidden like /tmp/". ."/ or similar. Keep a copy of it, you may be able to look at the code to see where the attack originated from.
So make sure you check these directories, run:
ls -l -a
This will list all files, including hidden files. Also, take note of the timestamps on the files.
You may also be able to find bots hiding by running the following commands:
find / -exec grep -l "sybnc" {} +
find / -name "*.set" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
find / -name "inst" | perl -pe 's/.\/\w+-(\w+)-.*/$1/' | sort | uniq
Check to see if netstat finds any remote connections between the port range 6660-7000:
netstat -tanp
My server isn't compromised, but here is what the output should look like if you run command above:
#BEGIN TERMINAL OUTPUT
root@bt:~# netstat -tanp
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 127.0.0.1:7175 0.0.0.0:*
LISTEN 1205/postgres
tcp 0 0 0.0.0.0:80 0.0.0.0:*
LISTEN 1264/apache2
tcp 0 0 127.0.0.1:631 0.0.0.0:*
LISTEN 1100/cupsd
tcp6 0 0 ::1:7175 :::*
LISTEN 1205/postgres
tcp6 0 0 ::1:631 :::*
LISTEN 1100/cupsd
#END TERMINAL OUTPUT
If you do find a connection to a remote port within the ranges of 6660-7000, run this command to find out what user/process it is running under:
lsof -i tcp:
Replace
above with one you want to investigate.
Start looking for any suspicious entries in your Apache error log associated with the timestamps of the files, if you found any, in /tmp, /var/tmp/ or /dev/shm/ or in a users home directory.
Here is an example of a suspicious entry:
217.170.53.79 - - [03/Dec/2011:19:48:13 +0100] "POST /phpmyadmin/index.php?session_to_unset=123token=42..._SESSION[!bla]=%7Cx...
This one exploited phpMyAdmin. Take a look at the article located here
http://pastebin.com/DUBeKQv5 on how to secure phpMyAdmin.
Setup Fail2ban. Fail2ban scans log files like /var/log/pwdfail or /var/log/apache/error_log and bans IP addresses that make too many
password failures. It updates firewall rules to reject the IP address.
Implement OSSEC-HIDS
http://www.ossec.net with inotify() to watch changes to your system and Apache directories including those that are HTTP writable.
Another good security step is to mount /tmp with noexec,nosuid,nodev
Here's how:
Check to see if your fstab has a /tmp file mounting option:
cat /etc/fstab | grep /tmp
The command above will display NO output if you do not have /tmp
mounted in its own partition.
cd /dev
Now we are going to create a 100MB file for our /tmp partition:
dd if=/dev/zero of=tmpMnt bs=1024 count=100000
#BEGIN TERMINAL OUTPUT
root@bt:/dev# dd if=/dev/zero of=tmpMnt bs=1024 count=100000
100000+0 records in
100000+0 records out
102400000 bytes (102 MB) copied, 0.535322 s, 191 MB/s
#END TERMINAL OUTPUT
Next make an extended filesystem for the tmpMnt file:
mke2fs /dev/tmpMnt
#BEGIN TERMINAL OUTPUT
root@bt:/dev# mke2fs /dev/tmpMnt
mke2fs 1.41.11 (14-Mar-2010)
/dev/tmpMnt is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
25064 inodes, 100000 blocks
5000 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
1928 inodes per group
Superblock backups stored on blocks:
8193, 24577, 40961, 57345, 73729
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
This filesystem will be automatically checked every 21 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override.
#END TERMINAL OUTPUT
Now backup your /tmp directory:
cp -R /tmp /tmp_backup
Check ownership of the files in your /tmp directory, make sure you
keep a note of this:
ls -al /tmp
#BEGIN TERMINAL OUTPUT
root@bt:/# ls -al /tmp
total 52
drwxrwxrwt 11 root root 4096 2011-12-05 11:45 .
drwxr-xr-x 30 root root 4096 2011-12-05 12:39 ..
drwxrwxrwt 2 root root 4096 2011-12-05 11:45 .ICE-unix
drwx------ 2 root root 4096 2011-12-05 11:45 kde-root
drwx------ 2 root root 4096 2011-12-05 11:45 ksocket-root
drwx------ 2 root root 4096 2011-12-05 11:45 orbit-root
drwx------ 2 root root 4096 2011-12-05 11:45 pulse-I5uG1yhc3uav
-rw------- 1 root root 141 2011-12-05 11:44 serverauth.wrvKiuVkuy
drwx------ 2 root root 4096 2011-12-05 11:44 ssh-dFpQYr1373
drwx------ 2 root root 4096 2011-12-05 11:45 vmware-root
drwxr-xr-x 2 root root 4096 2011-12-05 11:44 .winbindd
-r--r--r-- 1 root root 11 2011-12-05 11:44 .X0-lock
drwxrwxrwt 2 root root 4096 2011-12-05 11:44 .X11-unix
#END TERMINAL OUTPUT
Now modify your /etc/fstab file to mount the new partition:
vi /etc/fstab
and insert this line at the bottom:
/dev/tmpMnt /tmp ext2 loop,nosuid,noexec,nodev,
noatime,rw 0 0
What the file should look like (this may differ from what you already
have, but the relevance is the line at the bottom):
# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
#
proc /proc proc nodev,noexec,nosuid 0 0
# / was on /dev/sda1 during installation
UUID=03718980-9d4b-41eb-b8d6-a64c2c4b2017 / ext4
errors=remount-ro 0 1
# swap was on /dev/sda5 during installation
UUID=c7517411-ea02-4002-b641-9651ccf6e046 none swap sw
0 0
/dev/fd0 /media/floppy0 auto rw,user,noauto,exec,utf8 0 0
/dev/tmpMnt /tmp ext2 loop,nosuid,noexec,nodev,noatime,rw 0 0
Now we are going to mount the new filesystem we created with loop,nosuid,noexec,nodev,noatime:
mount -o loop,nosuid,noexec,nodev,noatime,rw /dev/tmpMnt /tmp
Run this command to chmod /tmp:
chmod 1777 /tmp
For precautions, run the remount command to display any errors (no ouput will be displayed if there are not any errors):
mount -o remount /tmp
Now copy everything back from the /tmp_backup directory to the new /tmp partition:
cp -R /tmp_backup/* /tmp/
Check to make sure everything was copied correctly:
ls -al /tmp
#BEGIN TERMINAL OUTPUT
root@bt:/# ls -al /tmp
total 24
drwxrwxrwt 9 root root 1024 2011-12-05 12:50 .
drwxr-xr-x 30 root root 4096 2011-12-05 12:39 ..
drwx------ 2 root root 1024 2011-12-05 12:50 kde-root
drwx------ 2 root root 1024 2011-12-05 12:50 ksocket-root
drwx------ 2 root root 12288 2011-12-05 12:38 lost+found
drwx------ 2 root root 1024 2011-12-05 12:50 orbit-root
drwx------ 2 root root 1024 2011-12-05 12:50 pulse-I5uG1yhc3uav
-rw------- 1 root root 141 2011-12-05 12:50 serverauth.wrvKiuVkuy
drwx------ 2 root root 1024 2011-12-05 12:50 ssh-dFpQYr1373
drwx------ 2 root root 1024 2011-12-05 12:50 vmware-root
#END TERMINAL OUTPUT
Remove the backup directory:
rm -rf /tmp_backup
Remove /var/tmp:
rm -rf /var/tmp
Create a symbolic link to /tmp:
ln -s /tmp /var/tmp
So, there you have ways to check for a compromised system and a few tweaks to security. Have fun!