Note: this post has been updated slightly to correct some minor errors. Also, commands may have been improperly formated due to WordPress’s treatment as such. Converted now to “code” formating.

1. Software Firewall – For security purposes, the software-based firewall that is included in all freshly deployed dedicated server operating system has been enabled and configured to allow on the minimal amount of connectivity required for you to access and configure your server. For Linux/UNIX users, this means that port 22 is permitting SSH connections. Port 80 (HTTP/Web) and port 443 (HTTPS/SSL Web) have been opened to allow all standard web traffic. In addition, the required ports for control panel access have been opened if you have ordered a control panel from ServePath. Finally, ICMP Ping has been permitted to allow our monitoring services the necessary access to aid in managing out network.

2. File Permission – There are certain files whose presence in the Linux file system can present a security risk and should be remedied as soon as possible. When the SUID (set user ID) or SGID (set group ID) bits are set on an executable, that program executes with the UID or GID of owner of the file as opposed to the user executing it. This means that all executables with SUID bit set and are owned by root are executed with the UID of root. This situation is a security risk and should be minimized unless the program is designed for this risk. To find all files on your file system that have the SUID or SGID bit set, execute the command:

# find / -path /proc –prune –o –type f –perm +6000 –ls

It is good practice to generate a list of SUID or SGID files on your server as soon as possible and re-run the above command on a regular basis to ensure new binaries with unsafe permissions are not being added to your server. World-writable files are a security risk as well. World-writable files and directories are dangerous since it allows anyone to modify them. World-writable directories allow anyone to add or delete files. To find all world-writable files and directories, execute the command:

# find / -path /proc –prune –o –perm -2 ! –type 1 –ls
 # find / -path /proc –prune –o –perm -2 ! –type l –ls

Another file permission issue is when files are not owned by any user or group. While this is not technically a security vulnerability, an audited system should not contain any unowned files. This is to prevent the situation where a new user is assigned a previous user’s UID so that the previous owner’s files, if any, are all owned by the new user. To find all files that are not owned by any user or group, execute the command:

# find / -path /proc –prune –o –nouser –o –nogroup\
 # find / -path /proc –prune –o –nouser –o –nogroup

3. Listening Ports – It is very important to ensure that all listening ports on your server are limited to only those that are necessary for you’re your server and its applications. To get a list of listening network ports, run the following command:

# netstat –tulp

Disable any ports that are not necessary. To do so, kill the PID (process ID) shown by netstat. The only port that your server must be listening on is SSH (port 22/tcp). Other ports that will need to be listening depend upon the specific purpose of your dedicated server. Note that by killing the PID of the process you are not preventing your server from starting the same service again on bootup. In order to see what programs your server is launching on startup, execute the following command:

# chkconfig –list |grep on (Red Hat systems)
 # chkconfig –list | grep on (Red Hat systems)

# ls -l /etc/rc2.d/S* | cut -d/ -f6 (Debian systems)

This command will show you which programs are to be executed in which run levels. In Red Hat, full multi-user mode is 3. To disable a service permanently, issue the following command:

# chkconfig <service_name> off where <service_name> equals the name of your service, such as httpd

To disable any service in Debian, simply execute the following command:

# rm –f /etc/rc2.d/S*<service_name>

Please note that the above commands do not actually disable the service, they simply prevent the service from being executed on startup.

4. Unlocked User Accounts – The first thing you should take stock of on a new server are the users with unlocked accounts. Users with unlocked accounts are allowed to login if assigned a valid shell, and should be kept to a minimum. To get a list of unlocked users, execute the following command:

# egrep –v ‘.*:\*|:!' /etc/shadow|awk -F: '{print $1}'
 # egrep –v '.*:\* | :!' /etc/shadow | awk -F: '{print $1}'

If you do not recognize any user returned by the above command, check to see if that user owns any files by executing the command:

# find / -path /proc -prune -o -user <user_name> -ls where <username> is the name of the user you do not recognize, such as jdoe

If the user does not own any files, or files that will not hinder the stability of your server, delete the user by executing the command:

# userdel –r <user_name>

5. Enable/Disable Features – All of the following lines and values should be added to the file /etc/sysctl.conf if you want to enable or disable the feature mentioned. You will need to restart your system for these changes to take effect:

6. Basic Access Control – One of the most important things you can do to protect your server is to implement very basic access control. Access control can eliminate a majority of the risk involved in running out of date services on the Internet. In order to implement an effective access control policy on your dedicated server, you will need the following pieces of information: The IP address or addresses of your Internet connection. For some, this may be one static address, while for others it is a pool of addresses. If you have more than one Internet connection, please be sure to get ALL the IP addresses you could be assigned at any time. You may need to contact your Internet Service Provider for this information.

7. Restrict SSH Connections – While we do not recommend anybody running outdated software, especially something as crucial as SSH, a not insignificant portion of the risks involved in running an outdated SSH server can be mitigated by only allowing certain IP networks to access your SSH server via iptables. Execute the following command to only allow SSH connections from certain IP address:

# $IPTABLES –A INPUT –p tcp –dport 22 –s <X.X.X.X/NN> –j ACCEPT

The above line will allow TCP packets destined for port 22 to be accepted if and only if the source of the packets are within the network denoted in <X.X.X.X/NN>. If you have more than one Internet connection, or have multiple networks, simply add another line, replacing <X.X.X.X/NN> with the proper values.

8. Access Control on Control Panel – If your server is running a control panel, you can also improve your security by implementing an access control policy on the control panel administrative port.

Plesk:

$IPTABLES –A INPUT –p tcp –dport 8443 –s X.X.X.X/NN –j ACCEPT

Ensim:

$IPTABLES –A INPUT –p tcp –dport 19638 –s X.X.X.X/NN –j ACCEPT

Cpanel:

$IPTABLES –A INPUT –p tcp –dport 2082 –s X.X.X.X/NN –j ACCEPT

9. Access Control on FTP – Another service you may want to implement an access control policy on is FTP. If you or a small handful of people are the only allowed users to FTP into your dedicated server, then you will certainly benefit from employing some iptables rules by entering the commands:

$IPTABLES –A INPUT –p tcp –s X.X.X.X/NN –dport 20 –syn –j ACCEPT
 $IPTABLES –A INPUT –p tcp –s X.X.X.X/NN –dport 21 –syn –j ACCEPT

Note that both of the above lines must be executed for each source network.

10. Enable IPTABLES – Lastly, if you do not have a hardware firewall you will want to enable iptables, the software firewall in Linux systems. For a detailed iptables tutorial from ServePath, please visit our Support Center pages at: http://www.servepath.com/support/iptables.htm.

Hope that helps you get your Linux Server get even more secured! For other helpful tips like this, be sure to visit the ServePath Knowledge Base.

Some users may experience slow FTP performance despite having high-bandwidth connections at both ends. The following hypothetical example illustrates this common issue.

For example, an FTP stream to one of our servers located in Texas from a server located within our network in San Francisco would peak at around 6 to 7 Mbps. This happens even though both machines at either end are connected to the Internet using a 100 Mbps link.

We ran several tests using a freely available utility called iperf, and obtained interesting results. While a single TCP stream would peak at around 6 Mbps, a single UDP stream would consume almost the whole available bandwidth at more than 95 Mbps. However, even though a single TCP stream would peak at around 6 Mbps, we could send multiple TCP streams and use much of the available link capacity.

The question then is why does one TCP stream peak at only 6 Mbps? This actually has to do with how flow-control is handled in a particular TCP implementation. Flow-control in TCP is dependent on window-size and the round-trip time (RTT) or latency, and both attributes can significantly affect TCP performance. For example, with the same link capacity and window size of (35,040 Bytes), if the RTT is 1ms, a TCP stream can use 280 Mbps of bandwidth. Increase the RTT to 50ms, and the TCP performance drops to 5.6 Mbps. Also, even the slightest packet loss can severely affect TCP performance. Our RTT from SF to the server in TX is around 45ms. That would explain the slow FTP performance.

Despite having plenty of bandwidth, we cannot circumvent the latency issue. It will take a certain amount of time for a packet to travel from one geographic location to another. Transmission is ultimately limited by the speed of light, which is not an insignificant factor. Furthermore, the paths over the Internet are not necessarily streamlined according to geography.

So then how can we achieve throughput of greater than 6 Mbps for a TCP stream? There are several solutions:

- Tuning TCP window size
- Using software that splits a single flow into multiple flows
- Leveraging Caching and Content Delivery Networks

There are several freely available papers on the topic of TCP performance over the Internet. This particular one explains it very clearly. Another useful link with vendor solutions.

Latest version of the KB article is here.

Below is a preview:

Title:	KB: Why is My FTP Transfer Speed Slower Than Expected
Description:	Some users may experience slow FTP performance despite having high-bandwidth connections at both ends.

By default, a Linux server doesn’t reboot after a kernel panic but waits for a user to manually reboot the server. To check this behavior, check /proc/sys/kernel/panic.

This represents the amount of time (in seconds) the kernel will wait before rebooting if it reaches a “kernel panic.” A setting of zero (0) seconds will disable rebooting on kernel panic. If the value is set to 1, the server will display the error message that a kernel panic occurred and will do nothing.

Latest version of KB article is here.

Below is a preview:

Title:	KB: Rebooting a Linux Server After a Kernel Panic
Description:	By default, a Linux server doesnt reboot after a kernel panic

Of the four points we’ve listed below, we’ve covered the what and the where; we still need to examine when and how to back up.

How to back up?

As I said below, ideally everything that needs to be backed up is in one spot. Ideally this spot is the backup media, which is in a government-controlled facility at the bottom of a mine under a mountain with four or five OC-768s. The real world is never ideal, unfortunately, so some of this just isn’t possible. However, you can get closer or further from ideal depending on how you set things up.

Try to keep whatever needs to be backed up on a separate partition; on a separate drive if you can manage it. This allows you to use tools such as dump (nww), which is pretty much the best tool for the job (on FreeBSD anyway). You can even keep configuration files in this space. The fewer places you have to hunt for files, the better.

If you use backups as a kind of undelete function (say, if you are backing up personal files and sometimes need to restore a file you’ve deleted), then you should keep incremental backups. Incremental backups store everything that has changed since the last backup. These allow you to make frequent, short, small backups. If your work changes often (for example if you are backing up an office fileserver) you can run such backups several times during the business day. Then, when a file is needed, you can grab it from an older incremental backup.

Alternatively you can make full backups each time. This minimizes restore time, since with only one backup file you can restore all your data.

More complicated structures, such as the Tower of Hanoi differential algorithm that is suggested in dump’s manpage, are nifty but not strictly necessary. They achieve an optimum between space consumed, time to back up, and time to restore.

A further consideration needs to be given to databases. You can’t simply dump the actual database files; when you restore, you’ll find the database is corrupted. Instead, regularly dump the database (a real database can dump without locking whole tables) and then back up the dumpfiles.

When to back up?

At what time to back up isn’t really hard to determine: whenever the load is lightest.

You might want to put some thought, however, into deciding how often to back up. As I said below, not everything need be backed up at once, but only as often as it changes significantly.

If you are making incremental archives or backups, you can run the backup three or four times daily, and then push those archives offsite at night.

Further reading

(all links open in a new window–sorry)

• Backup strategies
• Backup basics

Most of this has been directed at Unix-like operating systems. Some of this applies to Windows as well, though in Windows you can’t simply back up your configuration scripts and then restore them to have your applications work just as you remembered them. If you enjoy paying for software, there are many third party backup utilities availables, such as Veritas Backup Exec. But most of these guidelines should apply.

If you have a dedicated server, it’s almost a certainty that you’ve never seen your server, or have even been in the same city as your server. Thus it is almost a certainty that you can’t show up in person to change tapes and take them offsite. Yet, it’s also probably a pretty good bet that your data is so critical to you or your business that it would not be too much of an exaggeration to say losing it would be a calamity.

How can you ensure your data is safe?

Well first, like security, your backup strategy needs its own plan and its own budget. You can’t toss firewalls at your server to make it secure; neither can you copy files off into the void. There are a number of issues that need to be addressed: what to back up, where to store the data, how to back it up, and when to run backups.

Some of this might apply if you’ve got a colocation account somewhere, but a lot of it is completely different. Physical access to a machine can change circumstances (as well as what services an ISP provides — at ServePath we are far more hands-on with our (unmanaged) dedicated customers than we are with our colocation customers).

What to back up?

If you’re paranoid, you might say “everything”, but this really doesn’t make any kind of sense if you’ve got a dedicated server. Since you can’t get to the machine, you’re never going to interact with it unless it has a complete install of some operating system on it. This means that file system-level utilities (such as my all-time favorite, dump (nww)) are pretty useless unless you architect your server’s layout around them.

When deciding what to back up, you have to keep in mind how you’re going to restore. If you simply tar up /, when you restore you’re going to scribble all over system files (and tar has issues with certain kinds of files as well) with old versions. If you were backing up Debian but restoring to FC4, you could easily render your system unusable by restoring to it!

What you want to grab are all your data files, and leave the rest to hang. Data files are actually split into two groups, configuration files and actual data files. Config files live in places like /etc and /var/named. If you can, try to avoid putting them there; use /root or something under /usr/local, such as /usr/local/etc or /usr/local/var. This way you don’t find out, when restoring from a crash, that your 2500 line custom Sendmail script was under /etc, which you didn’t bother to back up. Data files can live in some funny places, but if there’s a service (such as httpd) that owns them, it’s best to give them a user (such as http or www) and put all the relevant files under this user’s home directory. This doesn’t have to be /usr/home/www. This helps to keep track of where everything is.

If money (and therefore space) is an issue, you can decide what can be risked. I don’t back up any of my media files, since the chance of losing them is very low, the cost of losing them is low, and the cost of backing them up is high. If you have a bunch of open source repositories on your site because you like to help out SourceForge or Fresh Meat, then don’t bother to back them up, because you can just download them again. If you have a bunch of open source repositories on your site because your business involves them, as one of our customers’ (nww) does, then you should be backing them up because the time it takes to download them is more expensive than the cost of backing them up.

Where to store the data?

Dedicated customers are in kind of a tricky position here. The answer to this question is almost an unequivocal “offsite”, but if you have enough data even a 100Mbps pipe can’t finish the first backup before the second begins.

Fortunately, you don’t need to pull your backups offsite every day. You can automatically perform remote backups weekly or even monthly, depending on your data, its temporal value, and the rate at which it updates or changes. For example, a group of servers that powers a small online store can backup the transaction database nightly, but the contents of the store itself probably don’t update nearly as often, and the actual scripts that run the store are even less critical. All need to be backed up, since without any one of them the site is offline, but they have different priorities.

The whole reason for pulling backups offline is so that if there is a major disaster, recovery is still possible. ServePath itself is fairly secure (impressively so, I’ll blog about it sometime), so a disaster big enough to destroy your onsite backups and your server would probably level this building and most of the rest of San Francisco. Still, if it happens, you still want your business to be able to pick itself up. So take the backups offsite at least once in a while.

Where to keep the data while it’s here? I want to take this time to clear up a misconception about RAID. RAID is not backups. They do not protect against the same kinds of errors. Backups provide crude protection against mistakenly deleted files, hackers, corrupt file systems, and hardware failures. RAID provides graceful protection against hardware failure, sometimes. Sometimes RAID simply improves performance. You can’t assume that RAID 1 (mirrored drives) will save your data, because if there is some other hardware failure, like the power supply, and your file systems are corrupted, then both drives are gonners. What RAID does let you do is to notice that a drive has failed, and then schedule downtime to replace that drive. It turns uncontrolled downtime into controlled downtime. Usually.

ServePath really offers three choices for onsite backups: a second drive in the same server, a second server which mirrors the first, or space on a NAS drive. I’m not sure what the standard offerings among dedicated server hosts are but I’m sure they’re much the same.

A second drive is a pretty good idea if you’re most often recovering from deleted files. Transfers are quick, both for backup and restore. However, if the server is offline and you need access to the data, you’re out of luck. And since really anything that affects the first drive can affect the second drive, too, it is not entirely safe. Very recently I had to help a customer restore his OS after a script ran “rm -r /usr”. Fortunately, he had other backups.

Something like a NAS prevents most of the troubles that a second drive can run into. Since data can be copied via rsync or ftp, it allows archives to be made without exposing them to the file system for errant scripts to destroy. Since it is on a LAN link with the server, backup and restore is relatively quick. This can be thought of as “semi offsite,” since even if your server explodes, your data will probably still be okay (depending of course on the energy in the explosion and where your computer is in the data center).

A second server is the most expensive but also the most reliable solution. With proper load balancing configured, any single server could die and your customers wouldn’t even notice, because the other servers would pick up the load. As long as all servers in the pool have all the data, no single failure should affect the rest.

Okay this is getting long. I’ll return to it next week.