ServePath Blog

With hacker and network attacks commonplace within the computer industry, ServePath, has taken further measures to combat security threats through an on-going partnership with Fortinet. Offering a variety of managed security products and services, ServePath continues its desire to provide its customers the most robust threat management packages available in the managed hosting industry.

Built using features within Fortinet’s FortiGate appliance, ServePath now offers its customers a Unified Threat Management suite though Fortinet’s virtual chassis-based firewall. ServePath recently launched an updated, more feature-rich yet less expensive combined threat prevention package with the following services:

• ServePath Internet Security Package (using Fortinet technology) includes:
  • Fortinet ASIC accelerated firewall
  • Unlimited IP addresses in a trusted interface
  • 100 Mpbs Internet traffic throughput
  • 100,000 concurrent sessions
  • Intrusion Prevention Services (ISP)
  • AntiVirus/AntiSpyware/AntiMalware Protection

Originally priced at $449.99 per month with a $299.99 setup charge, the ServePath Internet Security Package has been reduced in price to $99.99 per month with a $99.99 setup charge. This drastic reduction in price reflects ServePath’s desire and commitment to ensuring its users have the latest and most advanced threat prevention package available.

If you have questions or would like to order this for your managed hosting solution at ServePath, please contact a ServePath Sales representative at: 1-866-321-PATH (7284). International callers should call: +1-415-869-7000. Or you can fill out the contact form.

Note: this post has been updated slightly to correct some minor errors. Also, commands may have been improperly formated due to WordPress’s treatment as such. Converted now to “code” formating.

1. Software Firewall - For security purposes, the software-based firewall that is included in all freshly deployed dedicated server operating system has been enabled and configured to allow on the minimal amount of connectivity required for you to access and configure your server. For Linux/UNIX users, this means that port 22 is permitting SSH connections. Port 80 (HTTP/Web) and port 443 (HTTPS/SSL Web) have been opened to allow all standard web traffic. In addition, the required ports for control panel access have been opened if you have ordered a control panel from ServePath. Finally, ICMP Ping has been permitted to allow our monitoring services the necessary access to aid in managing out network.

2. File Permission - There are certain files whose presence in the Linux file system can present a security risk and should be remedied as soon as possible. When the SUID (set user ID) or SGID (set group ID) bits are set on an executable, that program executes with the UID or GID of owner of the file as opposed to the user executing it. This means that all executables with SUID bit set and are owned by root are executed with the UID of root. This situation is a security risk and should be minimized unless the program is designed for this risk. To find all files on your file system that have the SUID or SGID bit set, execute the command:

# find / -path /proc –prune –o –type f –perm +6000 –ls

It is good practice to generate a list of SUID or SGID files on your server as soon as possible and re-run the above command on a regular basis to ensure new binaries with unsafe permissions are not being added to your server. World-writable files are a security risk as well. World-writable files and directories are dangerous since it allows anyone to modify them. World-writable directories allow anyone to add or delete files. To find all world-writable files and directories, execute the command:

# find / -path /proc –prune –o –perm -2 ! –type 1 –ls
 # find / -path /proc –prune –o –perm -2 ! –type l –ls

Another file permission issue is when files are not owned by any user or group. While this is not technically a security vulnerability, an audited system should not contain any unowned files. This is to prevent the situation where a new user is assigned a previous user’s UID so that the previous owner’s files, if any, are all owned by the new user. To find all files that are not owned by any user or group, execute the command:

# find / -path /proc –prune –o –nouser –o –nogroup\
 # find / -path /proc –prune –o –nouser –o –nogroup

3. Listening Ports - It is very important to ensure that all listening ports on your server are limited to only those that are necessary for you’re your server and its applications. To get a list of listening network ports, run the following command:

# netstat –tulp

Disable any ports that are not necessary. To do so, kill the PID (process ID) shown by netstat. The only port that your server must be listening on is SSH (port 22/tcp). Other ports that will need to be listening depend upon the specific purpose of your dedicated server. Note that by killing the PID of the process you are not preventing your server from starting the same service again on bootup. In order to see what programs your server is launching on startup, execute the following command:

# chkconfig –list |grep on (Red Hat systems)
 # chkconfig –list | grep on (Red Hat systems)

# ls -l /etc/rc2.d/S* | cut -d/ -f6 (Debian systems)

This command will show you which programs are to be executed in which run levels. In Red Hat, full multi-user mode is 3. To disable a service permanently, issue the following command:

# chkconfig <service_name> off where <service_name> equals the name of your service, such as httpd

To disable any service in Debian, simply execute the following command:

# rm –f /etc/rc2.d/S*<service_name>

Please note that the above commands do not actually disable the service, they simply prevent the service from being executed on startup.

4. Unlocked User Accounts - The first thing you should take stock of on a new server are the users with unlocked accounts. Users with unlocked accounts are allowed to login if assigned a valid shell, and should be kept to a minimum. To get a list of unlocked users, execute the following command:

# egrep –v ‘.*:\*|:!' /etc/shadow|awk -F: '{print $1}'
 # egrep –v '.*:\* | :!' /etc/shadow | awk -F: '{print $1}'

If you do not recognize any user returned by the above command, check to see if that user owns any files by executing the command:

# find / -path /proc -prune -o -user <user_name> -ls where <username> is the name of the user you do not recognize, such as jdoe

If the user does not own any files, or files that will not hinder the stability of your server, delete the user by executing the command:

# userdel –r <user_name>

5. Enable/Disable Features - All of the following lines and values should be added to the file /etc/sysctl.conf if you want to enable or disable the feature mentioned. You will need to restart your system for these changes to take effect:

6. Basic Access Control - One of the most important things you can do to protect your server is to implement very basic access control. Access control can eliminate a majority of the risk involved in running out of date services on the Internet. In order to implement an effective access control policy on your dedicated server, you will need the following pieces of information: The IP address or addresses of your Internet connection. For some, this may be one static address, while for others it is a pool of addresses. If you have more than one Internet connection, please be sure to get ALL the IP addresses you could be assigned at any time. You may need to contact your Internet Service Provider for this information.

7. Restrict SSH Connections - While we do not recommend anybody running outdated software, especially something as crucial as SSH, a not insignificant portion of the risks involved in running an outdated SSH server can be mitigated by only allowing certain IP networks to access your SSH server via iptables. Execute the following command to only allow SSH connections from certain IP address:

# $IPTABLES –A INPUT –p tcp –dport 22 –s <X.X.X.X/NN> –j ACCEPT

The above line will allow TCP packets destined for port 22 to be accepted if and only if the source of the packets are within the network denoted in <X.X.X.X/NN>. If you have more than one Internet connection, or have multiple networks, simply add another line, replacing <X.X.X.X/NN> with the proper values.

8. Access Control on Control Panel - If your server is running a control panel, you can also improve your security by implementing an access control policy on the control panel administrative port.

Plesk:

$IPTABLES –A INPUT –p tcp –dport 8443 –s X.X.X.X/NN –j ACCEPT

Ensim:

$IPTABLES –A INPUT –p tcp –dport 19638 –s X.X.X.X/NN –j ACCEPT

Cpanel:

$IPTABLES –A INPUT –p tcp –dport 2082 –s X.X.X.X/NN –j ACCEPT

9. Access Control on FTP - Another service you may want to implement an access control policy on is FTP. If you or a small handful of people are the only allowed users to FTP into your dedicated server, then you will certainly benefit from employing some iptables rules by entering the commands:

$IPTABLES –A INPUT –p tcp –s X.X.X.X/NN –dport 20 –syn –j ACCEPT
 $IPTABLES –A INPUT –p tcp –s X.X.X.X/NN –dport 21 –syn –j ACCEPT

Note that both of the above lines must be executed for each source network.

10. Enable IPTABLES - Lastly, if you do not have a hardware firewall you will want to enable iptables, the software firewall in Linux systems. For a detailed iptables tutorial from ServePath, please visit our Support Center pages at: http://www.servepath.com/support/iptables.htm.

Hope that helps you get your Linux Server get even more secured! For other helpful tips like this, be sure to visit the ServePath Knowledge Base.

So I’ve been thinking lately about the title “System Administrator.” This is our official job title (it says Systems Administrator on our business cards–I guess the extra s is a nod to the fact that we have some 2500 systems in the data center). This is a slightly misleading title, however. I’m not really a system administrator as much as I am a system medic. I only see servers when they’re sick, I do whatever it takes to fix them as fast as possible, and I (hopefully) never see them again.

From what I’ve seen (working for ServePath, but actually far more often on IRC), people tend to think this is what a system administrator does.

It isn’t.

Just because a server is online doesn’t mean it is properly administered. This is akin to saying that if you’re alive, you must be healthy.

There are two very broad areas a server needs to be tuned for after its services have been set up, security and performance.

If you’re a sysadmin for, say, a FreeBSD server, some questions I might have regarding security are

• Do you know what version of SSL/SSH you have installed? Do you know whether you need to upgrade? Do you know how to upgrade these without breaking anything?
• Do you know what ipfw is, and how to use it?
• Do you know what pf is, and how to use it?
• Do you know what termlog is, and do you use it? Why?
• What logs do you keep, and where do you keep them?
• Do you know what a jail is, and should you be using them?

For performance,

• Which processes take up most of your resources, and which resources (disk I/O, network, CPU, etc)?
• At what point is a process taking too many resources?
• Do you know what inodes are? Do you have enough? How would you get more? (I had a client run out of inodes on two different file systems.)
• Do you know why /usr, /, /tmp, and /var are all on separate slices by default? When might you want to change this?
• What would you do if directories are taking a long time to list their contents?
• What network services do you run, and what kind of network performance do you get? How could you adjust your network buffers to get better performance? What about your firewall rules?
• Do you know what RFC1323 is, and when you’d need what it specifies?

Ultimately a server needs rather a lot of attention to be performing well and be secure. If you just turn a server on and plop it online, you’re probably not getting out of it all that you could.

And you’re also probably hosting movies for kids on IRC, even if you don’t know it.

This morning at approximately 10:00 AM EDT Hosting Matters was hit by a massive DoS attack apparently directed at several conservative blogs hosting with them. Hosting Matters announced in their forum that a specific web site was the target of the attack but did not identify the target by name. The web site in question was isolated from the rest of the Hosting Matters network but the attack still managed to affect several conservative blogs such as Instapundit and LittleGreenFootballs. According to blogger Michelle Malkin, who has been the target of previous DoS attacks but was not affected by this one, the targeted blog was Aaron’s CC. Most of the blogs affected seem to be back up and running now, unlike the original target.

According to reports, the attack originated in Saudi Arabia, although that doesn’t necessarily mean that the perpetrators were Saudi, just that the computers they hacked were situated in Saudi Arabia.

At ServePath we were able to deflect a similar attack thanks to our Riverhead Networks equipment which did an excellent job protecting our network and our customers. At about 8:30 AM PDT we started getting alerts which indicated an inbound flood of over 500,000 packets per second. Fortunately, the Riverhead system took over the incoming flow to the attacked IP and started mitigation pretty much immediately. The attack is still going on as I type this, but seems to finally be tapering off.

Bruce Schneier made the New York Times. This is for his “movie plot” contest: he’s tired of seeing resources diverted to stopping terrorist attacks that are gandiose and easy to imagine, but difficult to execute and thus unlikely.

In the computing world, Schneier is well known for his various books on security and cryptography. He’s probably best known for the Blowfish cypher, which I’d bet real money you use every day. I enjoy reading his blog, which is enjoyable and usually insightful. He thinks “big picture” security; for him, nearly everything has security implications.

Security is a big deal, and it’s an issue you can’t ignore. Unfortunately, most of us, as Schneier says, think in terms of movie plots. Hands up everyone who has taken steps to keep someone from snooping on their TCP sessions. This includes all of you who use ssh and shy from telnet. After all, telnet is unencrypted! Would you type your password into a telnet session? Only morons use telnet.

Right.

But do you know what the chances are of someone actually snooping on your telnet session are? Over the internet? If you are a bank, yes, you have a legitimate reason to encrypt everything. If you are mail.my-family-domain.net, then encryption is probably overkill. Now, of course, ssh is extremely easy to use, and in fact is even more convenient than telnet for, well, everything, but I’ve seen people get lambasted for using telnet when, really, who cares?

(Now, I’m guilty of this, too. When I back up my personal finances, I always encrypt them. I don’t have the skills to determine whether the program I use (ccrypt) is actually securely encrypting the files, I don’t care whether anyone really DOES read the backups, and I don’t think anyone cares to read the backups, but I do it anyway.)

The problem, though, isn’t paranoia. I mean, it’s a little bit of a problem; last night a customer locked himself out of his server, and asked me to open the firewall. It took far longer than it should have because he was using TWO firewalls. But generally paranoia is fine. So you’d rather use a dedicated line than a VPN. Great. If you have the cash for it, go right ahead. But what if you aren’t swimming in monies, but you buy that line anyway? After all, several banks recently got ripped off because their ATMs have been configured to go over the internet instead of dedicated lines, and now they’re being hacked. I know we only sell used automobiles, but I don’t like taking risks.

Meanwhile, our employees are writing their passwords down on stickeys. Or better yet, they all just use the same account.
This is the issue. The other night I had a colocation customer call frantically asking that some (former?) employees be taken off his access list. (The access list is the list of people allowed into the building.) He then wanted me to copy all his data off his server, in case something should happen to it; he had no backups. Going over this later, I determined he would probably have paid about $10k in emergency fees–minimum–for the kind of work he was requesting. Fortunately he had hot swappable drives configured in a RAID mirror. I removed one drive from each of his servers: there were his backups.
This customer probably has a firewall, but I doubt whether it has ever done more than repel automated ssh attacks. On the other hand, who looks at an employee–maybe even a friend–and says, “One day this person might be the most dangerous attacker I get.”?

Okay, so that’s also a movie plot scenerio. Don’t pour ALL your resources into defending from ex-employees and delimiting the access of current ones. You’ll only make the people who work for you ineffective and frustrated, and you’ll be taking money from legitimate departments.

As Schneier says, security is always a trade-off. You can’t say, “we have to defend against this at all cost”, where this is some horror story on reddit. You actually have to spend some time identifying appropriate threats, the costs of defense and disaster recovery, and determine how many resources any given attack actually warrants.

But if you do decide that you really do need two firewalls, please be kind and inform Support.

“It is a truth universally acknowledged…”

Once, I licked the bottom of my foot.

I won’t claim that my youth wasn’t full of foibles, but I like to think I have avoided many silly mistakes and that, on the whole, I have fairly good judgment. I bet you like to think that, too, since I fix your servers when they’re down.

Despite whatever more rational faculties I may have, there are times when judgment is overcome by other things, (I hasten to assure you that this never happens with servers or networks, unless you count the time in college when I decided to see if I could get better network performance by setting my NIC to 100MbpsFD instead of 10Mb. It crashed the hub, and my dorm was without net access for a day. Who knew?) and it’s during those times that I can be found buying soda cans from vending machines to drop them down stairwells.

Anyway, the point is that once I licked the bottom of my foot. But I can understand if you read it wrong. If I was reading some random blog and came across that sentence, I’d think it said “Once, I licked the bottom of my foot.” In fact, what it really says is “Once, I licked the bottom of my foot.” Once was enough. For a lot of the silly things I’ve ever done, once was enough.

Why am I writing about this on a blog that is about a data center? Well, once, I lost all my data. I was repartitioning a drive, or resizing a partition, or doing something in Partition Magic, which is a useful program that will do a lot of low-ish level data organization on your hard drives, when the power went out. My disks were left in an unstable state and I lost everything. This was years and years ago now, but it is also the last time I have ever lost data. I’ve deleted files, had hard drives crash, installed OSs over old ones, but I’ve never lost another bit of data. Once was enough.

In a future post I’ll get some backup basics together; it’s actually more complicated than you might think. Like good security, it requires a number of trade-offs. But for now, I just want to request that the reader think very carefully about whether she has any data to lose and, if so, what kind of precautions should be taken to prevent such a loss.

ColoServe’s parent company ServePath has just published some new “Security Checklists” for Red Hat, Debian, FreeBSD, Windows 2003 and Windows 2000 which lots of people are finding helpful reference guides for keeping security vulnerabilities closed. These were written by one of our Senior Systems Engineers, who has drawn on his experience to put together these quick reference pages. With over 3000 servers under management in our San Francisco data center, he has seen it all, and shares some of the basics at the links above. Keep safe out there in the wilds of the Internet!

For security reasons, only customers listed on the access list are allowed into ColoServe’s building and San Francisco data center. It is not sufficient to be escorted by someone on the access list, unless that person is a ColoServe employee. If you need to bring in a guest or contractor, please notify us ahead of time by opening a support ticket and we can add people to your access list account temporarily.

Please remember that current photo identification is required, so a driver’s license or passport must be left with the guard at the front desk for entry. Card keys are now being issued for on-site use at the guard desk, so you will not need to check in a second time at our data center entrance on the second floor.

For assistance with the access list, or technical support requests, please open a support ticket at My.ColoServe.com

For Emergencies, please do not hesitate to call our NOC number. We ask that whenever possible you open a ticket first so that we can properly track your issue.

ServePath is proud to announce our new MailFilter service, available exclusively to customers on our network.

ServePath MailFilter is a fully-managed, network-based content filter designed to help businesses and other domain administrators deal with the growing problem of unsolicited Email (spam) and viruses. Simply point the MX records for your mail server at ServePath’s MailFilter servers and this reliable and constantly updated system does the rest.

Best of all, this new service is one flat-rate for an entire domain.

Spam filtering is often sold for $5.00 / month per mailbox, but the wholesale ServePath price has no per user fees.

Learn more about the MailFilter>