ServePath Blog

So I’ve been thinking lately about the title “System Administrator.” This is our official job title (it says Systems Administrator on our business cards–I guess the extra s is a nod to the fact that we have some 2500 systems in the data center). This is a slightly misleading title, however. I’m not really a system administrator as much as I am a system medic. I only see servers when they’re sick, I do whatever it takes to fix them as fast as possible, and I (hopefully) never see them again.

From what I’ve seen (working for ServePath, but actually far more often on IRC), people tend to think this is what a system administrator does.

It isn’t.

Just because a server is online doesn’t mean it is properly administered. This is akin to saying that if you’re alive, you must be healthy.

There are two very broad areas a server needs to be tuned for after its services have been set up, security and performance.

If you’re a sysadmin for, say, a FreeBSD server, some questions I might have regarding security are

• Do you know what version of SSL/SSH you have installed? Do you know whether you need to upgrade? Do you know how to upgrade these without breaking anything?
• Do you know what ipfw is, and how to use it?
• Do you know what pf is, and how to use it?
• Do you know what termlog is, and do you use it? Why?
• What logs do you keep, and where do you keep them?
• Do you know what a jail is, and should you be using them?

For performance,

• Which processes take up most of your resources, and which resources (disk I/O, network, CPU, etc)?
• At what point is a process taking too many resources?
• Do you know what inodes are? Do you have enough? How would you get more? (I had a client run out of inodes on two different file systems.)
• Do you know why /usr, /, /tmp, and /var are all on separate slices by default? When might you want to change this?
• What would you do if directories are taking a long time to list their contents?
• What network services do you run, and what kind of network performance do you get? How could you adjust your network buffers to get better performance? What about your firewall rules?
• Do you know what RFC1323 is, and when you’d need what it specifies?

Ultimately a server needs rather a lot of attention to be performing well and be secure. If you just turn a server on and plop it online, you’re probably not getting out of it all that you could.

And you’re also probably hosting movies for kids on IRC, even if you don’t know it.

Here are a couple vid clips of some data center workers moving the new Liebert CRAC Units into the data center. These units are so big they had to take the back of the elevator off to fit the thing inside!

Every morning I would walk through our data center to get to my desk. I always liked walking through there, just to see what was going on. Sometimes a colo customer would be in there setting up his server, or the data center technicians would be at work on our Cat6 cabling upgrade. On a warm day it was nice to get cooled down even if the noise was a bit overwhelming.

But since we recently moved to a new office to make room for our data center expansion, I no longer get my morning walk-through. Instead, I have a sweet window cube with lots of sunlight and a view of the street, so that certainly makes up for it. And, of course, I can always pop over to the data center whenever I want to see what’s going on.

By moving our sales, marketing and accounting department we freed up 3,000 square feet of data center space. This will allow our existing customers to expand as they grow their business, plus it will enable us to meet increasing demand for San Francisco data center space. The expanded space features Cat6 cabling designed to support applications such as IPTV, and it allows up to 10 gigabits per second hand offs to our cabinets or cages.

This is what my former office looks like now:

I wanted to draw everyone’s attention to the fact that we now offer a variety of easy ways of subscribing to this blog using RSS feeds. As you can see on the right hand side of the page (scroll down a bit), we now offer everything from Bloglines to MyYahoo! and Feedster.

So come get your RSS feeds while they’re hot!