ServePath Blog

“It is a truth universally acknowledged…”

Once, I licked the bottom of my foot.

I won’t claim that my youth wasn’t full of foibles, but I like to think I have avoided many silly mistakes and that, on the whole, I have fairly good judgment. I bet you like to think that, too, since I fix your servers when they’re down.

Despite whatever more rational faculties I may have, there are times when judgment is overcome by other things, (I hasten to assure you that this never happens with servers or networks, unless you count the time in college when I decided to see if I could get better network performance by setting my NIC to 100MbpsFD instead of 10Mb. It crashed the hub, and my dorm was without net access for a day. Who knew?) and it’s during those times that I can be found buying soda cans from vending machines to drop them down stairwells.

Anyway, the point is that once I licked the bottom of my foot. But I can understand if you read it wrong. If I was reading some random blog and came across that sentence, I’d think it said “Once, I licked the bottom of my foot.” In fact, what it really says is “Once, I licked the bottom of my foot.” Once was enough. For a lot of the silly things I’ve ever done, once was enough.

Why am I writing about this on a blog that is about a data center? Well, once, I lost all my data. I was repartitioning a drive, or resizing a partition, or doing something in Partition Magic, which is a useful program that will do a lot of low-ish level data organization on your hard drives, when the power went out. My disks were left in an unstable state and I lost everything. This was years and years ago now, but it is also the last time I have ever lost data. I’ve deleted files, had hard drives crash, installed OSs over old ones, but I’ve never lost another bit of data. Once was enough.

In a future post I’ll get some backup basics together; it’s actually more complicated than you might think. Like good security, it requires a number of trade-offs. But for now, I just want to request that the reader think very carefully about whether she has any data to lose and, if so, what kind of precautions should be taken to prevent such a loss.

Before I worked here I wasn’t exactly sure what this kind of job consisted of. Of course this made for a very awkward résumé, since I had to get across that I knew a lot about computers without implying I was best suited for a completely different computer job, such as database management. But even for a while after I joined, there were moments when I was surprised, because what I had been expecting wasn’t anything like what actually happens. So I imagine there may be a lot of you who are speculative or curious about what kind of issues can happen in a data center.

All the issues that I have to deal with can be split into three categories, hardware, software and networking. Since we sell only unmanaged servers and colocation, ideally I’d only be responsible for hardware and networking (a fourth category, environment, is important but not in any way under my purview). However, operating systems being what they are, those things break all the time and of course I have to fix them.

Hardware

The hardware issues we see most often are bad RAM and bad hard drives. Very rarely we have bad RAID cards or NICs, and once I had to replace a CPU. These are always fairly easy fixes, once the problems have been identified. The only real issue is when a customer loses data due to a failed hard drive. RAID can sometimes (but not always) prevent that, which is one of a million reasons why backups are so necessary.

Under hardware I’m also going to throw all the scheduled upgrades we do. Fooling with hardware is the easiest part of this job, except for inventory management, which gets tres annoying, but even that isn’t so bad. This is the same monkey stuff you did for your family when you were 14.

Network

I don’t think I’ve seen the network actually break, but customers fall off it all the time. 95% of the time, this is because of Red Hat Linux. Oh man, do I hate Red Hat. Don’t take this personal, if you like, use, work for, or are Red Hat (well, take it personal if you are Red Hat), but the network configuration in this OS is such a mess. So if you use Red Hat, and you reboot, and suddenly you can’t get on the network, it’s because the network scripts, which used to work just fine, thank you, decided they didn’t like where the default gateway was defined, and now expect it to be defined in another of the 735 different network configuration files, which lives in another directory from the file previously used. Haha!

This is, of course, only my opinion.

Usually network upkeep involves protecting our network from customers. If customers get cracked, they tend to become members of zombie networks, and the UDP floods they send out can slow things down for other customers. We tend to head those off by limiting the compromised customer’s connection.

Less often, but not rarely, customers become victims of DoS or DDoS attacks. In fact, there’s one going on right now. If you happen to know 208.185.250.11, tell him I said to knock it off. There are nearly always handled automagically by our network infrastructure, but it’s good to keep an eye on it.

Software

Oh boy. Broken software. Where to start?

Well, there are the usual suspects. By default, Windows will only allow two active Terminal Services sessions at a time (Windows 2003 allows you to connect to the console remotely, which can count as a third session). If you run out of these, and Windows doesn’t reset them for some reason, We have to visit the box to reset them manually.

Control panels have been known to become unstable. This seems to happen when a user tends to be familiar enough with the command line to use that, but also has a control panel installed. The CP can become confused if a file is edited manually. This is why Ensim (for example) changes the motd to inform users that, if they edit files, they have voided their warranty.

Remotely upgrading OSs is also a tricky issue, for example kernel upgrades.

Then there are the day-to-day surprises, like that time up2date got confused and uninstalled OpenSSH.

So there are a myriad of different software issues that actually crop up, but the best way to classify them would be in two categories: those that break the OS and access to it, and those that break the services the server provides. We probably have an 90/10 split between them. Very rarely will we get involved in customer setups; our customers generally prefer to have their own IT staff take care of it.

In a way it’s almost disappointing that we don’t get to do the real Sysadmin work (that is, configure client servers with actual solutions to actual problems, instead of just making sure they’re online). But that would be impractical for the number of clients we have, and they’d basically be paying for our on-the-job training as we learned about their (unique, sometimes bizarre) setups. So probably it’s just as good we don’t.

On Wednesday March 22, ServePath hosted the Juniper Networks Demo Truck on its tour around the country. The demo truck is a completely rebuilt 18-wheeler decked out with the latest and greatest in cutting edge security and routing solutions.

The truck was parked outside our San Francisco data center all day, and there was a steady stream of people coming to check out the latest hardware. Juniper brought a bunch of engineers eager to show off the gadgets, and both industry people and curious pedestrians got a full show.

In the middle of the day, ServePath hosted guest speaker Roslyn Rissler, a member of Juniper’s “Infranet Controller” Management Team. Ms. Rissler gave an interesting talk about Juniper’s Unified Network Access Control solution before we all enjoyed a delicious lunch. ServePath also provided guided tours of our data center throughout the day.

There are many advantages to working on the night shift, but I think the greatest of them that I get to accept responsibility for the entire data center.

There are many disadvantages to working on the night shift, and I know the greatest of them is that the food is awful.

The food is awful because there are only three delivery outfits in San Francisco that are open after midnight (two pizza and one Chinese), and really after two days in any given week I’m already sick of it, but unless I want to come in three hours early (making my shift an even 12 hours) it is all that is to be had. Responsibility, on the other hand, is good because when an issue arises I know I’m the owner, and I have to fix it. Sometimes it is more difficult to make the decision to take ownership than it is to fix the problem; this way the decision is made for me. Anything to help my lazy self.

My name is Toby Burress and I work graveyard in Support from Tuesday through Saturday. I’m going to start posting here to help create a venue where our customers can start the beginning of something like a dialog with ServePath. Right now (as far as I know) the only situations when ServePath and its customers interact is either when (a) ServePath wants to sell something, or (b) something ServePath sold is broken. Neither is a great opportunity for more relaxed communication; thus, this blog.

I have been ruminating for a while on the advantages of the night shift. I took this job specifically because they offered me the graveyard, but then it was simply because I knew I was more alert at those times than during the day. Over time, however, my affection for the early morning hours metamorphosed into an appreciation for the early morning hours here. The night shift is my shift, and at night the data center is my data center. From my pleather throne in the NOC of my Olympus I survey the TCP/IP commerce, negotiations, or sometimes warfare between and among the various black-and-beige citadels you, my subjects and vassals, have established in my dominion. A young system logging process (call it, say, thucydidesd) might see only the acts and effects of lesser system administrators who, selfishly seeking only the good of their estates, maneuver for dominance in the Peloponnese of the Internet, but a greater and wiser program (homerd) would see beyond the immediate scenes of cause and effect, and in the grand course of events this blind process would detect what could only be the effects of my omnipotent and omnipresent hand benevolently guiding the sum of human events through the dark and stormy nights, sometimes censuring, sometimes dispensing luck and wealth, but always to a beneficent end.

Uh, yeah, I have a minor in classical literature. I don’t get to let it out much. Another benefit of the night shift. But when difficulties crop up, as they are bound to in any complicated system (and, hoo boy, some of our systems are complicated), I guess you could say that I take them personal-like. The total merit of this (benefit to the customer, the benefit to ServePath, the effect on my health) can be debated, but I think on the whole it’s good.

A further side effect of owning my own data center (however briefly) is that I always know what is happening to every ticket in the queue. I mean no slight to my fine coworkers (hi guys!) but I always feel more comfortable when I am in charge of an issue. This feeling in me is probably why, while I make a pretty good support peon, I’d probably make a lousy support manager. C’est la vie.

So that’s basically me. Possessive, suspicious, delusions of grandeur—your nightly Support Team. In future posts I hope to get a little more specific about what kind of issues tend to crop up in the NOC, what the solutions are, etc. I also plan to give stern lectures about proper system administration which I hope dearly can be taken to heart without also taking affront. System administration is certainly not glamorous (we don’t get the respect developers do), but there are definitely some deft ways to get out of tricky situations, and at the very least I could use an archive for myself. Who knows, maybe you’ll benefit too.

When deciding where to colocate there are several important factors to take into account. Here are some of the things you should look at before you decide on a data center.

Building Security
Building security is extremely important. You want to look for a building that offers quality security systems and guards on duty 24/7. Make sure that there are video cameras on all floors, so that anyone visiting the building is covered from the minute they enter to the time they leave again. Only authorized personnel should be allowed access to the facility, and there should be at least two ID checks before entering the data center itself. You should also make sure that there are locks on the actual cabinets or cages where your servers will be stored.

Read about ColoServe’s Secure San Francisco Data Center.

Power System
Ideally, the data center should be connected to two power grids with multiple entry points, but obviously you are only likely to find this in major cities. However, even if the facility is only on one power grid you should make sure that there are multiple entry points. You should also ask about generators and power backup plans. I would highly advise against ever colocating at a data center that doesn’t have its own generator(s). It is simply a must for a data center to have on-site generators and fuel, and a really great one will even have more than one.

Our San Francisco data center power is fed by conditioned UPS electricity, with redundant feeds from dual 2 Megawatt generators. One generator can power the entire building for several days time, and there is 20,000 gallons of reserve diesel fuel on site. These generators are tested monthly, and every third month the power is disconnected from the two utility feeds to ensure proper generator operation.

Network
One of the things to consider is your bandwidth requirements. In order to do that you need to know your current end-users (and the users you hope to reach in the future) in terms of where and how they access the Internet and compare that against the network connectivity and cross-connects afforded by each co-location vendor. The quality of the user’s experience is going to depend on how many hops it takes them to reach your site from their ISP. If it seems as if your users are going to need too many hops getting to and from their ISPs and the data center network, it is best to find another colocation facility; one that can better meet your requirements.

Read about ColoServe’s Screaming-Fast NetworkTM.

As a Microsoft Certified Partner, we are proud to be able to offer SQL Server 2005. At ServePath you can choose from SQL Server 2005 Workgroup and Standard editions, but we also offer SQL Server 2005 Express Edition preloaded and FREE on all our Windows Dedicated Servers.

SQL Server Express is the easy-to-use lightweight version of SQL Server 2005, and what is so great about it is that as your needs grow your applications will seamlessly work with the rest of the SQL Server product family.

Microsoft redesigned SQL Server 2005 to better meet customer needs and demands, and that definitely shows. It combines the best in analysis, reporting, integration, and notification. This enables your business to build and deploy cost-effective business intelligence solutions that help your team drive data into every corner of the business through scorecards, dashboards, Web services, and mobile devices.

SQL Server 2005 includes key features that allow you to easily manage and organize your data:
• 64-bit support
• Isolation model optimized for OLTP
• New Management Studio and Integration with Visual Studio 2005
• Failover Clustering

Scalability
• Up to 4 Processors
• Up to 4 GB RAM
• No limit on the size of the databases

Learn more about ServePath’s Microsoft SQL Server 2005 Dedicated Servers.

We have undertaken the huge project of upgrading the wiring in our data center with category 6 copper cable from Berk-Tek. This project is going to take several months to complete, and I’ll keep you updated on our progress right here on this blog.

So what does Cat 6 cabling do that Cat 5e doesn’t? Well, to begin with systems operating over Cat 6 cabling will have fewer errors for current applications due to its improved transmission performance and superior immunity from external noise. This means fewer re-transmissions of lost or corrupted data packets under certain conditions, which again translates into higher reliability for Category 6 networks compared to Category 5e networks. In addition to improved data throughput, the superior engineering delivers greater reliability and virtually eliminates “mystery” problems associated with antiquated Cat 5 cable plants that can haunt systems administrators.

The Cat 6 copper cable is specifically designed to absorb ambient energy sources such as Electromagnetic interference (EMI) and radio frequency interference (RFI), eliminating BER loss from these common cable associated problems. ColoServe’s Cat 6 cable plant has twice the bandwidth of Cat 5e systems with positive Power Sum Attenuation to Cross-talk Ratio (PSACR) up to 200 MHz.

But do you really need all that bandwidth since there isn’t yet an application that requires 200 MHz of bandwidth? Maybe not right at this instance, but all signs point to the fact that it won’t be long now. Data rates have been doubling every 18 months, and applications running at 1 GB/s are already pushing the limits of Cat 5e cabling. The demands for faster data rates will definitely increase as streaming media applications such as video and multi-media become more popular and commonplace.

Cabling infrastructure is generally considered a 10 year investment as opposed to two or three years for electronics. That’s why we are getting started on our wiring upgrade now. As always, it makes sense to plan ahead and to be prepared for whatever the future might bring.