The Hitchhiker’s Guide to the Galaxy

I would like to offer the following “Guide” of my own to those small companies or startups that are struggling now, on the verge of closing shop or just ready to give up. First, let me restate Adams’ words: “Don’t Panic!” There are methods to keep your company and vision moving forward, maybe in a different direction or perhaps a bit more slowly.

Here are a few things that you may want to consider as you re-architect your survival strategy:

1. Start Now – don’t take a “wait and see” attitude. If you have a great idea, keep moving forward, but DO start your cash conservation immediately. The mere fact that you are searching for information and reading this post is a great sign that you are being active!
2. Outsource – sure, doing things in-house can save you some cash, but only part of the time. Truly evaluate what makes sense (cents) in your day-to-day operations. If it takes your developer a week to do something and an outside “specialist” a day or two, evaluate the costs of both actions. If you aren’t paying your employees and they are doing work in their free time, try to factor in the time-to-market of that approach. While you may save money in up-front costs, you may lose it in terms of beating your competitors to market. Here are a couple factors within “outsourcing” to consider:
   • Human Capital – frequently small boutiques who are experts in a particular field can do things faster and better than you can in-house. They, too, might be effected by the economic downturn and may be willing to cut some of their costs just to have your business.
   • Operations – this can be both on the technical or just the basic company operations side. If you are providing healthcare benefits, see if you can change your plans or even ask your employees if they have other means to get healthcare (e.g., through a spouse).
3. Avoid Capital Expenditures (CapEx) – hardware costs money, lots of it. Of course this all depends on what stage your company is at. If you are just starting, you may be able to get away with repurposing old computers and sharing bandwidth, but as you grow and get closer to having to “prove your value proposition” to investors or even end-users, you do need some sort of infrastructure. Consider using Cloud Computing (e.g., GoGrid) to host your infrastructure, whether it be a development environment or eventually your production infrastructure. By using “the Cloud” you have zero CapEx, no monthly/yearly contracts and are billed by your usage. This is a great way to control your costs and scale only on-demand. You can easily control your capacity, and avoid having your infrastructure sitting around unused or idle.
4. Simplify/Set Realistic Goals – The more complex you make things, the harder they are to undo. Figure out what you really want to do with your business. Is it a hobby or your life work? Do you want to be the next Google? If so, you probably want to rethink that. While it is good to have lofty goals, keep them closer to earth. Did you hire someone who sold you on reaching something unattainable? You may want to reconsider that, hard as it may be, and let those people go. Re-architect your strategies. Clearly identify the most fundamental and basic goals you want to achieve. Sometimes through simplification, you can find a niche that others haven’t. That makes you more viable and attractive. If you are doing something that everyone else is gunning for, and you are struggling, it probably isn’t worth it. You don’t need to throw it all away, but you might want to evaluate what you want to do and choose 1-2 things that are (somewhat) unique. Remember K.I.S.S (Keep It Simple, Stupid).
5. Remain Flexible – the worst thing that you can do in these trying times is not move or be rigid. Being a startup or small company has definite advantages. You have the ability to move much more quickly than larger companies. Look to be flexibility in many areas: strategic direction, product or service feature set, tools and infrastructures and even work ethic. Keep your employees happy by finding out what works best for them. I’m reminded of a story I heard about a hair dresser who was looking for a change. After some soul searching and help from some personality profiles, they became a landscape architect: similar ideas of grooming but in a completely different field (literally). So stay flexible in your own thinking as well.
6. Network & Socialize – as you start to panic, the worst thing you can do is do it alone. Trust me, there are many people and business who are sharing your same concerns. Some may be further down the process of recovery or re-architecting and may be willing to share with you their experiences and what to or not to do. There are so many ways to connect with people nowadays. There is the obvious Social Media (e.g., Twitter, Facebook, FriendFeed, etc.) and I’m a big proponent of these methods. However, in this case, you may really need to just “get out” and talk to people face-to-face. If you live near a big metropolitan area, there is most likely some sort of meetup (check MeetUp.com for example – their video really is great!) or event that meets your needs. Go to them. Start talking to people. (If you are in San Francisco, check out an event I host called StartUpSF.) You may be surprised as to how many great ideas you may get and even how many people really want to help. New strategic partnerships are frequently started at events like this. It’s important to listen to new ideas and see how they apply to your own. Some of the things you hear may influence how you attack other points on this list. Remember too that you can socialize your Public Relations very easily now. Read some experts’ tips and you may save costs and time there too.
7. Conserve Expenses – this is obvious enough. Watch your energy expenditure especially. I mentioned the Cloud before; by outsourcing your IT infrastructure, you can save tremendous costs, especially if you are doing it yourself through your own server rack in your closet. Don’t travel if you don’t need to. With large bandwidth pipes, it’s easy to have video conferencing with almost the same result as a face-to-face. Save the face-to-face for closing the deal. Oh, and pack your own lunch.
8. Don’t Ignore the Rest of the World – the US economy may be seeing some hard times, but there are other markets out there that may want to spend their money with you. Invest some time in reviewing these other markets. Their currency may be a lot stronger than the US dollar, so they may be more willing to pay for your product or service, or even invest in your business. Don’t ignore the fact that with advances in technologies, the world is a much smaller place than we actually realize.
9. Cash Flow Management - this is true on both billables and receivables. If you have existing vendor contracts and they are part of your lifeblood, see if you can renegotiate them. As I mentioned before, you are not alone in this economic crunch and many vendors (and even lenders) would rather renegotiate than lose your business. If your own business model is not getting the traction that you desired, you might want to try to tweak it a bit. Consider offering pre-payment discounts (e.g., have people commit for a longer amount of time but discount it against a monthly or smaller cycle rate). ServePath does this type of thing with managed hosting, by offering “server specials” at a lower cost. Old inventory may be sitting around so leverage it through discounts as well.
10. Keep your Day Job – I had to end with some humor. If your startup is your life-long passion, it is your day job so take these tips to heart. If it isn’t, remember you have to pay your bills somehow.

Do you have any other ideas, tips or suggestions? If so, I would love to hear them. And remember, “Don’t Panic!”

[This was originally posted on the GoGrid blog.]

Built using features within Fortinet’s FortiGate appliance, ServePath now offers its customers a Unified Threat Management suite though Fortinet’s virtual chassis-based firewall. ServePath recently launched an updated, more feature-rich yet less expensive combined threat prevention package with the following services:

• ServePath Internet Security Package (using Fortinet technology) includes:
  • Fortinet ASIC accelerated firewall
  • Unlimited IP addresses in a trusted interface
  • 100 Mpbs Internet traffic throughput
  • 100,000 concurrent sessions
  • Intrusion Prevention Services (ISP)
  • AntiVirus/AntiSpyware/AntiMalware Protection

Originally priced at $449.99 per month with a $299.99 setup charge, the ServePath Internet Security Package has been reduced in price to $99.99 per month with a $99.99 setup charge. This drastic reduction in price reflects ServePath’s desire and commitment to ensuring its users have the latest and most advanced threat prevention package available.

If you have questions or would like to order this for your managed hosting solution at ServePath, please contact a ServePath Sales representative at: 1-866-321-PATH (7284). International callers should call: +1-415-869-7000. Or you can fill out the contact form.

From what I’ve seen (working for ServePath, but actually far more often on IRC), people tend to think this is what a system administrator does.

It isn’t.

Just because a server is online doesn’t mean it is properly administered. This is akin to saying that if you’re alive, you must be healthy.

There are two very broad areas a server needs to be tuned for after its services have been set up, security and performance.

If you’re a sysadmin for, say, a FreeBSD server, some questions I might have regarding security are

• Do you know what version of SSL/SSH you have installed? Do you know whether you need to upgrade? Do you know how to upgrade these without breaking anything?
• Do you know what ipfw is, and how to use it?
• Do you know what pf is, and how to use it?
• Do you know what termlog is, and do you use it? Why?
• What logs do you keep, and where do you keep them?
• Do you know what a jail is, and should you be using them?

For performance,

• Which processes take up most of your resources, and which resources (disk I/O, network, CPU, etc)?
• At what point is a process taking too many resources?
• Do you know what inodes are? Do you have enough? How would you get more? (I had a client run out of inodes on two different file systems.)
• Do you know why /usr, /, /tmp, and /var are all on separate slices by default? When might you want to change this?
• What would you do if directories are taking a long time to list their contents?
• What network services do you run, and what kind of network performance do you get? How could you adjust your network buffers to get better performance? What about your firewall rules?
• Do you know what RFC1323 is, and when you’d need what it specifies?

Ultimately a server needs rather a lot of attention to be performing well and be secure. If you just turn a server on and plop it online, you’re probably not getting out of it all that you could.

And you’re also probably hosting movies for kids on IRC, even if you don’t know it.

Of the four points we’ve listed below, we’ve covered the what and the where; we still need to examine when and how to back up.

How to back up?

As I said below, ideally everything that needs to be backed up is in one spot. Ideally this spot is the backup media, which is in a government-controlled facility at the bottom of a mine under a mountain with four or five OC-768s. The real world is never ideal, unfortunately, so some of this just isn’t possible. However, you can get closer or further from ideal depending on how you set things up.

Try to keep whatever needs to be backed up on a separate partition; on a separate drive if you can manage it. This allows you to use tools such as dump (nww), which is pretty much the best tool for the job (on FreeBSD anyway). You can even keep configuration files in this space. The fewer places you have to hunt for files, the better.

If you use backups as a kind of undelete function (say, if you are backing up personal files and sometimes need to restore a file you’ve deleted), then you should keep incremental backups. Incremental backups store everything that has changed since the last backup. These allow you to make frequent, short, small backups. If your work changes often (for example if you are backing up an office fileserver) you can run such backups several times during the business day. Then, when a file is needed, you can grab it from an older incremental backup.

Alternatively you can make full backups each time. This minimizes restore time, since with only one backup file you can restore all your data.

More complicated structures, such as the Tower of Hanoi differential algorithm that is suggested in dump’s manpage, are nifty but not strictly necessary. They achieve an optimum between space consumed, time to back up, and time to restore.

A further consideration needs to be given to databases. You can’t simply dump the actual database files; when you restore, you’ll find the database is corrupted. Instead, regularly dump the database (a real database can dump without locking whole tables) and then back up the dumpfiles.

When to back up?

At what time to back up isn’t really hard to determine: whenever the load is lightest.

You might want to put some thought, however, into deciding how often to back up. As I said below, not everything need be backed up at once, but only as often as it changes significantly.

If you are making incremental archives or backups, you can run the backup three or four times daily, and then push those archives offsite at night.

Further reading

(all links open in a new window–sorry)

• Backup strategies
• Backup basics

Most of this has been directed at Unix-like operating systems. Some of this applies to Windows as well, though in Windows you can’t simply back up your configuration scripts and then restore them to have your applications work just as you remembered them. If you enjoy paying for software, there are many third party backup utilities availables, such as Veritas Backup Exec. But most of these guidelines should apply.

If you have a dedicated server, it’s almost a certainty that you’ve never seen your server, or have even been in the same city as your server. Thus it is almost a certainty that you can’t show up in person to change tapes and take them offsite. Yet, it’s also probably a pretty good bet that your data is so critical to you or your business that it would not be too much of an exaggeration to say losing it would be a calamity.

How can you ensure your data is safe?

Well first, like security, your backup strategy needs its own plan and its own budget. You can’t toss firewalls at your server to make it secure; neither can you copy files off into the void. There are a number of issues that need to be addressed: what to back up, where to store the data, how to back it up, and when to run backups.

Some of this might apply if you’ve got a colocation account somewhere, but a lot of it is completely different. Physical access to a machine can change circumstances (as well as what services an ISP provides — at ServePath we are far more hands-on with our (unmanaged) dedicated customers than we are with our colocation customers).

What to back up?

If you’re paranoid, you might say “everything”, but this really doesn’t make any kind of sense if you’ve got a dedicated server. Since you can’t get to the machine, you’re never going to interact with it unless it has a complete install of some operating system on it. This means that file system-level utilities (such as my all-time favorite, dump (nww)) are pretty useless unless you architect your server’s layout around them.

When deciding what to back up, you have to keep in mind how you’re going to restore. If you simply tar up /, when you restore you’re going to scribble all over system files (and tar has issues with certain kinds of files as well) with old versions. If you were backing up Debian but restoring to FC4, you could easily render your system unusable by restoring to it!

What you want to grab are all your data files, and leave the rest to hang. Data files are actually split into two groups, configuration files and actual data files. Config files live in places like /etc and /var/named. If you can, try to avoid putting them there; use /root or something under /usr/local, such as /usr/local/etc or /usr/local/var. This way you don’t find out, when restoring from a crash, that your 2500 line custom Sendmail script was under /etc, which you didn’t bother to back up. Data files can live in some funny places, but if there’s a service (such as httpd) that owns them, it’s best to give them a user (such as http or www) and put all the relevant files under this user’s home directory. This doesn’t have to be /usr/home/www. This helps to keep track of where everything is.

If money (and therefore space) is an issue, you can decide what can be risked. I don’t back up any of my media files, since the chance of losing them is very low, the cost of losing them is low, and the cost of backing them up is high. If you have a bunch of open source repositories on your site because you like to help out SourceForge or Fresh Meat, then don’t bother to back them up, because you can just download them again. If you have a bunch of open source repositories on your site because your business involves them, as one of our customers’ (nww) does, then you should be backing them up because the time it takes to download them is more expensive than the cost of backing them up.

Where to store the data?

Dedicated customers are in kind of a tricky position here. The answer to this question is almost an unequivocal “offsite”, but if you have enough data even a 100Mbps pipe can’t finish the first backup before the second begins.

Fortunately, you don’t need to pull your backups offsite every day. You can automatically perform remote backups weekly or even monthly, depending on your data, its temporal value, and the rate at which it updates or changes. For example, a group of servers that powers a small online store can backup the transaction database nightly, but the contents of the store itself probably don’t update nearly as often, and the actual scripts that run the store are even less critical. All need to be backed up, since without any one of them the site is offline, but they have different priorities.

The whole reason for pulling backups offline is so that if there is a major disaster, recovery is still possible. ServePath itself is fairly secure (impressively so, I’ll blog about it sometime), so a disaster big enough to destroy your onsite backups and your server would probably level this building and most of the rest of San Francisco. Still, if it happens, you still want your business to be able to pick itself up. So take the backups offsite at least once in a while.

Where to keep the data while it’s here? I want to take this time to clear up a misconception about RAID. RAID is not backups. They do not protect against the same kinds of errors. Backups provide crude protection against mistakenly deleted files, hackers, corrupt file systems, and hardware failures. RAID provides graceful protection against hardware failure, sometimes. Sometimes RAID simply improves performance. You can’t assume that RAID 1 (mirrored drives) will save your data, because if there is some other hardware failure, like the power supply, and your file systems are corrupted, then both drives are gonners. What RAID does let you do is to notice that a drive has failed, and then schedule downtime to replace that drive. It turns uncontrolled downtime into controlled downtime. Usually.

ServePath really offers three choices for onsite backups: a second drive in the same server, a second server which mirrors the first, or space on a NAS drive. I’m not sure what the standard offerings among dedicated server hosts are but I’m sure they’re much the same.

A second drive is a pretty good idea if you’re most often recovering from deleted files. Transfers are quick, both for backup and restore. However, if the server is offline and you need access to the data, you’re out of luck. And since really anything that affects the first drive can affect the second drive, too, it is not entirely safe. Very recently I had to help a customer restore his OS after a script ran “rm -r /usr”. Fortunately, he had other backups.

Something like a NAS prevents most of the troubles that a second drive can run into. Since data can be copied via rsync or ftp, it allows archives to be made without exposing them to the file system for errant scripts to destroy. Since it is on a LAN link with the server, backup and restore is relatively quick. This can be thought of as “semi offsite,” since even if your server explodes, your data will probably still be okay (depending of course on the energy in the explosion and where your computer is in the data center).

A second server is the most expensive but also the most reliable solution. With proper load balancing configured, any single server could die and your customers wouldn’t even notice, because the other servers would pick up the load. As long as all servers in the pool have all the data, no single failure should affect the rest.

Okay this is getting long. I’ll return to it next week.

This is where ColoServe’s new monitoring services come in. This service ensures that you are proactively notified if your web site or Internet application go down, and it also offers you more visibility into the health of your server’s CPU, hard disk space and memory utilization.

With ActiveAlert monitoring we can help you make sure that your Internet-based services are always available. Should your server suddenly become unavailable, an alert will immediately be sent to you, and you can instantly take action.

Server Component Monitoring makes sure that your server is running at peak performance. Predetermined thresholds will be set for your CPU, disk and memory, and if any of these thresholds are reached, you will instantly be alerted via email.

Learn more about ColoServe’s Monitoring Services.

At ServePath we recently introduced a whole range of managed backup services that are designed to make it easy for you to make sure your data is safe and available should you experience any sort of hardware or software failure.

We offer several different solutions:

Network Storage
ServePath’s network storage servers are redundant RAID array servers available on a private network within our data center and are not publicly available to the Internet. Learn more about Network Storage.

Managed Backup
Our Managed Backup product combines a Network Storage account with a customized backup plan and automated scripts. The most commonly used media rotation schedule is the Grandfather-Father-Son schedule. With our customized plan, you can determine what files are backed up and on what schedule, how often archives are rotated, and much more. Learn more about Managed Backup.

Data Replication
With data replication you get identical on or off-site replicas of entire volumes or selected files, and configuration, management, server hardware, replication software and automatic failover are also included to give you the performance you need, not to mention peace of mind. Learn more about Data Replication.

Dedicated, Network Attached Storage
For solutions requiring 500 GB or more of high performance storage, we can build a Network Attached Storage array customized to your particular requirements. Network Attached Storage arrays are connected directly to your server or servers, so only your machines can access them.

This is where ServePath’s monitoring services come in. This service ensures that you are proactively notified if your web site or Internet application go down, and it also offers you more visibility into the health of your server’s CPU, hard disk space and memory utilization.

With ActiveAlert monitoring we can help you make sure that your Internet-based services are always available. Should your dedicated server suddenly become unavailable, an alert will immediately be sent to you, and you can take action right away.

Server Component Monitoring makes sure that your server is running at peak performance. Predetermined thresholds will be set for your CPU, disk and memory, and if any of these thresholds are reached, you will instantly be alerted via email.

Learn more about ServePath’s Dedicated Server Monitoring Services.

Our Ethernet services consist of three different products:
• LanExtender
• NetExtender
• DataExtender

LanExtender gives you the opportunity to extend your office network into our data center to get better performance and reliability. We can integrate your LAN with our data center, thereby allowing your business to maintain database, mail, file or any critical servers in a secure environment while users access them as if they were just a few feet away in your office server room.

NetExtender offers you high-speed quality Internet access through multiple Tier 1 backbone providers without the hassle of having to deal directly with the Telcos or buying multiple connections. It is an affordable way to increase your bandwidth and maximize the performance of your Internet-based applications.

DataExtender is designed to facilitate off-site backup of your data to a dedicated storage server using a high-speed Ethernet link. The server is located in our secure data center, and you are therefore ensured swift data recovery in the event of a disaster. You will be able to perform secure off-site backups directly from your office.

Learn more about ColoServe’s Ethernet Data Services.

These new features fill out the existing portal, which also allows colocation customers to view bandwidth and data transfer usage statistics, remotely reboot servers with the included power cycling interface, view invoices, view, open and update support tickets, and update their account information.