Another year done. Most of the team here are winding down in preparation for the holiday break. (Customer services and tech operations will still be around). The last few months have been rather crazy here and I haven’t been as good as I could with keeping the tech blog up to date. We’ve launched so many feature in the last 4 months that I haven’t written about including the Supplier Extranet, Questions and Answers, Widgets and more.
My list of resolutions for January will include to be more communicative.
Meanwhile in honour of International Backup Awareness Day here are some random musings on backup strategy…
Site Backup
Over the weekend I was going back and reviewing our backup strategy for the site as a result of this post from Jeff Atwood. For those that don’t know he lost his blog www.coding-horror.com last week as a result of a drive failure on the host server. The regular backup was backing up to the same drive on the host which was a bit of a FAIL. I’m sure Jeff is beating himself up quite nicely to that’s enough of the poking.
Backup is one of the few things that has the potential to keep me up at night. It suffers form being very uninteresting yet absolutely vital to the company. It’s a bit of a no-win in this respect as if everything is fine then no one notices. If it goes wrong (and by wrong I mean really really really wrong) you will probably lose your job.
Anyway, I had a look and found a couple of holes (I always seem to find some) but decided that overall I’m happy. The key to our production strategy (there is a separate model for office stuff) is to keep all the data we actually care about in one place, the database server. We currently have 9 databases storing everything from emails to financials to images. Having everything in the database means all I have to worry about is making sure that the database server is backed up.
My experience with hardware over the last 15 years has taught me is to trust the adage: DATA IN ONLY PLACE DOESN’T REALLY EXIST. This has lead me to be a little paranoid over potential data-loss so as the following diagram shows we have data in lots of places.
Production Db Backup Flow
- Data spread across RAID 5 Array.
- Nightly raw backup to separate HDD (kept for 2 weeks).
- Nightly compressed backup (from raw) to external HDD (kept for 6 months).
- Weekly transfer of backup to office (with restore to dev/uat to verify).
- Physical burn to DVD for long-term storage.
This of course raises the problem of having sensitive data in lots of places to be managed but that’s another story.
So if you haven’t done it recently use this window before your holiday to BACK UP YOUR DATA!
The last couple of months have been rather crazy here in the Wahanda office. After a number of weeks of supremely hard work by the whole tech team we’ve launched ourselves stateside at us.wahanda.com.
We already have one of the largest listings of wellness venues in the united states and are adding new ones at a rate of knots! The focus is really on community and venue content as we don’t support sales as yet.
Anyway, this post isn’t about that as I still haven’t recovered enough to face the prospect of writing it all up in detail. So in the meantime I thought I’d post some photos of our new (ish) server room here in London.
Problem
As a startup we really try to avoid spending money where possible. However after nearly a year of everything running in a big pile on the floor cooling issues with the development and office servers were prompting some of them to shut themselves down in protest. We needed some sort of rack system.
Pile of Servers
Yes, they were actually on the floor of a cupboard propped up using scraps of wood we found in the street.
Rather than spend £300+ buying ourselves a rack I thought I’d make one. My tendency towards NIH syndrome is well known to those of you that have had the pleasure of my company but I quite like making real things occasionally. You know, things involving metal, wood, splinters, screws etc. As opposed to virtual stuff which you can’t hit people over the head with (not counting metaphorically).
I also know you can get racks second hand on ebay but even then I was looking at £100 for anything half decent plus a lot of hassle getting it from wherever it was to the office (I don’t own a white van).
Building the base
I make quite a lot of things out of wood for home so It was the work of about an hour to cut and screw some 2×2 batons and some sheet mdf into a simple base.
[\caption]
I then bought 4 2m strips of rack rail from Maplin, plus 4 blanking plates and a rack-mountable power strip. In hindsight the plates probably weren’t necessary and I could have got away with some simpler. They did make it very easy to create the box-frame though, which saved time on construction.
This was then all assembled on site and the rack (albeit a bit wobbly) was in place.
Salim helping with the construction
The base and the rails
Installation
The principle of the rack is that the servers themselves act as the main stabilising component. As such there is no fixed top or sides which makes it nice and easy to get to the machines themselves. To power the whole thing we installed a couple of APC Power Chute UPS units we already had.
The Finished Product
Conclusion
So, no doors for security or inbuilt cooling (i.e. cheap fans) and no fancy glass panel I can watch the flashing lights through but for less than £70 in parts and a few hours work we now have a full 42U rack unit which we can dismantle and take to the next office with very little hassle.
After a lot of deliberations and a fairly huge amount of effort from all involved we now have a uk based hosting solution up and running with our new provider Clara.net.
We have 6 servers involved in running the wahanda.com site. It’s a fairly standard configuration that should give us enough capacity to support 10x current peak traffic and is easily scalable beyond that by adding more kit.
UK Hosting Overview
One of the big unknowns in the process was how much power we would actually end up using. Working out the exact power consumption for a server is notoriously difficult. So many factors are involved and the way they interact is not always predictable. I’m sure someone who has a better background in electronics could do it better but generally I find the easiest thing to do is to buy some hardware and measure. Once it was all in place we measured 3.0A in total, which actually gives us some headroom for more kit should we need it!
The atom servers are discussed in a previous post and worked out at 0.35A when measured individually. That’s more than we originally hoped but the dual-core Atom is a little hungrier than the single and we’ve got 2 sata drives in each one (mirrored) which pushes us up a bit as well.
Our web servers are single socket quad-core Dell R200 rack servers. Our underlying architecture is ideally suited to 3 cores per host so these work pretty well. We’ve also finally moved to a full 64-bit stack, making full use of the 4Gb RAM in each one. We balance load equally across the 3 web servers which gives us good redundancy as well as performance. 0.5A each for these.
The database server is the only really expensive piece of kit, being a hefty Dell 2950 III with dual quad, 8GB RAM and lots of disks. We rely on a highly normalised database model so really need this to perform. As expected this machine draws about a third of the total consumption in the rack at 1.0A.
Shiny lights...
One thing I’ve kept very much in mind during this process is how little hardware we should need to support the amount of traffic we have. Coming from a high traffic corporate environment (previously was at lastminute.com) you get used to having a very large amount of hardware running anything and everything. Some of this is justified but a lot is down to a legacy of throwing hardware at problems rather than fixing the underlying apps.
the most impressive site I know of in terms of pageviews per server has to be Plenty of fish. They serve 12k+ pages per second on a mere handful of servers. There is a whole load of de-normalisation going on to make this possible, and I would argue that this sort of model where the pages are completely simplified / optimised / written inline doesn’t work for larger dev teams but it’s still damn impressive.
Anyway, we’re live now so hopefully you’ll all notice the improvement in performance if you’re in the uk. We’ll be launching our u.s.a. hosting setup in the next couple of weeks so if you’re located across the atlantic and have noticed a slowdown it won’t last long.
We’re currently in the process of setting up a bunch of servers in the u.k. to deal with European traffic on the site.
Something I was surprised to find out last year was that the most limited commodity in London regarding hosting is not space or bandwidth, it’s power. One nameless ISP told me that it will remain this way until after the London 2012 Olympics as they’re sucking up all the excess capacity!
Our initial allotment is for only 4A. This really isn’t much when a single one of our dev machines (the ones we use to host internally, build things on and test on etc.) draws around 0.9A. Out initial estimate for servers was for a base install of 6 so these machines aren’t really going to work.
Those guys at Dell are producing some more efficient boxes these days which means our primary web stack can run in just 0.5A (Single Quad-core Xeons). Our main db server is likely to be at least 1A so we needed something even more efficient than the dells for the network config.
At home I’ve built a number of machines as media servers or lightweight web machines using the Mini-ITX format. I was therefore quite excited to hear about a bundle being offered by mini-itx.com which included a dual-core Atom based ITX Mainboard, a couple of gigs of RAM, all housed in a 1U Rack Case.
Inside the Atom 1U Rackserver (aeryn)
As you can see we have an ITX board, 2 SATA drives (80Gb mirrored) and not much else in the box. Witbh this config we have a power draw of around 0.31A. Not quite as low as the 0.15A advertised by MiniITX but we have gone for the dual core atom + 2 drives so not surprising it’s higher.
We also experimented with a 2Gb flash drive in place of once of the HDD to see if it would bring down the power. It didn’t that much so we went with the slightly more robust full RAID1 mirror instead. This seems to be the way things are going though so hopefully we’ll be able to move all of the infrastructure type servers to full SSD in the medium term.
2Gb SATA Drive (Flash)