When we first started Wahanda we made the decision to have an all Mac OSX development environment. This has served us pretty well over the last year or 2 and the Mac is a great platform. For those like myself who only gets involved in PHP development with a single windows VM for SQL Server it’s still working great.
However our primary site platform now spans a fair few Java Webapps. Running 2 tomcats + VM’s for IE6, IE8 etc. has got to the point where the standard test-amend-compile-test cycle can be in the order of minutes. This was understandably causing a lot of complaints!
So, we all sat down and had a long discussion about what to do. I would love to be in a position to buy everyone dual quad-core Mac Pros but we really don’t have that sort of money at the moment. So with general agreement we decided to move to a more generic hardware platform running Debian Linux (We did debate using Ubuntu but as the servers are all Debian it was decided to go for consistency).
Chris went away and played with some options and ended up with this spec:
- Asus P5QL Pro Motherboard
- Intel Core2 Quad @ 2.66Ghz
- 2 x Crucial 4Gb DDR2 RAM
- GeForce 9600GT Dual DVI Video
- 2 x 250Gb Seagate Barracuda 7200rpm HDD
- Various cheepy cheep ATX cases
This is at least a doubling of the previous developer MacBooks. 2 more cores, twice the RAM and RAIDED drives rather than the rather pathetic 5400rpm notebook drives from before.
The following diagram is just the 2 tomcats starting up. That capacity really is needed!
This one is building the entire site stack. Doesn’t suck up quite as much as an init of the environment but the spread of load across the 4 cores is great to see.
Overall we’ve reduced the test-amend-compile-test back to only a few seconds. An enormous boost to productivity in the team. The downside was that we lost everyone who made the switch for 2 straight days why they got everything set up.
The marketing / design / copy team are also very happy with their pass-me-down MacBooks. 
The last couple of months have been rather crazy here in the Wahanda office. After a number of weeks of supremely hard work by the whole tech team we’ve launched ourselves stateside at us.wahanda.com.
We already have one of the largest listings of wellness venues in the united states and are adding new ones at a rate of knots! The focus is really on community and venue content as we don’t support sales as yet.
Anyway, this post isn’t about that as I still haven’t recovered enough to face the prospect of writing it all up in detail. So in the meantime I thought I’d post some photos of our new (ish) server room here in London.
Problem
As a startup we really try to avoid spending money where possible. However after nearly a year of everything running in a big pile on the floor cooling issues with the development and office servers were prompting some of them to shut themselves down in protest. We needed some sort of rack system.
Pile of Servers
Yes, they were actually on the floor of a cupboard propped up using scraps of wood we found in the street.
Rather than spend £300+ buying ourselves a rack I thought I’d make one. My tendency towards NIH syndrome is well known to those of you that have had the pleasure of my company but I quite like making real things occasionally. You know, things involving metal, wood, splinters, screws etc. As opposed to virtual stuff which you can’t hit people over the head with (not counting metaphorically).
I also know you can get racks second hand on ebay but even then I was looking at £100 for anything half decent plus a lot of hassle getting it from wherever it was to the office (I don’t own a white van).
Building the base
I make quite a lot of things out of wood for home so It was the work of about an hour to cut and screw some 2×2 batons and some sheet mdf into a simple base.
[\caption]
I then bought 4 2m strips of rack rail from Maplin, plus 4 blanking plates and a rack-mountable power strip. In hindsight the plates probably weren’t necessary and I could have got away with some simpler. They did make it very easy to create the box-frame though, which saved time on construction.
This was then all assembled on site and the rack (albeit a bit wobbly) was in place.
Salim helping with the construction
The base and the rails
Installation
The principle of the rack is that the servers themselves act as the main stabilising component. As such there is no fixed top or sides which makes it nice and easy to get to the machines themselves. To power the whole thing we installed a couple of APC Power Chute UPS units we already had.
The Finished Product
Conclusion
So, no doors for security or inbuilt cooling (i.e. cheap fans) and no fancy glass panel I can watch the flashing lights through but for less than £70 in parts and a few hours work we now have a full 42U rack unit which we can dismantle and take to the next office with very little hassle.
After a lot of deliberations and a fairly huge amount of effort from all involved we now have a uk based hosting solution up and running with our new provider Clara.net.
We have 6 servers involved in running the wahanda.com site. It’s a fairly standard configuration that should give us enough capacity to support 10x current peak traffic and is easily scalable beyond that by adding more kit.
UK Hosting Overview
One of the big unknowns in the process was how much power we would actually end up using. Working out the exact power consumption for a server is notoriously difficult. So many factors are involved and the way they interact is not always predictable. I’m sure someone who has a better background in electronics could do it better but generally I find the easiest thing to do is to buy some hardware and measure. Once it was all in place we measured 3.0A in total, which actually gives us some headroom for more kit should we need it!
The atom servers are discussed in a previous post and worked out at 0.35A when measured individually. That’s more than we originally hoped but the dual-core Atom is a little hungrier than the single and we’ve got 2 sata drives in each one (mirrored) which pushes us up a bit as well.
Our web servers are single socket quad-core Dell R200 rack servers. Our underlying architecture is ideally suited to 3 cores per host so these work pretty well. We’ve also finally moved to a full 64-bit stack, making full use of the 4Gb RAM in each one. We balance load equally across the 3 web servers which gives us good redundancy as well as performance. 0.5A each for these.
The database server is the only really expensive piece of kit, being a hefty Dell 2950 III with dual quad, 8GB RAM and lots of disks. We rely on a highly normalised database model so really need this to perform. As expected this machine draws about a third of the total consumption in the rack at 1.0A.
Shiny lights...
One thing I’ve kept very much in mind during this process is how little hardware we should need to support the amount of traffic we have. Coming from a high traffic corporate environment (previously was at lastminute.com) you get used to having a very large amount of hardware running anything and everything. Some of this is justified but a lot is down to a legacy of throwing hardware at problems rather than fixing the underlying apps.
the most impressive site I know of in terms of pageviews per server has to be Plenty of fish. They serve 12k+ pages per second on a mere handful of servers. There is a whole load of de-normalisation going on to make this possible, and I would argue that this sort of model where the pages are completely simplified / optimised / written inline doesn’t work for larger dev teams but it’s still damn impressive.
Anyway, we’re live now so hopefully you’ll all notice the improvement in performance if you’re in the uk. We’ll be launching our u.s.a. hosting setup in the next couple of weeks so if you’re located across the atlantic and have noticed a slowdown it won’t last long.
We’re currently in the process of setting up a bunch of servers in the u.k. to deal with European traffic on the site.
Something I was surprised to find out last year was that the most limited commodity in London regarding hosting is not space or bandwidth, it’s power. One nameless ISP told me that it will remain this way until after the London 2012 Olympics as they’re sucking up all the excess capacity!
Our initial allotment is for only 4A. This really isn’t much when a single one of our dev machines (the ones we use to host internally, build things on and test on etc.) draws around 0.9A. Out initial estimate for servers was for a base install of 6 so these machines aren’t really going to work.
Those guys at Dell are producing some more efficient boxes these days which means our primary web stack can run in just 0.5A (Single Quad-core Xeons). Our main db server is likely to be at least 1A so we needed something even more efficient than the dells for the network config.
At home I’ve built a number of machines as media servers or lightweight web machines using the Mini-ITX format. I was therefore quite excited to hear about a bundle being offered by mini-itx.com which included a dual-core Atom based ITX Mainboard, a couple of gigs of RAM, all housed in a 1U Rack Case.
Inside the Atom 1U Rackserver (aeryn)
As you can see we have an ITX board, 2 SATA drives (80Gb mirrored) and not much else in the box. Witbh this config we have a power draw of around 0.31A. Not quite as low as the 0.15A advertised by MiniITX but we have gone for the dual core atom + 2 drives so not surprising it’s higher.
We also experimented with a 2Gb flash drive in place of once of the HDD to see if it would bring down the power. It didn’t that much so we went with the slightly more robust full RAID1 mirror instead. This seems to be the way things are going though so hopefully we’ll be able to move all of the infrastructure type servers to full SSD in the medium term.
2Gb SATA Drive (Flash)