I had the privilege of attending what I think will end up being a historic moment in the technology industry. The event was Facebook’s Open Compute launch, in which they opened up two technology domains known more for their secrecy than knowledge sharing and, in the process, broke down the walls between the domains themselves. I captured this video of Mark Zuckerberg introducing the concept at their headquarters in Palo Alto April 7th, 2011:

Mark Zuckerberg introduces Open Compute April 7th, 2011 from OpenStack on Vimeo.

The two domains, historically considered separate, are Data Center Design and Server Design. In the old world, it made sense to look at a server as a single unit, because applications typically lived on a single server (or, at most, a cluster of servers). In the new world, an application can span tens of thousands of servers (as I believe Facebook does), or hundreds of thousands of servers, as I believe Google does. No one knows for sure how many they have, but it’s a lot!

Given that Rackspace (my employer) operates datacenters globally and has tens of thousands of servers as well, our datacenter teams have been collaborating with Facebook for many months on next generation designs. When they asked our chairman Graham Weston to attend the event, I immediately saw an opportunity to jump on his coat tails with an openstack flag hidden in my back pocket. I just had a hunch there was more to this movement than just hardware… and here was my chance to sneak in and represent for all my OpenStack peeps out there…

Much has been written about cloud computing (commonly thought of as a large collection of servers running such apps “in the cloud”), but much of the discussion centers around distributed software systems to handle this new shape of problems (HADOOP, Cassandra, OpenStack) or the services implementing such software designs (Amazon, Rackspace).

At that software layer, clearly a wave of innovation is being unleashed by the open source software development model, in which like minded individuals are free to collaborate, regardless of company affiliation, to solve interesting problems. From my vantage point as one of the founders of OpenStack, I can tell you first hand that this model is unbelievable powerful as it attracts the brightest minds on the planet to come together to solve problems. It’s also redefining what it means to do “business development” in this era, as the mortal enemies of the past become the collaborators of the future, but that’s best left to a future post.

Now, Facebook is doing the same for two previously distinct domains: Data Centers and Servers. Jonathan Heiliger’s team presented some great background:

Facebook’s VP discusses Open Compute at Facebook, April 7th, 2011 from OpenStack on Vimeo.

Facebook’s Open Compute Launch, April 7th, 2011 from OpenStack on Vimeo.

The key word, in my mind, is previously. When viewed as a system — one giant supercomputer (for lack of a better word) with a hundred thousand processor cores being powered and cooled by massive fans, air ducts, and even higher voltages than normal all the way to the motherboard, it hits you: the building is the chassis for this computer! The open software platforms that enable apps to run on a computer the size of an airport are, then, in a sense the new operating system (OpenStack, Cassandra, Cloud Foundry, etc). Scoble has some great photos of the facility here (one stolen below).

By looking at the datacenter as one giant computer, Facebook was able to discard components that previously seemed essential, like the front “face plate” of each server (now just one of thousands of nodes in side a giant computer). They call this “vanity free” design, and I love it. Here’s a picture of Graham Weston in front of a “triplet” which is a 3 rack design with tons of these vanity free servers inside:

Amir Michael explains the history and goals in this video I captured at the event. Love the enthusiasm!!:

Facebook’s Open Compute Launch, April 7th, 2011 from OpenStack on Vimeo.

In my mind, the next wave of innovation comes from looking at the software and hardware (datacenter included) as one complete system. At the risk of invoking a technology cliche and inviting a fan boy war: The Apple Way. Putting aside the obvious not-very-open approach to innovation Apple is known for, my point is this: By looking at the problem from the software application level all the way down to the strands of electrical wire running into individual servers and even the physical location of the “giant computer” itself (near plentiful electricity, low risk of natural disasters etc), a whole new wave of innovation has yet to be realized.

Although data center folks like to use something called “PUE” to measure the efficiency of their designs, it really just measures the waste at the physical level (i.e. electricity that did NOT go to something “useful” like making the processor run), this does not measure the waste of bad software. I use the term “bad software” for simplicity, but what I mean in practical terms is that a lot of the “work” being done by those processors — even in the Open Computer “hyper efficient” design, is wasted and we may not even know it. The biggest reason we’re in the dark when trying to look at this problem holistically is the closed, proprietary nature of the players building so many of the technology pieces of the stack – until now.

Starting today, communities like Open Compute can combined forces with other open communities like OpenStack and Cloud Foundry to get real world data on how a given workload (say, serving up TMZ.com to 10M users or something less critical, like storing images from Mars) actually consumes power to find the waste and come up with innovative solutions across domains. To prove that collaboration is about action and not talk, Jesse Andrews, Jim Curry and myself approached the Facebook team after the event and said “hey! lend us some of these servers and we’ll take them to our office in San Francisco and have OpenStack running on them tonight!”. Something that might have taken months to arrange was now in motion in a matter of hours! A couple of videos of our secret plan in action:

OpenStack running on OpenCompute — on launch day!

Going back to the Apple comparison for a moment – there is a reason the iPad gets 10+ hours of battery life, and it is not purely about the capacity of the battery (hardware design & decisions) or the lightweight iOS operating system (software design) or the custom ARM processor (chip design), or the well understood nature of the applications and the resources they use (software platform design), it’s about all of these things taken as a whole. While there may be one company on the planet capable of taking this approach in consumer electronics you can hold in your hand, IMHO there are ZERO companies on the planet who can do it at the “datacenter computer” level. This is why it is more essential than ever that we embrace openness at every level of the stack, and get to work on this critical problem: together!

While I applaud all of the smart minds working on alternative energy sources, which will no doubt be essential to the future of our planet (and economy!), I’m equally excited to see the walls coming down amongst the practitioners that will be putting a massive amount of that energy to work (via cloud computing) in dramatically more efficient ways through open source and open design in the years to come.