Archive for Visits

Redress your apps for Cloud

This week, Alex Stamos of iSEC Partners visited and gave a great talk titled “Securely Moving Your Business into the Cloud”.  Much of that material is publicly available here. Alex is a straight shooter and a straight talker.  By the second slide, he’s already warmed up and delivers quite a punch line:  You cannot securely move into the cloud without re-writing your software.

I subscribe to that line. And there’s more to it than security. Earlier on, I’ve reached the same conclusion when thinking about availability and all the *-abilities that an enterprise needs for its business-critical operations.

Every so often, the IT industry falls for the holy grail of horizontally scaling applications, blindly and effortlessly, without touching a line of code. It happened with Grid Computing before Clouds. The early wins in their respective stomping grounds (HPC for Grids, entrepreneurs for Cloud) don’t necessarily scale to become F500 wins. Rather, reality sinks in, that one needs to rework the application stack and, worse yet, needs to recruit several PhD types to do that. We cannot defy gravity nor the laws of distributed systems.

In learning this all over again, there’s some forward progress. Those who venture into retooling their stack will most likely achieve superior security and *-abilities in general. In their dollar and sense considerations, they will have to contrast Cloud savings with the budget and timeline to implement and operationalize the new stack. Some others will justifiably punt and wait for a Hail Mary pass* by whatever will come next after Grid and Cloud.

*Not quite Dave Patterson’s Hail Mary pass, even though there’s a striking similarity with what’s happening with multi-core at micron scale and the annex arguments pro/con application re-write.

Leave a Comment

My other computer is a Utah datacenter

I was recently in Salt Lake City and visited the new datacenter that opened back in May.

Quick facts:

  • ~300M$ investment
  • 240,000 square foot building; inside, 3x rooms with 20,000 sqf each of rack-worthy raised floor
  • fault tolerant Tier IV data center
  • designed PUE of 1.4
  • 7.2 MW of total server load
  • 400V/230V power distribution (230V to servers)
  • Outside air used for cooling at least half the year (water-side economizer)
  • Total hot-aisle/cold-aisle containment
  • Deploy a rack anywhere/anytime, thanks to ToRs plus optional in-row adaptive cooling

Its opening was covered quite well in the press and blogosphere, see for instance 1, 2, 3, 4 for details, pics, color.

Some observations:

  • Internet-scale Maestro James Hamilton is right on when he says that there are significant economies of scale in building and operating an Internet-scale datacenter, albeit with a very high cost of entry;
  • How low can you go in the layers. I developed system, server, network aspects for this datacenter and thought of myself covering low layers of infrastructure. Move 800 miles East from the office, and these same aspects now look like the tip of an iceberg. That is, they are the topmost layer in a deep stack of power distribution layers, cooling layers, backups of backups before terminating at the power substation and the high-voltage power lines;
  • It’s hard to manage all dependencies, especially when the overall system of systems is mission-critical. Kudos to the hardware folks who are so much better at this than us software types;
  • Then there’s Cloud Computing… Given the cost of entry and the long-term commitment to an Internet-scale datacenter, it’s no surprise that Clouds are becoming increasingly competitive against traditional options (e.g., lease colo space or roll your own).

Comments off


The word generativity jumps at me while I’m reading Jonathan Zittrain’s new book, “The Future of the Internet – and How to Stop it”. Zittrain defines generativity as  a “system’s capacity to produce unanticipated change through contributions from broad and varied audiences”. Internet, PC, wiki/wikipedia best exemplify generativity. It’s “generativity” what I had in mind and tried to say when I wrote about Internet’s virtuous wheel of innovation.

Generativity hits home. It’s the reason why I’m so genuinely interested in the Android platform (I got to carry one such phone alongside my iPhone). It’s why I put my TV set in early retirement and replaced it with an Internet-enabled one equipped with widget SDK – a generative TV in the making, hopefully. I know that I have given and will be giving my 150% in those jobs that have to do with generative artifacts (luckily, I have had a few of those jobs throughout my career).

Generativity is quite a litmus test for new directions in technology. Take cloud computing. Does it mark a new epoch in generativity? Or is this a mere TCO optimizer?

For sure, security, regulations, net-neutrality pose some great challenges to our collective journey in generativity. I look forward to reading the second half of Zittrain’s book and learning about his proposed solutions.

Zittrain came to visit us at eBay and gave an excellent lecture on “Minds for Sale” — an eye opener on both the positive and negative outcomes of long-tail participation in cyberspace.

Leave a Comment

Teach programming to your littl’ digital natives

In my monthly CACM issue, I found a delightful and somewhat unusual article on “Scratch“. With Scratch, Mitch Resnick et al.  at the MIT Media Lab have created a programming environment with the lowest up front investment for children and teenagers. As you would expect in a platform that speaks to digital natives, Scratch comes with a host of rich media and social networking components built in.

My children love Scratch. They were able to program in Scratch and do things that appealed to them from the very first session. I like them to spend time with Scratch because it lifts the curtain on how computer games and digital entertainment work. It stimulates their creativity and a can-do attitude towards technology.

In the mid ’90s, I had the fortune to meet Mitch Resnick at the Media Lab. My company back then was a top tier sponsor. I saw the first prototypes of what became Lego Mindstorms (whose programming user experience put the early seeds for Scratch). It’s fascinating how Resnick repeatedly gets it. He might as well be the Steve Jobs of under age computer human interface.

Leave a Comment

Berkeley visit

This Monday, Prof. Ion Stoica invited me to Berkeley to give a lecture on eBay’s scaling, challenges, and lessons learned, as part of their Cloud Computing class. I really enjoyed the opportunity to present and the open, lively discussion that resulted. Questions clustered around the transactional nature of the eBay workload, eventual consistency, and the this-is-life-in-a-big-city kind of scaling practices (inclusive of people and process implications, which play a key role alongside the technology). With a Fortune 500 mindset, I shared my view of Clouds and some research big rocks that in my opinion can profoundly impact Cloud adoption.  

While visiting, I learned that Berkeley’s magnum opus paper on Cloud Computing will be unveiled to the public on Thu Feb 12th. In January, I had a sneak preview of the paper and was impressed by its breadth and depth.

Leave a Comment

UC Berkeley’s RADlab retreat

I’m on my way back from the UC Berkeley’s RADlab winter retreat at Lake Tahoe. I’m really grateful to Prof. Randy Katz and the UC Berkeley faculty for inviting me.

RADlab’s vision is to enable one person (possibly the next Pierre Omidyar) to single-handedly invent and operationalize the next multi-million-user service (possibly the next eBay) over the course of a long weekend. RADlab’s retreat is an opportunity for industry leaders to mix with faculty and students to review the progress and provide feedback.

Over these intense three days, students and faculty gave excellent talks and demonstrations. Things that I can call-out in a (very) short list:

  • By now, it is commonplace for researchers and students to develop, test, and run on cloud;
  • Ruby has made it into the classroom. I’ve seen some impressive term projects with very little code written (and, interestingly enough, the test code has more lines than the actual code);
  • Machine learning is alive! I have found some encouraging proof points in scaling up/down resources, generating equivalent synthetic workloads, timely detecting datacenter problems by way of signature correlation, etc.;
  • A scalable data store (scads) for which cost/user doesn’t increase and there’s a declarative language to set performance/consistency tradeoffs;
  • Use of a performance model to perform accurate diagnosis while using just about 10% of log data;

I often think about the scale divide between those who can get behind the curtains of internet-scale data centers (often times, in the 10^5 servers realm) and those who cannot. Those who have access typically have limited freedom to chase high-risk, high-payout propositions. Next-gen infrastructure researchers must get a chance to see their artifacts operating at scale. What can researchers and practitioners do to bridge the scale divide to a mutual advantage? I’ve seen that students clearly extract a lot of value out of their internships. Without a doubt, internships are a great win-win-win. Also, I’ve heard speakers asking for anonymized production traces. Unfortunately, this is a tough nut to crack … in the web age, the unintended ripple effects of taking traces out of the house give many corporations the heebie jeebies.

I look forward to monitoring the continued progress of RADlab.

Leave a Comment

Visit to MSR SV

I recently had the pleasure of visiting Microsoft Research Silicon Valley. Mihai Budiu (whom I met at a November event) kindly invited me over. There, I found some great and vibrant teams. Their logistics are nice too — I love their walled offices and lounge areas, it reminded me of what we had at the OSF RI in Cambridge MA (researchers need their space!).

I talked about datacenter issues, scale-out themes and “clouds” (in fact, I covered much of the material that I presented at the Cisco Cloud Symposium back in November). I had the privilege to meet Chuck Thacker in person. I sat down with Chuck’s team first and then with the DryadLINQ team (who were just back from presenting their work at OSDI 08). Dryad + LINQ piqued my interest for this particular use case …. I need to crawl log entries (at eBay, we generate 2+ TB worth every day) and mine the relationships amongst logical/physical entities, in a cross-tier fashion. 

Leave a Comment

Video interview on Clouds

In June, Dejan Milojicic of HP Labs hosted a fireside chat on Cloud Computing with Russ Daniels (VP and CTO of Cloud Services Strategy at HP) and me. The interview is part of an ongoing series of talks sponsored by the the IEEE Computer Society. The three of us engaged in a lively conversation (despite being bright and early in the morning!).

Dejan has just notified us that the video interview has been web-published here.

Leave a Comment