Dec
10
WRITTEN BY:
David Cummins
Thursday, 10 December 2009
Distributed computing is not new, but the internet and new business models are making it affordable for smaller organisations to leverage the power of these computing networks.
A distributed system consists of multiple autonomous computers that communicate through a computer network. The computers interact with each other in order to achieve a common goal. Distributed computing platforms have been around almost as long as the network and there are many uses for this style of computing in evidence around the world today. Folding@Home is a project that uses networks of computers to simulate protein strands, while SETI@Home has been set up to search for extra-terrestrial intelligence.
As the cost of creating and connecting these network platforms becomes more affordable to the enterprise (as opposed to research labs) it was only a matter of time before organisations started to make distributed computing clusters available for a fee (or free). Companies such as
Google (AppEngine),
Microsoft (Azure) and
Amazon (EC2) all have distributed computing platforms available for rent.
As an example of what’s possible, let’s look at
80legs.
80legs is a service platform using a cluster of over 50,000 computers to deliver custom web crawling applications at low costs. 80legs offers a number of different ways to create custom crawl jobs, available through a web user interface or through custom applications.
The web user interface allows you to create jobs based on a set of criteria such as the seed URLs you would like to crawl, how deep you wish the job to follow links, and the processing that you would like to perform on the pages that are discovered.
At Frame, we used this successfully to determine which organisations, within a target sector, have some level of IT engagement in web technologies (such as wiki, Twitter, blogging, social media and Facebook). We started the seed search at a list of all relevant websites and let 80legs crawl its way through discovered pages (up to 1 million). Out of this, we received over 200,000 hits which we were able to narrow down to key areas within the organisations for our consultants to arrange interviews.
This processing capability is not something that organisations are able to achieve easily, but 80legs does make it affordable. The entire processing job for the study cost US$2.30.
Other organisations are taking this on board and are offering commodity services at fractional prices. For example, one organisation enables you to test websites for browser compatibility on an ad hoc or scheduled basis—this is great for web developers who don't want the hassle of maintaining multiple virtual (or physical) operating systems for the sole purpose of browser checks. Another organisation offers distributed load testing for your web application from machines located throughout the world.
Distributed computing platforms are a great opportunity for organisations to outsource computing problems that are traditionally expensive to setup and maintain whilst retaining control of how and when these jobs are to be run. All that is left for you is to determine how to interpret the results!