|
| Cloud, Digital, SaaS, Enterprise 2.0, Enterprise Software, CIO, Social Media, Mobility, Trends, Markets, Thoughts, Technologies, Outsourcing |
ContactContact Me:sadagopan@gmail.com Linkedin Facebook Twitter Google Profile SearchResources
LabelsArchives
|
Saturday, September 18, 2004Google's cluster architectureFew Web services require as much computation per request as search engines. On average, a single query on Google reads hundreds of megabytes of data and consumes tens of billions of CPU cycles. Supporting a peak request stream of thousands of queries persecond requires an infrastructure comparable in size to that of the largest supercomputer installations. Combining more than 15,000 commodity-class PCs with fault-tolerant software creates a solution that is more cost-effective than a comparable system built out of a smaller number of high-end servers. The architecture of the Google cluster,and its design is influenced by -energy efficiency and price-performance ratio. Energy efficiency is key at Google's scale of operation, as power consumption and cooling issues become significant operational factors, taxing the limits of available data center power densities.Google's application affords easy parallelization: Different queries can run on different processors, and the overall index is partitioned so that a single query can use multiple processors. Consequently, peak processor performance is less important than its price/ performance. As such, Google is an example of a throughput-oriented workload, and should benefit from processor architectures that offer more on-chip parallelism, such as simultaneous multithreading or on-chip multiprocessors.Google’s software architecture arises from two basic insights. First,Google provides reliability in software rather than in server-class hardware, so commodity PCs can be used to build a high-end computing cluster at a low-end price. Second, Google tailors the design for best aggregate request throughput, not peak server response time, since response times can be managed by parallelizing individual requests. Operating thousands of mid-range PCs instead of a few high-end multiprocessor servers incurs significant system administration and repair costs. However, for a relatively homogenous application like Google, where most servers run one of very few applications,these costs are manageable. Assuming tools to install and upgrade software on groups of machines are available, the time and cost to maintain 1,000 servers isn’t much more than the cost of maintaining 100 servers because all machines have identical configurations. Similarly, the cost of monitoring a cluster using a scalable application-monitoring system does not increase greatly with cluster size. Furthermore, the repair costs can be kept reasonably low by batching repairs and ensuring that we can easily swap out components with the highest failure rates, such as disks and power supplies. An excellent article that details out the marvel - Google's architecture. | |
| Sadagopan's Weblog on Emerging Technologies, Trends,Thoughts, Ideas & Cyberworld |
