Amazon web services and Hadoop

By
This is a prettyPublish Post cool project using Hadoop and Amazon web services

From New York Time Article Cited Below
This all adds up to terabytes of data, in a less-than-web-friendly format. So reusing the EC2/S3/Hadoop method I discussed back in November, I got to work writing a few lines of code. Using Amazon Web Services, Hadoop and our own code, we ingested 405,000 very large TIFF images, 3.3 million articles in SGML and 405,000 xml files mapping articles to rectangular regions in the TIFF’s. This data was converted to a more web-friendly 810,000 PNG images (thumbnails and full images) and 405,000 JavaScript files — all of it ready to be assembled into a TimesMachine. By leveraging the power of AWS and Hadoop, we were able to utilize hundreds of machines concurrently and process all the data in less than 36 hours.


http://open.blogs.nytimes.com/2008/05/21/the-new-york-times-archives-amazon-web-services-timesmachine/?

Comments

Post a Comment

Popular posts from this blog

Vim vi how to reload a file your editing

By
5.8. How do I reload/re-edit the current file? You can use the ":edit" command, without specifying a file name, to reload the current file. If you have made modifications to the file, you can use ":edit!" to force the reload of the current file (you will lose your modifications). For more information, read :help :edit :help :edit! :help 'confirm' from - http://vimdoc.sourceforge.net/cgi-bin/vimfaq2html3.pl#5.8
Read more