Amazon S3 architecture

While the post @ http://highscalability.com/amazon-architecture explains Amazon's architecture in general, I am interested in knowing how Amazon S3 is implemented.

Some of my guesses are

1. A distributed file system like HDFS http://hadoop.apache.org/core/docs/current/hdfs_design.html
2. A non relational persistent DB like CouchDB http://couchdb.apache.org/

Is it be possible to implement something similar to this on a much smaller scale using scripting languages like Python or PHP?

From stackoverflow

• There is at least one S3 equivalent solution that you can use for testing clients against. You might find some information by studying the source code and maybe you could use it in your own project.

• Look at MogileFS. Its not API compatible with S3, and has a lot less features than S3, but it does the basic process of distributing static files over a set of low-cost storage servers.

• Amazon S3 is implemented using the architecture described in the Dynamo Paper:

  http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

  The paper explains consistent hashing, and how and why the guarantee is "eventual consistency".

  The conflict resolution they talk about for Dynamo is not exposed to users of S3. It is used internally in Amazon's applications, but for S3, the only conflict resolution is last write wins.

  Edit: Werner Vogels has said "Dynamo is not directly exposed externally as a web service; however, Dynamo and similar Amazon technologies are used to power parts of our Amazon Web Services, such as S3." http://www.allthingsdistributed.com/2007/10/amazons_dynamo.html

  I would emphasize that he isn't saying S3 and Dynamo share components, he explicitly says that Dynamo itself is one of the technologies that power S3. Everything I've seen from S3, including the caveats, is accounted for by assuming S3 is a fancy web services wrapper around Dynamo with authentication, accounting, and a last-write-wins conflict resolve that is invisible to the user.

  The original question was about the underlying storage mechanism for S3. It is explicitly not a distributed file system like HDFS or a non-relational database like CouchDB. Dynamo fills this role.

  StaxMan : I don't think this is quite true. Dynamo's goals are rather different from S3 (Dynamo is used for shopping carts and similar use cases, lots of writes, app-provided conflict resoluton, short values) than S3's (simple key/value store, with large values, no content-based conflict resolution or merging. And as such, internal impl & design are quite different (AFAIK). Architecturally they share many things, and some internal components too.

• It's closer to 2, although with content stored as "BLOBs" without system caring about contents, whereas CouchDB does. Backend storage uses a local DB (BDB?) for nodes of clusters used to store multiple copies. Reads can go to any node that has a copy, as can writes, but writes need to be resolved to get rid of conflicts. As Kevin mentions, this guarantees "eventual consistency", but gives no strict guarantees of when, or which write wins (from external POV; internally that is defined).

  Reading Dynamo docs is useful in understanding many of the concepts, but AFAIK implementation is different. Dynamo is used internally by Amazon for other uses. There are also open source implementations of both; one interesting one is Project Voldemort. CouchDB is obviously very interesting as well.