喜欢设计高可用系统的朋友可以借鉴
HDFS 有两个核心组件，fsimage 和 edits 操作日志
fsimage -- contains a serialized form of all the directory and file inodes in the filesystem. Each inode is an internal representation of a file or directory's metadata and contains such information as the file's replication level , modification and access times , access permissions, block size, and the blocks a file is made up of. For directories , the modification time, permissions, and quota metata is stored.
edits -- grow without bound.
主从复制要解决的问题
If the namenode is restarted, it would take a long time to apply each of the operation in its edit log. During this time, the filesystem would be offline, which is generally undesirable.
解决方案：
Run the secondary namenode, whose purpose is to produce check-points of the primary's in-memory metadata. The checkpointing process proceeds as follows:
1）The secondary asks the primary to roll its edits file, so new edits go to a new file
2) The secondary retrieves fsimage and edits from primary (using HTTP GET).
3) The secondary loads fsimage into memory, applies each operation from edits, then creates a new consolidated file.
4) The secondary sends the new fsimage back to primary (using HTTP POST)
5) The primary replaces the old fsimage with the new one from the secondary, and the old edits file with the new one it started in step 1. It also updates the fsimage file to record the time that the checkpoint was taken.
此算法也可以到 MongoDb 设计中的 脏数据整理的过程。
比如说
File 1 added -> File 1 modified -> File 1 deleted
这个过程会产生两个拷贝，已经没有索引会指向这个上文档了，
如果在发生前两个步骤的时候做 check point 而且 文档数据库是多个文件rotate模式， 就可以多线程并行的运行这个算法来合并对同一个文档的修改
阅读(1514) | 评论(0) | 转发(0) |
