diff options
Diffstat (limited to 'doc')
-rw-r--r-- | doc/Makefile | 6 | ||||
-rw-r--r-- | doc/db.md | 95 |
2 files changed, 101 insertions, 0 deletions
diff --git a/doc/Makefile b/doc/Makefile new file mode 100644 index 0000000..05b987d --- /dev/null +++ b/doc/Makefile @@ -0,0 +1,6 @@ +ALL = db.html + +all: $(ALL) + +%.html: %.md + markdown $< > $@ diff --git a/doc/db.md b/doc/db.md new file mode 100644 index 0000000..9b7f9d3 --- /dev/null +++ b/doc/db.md @@ -0,0 +1,95 @@ +# plop database + +The plop database is a distributed, single-master, append-only +database suitable for transparency systems like Certificate +Transparency. + +Data entries are stored together with three attributes: + +- index + + integer; the first entry has index 0, the next one 1 and so on + +- entry hash + + the hash over the entry, used for duplicate detection + +- leaf hash + + hash over specific parts of the entry, usually together with a + timestamp for use in a merkle tree + +## Storage in a file system + +Two files (catlfish names in parentheses): + +- treesize (treesize) + + filename is static, contains one line -- the number of entries in + the database + +- index (index) + + filename is static, contains one line per entry -- the leafhash + +Three directories, "bucketed" in three levels, one file per database +entry (catlfish names in parentheses): + +- entry (certentries) + + filename=leafhash, content=the actual data of the entry + +- entryhash (entryhash) + + filename=entryhash, content=leafhash + +- indexforhash (certindex) + + filename=leafhash, content=index + +## Distributed + +TODO: describe distribution + +## Erlang code in src/ + +- db.erl + + public interface for adding entries and getting entries by index, + leaf hash and entry hash + +- index.erl + + file-based storage for ordered append-only lists of fixed-sized + entries, retrievable by index + +- perm.erl + + reading and writing of files + +- atomic.erl + + atomic file operations + +- util.erl + + helper functions for lower level file handling + +- fsyncport.erl + + interface to C implementation for fsync(2) syscall + +## C code in c_src/ + +- net_read_write.c + + read and write to/from a file descriptor, using fsync(2) to increase + probability that data lands on disk + +- fsynchelper.c + + erlang port for net_read_write + +- erlport.c + + glue |