diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/design.txt | 162 | ||||
| -rw-r--r-- | doc/minimalsystem.txt | 92 |
2 files changed, 0 insertions, 254 deletions
diff --git a/doc/design.txt b/doc/design.txt deleted file mode 100644 index 7a76c6a..0000000 --- a/doc/design.txt +++ /dev/null @@ -1,162 +0,0 @@ --*- markdown -*- - -Overview -======== - -This document describes the design of catlfish, an implementation of a -Certificate Transparency (RFC6962) log server. - - - - +------------------------------------------------+ - | front end nodes | - +------------------------------------------------+ - ^ | | - | v v - | +---------------+ +---------------+ - | | storage nodes | | signing nodes | - | +---------------+ +---------------+ - | ^ ^ - | | | - +------------------------------------------------+ - | primary merge node | - +------------------------------------------------+ - ^ | - | v - | +----------------------------------+ - | | secondary merge nodes | - | +----------------------------------+ - | ^ - | | - +-------------------+ - | merge-repair node | - +-------------------+ - - - -Design assumptions ------------------- -* The database grows with 5 GB per year, based on 5,000 3 kB - submissions per day -* Max size is 300 GB, based on 100e6 certificates -* submissions: less than 0.1 qps, based on 5,000 submissions per day -* monitors: 6 qps, based on 100 monitors -* auditors: 8,000 qps, based on 2.5e9 browsers visiting 100 sites - (with a 1y certificate) per month (assuming a single combined - request for doing get-sth + get-sth-consistency + get-proof-by-hash) - - -Terminology -=========== - -CC = Certificate Chain -CT log = Certificate Transparency log - - -Front-end node -============== - -* Handles all http requests. -* Has a complete copy of the published data locally. -* Read requests are answered directly by reading local files - and calculating the answers. -* Add requests are validated and then sent to all storage - nodes. At the same time, a signing request is sent to one or - more of the signing nodes. When responses have been received - from a predetermined number of storage nodes and one signing - response has been received, a response is sent to the client. -* Has an inward-facing API with the entry points SendLog(Hashes), - MissingEntries() (returns a list of hashes), SendEntry(Entry), - SendSTH(STH), CurrentPosition(). - - -Storage node -============ - -* Stores certificate chains and SCTs. -* Has a write API SendEntry(Entry) that stores the certificate chain - in a database, indexed by its hash. Then stores the hash in a list - NewEntries. -* Takes reasonable measures to ensure that data is in permanent - storage before sending a response. -* When seeing a new STH, moves the variable start to the index of the - first unpublished hash. -* Has a read API FetchNewEntries() which returns - NewEntries[start...length(NewEntries)-1]. - - -Signing node -============ - -* Has the signing key for the log or talks to an HSM. - - -Merging node -============ - -* The primary merge node is determined by configuration. -* The other merging nodes are called "secondaries". -* The primary merge node has two phases, merging and distributing. - -Merging (primary) ------------------ - -* Fetches CCs by calling FetchNewEntries() on storage node i - where i = 0...(n-1) -* Determines the order of the new entries in the CT log. -* Sends the entries to the secondary merge nodes. -* Calculates the tree head and asks a signing node to sign it. -* When a majority of the secondaries have acknowledged the entries, - compares the calculated tree head to the tree heads of the secondaries. - If they match, considers the additions to the CT log final and - begins the distributing phase. - -Merging (secondaries) ---------------------- - -* Receives entries from the primary merge node. The node must be certain - that the request comes from the current primary, and not - an old one. -* Takes reasonable measures to ensure that data is in - permanent storage. -* Calculates the new tree head and returns it to the primary merge node. - -Distributing ------------- - -* Performs the following steps for all front-end nodes: - * Fetches curpos by calling CurrentPosition(). - * Calls SendLog() with the hashes of CCs from curpos to newpos. - * Fetches missing_entries by calling MissingEntries(), a list - of hashes for the CCs that the front-end nodes does not - have. - * For each hash in missing_entries, upload the CC by calling - SendEntry(CC). - * Send the STH with the SendSTH(STH) call. - - -Merge-repair node -================= - -* There is only one of these nodes. -* When this node detects that an STH has not been published - in t seconds, it begins the automatic repair process. - -Automatic repair process ------------------------- - -* Turn off all reachable merge nodes. -* If a majority of the merge nodes cannot be reached, - die and report. -* Fetch the CT log order from the merge nodes. -* Determine the latest version of the log. -* Select a new primary merge node. -* Change the configuration of the merge nodes so that - they know who the new primary merge node is. -* Start all merge nodes. -* If any of these steps fail, die and report. -* If all steps succeed, die and report anyway. The automatic - repair process must not be restarted without manual - intervention. - - diff --git a/doc/minimalsystem.txt b/doc/minimalsystem.txt deleted file mode 100644 index 061b6cc..0000000 --- a/doc/minimalsystem.txt +++ /dev/null @@ -1,92 +0,0 @@ -<!-- -*- markdown -*- --> - -Setting up a minimal system -=========================== - -To setup a minimal system, first make sure you have the "Requirements" -from README.md and then do: - - make - make release - - -Removing files from your previous session ---------------------------------------- - - rm -f /tmp/cert1-sct - rm -rf /tmp/testcert1 - - -Setting up the environment ----------------------- - - mkdir /tmp/testcert1 - make tests-prepare - -This will configure the servers to run on 127.0.0.1 port 8080, 8081, -and 8082. If you want to change that, change the configuration files -`test/*.cfg`. - -Starting the servers --------------------- - - make tests-start - -Submitting certificates ------------------------ - - tools/submitcert.py --store tools/testcerts/cert1.txt \ - --check-sct --sct-file=/tmp/cert1-sct https://127.0.0.1:8080/ - -This submits the certificate (in PEM format) in the file -`tools/testcerts/cert1.txt` and appends the resulting SCT to the file -`/tmp/cert1-sct`. It also checks the signature of the SCT. - -If you want to submit all the files in a directory, name directory -with a `/` at the end, for example `tools/testcerts/`. - - -Running merge once ------------------- - - tools/merge.py --config test/catlfish-test.cfg - --localconfig test/catlfish-test-local-merge.cfg - -This will read the submitted certificates from the storage node, -decide the order, and publish the certificates to the frontend server. - -If you want to run the system continuously, run the merge command in -cron or in a while loop. See `packaging/docker/catlfish-dev/merge.sh` -for an example of the latter. - -Verifying SCT:s ---------------- - - tools/verifysct.py --sct-file /tmp/cert1-sct \ - --publickey=tests/keys/logkey.pem \ - --cafile tests/httpsca/demoCA/cacert.pem \ - https://127.0.0.1:8080/ - -This verifies that all the certs corresponding to the SCT:s in the -file `/tmp/cert1-sct` are actually present in the log. - -Running a monitor ------------------ -An easy way to run a monitor is: - - while true; do - ./fetchallcerts.py --store /tmp/testcert1 \ - --write-sth https://127.0.0.1:8080/; - sleep 10 - done - -This will fetch all the certificates in a log, verify the consistency -proof between the old and new versions, and save the new STH. - -Stopping the system -------------------- - - make tests-stop - -It is important that the system is stopped before running `make -release` again, or the release build will fail. |