diff options
| -rw-r--r-- | docs/Makefile | 10 | ||||
| -rw-r--r-- | docs/cosmos-puppet-ops.mkd | 421 |
2 files changed, 431 insertions, 0 deletions
diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 0000000..2b85134 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,10 @@ + +.PHONY: all +DOCS := $(wildcard *.mkd) +all: $(DOCS:.mkd=.pdf) + +%.pdf: %.mkd + pandoc -o $@ $< + +clean: + rm -f *.html *.pdf *~ diff --git a/docs/cosmos-puppet-ops.mkd b/docs/cosmos-puppet-ops.mkd new file mode 100644 index 0000000..614aa12 --- /dev/null +++ b/docs/cosmos-puppet-ops.mkd @@ -0,0 +1,421 @@ +% System Operations using Cosmos & Puppet +% Leif Johansson / SUNET / 2013 / v0.0.3 + + +Introduction +============ + +This document describes how to setup and run systems and service operations for a small to midsized +systems collection while maintaining scalability, security and auditability for changes. +The process described below is based on opensource components and assumes a Linux-based hosting +infrastructure. These limitations could easily be removed though. This document describes the +multiverse template for combining cosmos and puppet. + + +Design Requirements +=================== + +The cosmos system has been used to operate security-critical infrastructure for a few years before +it was combined with puppet into the multiverse template. + +Several of the design requirements below are fulfilled by comos alone, while some (eg consistency) +are easier to achieve using puppet than with cosmos alone. + +Consistency +----------- + +Changes should be applied atomically across multiple system components on multiple +physical and logical hosts (aka system state). The change mechanism should permit verification of state +consistency and all modifications should be idempotents, i.e the same operation +performend twice on the same system state should not in itself cause a problem. + +Auditability +------------ + +It must be possible to review changes in advance of applying them to system state. It +must also be possible to trace changes that have already been applied to privileged +system operators. + +Authenticity +------------ + +All changes must be authenticated by private keys in the personal possession of privileged +system operators before applied to system state aswell as at any point in the future. + +Simplicity +---------- + +The system must be simple and must not rely on external services to be online to maintain +state except when new state is being requested and applied. When new state is being requested +external dependencies must be kept to a minimum. + +Architecture +============ + +The basic architecture of puppet is to use a VCS (git) to manage and distribute changes to a +staging area on each managed host. At the staging area the changes are authenticated (using +tag signatures) and if valid, distributed to the host using local rsync. Before and after +hooks (using run-parts) are used to provide programmatic hooks. + +Administrative Scope +-------------------- + +The repository constitutes the administrative domain of a multiverse setup: each host is +connected to (i.e runs cosmos off of) a single GIT repository and derives trust from signed +tags on that repository. A host cannot belong to more than 1 administratve domain but each +administrative domains can host multiple DNS domains - all hosts in a single repository +doesn't need to be in the same zone. + +The role of Puppet +------------------ + +In the multiverse template, the cosmos system is used to authenticate and distribute changes +and prepare the system state for running puppet. Puppet is used to apply idempotent changes +to the system state using "puppet apply". + +~~~~~ {.ditaa .no-separation} ++------------+ +------+ +| cosmo repo |---->| host |-----+ ++------------+ +------+ | + ^ | + | | + (change) (manifests) + | | + +--------+ | + | puppet |<---+ + +--------+ +~~~~~ + +Note that there is no puppet master in this setup so collective resources cannot be used +in multiverse. Instead 'fabric' is used to provide a simple way to loop over subsets of +the hosts in a managed domain. + +Private data (eg system credentials, application passwords, or private keys) are encrypted +to a master host PGP key before stored in the cosmos repo. + +System state can be tied to classes used to classify systems into roles (eg "database server" +or "webserver"). System classes can be assigned by regular expressions on the fqdn (eg all +hosts named db-\* is assigned to the "database server" class) using a custom puppet ENC. + +The system classes are also made available to 'fabric' in a custom fabfile. Fabric (or fab) +is a simple frontend to ssh that allows an operator to run commands on multiple remote +hosts at once. + +Trust +----- + +All data in the system is maintained in a cosmos GIT repository. A change is +requested by signing a tag in the repository with a system-wide well-known name-prefix. +The tag name typically includes the date and a counter to make it unique. + +The signature on the tag is authenticated against a set of trusted keys maintained in the +repository itself - so that one trusted system operator must be present to authenticate addition or +removal of another trusted system operator. This authentication of tags is done in addition +to authenticating access to the GIT repository when the changes are pushed. Trust is typically +bootstrapped when a repository is first established. This model also serves to provide auditability +of all changes for as long as repository history is retained. + +Access to hosts is done through ssh with ssh-key access. The ssh keys are typically maintained +using either puppet or cosmos natively. + +Consistency +----------- + +As a master-less architecture, multiverse relies on _eventual consistency_: changes will eventually +be applied to all hosts. In such a model it becomes very imporant that changes are idempotent, so +that applying a change multiple times (in an effort to get dependent changes through) won't cause +an issue. Using native cosmos, such changes are achived using timestamp-files that control entry +into code-blocks: + +``` +stamp="${COSMOS\_BASE}/stamps/foo-v04.stamp" +if ! test -f $stamp; then +# do something here +touch $stamp +fi +``` + +This pattern is mostly replaced in multiverse by using puppet manifests and modules that +are inherently indempotent but it can nevertheless be a useful addition to the toolchain. + +Implementation +============== + +Implementation is based on two major components: cosmos and puppet. The cosmos system was +created by Simon Josefsson and Fredrik Tulin as a simple and secure way to distribute files +and run pre- and post-processors (using run-parts). This allows for a simple, yet complete +mechanism for updating system state. + +The second component is puppet which is run in masterless (aka puppet apply) mode on files +distributed and authenticated using cosmos. Puppet is a widely deployed way to describe +system state using a set of idempotent operations. In theory, anything that can de done +using puppet can be done using cosmos post-processors but puppet allows for greater +abstraction which greatly increases readability. + +The combination of puppet and cosmos is maintained on github in the 'leifj/multiverse' +project. + + +Operations +========== + +Setting up a new administrative domain +-------------------------------------- + +The simplest way is to clone the multiverse repository. First install 'git'. On ubuntu/debian +this is in the 'git-core' package: + +``` +# apt-get install git-core +``` + +Also install 'fabric' - a very useful too for multiple-host-ssh that is integrated into +multiverse. Fabric provides the 'fab' command which will be introduced later on. + +``` +# apt-get install fabric +``` + +These two tools (git & fabric) are only needed on mashines where system operators work. + +Next clone git://github.com/leifj/multiverse.git - this will form the basis of your cosmos+puppet +repository: + +``` +# git clone git://github.com/leifj/multiverse.git myproj-cosmos +# cd myproj-cosmos +``` + +Next rename the upstream from github - you will want to keep this around to get new +features as the multiverse codebase evolves. + +``` +# git remote rename origin multiverse +``` + +Now add a new remote pointing to the git repo where you are going to be pushing +changes for your administrative domain. Also add a read-only version of this remote +as 'ro'. The read-only remote is used by multiverse scripts during host bootstrap. + +``` +# git remote add origin git@yourhost:myproj-cosmos.git +# git remote add ro git://yourhost/myproj-cosmos.git +``` + +Note that you can maintain your repo on just about any git hosting platform, including +github, gitorius or your own local setup as long as it supports read-only "git://" access +to your repository. It is important that the remotes called 'origin' and 'ro' refer to +your repository and not to anything else (like a private version of multiverse). + +Now add at least one key to 'global/overlay/etc/cosmos/keys/' in a file with a .pub extension +(eg 'operator.pub') - the name of the file doesn't matter other than the extension. + +``` +# cp mykey.pub global/overlay/etc/cosmos/keys/ +# git add global/overlay/etc/cosmos/keys/mykey.pub +# git commit -m "initial trust" global/overlay/etc/cosmos/keys/mykey.pub +``` + +At this point you should create and sign your first tag: + +``` +# ./bump-tag +``` + +Make sure that you are using the key whose public key you just added to the repository! You +can now start adding hosts. + +Adding a host +------------- + +Bootstrapping a host is done using the 'addhost' command: + +``` +# ./addhost [-b] $fqdn +``` + +The -b flag causes addhost to attempt to bootstrap cosmos on the remote host using +ssh as root. This requires that root key trust be established in advance. The addhost +command creates and commits the necessary changes to the repository to add a host named +$fqdn. Only fully qualified hostnames should ever be used in cosmos+puppet. + +The boostrap process will create a cron-job on $fqdn that runs + +``` +# cosmos update && cosmos apply +``` + +every 15 minutes. This should be a good starting point for your domain. Now you may +want to add some 'naming rules'. + +Defining naming rules +--------------------- + +A naming rule is a mapping from a name to a set of puppet classes. These are defined in +the file 'global/overlay/etc/puppet/cosmos-rules.yaml' (linked to the toplevel directory +in multiverse). This is a YAML format file whose keys are regular expressions and whose +values are lists of puppet class definitions. Here is an example that assigns all hosts +with names on the form ns\<number\>.example.com to the 'nameserver' class. + +``` +'ns[0-9]?.example.com$': + nameserver: +``` + +Note that the value is a hash with an empty value ('namserver:') and not just a string +value. + +Since regular expressions can also match on whole strings so the following is also +valid: + +``` +smtp.example.com: + mailserver: + relay: smtp.upstream.example.com +``` + +In this example the mailserver puppet class is given the relay argument (cf puppet +documentation). + +Fabric integration +------------------ + +Given the above example the following command would reload all nameservers: + +``` +# fab --roles=nameservers -- rndc reload +``` + + +Creating a change-request +------------------------- + +After performing whatever changes you want to the reqpository, commit the changes as usual +and then sign an appropriately formatted tag. This last operation is wrapped in the 'bump-tag' command: + +``` +# git commit -m "some changes" global/overlay/somethig or/other/files +# ./bump-tag +``` + +The bump-tag command will ask for confirmation before signing and will rely on the git and +gpg commands to create, sign and push the correct tag. + +Puppet modules +-------------- + +Puppet modules can be maintained using a designated cosmos pre-task that reads a file +global/overlay/etc/puppet/cosmos-modules.conf. This file is a simple text-format file +with 3 columns: + +``` +# +# name source (puppetlabs fq name or git url) upgrade (yes/no) +# +concat puppetlabs/concat no +stdlib puppetlabs/stdlib no +cosmos git://github.com/leifj/puppet-cosmos.git yes +ufw git://github.com/fredrikt/puppet-module-ufw.git yes +apt puppetlabs/apt no +vcsrepo puppetlabs/vcsrepo no +xinetd puppetlabs/xinetd no +#golang elithrar/golang yes +python git://github.com/fredrikt/puppet-python.git yes +hiera-gpg git://github.com/fredrikt/hiera-gpg.git no +``` + +This is an example file - the first field is the name of the module, the second is +the source: either a puppetlabs path or a git URL. The final field is 'yes' if the +module should be automatically updated or 'no' if it should only be installed. As usual +lines beginning with '#' are silently ignored. + +This file is processed in a cosmos pre-hook so the modules should be available for +use in the puppet post-hook. By default the file contains several lines that are +commented out so review this file as you start a new multiverse setup. + +In order to add a new module, the best way is to commit a change to this file and +tag this change, allowing time for the module to get installed everywhere before +adding a change that relies on this module. + +HOWTO and Common Tasks +====================== + +Adding a new operator +--------------------- + +Add the ascii-armoured key in a file in `global/overlay/etc/cosmos/keys` with a `.pub` extension + +``` +# git add global/overlay/etc/cosmos/keys/thenewoperator.pub +# git commit -m "the new operator" \ + global/overlay/etc/cosmos/keys/thenewoperator.pub +# ./bump-tag +``` + +Removing an operator +-------------------- + +Identitfy the public key file in `global/overlay/etc/cosmos/keys` + +``` +# git rm global/overlay/etc/cosmos/keys/X.pub +# git commit -m "remove operator X" \ + global/overlay/etc/cosmos/keys/X.pub +# ./bump-tag +``` + +Changing administrative domain for a host +----------------------------------------- + +In the `$new` repository: + +``` +# ./addhost -b $hostname +# fab -H $hostname chrepo repository:git://other/repo.git +``` + +In the `$old` repository: + +``` +# rsync -avz $hostname/ $new/$hostname/ +``` + +In the `$new` repository: + +``` +# git add $hostname/ +# git commit -m "transfer from $old" $hostname +# ./bump-tag +``` + +In the `$old` repository: + +``` +# git rm -rf $hostname +# git commit -m "remove $hostname" +# ./bump-tag +``` + +Reviewing all changes +--------------------- + +Running a command on multiple hosts +----------------------------------- + +On a single host: + +``` +# fab -H $hostname -- command -a one -b another -c +``` + +On multiple hosts based on category: + +``` +# fab --roles=webserver -- ls /tmp +``` + +On all hosts: + +``` +# fab -- reboot # danger Will Robinsson! +``` |