1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
|
# soc_collector -- Gathering vulnerability information and presenting it
## The oneliner
The collector answers the fundamental question constantly posed by all
SOC staff ever: Can we have lunch now?
## The elevator pitch
You're working as a Security Operations Center engineer and your job
is to, one, know when any part of your infrastructure is vulnerable
and, two, if it is, do something smart about it.
The collector compiles data from vulnerability scanners and stores the
data in a database. You query the collector for the current
vulnerability status of your network infrastructure.
Without a summary of your vulnerability status and the ability to quickly
deepen your knowledge of a given system, your chances of ever eating
lunch with a clear conscience are slim.
## The user interface
TODO
## The gory^Wtechnical details
TODO
## The name
The "soc" part means Security Operations Center.
The "collector" part is correct but misleading since `soc_collector`
also processes and presents.
## The license
This code is licensed under the 2-Clause BSD License, see LICENSE for
the full text.
## How to test it out
The collector has been tested on Debian 11 (Bullseye). Other Unix
systems should also be capable of running a collector.
Clone the repository.
git clone https://git.sunet.se/soc_collector.git
Install dependencies (Debian).
sudo apt install docker.io docker-compose jq curl apache2-utils
Start the collector and JWT server, and generate certificates for JWT signing:
./quickstart.sh
Now the database and the API server should be running, now we can try
adding some observations. First, get a JWT for the default user `usr`:
JWT=$(curl http://localhost:8000/api/v1.0/auth -X POST -p -u usr:pwd | jq -r .access_token)
Then we use the JWT to add an observation (defined in `example_data.json`):
curl -s --data-binary @example_data.json -H "Authorization: Bearer $JWT" http://localhost:80/sc/v0/add
Try retreiving all observations permitted by our JWT:
curl -s -H "Authorization: Bearer $JWT" http://localhost:80/sc/v0/get | json_pp -json_opt utf8,pretty
We might also filter the data:
curl -s -H "Authorization: Bearer $JWT" http://localhost:80/sc/v0/get?port=111 | json_pp -json_opt utf8,pretty
Believe it or not, but we can also get a single observation by looking up its key (_id):
curl -s -H "Authorization: Bearer $JWT" http://localhost:80/sc/v0/get/1633633714355 | json_pp -json_opt utf8,pretty
We can also limit the number of results and skip N results forward with the parameters limit and skip:
curl -s -H "Authorization: Bearer $JWT" 'http://localhost:80/sc/v0/get?limit=5&skip=2' | json_pp -json_opt utf8,pretty
## Development
There are two docker-compose files used for development:
- `docker/docker-compose-dev.yaml` for the collector, and
- `auth-server-poc/docker-compose.yml` for the JWT server.
To apply changes, build with `docker-compose build -f docker/docker-compose-dev.yaml` or `docker-compose -f auth-server-poc/docker-compose.yml` (depending on what has changed) and then restart the containers with `./quickstart.sh`.
If you want to save build time you can also pass a service name to `docker-compose build`, i.e. `docker-compose build -f docker/docker-compose-dev.yaml collector`.
## JWT mechanics (work in progress)
2021-11-24: Currently no checks except that the JWT is valid are performed when
adding observations. When retrieving observations, the JWTs "domains" claim is
used. In auth-server-poc, domains is hard-coded to `["sunet.se"]` as an example.
|
