# Infrastructure Familiarization

## Setting a Baseline

Before we start introducing chaos let's take a bit of time to ensure we are meeting a baseline of:

* Service failures can be detected.
* Failed services should restart automatically.
* Logs are collected. (preferably we can collect all debug logs as well)
* Metrics are collected.
* There is some way to view system wide logs and metrics. (Eg Kibana/Grafana)

If you're already doing these things feel free to skip the pages related to them.

## Understanding Your System

Try to collect some information about **each component in the system**:

* What is the minimum number of nodes fulfilling this role that need to be in a healthy state in order to function? (For example, a raft group requires a majority online)
* Is the component stateful? If so, where is state saved? Which data is required by the system to not be lost/corrupted?
* Is the component mission critical? If it entirely fails should the system still work? (albeit in a degraded state)
* Are there any metrics you should be monitoring on this component during your testing? (Eg QPS on a SQL database)

Additionally, from the perspective of **the entire system**:

* What are the main metrics of the system?
* What is a realistic workload for the system? How can you reproduce this?
* At what point do you expect the system to fail? (Eg if you switch off the SQL databases entirely the system should break)
* What is the average cluster topology and size of the system? What is the breaking point of this system? (Eg the majority of nodes of each component is alive)

##