A Menagerie of Tools

In the following section we'll talk about a group of tools which can be used for Chaos testing. Some, such as kill may already be familiar to you. Others like pumba may be entirely new.

These tools only represent some of the tools available. You may find others, or even make your own.

Triggering Events

While you can run many of these commands in a manual way, you can also orchestrate running these commands by using an agent installed on your nodes.

You could also use the ideas below for some simple testing.

Timers

You could also use things such as systemd timers (utilizing RandomizedDelaySec) like so:

/etc/systemd/system/kill-postgres.timer

[Unit]
Description = Kill postgres eventually.

[Timer]
OnActiveSec = 0
RandomizedDelaySec = 1000
Unit = kill-postgres.service

[Install]
WantedBy = timers.target

/etc/systemd/system/kill-postgres.service

[Unit]
Description = Kill postgres.

[Service]
ExecStart = /usr/bin/killall -s KILL postgres

Socket Activation

You can alternatively trigger failures via systemd's socket activation feature:

/etc/systemd/system/kill-postgres.socket

[Unit]
Description = Kill postgres on message.

[Socket]
ListenStream = 127.0.0.1:8000

[Install]
WantedBy = sockets.target