> For the complete documentation index, see [llms.txt](https://pingcap.gitbook.io/chaos-workshop/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://pingcap.gitbook.io/chaos-workshop/failure-as-a-feature.md).

# Failure as a Feature

Sometimes just creating failures at random, or introducing fuzziness external to the executable in question isn't quite enough.

Perhaps you suspect your system may be susceptible to failures during one particular part of code, or you'd like to reliably cause a failure in one component, and introduce fuzziness to another.

One tactic for doing this is to use "Fail points."

## What's a Fail Point?

A fail point is a configurable defined marker in your code.

You can find fail point libraries for some common programming languages.

* <https://github.com/pingcap/fail-rs>
* <https://github.com/etcd-io/gofail>

What does a fail point look like?

```rust
fn example_function() {
    println!("I hope I don't fail...");
    fail_point!("here");
    println!("Whew, I survived!");
}
```

Or inject an error (or something else), instead of a panic:

```rust
fn get(&self, key: &Key) -> Result<Value> {
    fail_point!("get_fails_with_error", |_| Err(box_err!(
        "injected error for get"
    )));
    // ...
}
```

Running it:

```bash
FAILPOINTS="here=panic" cargo run
# I hope I don't fail...
# thread 'main' panicked at 'failpoint here panic', /home/hoverbear/.cargo/registry/src/github.com-1ecc6299db9ec823/fail-0.2.0/src/lib.rs:286:25
# note: Run with `RUST_BACKTRACE=1` for a backtrace.
```

## Where to Place Fail Points

You can use fail points to simulate simple failures, such as a write failure:

```rust
impl<S: RaftStoreRouter> Engine for RaftKv<S> {
    type Iter = RegionIterator;
    type Snap = RegionSnapshot;

    fn async_write(
        &self,
        ctx: &Context,
        modifies: Vec<Modify>,
        cb: Callback<()>,
    ) -> engine::Result<()> {
        fail_point!("raftkv_async_write");
        // ...
```

Placing failures at critical sections of your code can allow you to, for example, cause a panic after getting a snapshot sent from a peer:

```rust
/// Event handler for the completion of get snapshot.
///
/// Delivers the command along with the snapshot to a worker thread to execute.
fn on_snapshot_finished(
    // ...
) {
    fail_point!("scheduler_async_snapshot_finish");
    // ...
```

Or failure during an iterator:

```rust
impl Snapshot for RegionSnapshot {
    type Iter = RegionIterator;

    // ...
    fn iter(&self, iter_opt: IterOption, mode: ScanMode) -> engine::Result<Cursor<Self::Iter>> {
        fail_point!("raftkv_snapshot_iter", |_| Err(box_err!(
            "injected error for iter"
        )));
        Ok(Cursor::new(RegionSnapshot::iter(self, iter_opt), mode))
    }
    
    // ...
```


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://pingcap.gitbook.io/chaos-workshop/failure-as-a-feature.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.