Finding Fractures: An Intro to Differential Fuzzing in Rust

In the year 2050, a malformed JSON input lead to the complete shutdown of the Replicant P2P network. Today, we'll reproduce this bug class in ~100 lines of code.

This post will serve as an introduction to differential fuzzing, with the goal of empowering readers to construct their own differential fuzzing harnesses. At Asymmetric Research, we’ve had a lot of success using differential fuzzing to uncover consensus bugs in various areas, including between the Agave and Firedancer implementations of the Solana validator client.

By comparing the behavior of multiple implementations against the same inputs, it can help uncover discrepancies that might otherwise go unnoticed in individual testing. This approach is especially valuable in decentralized systems, where even minor deviations in client behavior can lead to network forks or security vulnerabilities.

We'll be using LibAFL to build our fuzzers. In the next section, we’ll create a basic JSON round-trip (serialize-then-deserialize) fuzzer using LibAFL and serde_json. After that, we'll turn our basic fuzzer into a differential harness. JSON parsing was chosen as an example target for this exercise, primarily due to the fact that the inputs to the harness are human-readable. The authors will also note that the majority of user interactions with the Solana blockchain happen over JSON-RPC. Bishop Fox has a nice writeup on the topic.

We've created this accompanying repo so that the reader may follow along. Our test system was Ubuntu 24.04 with llvm-18 installed and available, using Rust nightly.

If you're already lost, fear not: our friends at Atredis have written an excellent LibAFL Introductory Workshop, which we highly recommend perusing if you're new to LibAFL or even fuzzing in general.

Rust Fuzzing 101

  [target.x86_64-unknown-linux-gnu]
linker = "/usr/bin/clang-18"
rustflags = ["-Cpasses=sancov-module",
"-Cllvm-args=-sanitizer-coverage-level=4",
"-Cllvm-args=-sanitizer-coverage-trace-pc-guard",
"-Clink-arg=-fsanitize-coverage=trace-pc-guard",    
"-Clink-dead-code",
"-Clink-arg=--ld-path=/usr/bin/ld.lld-18"
            ]

libafl =  {version = "0.15.2", features = ["prelude"]}
libafl_targets =  {version = "0.15.2", features = [
    "sancov_pcguard_edges",
]}
libafl_bolts = {version = "0.15.2", features = ["prelude"]}

[profile.release]
codegen-units = 1
  

let mon = SimpleMonitor::new(|s| println!("{s}"));
  

let mut mgr = SimpleEventManager::new(mon);
  

let edges_observer =
    HitcountsMapObserver::new(unsafe { StdMapObserver::new("edges", &mut EDGES_MAP) });

let time_observer = TimeObserver::new("time");

let map_feedback = AflMapFeedback::new(&edges_observer);
let mut feedback = feedback_or!(map_feedback, TimeFeedback::new(&time_observer));
  

let mut objective = feedback_or_fast!(CrashFeedback::new(), TimeoutFeedback::new());
  

let mut state = StdState::new(
    libafl_bolts::rands::StdRand::with_seed(0),
    CachedOnDiskCorpus::::new("/tmp/corpus", 1024).unwrap(),
    OnDiskCorpus::new(PathBuf::from("/tmp/crashes")).unwrap(),
    &mut feedback,
    &mut objective,
).unwrap();

let scheduler = QueueScheduler::new();
let mut fuzzer = StdFuzzer::new(scheduler, feedback, objective);
  

let mut harness = |input: &BytesInput| -> ExitKind {
    let input_bytes = input.target_bytes();
    if let Ok(input_str) = str::from_utf8(&input_bytes) {
        let serde_json_parsed: Result =
            serde_json::from_str(input_str);

        if let Ok(parsed_value) = serde_json_parsed {
            if let Ok(serde_string) = serde_json::to_string(&parsed_value) {
                let round_trip: Result =
                    serde_json::from_str(&serde_string);

                if let Ok(round_trip_value) = round_trip {
                    if round_trip_value != parsed_value {
                        return ExitKind::Crash;
                    }
                }
            }
        }
    }
    ExitKind::Ok
};
  

let mut executor = InProcessExecutor::new(
    &mut harness,
    tuple_list!(edges_observer, time_observer),
    &mut fuzzer,
    &mut state,
    &mut mgr,
).unwrap();

let mutator = StdScheduledMutator::new(havoc_mutations());
let mut stages = tuple_list!(StdMutationalStage::new(mutator));

if state.corpus().count() < 1 {
    state
        .load_initial_inputs_forced(&mut fuzzer, &mut executor, &mut mgr, &["./corpus".into()])
        .unwrap();
}

println!("Starting fuzzing loop...");
fuzzer
    .fuzz_loop(&mut stages, &mut executor, &mut state, &mut mgr)
    .unwrap();

println!("Fuzzing finished.");
  

Differential Fuzzing 101

Now that we’ve created a standalone pure-Rust fuzzer with coverage feedback using LibAFL, we can begin working on a differential harness. The majority of the harness is exactly the same, so we’ll only show the key differences, namely, a second json parser (the json crate on crates.io), and our basic “verifier” that diffs the two parsing results:

let mut harness = |input: &BytesInput| -> ExitKind {


       let input_bytes = input.target_bytes();
       if let Ok(input_str) = str::from_utf8(&input_bytes) {
	    // run the string through both implementations
           let serde_json_parsed: Result = serde_json::from_str(input_str);
           let json_rs_parsed = json::parse(input_str);


           // baby “verifier”
           if serde_json_parsed.is_ok() != json_rs_parsed.is_ok() {
               eprintln!("mismatch on return type:\n{:?}\n{:?}", serde_json_parsed, json_rs_parsed);
               return ExitKind::Crash;
           }
           // TODO: add state diffing
       }


       ExitKind::Ok
   };

That’s it! We’re off to the races, and our basic “verifier” checks that both implementations are able to successfully parse the same input string. We can run it with:

cargo run --release --target x86_64-unknown-linux-gnu --bin json-diff -- corpus/

We’ll see after a few thousand executions that we begin to find crashes!

[UserStats   #0]  (GLOBAL) run time: 0h-1m-48s, clients: 1, corpus: 126, objectives: 0, executions: 13322, exec/sec: 123.3, edges: 0.027%
                  (CLIENT) corpus: 126, objectives: 0, executions: 13322, exec/sec: 123.3, edges: 710/2621440 (0%)
[Testcase    #0]  (GLOBAL) run time: 0h-1m-48s, clients: 1, corpus: 127, objectives: 0, executions: 13322, exec/sec: 123.3, edges: 0.027%
                  (CLIENT) corpus: 127, objectives: 0, executions: 13322, exec/sec: 123.3, edges: 710/2621440 (0%)
mismatch on return type:
Err(Error("number out of range", line: 1, column: 9))
Ok(Number(Number { category: 1, exponent: 2322, mantissa: 2222 }))
[UserStats   #0]  (GLOBAL) run time: 0h-1m-51s, clients: 1, corpus: 127, objectives: 0, executions: 13322, exec/sec: 119.5, edges: 0.028%
                  (CLIENT) corpus: 127, objectives: 0, executions: 13322, exec/sec: 119.5, edges: 736/2621440 (0%)

Unfortunately, this looks like a crash that we don’t care about - serde_json thinks that the number (222E2322) is out of range and rejects the input, but json-rust thinks that it’s a perfectly fine number - one of the best numbers, in fact. We can work with this. Note: a parser differential like this could be a valid finding for some arbitrary target application, but we don't particularly find this case interesting.

Let’s patch our verifier to ignore this specific case:

     if serde_json_parsed.is_ok() != json_parsed.is_ok() {
               // account for big numbers that serde_json rejects
               if let Err(ref e) = serde_json_parsed {
                   if e.to_string().contains("number out of range") {
                       return ExitKind::Ok
                   }
               }
               eprintln!("mismatch on return type:\n{:?}\n{:?}", serde_json_parsed, json_parsed);
               return ExitKind::Crash;
           }

After a short while running again with this “mismatch” discarded, we get some real bugs!

[Client Heartbeat #0]  (GLOBAL) run time: 0h-7m-40s, clients: 1, corpus: 203, objectives: 1, executions: 56267, exec/sec: 122.3, edges: 0.041%
                       (CLIENT) corpus: 203, objectives: 1, executions: 56267, exec/sec: 122.3, edges: 1075/2621440 (0%)
mismatch on return type:
Err(Error("expected `,` or `}`", line: 9, column: 12))
Ok(Object(Object { store: [("mixed_types_array", Array([Number(Number { category: 1, 
...

Let’s take a look:

Okay, python pygments doesn’t seem to have a problem rendering/colorizing our JSON. Maybe we can spot the error better with hexdump?

> hexdump -C crashes/73e91b865a3791e6 
00000000  7b 0a 20 20 22 6d 69 78  65 64 5f 74 79 70 65 73  |{.  "mixed_types|
00000010  5f 61 72 72 61 79 22 3a  20 5b 31 2c 20 22 73 74  |_array": [1, "st|
00000020  72 69 6e 67 22 2c 20 6e  75 6c 6c 2c 20 74 72 75  |ring", null, tru|
00000030  65 2c 20 7b 22 6f 62 6a  65 63 74 22 3a 20 22 76  |e, {"object": "v|
00000040  61 6c 75 65 22 7d 2c 20  5b 31 2c 20 32 2c 20 33  |alue"}, [1, 2, 3|
00000050  5d 5d 2c 0a 20 20 22 72  65 63 75 72 73 69 76 65  |]],.  "recursive|
00000060  5f 73 74 72 75 63 74 75  72 65 22 3a 20 7b 0a 20  |_structure": {. |
00000070  20 20 20 22 70 72 6f 70  65 72 74 69 65 73 22 3a  |   "properties":|
00000080  20 7b 0a 20 20 20 20 20  20 22 72 65 63 75 72 73  | {.      "recurs|
00000090  69 76 65 22 3a 20 7b 0a  20 20 20 20 20 20 20 20  |ive": {.        |
000000a0  22 70 72 6f 70 65 72 74  69 65 73 22 3a 20 7b 0a  |"properties": {.|
000000b0  20 20 20 20 20 20 20 20  20 20 22 72 65 63 75 72  |          "recur|
000000c0  73 69 76 65 22 3a 20 7b  0a 20 20 20 20 20 20 20  |sive": {.       |
000000d0  20 20 20 20 20 22 70 72  6f 70 66 72 74 69 65 73  |     "propfrties|
000000e0  22 3a 20 7b 7d 0a 20 20  20 20 20 20 20 20 20 20  |": {}.          |
000000f0  7d 0b 20 20 20 20 20 20  20 20 7d 0a 20 20 20 20  |}.        }.    |
00000100  20 20 7d 0a 20 20 20 20  7d 0a 20 20 7d 2c 0a 20  |  }.    }.  },. |
00000110  20 22 77 68 69 74 65 73  70 61 63 65 22 3a 20 22  | "whitespace": "|
00000120  20 20 20 20 6c 65 61 64  69 6e 67 20 61 6e 64 20  |    leading and |
00000130  74 72 61 69 6c 69 6e 67  20 73 70 61 63 65 73 20  |trailing spaces |
00000140  20 20 20 22 0a 7d 0a                              |   ".}.|
00000147

Do you see it?

> bat crashes/73e91b865a3791e6
───────┬────────────────────────────────────────────────────────────────────────────────
       │ File: crashes/73e91b865a3791e6
───────┼────────────────────────────────────────────────────────────────────────────────
   1   │ {
   2   │   "mixed_types_array": [1, "string", null, true, {"object": "value"}, [1, 2, 3]],
   3   │   "recursive_structure": {
   4   │     "properties": {
   5   │       "recursive": {
   6   │         "properties": {
   7   │           "recursive": {
   8   │             "propfrties": {}
   9   │           }^K        }
  10   │       }
  11   │     }
  12   │   },
  13   │   "whitespace": "    leading and trailing spaces    "
  14   │ }
───────┴────────────────────────────────────────────────────────────────────────────────

A control character!

> python3 -c 'print("hello\vworld")'
hello
     world

Once again, serde_json is much stricter than json-rust, and we can see that strictness is explicitly a non-goal of json-rust:

“Mapping that (very loose format) to idiomatic Rust structs introduces friction...this crate intends to avoid that friction.”

Next Steps

There might be many more “uninteresting” differences between the two json libraries that the programmer would have to iterate through and “discard,” or, one of the differences we’ve already found might be enough to cause a consensus divergence between different blockchain implementations! The less-opinionated json-rust crate still gets tens of thousands of downloads per day, which isn't a problem by itself. However, if someone building a new Rust consensus client wasn't aware that there's no json crate in the rust stdlib, and haphazardly went with this crate, a two-byte difference can fork consensus - here's proof. GitHub CodeSearch can be useful when looking for potentially vulnerable projects.

There are many things outstanding in order to improve the fuzzer, which are left as an exercise to the reader (until the next blog in this series 😉):

• LibAFL supports different types of Inputs - right now, we’re using an uninformed binary input to fuzz JSON, which is likely inefficient, given the structured nature of JSON.
• Our fuzzer, at present, only runs on a single core, with no idea of parallelization.
• There’s no convenient way to “replay” a generated crash file through the harness.
• Perhaps most importantly, there's no check for consistency/equivalence between the two generated data structures - our harness is only searching for cases where one input is rejected and the other is accepted.

Takeaways

After navigating the treacherous waters of pure-Rust coverage instrumentation, we were able to write, from scratch, a pure Rust JSON round-trip fuzzer. We then extended it into a differential fuzzing harness that’s capable of finding legitimate mismatches in behavior between target JSON implementations.

In the world of web3, a parser differential such as the one we discovered in this blog post could be (and often is) load-bearing for consensus interactions in a multi-client blockchain ecosystem. We'll see you soon for part two.