Add --release flag

[YanVictorSN]

• Add --release flag

Close #20

@smoelius

[smoelius]

@YanVictorSN Thanks very much for doing this.

Do you have any interest in writing a test for the new flag?

I realize I didn't say anything about that in #20. So "no" is a perfectly acceptable answer.

[YanVictorSN]

@YanVictorSN Thanks very much for doing this.

Do you have any interest in writing a test for the new flag?

I realize I didn't say anything about that in #20. So "no" is a perfectly acceptable answer.

Yes, sure! Is it possible for me to do that in another PR? Also, I'm just learning Rust—if you can point me to examples in the code or share any tips, I’d really appreciate it.

[smoelius]

Yes, sure! Is it possible for me to do that in another PR?

Sure.

Also, I'm just learning Rust—if you can point me to examples in the code or share any tips, I’d really appreciate it.

I'll just record some notes here.

I imagine building a test target whose behavior changes based on whether --release is passed.

The easiest way I can think to do this is:

#[test]
fn test() {
    target();
}

#[test_fuzz::test_fuzz]
fn target() {
    #[cfg(not(debug_assertions))]
    assert(false);
}

This should work because debug_assertions is off when building in release mode.

The above target code should go in a file in the examples/tests directory.

Then there should be a test in the cargo-test-fuzz/tests/integration directory that exercises the target code as follows:

• test it with cargo test
• run cargo test-fuzz on it in two ways:
  • without --release and verify success (i.e., no crash)
  • with --releaseand verify failure (i.e., crash)

The "run cargo test-fuzz" step should use this command:

test-fuzz/cargo-test-fuzz/tests/integration/fuzz.rs

Line 35 in 2543774

let mut command = examples::test_fuzz(krate, "target").unwrap();

krate will be the name of the file containing the target code.

Finally, the "run cargo test-fuzz" step should specify a timeout so that if the target code doesn't crash, the test doesn't run forever.

I think 80% of this could be achieved by copying one of the fuzz_* tests in the cargo-test-fuzz/tests/integration directory. But I am happy to answer questions if anything is unclear.

[YanVictorSN]

@smoelius Thank you so much. I’ll do that. And if I have any questions, I will definitely reach out to you.

[smoelius]

I've done some experiments locally, and I think this is working as expected. I'm going to hold off on releasing this until we have a test in CI, though.

Thanks again!

[YanVictorSN]

Thank you! @smoelius I'm trying to create the test, but I'm having some trouble. Should I open a draft or can we chat here?

[smoelius]

Either is fine. Whatever is easiest for you.

[YanVictorSN]

@smoelius Do I need to pass any value in the target? I know I need to test the release tag, but I need to initialize the corpus directory like in the fuzz file to check if it will crash, right?

[smoelius]

Direct answer to your question: yes, I was making a mistake, because AFL++ doesn't like and rejects empty corpus files, and "no arguments" would serialize to an empty corpus file.

But I just realized that my debug_assertions idea won't work, because cargo-afl spefically enables debug_assertions, regardless of whether building in release mode: https://github.com/rust-fuzz/afl.rs/blob/e586a66aadc36977501257ee8b8201d61a452021/cargo-afl/src/main.rs#L285

Here are the things that are enabled in release mode: https://doc.rust-lang.org/cargo/reference/profiles.html#release

Hmm... What else in there could be used to produce a crash in release mode? (It's not immediately obvious to me.)

[smoelius]

Maybe there's a way to turn the absence of overflow checks into a crash?

[smoelius]

Something along these lines maybe? https://stackoverflow.com/a/73224634

Is this something you would like to play with?

I realize I've imposed upon you by asking you to write this test. If this has become too much of a hassle, I completely understand.

[YanVictorSN]

Yeah, sure. Don’t worry, I’m learning a lot. You make things easier for a beginner like me. So, the idea here is to try to force an overflow to check if the release tag is working or not, right?

[smoelius]

Overflow checks will be enabled in debug mode, but not in release mode.

On the face of it, that means one could get a panic/crash in debug mode, but not in release mode. But we want the other way around.

So, I'm wondering, could we catch the panic with std::panic::catch_unwind, and then do the opposite, i.e.:

• if a panic is caught, silently return.
• if no panic is caught, panic.

Does that make sense?

[smoelius]

🤦 No that's not going to work either! https://github.com/rust-fuzz/afl.rs/blob/e586a66aadc36977501257ee8b8201d61a452021/cargo-afl/src/main.rs#L286

[smoelius]

Ok, this has become more complicated than I expected.

It might be possible to use the PROFILE environment variable: https://doc.rust-lang.org/cargo/reference/environment-variables.html#environment-variables-cargo-sets-for-build-scripts

But the documentation suggests that variable is set only for build scripts. In other words, writing a test that relies on that variable would require writing a build script. That could be done, but I can't quite see how all of the pieces would fit together.

I'm going to have to thing about this some more, unless you can see a solution.

[YanVictorSN]

Not yet. I’ll think more about it and, in the meantime, look for another good_first_issue in the repo. Thanks!