C++ Weekly - Ep 352 - Not Doing This Should Be Illegal! (Always Fuzz Your C++!)
Vložit
- čas přidán 27. 11. 2022
- ☟☟ Awesome T-Shirts! Sponsors! Books! ☟☟
Upcoming Workshop: Understanding Object Lifetime, C++ On Sea, July 2, 2024
► cpponsea.uk/2024/sessions/und...
Upcoming Workshop: C++ Best Practices, NDC TechTown, Sept 9-10, 2024
► ndctechtown.com/workshops/c-b...
Episode Notes and code samples: github.com/lefticus/cpp_weekl...
T-SHIRTS AVAILABLE!
► The best C++ T-Shirts anywhere! my-store-d16a2f.creator-sprin...
WANT MORE JASON?
► My Training Classes: emptycrate.com/training.html
► Follow me on twitter: / lefticus
SUPPORT THE CHANNEL
► Patreon: / lefticus
► Github Sponsors: github.com/sponsors/lefticus
► Paypal Donation: www.paypal.com/donate/?hosted...
GET INVOLVED
► Video Idea List: github.com/lefticus/cpp_weekl...
JASON'S BOOKS
► C++23 Best Practices
Leanpub Ebook: leanpub.com/cpp23_best_practi...
► C++ Best Practices
Amazon Paperback: amzn.to/3wpAU3Z
Leanpub Ebook: leanpub.com/cppbestpractices
JASON'S PUZZLE BOOKS
► Object Lifetime Puzzlers Book 1
Amazon Paperback: amzn.to/3g6Ervj
Leanpub Ebook: leanpub.com/objectlifetimepuz...
► Object Lifetime Puzzlers Book 2
Amazon Paperback: amzn.to/3whdUDU
Leanpub Ebook: leanpub.com/objectlifetimepuz...
► Object Lifetime Puzzlers Book 3
Leanpub Ebook: leanpub.com/objectlifetimepuz...
► Copy and Reference Puzzlers Book 1
Amazon Paperback: amzn.to/3g7ZVb9
Leanpub Ebook: leanpub.com/copyandreferencep...
► Copy and Reference Puzzlers Book 2
Amazon Paperback: amzn.to/3X1LOIx
Leanpub Ebook: leanpub.com/copyandreferencep...
► Copy and Reference Puzzlers Book 3
Leanpub Ebook: leanpub.com/copyandreferencep...
► OpCode Puzzlers Book 1
Amazon Paperback: amzn.to/3KCNJg6
Leanpub Ebook: leanpub.com/opcodepuzzlers_book1
RECOMMENDED BOOKS
► Bjarne Stroustrup's A Tour of C++ (now with C++20/23!): amzn.to/3X4Wypr
AWESOME PROJECTS
► The C++ Starter Project - Gets you started with Best Practices Quickly - github.com/cpp-best-practices...
► C++ Best Practices Forkable Coding Standards - github.com/cpp-best-practices...
O'Reilly VIDEOS
► Inheritance and Polymorphism in C++ - www.oreilly.com/library/view/...
► Learning C++ Best Practices - www.oreilly.com/library/view/... - Věda a technologie
I prefer the less clickbaity titles personally.
Coming from an FPGA background, constrained random verification, or fuzz testing, as you're calling it, is a known technique since the mid 2000s. When it gained popularity there was a push to integrate random tests into all designs, as can be read from some books at that time. But simply put, not every design stands to benefit much from fuzzing.
If you're testing a public API for a large popular project and you have the manpower to integrate, and, more importantly automate fuzz test handling, yes, this is an incredibly useful feature. But for a small project with well known and understood input/output limitations a few directed tests will bring much more immediate benefit. The main benefit of fuzzing is finding unusual input sequences. If your inputs are constrained, you will need to constrain the fuzzer and automatically discern false positives from real errors. This is a task that can cost a non trivial amount of manpower for small teams, and the potential benefit will simply not always be there, when compared to directed tests.
Anyway, this definetly shouldn't be illegal, not even the first recommended tool in many situations.
Sadly, also gonna agree. The clickbait title is terrible for ever finding this video again :(
"youtube jason turner fuzzing" ?
@@GeorgeTsiros The title was adjusted sometime after I made this comment. The original title was just "C++ Weekly - Ep 352 - Not Doing This Should Be Illegal!"
Thanks Jason for the great video. I have to say, contrary to what you say at 7:20, there is (at least) one static tool that can detect this problem. It's Polyspace Code Prover from MathWorks (disclaimer : I am working for this company).
An empty string should also trip the fuzz testing instrumentation, since you check for the open brace without first checking that the string has a front() character.
Yes! Recently, CppCon video of Herb Sutter's cppfront talk was released. I said to myself "how hard could it be to fuzz it?" it turned out, not much. I'm only on Windows with Visual Studio, but I already managed to find a bug in his tool in hour or two thanks to fuzzing. It was not security vulnerability, it was infinite loop while parsing bad input. Now, when I'm thinking about it, it could be used to DoS/timeout/increase cost of CompilerExplorer virtual machines.
Excellent vid, thank you Jason.
From the legal perspective, it is somewhat defined what you need to do should your project be deployed in an environment where public safety/security is at risk.
The standards I know of that describe, among much much more, all the testing techniques you need to do to conform to this standard. For safety, the standard is IEC 61508 - it defines Safety Integrity Levels (SIL) where SIL4 (the highest level) requires you to do fuzzing. Then, for security, there is IEC 62443 - it defines Security Levels, where each level also mandates you to do certain kinds of tests (including fuzzing). In some applications, conformity to those two standards is required.
Oh wow thanks, I was not aware of those standards. I'll check them out.
The description has zero information in it
Not true. It tells you how to buy his merch ;)
And it has the link to the github issue with all of the code samples!
Awesome, I'll check that out and add it to my list.
I’d love to see you also cover property based testing in C++ with RapidCheck, which uses similar techniques but for correctness rather than vulnerabilities, (though it can also catch vulnerabilities, of course)
people know they should fuzz but everyone knows you also cant just go "open fuzzer, open exe, run" you have to manually go in and edit your code to tie the fuzzer into the input field. The sanitizers are not like this so they get more use.
How would you start fuzzing in a big legacy codebase (closed source code)?
I have limited resources so I would like to get the most out of this.
start at the beginning if you plan to get full coverage, or start with the functions that have the most complex interactions of parameters if you don't need full coverage. For example, you don't need to fuzz test the function:
double GetAverage(const vector elements) {
double avg = 0;
if (elements.size())
for (double el : elements) avg += el / elements.size();
return avg;
}
because it has very simple inputs and you can easily manually check that the corner cases are properly handled by looking at it for 2 seconds. But if you have a large function with a lot of inputs and a lot of "if" statements that take all sorts of different paths based on complex boolean conditions involving multiple inputs at a time, that might be a good one to fuzz test because properly analyzing it would require a giant, multidimensional table to.
Also, like he mentioned in the video, you ALWAYS want to fuzz test any functions that are on the user side of a trust boundary, meaning any function that has inputs that come directly or indirectly from user input that hasn't been sanitized yet. Like if you have a function that parses JSON, you definitely want to fuzz test that because you don't want some cheeky little 7th grader hacker-in-training to crash your entire server by making a web request that contains an invalid JSON string.
Most click-bait title you’ve ever had… like button immediately clicked.
One question about fuzzing: a lot of high-performance code is about contracts to omit unnecessary checks. How can this code be fuzz-tested? For example, an algorithm could have some documented (or even debug-asserted) conditions on the input. The fuzz-tester however does not care for those. Should I just write some code that does a check before calling the function and "discard" the fuzz input if necessary or are there other approaches?
I'd be glad for any hint on how these cases are handled, thanks!
If your API is public, it *will* be used wrong by someone. I am not a fan at all of the "they promise to use it correctly" contract. This is just an opening for someone to exploit. Internally or externally to the project.
@@cppweekly Nor am I! But they can't be avoided altogether. There exist, for example, specialized algorithms with better complexity for ranges that are sorted. Also in the standard library you can find such algorithms that require the input to be sorted (in debug build an assert warns you if it isn't). Actually, a lot of code uses asserts on the input (even things like vector on access operator etc). How do you fuzz test such code (especially without the assert terminating the program :D)? Or is the answer just: use ordinary unit tests for those cases?
I use AFL++ to fuzz C++ code. I'm planning to develop in Rust and Haskell as well. Can programs in them be set up so that AFL++ can fuzz them?
I have no experience with this, but I know it is possible to fuzz code in any language.
Any good tutorials on fuzz testing for self-taught programmers?
The thing about fuzz testing is that you don't really need a tutorial, especially just to get started. You basically do exactly what he did in the video and the fuzz tester does the rest and gives you a report.
I don't think I have a project ready to be published that would require fuzz testing but I have no idea how to implement fuzz test. I understand the principle, heard about it since many years, I am already convinced but I never found good tutorial/resource to get fuzz testing right and nice (unit testing is also not fun/easy in C++ compared with other modern language, catch2 was actually bad when I used it compared to GTest). The boilerplate code doesn't look great to me, actually my main concern is code structure.
Still a very nice video, pushing me to be a better coder. I will take the time one day to get it right.
He just showed in the video. You define a function that takes inputs and passes them to your function and then run the fuzz tester. That's it. You just do that for every function that you want to fuzz test.
You have shown std apply, you used to have a much wilder example prior to apply showing being a thing where you would dispatch code with make tuple and eventually you showed how to do std apply instead
Compilers can't report this code can hidden assumption could be that `}` need to exists (like you check it before function call).
Recently I avoided checking if my algorithm exit working area by guarantying that data that it work with have encoded borders in it and processing end before I reach border. If I could not fully control this data then checking for borders would be required.
you are focusing on the wrong part. he picked this as an example only for when some logic happens based on user input, and the logic is just wrong.
so you can do your little "border" checks ("bounds-checking" is the word), that may help avoid this contrived example, but you need this fuzzing for when you DON'T think you did anything wrong, and then the fuzzer says that you did make a mistake! Your mentality of "but I did check everything already" is never good enough because humans are fallible.
If your function is assuming that the caller has made certain checks, then you can wrap the function with a new function that makes those checks explicitly before calling that function, and fuzz the wrapper function.
@@mrlithium69 I was referring more to why compiler will never diagnose code like this than usefulness of fuzze testing. If you have function with wide contract then you should fuzze test it thoroughly, but if it have narrow contract then any fuzze testing is pintless in most cases because by definition any value outside small permitted set will UB this function (and exposing this function to external inputs is felony crime :> )
@@Earthcomputer This would be problematic when one function is called during game load and another during update of map, this mean I can't create wrapper that will contains both functions because it will be effective `main` :)
Of corse you should not permit any one to alter this data in between this two calls, thus making data private/const will help greatly in case like this.
I may not fully understand fuzzing, but I find it hard to apply it in the projects I'm working on at present. Right now I'm writing a C function that takes a char* string. I could try to pass garbage to it, but I don't see the point because (1) my function expects a valid JSON string, (2) it's a private function and when I call it in other places of the code, I know for a fact that I pass a JSON-formatted string because the string comes from another library that returns a JSON string, (3) if it's not a JSON string, it's going to trip the call to a JSON parser (implemented in a third-party lib) at the beginning of the function and just return an error code. Overall, the only way to trip the rest of my function is to provide a valid JSON string (to pass the first couple of lines which are just parsing it) but with specific things that could trip it (such as having a JSON array instead of an object, having a key that I'm expecting not be present, or have the wrong type of value, etc.).
Overall if I wanted to fuzz-test this function I would need to write something that generates a JSON string that's invalid in some specific way, and this is exactly what I'm doing in my unit tests. How could fuzz-testing help in this case?
I don't believe that in your unit tests you test all possible valid JSON strings. A fuzz tester would help you by testing more cases, you just need one that only generate valid JSON strings so you won't waste your time testing your third party library instead of your code.
@@LEpigeon888 Ok I read more about fuzz testing and what's missing in this video (and all the videos I had seen before saying "you should fuzz test") is that (1) fuzz testing is not just about feeding random data, there is a program that tries its best to reach branches of the program (e.g. using genetic algorithms), (2) you can usually provide your own way of generating random inputs (e.g. making sure they're all JSON-formatted), and (3) you can also provide your own ways of doing mutations of input data.
That's very different from just the notion of feeding random data, which is usually presented when talking about fuzzing.
Using that nasty libpng code in modern C++ should be illegal too! 😂
Chill out with the clickbait
@Jason Turner, please note this. I came here to say just what this guy already has.
You don’t need to this, you have much better content already!
I totally agree
Please treat youtube video titles like code comments: make them expressive and not cryptic.
He needs a sanitizer for his video titles.
The truth is, click bait works
@@HylianEvil depends on the genre. For informative videos like this, it makes it impossible to go back and find. I cannot simply search "fuzz testing" and look for that in titles. Click bait might work for initial clicks, but it makes no sense for content that you want to treat like documentation or rewatch.
I know I should fuzz. I feel like it'd be easier if I was a linux native. Also my code doesn't compile on clang. So maybe I need to rewrite it.
Use Dr Memory's Fuzz Tester then drmemory.org/page_fuzzer.html
It's not just open-source software. "Security researchers" constantly fuzz every aspect of commercial products that they can think of.
Oh no, why is Jason click baiting us :<
Still good video though, 5/7 would recommend.
I would give 6 hours of bleed out of 1 heart. Definitely recommend.
Doctors (PhDs) hate that it's legal to avoid his one weird trick.
Hmmmm...well your click bait is on base but your avatar needs a big smile to let the viewer know they're about to watch an awesome video also high contrast thumbnails with no more than 3 words
Fuzzing?
Nah, will not help..... we are using clang-tidy and some other checks and they are DISABLED on big chunks of the code-base cause it is "legacy"-code and has too many warnings/errors. I would be surprised if you couldn't find 10 ways of crashing the entire system and 4 ways of getting complete control in the time it would take you to write and compile a single test. (There are still 5 year old bug-tickets open about unsanitised user-input being added directly into SQL-statments or bash )
oh no
I can feel your pain. After I introduced annotation in the source code for a false positive warning (for Coverity static analyzer), some of my colleagues on the other side of the globe started using the annotation for every warning they received.
@@hiftu Just today i saw some new code added that disabled conversion, overflow and out-of-bounds warnings cause supposedly they were all "false positive" .... just that it was doing calculations with int32_t and size_t and assumed they were equal -.-
Your CTO or software VP or whoever needs to be fired. You've been knowingly vulnerable to little Johnny ';Drop Table for 5 years and that isn't the number 1 priority of the entire software department? You must have constant headaches about subtler things if they can't even get priorities straight on SQL injection.
@@Dziaji Thankfully the software is not open or accessible to anybody and those userinterfaces are used by trained personal (.... as far as that training helps).
I have mostly given up on automated test suits cause we can't even do actual unit-tests, let alone fuzzy testing of individual functions, due to just how interdependent everything is.
it is a nightmare and even getting anything fixed can take months if you don't do it your self.
May be I've always worked in incorrect environments (badly managed teams, not so "smart" managers, etc), but I've never seen a project where the tests were included in the estimation of development cost (time). The only time there was automated tests, the tests themselves were never considered as part of the dev. And coding these tests takes time, doesn't it ?
In this case, they were "fuzzy tests" in a way, more something like very far away things, you know they exists, but you never see them ...
Don't worry, I never worked on open source software ! Nothing I/we made is public.
At least, I hope. It would bother me if some train begins to have random behaviors, or if some metal coils were cut in strange directions ...
Tel me, I'm not the only one who worked on projets that cut corners on testing ?
May be it really needs to be put in a criminal law ...
I saw cutting out testing a lot in industry. It just big additional cost, customers are not willing to pay, thinking that benefits are not worth it. Especially if your team manages to deliver thing with small number of bugs which are found and fixed easily. In that case for customer, it looks like you are doing good without automated testing, and it is not worth money. Also if there already exists huge project, which has been always without tests, adding tests to it will cost basically as much as whole project before, so it will be ridiculous to customer and nobody will pay for that.
A fuzztesting crash is not necessarily a bug. For performant code, and sometimes as a method of catching bugs, you don't necessarily want to sanitize all the inputs coming into a function. For example, if you have some robot running on micro-controllers, and you need to squeeze every cycle out of the processor, you might not range check your array accesses. The fuzz tester will die, but that doesn't mean there is a bug. There will only be a bug if something in the program actually causes an out of bounds access. So yes, the fuzz tester found a set of inputs that will crash the program, but those inputs might not be possible in the full context of the program.
Also, just because the fuzz tester found a set of inputs to crash your program doesn't mean you want to address it, even if you aren't trying to squeeze out extra performance. At my job, we intentionally leave divide by zero risks and things like that in the code if it SHOULD be impossible for the divisor to be zero. That way, if a bug is introduced that sets the divisor to 0, the program crashes, which is a good thing, because it lets us know that some thing is very wrong. If you sanitize all division to make sure you never divide by zero, you could cause the introduced bug not to be noticed. No one checks every log statement after every test of every code change, so it is possible that sanitizing your division leads to a bug in the code that is only noticeable by a log statement that says "Error: division by zero", and then the program continues on, effectively treating the divisor as 1 instead of 0, which may lead to less noticeable, but still very serious, bugs. You wouldn't want to use this sort of intentional crash debugging technique on like a cpap machine, or an oxygen system on a space station, because you want those things to continue to try to run no matter what is going wrong in the software, but for something like a trading bot or accounting software, you DEFINITELY do want the program to crash rather than continue purring along silently in an error state.
So if you use fuzz testing, don't necessarily fix every single crash that the fuzz tester finds unless you have a well thought out policy to do so that has a method for detecting crashes avoided by sanitation.
> That way, if a bug is introduced that sets the divisor to 0, the program crashes
No. There is nothing that guarantee you that the program will crash if there is a division by 0. The compiler can choose to generate assembly that somehow won't make a division, for example if it find that the divisor is always a power of 2 (excluding 0 because it's undefined behavior so the compile will just ignore that case), then it may do a bitshift instead of a division, and a bitshift of 0 won't crash your program.
The chances that the compiler generate assembly that won't crash for a division by 0 are super low I agree, but they aren't null, even with optimization disabled. You should never except a crash when undefined behavior happens.
> If you sanitize all division to make sure you never divide by zero, you could cause the introduced bug not to be noticed.
No, because if your policy is to stop the program in such cases, you should just call abort(). If you want your program to stop when you have a division by 0, you should explicitly write it, relying on undefined behavior to do what you expect is dangerous, because like its name imply, it's undefined.
And for fuzz-testing you'll probably want to replace these abort by a throw or another mechanism that will let you move on to the next iteration of the fuzz test without stopping your program.
@@LEpigeon888 You seem to be completely missing the point entirely. I'm not talking about an official policy with dealing with division by 0. I'm talking about tactically and intentionally leaving the program open to segfault or throw unhandled exceptions in places instead of gracefully handling all error cases. Division by 0 was just the first thing I thought of.
@@LEpigeon888 Also, undefined behavior is almost never undefined. It is undefined by the standard, but many instances of "undefined behavior" are deterministically defined if your compiler remains consistent.
@@Dziaji I perfectly understood, and my reply exactly address that. Segfault are always bugs as well and you NEVER want to let that kind of bug happens, it's a big security risk. You should really reconsider how you manage your edges cases.
Some compiler may define some undefined behavior, that's true, but unless you've seen it explicitly written in their documentation then it's not the case. And if you rely on this behavior your code isn't standard C++ anymore and can only be used with one compiler.
And dereferencing null or out of bound access are NEVER defined behavior in any compilers, segfault are NEVER guaranteed in such cases.
@@LEpigeon888 most programs only use a single compiler.
Yeah, segfault is not guaranteed in those cases just like you might guess someones 2048 bit private key on the first try.
You clearly DON'T understand the point because again you reiterated that "segfaults are always bugs and you never want to let them happen." Which is exactly the point I am making. When your program segfaults during manual testing or unit testing, you know you have a bug that needs to be fixed.
You may not be being hyperbolic, but I'm certain that _some_ conic section is involved here.
I don't agree with calling it clickbait. While I don't believe it should be illegal as a blanket statement, certainly for high value targets to not perform fuzz testing on the software they use should be. Consider how many hospitals and small governmental entities such as city and county governments have had their security compromised and thusly compromised the security and sometimes safety of people who can't merely delete their data from someone else's database. It seems to happen more and more these days to the point where the utter negligence should be punished in some way. Although, obviously governments are going to exempt themselves, so it'll never happen that the most culpable will actually be punished.
It is clickbait because it has no relevance to the content of the video, not because it uses hyperbole with the word "illegal". You have to click the video to know *what* should be illegal. It should be something like "C++ Weekly - Ep 352 - Fuzzing: Not Doing This Should Be Illegal!", and then he wouldn't be inflating his views at the expense of people's time that are trying to find information on fuzz testing.
I use your channel as a useful reference. The clickbait titles are really not appreciated.
c++ shud be illegal
std::uint32_t rot32(std::uint32_t x, std::uint8_t r){return (x > (32 - (r & 0x1f)))}