Property-Based Testing In Python: Hypothesis is AWESOME
VloĆŸit
- Äas pĆidĂĄn 8. 07. 2024
- Hypothesis is an awesome Python package for property-based testing. In this video I talk about what property-based testing is, and show you a practical example of how to use it to make writing software tests in Python a breeze.
The code I worked on in this episode is available here: github.com/ArjanCodes/2022-hy....
đĄ Get my FREE 7-step guide to help you consistently design great software: arjancodes.com/designguide.
đ» ArjanCodes Blog: www.arjancodes.com/blog
đ Courses:
The Software Designer Mindset: www.arjancodes.com/mindset
The Software Designer Mindset Team Packages: www.arjancodes.com/sas
The Software Architect Mindset: Pre-register now! www.arjancodes.com/architect
Next Level Python: Become a Python Expert: www.arjancodes.com/next-level...
The 30-Day Design Challenge: www.arjancodes.com/30ddc
đ GEAR & RECOMMENDED BOOKS: kit.co/arjancodes.
đ If you enjoyed this content, give this video a like. If you want to watch more of my upcoming videos, consider subscribing to my channel!
đŹ Discord: discord.arjan.codes
đŠTwitter: / arjancodes
đLinkedIn: / arjancodes
đ”Facebook: / arjancodes
đ Code reviewers:
- Yoriz
- Ryan Laursen
- James Dooley
- Dale Hagglund
đ„ Video edited by Mark Bacskai: / bacskaimark
đ Chapters:
0:00 Intro
1:11 About unit tests
1:44 What is property-based testing
3:09 A basic example of property-based test
5:28 Controlling what example data is generated
6:22 Adding a second property based test
7:29 A more complex example
10:15 Adding a test for negative team sizes
11:43 Adding more property-based tests
13:46 Creating a custom strategy
16:07 Using the custom strategy
16:54 A few more examples of using strategies
18:40 Generating testing code automatically
19:44 Final thoughts
#arjancodes #softwaredesign #python
DISCLAIMER - The links in this description might be affiliate links. If you purchase a product or service through one of those links, I may receive a small commission. There is no additional charge to you. Thanks for supporting my channel so I can continue to provide you with free content each week!
đĄ Get my FREE 7-step guide to help you consistently design great software: arjancodes.com/designguide.
Excellent vid - much clearer than the documentation. One feature worth emphasizing is the automatic testing code generation. Thatâs the big gun for creating a ton of test coverage.
The amount of bugs hypothesis has uncovered on projects I've worked with makes learning it worthwhile
What would be really cool would be a comparison video, first write a small project (idk, sudoku, battleship, whatever) and then write appropriate unit tests for it, and then do it again but using test-driven-development. To me it was hugely interesting how it influenced the way I design my code and how sensible the coupling between tests and code almost automatically becomes.
I told you property-based testing is awesome :). Great video, Arjan! One approach I like a lot is to use a test oracle. When I have an optimized and therefore complicated implementation of an algorithm, I often also implement a simple version of the algorithm which is easy to understand, but not optimized and not efficient. I use Hypothesis to generate random input, run both implementations, and then assert that they return the same result.
Nice one!
Another awesome feature of property-based testing and also Hypothesis is that the failure-inducing input is automatically simplified to give you a small and simple counter example. This is beneficial for debugging.
Excellent video! I am using pytest quite often but didn't know about hypothesis. This would take my tests to the next level! Keep these videos coming, Arjan
You bring up an interesting capability and much misunderstood feature of Python; Decorators.
Perhaps this could be a topic for the future?
Thanks for sharing!
I had never heard of property testing and found the subject very very interesting. I would really appreciate more videos on it!
Thank you so much for this video. I also had an excellent experience with hypothesis. You covered almost every case, probably except using map and flatmap to generate complex strategies. We found lots of very strange bugs thanks to this framework. Also property based testing is kinda mind blowing at first, but when it becomes a second nature, you even write regular tests much better :)
Thank you so much for all of your videos, i enjoy every second of it. Best coding channel on youtube â€
Nice, Arjan. I like your clear way of explaining sometimes complicated concepts. You taught me Python development the right way!
Love the videos Arjan keep them coming.. Can't wait every time!
On it đ
I love hypothesis! It's an awesome package and I think it should be used much more. However I recognise that it can be tricky to know when to use it if you are used to regular ol' unit tests.
Your videos are simply awesome. Please provide us with more valuable information specially about hypothesis in depth (as u mentioned).
Keep up the good work! âđ»
Very good video! I saw hypothesis package one time in our company, but I didn't have time to dive into it. After that vid I'll back there for sure :D
Didn't know about Hypothesis but I was aware of property-based testing (i.e. verifying a property of the system more than asserting a specific response for a given input). In my experience I was using a combination of unitest and Faker: it seems this approach would be more structured and synthetic. Definitely worth to try in one of my next projects.
Great video, congrats and keep posting!
Thanks so much Javier, glad you liked it!
The back lighting in your videos are đ
Thanks!
Excellent!
Thank you!
This is way better than what I could gather from documentation, kudos!
Thanks so much Vladimir, glad it was helpful!
Awesome! Thanks!
Happy you liked it!
Great video! Iâd love it if you could do a vid on unit testing, integration testing and data quality checking for a data engineering shop. I know itâs not exactly your wheel house, but curious what youâd come up with.
Thank you for this! I was already kinda doing this, but hypothesis will make the tests clearer and easier to make.
Youâre welcome! âșïž
Iâm sold on the idea, but translating it to a real project and not the toy examples is difficult. I think I need a real OSS code base that does it well to demonstrate non-trivial uses in like half a dozen use cases. As a data engineer, I can see the value of treating the inbound data as a random variable in tests but the cost to learn hypothesis always seems a little too steep to implement in a real world scenario.
I'm on the fence about this, too. I very much like a data-driven approach to testing, as it allows you to collect real-world test cases and add failure cases over time without having to touch the actual test code. The library could be very useful for fuzzing, though, but I don't know whether *all* output of the given generator (e.g. "integers()") is actually tested.
Would be quite irritating to have the test randomly fail if only a subset is drawn each time. OTOH the run time could be prohibitively long if the number of test cases becomes overwhelmingly large (think matrixes or lists).
I guess the utility of the library depends heavily on the type of software you're working with.
@@totalermist yeah I looked into ScalaCheck which is the equivalent in that ecosystem which is based off the QuickCheck in Haskell. Apparently the strategies run a limited sample of values but hold the seed. So if there is a failure it spits out the failing value and the RNG seed to the failure is repeatable.
I think I heard in the ScalaCheck equivalent, that when it finds a failing case it tries to reduce it in some way to the simplest failing case. Eg if -1234 fails itâll try -1. But I also heard the strategies can be biased / weighted to try common failure cases. Like strings() will try an empty string more often than even odds.
But yeah⊠a walk through of a real OSS code base with the author of said code base is the master class I am after.
Excellent video would love to hear more about hypothesis!
On it, Hector ;).
Arjan..
I absolutely â€ïž your videos. Youâve rekindled my passion for Software development. Iâll definitely use it for QA activities.đ
BTW can you create a detailed video on code documentation đand cover tools like Pdoc3 Sphinx etc. that would be phenomenal
Thanks! Very nice. Feels like hypothesis with the faker package could be an elegant way to create, test & stub data.
Youâre welcome! Indeed, thatâs an interesting combination to explore
Great video, as usual, thanks!
This topic was new to me and I would be very interested in a second video on this subjet!
Maybe as a suggestion: apply this to a more "real life" example? Given a small simple project, what parts would you want to test with normal unit tests, what parts would you want to cover with property-based tests, where to do both? What is really the added value of those extra property-based tests over the classic unit tests (including parametrized tests)? Maybe there are typical things that are very commonly tested with property-based tests? Also, I guess looking at the code coverage is not really the point... is it?
It's just some thoughts :) I really liked the video, I see that this tool is extremely powerful, but I'm not sure how to apply this to my projects...
Thank you for this informative video. Could you make a video to showcase property-based testing focused in Django API testing?
every time i read about something cool i can count on you to make a video about it within the next few months instead of trying it myself xD
this will be one i have to add into the mix
At your service đ
I've seen this hypothesis in some project before. This is the first video approaching this package, btw
would love to see how this is used with dictionaries/pandas data!
The thing that blew my mind about QuickCheck is that when it finds a failing example it shrinks it down to the smallest possible failing example. There are methods to do this with custom strategies but I had a lot of trouble wrapping my brain around them. Can Hypothesis do this? If so, can you do a follow-up about how to shrink your own custom examples?
Some pedantic comments ^^: test_negative_team_size should be called test_non_positive_team_size since 0 is typically considered neither positive nor negative. It is definitely not negative. I also propose to have a simple unit test for 0 since this is typically an interesting special case. If you just specify max_value to be 0 it is not guaranteed that 0 will be part of the generated random inputs.
Thanks for the feedback, Robert. Indeed, the naming you suggest is more precise. One way to add the 0 example is by using the @example decorator in Hypothesis and this will make sure Hypothesis includes it in the test data.
@@ArjanCodes Hi Arjan! The example decorator is even better than my proposal. In this way one does not need an additional unit test. Awesome! Best wishes from Austria :)
Haven't watched it yet but, yes hypothesis is amazing
Hi, I Always follow your videos and find them really useful. I would really like a video about event driven architecture showing both event and command handling, I wasn't able to find any good one of that around the youtube ether..
Quite interesting
Thanks Kriss, happy youâre enjoying the content!
whats your favorite for software development ,windows ,linux or mac?
Thanks a lot! Please cover `mutmut` next.
Youâre welcome. And thatâs also on the list. đ
Arjan, did you have a video explaining how to log? I have a problem with logging my program with multiprocessing and filerotating handlers. đą
Hi! Looking all your videos while I am in the train đ really great channel!
I have a question: do you thing itâs a good practice to use something like âtest()â to generate values?
I do not like this approach, since you donât have the control of the inputs your are using to test your function.
While writing I was thinking that maybe this type of testing can be part of the fuzzing testing that maybe can be separated from the unit one
How does hypothesis generate values.
Since they can't try every possible values (too slow), does the library use a specific strategy to find corner cases ?
ok. it does not verify that the code matches specification, but it tracks bugs by matching invariants of function and reciprocical functions?
I don't get the Python highlighting in VSCode... how the settings decorator is green but all the others are yellow.
Still not entirely sold on this.. the basic tests (encoding) pass also if I replace the functions with "return inp"... and even adding a type checking property test won't cut it either.. sure you don't care about what data to feed the function with, but you need to make sure that for a given input the function return the expected! Or am I missing something here?
Let's imagine that we have a function than gets 2 arguments: string and integer and returns the boolean.
For example, it returns True if the second argument (int) == len(first argument)
Actually, I'd provide parametrized tests for the most common cases with the next structure:
@parametrize("data, expected_result",
[
(("Arjan", 5), True),
(("Hello", 3), False)
]
)
Could I do something similar with @composite for example?
Or any usage with parametrized tests / fixtures / lazy_fixtures / ...
Would be a great video, IMHO
hypothesis seems like a nice tool, will definitely try it in my next project.
One thing I'm slightly confused about is that you suggest property-based testing as better than unit testing. I would call the kind of tests you wrote unit tests.
Is unit testing really just tests of the form input -> expected output? Hard coding input/ouput examples sounds like a terrible way to write tests. Do people actually do that?
> unit testing really just tests of the form input -> expected output?
I'm here to say, yes, that is what some, most, people think is unit testing. And even with property-based testing, you still have to write a good number of this variety of test. How else does the machine know what is "supposed" to happen, without re-writing all the code you are testing in the test itself?
i gave u a like bello đ
What do you think about using hypothesis for testing an API? Would generating payload data using hypothesis help discover any bugs?
Why not?
There is schemathesis for APIs
Sure. For example, if you have an API endpoint for searching documents with a skip and limit parameter, you could use Hypothesis to run tests with various values of skip and limit. Or test endpoints that expect dates with various date values, etc.
Does anyone know if hypothesis package is smart enough to avoid redundant tests OR does it always generate data randomly ? A redundant test has a implies relationship. For e.g: If a number is divisible by 4, then its obviously divisible by 2. Hence, i say the latter test is redundant.
Could somebody tell me how should test for an example a function to calculate sum of squares would look like:
I got a function:
def my_func(a, b):
return a**2 + b**2
so now for tests
@given(integers())
def test_my_func(a, b):
assert a**2 + b**2 == my_func(a, b)
so my question is what's the point of a test like it or how to test math inside python? Because for me testing math looks like copying that func to test and checking if they are equal.
Ofc I am not asking normal unit test:
@pytest.fixture(a, b, result, [(1, 2, 5), (2, 3, 13)])
def test_my_func(a, b, result):
assert my_func(a, b) == result
so If I think correctly hypothesis for testing math has no value right?
I agree that testing in this way doesnât really help because a bug in the math is not found since itâs also in the testing code.
What you could do instead is test for properties. For example, the sum of two squares is always a positive number or zero. So you could write a test that verifies that this property holds for a bunch of different aâs and bâs. Iâm not a mathematician, but there are probably other properties of the sum of two squares that you could write tests for as well.
@@ArjanCodes That totally makes sense :)
I don't get the point behind testing plain arithmetic functions, you need to trust the software at least for this basic task đ
@Aflous, I think this was intended as a simple example to illustrate testing mathematics functions. In a real software application, you probably wouldnât have a function like this.
@@ArjanCodes As a mathematician, another one is using the fact that (a - b)^2 >= 0, so a^2 - 2ab + b^2 >= 0, so a^2 + b^2 >= 2ab. But also the opposite, (a + b)^2 >= 0 so a^2 + b^2 >= -2ab, which helps if one of a or b is negative.
Another idea is testing the sum of two integers is also an integer.
Of course we could think about extending to floats, complex numbers, lists of a,b pairs, etc. and you would need to check for different properties.
Video is cool and enriching. But I have an error when trying to run the hypothesis through the generation. "ValueError: source code string cannot contain null bytes".
I found the problem. When use the terminal to generate the hypothesis, it will create a file with format UTF 16. Need to change to UTF 8 to work
You all know how to make the hypothesis generate the test file in UTF 8 instead of UTF 16 through pycharm. No matter how I change the file encoding in pycharm, it will always create the test file in UTF 16.
Is there an equivalent for JS?
There is fast-check: czcams.com/video/a2J_FSkxWKo/video.html
Also jsverify: dev.to/glebec/property-testing-with-jsverify-part-i-3one
Both don't use decorators, though. Might be a fun project to port the hypothesis library to JS or TS while keeping the decorator approach.
Tangent: You've nailed the lighting, but the background sound is a little too loud and distracting, at least to my ears.
Shouldn't you write tests before you code? Or is that only TDD?
That's only TDD in its strictest form. In practice, though, I've never seen it applied consistently.
Property-based testing is more of an in-between of unit tests (e.g. what TDD is based on), BDD, and integration testing.
If you only test single code units or functions, it's more towards the UT side, if you use it to test against specs (given-when-then), it's more BDD, etc.
I put my email in the code guide site box several times. Never came. Not in spam.
If you send me an email (support@arjancodes.com), Iâll make sure you get the guide.
So this is pythons fuzzer.
It is _a_ Python fuzzer, yes.
Elon Musk explaining his Code after acquiring Twitter: 8:20
7:41 don't do that again pls
I saw a great way to make hypothesis tests cleaner. When importing you do:
from hypothesis import strategies as some
And then
@given(some.text())
This is an awesome idea! I can't believe I haven't seen this before
Stealing this, thank you!