How to Use Hypothesis for Model-based Testing (Step by Step)
Vložit
- čas přidán 23. 07. 2024
- Do you want to make software testing easier? Model-based, or stateful, testing can be a great way to save time and extra work. In this video, I'll show you how to use Hypothesis for model-based testing so that you can streamline your testing process and get better results.
GitHub link: github.com/ArjanCodes/2023-mb...
👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
💻 ArjanCodes Blog: www.arjancodes.com/blog
🎓 Courses:
The Software Designer Mindset: www.arjancodes.com/mindset
The Software Designer Mindset Team Packages: www.arjancodes.com/sas
The Software Architect Mindset: Pre-register now! www.arjancodes.com/architect
Next Level Python: Become a Python Expert: www.arjancodes.com/next-level...
The 30-Day Design Challenge: www.arjancodes.com/30ddc
🛒 GEAR & RECOMMENDED BOOKS: kit.co/arjancodes.
👍 If you enjoyed this content, give this video a like. If you want to watch more of my upcoming videos, consider subscribing to my channel!
💬 Discord: discord.arjan.codes
🐦Twitter: / arjancodes
🌍LinkedIn: / arjancodes
🕵Facebook: / arjancodes
📱Instagram: / arjancodes
👀 Code reviewers:
- Yoriz
- Ryan Laursen
- James Dooley
- Dale Hagglund
🎥 Video edited by Mark Bacskai: / bacskaimark
🔖 Chapters:
0:00 Intro
1:23 Example explanation
3:39 What is model-based testing?
4:02 Writing a unit test
4:40 Property-based testing
6:00 Model-based testing with Hypothesis
7:38 Writing stateful tests for the Order example
20:52 Outro
#arjancodes #softwaredesign #python
DISCLAIMER - The links in this description might be affiliate links. If you purchase a product or service through one of those links, I may receive a small commission. There is no additional charge to you. Thanks for supporting my channel so I can continue to provide you with free content each week!
👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
The difficulty I have with this kind of testing, is that the testing quickly gets more complicated than the code. This video is a actually an example of that. Furthermore, to change the structure of the original code, you also have to fix all the tests, which is not always easy and it's double work. It actually discourages cleaning your code structure and solving technical debt.
I like the thought of automated testing, but due to the overhead it creates I find myself avoiding it.
I think it's very reasonable to avoid tests because it feels like more work, but I don't think it's fair to say that it's double the work. Taking the time to write tests in the beginning will save future time spent testing/debugging the project. In my opinion, not writing tests will in most cases easily add up to double the amount of time or more spent debugging/testing. So it may be more work upfront, but it's less work overall.
I've found in the past (as a hobby dev), that I'm more likely to do large refactoring, because I know that tests will have my back.
I also find TDD pretty fun, like achievements in a game.
When I have to change my tests, that means I'm breaking user code, so it ought to be a pain in the neck.
It helps a lot to test the behavior and not the implementation. This allows more opportunity to refactor without impacting too much on the tests.
totally agree. It is an interesting topic, and maybe fun for pet projects and so, but not for real business stuff.
Hi Arjan,
I like watching your videos and learned a lot of things from you. I watched 90 percent of your videos probably. I can give you some video topic suggestions that you haven't covered yet.
1 - Pre-commit: How to use it efficiently to improve our git commits. Affiliated with linters(Pylint etc.), formatters(Black etc.), import sorters(isort), documentation helpers(pdoc) etc. A separate video for each acategory(linter, formatter, import sorter, documentation helpers) etc.
2 - Cookiecutter: A python library to reuse the same codebase for further projects
3 - Pytest Fixtures: How to use Pytest's fixtures efficiently
4 - DVC: A data versioning tool. Similar to git, but versions data instead of code.
5 - MLFlow: A python library to track Machine Learning experiments
Also, I want you to share you cryptocurrency addresses with your followers. I want to donate some amount of money directly to you.
Thank you for your great videos.
Yes I'd like to second these! @ArjanCodes
I still miss the background signboard with the amusing quotes!
This man never disapoints 💯🙌Thank you
This is what I looking for, great inspiration 👍
Glad it was helpful!
Excellent content, once again! Thanks a lot.
Around 19:30, you fix the bug by introducing the method `_update_total_cache` but I found that simply adding a copy of a line item (copy, not even deepcopy) in `add_line_item` works as well and is a bit clearer (one line change).
Just sharing back!
Pleasure :)
Thanks so much!
Thank you too!
Super useful video, I will see how I can try to integration such testing in my project
Hypothesis seems to be really powerful!!
Glad you liked it, Loïc!
Thank you for the video. Clear, informative and interesting.
What bothers me with model based testing is that it seems to me that the complexity of the tests is equal to the complexity of the code itself, and if I can't trust my code without testing, I obviously cannot trust the test.
In good old tests I think of specific cases that I want to test, so I can, for example, take those cases from my requirements document etc.. But here I need to write the full logic of the program in my tests, right? You wrote the same logic of calculating the total twice (sum all line totals).
Maybe it's just because of the over simplified example?
The trick with property-based testing is how to write test logic without re-implementing the main logic. Thinking in properties of your code is a mental shift compared to example-based tests. Not that easy to adopt :(
But the mental shift to property-based thinking reflects your domain knwoledge
Remember that the "property" in property-based testing refers to MATHEMATICAL properties like commutative, associative, distributive, and identity properties of objects!
It does not refer to the properties of an object.
Thank you. The one answer I couldn't find anywhere else
My first thought was that or seems so complicated BUT I think that this might actually be great for integration testing and not as a a replacement for regular “testing this one function” kind of tests.
And now you need to cover this test-producing monstrosity itself with tests :)
Yup, or test that you actually test the edge cases. -and that's where the library Mutmut shines. Quite fun to try on a codebase.
As I watched you developing tests, I kept thinking, "But you didn't write a test for this case... or this case..." That's the problem I have with such things, the more you think about it, the more unique test cases you need. What if quantity is negative? What if you add another item with same description but different price? If similar line-items are allowed, which one should be removed?
One project, we had the testing group really put our application through a lot of scenarios. They seemed to ALWAYS find a use case we hadn't anticipated. We tried to write a test case in our testing suite for each failure the test group found, so as we progressed and refactored and developed more and more features (used a modified Agile approach), the test suite grew and grew. In the end we were somewhat confident that we didn't break things with each release, but the test code was monstrous.
Hypothesis does an excellent job of finding corner cases I would never have thought of.
what do you think of modeling languages like alloy?
I don't know if it would be worth the time to learn tools like that
One question, from the explanation I understood that actions were described by "rules" and assertions as "invariants". In 12:00 and 20:25 the assertions were created using the "rule" decorator, shouldn't it be done with the "invariant" decorator instead? Great video, thanks so much. It looks a bit too bulky at first glance, but I get the benefits you can obtain by using this kind of testing.
I would use the invariant decorator for the mentioned sanity checks as well. If you read the documentation, it mentions the key difference between rules and invariants is that rules may be executed zero or multiple times inbetween other rules and invariants are guaranteed to run after each rule.
you have a finite state machine in this class.
Perhaps you can determine tests by looking at the state transition diagram for this class?
After extensive experience with pattern oriented style using C# and Java, DL projects on Python look for me like a ball of muddy stuff. You give the hope that good code can also be written on Python. Thank you very much!!!
Very useful article thanks. What VSCode theme do you use?
Thank you! I'm using the default dark theme of VS Code, nothing fancy :).
I really like the concept of property-based testing, but am curious how Hypothesis ensures that the tests are run to give reproducible outcomes? As others have mentioned, it's not clear from the output how many scenarios are run or whether they are exhaustive.
Hypothesis runs tests differently, randomly, each time. When it finds a failure, it records it in a database and ensures that particular example always gets run, in addition to a variety of random new examples.
How many combinations or tests is this code executing? The execution time is high, pytest only show that it is executing two tests.
The documentation says it runs 200 by default. It is configurable.
16:04 But the test fails on total_agrees(), not on remove_line_item_from_order_raises_error()
This feels more complicated than writing the test cases yourself. Still yet to grasp it esp the usefulness. Pytest has parameterization which seems similar to what am seeing here. I will rewatch the video, maybe I am missing something
You can write property-based tests using parameterization, but if there is a large space of possible situations, parameterization requires you to explore the entire space every time; hypothesis will explore the space in a way designed to find the simplest failing example.
You had me at property-based testing over example-based unit tests, but my impression from this video is that model-based goes too far and is too much overhead and too complex code for tests.
Yeah this seems pretty silly, as others have said this ends up in creating so much complexity in the test and uncertainty around what values are being generated that it's hard to have faith the test is behaving properly, next thing you know you need a test for the test.
First comment. For sale
As with almost all the "advanced" testing strategies I've seen, writing the test is almost as complicated as writing the software in the first place. To reduce the work you need to do you have to jump through fascinating hoops ("data: st.SearchStrategy" etc.). You almost get to the point where the test suite needs a test suite.
Just write the tests yourself has been my strategy. You know what you're testing, there's no magic hidden stuff, and your limited brain horsepower is devoted to the product as a whole as opposed to "clever" testing. YMMV.
EDIT: just saw @YCLP 's comment from earlier.
Next day EDIT: will read about hypothesis anyway