TDD vs BDD. The giants of test go head-to-head (2)
Vložit
- čas přidán 9. 07. 2024
- This time, we'll be focusing on Code Coverage, Maintainability and Portability. And crowning a winner. Will it be TDD or BDD?
= = = = = = = = = = = =
New for 2024: my best-ever training:
"How Your Agile Teams Can Achieve Predictability and Productivity WITHOUT Burnout"
→ www.developmentthatpays.com/w...
= = = = = = = = = = = =
Grab your FREE TDD vs BDD Cheat Sheet: www.developmentthatpays.com/c...
It's time to bring the TDD vs BDD battle to a bruising conclusion. Which will emerge victorious?
Will Test Driven Development be pinned to ropes? Or will Behavior Driven Development be out for the count?
ROUNDS:
- 2:18 - Round 5: Code Coverage
- 2:54 - Round 6: Maintainability
- 4:07 - Round 7: Portability
PREVIOUS EPISODE:
- • TDD vs BDD. The giants...
Music: 260809 Funky Nurykabe: ccmixter.org/files/jlbrock44/29186
-------------------
63. TDD vs BDD. The giants of test go head-to-head (2)
#DevelopmentThatPays
Previously... TDD and BDD went head to head. Round 1 [Focus] was a narrow win for BDD. Round 2 [Writer/Reader] was a decisive win for BDD. But then the tide turned: Round 3 [Speed] went to TDD. And Round 4 [Specificity] went the same way. That's two rounds a piece. But this bout is far from over. Go live ------ Welcome to Development That Pays. I'm Gary Straughan. And welcome back to second and final part of our TDD vs BDD battle. Before we get back to the fight, there's the small matter of the question I left you with last time. Here's a quick reminder: "You're responsible for a major system. It has excellent test coverage, with a full complement of Unit Tests and Behavioural Tests. A major deadline is looming and you're gearing up for the release. Vexingly, there's one failing test. And the Development Team say they won't be able to fix it in time for the planned release Question 1: Do you go ahead and release to live Question 2: Does your answer to question 1 depend on the TYPE of test that is failing." So what do you think. Would you release Alistair, you were clear... "If there is a failing test then the system is not production ready. I would argue that it (almost) never depends." Although you did go on to say that might be mitigating circumstances, such as the value of the new features in the release outweighing negative impact of a bug. Andrew, you were more bullish. And I liked that you expected the Product Owner to be part of the Go / No Go decision My answer Well, I'm not a brave man, so I probably wouldn't... But let's dig just a little bit deeper. If all of the Unit Tests are passing, and one Behavioural Tests is failing This is bad news. A failing Behavioural Test means that the "contract" has been broken. That's a clear DON'T RELEASE situation in my book. The other case - all the Behavioural Tests passing and one Unit Test failing - is more nuanced. If all the behavioural test are passing, then the contract is intact. Everything that the system promises to do, it does. Whatever is failing is having no impact* on the external functionality. One possibility is that application code covered by the failing test doesn't do anything. So the question becomes: "Why NOT go live " Did you spot the asterisks These arguments only hold water ONLY if the level of code coverage is very high. Which brings us right back to our bout. Round 5: Code Coverage. ------- A high level of code coverage is a good thing. The higher the level, the better you sleep at night. But 100% code coverage is rare and difficult - especially where unit tests are concerned. Levels of 70% and 80% would be considered by many to be good. For BDD, the picture is rosier - the code coverage percentages tend to be higher. For a microservice with a well-defined API, there's no reason NOT to have 100% code coverage. Round 6: Maintainability -------- The only constant is change. So a good question to ask is what will cause the tests to change. Any change to the functionality of the code is going to require the unit tests to change. And by the way, if the functionality Is to change, then the tests change first. A change is still development, and the development is still test driven! What about on the BDD side Not all functional changes impact the external behaviour. Going back to the cash machine example for a moment: I could exchange the card transport mechanism for another model. I could exchange the screen for one with better resolution. I could upgrade the keyboard None of these substitutions would require a change to this Behavioural Test. For the sake of fairness, I should say that the the inverse situation is possible: If new and di
• TDD vs BDD. The giants...
• BDD vs TDD (explained) - Jak na to + styl
I don't know why you are not having a billion followers. Thank you
Thank you!
Loved the conclusion !
Love cake!
Simply THANK YOU for all your videos! What a great production, method and synthesis in a short time! They clearly outstand the majory of other tech teaching videos on YT. Thank you once again! :D
What a lovely comment! Thank you - much appreciated!
Coverage: 100% is probably not desirable, even if it would be feasible. Also, I feel it is probably very much easier to achieve higher coverage with unit tests than with functional tests. That is because functional tests cover the big picture of execution through the code in a limited number of predetermined paths. Unit tests cover the fine-grained details of each unit; to achieve something similar with functional testing would require sets of functional conditions with exponentially more permutations in various details of each.
Thank you so much. Your videos are such an inspiration. Please do sprint simulation videos. How to use Jira in a typical work day...
Just loved the way the concept made clearer... Thanks!!!
You are most welcome! Thanks for your comment!
Very helpful and good video! You made the concept very clear! Thanks
you're super awesome, I really loved the way you explain concepts
Well presented Gary. Interesting outcome and summary.
Amazing series with outstanding quality! Thank you for producing these videos!
Really glad you like them!
Why don't you upload new videos? They are great and helped me a lot with starting a wholly different position in the IT sector from Sales.
I'm working on two new videos right now. 📽
Great videos!!!!!! Really useful and entertaining
Feel like BDD is a must to finish a project. Also to start. Any other way is bad management and amateur's company.
Yes the company manger I work for just like to 'put bandages' on bleeding wounds, that the entire method.
So...what does a behavioural test "look" like? I know precisely what a unit test is, how to write it, and how to execute it. I'm less clear on a behavioural test. Is it necessarily a whole-system test, like starting the bike? Is it automated, as with unit tests? Or, when I wrote a testing script (script as in an actor's lines) for myself to 1) dial into my telephony system 2) update the sensor-listening time to between X and Y which includes the current 5 minute window 3) trigger the sensor pathway by opening the door, and 4) verify that an alarm went off; was I actually writing a kind of behavioural test? I suspect that what you are referring to as a behavioural test is in fact automated, but am having a hard time wrapping my head around just how to test enough code to comprise a behaviour without testing the whole system on the one hand, and breaking down into a functional test on the other. Does that make sense?
Those are fantastic questions! I think another video may be in order :)
The test you describe sounds to me like a Behavioural Test. Think it also qualifies as a System Test. And a Functional Test.
And yes, behavioural tests can be automated. The ones I'm most familiar with run something like this. For each test:
- Wipe the database. (Yes, really!)
- Seed the database (with just the data required for the test). This is putting the system into a specific "state"
- Run the test
A very simple test might look like this:
- Given I have a blog post with an ID of 77
- When I visit page post/77
- Then I should see a blog post
Behind the scenes, the first line causes the database to be wiped, and a single record (id = 77) to be added to the "posts" table.
Does that make sense?
Yes, that makes sense. I have written automated tests that work just like that: establish a clean state in the database, trigger a specific pathway, and check that the database contents reflect what just happened. Hmm the lines seem to blur a bit between functional/behavioural, and as you say, possibly system as well. Words, words, words. I guess the important thing is to decide which level of granularity makes the most sense (i.e., has the most business value) for a given project and test at that level, regardless of what you call it.
With that said, can you give a example for TDD? Possibly the same example.
Excellent question. Here's a blog post-related example:
It's common for the last part of the url of a blog post to be auto-generated from the title. I might decide that I want a function called slugify() . Here's how it might play out:
- Write a test called testCanInstantiateSlugify()
- Verify that the test fails. (IMPORTANT!)
- Write just enough of slugify() to get the test to pass
- Write a test that expects an input of "Hello Mum" gives an output of "hello-mum"
- Verify that the test fails. (IMPORTANT!)
- Change slugify() to get the test to pass
(And carry on in a similar vein to test special characters, numbers, etc.)
Does that make sense?
An amazing explanation,but. End of the day, both of methodologies are good for specific scenarios and purposes. It is not math, it is more of approach to cover specific functionality. If Developer or QA Engineer deliver something to production and both of methodologies passed ,but not catched a bug, who responsible in this case for missing a bug/bugs? Methodology, approaches, vision? Especially today, when everything in agile prospective, both of methodologies give you a added value to reach a goal.
Can't argue with that: I mix and match both TDD and BDD.
Nice conclusion
Great content, presented very well. A rear duo, well done.
Thank you - much appreciated.
Do you use gerkin/cucumber for your BDD test?
Another great couple of videos, many thanks :)
Glad you're enjoying the videos.
Haven't used Cucumber in years (since I left the BBC). In recent times, I've mainly used Gherkin via Behat (PHP).
What do you use?
Have seen JBehave and Cucumber for Java used. Not sure what the difference is there though!
The difficulty we found was getting business buy in to it. Any advice? Cheers
Excellent Video!!! Thank you
You are most welcome! 👍
Fantastic series.
Thank you!
Can you summarise how BDD is implemented? What tools? I have used those testing tools that test the behaviour of the GUI interface at run-time. There is no way they are portable if the application is rewritten. They change far more than unit tests. Units tests only change when the class/method/function interface changes. It could be argued that if the interface changes then their should be a another version of the method with new unit tests. As a developer I would rather work on an application that has unit tests by a country mile.
You're a great teacher, thanks.
Thank you. You just made my day!
@@Developmentthatpays I'm so glad 🌸, keep going 👏
Can you go live with a failing behavior test? Yes, sometimes. I rode my motorbike to the store today, even though the clock on the handlebar failed its behavior test. The new ATM machine gave me the right amount of cash but said "Thank yoo" in the end. Maybe the bank decided that this was less embarrassing than an "down for maintenance" sign. There are no absolutes; just tradeoffs.
VERY smart answer 👍👍👍
@@Developmentthatpays Glad you enjoyed it. There's more. Are you OK, if all behavior tests pass, but a unit test fails? Not always. The ATM machine could pass the behavior because it displays your balance correctly as $80.00, but the unit test fails because the balance is stored as 80.0001 for all motorcycle owners.
There are times when we might get into trouble with the "usability is an implementation detail" mindset. At one extreme one could argue that the user just needs to withdraw $20, and the entire ATM machine is an implementation detail. That may be a useful abstraction, but a bad way to build and test ATM machines.
We may be in an environment where developers code based on the behavior specification. Now we are asking the coders to make UI/UX decisions. They might implement slot machine handles instead of dropdowns.
We may be in an environment where the implementation-free behavior specification is a starting point for the product and design teams. There might be many feedback loops between the requirements and the design. In this world, the behavior specification that finally goes to developers is rich with implementation details of screens and buttons and style sheets etc. Once again, there are no absolutes, just tradeoffs.
Gary, just curious what was your objective in this series suggesting a battle that one was 'better' than the other? As you point out, applications should contain unit tests, UI tests - and other contract / component / integration test levels of the test automation pyramid. Did you mean to be controversial to incite more CZcams viewers or have you experienced this test rivalry in a real work place? :-(
I first experienced the battle between my ears: as a long-time Unit Tester, the rise of BDD was, err, challenging!
More recently, a colleague was massively challenged by the suggestion that we should work in a BDD fashion. After all, when something goes wrong... "How do you know what went wrong?"
[Side note: my colleague is a big motorbike fan. It's no coincidence that this series of videos began with testing... a motorbike!]
very good
Thank you!
Uhhhh... i liked the series and the comparison, nice work and all... but you can't take BDD tests out of a language context. You could argue that all that is required for a BDD test is the feature specification (Given, When, Then) as you are doing. On the other hand, what makes this high-level, human language specification actually serve as a test for a feature of code in a programming language... is that BDD frameworks (rspec, mocha, etc.) all include code. Further more, you can't divorce BDD tests from additional abstraction layers and interfaces including headless browsers. Since we are all talking about code, given that we are developers, then we can all agree that TDD unit tests and BDD (acceptance/integration?) tests are similarly portable, if you define portability as cross-platform usage or reliance on frameworks. BDD isn't magically more portable than TDD. You're talking about generalization at the specific abstraction layer that BDD is designed to test around. You can write BDD tests for functions... it doesn't really matter that the abstraction level is low. If your testing framework has multiple language support, then your specs should still translate. Alternatively, you could write BDD tests for web UIs, and then change the whole API layer or web service layer and the BDD tests should still work. BDD is only as portable as the abstraction layer with which it is used.
Matthew - Great comment. Generally speaking, tests are much less portable in practice than in theory. Here's one case where it worked like a charm: an API written in PHP, with tests in Behat (also PHP). I re-wrote the API in Go, and pointed the Behat tests at the new API. Test-wise, the sum total of the change was one line of CONFIGURATION. That's what I meant by portability.
I had to dislike this as I think that there are too many problems with it. These videos seem to be created by someone who has read about the different kinds if testing but never used them, or at least not very much.
1. You are not really comparing TDD and BDD, but rather unit tests and behavioral tests. TDD and BDD are methodologies where the test is written first just using different kinds of tests (as you stated in this series), but all of the discussion here is about unit tests vs. behavioral tests.
2. Unit tests and behavioral tests are complementary, not competitive, testing methodologies, so the whole idea of pitting them against each other and keeping a scorecard to determine a winner is problematic. There can be no winner or loser, because they provide different insight into the system and the code that implements it. As you said in the prior video, unit tests help ensure that the code is "Doing the thing right", while behavioral tests are about "Doing the right thing" or put differently, unit tests ensure that the implementation is correct while behavioral tests ensure that the requirements are met. Comparing and contrasting the two is a good thing as it helps in better understanding them, but making it a competition is problematic in that it that it suggests that we should use only one or the other, when we should use both to achieve both high quality code and code that meets the requirements.
3. Round 5 (coverage): Where are you getting your data that code coverage tends to be higher for behavioral tests than for unit tests? That seems counter-intuitive to me, and it does not match my experience. It is usually harder to get better coverage from behavioral tests, requiring 10 to 100 times more behavioral tests (in a complex system) to cover all of the various use cases and edge cases, and often many of the edge cases are impossible to test with behavioral tests, because they depend on a technical situation that cannot be controlled by the behavioral test as it is testing from the outside in (e.g. How do you get the database to go offline in the middle of a behavioral test?).
Also, what do you mean by coverage? If you are only talking about lines of code or branches and the like that code coverage tools measure, then that is a very limited kind of coverage, Coverage of different scenarios is much more meaningful. A single test could take your tool measured code coverage from 0 to 90%, but it is unlikely that one test covers 90% of the scenarios that could be tested.
As such, I think that you misrepresented the facts in this round, which skewed the outcome, but I recognize that my experience is limited, and you might have broader experience or at least a broader source of data to back this up.
4. Round 7 (portability): You completely mislead the viewer by saying behavioral tests "Aren't coupled to the code." Behavioral tests can be, and often are, coupled to code at the glue layer (in cucumber this is called the steps), so as a whole, behavioral tests can also require changes when the code is ported. Without the glue layer, the behavioral "tests" are not executable, and they don't actually test anything, they are really just requirements.
Tests of services via the web service API might not need to change if you can use the same web service API(s) or the exact same tools to set-up all of your context (e.g. pre-existing data, authentication and authorization, configuration, etc.). If you need to use something new with the ported service, then the tests (at least the glue layer) have to change to work with the ported service code.
When it comes to UI level testing, then you will nearly always need to change at least the glue layer that actually executes the tests against a UI, and if the tests themselves refer to UI elements (input fields, buttons, text, etc.), then they will also likely need to change as it is a rare case that the UI will remain exactly the same when the code is ported.
If the tests are for a library or a framework being implemented in a different technology, then the glue layer will also need to change.
Even if you could reuse the behavioral tests without change, there is a question of whether you would want to, since that might require using different technologies (e.g. languages) for the glue layer than what is used in the code, which might limit who can work on making the behavioral tests executable.
I will grant you that there can be more reuse with behavioral tests than unit tests (depending on the system being tested), but not the complete reuse that you imply by saying that the behavioral tests "Aren't coupled to the code", which is often not true. There is usually still some coupling, but it can be a looser coupling than with unit tests, so porting behavioral tests is often easier.
5. Round 7 (portability): Comparing the ability to port the tests is also a misguided comparison, because even if the port is to use the exact same technology, where the unit tests could be reused, you almost never want to reuse the unit tests, because that would tend to force you to rewrite the same implementation that you had before, because the unit tests are specific to that prior implementation. That would usually defeat the purpose of the rewrite, which is likely to get a different (better) implementation. As such, you would rarely, if ever, want to port the unit tests as that would prevent you from making improvements.
As such, I would say that round 7 is meaningless since portability is only a desirable trait for behavioral tests.
Really appreciate the time you put in to replying. Think you're mostly right and I'm mostly wrong: I'm especially embarrassed to have conflated TTD and unit tests (and BDD and unit tests).
In my defense, my mental model here was a system with an API - with the API being the entry point for the "behaviour". This being the case:
- There could be 1000 functions serving just a handful of endpoints (many units tests, few behavioural tests)
- Behavioural test code is portable (if the api contract is maintaned