I Bet You’re Overengineering Your Software
Vložit
- čas přidán 23. 05. 2023
- One of the common causes of poor, overly bureaucratic software development process is a fear of over-engineering, but what is overengineered software, how do we stop overengineering and is overengineering really the problem anyway?
In this episode Dave Farley author of best-selling books “Continuous Delivery”, “Modern Software Engineering” and “CD Pipelines”, describes 2 different causes of over-engineering and how to avoid them, Dave also explains the difference between over-engineering and designing in a way that leaves your system open to change.
-
⭐ PATREON:
Join the Continuous Delivery community and access extra perks & content!
JOIN HERE for as little as £2 ➡️ bit.ly/ContinuousDeliveryPatreon
-
👕 T-SHIRTS:
A fan of the T-shirts I wear in my videos? Grab your own, at reduced prices EXCLUSIVE TO CONTINUOUS DELIVERY FOLLOWERS! Get money off the already reasonably priced t-shirts!
🔗 Check out their collection HERE: bit.ly/3vTkWy3
🚨 DON'T FORGET TO USE THIS DISCOUNT CODE: ContinuousDelivery
-
🔗 LINKS:
“Software Quality”, Wikipedia ➡️ en.wikipedia.org/wiki/Softwar...
“Over Engineering”, Wikipedia ➡️ en.wikipedia.org/wiki/Overeng...
“Accelerate Review”, San Tuon ➡️ www.santuon.com/accelerate-th...
“Over Engineered Bike” ➡️ • Just a Normal Bike Mat...
“Over Engineered Archery” ➡️ • Auto-aiming bow vs. FL...
-
BOOKS:
📖 Domain Driven Design, by Eric Evans ➡️ amzn.to/2WXJ94m
📖 "Extreme Programming Explained: Embrace Change", Kent Beck ➡️ amzn.to/3K5fhg6
📖 Release It!, Michael Nygard ➡️ amzn.to/38zrINu
📖 Test Driven Development: By Example (The Addison-Wesley Signature Series), Kent Beck ➡️ amzn.to/2NcqgGh
📖 Dave’s NEW BOOK "Modern Software Engineering" is available as paperback, or kindle here ➡️ amzn.to/3DwdwT3
and NOW as an AUDIOBOOK available on iTunes, Amazon and Audible.
📖 The original, award-winning "Continuous Delivery" book by Dave Farley and Jez Humble ➡️ amzn.to/2WxRYmx
📖 "Continuous Delivery Pipelines" by Dave Farley
Paperback ➡️ amzn.to/3gIULlA
ebook version ➡️ leanpub.com/cd-pipelines
NOTE: If you click on one of the Amazon Affiliate links and buy the book, Continuous Delivery Ltd. will get a small fee for the recommendation with NO increase in cost to you.
-
CHANNEL SPONSORS:
Equal Experts is a product software development consultancy with a network of over 1,000 experienced technology consultants globally. They increase the pace of innovation by using modern software engineering practices that embrace Continuous Delivery, Security, and Operability from the outset ➡️ bit.ly/3ASy8n0
Sleuth is the #1 most accurate and actionable DORA metrics tracker for improving engineering efficiency. Sleuth models your entire development cycle by integrating with the tools you already invest in. You get a full and accurate view of your deployments, see where true bottlenecks lie, and keep your team’s unique processes and workflows. With accurate data, Sleuth surfaces insights that your engineers can act on to improve - with real impact. ➡️ www.sleuth.io/
IcePanel is a collaborative diagramming tool to align software engineering and product teams on technical decisions across the business. Create an interactive map of your software systems and give your teams full context about how things work now and in the future. ➡️ u.icepanel.io/1f7b2db3
Tricentis is an AI-powered platform helping you to deliver digital innovation faster and with less risk by providing a fundamentally better approach to test automation. Discover the power of continuous testing with Tricentis. ➡️ bit.ly/TricentisCD
TransFICC provides low-latency connectivity, automated trading workflows and e-trading systems for Fixed Income and Derivatives. TransFICC resolves the issue of market fragmentation by providing banks and asset managers with a unified low-latency, robust and scalable API, which provides connectivity to multiple trading venues while supporting numerous complex workflows across asset classes such as Rates and Credit Bonds, Repos, Mortgage-Backed Securities and Interest Rate Swaps ➡️ transficc.com - Věda a technologie
📖 Dave’s NEW BOOK "Modern Software Engineering" aims to help you think about your work more effectively, manage it more successfully, and genuinely improve the quality of your applications, your working life, and the lives of your colleagues. It's available as paperback, or kindle here ➡ amzn.to/3DwdwT3
and NOW as an AUDIOBOOK available on iTunes, Amazon and Audible.
I have definitely overengineered in the past. I hope to have learned from these experiences.
I concur with what Dave presented here 100%. I now try to practice YAGNI, but I also try to keep the design flexible enough to accommodate what's coming in the future, even if I cannot predict it.
In my current experience, I think the problem of underengineered is more prevalent than overengineered. I see a lot of Big Ball of Mud. I suspect most of us here have probably experienced the same.
I agree Jim, under-engineering is the more common outcome, I wonder though if sometimes this is the result of the pressures that are, wrongly, applied to avoid over-engineering. ‘Under-engineering” is probably a good topic for another video 😉😁😎
Yeah... it would be interesting to talk about over and under engineering... and finding the "goldilocks zone" where there's just the right amount of engineering.
Some people interpret agile as meaning aggressive underengineering. When the defect rate torpedoes the velocity, then there is and agreement to do a "refactoring" sprint. Usually this sprint happens over a December and practically nothing that is planned gets finished, but the little bit helped enough to eek the velocity up again to a level that the stakeholders accept. Then rinse and repeat every year. I've had this unfortunate experience in more than one workplace.
Over engineering can also lead to a big ball of mud when new devs don't understand the intended architecture or patterns and either ignore it it mess it up. Or become too afraid to change it 😬
❤ I call my system of working just-in-time development where you only implement what you need right now: start with running code and keep it running. Why just-in-time? Because JIT happens...
That’s fine for a job but developing this way can lead to duct taping something very brittle.
@@mecanuktutorials6476 I totally disagree. Building incrementally creates robust code because it is constantly run and tested. The software grows organically. There is never a need for tape of any kind.
@@brownhorsesoftware3605 I don’t disagree that it works on a team of 1 or 2 that grows properly but often, you have a team of say 5-10 people who all have different tastes, priorities, and incentives. The initial bring up is the most important and if you leave it to JIT, it’ll most likely be 5-10 disparate, in cohesive pieces of work.
JIT is something I used in my own projects because it’s natural but it wouldn’t ever work in any corporate job where “what you need right now” is not determined by you. Everyone needs different things right now. Now you have integration problems because people work in parallel. And their features are incompatible or overlap with yours. That’s the challenge, why planning becomes important.
At the end of the day, if everyone does JIT programming, it doesn’t mitigate the project becoming a mess. That’s why most corporate proprietary repositories tend to be monstrosities despite having all of the tests, automation pipelines, robustness, etc.
Applying The Iron Triangle to software is also another symptom of the delusion that what programmers are doing is Production.
I'm in my 30's. I spent most of my life working as a truck crane operator. I transitioned into IT 1.5 years ago in a non technical role so invested time in learning about both project management and lean-agile strategic management. The fact that anyone would apply a project management mindset to software development I find absolutely baffling and shows a complete lack of understanding of what software development is. Pretty sure I'm quoting Dave here with a very simple explanation for business managers that what we're doing "is solving real world problems with working software." It doesn't make any sense to focus on delivering a thing. Focus on the value that's being delivered, which is the solving of the problems.
Even in production the iron triangle is not always applicable. If the product is not good you have a lot of discard. That is neither fast nor cheap.
In software development it sometimes serves as an excuse for delivering a mediocre product and missing deadlines and /or budgets. In my enviroment I observe a shift to more test driven work. I adopted it some years ago and it really helps to get it right the first time and facilitates changes. Fewer test and rework cycles for a good product make it cheaper and faster just as Dave said.
*_All_* of this, Dave! Once again you're telling my story.
For so many years that I'm afraid to count, I've been the sole developer on an OS for Creators. I *did* have a clear objective: a platform that allowed me to easily build UIs and features as easily as creating drawings; this was to be a proprietary tool solely for my content. However, it took me far too long to realize that I, as an artist AND developer, was a very poor target. I'd over-engineered the platform so much that it was far too easy to lose sight of my objectives.
I realized that I needed *_external_* targets, so I tapped long-time users of a CG application that I'd developed in the late 1990's - early 2000's. *_That was the ticket!_*
Even though the feedback that I'd gotten contained nothing really new, it gave me clarity. I redefined the product objectives, swept out a lot of over-engineered subsystems, then approached the design from the view of myself purely as creator. Yes, not just a user or "end user," but a _person_ creating art. This changed the entire paradigm of the OS. Simplified it. And made it the most flexible piece of software that I'd ever seen!
Seems like a simple thing, that realization, but only in hindsight. It was a long and arduous journey, but it looks like I'll be able to release in Q4 of this year. So, to whomever read all of this, pay close attention to this video in particular. Whether you're an individual or part of a team, it will save you a world of hurt. Hell, I wish *_I_* could have seen it a decade ago....... 😔
The most valuable meme from this video: 'The most important quality of software is to be easily changeable, even more important than to actually work! Because, if it does not work, we can easily and quickly fix it!'. Exactly my thoughts too, but I could not express this meme as clear as Dave, of course.
Thank you Dave for the content you create. It is extremely helpful. I see you as an industry senior and an inspiration for me.
Something about the video itself. When you are talking, there are these animations and text points popping up on the screen, which takes away focus from what you are saying. I mean, the mind has to switch context to read what's on the screen, and while doing that I miss on what you were saying at that moment. The animations have increased in your videos lately, when compared to your older videos. I find your older videos to be more useful and understandable solely because of this.
Always love the perspective on these things Dave. Trying to future proof things is a great callout and agree on some level it's driven by fear. In a lot of cases that fear is likely justified due to poor internal processes that decimate throughput and make it basically impossible to respond to change. I can't see many high performing teams having fear of being able to build and adapt, only the places that are a total circus.
On a lower level, this is why BDD is such a useful process for solving problems with code. You've broken down a requirement into a small piece that will tangibly benefit an end user, and you are coaxed to write the minimum amount of code to get a test to pass and deliver the feature. It's very difficult to overengineer a solution when solving problems like this.
The killer question to the over-engineerer is: how does what you are doing make the company richer? Most of the time the answer is based on some contingency that will probably never happen. It's still amazingly hard to make them stop.
"The point of software is that it's soft." Surprisingly profound.
This is one of the best videos I've ever seen on software engineering. My biggest problem: My lack of ability and charisma to communicate these things to the proper folks at my company. I'll keep trying though.
Excellent video!! I've been involved in a couple of "framework" projects over the years, and have suffered from building things that were never used. Since the last one in the early '00s, I've tried my level best to follow exactly the approach Dave mentions in this video: build incrementally and only for the problems you need to solve right now!
Well, lets see this example. Lets take revision control for source code as a talking point. There was RCS, then CVS, and SVN, ClearCase, Perforce, BitKeeper, to name a few. (ClearCase was always described as "CVS you had to pay for"). Then the git source control software and it was written by the linux developers FOR the linux developers, they knew their own functional requirements and had to switch for licensing arguments with bitkeeper... They needed something reliable, scalable... they solved their complex needs so well, it became the defacto source control standard for the modern age and it has TONS of features that are amazing. Secure features like PGP signed commits and quality of life features like rerere.
I think we could all agree that GIT looks like a quality piece of software that is definitely not "over engineered"
SOLID should be applied after YAGNI and KISS. Critically, you should not apply SOLID to things you YAGNI away. I think this is where a lot of over-engineering comes from.
One thing I've noticed is that the amount of speedup you get from deferring quality doesn't last very long. On the order of a month or two. If you're deferring quality for speed for several months, you're actually moving slower. I've been on some projects where we were "6 months from launch" and people were deferring on quality left and right. The codebase quickly turns into a disaster.
To keep this working though you need to allow for people to constantly refactor the design. Which I think is important, and you shouldn't have to get a manager's permission every time you need to refactor things. Because see the above paragraph: skipping it will make you slower very quickly.
I’m not 100% sure I follow your statement that “SOLID should be applied after YAGNI and KISS.” It seems like what you are saying is that you shouldn’t apply SOLID to code you do not write, which, yeah - seems natural but needs not be said. So, I wonder if I understand you correctly. I thought you may be trying to say “do not spend any extra time writing code that aligns with SOLID principles until you are sure the code will stick around, then refactor it.” I am not sure that is a statement I could agree with. There are times when “quick and dirty” is perfectly acceptable. One-off scripts that I write to transform a data file or some such are not worth the effort and are usually better to be written using the imperative, top to bottom ways of old.
@@pchasco only apply SOLID to your requirements, not "what if". I've seen people trying to apply the open-closed principle to things that weren't features or requirements.
For the quick and dirty stuff. Yeah. That's why I time boxed it. If you're going to be cutting corners for months, you're slowing down. If you cut corners for a week or even a month, that's fine.
@@bobbycrosby9765 Still not sure I follow. Open-closed principle has to do with how we implement a class and doesn’t have anything to do with requirements outside the detail that we build programs to fit requirements, and programs are composed of classes in most OO languages.
@@pchasco the scope of the open-closed principle is limitless.
Your requirements are not.
If you "violate" the open-closed principle only if your requirements change, then it's not a violation.
If your requirements do change, you can refactor at that point.
Great insights there. Some of the major contradictions/confusion I have encountered are the "open/closed principle" ... and the misinterpretation of "anticipate change", also abstracting beyond the "problem "domain. Also too many unit tests which test interactions "in the guts" of the code base - making it virtually impossible to make small, incremental, refactorings.
What you say is 100% spot-on. As you have data, please provide some concrete examples!
"In programming, if someone tells you 'you’re overcomplicating it,' they’re either 10 steps behind you or 10 steps ahead of you." - Andrew Clark
Design for expandability, but don't implement a feature unless you have a good reason to believe that the user wants it.
I would like to hear more about this topic as it relates to the must haves and nice to haves as communicated by product teams. I'm personally not a fan of building product organizations for building your in-house technology needs, but it will always be a tough balance walking the fine yet distinct line between good separation of concerns and overengineering as you add more stakeholders.
in re the Iron Triangle and speed, as a solo developer I often base pricing on how much of my daily schedule is even available to a client, so they will pay more to get a nearer deadline out of me for a given scope of work.
it really is a balancing act. i always find that the tools are what limit what can be expressed, and that over engineering usually happens when trying really hard to get around these limitations. sometimes the limitations are due to a lack knowledge, and therefore making things more complicated than necessary. and yes you always have to ask yourself, whether the added complexity is worth it, like you said. with tools i mean pretty much anything given to you, like: frameworks, protocols, IDEs, the operating system, but especially programming languages
I agree, I've seen this a lot in my current organisation. When I started we were a very young operation and had only gone to production that week. As it happens we became successful quite quickly and user numbers were about 1 year ahead of projections within 3 months. So we had to mature the product very quickly, and there was a period of a few months when developers were getting very agitated with our CICD team because they were suddenly needing certain features in builds and pipelines that there simply hadn't been time to implement when the goal was to get a working product. We very much had to go through a "workaround" period and you can easily tell which projects those are when you look at them today. So much unnecssary "functionality" along with all the testing and validation, and that testing and validation has to be more thorough because of the non-standard patterns being used. I would agree pretty strongly that a lot of overengineering comes from a place of needing to vault a barrier that, in principle, shouldn't really be there in the first place. Although I have seen some that comes from a place of wanting to be the coolest dev out there 😅
I don't know what to say other than spot on. Great video. ❤
Great video!
This is the video I needed today.
One of the managers in my team asked us multiple times to make our product configurable and to address as many use cases as possible, thinking at the future, and he has wild ideas about what the future may bring at the point that it seemed he wanted to build the Musk's everything app. I told him that we can start to think about these hypothetical features and use cases only after we defined what's the specific problem and our business requirements we want to address. I managed to save the project from him so far.
The problem is that he's obsessed in allow new use case without changing the software and is afraid of touching the code, in his dreams we should get a new feature simply by configuration.
Yikes
I've been around that cycle a few times. Eventually you introduce some sort of scripting language to make your process completely customisable. Then you've made the configuration so complex it takes a programmer to set anything up. Usually that scripting language is not as good as the underlying language of the app, so you're still relying on developers to make changes, and you still have to test any change - but changes now take longer than when you were directly changing the code!
@@robinbennett5994 yes, exactly my point! It's madness. And the worst thing is that no matter how many configurations you put, it will never cover all the cases and you'll always need a developer to add stuff and configuration. It's a vicious cycle that kills project transforming them in big complicated and scary monsters. The classic project nobody want to works on, that causes good developers to resign and the on which ultimately you put only junior devs that will make the situation worse and worse
Each and Every word is so much true. People are trying to use design patterns even there is not any need there.
In my perspective these things have nothing to do with each other.
I'll take the definition of engineering that is commonly cited, the reference I find is Koen 2003, but I got it from Bill Hammack's latest book: "Engineering is the use of heuristics to cause the best change in a poorly understood situation within the available resources".
To me, most of the things we talk about in software engineering in terms of quality software design practices go into the "heuristic" bit: our ideas about things like coupling, cohesion, and encapsulation are employed as engineering heuristics. We are doing a good job of quality insofar as we are applying appropriately applicable heuristics, the best tried-and-tested that we have available to us, and adjusted and reinforced necessary to fit our operating context.
Whereas "over-engineering" refers to violating some part of the definition of engineering in some other way: usually not delivering the "best change" in some way.
"Best change" here is always understand in a cultural context. E.g. writing a payroll app in hand-coded assembly is "over-engineering", in the sense that it does not deliver the "best outcome": the software may be much faster than a payroll app written in Java, but the software will have taken too long to develop, it will be harder to change, it will be hard to hire new engineers to work on it etc. Over-engineering usually means placing too much weight or emphasis on one aspect of the "best outcome" at expense of others.
But everything is broke these days.
I don't think enough time is spent before the release.
This problem has very little to do with software engineering per se and everything to do with the politics and power structures around the software industry.
Most software these days is developed using iterative methodologies, where software engineers are delivering software artifacts and executables up to the business on a cadence of anywhere between one to four weeks. This gives the business insight regularly on where the software is, in whatever state it is at that week/month etc: features, bugs, warts and all.
After an iteration, the business can see and test for themselves clearly whether the software is ready, they can use and test the software and see exactly where the defects are and where it is lacking, and whether it needs more iterations to improve it.
But usually the business simply decides to release the software as is. There are many reasons for this, starting with that hiring your own in-house user testers to harden products over multiple iterations before release is very expensive, whereas releasing to production and treating your end-users as guinea pigs is free, and they don't seem to mind it.
And then on top of that, even after releasing broken stuff to the market, the businesses rarely slow down their "product roadmap" to stop and focus on bugs that they find in that guinea pig stage. Instead, they simply plow ahead, making developers work on new features and functionality when the software is already riddled with bugs and technical debt.
Usually this is pure politics: incremental methodologies present harsh reality to the business in plain terms, but the CEO has his "product roadmap" and doesn't care at all for reality. He has investors to make happy and trick into believing he's competent/in charge of things, and ticking boxes on his "product roadmap" is the easiest way to do that.
It stands to reason that there certainly *should* be a lot of broken software out there. After all, there's a hell of a lot more of it today than, say, in the early 2000's. A lot more people are coding who probably shouldn't be. A lot more reliance on frameworks and libraries; while that should be a good thing, too many have no idea what using so many external components does to their software. Add to this jumping around to the latest language or whatever like it's the fashion isn't helping matters: too many don't really specialize, but generalize. I swear, I wouldn't want to be starting out today.
Yeah, 5 decades of talking about how to make software better and all it does is get worse and slower.
I have a very limited experience with production level software but, from what I've seen so far, the code resulting from "technical masturbation" is regarded as safe, mature, and well-though out whereas simple-straightforward solutions are barely taken into consideration. For gods' sake, I witnessed a tech-lead use inheritance to avoid repeating a couple lines of code twice.
I always recommend to my peer this excellent channel. But this video is confused.
Type 1 master vision don't exist, unless you count "monolith is bad" or "kurbenetes is good" or "scrum is agile" or "I code in my garage" as "master vision". Which are just an instance of "preferring style over substance"
Type 2 future proofing. This *IS* engineering, it is perfectly described at 19:00, except it is a contradiction. How can you "leave things open", if you are not "attempting to predict the future ?". What "close" would even mean outside of this "predict future" concept ? What is "change" BUT a future event ?
How can you spent time "NOT delivering value to customer", by abstracting your cloud provider or your communication protocol, if not because you expect it to change in the future ? How many of us have tried to explain all those perfectly sound concept like SOLID to "management" and get the message to stop "overengeneering" this and JDI (just do it) ?
As a side note, I don't expect DORA is able to measure "speed of change" in any meaningful way.
100% Agree
This video is two in one: over-engineering and a definition for quality. I enjoyed both subjects. This is all advice I will take to heart.
In the Dora metric, what is the meaning of "throughput"? When you said "speed and quality (stability and throughput)".
"Throughput" is an indicator of efficiency, in terms of frequency of releases into production and lead-time. I have several more videos on measuring quality and efficiency of code. For example: czcams.com/video/hbeyCECbLhk/video.html or the full playlist: czcams.com/play/PLwLLcwQlnXBwvH8Iqs9zqkbSWdvWoyX4v.html
At 1:20, some person trying to drive centred on the yellow line, with a slight oscillation. Totally distracts me as I imagine having to avoid having an "accident" with that quality-focused genius.
I got a better grip on my tendency for perfectionism after I heard Alan Kay talk about "What Is Actually Needed".
I think TDD works well to limit this tendency. f
It seams a lot of the webpages made is bloat, and bad ideas.
The iron triangle is real when management wants to throw features over the fence. In their eyes production of software should be fast as possible, that's y they bought into agile for the 1st place. So they make devs trade quality for speed and they expect to deliver as much story points as possible. Then delivery is done, we got our money from the customer, all good. Any complaints? Devs are to blame and a whole new chapter begins.
You may think this is madness but it's quite common actually.
“Real” but “not true” there’s a difference. Sure people act as though this is true, but it is an illusion, so acting this way is a pretty bad mistake, always.
@@ContinuousDelivery OC it is an illusion. Sooner or later, someone is going to pay for the ignorance.
I need a "refrain from technical masturbation" T-shirt!
4:40 I think u minunderstood "quality" here. It does not mean code quality, but rather functional quallity, ie. the number of features provided by the software.
What do you mean? If your code quality is low you must spend more time finding bugs.
So if your code quality is low it takes longer to develop new features. So your definition of having a growing set of features is not possible.
I prefer J B Rainsberger's Simple Design Dynamo. Who'd disagree with Kent Beck's Four Elements of Simple Design?
Similarly, I appreciate Kevlin's "Simplicity Before Generality, Use Before Reuse." I appreciate that you can easily pivot in any direction starting from simplicity.
What a blōdy wise gent
Oh. Absolutely. Guaranteed we over engineer our software all the time.
Curious thought on my head while watching the video: even Nature chooses the more adaptable to environment changes individuals for survival and reproduction, on its process of Natural Selection.
You don’t need a Ferrari to drive to the shops!
My overengineering *is* a problem.