Pipeline-oriented programming - Scott Wlaschin - NDC Porto 2023
Vložit
- čas přidán 18. 01. 2024
- This talk was recorded at NDC Porto in Porto, Portugal. #ndcporto #ndcconferences #functionalprogramming #developer #softwaredeveloper
Attend the next NDC conference near you:
ndcconferences.com
ndcporto.com/
Subscribe to our CZcams channel and learn every day:
/@NDC
Follow our Social Media!
/ ndcconferences
/ ndc_conferences
/ ndc_conferences
Passing data through a pipeline of transformations is an alternative approach to classic OOP. The LINQ methods in .NET are designed around this, but the pipeline approach can be used for so much more than manipulating collections.
In this talk, I'll look at pipeline-oriented programming and how it relates to functional programming, the open-closed principle, unit testing, the onion architecture, and more. I'll finish up by showing how you can build a complete web app using only this approach. - Věda a technologie
Any Scott Wlaschin's speech is a quality mark. Thanks!
A presentation by Scott Wlaschin - instant click!
I've been using this pattern for some years. Data train goes from station from station, no side effects. Just accumulating operations as data (makes testing very easy is well). On some stations, some choo-choo is performed (fire the side-effects) and then the train continues its journey...
"Pipeline-oriented programming" is more properly termed "concatenative programming". There is an entire family of languages devoted to it, namely Forth, Joy, Factor, and Listack. There is even a concatenative hack of Clojure called Gershwin. Manfred von Thun did a lot of the theoretical work on this programming style 20 years ago. It's really the ultimate extension of function composition. The basic principle is simple - you send data through functions in the order in which they act upon the data. (As opposed to the usual inside-out or backwards function call style.)
Or dataflow programming.
I'd like to stick to pipeline as terminology. Thanks to both of commentators for invaluable input as to what "proper" might be, but pipeline works just great so if you don't mind - I'll use that one. Cheers.
Also Functional Programming
Or functional reactive programming. There are so many names for it. Even monads do that, they are "pipeline-oriented", and they solve, what a coincidence, the issue of IO in pure functional languages such as Haskell.
Scott's talks are marvelous. Here's a list of all his best talks so far ordered by the number of views:
1. "The Functional Programmer's Toolkit - Scott Wlaschin" - czcams.com/video/Nrp_LZ-XGsY/video.html
2. "The lazy programmer's guide to writing thousands of tests - Scott Wlaschin" - czcams.com/video/IYzDFHx6QPY/video.html
3. "Four Languages from Forty Years Ago - Scott Wlaschin" - czcams.com/video/0fpDlAEQio4/video.html
4. "The Power of Composition - Scott Wlaschin" - czcams.com/video/vDe-4o8Uwl8/video.html
5. "Pipeline-oriented programming - Scott Wlaschin - NDC Porto 2023" - czcams.com/video/ipceTuJlw-M/video.html
6. "Reinventing the Transaction Script - Scott Wlaschin" - czcams.com/video/USSkidmaS6w/video.html
7. "Thirteen ways of looking at a Turtle - Scott Wlaschin - NDC London 2021" - czcams.com/video/g06igkxbF78/video.html
8. "Building confidence in concurrent code with a model checker - Scott Wlaschin - NDC Oslo 2020" - czcams.com/video/tqwcz-Yt9gQ/video.html
btw. I'm building a newsletter called Tech Talks Weekly (techtalksweekly.substack.com/) where once a week I send out all the recently uploaded tech conference talks across +100 engineering conferences including NDC, GOTO, Devoxx and more. Consider subscribing if this sounds useful.
btw. I highly recommend his book called Domain Modeling Made Functional where he presents Functional Programming as a perfect paradigm for doing DDD.
I subscribed. Also added all of those in a playlist
czcams.com/play/PLdbbFSJJ_c_RC2c5_te9aKtS7B3BdQD0J.html&si=AdqVlojwkehvs-Be
@@joaomonteiro6245 Great idea with the playlist, thanks for putting this together!
And also thanks for subscribing :)
@@TechTalksWeekly no, thank You for putting the links together and the Newsletter!
Amazing presentation.
To me, in the narrow realm of software engineering, systems and software design are the two of the most sexiest areas ever. The best part is it will never get old or out of fashion.
A nice approach to software development and coding 👨💻😎, but I find it somewhat similar to the middleware and loop pipe approach that was introduced in .NET Core 3.2, and also bears resemblance to the Aspect-Oriented Programming approach that focuses on the business management layer, so I see it as taking an advantage from AOP. And an advantage of middleware, and the result was this beautiful approach. I do not deny that it contains its own features, but the basic things that represent the essence of this approach are what I mentioned previously. Thank you, teacher, for this nice lecture.
I'm a fan of fluent interfaces as well as functional programming styles, but when it comes to pipelining, I'd love to see an example of connecting functions with more complex signatures.
that can get tricky..
What exactly can get tricky? "More complex signatures"? What does that mean? F# arguments are always curried.
A let binding can hold any data or partially applied function.
This is an OO answer, but you can use the Command pattern to encapsulate all the details of an operation, thus making it possible to build a chain of commands. The output of a command could be used as input to the next command and so on
Hmm. This is what I've been doing for years, but I've been calling it OOP. I probably started building like this around the time I discovered TDD or Clean. What a great talk.
52:15 It’d be nice if it was possible to use React in this way. React being the pure code parts (explicitly calling React.render) and I/O being user input or network requests. Either you could have all fetches in the top level, respecting the purity of this Onion architecture (aka. ‘functional core, imperative shell’). Alternatively, the React.render call could collect all side-effects (like fetches) and then you could batch them and execute them in the I/O step. Like a structured concurrency approach with EffectJS. It would solve the fetch waterfall problem.
maybe you could co-locate fetches in XState machines next to the components, but pass that machine down from the top of the page, so it is always called (with all fetches for each component on the page) on page load?
I've been programming like this for a few years now and it has been a game changer. Being able to reason about pure logic and complicated runtime state changes separately just scales so much better, is easier to debug, maintain, understand, etc.
This is btw. implemented in frameworks such as Redux, ELM (view-update-model), FRP and similar. I would argue that even React works like that: you have pure, declarative components, event handlers and state management. While they can be kind of put together they are conceptually separated.
The idea is always the same: you have a functional/pure part separated from the IO part. They all only disagree on, or have a different vision of, how to do this exactly.
Actually, this pattern is very widespread when I think about it - seems the entire industry silently embraced it without many people even noticing. My theory is anyway that sooner or later FP will dominate development. There is hardly any mainstream toolkit, language or framework that doesn't include at least a few FP-concepts - and it's getting more.
fantastic talk !
Didn't Scott originally use the name "Railway-oriented-programming" instead of "Pipeline-oriented-programming"?
Railway is for error handling.
Is this any different than Clojure's thread[first|last|as] macro with a sequence?
can we overload operator | in C#
actually functional programming
Just when I thought that I need to learn more about pipelines 😊
How is this diffrent form the fluent interface design pattern?
It’s the same but the pipe generic is interesting addition.
Interesting, but how do you break a pipeline if the remaining functions should not be executed? With a for loop you can break or return early.
I suspect you'll get the answer on the other talks. There were several examples of this in the simple web server: GET fails if the request is not valid and the following functions (route and setStatusCode) are not executed.
Mix it with the Result pattern
basically you prepare each function to short-cut. See more on this in railway-oriented programming: czcams.com/video/fYo3LN9Vf_M/video.html
He showed it in the webapp example, it uses maybe monad without saying a word monad and making railway analogy instead.
PowerShell!
Aren't these just called streams?
"Grokking Simplicity" (2021, Eric Normand) is a more thorough introduction to Functional Programming (also the book crashes a lot less than this guys laptop, yeesh windows)
Hey, it's not his fault that Windows isn't FP and pipeline-oriented ;)
Guru Meditation...
amazing its the same talk he has been giving the last 5 years.. NOTHING NEW!!! Read his book (years old by now).. same content..
education never stops, there are always more people and new generations.
Yeah, because these are timeless concepts.
Great talk but Windows let him down, switch to Linux and say bye to your blue screens
As if Linux is more stable than windows on a desktop
This appears to be nothing more than pure functions plus method chaining, which is indeed a useful idiom, but it's hardly a "paradigm". I don't see how it's more composable or testable than pure functions in general, nor do I see how it is an alternative to OOP. Actually, in most languages, this is only possible thanks to OO method call syntax.
I think you are wrong. He just misnames it. What he is talking about seems like DDD, but he for some reason calls it pipeline instead.
Was thinking the same. We do this all the time in pure functional code. there is literally no other way. He most likely just tries to find a less intimidating name for functional programming. For whatever his reason is...
I think this talk is really great for people who are unfamiliar with FP. It highlights the benefits of composition without putting FP in the forefront. I still work with "senior" developers that refuse to learn or use LINQ based solely on opinion and dogmatic principles. Giving it a new name might make those people tune in.
Yeah, I actually give Scott a lot of merit for simplifying FP concepts and design for all the tech leads and CTOs out there that wouldn't otherwise sit for even 5mins to listen to an FP talk, unfortunately for them and for all the developers actually interested in finding better ways to write composable code
Yeah, it's just FP explained to non-FP programmers. But then also why do we even need an OOP-alternative? To this day no one could explain to me what kind of problem it really solves.
> Actually, in most languages, this is only possible thanks to OO method call syntax.
No. How so? In most languages the OO method call syntax is just syntatic sugar for function(self/this, ....). Even virtual functions are nothing more than fancier function pointers.
In C++ for example a class is pretty much equivalent to a struct (the only difference is that class members in a struct are public by default and private in a class). Inheritance can be easily replaced by composition (which is superior anyway).
There is absolutely nothing(!) that can be solved easier with OOP than without. It's just a non-sensical abstraction put on top. Not only did it's inventor (Alan Kay) had only a vague idea what it should actually be, no he also admitted that he didn't understand FP and said that OO was wrongly implemented in pretty much every modern language (at least not according to his idea). It was a huge mistake, but companies invested billions in it (Java *cough* *cough*) so many don't want to admit it.
Компьютер перезагружался потому, что народ охренивал от того, что он снова в очередной раз рассказывает одно и тоже! Сколько можно уже одно и тоже рассказывать-то? Может уже пора на пенсию?!
😂
Wow - just like core idea in UNIX in 1975.... LOL!
When world is going to completely reject this madness with OOP?
When Java dies... 😅
@@raptoress6131 which is not happening
That process has started. It won't be a complete rejection, but with 'Out of the Tar Pit', OO-everwhere began its descent.
OOP is good
This is just OOP. The arrows are the class data the boxes are methods. The good thing about OOP is scope. The data in the arrow is private