Building a Request Pipeline for Separating Concerns with Pipes & Filters
Vložit
- čas přidán 9. 07. 2024
- How do you separate concerns when processing a request? Typical concerns such as Logging, Validation, Exception Handling, Retries, and many more. One way is to build a request pipeline for separating concerns by using the Pipes and Filters pattern. You can also build a pipeline using the Russian Doll model that allows you to short circuit at any point throughout the pipeline.
🔗 EventStoreDB
eventsto.re/codeopinion
🔔 Subscribe: / @codeopinion
💥 Join this channel to get access to source code & demos!
/ @codeopinion
🔥 Don't have the JOIN button? Support me on Patreon!
/ codeopinion
📝 Blog: codeopinion.com
👋 Twitter: / codeopinion
✨ LinkedIn: / dcomartin
0:00 Pipes & Filters
2:50 Russian Doll
5:02 Code Example - Věda a technologie
Never heard before of the term "Russian doll", but is interesting to know that this can be another name to it. I usually implement it through the "decorator pattern", and its usage is great to keep things adhering to OCP.
Great video man! Thanks for sharing
We do the Russian Doll approach through DI based decoration. The DI container makes sure the right decorators are used for each given request/pipeline. We typically do validation/caching/logging/cache invalidation (with successful commands)/authN/Z with this approach. The best part about this is that the cross-cutting concern code stays outside of the "action" logic.
Do you have an example?
Thanks for another great video. I use MediaR PipelineBehaviors for this but good to see that Brighter/Darker also has a similar concept.
You said temporal coupling! You're one of the few content creators who seem to know what they're talking about, not all that SOLID stuff presented by amateurs who don't even know what SRP actually is or who Ward Cunningham is.
Here's a secret, a lot of the videos I post are rooted in coupling and cohesion!
Congrats on the sponsor!
Thanks for the nice video explaining the Pipes and filters and russian doll concept too. I remember using PSR-15 Request dispatcher with middlewares in php, we being able to compose a list of middlewares in the config file for different scenarios and see that request is handled by them as it is received when creating the response
It's a common pattern that I suspect a lot of people use but aren't even aware of it or the names of them.
@@CodeOpinion makes sense
Love you videos!! I implemented something similar in a recent project especially at the end when you mentioned Execution context, here is how its composed ->
1. I have context filters that build context data needed to process the transaction (eg. getting product information)
2. After we have all the context data I have pre-receiver filters that do pre processing (eg. payments) then the request is sent to the "Receiver"
3. After the "Receiver" has finished I have post-receiver filters (eg. save transaction to DB)
All this is a move from a monolith to a microservices architecture with the goal being orchestration. I'm looking forward to your Execution context video so as to learn and compare how we implemented it ourselves
Yes, what you described is common use case where you have a pre/post filters. Ultimately you don't have a final destination, just a series of request handlers that are chained together and using a execution context to "communicate' between each other. The benefits are it allows you test each filter independently, however the filters are temporally coupled.
Good video. Would be great to see an example of a middleware-style implementation from scratch vs. using a library. Our team is having a discussion about this exact topic at the moment so your video is timely. I've found the 'Russian Doll" method to be effective but it does mean you have to remember to not break the chain by calling the next handler. I've implemented this pattern by registering my IRequestHandler implementation types in "Startup" using a Service Descriptor (with DI), then injecting them in a pipeline manager which calls each handler in the order they were registered. This way each handler is distinct, separate, reusable, and has no knowledge of a "next" handler to call. Either way, great stuff and keep up the amazing work :)
Yes, Russian doll is interesting because you have the option to not call the next filter. In my experience, that's generally not very often and you generally do want to go through the entire pipeline. Depending on how the API is designed it still can be fairly seamless to make that painless. Brighter does this by making you return the request, which you can do simply by calling the base class like my example, which ultimately just calls the next filter for you.
Nice video, thank you! My question about what with the context passed between next steps (if we call this sequence pattern) was waiting till last second, but you've alread thought about that - I'm waiting for video :D Cheers.
Yes, I'll be covering it in a future video! Thanks for watching and the feedback.
Thanks for the video. Have you ever shown how you can capture trace level logging from the receiver based on a condition? For example if I want to turn on tracing for one particular user or record that will process but don't want to enable trace logging for everything. Would love to see a video on it in the future.
Create a custom filter that has logic in it to know what user/record. If it needs to be dynamic the possibly using a distributed or in-memory cache where you can fetch that value quickly (could be something like AWS parameter store). Just my immediate thought.
@@CodeOpinion That works for logging the request but not capturing trace application logs from the receiver code.
Request PipeLine = Orchestration, Russian Doll = Choreography .
Hmmm not so sure, when I think of those terms related to Event Driven Architecture. It really depends on if they are composable or not. I mentioned at the end of the video that you could pass data around between filters, but that has coupling implications.
Please create some videos on how .Net core CLR works and how it is able to run cross platform.
Wow that's great, I have implemented similar pattern in Java, specially using the Russian doll model. Just one question, what's the difference between the Request pipeline pattern vs Chain of responsibility pattern ?
Pipes & filters are about transforming or filtering the data as it goes through the pipeline of filters.
How do we define orthogonal concerns. How do you know when someone is doing something that doesn't belong in a filter? and should maybe be a command within in an event handler. In our system there is code that updates a search index. IT pretty much runs after every event handler , so they have made this a filter in brighter. IS that correct? WHen i intially read the word orthorganal i thought of generic cross cutting concerns like logging or validation. However the definition of the word just means perpendicular. Could you define what should be in a filter for me please?
It could be run after every event handler via a filter. Or the event handler could publish another message to do the update search index so that it's done in isolation.
How would it fit in an architecture where each filter is a microservice? Lets say for example every MS does an operation to an image (restoration MS, Object Detection MS),
Should each microservice pass the data directly to the next? Or should it be sent back and forth to an orcheatrator?
Also, if every MS needs to communicate to another, ahould it be done by using RPC, or is it preferred to use REST?
I have a hard time answering this question because I can't imagine why you would need/require each service to be it's own independently deployable unit? I'm not a fan of the term microservices, rather a fan of services that own a *set* of business capabilities.
@@CodeOpinion One reason to advocate making each service to be a deployable unit is mostly scaling (at least how I see it) - if for example one service is needed more than the other it is possible to simply deploy more instances, and for some reason it feels more flexible to me. But I get what you're saying - it would be much simpler to have a service that contains all of those image processing capabilities, and when you split it up to many tiny services it could get very complex (although the services themselves may be very simple, the overall design would be more complex - please correct me if I'm wrong).
I also want to thank you for taking the time to answer my question and creating wonderful content :)
This reminds me of the decorator pattern.
Similar but it's not wrapping.
Awesome video as always.
Question:
Lets say I have a Clean Architecture based application, my application layer has a public interface for my use cases, that concrete implementation of this interface gets injected into a WebAPI That consumes it.
Where should the Authentication and Authorization logic leave?
The reason why I am asking this is to understand where I should have a filter that would for example handle if the user making the request is allowed or not to make that request.
Where do you put Authorization now?
Just read this comment and that's something I'd love to know too.
In my case it's a Blazor Server project, no WebAPI, no requests in the sense of HTTP requests. Still requests tho, in the sense of the UI client calling use case/features handlers by passing appropriate command/query objects.
My project is pretty much organized in terms of "vertical slices" - some request object, validator, and all the rest of it + a handler for each feature. I am not using MediatR.
As I'm inexperienced and still lack a grasp of many design patterns, the best I can do right now is to define a class containing the standard access rules for my request and from there to remember to guard against them in each and every handler.
Say I have a feature handler called DeleteAccount, in its Execute method I must remember to start with accessControl.ClientMustBeLoggedInWithRole("admin"); before I proceed with the validator and so on.
Seems better then MediatR. But I did not get the point. Sorry for that.. Does IRequest , RequestHandler come from Brighter library...??
Yes IRequest comes from Brighter. I was using Brighter as the example just to illustrate. Brighter is a full messaging library that supports different transport. Similar to MassTransit or NServiceBus. MediatR is just an in-memory dispatcher.
@@CodeOpinion Thank you for the response and video.
These filters are equal to mediateR pipeline, correct? Is this more specific for each request than a generic pipeline?
Yes, MediatR you can create a pipeline. It's one of the benefits to using it IMO.
How is this different or better than Mediatr.NET?
It's not. MediatR supports "behaviors" which are for creating a Pipeline using the IPipelineBehavior.
This Russian Doll filter sounds like Choreography from SAGA...isn't it?
I don't think of it that way. If anything I'd say closer to orchestration if you were to make it asynchronous (compared to how I illustrated).