Understanding Streams | Understanding Node.js Core Concepts FREE VERSION
Vložit
- čas přidán 7. 09. 2024
- Grab the complete and premium version of the course using this link:
www.udemy.com/...
Let's dive deep and understand all about Node.js Streams, one of the most important concepts every Node.js developer needs to master. We'll first understand how to use existing Writable Streams, then we'll learn how to develop our own Writable Streams, and then we'll move on to Readable Streams, and then Duplex Streams, and finally, we'll understand Transform Streams by creating a simple encryption/decryption application.
Final Source Code: github.com/agi...
The Course - Free Version: • Understanding Node.js ...
Chapters:
Understanding Writable Streams: 00:05:28
Understanding Readable Streams: 1:51:42
Understanding Stream Piping: 2:59:35
Implementing Our Own Writable Stream: 3:51:13
Implementing Our Own Readable Stream: 4:33:32
Understanding & Implementing Duplex Streams: 4:53:21
Understanding & Implementing Transform Streams: 5:09:38
Outro: 5:48:30
00:00 Introduction
05:30 Code: Benchmarking different ways to write to a file
31:18 Code: Benchmarking writing to a file using streams naively (faster but not memory efficient)
38:05 What are streams in NodeJS? Explanation with analogies
46:36 Different types of streams (with examples)
53:10 How do writable streams work?
1:01:05 How do readable streams work?
1:04:16 Code: Using writable streams to make previous program more memory-efficient
1:41:06 Reading NodeJS docs about writable streams and recap
1:51:44 Code: Using readable and writable streams in conjunction
2:12:45 Code: Selectively write chunk of data using streams
2:21:32 Explanation of what's happening with numbers being split when chunking the data
2:38:20 Code: Implementing the solution that solves the splitting issue
2:54:36 Reading NodeJS docs about streams and recap
3:02:16 Code: Example of piping
3:25:38 Previous problem using piped streams
3:31:36 Reading docs on pipes
3:40:10 Why you shouldn't stream.pipe in production
3:44:35 Example with stream.pipeline
3:48:42 The stream.finished function for cleaning up
3:49:18 Discussion about readable/writable streams, duplexes and transformers
3:51:15 Implementing our own writable stream
4:34:00 Implementing our own readable stream
4:53:22 Duplex and transform streams
5:09:38 Creating an encryption and decryption application
5:48:35 Done with streams: Final notes, advice, stream-based NPM packages, etc.
thank you for this.
Thanks man
Thanks a lot my dear.
Have a great life
I'm really glad that I found your channel randomly. By watching all of your previous nodejs videos I feel much more confident working with nodejs ❤
after watching this great tutorial i feel like i'm the best nodeJS developer in the world hhhhhh thank you so much really such GREAT GREAT GREAT Tutorial
Simply the fact you backed yourself into a corner and then walked yourself and the viewer through a possible solution, instead of using a cookie-cutter example, places you above the rest. And this is leaving out the fact you're such a good pedagogue.
A video like this is an absolute TREASURE FOR LEARNING FUNDAMENTAL NODE CONCEPTS! Thank you very very much for creating helpful tutorials like this one. 😀
wow, a video last 6 hours, can't wait to enjoy it. Thank you so much
It's rarely the time when we get to learn so much on one topic from one video. Thank you so much man. I am eagerly waiting for your next video.
Loved this series so far. Would also like to see a condensed crash course on the same topic.
No words to describe this explanation. I already used streams, but having this understanding of how all works it's another level. Plus, if you are a full stack developer you can use this concepts in the front-end with web streams.
WOW! 6 hours on streams! amazing stuff, well explained. keep up the good work
Thanks, will do!
Hey you are doing great work there is no content like this on youtube...
It'll help a lot if continue this level content....
man you re a legend like you explained the hardest subject in node better than anyone I watched, going in depth on the underline implementation of streams thank you so much for the content
Man, mannnnn, maaannn!!!!! Seriously!! I have been looking for this for over a week and couldn't find thorough details about node stream. So much grateful for this 🙏🙏🙏🙏 Thank you
it took me sometime, but i finished!! I really love your advice of looking into the source code of libraries and of nodejs itself. since I started your series I feel i can read the nodejs docs more confidently, before they were obscure to me lol.
Erik Wendel has a bunch of videos with project with streams, but he explained superficially, for a slow learner like me, I was lost. Thanks to you i feel I understand streams now. I implemented the percentage thing of the encryption app. now I going to start your networking video and also watch the projects of erik wendel, this time, understanding what he is doing haha. thanks for the great content
Amazing course understanding node js fundamentals and going in depth is what i like about your courses unlike other node js courses they just into using Express js and make everything unclear and abstracted thanks for your efforts it really changed my pov in node and backend fundamentals and if anyone is wondering ,I highly recommand this course watch plus code along with him and take notes ,Cant wait for you next one .
Much appreciated!
a minor bug at 1:29:25 - breaking the loop before incrementing the counter. It will result in some duplicate numbers - 2915, 5645, etc. The counter should be incremented before breaking the loop
Thank you very much for mentioning this!
the most intensive nodejs stream course. thanks for sharing.
Thank you for your detailed explanation on what happens internally. I couldn't wrap my head around streams by just looking at the sample codes online.
Your voice is truly expressive and wonderful. It's very gentle and easy to absorb. A slight improvement would be to create engaging slides before diving into code demonstrations. That would be even better
Thanks Emma! I'll keep that in mind!
Stream has always been i topic that interests me the most. This will be great for revising my knowledge and learning smt new along the way!!
man i am excited to begin this journey, already watched all the previous videos, let's gooo
Keep it up buddy! Glad you’re learning so much!
I thank you very much for this tutorial. Glad that I found you channel. Learned a lot from you. Hope you are doing good and please please don't stop this series, waiting to learn more from you
37:24 you spin up 10_000_000 writes and prevent the buff from being freed as long as the write is not completed. This instantiation is a killer.
OMG WOW.! I have been trying to understand streams for so long, I have never came across such a detailed video on streams till date. YOU ARE A GEM. I really really appreciate what you are doing and request you to keep posting more videos when you get time. Love from India.!
You deserve to have a million subscriber lucky to find this video
you are awesome I never see any tutorail on internet using approch for learning like you use. Just go deeper and deeper.
The emphasize to use pipeline() instead of pipe() is very important.
More than a nodeJS tutorial.You are the best
Wow i just found a hidden diamond, im gonna watch all of this !!
Thank you so much ! I have learned a lot from this video about streams. And the way you explained each and every concepts is AMAZING !
Great videos, explained in a manner for all public.
thank you for your tutorials!! please continue making such high quality videos! really a treasure!
Wawww.... Worth waiting for a long time 👏👏👏
This is a long one. I am 3hr in. Very good content!
i'm really glad that see this video about nodejs stream
on 5:30:13, the 92 which is 5c in hexa is the result from the operation: 1116 % 256 = 92
Thank you for all your efforts explaining those concepts that seems hard to master at first glance. Your videos make the understanding of node core concepts much more easy and this is great quality content and plus, for free. You are thrully contributing to make the world better
You made a 6 hours+ video on stream!!! Thank you.
would love to see a video explaining how express builds upon node
Wow...kindly dnt make us wait for these type of awesome videos...kindly keep loading❤️❤️
Thank you for making this available to all for free. Your content is very high quality - far better than the Node.js content on paid platforms like Pluralsight. Do you have a link to donate directly without giving 30% to CZcams?
Thank you very much Joseph! I don't have a link for donations, but if you want, you could purchase the new version of the course using my link from the latest CZcams video. Udemy will give 97% of the revenue to the instructor if a purchase is made using the link. Otherwise, the instructor only gets 37%, so that would be a great support! Thanks again!
@@Cododev Just purchased it. It's the least I can do. I've learned a lot already! Very few instructors go as deep as you do on the material. You rock man. Looking forward to seeing what comes next.
wow... it's there, can't wait to watch it, thank you.
thank you so much for the tutorial. This helps me understand streams a lot.
It's a very important topic of CS.Great great great thank to you.
this is epic
i like your teaching style by giving examples and optimizing as max as possible
from where you learnt this much 😁
Aww thanks! Well just experience and a good deal amount of research and experimentation!
UTF-8 decoding a chunk only works if the characters are ascii characters ( 1 character = 1 byte ), anything else (example Chinese characters) uses more than 1 byte, so if you try to decode the chuck it can either throw an error or give you incorrect characters
That’s an excellent point! I’ll soon add a description in the pinned comment and point this out. I should’ve said every ASCII character is 1 byte, not every utf-8 character. Thanks a lot for mentioning this!
@@Cododev I've even noticed the Node documentation do this in their http request stream example where they immediately decode the chunks as utf8
Hii,
It seems you're understanding how numbers are being split.
I need a little bit to understand it. Can you help?
You're the best bro,
Thanks for this series.
Hats off brother.
God bless you
bro lots of love for this.
During the "Implementing our own writable stream" section at 3:51:15, can some one verify that when subclassing the Writable class, the Writable class provides us an internal buffer and also manages it for us too. And that we don't actually have to implement/manage our own internal buffer in the subclass like we did there with the chunks array. tysm.
Thank you, your course is extremely helpful!
This video is pure gold!
OMG. What a u doing bro. this is just perfect. Greetings from Moscow
You deserve way more subscribers
5:07:30 If your duplex.write is not writing to the file, make sure to add 'duplex.end(Buffer.from...) for the last line, to trigger the _final method. Happy coding
Thanks for the new year present!
Wow, another amazing video.
Many thanks❤
👍 Really helpful please continue making this type of content
Hi , @ 04:57:40 you suggest that in case of transform stream , data chunks move from readable internal buffer to writable internal buffer. I think it should be the other way around , we write chunk to the writable internal buffer which then goes through the transform function where we push the transformed chunk to the readable internal buffer from where it can be consumed.
That is an excellent point, thanks a lot for mentioning it! I’ll soon do an update and fix this, thanks again!
@@Cododev All thanks to you man for your efforts. Great videos 👍
You are My God in human shape.
Could you please restructure the course, create additional lecture slides, and publish them on Udemy? I have many questions to ask you, but on CZcams, it's difficult to engage in a Q&A session because the platform lacks support for such interactions
Hi there! That’s exactly what I’m doing right now. Unfortunately CZcams is not the best platform to publish a 45h+ course on. At the end of the course, I’m also planning to give the students a massive real-world project with tens of thousands of lines of code and learn how to deploy that and see how every single part works together in production, and doing that is practically impossible on CZcams. I have completed the next section of the course, but I’m now working on restructuring the course and moving it to a platform that’s dedicated to education and much more suitable for long-length courses. And you’re absolutely right, the CZcams comments section is a pain for having excellent Q&A support. I will soon create a post and talk more about this. Thanks!
@@Cododev Dear Cododev, I felt incredibly happy when I discovered your lectures, and I admire your expertise in the world of Node.js. If you plan to publish a course on Udemy, possibly titled 'Understanding How Node.js Core Works in Depth,' I hope you'll take care to create beautifully designed lecture slides and provide detailed content. It would be great if after each lecture video, you could include a coding demonstration video to illustrate the concepts. If necessary, you can use the WhiteBoard application on Windows or a similar software on macOS for for additional visual aids.
I prefer dark-themed slides over bright ones, as dark themes are more suitable for programmers, especially those with nearsightedness. If you feel that the course would benefit from teaching some foundational knowledge about C++ concurrency, C++ networking, and so on, I would be delighted if you could include those topics before proceeding with later parts of the course.
Thank you!
awesome explaination. GREAT WORK! Thank you :)
2:24:00
If im not mistaken, there is a possibility that last number in file can be split into two separate chunks, so it is not certain that last item in last array is 'expected string number'?
Hey there!
I was wondering if you could consider including your course in the Udemy subscription plan. It would be fantastic to have access to your content through such a convenient platform. Thanks for considering it!
Yes I’m looking forward to doing that, hopefully it will happen sometime soon if Udemy doesn’t require me to remove all the Node.js videos from the channel.
One more Nodejs Core Concept ❤
Great job! Just bought your udemy course, keep it up! I´m wondering wht´s the name of the vscode theme in the video?
thank you very mush for all the information you shared with us ,it really helpful 🙏🙏❤
Great stuff!
Thanks so much for providing in depth explanation. ❤
3:32:54 Is it good to use timeEnd in writeStream.on("finish") instead of readStream.on("end") ?
why on 4:32:16, number of drains is 480 while number of writes is 481? This hasn't account for the last write which flushed the remaining data
I am Windows user, with ryzen 5600U. When i tried to run the "splitting chunk 2:38:00 " code with 10^9 size file, memory and Cpu work as it is, but disk r/w become too low, may be around 10-20MB. But in your code whenever you execute that code, your disk r/w go up to 1GB. One more thing is, if i perform stream write() operation with chunk, outside the foreach loop, disk r/w become high up to 800MB. Please help, me how to increase this disk w/r speed. This 10^9 file took at least 39minit to add all the even number in "dest.txt" file in my system.
Very important topic, thx!
41:50 Having more memory is not bad but for an operation that will happen only once or twice it's waste of resources. That's one good way to think about this imo.
53:17 First
01:20:32 second
1:31:37 third
2:50:20 fourth
Hi, a very good video. can you tell me the font name in your code editor? It looks pretty.
WOOOOOOOOOOOW , thx for your efforts and time ♥♥
Another great video! Thank you!
Hi question for readStream @2:40:00 , for the split issue. We are able to solve the issue where first number and last number of chunk are getting messed when we know the chunk array (all elements of chunk are in increasing order) but if we dont know the content of readStream chunk , how can we resolve this issue.
Hi there! Well that's why it's important to understand the underlying format of the file you're dealing with! So suppose instead of a text file, it was an image file, then you would need to know how the pixels and colours are represented in binary so that you could do some modifications like applying filters, cropping, resizing, etc.
In this example we made use of the fact that numbers are in ascending order, but honestly, for each problem you will need a specific solution that works best for that. So if you could maybe narrow your question down a bit, I might be able to explain better what to do in that specific situation. Thanks!
a chunk just support 65536 bytes. if it has just 2 bytes lefting to full it, and the last string have 5 bytes, then the string will be splitted, two bytes would left behing to the previous chunk and the another three bytes will to the next chunk.
Then, to solve you problem, you can see the full chunk to see the content of the chunk and then do some action based on what you need.
see the full chunk content logging:
console.log('------------staring chunk------------')
console.log(chunk.toString())
console.log('------------ending chunk------------')
@@Cododev Hi thanks for the reply.. I'm not a developer or programmer..its just a hobby so I might not be able to frame question very accurately.
Suppose I'm reading a large mkv file from server how can I write it into another file without messing the data.is it possible without using pipe.Am I suppose to know the mkv formate
One more question other than this😅. In readstream the default highwaterMark is 64 KiB and for the writeStream its 16KiB..my question is when data event is fired the Read stream has 64KiB data but write Stream can only take upto 16Kib before it needs to drain so does the 48KiB data rest into memory or how does it play out..does write stream writes into destination file in 3 chunks(16* 3 = 48) or all the 48KiB data gets written at once before reading another chunk.
@@gabrielmachado5391 thanks for the input gabriel..but if we don't know the underlying format how can we spot where the error has occurred that we need to fix by adding the last byte of previous chunk with the first byte of next chunk
@@RaGa_BABA in node docs, they explain that chunk could be either a buffer or a string.
otherwise, to catch errors you can use the 'error' event.
descripition of this event, according the docs:
"The 'error' event may be emitted by a Readable implementation at any time. Typically, this may occur if the underlying stream is unable to generate data due to an underlying internal failure, or when a stream implementation attempts to push an invalid chunk of data.
The listener callback will be passed a single Error object."
Could you please create more videos in nodejs
4:32:36 How come you didn't have to call stream.end when you used your own implementation of a writable stream? also why is there +2 more writes than drains? doesn't it call drain every time you write?
man great content, congratulations!
Please i can't find the video about buffers
Another approach to solve the 'drain event' problem is to use a generator. It will let us avoid keeping the "i" variable in the closure and you can safely break the loop. Next time you run it, it'll continue from where the generator yielded the last value bc it can save its state.
const fs = require('fs').promises;
function* bufferGenerator() {
for (let i = 0; i < 1000000; i++) {
yield Buffer.from(` ${i} `, 'utf-8');
}
}
async function writeToFile() {
console.time('write many');
const fileHandle = await fs.open('./text.txt', 'w');
const stream = fileHandle.createWriteStream();
const generator = bufferGenerator();
const writeMany = () => {
let result = generator.next();
while (!result.done) {
if (!stream.write(result.value)) {
break;
}
result = generator.next();
}
if (result.done) {
stream.end();
}
};
writeMany();
stream.on('drain', writeMany);
stream.on('finish', () => {
console.timeEnd('write many');
fileHandle.close();
});
}
writeToFile().catch(console.error);
1:22:46
You said Buffer.alloc(16383, 10) doesn't mean we're writing 10 to the file and it was character "10" then also it would mean the same thing but in case of character it's writing to the file.
Just commenting to get you more reach, upload more content, thank you
Really thanks for your video~❤
What is the name of the background music?
Starting it on february 12.
did you finish?
Just finished it on february 22, now it's time to start practicing and getting better :D
oloco dog
@@gabrielmachado5391 Leia a documentação brother, faça muitos testes mirabolantes que uma hora você fica craque. Faz até de olho fechado kkkk
great stuff, a question though @01:22:04 , if we allocate internal stream lets say 16383 bytes (writablelength = 16383) and then we allocate same stream a single byte but with setTimeout , the writableLenght is just one byte.
for example:
stream.write(Buffer.alloc(16383, "a"));
console.log(stream.writableLength); // 16383
setTimeout(function () {
stream.write(Buffer.alloc(1, "b"));
console.log(stream.writableLength); // 1 ---why?
}, 1000);
Well, it could be because this example is not really considered an example of repeated writes. We are writing once to the file (assuming it is a file stream), and then there's some time for the stream to flush the data to the underlying resource because of that setTimeout.
So after the first write, Node sees the setTimeout, and then the main execution thread is done. And after a little while, the main execution thread will kick in again and handle the second write (learn about the event loop please if this doesn't make sense). I'm not sure but I think when we are going to the next tick (stopping the execution thread and then starting it again), Node streams will do an automatic flushing and draining so that the write is now done and the internal buffer is empty again. Let me look more into this and I'll let you know if I find more info about it! Thanks for the question!
@@Cododev yeahhh..it makes sense.. thankyou for the detailed answer.
Will you do an async video? (promises, callback, Promise class..., How it works...)
Hi I have a doubt. Please look at the below code.
const Stream = require("stream");
const readableStream = new Stream.Readable({
highWaterMark: 200,
read() {},
});
readableStream.push("H");
readableStream.on("data", (chunk) => {
console.log(chunk.toString());
});
In above code, I mentioned the internal buffer size as 200. I pushed only the letter H onto the stream.
According to the tutorial at 1:02:14, data event is raised only when internal buffer is filled. But the above code is printing the letter H(meaning - data event is raised before buffer is filled).
Could someone explain me please?
You're awesome bro
I'm not finding the course on nodejs process
I'm encountering an issue with a write-many.js. When I write a series of simple indexes ranging from 0 to a set number of writes, I consistently find two instances of the number 103978. I checked every number while writing even numbers from a file named src.txt, and discovered that the source of the problem is within `src.txt’ itself, which contains two identical numbers (103978). What could be causing this duplication?
Thank you bro❤
Bro, you're the best 👌
THANK YOU!
Where can i found more information? is there a roadmap?
At 24:33 can anyone explain why it’s taking 700mb of memory?
Reay to go to next video. Any idea when its coming?
Please keep making videos