Building an upload backend service with No libraries, No forms, just vanilla JS with progress)
Vložit
- čas přidán 2. 06. 2024
- OK, I lied I used the HTTP and the fs library on node js but nothing on the backend but nothing but the built-in functions on the browser.
In this video, I’ll build a simple progressor upload service and explain how it works. Let us have fun.
Source Code
github.com/hnasr/javascript_p...
Resources
nodejs.org/api/http.html#http...
developer.mozilla.org/en-US/d...
developer.mozilla.org/en-US/d...
0:00 Intro
3:00 Preparing Frontend/Backend
7:00 Reading file with FileRead in Frontend
14:00 Slicing the file into Chunks
19:00 Sending Chunks through fetch to Backend
23:00 Received Chunks on Backend
28:00 Progress on Frontend
30:00 Increase chunk size
31:00 Possible Improvement and Scalability
'
Become a Member on CZcams
/ @hnasr
🔥 Members Only Content
• Members-only videos
Support my work on PayPal
bit.ly/33ENps4
🧑🏫 Courses I Teach
husseinnasser.com/courses
🏭 Backend Engineering Videos in Order
backend.husseinnasser.com
💾 Database Engineering Videos
• Database Engineering
🎙️Listen to the Backend Engineering Podcast
husseinnasser.com/podcast
Gears and tools used on the Channel (affiliates)
🖼️ Slides and Thumbnail Design
Canva
partner.canva.com/c/2766475/6...
🎙️ Mic Gear
Shure SM7B Cardioid Dynamic Microphone
amzn.to/3o1NiBi
Cloudlifter
amzn.to/2RAeyLo
XLR cables
amzn.to/3tvMJRu
Focusrite Audio Interface
amzn.to/3f2vjGY
📷 Camera Gear
Canon M50 Mark II
amzn.to/3o2ed0c
Micro HDMI to HDMI
amzn.to/3uwCxK3
Video capture card
amzn.to/3f34pyD
AC Wall for constant power
amzn.to/3eueoxP
Stay Awesome,
Hussein - Věda a technologie
Please never stop doing this "from scratch" videos! They are super helpful and I love learning more about this from your videos.
Well he stopped
never judge a book by its cover!
the first moment I saw the title of the video I felt disappointed🤦🏽♂️ as you are known by the low-level guy (at least for me) and I thought you gonna do the same video that everyone on the internet record right now 😀. to be honest I totally regreted that feeling after 2 or 3 minutes 😀.
thank you. this is an eye opener.
I would suggest that you keep building on top of this and make another video for building the distributed stateless upload service. and from the frontend side you can utilize some DS concepts like queue to receive the chunks and release/send them in a controlled manner.
it could also be nice to explain the similarities between this approach and the native multipart/form-data.
again, thanks a million for all of your amazing content. 😍
😂 😂 same fealing when i read the title
Actually I thought the exact opposite, this is one of Hussain's best videos! Usually he tends not to go into as much depth as I'd like to see. Here he builds the whole example, and equally importantly covers its deficiencies and options for improving and scaling it.
@@MadAndy24 I also think it is one of the best videos, but what I am saying is, it wasn't my first thought when reading the title only.
You channel is a goldmine, thanks!
This is nice continuity from node http and node fetch! Thank you, I learned a lot from your contents
OMG😱, This is amazing, loving your content Hussain
I always use form data to upload a file, but this way is more scalable and give you more control. I will wait more videos like this one.
This is a really valuable video. Not just because you show how to build an upload service with vanilla JS but also because of the last 10 mins where you explain how one can take it forward. Thanks Hussein!
I started learning api from express ignoring the core nodejs modules. Now i see how important this was for me to learn all these. Thanks a lot for this video.
Great stuff! Something that I've been looking at, but with multiple file upload and directories. I like the commentary about scalability, though a database seems like overkill
Thanks! Amazing video... I really enjoyed to the core! Gave me insight and motivation to learn more :)
Awesome video, we need more content like this. If you think that is a good idea i'll love to see another video doing this in the "right way" or at least the way you explain in the ending of the video!!!
I was waiting for this video... Thank you
You have no idea how helpful it is that you mention "this is a custom header". So many guys forget to identify built-in/belonging-to-some-api-specification vs custom in many respects of programming
i was not previously interested in node.js but seeing Hussein videos - i am starting to catch a glimpse...
Many of my friends complains that your videos are too lengthy and you speak too slowly, but I always say to them that your videos are like watching movies which are full of knowledge and fun.
Hands down one of the best (if not the best) channels on youtube
Amazing video. This video would have saved me a lot of time of searching a year ago
you make such good content!
but I swear, the spaces in your html attributes trigger my fight or flight!
keep up the great work 👍
Great video thanks so much would love more videos like this
This is absolutely amazing
This one is too funny and knowledgeable. I love it.
right? I was watching this with headphones inside a college building. People probably thought I was crazy laughing at a coding video
Too late for me to watch rn. Putting in my watch queue for later. Thanks for awesome content!
This is amazing, could you please show how multipart-form work?
Thank you for your content
Good question, I suppose I should have started with that☺️
A beautiful guide
I'm lucky I subscribed to your channel
My friend
this was so fun
You videos are awesome. One question, you are sending res back even before chunks are actually appended to the file. Is it fine?
Awesome ! Just AWESOME
Hello Hussein,
Thank you for the content. Amazing as always.
Is the server able to make any other operations while writing each chunk to the file?
Fantastic question ! In my code no since I am using AppendSync which uses the main thread. Using Append (async) will use the thread pool and the server will be free to serve other requests/do other tasks.
We are obviously talking about nodejs here. Other runtimes/languages are completely different story.
@@hnasr Thank you for the response Hussein.
Just out of curiosity, is it safe to use Append (async) though? Was considering what's going to happen if for any reason the chunks are getting processed in a different order.
@@yannistheodorakopoulos5916 if you really want to upload large files frequently, then its better to scale your uploading services as he recommend at the end of video.
beautiful stuff
Great video 👍.
I have a question is it guarantee that chunks will arrive in the same order that were sent since you are appending them in the backend. In a sense this could lead to corrupted file if they didn't arrive in the same order
In this particular example I am sending them in order and serially. I don’t send a chunk unless I get a response from the server that the previous chunk arrived. (The await in the loop)
If I started sending chunks in parallel them we are the mercy of both the frontend queuing and the backend queue. That is where we will need each chunk to have a unique and sequential id at layer 7 so we can assemble it
This is really a valuable video and you made it in a way that anyone can take this and implement their own upload service logic of any scale.... I love that .... There is one question i have though about the resumability part that we can an endpoint in backend right to check where we left off like you mentioned that chunk id from that we can send that chunk id to frontend and continue upload from that can we not? or maybe i am missing something please do tell me
Thank you for your content
brilliant, thank you sir
Why chunk at all if TCP itself chunks?
Valid question. If we were using raw TCP directly, the os would have broke the data into TCP segments each based on the window size.
However, we are using HTTP which does use TCP except we don’t have that low level control. The proxies and servers have all sorts of protection at HTTP layer to timeout large requests.
@@hnasr Thanks. Is `multipart/form-data` somehow immune from this kind of protection?
Very informative
Thanks for the great video! One question, how would you decide whether to use client-side upload to S3 or server-side upload S3? I'm thinking that at a smaller team, do we really need a upload service or we can just get the presigned url and upload to S3 directly from client side.
if you want to upload from client to s3, then its needed to put s3 config in the client. I am not sure its good idea.
@hai doan
Using presigned url, You don't have to put any secret info on the client. The server shouldn't be involved in the upload process at all. Only store the url in db.
@@davidnaiz2417 thanks for your info, btw uploading fron backend will be good for scale, especially uploading large image by chunk.
const chunkCount = Math.ceil(ev.target.result.byteLength/CHUNK_SIZE) // That rounds a number up to the next largest integer, by example 3001/1000 -> 3.001 -> 4
Great content....
Beautiful little example. You wonder why these open source upload libraries are several MEGABYTES of code and that’s just client code.
This is a really awesome tutorial.
But nowadays we generally upload these files to cloud storage (like Google files storage, AWS file storage). In then we upload in one request or do me do multiple requests.
Is there any good article to go though it
Cool, thanks!
This was an amazing video, it's helps me a lot. could you please show how to upload more that 2 GB file?
how can i get more information about this topic, transfer bytes via network?
I love it
Superb 📁📁📁
can u make a video on downloading files from server to client
nice video
is this what an html form does when you upload a file ?
Great content!!
Do you have a guess for why onprogress event doesn’t let us access the bytes it read?
I mean it’s already reading it in chunks, and using the onload event and chunking it afterwards I really feel stupid and wasteful.
Really can’t tell why, and I’m curious for your guess on the matter
And again - really really great vid. Good developers will appreciate all this vanilla stuff that is going on here.
It's most likely a security thing
Was thinking about that also and I suspect that it would be more troublesome if it would behave like that. Remember FileReader can prompt an error (description from moz):
"The error event is fired when the read failed due to an error (for example, because the file was not found or not readable)" and now imaging reading and streaming an invalid file (which you don't know its invalid) to the server just to notice that at 98% there is an issue. Now you would have to code a logic to remove all the data from the server and you would waste the connection, transfer and space.
It's simpler for the browser to check and validate the file from the user on their own device *locally* and when it's valid FIleReader gives a code or developer a green light to work with the file.
how would you do it so that you don't have to load the entire file into memory first? save a fixed size into the array buffer, upload them all, then replace the array buffer with the next bytes of the file, and so on...?
1- Speaking of scalable backend, how does backend knows if the upload ended if you can just send the next request to another server?
2- Why does completety different fetch request handled by single req.on(“data”) on backend? I mean How does backend know those are the part of the same file ?
Is there any possibility that chucks are received in different order ??
More like this please ! Thank you
Best video on file upload?
Hello Hussein
Can you please make a detailed video on how to process 2gb file on 1 gb ram etc
Also if possible you can make paid courses for system design from absolute basics to Hussein Nasser level
How do you account for this in a real life application having multiple users assuming they would be duplicate names thereby causing chunks get attached to different data..how would you handle creating unique names in the backend or front
In order to read file in the chunk, instead of passing theFile blob, use slice on the theFile and then read it via filereader.
```
const f_chunk = theFile.slice(chunkId*CHUNK_SIZE, (chunkId+2)*CHUNK_SIZR)
const filereader = new FileReader ();
filereader.onload = ev = { sendChunkToServer(ev.target.result)}
filereader.readAsArrayBuffer(f_chunk);
```
26:26 why you are blocking the main thread with read Sync ?
Hi, Hussein. What do you think about gun.js?
Nice tutorial... But I think you should take it one step further to implement what u Suggest at the end
Idk why, but for me the image doesnt display in vscode. Says it's in some unsupported format
Awesome
I search how to do this without libraries for hours. Thanks!
I haven't watch the whole video yet but im building in this type of function for a clientt but im wondering if this will this be secure enought so that my client wont be hacked or get in virus on his computer true the file. 🤔
what if the chunk count of chunk value is in float?
lol the Japanese making it hit different 😁
amazing content. does anyone know other resources like this where we get to see things from scratch and not very advance to keep up with
i'm uploading to nowhere. the upload completes but doesn't appear anywhere.
at the file name creation, you could've used the new Date() instead of Math.random().
I developed same solution for a company as a scalable & resumable file upload service, they didnt give a shit.
browsers dont use require how tf can i do a file upload
I think your code is vulnerable to the directory traversal attack.
Solid stuff from scratch 🔥
Aside from this content I have interest in learning about postgres write ahead log and change data capture stuff and publishing changes to message broker e.g. rabbitmq
Your chunk count formula is wrong. It has the potential to send one empty chunk at the end.
Yup, rather than just adding one, you should round the chunk count up.
@@MadAndy24 or add chunk_size - 1 before dividing
Better way for unique name is append a timestamp
Js is just script for form validation not language but standing its nature.