How I load lots of data in my React application

In my React application I use infinite scrolling instead of pagination to show users lots of data and in this video I show how exactly I do that, first giving a brief overview of the difference between two techniques and then demonstrating my React frontend and my ExpressJS/MongoDB backend.
UPDATE: In the video I say "virtualisation" but that is not correct term, since probably more appropriate one is infinite scrolling or lazy loading, or something along that line.
You can reach out:
Discord: / discord

[@memeproductions4182]

Infinite scrolling even if used together with virtualization still has its limits, for example if you were to scroll to the end of a very large list you would still have all the elements in memory, if they are a lot they could even lead to the application crashing

[@LifeLoveAndMonads]

I agree, thats a very good take. Since virtualisation takes care of elements unmounting, but you are correct that, especially in the shown example, the data itself will be in memory. But I guess when you are measurably can end up in that situation you would also implement data off-loading.

[@memeproductions4182]

​@@LifeLoveAndMonadsyeah, I don't envy the guy who would have to implement that though lol

[@LifeLoveAndMonads]

I honestly don't think that it is THAT hard, but surely there are plenty of nuances and edge cases. But that is true for any serious application out there. But since most places where virtualisation is present are rather big companies: Discord, Facebook, YT, etc, they will find a bunch of smart people to solve those issues :)

[@jamjam3448]

What if you use intersection observer to unmount when it's not in view.

[@LifeLoveAndMonads]

you can use intersection observer to unmount things that are not in the view, I believe, but it was more about making sure you do not build too much data in memory I guess, but you can certainly figure something out, like I have said, I do not think it is that much of a problem, as long as you know exact requirements and clear formulation of a problem

[@anj000]

For most cases do not use infinite scrolling.
Couple of years ago there was a trend to use it everywhere because it was rather new and fancy, but it is pain in the as for the users. And it can cause accessibility, seo and performance issues.

[@LifeLoveAndMonads]

Valuable points!

[@ManuelBorges1979]

I liked the video. Good job. 👏🏼 I subscribed.

[@LifeLoveAndMonads]

Thank you!

[@ehza]

Nice. Keep it up.

[@ShighetariVlogs]

I like your content thank you for sharing 🎉

[@LifeLoveAndMonads]

Thank you for watching :)

[@Alex.Shalda]

awesome video

[@LifeLoveAndMonads]

Thanks mate!

[@sthernito]

This is a really interesting topic and great to see how to implement Virtualization, but at the end it depends entirely in the project, there will be cases where pagination is a better experience for the users and cases where Virtualization could be, just remember the rule, you are not your user.

[@arshiagholami7611]

use an index 🎉

[@federicoreina7732]

If you have enough data in your db (and supposing the user scrolls a lot) I think your performance will degrade over time as the DB needs to traverse all the elements you are skipping. For this case having a search token that keeps account of the search position is a possible solution

[@LifeLoveAndMonads]

Thanks for the advice!

[@ducklingzpl46u36]

Can you make more performance videos for react. Would love you. Already subscribed.

[@LifeLoveAndMonads]

Thanks mate! In full honesty I would not consider this current video has anything to do with performance, only from the perspective that we do not load all data at once and exploring, sort, two path we can take to achieve that. So no promises, but hopefully it will be more videos to like :)

[@VuNguyen-yj5wi]

How about a case when you scroll down to get next data for next page from server, but there is a some records whichs were inserted before you scroll down, therefore the datas for next page will include records whichs were loaded before

[@LifeLoveAndMonads]

Thats exactly what happened behind the scenes of this video :) But because I think something was every now and then happening twice, so I have handled that in my "merge" data function. But you can always find more sophisticated way. And of course you can try to make sure that each page contains unique data. I can see that sometimes its not up to you, since for example while you were requesting for data, new items were added and concept of "pages" shifted. In that case you can try to do something like firebase does, when they return pages they are sort of like linked lists of chunks of data, since data should be ordered on the backend before pagination and pagination itself has to be a bit more complicated that just skip/limit.

[@teujorge]

use a "cursor"...
say you fetch 10 items. you get the ID of the last item and that becomes your cursor. your next fetch will be the next 10 items after the cursor. you may also need to sort your data...

[@twitchizle]

Just get the next 10 record after the id of last record.

[@FrontendNerd-lg3oh]

Video is neat

[@LifeLoveAndMonads]

Thank you!

[@rawallon]

No, you're neat

[@Rodrig79863]

Clicked cause of the thumbnail 👍

[@LifeLoveAndMonads]

We call it psychology.

[@karansharma-fv2dd]

Can anyone please help how to retain CTRL + F functionality of the browser as a normal user you would expect that functionality to work but due to this. it will break.

[@ChristopherJamesCalo]

There's no way to Ctrl F over content that is not in the DOM. But you could try building a filtering search box that searches over all records, whether they be in memory (but not in the DOM) or on the server. Then what you do to update the list in the DOM is up to you.

[@dgcp354]

add an event on button push for CTRL that focus on search input

[@Mashwishi]

Is this good for like social media web apps ?

[@LifeLoveAndMonads]

I think it is good for any case where you don't want pagination for some reason - Messenger, Telegram or Discord chat, social media apps like Facebook, Reddit, etc.

[@jobasti]

Why is this called "virtualization" and not doom-scrolling, endless-scrolling or lazy-loading? Virtualization is a totally different thing and well known since decades .... i dont get it ... please help.

[@tedchirvasiu]

Everything needs a fancy name to stay relevant.

[@jobasti]

@@tedchirvasiu Well if Doom-Scrolling isnt fancy i dont know what is xD

[@LifeLoveAndMonads]

never heard that term! I will call it Doom-scrolling from now on :)

[@jobasti]

@@LifeLoveAndMonads Facebok, Twitter and almost every "community" site uses that since forever. :)

[@titbarros]

Virtualization is not necessary infinite. You might be virtualizing a well known number of items. And is not necessarily scroll either. Think diagrams.

[@user-qr4jf4tv2x]

i like to put pagination on an api. load everything needed then slice it up and store it in s3

[@backupmemories897]

let me introduce you to virtual scrolling xD you can render 10m list and it wont lag because it only gonna show u list that fits on the container. in any direction.. if u look at dom u gonna see the list are being deleted and inserted dynamically.. instead of 10m divs on the inspect element u gonna only see maybe 20-30 items depends on your item height.. and how much of that fits on the container.

[@blackwatch2150]

Ohh, means virtualization won't affect performance.

[@jma42]

basically that is still in some way, a form of caching, which is another can full of worms

[@LifeLoveAndMonads]

In full honesty, I do not see a connection with caching, but implementing a proper list virtualisation can be not super easy. In my case and in the video its kinda a very simple case.

[@aer1th621]

can i have github repos?

[@LifeLoveAndMonads]

sure, github.com/PompolutZ/wuclub_monorepo

[@towu]

Nice stuff! Do you have a discord server?

[@LifeLoveAndMonads]

Not yet, never had a reason to have one :) And I have never used Twitter :)
UPD: Now I have - discord.gg/cs9uKmCZ

[@IvanAndrieiev]

[@LifeLoveAndMonads]

Thank you!