self-host the INTERNET! (before it's too late)
Vložit
- čas přidán 17. 05. 2024
- Ugreen NASync Private Cloud Storage Solution： bit.ly/3IRQY3T
40% OFF on (Mar 26th 8: 00 am PDT-28th 8：00 am PDT)-48Hours
40% OFF on (Mar 26th 11:00 am EDT-28th 11:00 am EDT)-48Hours
Up to 35% OFF during 3/28-5/9
People who pays a $5 deposit on the pre-sale will receive 40% off from 3/26-5/9
Using a few tools, we can self-host much of the important data the internet has to offer.
RESOURCES
Docker Guide: • WATCH THIS to get star...
Kiwix Server: github.com/kiwix/kiwix-tools/...
ZIM Files: library.kiwix.org/#lang=eng
OpenStreetMaps Tile Server: github.com/fsievers/openstree...
OpenStreetMaps Data: download.geofabrik.de/
Docker Volume Tip: forums.docker.com/t/how-to-ma...
👏SUPPORT TECHHUT
CZcams MEMBER: bit.ly/members-techhut
TECHHUT.TV: bit.ly/techhut-join
DONATE PAYPAL: bit.ly/paypal-techhut
$100 LINODE CREDIT: bit.ly/linode-techhut
$5 PBCWAY CREDIT: bit.ly/techhut-join
🏆FOLOW TECHHUT
JOIN DISCORD: bit.ly/discord-techhut
TWITTER: bit.ly/twitter-techhut
MASTODON: bit.ly/mastodon-techhut
INSTAGRAM: bit.ly/personal-insta
📷MY GEAR (PAID LINKS)
ASUS ROG M16: amzn.to/3t8Xgpo
DeepCool MATREXX 40: amzn.to/3q3K8Qn
AMD 3700x: amzn.to/31AKX9N
ASRock B550M: amzn.to/3qeymTv
G.Skill Trident Z Neo: amzn.to/3JRbeSF
Radeon RX 580: amzn.to/3n7Ax9g
Cannon M50: amzn.to/3xLfhuA
00:00 - WHY
01:15 - UGREEN NAS (Sponsor)
03:00 - prerequisites
03:58 - Network Share
05:02 - Kiwix
06:50 - Kiwis Server Setup
09:28 - OpenStreetMaps Server Setup
14:53 - ty - Věda a technologie
Just think, a AAA modern game is over 100 gigs and yet the entire Wikipedia is only 109GB!!!
thats because wikipedia is mostly text, and games are mostly binaries, textures, audio, amongst others.
That's because: Textures are pretty damn high-res nowadays, and there are several layers instead of just having a single texture (like back in 2002-ish).
You've got diffuse textures, specular textures, normal maps (bump mapping), emissive textures, ambient occlusion maps, detail textures, reflection and refraction maps, subsurface scattering maps, metalness maps. So take the size of a single texture and 10x it to cover all the different types of extra textures needed for special effects.
THAT is why games are 100+GB. Binaries (.dll and .exe files), audio, map data and others fade into naught compared to textures.
I mean famous Titanfall 's 35GB of uncompressed audio are thing.
What else you want? Maybe remove gigabytes of unused resources and locations? Or beta test gameplay?
Workers of multibillion corporation have much more important things to do, like triple check working on in game shop or implement new gatcha.
Trash takes up less space than I thought.
But 100gb are "maxi" with media, text only are far less
It's getting more important for people to collectively start saving as much information as possible, in bulk, than ever before. I very recently just had a look at my CZcams playlists and rediscovered one I made during Covid times called "important historical context" which would have been filled with videos of certain political developments. ALL of the videos in that playlist were removed from CZcams. All of them. The scary thing is, while we now have the "easiest" way to access information, the ministry of truth also now has the easiest way to curate and re-write history.
Very smart
The winners rewrite history.
I love that reference
It’s insane that I can not find a fraction of the documentaries that I used to watch on controversial topics anymore. I wish more people were aware that this type of thing is happening.
@@jayman94fly the losers can also rewrite history to pretend like they won does that make them the winners?
MOM?! WHERE DID YOU PUT MY INTERNET?!
Your sister took it
"I uh, put it away"
You don't need to get Internet, we have Internet at home
(internet at home = 56k dial up, or even better, an old timey tv arial on the roof)@@Honeybearsphone
I printed it, and then sold the gear
Next project fitting the entire internet on a 1.5 TB sandisk mini SD card.
All human knowledge in my thumbnail
We would need hella compression for that since the internet is many zettabytes big
Only the most essential of the internet, like Wikipedia,dictionary, and world time software stuff like that , compress it so it fits.
sure. backup entire WWW onto a smart pregnancy test. what can possibly go wrong.
@teknastyk Don't be a dummy. I mean the essentials.
You forgot the most important piece of the internet!
Stack Overflow.
real
Just localize gpt 4 turbo bro.
@@mahpell7173 you can do that?
@mahpell7173 Your pfp in junction with your comment makes it funnier than it would otherwise be because it looks like you're snickering at your own joke.
Oh, nm. My mind went a complete different direction.
Literally what the guy in half life Alyx did
"You downloaded the Internet?!"
"Well, most of it anyway"
Still gotta play this game, hopefully i'll get a better gpu soon
@@antonshotpoteita Alyx is a good optimised game actually, I've ran it on my 1050 ti, it was stable 30 fps, but there may be places where it will drop
@@stehouse I played Alex on 1650 Super and in some moments the FPS was really good, but in some moments, due to the low FPS, I started to feel sick, although I had never felt sick in VR before.
@@antonshotpoteita Absolutely worth it.
This video, like many others, created an overwhelming feeling along the lines of, "oh my gosh of course, he's right. I need to save at LEAST the survival information, then everything else in rank of importance". Something like that. It urges me to start prepping again. Which honestly is probably something like trying to create a safety to cope with the anxiety. But you know what, friends, I realized that even if SHTF as big and as hard as some of us prep for, I'm okay with probably dying. I mean I would give it my all, there certainly would be a lot of stress and pain. And in that scenario, most preppers would say going into it unprepared will likely spell demise. And they're probably right. But that's okay.
I recently was blessed to be able to somehow manage to put money down and start paying mortgage on a house (escape the rent trap am I right?) and over several months of living @ the new crib I realized it really is a house of cards. Between the chaos in government, ambitions of larry fink and the 1%er's, natural disasters, changes in economy, being stretched thin financially, almost getting laid off at work, fire, flood, powerful HOA's (mine is pretty chill TG), family stresses etc etc, you quickly realize that literally anything and everything can take you down and lose you everything you've ever worked for.
Life can wipe you out in more ways than you can imagine. We can certainly insure ourselves against common things and we do, but we can never prep for it all. Even if we had the most hunk of a bunker we could walk outside tomorrow, and tree branch falls on us and we get the speed nap. Make your home a home, that is, make it comfortable. But damn we've got to live our lives and not create suffering for ourselves by worrying about whether or not we have the head knowledge to find and make kindling in apocalyptic urban setting #2.
~an exaggeration that goes way beyond what this video even touches just to make a point written by a guy that will probably die someday.
sorry to be so serious but hoping this posture helps someone like it helps me.
Well said..
you know what, i appreciate the honesty
The first thing is make sure you have electricity to power your self-hosted internet if world, national or local disaster occurs.
I feel the same, except I have kids, so I have to keep them alive.
bro its such a mental hurdle to compile everything needed and back it up twice. but having a llm suite to answer questions post event would be too valuable.
Me: "Mom, can we have Internet?"
Mom: "We have Internet at home"
Internet at home:
Coming home after a long day at school to relax with some wikipedia articles.
dude, this is such a good idea!! you should expand on this.
come up with a whole doomsday essentials pack - not just wiki, but also books, etc. key resources to survive and thrive
Would love to get my hands on a printed Encylcopaedia Britannica from say 1970 and a bunch of yearly updates.
The Next Step on Prepperism.
It already exists, it is called a Cyberdeck.
@@Storin_of_Kel But cyberdecks are mostly done for the challenge and the cyberpunk aesthetic. This has a more serious orientation to it.
I kinda suspect that in a doomsday scenario, being able to reliably power on any of those devices would be quite a difficult problem. If it's a man made disaster, tech infrastructure and power delivery would be an immediate target. If it's a natural disaster, I could sorta see stuff working? Idk, solar power and a whole lot of low energy devices can help.
what about out of copyright books? project gutenberg type stuff? Store an entire library of stuff to read, that would be awesome
I've been doing a lot of archiving old books over the past six months- mainly on first and second hand accounts of the Civil War. Interesting stuff.
Internet Archive and Project Gutenberg both have a ton, also PDFdrive is your friend, but you can download a lot of these things and store them on a solid state drive (or a few of them if you’re a hoarder) quite easily.
@@Hypnotically_Caucasianactual hero
Or in copyright books, LibGen comes to mind :P
libgen? Annas archive? Hey @Hypnotically_Caucasian, just check there.
Ah, the IT crowd moment :) Jane, this is the internet!
i immediately thought of that, lol.
The ELDERS of the INTERNET? The ELDERS of the INTERNET KNOW MY NAME?!
We are bunking off.....!!!!! 😂😂😂
Well…. If it’s ok with the Hawk
Jen not Jane.
internet archive is another cool place to download stuff
Unfortunately being sued into oblivion by book publishers, so use it while you can
Very true
Hmm.. only 109GB for Wikipedia? I'll add that to todo list for my server. That would have been helpful back in early 2000s when I had slow Internet and doing school work.. lol 😅
But if you started downloading 109 GB back in 2000, you'd be finished just about.... now :P
That's just the text of the English Wikipedia, with images and the other languages it's like 1TB
@@ANT-jm4qxjust the text of English Wikipedia compressed is like 35gb
My man would have graduated before the download finished
@wrOngplan3t you wouldnt be finished, since youd be stuck with 2000's outdated info 😂
Local mapping data is such a godsend. Years ago we were crossing from argentina to chile through a small crossing in northern san juan, middle of the mountains aka no internet and no phobe signal, OSM was super useful for navigation :D
Tomorrow I will download the internet, and then cut the cable! Take that, ISP!
unfortunately the internet constantly updates...
also im in australia so it'll take about 8 hours to download wikipedia as it is
I think I want to integrate one of those ChatGPT OpenAI AIs and run the AI offline with this offline data set.
@@zonk1477 the AI would need crazy amounts of RAM and data to be usefull.
"Internet's most important resources: Arch Linux Documentation." What am I going to do with Arch Linux if the world's internet disappears lol. I think at that point there will be more important things to worry about. "Okay, all the cities are in shambles.. i'm starving.. I'm cold.. I've got it, let me grab my localized Arch Linux docs. At least I won't have a peasent operating system like Ubuntu."
You're not going to do anything with Arch Linux. That's not the point.
The point of the arch docs is that they are the most concise and consistently useful set of Linux documentation yet made.
*Critically*, the vast majority of the information in those docs is distro-agnostic.
I don't use Arch Linux on any of my devices. Everything in my house runs something that's Debian-based. Even with that, having access to a good-quality set of mostly system-agnostic documentation is absolutely invaluable, so I've always got the arch docs bookmarked, and it's one of the first things I'd save if I were building one of these.
I of course would also save the Debian documentation, and perhaps dump the AskUbuntu forums, in addition to Stackoverflow and Serverfault.
That being said though, what the Arch Wiki has that those don't, is brevity.
*That* is why I still use the Arch Wiki Constantly even though I haven't used Arch in years.
>decide to back up resources to be prepared when the world ends
>don't download anything technical because "i'll have more important things to worry about than my computer"
>computer breaks
>can't fix it because i didn't download the relevant resources
mfw
@@nibbletrinnal2289> be too stingy to download anything because “there are bigger issues”
> have no technology that you can use
> be in the stone age
> mfw
Kinda my thoughts too. Especially frustrating if needed to download additional packages in order to solve a problem
in a worst case scenario; internet will still be operational. all of the core infrastructure is built deep underground. we also use satellites. to say that internet wouldn't exist anymore is a myth. go play "outriders" . humanity took "all of internet" with them; when they left earth.
It's a nice to know that wikipedia is small enough for anybody to make a backup of.
perfect application for my 6W TDP N100 Proxmox sever running on solar power
This reminds me of that IT Crowd episode where they convince people the internet is in a black box.
Because it will be
Being a Venezuelan, I totally understand you and backup your actions. Thanks you very much for sharing your expertise with those of us who may need it in the future. (God forbid it though, because that would mean the SHTF).
You'd better shield that NAS from magnetic pulse/solar flare events or it will be wiped/fried. You MUST GROUND YOUR SHIELD (screen or copper metal fabric) !!!
Hard drives are literally faraday cages by design. At most the PCB fuse would burn, which is easily replaced or bypassed. The data on the platter is safe.
Love it, awsome knowledge and content. Thank you for the share.
I have a Boox Palma and bought a 1 Tb micro SD card. I’ve got almost all of what Kiwix offers, my entire state maps downloaded, and numerous videos, documents, books, libraries, etc.
All of this in an e-reader the size of a smartphone with a battery that can last weeks, if used sparingly. It’s a pretty cool setup.
Sweet
Where do you prioritize/ find valuable documents to save etc. Maps of the area where you live makes sense, but then how to prioritize what comes next? Farming / survival skills?
@@shanoahmontano7077 that’s where the 1 TB micro SD comes into play. You don’t have to prioritize. It fits everything. Or at least everything I could think of. There’s a whole datahoarder subreddit that would scoff at it, but I’m content
This is the best! So glad I subscribed to this channel!
Not sure if I can afford my little bits of the internet.
This is the best tech channel ive seen and im still subscribed
What a neat concept! I'd heard of Kiwix before but didn't realize what you could do with it. Cool that you can load mapping data too.
Dude, I'm so proud of you. 198,000 subscribers? Crazy. Keep killing it dude. from Warpiggies, to techhut.
Oh so very cool, this would be a very good VM for me to try thanks so much for a great video!!
Hi , sorry for the of topic. I would advise to turn down the fill ligh by about 2 stops. So there will be more volume. Used to work as second DP , or as a gaffer in cinema production for almost a decade .
Best wishes
I'll give it a try. 😎
Thanks again.
My idea of a nas is plugging an external drive into the CASA server
For some reason I love watching videos about setting up servers and networks.
The absolute muscle memory of clicking the word trains.
Wow, great h/w specs!
This has been something I've been planning to do for a long time but wasn't quite sure how. I really wanted to build a sort of STC (Standard Template Construct) server for myself if basically everything went to shit. Thanks very much! I'll be setting this up ASAP! :)
Better put it in a faraday cage and keep it off direct contact with the net.
Born and raised in Sunnyside, WA! Spent my life bouncing around the West but mostly WA and AZ. Thank goodness, projects like this mean that society will be much easier to restart if, God forbid, the worst case scenario does occur.
This is unironically the best present for Mother’s Day. Hell yeah
Out of curiosity. noticed you are using Edge on Linux ... is the hardware decoding working for you on the browser, vp9, h.264 and av1 or is it using CPU when watching videos, like on CZcams?
I was going to back green, but I'm concerned that the software will either have my data hit their server or they will somehow have access to the data. I wanted it for work but can't risk client data confidentiality.
I love where you’re going with this… for some of us.. Web 3.0 being decentralized… if you think about it… initially we all were decentralized before off the cloud. ☺️ I see that becoming a thing again. Loved the video.
Every company should store there sites everyday on a hard drive and then have a second feed source like satellite land line and signal so it stays active no matter what happens
This is actually a great idea. Similar to having a library of information in your private collection.
Thanks now my bunker is complete.
This , Jen , is the Internet.
The investigative reporter - Whitney Webb - highly recommended people doing this type of thing.She predicted (based on evidence she found) that there is a plan in place to take down the internet (and later do a "3.0 reboot of it ) to aid in forcing everyone on CBDCs and massive censorship / blocking access to certain information. Although, I think the option featured here is more sophisticated than what she had in mind for grabbing stuff off the internet.
Jen, this is THE INTERNET!!!
Need this! Especially I have a raspberry pi tower sever along Raspberry Pi 3 client.
Did you install Proxmox on the UGreen NAS?
I have self hosted many things, but not yet the Internet
Honestly if space travel ever becomes more mainstream each ship is going to need its own ‘internet’ that can be updated when a connection is available.
I've been downloading my vidoes. and PDF's in my collection but this is next level!
Time to archive the internet archive
I already do this with a series of XHDD's. Toshiba and Seagate sell reliable hardware at a [mostly] decent price. Every time they knock the price down to 100 USD i grab one of the 4TB drives and fill it up. Between work software (IDE's mostly), movies, shows, music & games, archived websites and internet videos; ive got enough entertainment to out last the apocalypse!
7:50 lmao the notification got me
I think that a GPT-4 trained model (a recent version of it got leaked, so it's widely available if you search) would be an excellent addition!
Besides being trained into the larger context of today's Internet, it's "knowledge" is stored into the trained model, so theoretically you wouldn't need Wikipedia (although I REALLY recommend it).
They need to make a Vault (or even multiple) for the internet like the Svalbard Global Seed Vault.
The main purpose would be for re-uploading in the event of unforeseen events causing the internet to go offline, such as major Solar Storms if they are ever to occur!
As for viability:
Estimates suggest that it contains hundreds of exabytes (1 exabyte = 1 billion gigabytes) of data. This includes everything from web pages, images, videos, documents, emails, social media posts, and more.
Our largest yet known to the public is Microsoft’s Azure Data Center, which has 15 exabytes of storage.
Microsoft also came up with extremely resilient storage disks that can withstand major damage of just about any kind, called Project Silica.
These can already store 7 TB of data per 75mm * 75 mm * 2 mm
Right now it would require about a billion of these disks to store it all
But allegedly Microsoft is planning to apply this tech to their 15 exabyte Azure Data Center, it might be feasible after all!
Better idea. Wifi or Bluetooth compressed file data sharing that connects automatically to multiple people. Maybe 2 other devices or more. Theres a similar idea in bluetooth walkie talkies, but no one has been able to get it to work well
Also a good reason: protecting your children from the internet by just providing an offline version of wikipedia and so on so they can still research for school projects etc. And then adding whichever other sources they need manually later.
so, it's limited to Kiwix right now? any other source platforms like that? or self-hosted scraping+compression maybe?
Currently downloading youtube.
Great youtuber on the Internet 😊
TY very valuable
Next video get a list of github project to backup along side, for linux enthusiasts, hackers and anything useful that is on github that has to be there on an old hdd. I will be using 1 Tb old sshd for this, no nas or anything facy just putting something to work. Any tips and tricks to do with that are welcome follow :) #theendisnear
Do you have any recommendations for more time tested equipment? I’m not doing all that and putting it on a kickstarter product ESPECIALLY when they have that weird software lol.
Perfect for meshnet
Save everything you find important to your life. (Pictures, music, movies, research, books, etc) aside from what you like. You don’t have a prayer of storing the entire internet. Just focus on what you can’t live the rest of your life without. I personally have a few drives containing everything I love about the digital age. If the apocalypse comes, I’ll be set in terms of entertainment 😂
Good video!
my reason would be getting this before WW3 💀💀💀 (idk if anyone else thought the same when they saw "before its too late" lmao)
is it necessary to go via VM & docker with Proxmox when there is also option for like lxc?
Curiously, today at work I needed to do something like this and I think the best option is wget with --mirror --convert-links and other flags
can you please explain why ?
Totally doing this
Super YT channel 👍💪
Yet another UGreen NAS promo? Yikes!
Kindest regards, neighbours and friends.
Much simpler than hosting a tile server is to use cruiser (available on aur) and just have the files available for use offline.
I think we’re close to the point where we could ask an AI to use Wikipedia to make an update to the last 2010 print version of the encyclopedia with updated information. One that would be printed out. You could even get it to make a 1 book summary or 5 book or what ever. Depending on how much you actually want to print or read. But training an AI on the old encyclopedia Britannica might require uploading multiple years so the AI can learn what type of information gets added or removed over the years. So the goal being that the AI would know to add important scientific advances or major political events. With the Wikipedia data the AI could cross reference those changes with the much larger Wikipedia data set. Obviously it would be copyright infringement unless Brittanica signed off on it or participated. I think the company still exist but is much smaller and just does online: But that would be a cool project.
A simpler solution might be to just use Wikipedia to make an update book or books for any version of encyclopedia you own. So if you have the 2010 you would have a book that has the updated events since that edition came out. If you own a 1980 set you might need 3 update books for example.
Wouldnt be better to use NFS for the storage?
Reminds me of the Brit comedy TV series 'The IT Crowd.' episode when they convince one staff the whole internet is in a little black box.
Hey! I have a question. I have Jetson Nano running Xubuntu, that is running Portainer, a Docker Manager, could I have my Jetson Mount my TrueNAS Server for storage and run my own internet and hold the data on my truenas server, but have the Jetson Nano access it?
I really like this idea, especially since, and correct me if I'm wrong, more and more internet resources are now being locked behind paywalls and subscriptions
Jellyfin is another great option to put into this
Always thought about doing this
what is that beatyfull linux distro YOU are using in this vid? Also nicest vid about this topic that ive seen so far. :D
"Do this if you dont trust your government." Well looks like WE ALL have a new weekend project lol!
Already had this in mind for over 10 years but it's expensive. All movies TV shows and games id want during the apocalypse.
i wish i had a sponsor to think like u
Wow! So cool!
you look exactly what you sound like, and you look exactly like someone who would do this
i'm genuinely impressed
good video
I'm both offended and honored. Thank you 🙏
Honestly I would do this for peace of mind. Just knowing when it’s offline the knowledge it can no longer be messed with negatively. Technically the amount of knowledge that you get from this would be next to impossible for an individual to get through it all. You’re only going to fact check and update interests of information.
This should actually be the way PCs evolve into. The OS automatically comes with the most important information and education from the internet. The rest could be added if you choose to. Just an extra drive already setup on machines with access. It would also make sense energy wise to. Less power to run access over and over again. Each time you access that website a ton of infrastructure needs to turn on just to get it to you. The bonus of this too is that companies can offer faster speeds for cheaper because they are not being bogged down by consumers. Of course that would be a balancing act. Now I just need to figure out communications like phone and texting so I don’t need to rely on companies for it. This internet download solution is a huge help to my goals.
The only way to do your own comms right now is LORA, so I hope all your friends live close by. Well, there's also HAM.
Then again, having friends close by is the best thing in a disaster.
@@joemerino3243 Thanks for the info. I’m going through a challenge where trying to figure out if it’s possible to make everything you would pay a bill for could be substituted for something self sufficient. So the most challenging one is comms. Everything else can be done as long as sufficient power is available through things like solar and wind. Water is provided by Rain and Dehumidification. Food through having a garden and taking care of animals. I mean you could do all these without power but it’d take ten times the amount of labour and time.
@@daniellapain1576 I think as you research you'll find solar and wind to be extremely insufficient for your purposes.
They both are very uneven in output and thus require batteries. Power storage wears out much faster than people think. Especially if you're trying to get your drinking water through dehumidification.
What you'll want is to be by a stream and dam it, then build a small hydroelectric setup. These can be built by hand from old washing machines or similar electric motors. This will give you more stable power that doesn't fluctuate so much. Your energy would still be quite restricted compared to metered power, though.
@@joemerino3243 It’s awesome that you just replied. I was on another thread and me and a few others are discussing a heating alternative using friction. Using a simple Dewalt battery drill a smooth piece of bar on concrete. I was able to get 200 degrees Fahrenheit in a short amount of time. There is something to it but it will take a lot of trials with different configurations. My input to this idea is to trial this concept inside a Sand Battery. Originally the fellow working on this idea was working out how to heat water with a wind turbine.
I think the biggest flaw to this end O' the world storage plan, is the fact that all the data is stored on a hard drive & not an SSD.
You could also just get an open llm model like Llama 3, which is basically the entire Internet compressed in an usb stick
plenty of times ive needed offline mapping because I was out of range of cellular or in a hole that wasn't covered by cell service.
Next you're going to have to get an EMP shielded case 😂
I have my Internet on a floppy disk since the 90's
Endless OS is an awesome and underrated distro I think.
Idea: Have a collection of the offline installers for all the most important softwares aswell as iso files for various operating systems
what about having endless os as a liveboot flashdrive?
Also add an offline ai model in the mix. It's surprising how much data it can provide with how small they are and it truly feels like still having access to the internet.
I think it would be interesting if you could have this done with pdfs. Make a virtual library of sorts.
A version of this with an LLM and RAG would be nice.
Hi! What about websites that are not on kwix? I'd love to host offline Word Reference as I use it for work and sometimes the internet is off
Wikipedia is just a good start for research, but, it's not a credible source.
I just want to put my ISO Linux, games, and movies on my NAS.
Wikipedia does not pretend to be authoritative, but gives links to all the source documents which ARE, at least within the limits of current human knowledge.
What would interest me is a means of downloading what actually changed in a target for localisation, so that it can be incrementally upgraded.
How do you keep the kiwix data updated?