Linux Dot Files Were Never Meant To Exist
Vložit
- čas přidán 9. 05. 2023
- Dot files or hidden files are a core part of the modern Linux experience but what if I told you that dot files were never really meant to exist and we've just been living through a 50 year old bug.
==========Support The Channel==========
► $100 Linode Credit: brodierobertson.xyz/linode
► Patreon: brodierobertson.xyz/patreon
► Paypal: brodierobertson.xyz/paypal
► Liberapay: brodierobertson.xyz/liberapay
► Amazon USA: brodierobertson.xyz/amazonusa
==========Resources==========
Rob Pike Blog Post: web.archive.org/web/201808271...
=========Video Platforms==========
🎥 Odysee: brodierobertson.xyz/odysee
🎥 Podcast: techovertea.xyz/youtube
🎮 Gaming: brodierobertson.xyz/gaming
==========Social Media==========
🎤 Discord: brodierobertson.xyz/discord
🎤 Matrix Space: brodierobertson.xyz/matrix
🐦 Twitter: brodierobertson.xyz/twitter
🌐 Mastodon: brodierobertson.xyz/mastodon
🖥️ GitHub: brodierobertson.xyz/github
==========Credits==========
🎨 Channel Art:
Profile Picture:
/ supercozman_draws
#Linux #OpenSource #FOSS #LinuxDesktop
🎵 Ending music
Track: Debris & Jonth - Game Time [NCS Release]
Music provided by NoCopyrightSounds.
Watch: • Debris & Jonth - Game ...
Free Download / Stream: ncs.io/GameTime
DISCLOSURE: Wherever possible I use referral links, which means if you click one of the links in this video or description and make a purchase I may receive a small commission or other compensation. - Věda a technologie
"The time machine is finally complete. You know what that means."
"We go back and kill Hi-"
"We go back and fix dot files, yes"
l often wonder if we should go back and take out Karl Marx Instead. LOL. Thing is, he had no idea how his ideas would be twisted. Millions died because or Marxism?
Killing Hitler and skipping 60+ years of relative peace is probably a bad idea anyway. Hitler took away a lot of excuses bad people have for creating a corrupt government in their favor. I doubt without the crimes of national socialism, and the destruction showing nobody wins in the end, the concept of human rights would have been taken serious for any peroid of history. Let's be happy there was such a peroid.
Reminds me of a sketch about an engineer being bad at naming things, and make things like a stain remover that prevents you from making the stain in the first place.
Just make sure there are no dotfiles on the time machine itself, otherwise you'll create a time paradox.
It's like a video game glitch that becomes an accepted tech like bunny hopping in source games.
the magic world of prpgramming
*Quake
*minecraft*
Well...Unix...at least the kernel, started out (kinda) as a videogame (Space something...I don't know, Thompson was simply bored out of his ass when he made it)
So it seems reasonable Unix would have all the fundamental hax0rs, I mean features, of a videogame XD
And as a result, we're all just very happy with the New Jersey approach to computing :P
@@SuperTort0ise I was specifically thinking of Valve trying to patch HL2's bunny hop and making it worse. But yea I also thought of Quake.
It’s not a bug, it’s a feature :)
**Da creeper starts to sizzz behind you**
69th like :)
True
*League Of Legends like it*
I still consider it a bug. Dot files are a hacly alternative to a good file hierarchy...
I like dot/hidden files & folders. It allows for some potentially breaking things to be hidden from users, but still allows them to be seen for others. It deliminates settings folders from data folders. Most importantly of all, keeping them in the home folder makes it easy to backup & move etc. I do agree though that better standards should be followed to minimise the amount of dot folders at the root of a home folder. The .local & .config are a good start.
The programs could easily be set so that the next update would move the dot file to the .config folder. It is not an insurmountable task.
@@act.13.41 very true. Though I think it may still be down to the standards being not that standard yet. I'd assume some don't want to make a change until things settle, of which they won't if people don't push them. That old chicken & egg.
But, ~/.config and ~/.local should be a part of backup routine anyway, so they're already easy to backup and being in proper folders is safer than in home's root as people won't think they can just be deleted (which is more common if in home's root).
At least both ways are easier for backups than Windows, which has config files scattered all over the place, making reinstalls with profile backup and restore fail for multiple things, whereas Linux hardly ever fails for me for anything (though I can't think of anything, there's probably been something that failed, so I'm avoiding "never fails").
@@ChrispyNut yep, Windows backups & migrations are a pain without dedicated software. Not only files all over the place, but registry entries too. I'm glad to be out of that, though I still have to keep up with it for friends. As for having a second dedicated user settings folder separate from home is a good idea, but that would take even more effort to get the community to move. Though that may just cause more issues, look at how the Programdata folder on Windows is used for things that it probably shouldn't be.
@@foznoth It's happening though, thanks for the XDG base standard.
Probably gonna need some applications which refuse to adopt, to be usurped/forked by those who have/will, but I expect many will implement migration scripts in updates down the line.
Maybe there will be some folder abuse, but part of what makes [GNU]Linux/FOSS so great is the ability to fork projects [which misbehave].
Nothing's ever perfect, but that shouldn't hinder the pursuit of progress.
At least they changed the convention and created the .config folder. Now most of the .dotfiles reside inside that directory so the home folder is not filled with a shit ton of files. also the fact that folders in the .config have the program name as their name makes it easy to sort everything out, delete unsused programs' configs, etc. much better overall than what it used to be.
This line of thinking shows why this bug became a feature, which hey if you make it work it still works.
The command
ls -d ~/.[!.]* | wc -l
reports that I currently have 251 dot files/directories in my home directory.
Also, think of the privacy issues on a multiuser system: even if the files/directories themselves are not readable by others, it is difficult to hide their existence (or nonexistence) from other users.
You can disallow the read of your home directory, and than only root can see it
It's too bad they didn't use the opportunity to create a true "local root" directory and fold ".config/" under ".local/etc/", mirroring the standard base directories, rather than separating all the XDG directories in their own directories / environment variables.
@@schwingedeshaehers That makes it difficult if you want to share some files with other users.
I also think hidden files are more a feature than a bug... The use of a hidden .git directory in a git repo feels like a very elegant solution - a lot more elegant than a visible directory. True that it would be better to organize the dotfiles in home. But... I just don't feel it is all that annoying.
But dotfiles are older than git. Just because dotfiles are super useful and every program uses them doesn't mean they weren't originally unintended
I think the concepts are mixed. Dot files and hidden files.
The fact that You hide a file by putting a dot in the front is totally irrelevant. The important feature is the fact that you can "hide" files...
*I guess this is a certified "It isn't a bug, it's a feature." moment.*
10:41 “This is a certified UNIX® system! I know this!”
This was actually quite interesting to me because when I first saw the files called . and .. I had learnt that all files starting with dot are hidden. This made me think that . and .. names were chosen for current and parent directory because it is the shortest way to express a hidden file/directory. Now, learning that it happened the other way around is kind of mindblowing 🤯
One folder I hate being "hidden" is '.minecraft' being dumped in %APPDATA%/Roaming on Windows particularly - chewing up precious OS Drive space when I got a nice big meaty D/E/F drive to dump my games to.
S'why I use Prism Launcher (stay the hell away from MultiMC nowadays) and GDLauncher in Portable Mode - forces all my Minecraft Instances to be stored in a folder dedicated for them *away from the OS drive*.
That is just Notch royally misunderstanding how dot folders are supposed to be used. They are for configuration and cache files, and not for the whole filling game.
I do see the use for the .files and .folders as a simple way to be able to easily declutter folder content views. What I am NOT a fan of is filling the home folder with them. It's something that messes with my idea of being organized and just knowing they are there makes me grind my teeth every now and then. Before this video, I had no idea why the home folder was getting filled with all these files and folders. Now that I know the reason I'm just... really annoyed. It's understandable how it developed but my sense of organization just wouldn't allow me to do it.
So thanks for making the issue a bit more bearable, I guess. 😁
Most software supports XDG_CONFIG_HOME ($HOME/.config) now.
brodie have a 'download' and a 'Download' folder,has the same vibe as 'alias sl=ls'
@n0tjakI love that program xD
I once trolled a colleague who had not locked his workstation. I installed sl and added it to his bashrc
Now everytime he opened a terminal he had to wait for the train. And because he did not know what Program this was, it took him a good amount of time of figuring out what was going on xD
I like them because now when I look in my home directory I don't see a hundred files/folders I don't mess with on a regular basis. Makes it easier to find the stuff that I do use.
yes but those files should never have been put there in the first place
7:32 a good reason to invent time machines XD We want less junk in our home folders
I like dotfiles, but I agree that home directory clutters up really fast with them. I wish the home directory contained only files I've created myself, with the exception for config files. I currently have 41 dot directories in my home. But even more annoying than having a bunch of dot directories is having a bunch of visible directories that one hasn't created themselves. For example, I wish the snap directory would be .snap if it absolutely has to reside in my home directory at all.
This would be where you create an operating system that is exactly the way you like it.
BTW, for snap, at least on gnome you can create a file called .hidden in your home directory to hide additional files/directories. Might work on other DEs too, idk
@@lunlunnnnn There's really no limit to what you can do in Linux if you are willing to make the effort.
@@lunlunnnnn Thanks! I use KDE Plasma and just tested the .hidden file. It worked. Cool!
50% of home .dot files are config files hahaha
The other 49% is the concept that your user stuff has to be on your home directory. To fix that we would need to have a different concept for storing user installed content.
The 1% is the .porn folder hahaahhaha
I love videos about Unix/Linux history and things like that, nice video! :)
Huh, I wasn't sure who'd done the video on XDG Ninja a while ago, so I checked to give it a mention aannnnndddd, it was Brodie, so I'm surprised he didn't mention in this video.
I'll do so for him (even though I'm probably lazier). App with SOME ability to automatically move dot files from home to proper XDG folders if already supported by those apps: czcams.com/video/Tp4fkmJ6qXk/video.html
Maybe for outsiders such a concept is out there but I think for those in the community it's not that crazy. Considering the way Linus himself speak, the man himself champions that bugs that become conveniences, are no longer a bug but a feature...
It shows that this line of thinking is common in this space. Considering Linus is one of the olds in the realm
Granted I get where Pike is coming from, and in this case not something they wanted to become a feature but it did...
Rob Pike did a guest lecture at my university a couple of months back, and it was by far one of the best lectures I've watched. He's a super awesome guy and a huge inspiration!
You mention that there are other bugs in the Linux kernel that have since been incorporated into workflows and used by other components of the system to the point that it would no longer be practical to remove them. I think it would be interesting to know more about them too, and how different, perhaps better, things would be if those bugs had been avoided from the beginning.
There is a special cauldron in hell for those who write apps that dump dot files in home directory
I actually like the hidden file concept. It isn’t fun finding a hidden file/folder somewhere, but it allows me to clean up my system. If I don’t want to see something, I just put s period in front of it. It’s like deleting without deleting a file.
The behvior is a bit wonky.
If you do:
ls .*
The first expectation is that you simply get a list of all dot files. But that's not exactly what happens.
Even worse:
chown -R user:group ..
I once made the mistake of doing that and screwed up my entire installation. It was a messy thing to fix, so I think I just reinstalled the OS.
So dot files and directories require special treatment when you are at the command-line.
@@friedrichdergroe9664 First, there are hidden folders so you should use `ls -d`, and second to list everything that's not '.' and '..' you could do `ls -d .[!.]*`. Third, don't ever use `chown -R`, especially if you don't understand why.
@@anon_y_mousse I understand why, as l have been doing this for decades. But even best of us is entitled to a slipup once in a blue moon. LOL
One of my biggest gaffs was deleting a live database. I was able to fix that by hacking the inodes of the still open files. Scary moment.
@@friedrichdergroe9664 How did you delete it? Did you use a wildcard where you shouldn't have?
Was there not a file system function for hidden files like there is today in FAT or NTFS?
I feel like in today's world where computers are mainstream it makes sense to keep some files hidden from people to protect thrm from doing dumb things.
Make them owned by a different user and then they can't change them, and make their home directory have the sticky bit set so that they can't delete them...
Or just use chatter to make it unchangeable.
To be honest it’s probably best to have those files hidden from basic users. Let us not forget the days of deleting sys32.
Or just have a "hidden" bit built in the filesystem. This is one of a few things that Micro$oft did kind of right
@@leogama3422 "Or just have a "hidden" bit built in the filesystem."
There is no THE filesystem in Linux. Many filesystems are supported and some have more meta-data than others.
@@leogama3422I have an idea, put a dot in front of the filename. So its filesystem agnostic.
Oh, wait... Hahaha
> Now defunct Google+
Right in the feels.
I think them being hidden is more of a visual thing, because I have dealt with many people that were learning computers, and any folder they did not need, needed to be deleted or hidden. So all of the config folders that are needed were ideal to be hidden. I know that is subjective and preference, but most lay people I encountered that was the case. Personally I do not care, if they were not hidden then I would like them all in one folder for ease, but as it is, being able to hidden is similar to that.
I think dot files in unix/linux are a great way to hint to people that they shouldn't touch these files unless they know what they're doing, without enforcing the issue through some complex permissions nonsense.
Similar to private members in Python classes being prefixed with an underscore: "it isn't private, but let's pretend it is."
Mac OS X has basically reinvented this with their ~/Library directory. Originally it was visible in the file browser GUI, but more recently they have started hiding it by default. At a high level, the file flag (chflags [no]hidden {file}) is more a explicit solution with an exactly stated purpose and so less likely to cause other program bugs like the counting one originally described. But it is also another example of the XKCD 927 "There are 14 competing standards" situation.
Okay, he can have his opinion. By the time I started using Unix with Unix V7, the .bashrc and .profile and a handful of other 'dot' files were happily being used. It's an extremely useful feature even if it was due to what Pike calls an accident.
While a time machine would be required to fix this problem, I think a step in the right direction would be to put pressure on projects to use .config or .local. Perhaps by putting a "preferred" tag on applications in package managers for applications that do? Then projects might put this on the TODO list for the next major version because it makes their project seem more professional The problem is that there are many programs used in distributions that are essentially unmaintained but are liked and used by many users. I like the fact that I can back up my home directory and get all the settings etc for the system installed applications, so the settings and data being in some structure is good - I'm not sure about those applications like Arduino and MPLABX that throw non-hidden folders into your home directory, because this is essentially the same problem. But its certainly worth highlighting this issue so that new projects have a standard to follow.
I see your point and it is a really good point, but I do see a use for dot files (or hidden files) for things that you actually want hidden, you don't want people to see or find just on a glance. Though as you said, because of people who like to tinker on things have hidden files shown at all times, it is kinda useless. I would like for files to exist but for things to not use them at all when they aren't necessarily, which would then only leave files you actually want hidden for a purple to be hidden.
Lets move all dot files into ONE directory and build into the filesystem that whenever a dot file is called its always in that directory tree
I definitely don't want *additional* always visible directories, so I'll be content with my dot files. The XDG Base Directory Specification is a godsend and more people should rely on it. It's much easier for understanding, backing up, and managing those important files when they're in SANE places like $XDG_CONFIG_HOME/ProgramName and not $HOME/.lolololol or some crazy thing that has seemingly nothing to do with the software.
And lets be honest, if $HOME/cfg were the standard, people would add ways to hide them in file managers anyway.
that is a very interesting piece of history. I agree with this assessment though: this could have easily been done with a /home/cfg and subsequent subfolders within the cfg folder. like /home/user/cfg/local or share, etc. I kinda always wondered why this was the case as hidden folders within a users folder never made sense to me. Apparently the answer is almost always laziness(which is weird because it takes more effort to do it this way). 🤣
i think the general idea or the agreed upon convention (whether a good idea or not) is that no program should ever put a file or directory in the home directory that doesn't have a dot in front of it, the only things that should be in the home directory without a dot are things that the user put there themselves, so that includes a $HOME/cfg or $HOME/local, i think the idea is that the user should never have to see things in a normal ls that they didn't put there if they don't want to, because otherwise it would just be clutter that they didn't ask for; the home directory is typically completely empty, except on typical modern systems that nowadays include the silly XDG directories like "Downloads" "Documents", etc to try to mimic a typical Windows or MacOS installation
You would still want a mechanism for hidden files though. I don't personally think that naming conversions are a bad way of handling it. It's simple to just rename a file, that's easy for the user, easy for scripts, easy for other programs that want to do similar things like .gitignore . Alternatives are things like msdos attributes, or stuffing metadata in separate files like on mac, which creates even more clutter. So in the scheme of things I think dot files actually are one of the better solutions.
@@shallex5744 to be fair, *I* will create those folders if they don't already exist, and so will most other people. This is kind of why that's mostly done by default in the majority of distros. Organization is kind of important, ya know? If you're not organizing your home folder in a common sense way to prevent cluttering the folder, you're barely a step removed from those barbarians that leave all their stuff scattered all over their desktop! 🤣
Personally... I'd rather config files not be in the home folder at all. the system already has a place for config files, and it's far from an inconvenience for most people to just su to edit them, but if they're going to dump crap in your home folder for no good reason, I'd rather SEE it, so that we can tell it wasn't just tossed there by a random bit of malware. IMO, there's no valid, non-malicious reason for anyone to HIDE files. Back when it was created and compute cycles were a legit concern sure maybe there was a good case for it as the video explained, but definitely not today.
Besides.. can't you just apply a switch after "ls" that shows only a page at a time that loads the next page of results upon hitting enter or something? I mean, surely if DOS has this feature, linux's shell must. 🤔
@@entelinThe dot files were just, as noted, a way to originally hide . (current directory) and .. (parent directory) by the ls utility.
Using attributes may be useful to DOS where names were stuck at 8 characters with an extra 3 characters to describe the file type and what to so with it (after a dot). Windwos file explorer defaults to hiding these "extensions" just like ls hid the files that started with a dot!
The thing about *nix files is that the attributes are stored in the inode whereas all the directory entries are just links from the name to the inode (or a path name "symbolic" link). It is this linking that means the same file (actual data and attributes - everything [except name]) can appear in more than one place in the same file system.
There is also a neat trick in that once you've opened a file, you no longer need the directory entry and can unlink the inode from the file system, and the disc space will only be released back for reuse by the file system when the file closes[1]. If the file is a temporary file by unlinking it nobody else can then open it, and it also frees up the name for reuse.
[1]if the system crashes with the file inode with no directory link, a fsck on reboot can find the in use data and create a file in the lost+found directory.
I could see one of the issues behind 'not fixing' the issue, many newbie users would likely erase the directories and then wonder why their emails (for instance) disappeared if they were visible by default.
This is not a issue, it is a good way to learn!!
@@joaomaria2398 /opinion
@@bigbay1159 ?
Mac OS X uses ~/Library with subdictrories like Application Support, Caches, Preferences, Mail, etc (dot file behavior exists and some programs use the, it is still a UNIX). However recent versions of the OS have begun to hiding these form the Finder (file browser GUI) via other file attributes.
One has to assume they started hiding them from the GUI for exactly the concern you mention, that too many people were deleting things before understanding why they might want them. The mac will recreate these directories and files in their default state as needed, so you can delete ~/Library without seriously breaking anything, but it may be confusing for a user that this "weird" directory keeps coming back and your programs lose their preferences.
Back when what has become known as MAXPATHLEN was in the low 3-digits (128 or 255) and the maximum length of a shebang line (#! plus path of executable plus options) was 32, file names had to be short and sometimes directory trees had to be flattened. People counted and saved every possible byte. Dotfiles were a comparatively reasonable compromise to cram a lot of crap into a short string. Aging tools like tar and cpio still have other old limitations, like short path lengths and inode number overflows.
While this unintended decision resulted in one type of file masking, hidden files are not a bug. Many file systems even incorporate summer off these features as metadata or similar datastreams that are not visible as files. By comparison dot does are much easier to deal with, edit, back up and transfer, which is really why they have such around.
That Linux needs to clean up the current rat's nest is unrelated, though it does need to happen. Personally I don't want extra data cluttering up directions visibly, nor as metadata or streams, so dot files feel like a reasonable compromise.
Years passing I tend to put « human » visible data outside $HOME - in another partition, usual personal XDG folders are just symlinks.
I mostly let dot/hidden elements in legacy $HOME.
The « mistake » for me is having two different families ( or purposes ) of data in the same place : { configs, settings, preferences, caches } and { documents, medias } because first family is more or less tied to the versions of system and applications whilst the second is more neutral, agnostic, the second family has no consequence on the functioning of the OS or DE. First family has.
Otherwise there's another way to hide things : list them in a file called .hidden that sits at the root of a folder ( and bye bye ~/snap which by its nature SHOULD be hidden. )
This is the ultimate example of turning a bug into a feature.
Never knew that, quite informative as always. :)
LOL I recall when I resisted the idea of dot files or hidden files, but eventually this is one quirk that actually won me over. I like this Video -- good job.
the funniest thing about dotfiles in the home directory specifically is that a BUNCH of things have env variables you can set to move them... which works great... for a week. Then they're all back in your home for some reason?? And then there's thing like yarn (the node package manager) which has options for some features to move but your .yarnrc file lives in home and cannot be configured to live elsewhere, even tho the developers are well aware of XDG_CONFIG_HOME existing.
6:02 One way to tell whether something is still needed is to move it aside into a separate directory somewhere you call “delete-pending”, or some name to that effect. If after, say, 1 month, 6 months, however long it takes you feel happy, nothing untoward happens, then you can delete the contents of that directory.
I don't know, something like… a trash folder? 😜
@@BogdanSzczurek, problem is that sometimes the system manages the trash folder, but what happens to a delete-pending folder is totally up to you.
@@BogdanSzczurekYou may clean the trash by accident.
Which is more proper to write, config/mozilla or mozilla/config? I'd argue that they should be the same. While we're talking about 50 year old bugs which are now features, this is one which could be corrected today allowing the path to represent tags rather than an explicit path. Were that design retro-fitted into *nix today, only where there are some overlap like the aforementioned config mozilla directories, where the same filename currently exists in both, would there be a problem. It also has some undetermined functionality with for example how sbin and bin are in multiple different subs, so should those subs be listed? What is they are several subs deep, like having a bin directory under source in a user directory. So there are still some things to solve, but how convenient would it be to have everything available where the actual order wasn't important so that you could apply path filters until the path was the same as what we have today?
My first personal encounter of introducing unintended features was when I had to write my won parser for a limited scripting language.
There where only 2 people officially using it at that time and they already started using undocumented quirks of the parser that reliably work. As in I officially documented some of them after checking if there was an obvious way they could break in an unpredictable way.
Most of them related to writing comments in the code. So the damage scope was very limited. But it also gave me an explanation on certain quirks of Assembler. The parser due to being as basic simply has much of the same "bugs" as mine. And no one ever bothered fixing them because most of them where documented as features.
When you put it like that, Flatpacks and AppImages make a lot of sense.
quick question. Which status bar are you using for Hyprland? I've been trying to find a good one for a while as a stopgap for learning eww
Waybar for now
@@BrodieRobertson Huh, didn't know you could make it look that nice.
It sounds to me like if I did go back in time and prevent the concept of hidden files then it would only be creating an exploit within the Unix system just waiting for someone to take advantage of. It's a good thing I am more responsible with my time machine than you would have been.
I knew of the dot hidden files because I have used Linux off and on over the years. Also with the Steam Deck in order to do some modding you need to see the dot files because that's where is Steam is located if you installed any games on the SSD rather than the microSD card.
In Unix it may have been a happy accident using a "hacky" method, but by the time Linus rewrote it for Linux, I'm sure it was %100 intentional. (Remember Linux is not Unix but a new operating system that was modeled after it)
I can imagine its been put there for compatibility, e.g. not to break scripts that run on UNIX and should also run on Linux
What’s that shell you’re using? It looks awesome.
I do appreciate the fact that when I do a listing of my home directory in particular there’s not a lot of clutter from apps, all that config mess is hidden and I can see just my files. It’s also handy for lame-ass version control, I simply Rename the older version of a script etc by prefixing with a dot, and postfixing with the version number .2 .3 etc so that the most current version of the script is the only one that shows in a standard listing of the directory. Dot files have never bothered me and I’ve been using Linux, AIX and OpenBSD for about 25 years now.
I think another part of it is that by default the IDE will choose the linux home directory as save location. It can be changed or configured, but normally with no configuration they usually default to the home directory in linux, and the equivalent in windows. I can't remember about mac, but I believe it's similar to linux as well.
It is nice to be able to hide files. Some people have files they don't want anyone messing with. It would be useful for parts of an OS that were never really meant to be seen.
It would be nice if they could be hidden
This is mind-blowing, always thought they were intended
Looks more like an accidental feature to me. What I would say though is that if the config is in some subdirectory of the the .config directory, it is much easier to version control it.
I like dot file and directory only if is created by myself. I hate to have config files on the home directory or even temp file, component file these are just annoying.
The worst design was exposing to the user that "." and ".." are real files and not just a concept. That has prevented future implementations from having hard links to directories. And that feature has a ton of useful applications like "time machine" style backups. Also, whoever changed "ls" made a minor transgression... whoever changed shell globbing did the most damage. Having to deal separately with "*" and ".*" is the biggest mess.
Very interesting story. Thank you for sharing this!
I like plan9 (OS). The thing I don't like about dot tiles in my home directory, is that I have to treat the root of my home directory as off limits and put stuff I create in a sub folder. It I happen to name that folder the same as a package I install later, I can have a problem with the files I created. I saw a ZFS setup where dot files were put on a different dataset.
I think an actual replacement would be another bit for hidden files, so you don't need to change the file name. That was what confused me when I switched to Linux from Windows.
I think an attribute bit is the wrong solution to the problem. Think about why the dot prefix exists: it gives applications a separate namespace to use for their own housekeeping, to minimize collisions with names that users might choose for their own files.
Think if you had a hidden “config” folder, and the user, for whatever reason, tried to create a file or directory called “config”, they would either overwrite it, or get an error saying it already existed. The first possibility is unfortunate, the second is just mystifying, because they cannot see the item that supposedly already exists.
I'd say having a special subfolder in each folder that's hidden, like e.g. "...", would be a solution with fewer drawbacks.
It certainly would solve the "where is that config file" better than what other operating systems do. It's somewhere in "~/.../", not in roaming, application support, application, appdata, My Documents, local, Preferences, %USER%, etc.
Linux supports many kinds of filesystem (EXT2, EXT3, EXT4, FAT, NTFS, ZFS) and so there is no universally applicable method BUT every one of these understands the ASCII character set. So a dot in front of the file name is not "hidden" to the file system (which may actually include a hidden attribute), but is hidden from BASH. Such files are also not hidden to applications since the dot is just part of the file name.
That's a lot of home dotdir/files. It didn't use to bother me, but these days I'd really prefer for programs I rarely use or only use once to not litter the home dotdir. I wish everything follows xdg folders specs, but that's not the case.
I legit use containers like distrobox and conty since they can set custom home while still able to connect to existing folders if I want to (not sure if Junest have it as well, haven't used it in a while).
I remember the first time I accidentally renamed a folder as a dot file in mac os. I was fully unprepared, thought that I accidentally deleted it. It was what introduced me to using a terminal, as there were no other ways to make hidden files accessible. Which was kind of amusing, being able to send and retrieve files from the banished zone, it made me feel like I was some kind of necromancer communing with the ghosts within my filesystem. If I were to change anything it would be to make the linux hidden file experience more like Mac OS.
"If I were to change anything it would be to make the linux hidden file experience more like Mac OS."
That would be circular since Mac OS has been made to be more like Linux. Well, not really, Mac OS was made to resemble a Unix variant, and so is Linux. So there's a lot of command line similarity but under the hood nothing is deliberately similar.
@@thomasmaughan4798 I described exactly what I meant by that, unlike Linux Mac OS doesn't have a "show hidden files" option in finder and it's funny
I very rarely have to mess with hidden files. And with KDE’s ability to temporarily show hidden files is genuinely pretty good
Something I really like about Linux is that you can see pretty much everything you need to know about the permissions and properties of a file in so little space. Take the following two lines from when I do ls -la in my home directory:
-rw-r--r-- 1 ubuntu ubuntu 3771 Jun 29 20:33 .bashrc
drwx------ 3 ubuntu ubuntu 4096 Jun 29 21:32 .cache
This shows me:
1) whether something is a file or a directory
2) what read, write or execute permissions I, people in the owner's group, and people not in the owner's group have
3) The number of file hard links (had to google this, no idea what it means)
4) the owner of the file and the primary group of the file
5) file size in bytes
6) the last time the file was edited
7) the file name, which also tells me if it's a hidden file
All in one easy to read and parse line, easily accessible through a very convenient command.
Compare that to the way that Windows does this kinda stuff. Yeah, basically what Windows is actively telling you is "please don't mess with file and folder permissions, because it's going to make it a pain in the ass for you and anyone else".
That's what you call an "unintentional feature".
I can see you. I just can't unsee you. 🤣
Good job Brodie.👍
It's why I don't use my home directory on any operating system. I always use the root directory of a new drive or I make a new partition on the only drive I have that has everything in it.
I just noticed, you have the hiragana fu in your browser addons bar? What's that about?
I think it is an extension to turn kanji into furigana.
@@leucome
I just looked in the Firefox addons store for "furigana" and then realized what "furigana" must be, inserting phonetic hiragana above kanji, like also shown in the Wikipedia article about Kanji (where there was an image showing "kanji" written in kanji with hiragana phonetics next to it).
@@Lampe2020 Yes. Usually they add these in kid books so they can learn kanji while reading the tiny hiragana.
I do think it's a bit weird that hidden files are created by using a period in front, I think if it had to be done it should have been done more like Windows/DOS with hidden being a file attribute. Doing it that way would have made it so that a file being visible/hidden could be changed with chmod without having to rename it, and also likely had the advantage that when logged in as root, you can see all hidden files, whereas the files you shouldn't see are hidden from normal users.
or chattr if you still want to be able to access the hidden files/directories
Yeah, but be careful, you may trigger some linux user for saying Windows does it better
@@Max128ping As explained in the Video, Linux doesn't intend to do it at all, it's just a side effect. So saying "Windows does it better" doesn't make sense. And I personally don't think, hiding files is anything good at all.
@@Linuxdirk Actually, the video says that the original Unix didn't intend to do it. Later clones of Unix, like GNU/Linux, copied it for compatibility from the start. So the GNU userland utilities and Linux itself were designed this way from the ground up, admittedly copying what was a mistake in Unix when it was first introduced for the sake of compatibility.
@@jeremyandrews3292 A minor point, the kernel doesn't care at all. It's only userland stuff.
Like the windows registry is any better?
What bothers me more is the inconsistency in the XDG base dir implementation as-is, namely, why ".config" and not ".local/etc", especially since ".local/share" seems to be the only standardized dir in ".local", and I've only seen one other app use ".local/state", which makes the ".local" dir rather redundant currently (why not just use ".share")
Seems to me like it would have been simpler and more consistent to recommend devs store any user data in the same place they'd store it from root but under "~/.local" instead, and properly, ie;
.local/etc
.local/share
.local/var/cache
.local/var/log
...and so on, just keep all the application junk in a single dir under home and leave the rest of it for the user to organize (or not) as desired.
Question, you bother the dot concept of a hidden files or hiding files as a feature?
If they start using metadata/attributes in the files for hiding, will that help :) ?
No, because all links to the file inode would either be all hidden or all visible, unlike the mix you can now have by having one link starting with a dot whilst another doesn't. Plus it'd also mean that even if you're only just listing the contents of a directory, the ls utility would have to check every entry's inode to check its hidden status (just like it does when using -t, -l, etc).
@@cigmorfil4101 so, overall, hidden files sucks?
Is this GNU/Linux specific or bsd and other uni*es?
@@DamjanDimitrioski
It runs all the way back to Sys V Unix (or possibly earlier - my source detailed information is about Sys V Unix) - directory entries just link a name to an inode which the kernel uses for the actual file data access. In Sys V directory entries were 16 bytes each, allowing upto 14 bytes for the name and 2 bytes for the inode link.
When just listing a directory's contents ls would just use the names in the directory file; if more information (access permissions, file length, etc) were needed it would then follow the inode link to get the required information (or more accurately call the kernel for the data stored in the inode associated with that file name). By doing this and using the dot prefix, files could be ignored just from the directory entry without having to get the file (inode) stat data.
If you start using a hidden flag it will either have to be in the inode data, which then means every file's inode would need to be interrogated even for a simple name listing, or it would have to be stored in the directory entry itself which would require a rewrite of that structure.
Putting the hidden flag in the inode is a bad idea as the name of a subdirectory you would want to be able to see, but the dot link to the same inode within that directory you would want hidden - you have an inode linked from two different places with different hidden requirements.
I like how the whiteboard says "you can't see me"
On the directory structure. What I do on my backup USB drives is having ./data where I put the data. Then there is ./lost+found from EXT4, and I created ./icon.png and ./meta.db. So I basically use ./ as root and ./data for the actual backup. With this the backup is not cluttered with technical and additional data.
This could be done for home directories, too. have all your data im $HOME/data/ and use $HOME/ for configuration files and directories alone. You could also place your "system data" like scripts or your profile picture, etc. there without cluttering your "daily use data".
"less CPU cycles than doing it properly"
Be careful, maybe you can summon an actual demon from hell by saying something so disgustingly cursed.
I consider the very concept of anything being hidden to be bad. It's MY system, running MY software, doing what I want, but if anything is hidden how can I truly know what's it doing and why. How can I clean my system if i can't know what application something is for.
“Checks show hidden files”
Phew crisis averted
When you have shitload of junk in a directory, just sort the files and directories by last access or modified date and move all entries over 6 months to another folder... If shit doesn't break loose, delete them... Some programs are limited by string length, so using single character saves a lot of bytes when the command-line has a lot of parameters. When you have a lot of mounts outside /mnt or root directory, it makes sense to keep everything as short as possible if the path to the mount root is already long. Same applies to symbolic links to directories.
I quite like the .files tho
Only problem is, too many apps dumping stuff on the home dir. They gotta learn to use .local and .config
I did not know that story about how dot files were hidden originally. It seems useful to mark files that basic users of macOS or Linux don’t see normally because they don’t need to interact with those files directly. The macOS has the $HOME/Library that is hidden in the Finder and is a modern way to do it, but they also hide all the dot files, so it has both.
I love them, there are some things you just don't want to see normally (well in my case I choose to see them always but the average user is not gonna wanna see them normally), I'm actually use a dot folder for a 2 part solution to cross-platform programming I making. The reason I chose it is because most users will not want to see the folder normally since it is only supposed to contain binaries & libraries that will be mapped into PATH prior to usage rather than make them a permanent part of PATH. The reason they'll need to be mapped in because I'm designing for ABI portability of the binaries & libraries, rather than do a lot of #ifdefs in code per ABI for dynamic libaries etc, just use 1 name and expect the PATH variable to lead to the right directory, the only time ABI/API becomes an issue is if you start linking native libraries and even then that should be done dynamically only to so that your software adapts to the system it is run on, if the libraries you want aren't there then you just don't provide the extra features that library enables unless another library should be available in it's place for said features. I want devs to stop developing only for 1 windows or only for android or only for osx or only for ubuntu etc, I want them to design for the solution I'm making and extend their features when the expected system is detected through the main API. I'm not sacrificing the global software directory aspect but rather expecting multiple versions to be made and installed in the sub-global directories which become global directories to the version that was made. Here's an example of my rough idea for launching one of these programs:
paw --start : PATH=$(OTG_DIR)/.paw/release/$(PAW_ABI)/bin$(PAW_PATH_SEP)$(PATH) &&
paw --debug : PATH=$(OTG_DIR)/.paw/debug/$(PAW_ABI)/bin$(PAW_PATH_SEP)$(OTG_DIR)/.paw/release/$(PAW_ABI)/bin$(PAW_PATH_SEP)$(PATH) &&
paw --timed : PATH=$(OTG_DIR)/.paw/timed/$(PAW_ABI)/bin$(PAW_PATH_SEP)$(OTG_DIR)/.paw/release/$(PAW_ABI)/bin$(PAW_PATH_SEP)$(PATH) &&
If OTG_DIR is not defined then paw will define it to $(HOME) or $(USER_PROFILE) or whatever the equivalent is, the software contained in said directories is expected to operate as portably as possible, going local only if it is necessary (which 9 times out of 10 it is not). As for libpaw (the other part of the solution) that is the only thing that is required to have local dependencies for supporting files, threads, 2d/3d graphics etc. It's aimed at being the standard stdlib shoud've been but isn't, however it will not stop devs from using libc etc, just be able to work with or without them.
So you want an AppData for Linux? I'm quite o.k. with ~/.config ~/.cache etc.
Just this week I went through all my home dotfiles moving everything I could into XDG_*_HOME folders. Most of them could be simply moved and the application picked it up, some required setting an env var and only those I could not move:
.dir_colors
.dropbox
.flutter
.gdrive
.jenv
.ssh
.zshenv
In the process I realized that KDE config files are a mess inside .config. Why coulnd't they just put everything inside a kde folder? It would be so much easier to know what you can/cannot delete and to move configs between computers...
You talked about the . and .. as if they were for the use of those using the terminal. Although that is definitely one use it is not the main use today. The .. is used in all operating systems for relative paths in both scripts and programming. The . is used more on the command line to essentially say 'don't look in the PATH variable'.
The terminal where they are originated. Or specifically "ls" for unix and "DIR" for DOS, the two programs that people used to use to see, what's in a directory. See, hidden files don't really exist. They can be opened and used just like any other file. But when people make more advanced file mangers, they followed the conventions set by "ls" and "DIR".
And the reason this only happened on unix: Only there "." and ".." were not just symbols, with special meaning, but actual existing folders (hardlinks) in the filesystem. They were automatically created by the system, but it theory you could've changed ".." to be an actual folder and not a hardlink (all you'd need is edit the inode number in the entry). That's why "ls" needed to skip them, because they were actual hardlinks that existed in every directory.
ls didn't "need" to skip them, it just made sense that as they in every directory you knew they were going to be there, expected them to be there and so didn't need to told they were there. There would be times when you would possibly need to know about them, particularly their attributes, so a mechanism was added to show them by the -a (all) flag. As people made use of the feature of a file name not being displayed if it started with a dot, the -A flag was added which did the same as -a except that dot and dot-dot are still not displayed.
@@cigmorfil4101 My point was that "ls" needed special code to if it wanted to _skip_ "." and "..", while DIR needed special code to _add_ them. And that every file manager after that copied the conventions set by ls and DIR.
Thank you for this stimulating story.
Out of force of habit, I always insert A (or a, if I'm lazy) to ls options, and suffer the cruft in ~.
Any excuse to type lart is worth it.
How about `ls -FArt`?
I keep dealing with a recurring bug involving a cheap @$$ embedded browser that keeps breaking. The developers had asked ‘can we do this thing?’ And sure enough they could. Then they gave it to users. They never stopped to ask ‘Should we do this thing?’ So now we have a mess of a browser we don’t own that we can’t predict failures for, but about every other year just randomly quits working for some sites until it’s upgraded. And if we remove it, the users will scream at us. But if it had never been added, we’d not have the associated mess it’s created.
Realise that . files are also not expanded by the shell when using *. Quite useful since rm -rf * would otherwise include ..
I CAN'T STAND the sludge in my home folder. It drives me BATTY, every time software supports files in both home and elsewhere I ALWAYS pick elsewhere. As a software developer who has written(popularish?) Linux software I made sure my software follows the XDG spec because it's SO ANNOYING when software doesn't.
Thank you for your service.
the problem don't have a way move those file a way to certain folder
I think they are useful for people not to mess with important files but to also be in a easy place to find. You shouldnt end up with that many but i do rebuild my pc every 2 years as i find upgrading xubuntu always causes issues over a fresh install
Bro took learn from mistakes too far, now we have to learn the specifications of 6.02 × 10^23 configuration files
... I like using dot prefix to keep the important parts of a directory on the top.
There is also the .hidden file that can hide non-dot files, this way you can hide things that don't have a dot but cannot have their name changed without breaking something. I bet they hate that addition as well :)
I was already aware of the story, but it amuses me.
Having said that, I think having dot files are *very* useful -- sure, directories and setup files can be explicit rather than hidden, but I have dozens of directories I personally created, and sometimes struggle to recall why I created them -- the situation would be even worse, if we didn't have dot files! If a file has a dot, I can expect it to by a system file of some sort, which sometimes helps me know I can ignore it, and sometimes helps me to know I need to focus on it.
Regardless, dot files -- like all files, really -- could certainly benefit from being well-named!
I have a time machine. Your \ls idea created major issues back in 2094.
Made me think of the sega genesis. Some games are harder to emulate and even wouldn't work on the newer consoles because they were programed around a glitch with the original console design that they fixed in later versions.
IMO, user config files should be located in $HOME/etc . The base OS config files are in /etc , then there are, in theory, /usr/etc and /use/local/etc , so finally there should be $HOME/etc . Makes sense to me.
I like dofiles simply because it allows me to hide the system level files, when I'm not actively working with them and makes my screen less cluttered when working with files.
I think some home folder bloat would go away, or new conventions would emerge, if dotfiles weren't a thing.
But home folder shortcuts are really easy to "abuse" as a developer.
Some of the hardest code to maintain and test, especially for cross platform apps, is anything todo with files and folders.
All modern OS have home folder paths and shortcuts, so its easy to define your own app folder at the users home.
Otherwise, i gotta maintain locations for linux, Mac, and windows All of which have different standards and specs.
I honestly don't get why people are complaining. It's still better system than Windows, but why are people working directly in their home folder? That's what subdirectories are for and then you won't see dot files unless you explicitly create them. For that matter, I put commonly used directories in my ~/.bash_aliases so I can `cd $g` for instance to play a game, and since bash is set to auto-complete through shell variables I can navigate to a particular game rather easily.
macOS also uses dot files to hide files and folders. But it’s almost always system related stuff. Apple does it to prevent you from interacting with them. You can view them without much trouble though.