How Microsoft Made A Mockery Of Government Standards (ft. POSIX Subsystem)
Vložit
- čas přidán 29. 07. 2024
- Support the Channel
Ko-Fi: ko-fi.com/ncommander
Patreon: / ncommander
Come Hang Out:
Discord: / discord
Mastodon: social.restless.systems/@ncom...
Wiki: restless.systems
Research Notes PDF: drive.google.com/file/d/1Mzl8...
Thank you to BlackCoffeeDrinker for editing!
Was originally called: How NOT To Implement the POSIX Standard, Featuring Windows NT ...
--
There's a lot of half assed computing in the world, but one of the larger cases of when Microsoft implemented the POSIX standard. Originally created to help increase standards compatibility, Microsoft's implementation of the POSIX.1 specification is legendary in just how utter trash it is.
Stepping once again into the aegis of NCommander in Realtime, we're once again going to dig out what it takes to build POSIX mode applications for Windows, what exactly the POSIX subsystem is, and see just how much Microsoft half-baked their implementation.
Strap yourself ins folks, this one is going to be a wild ride.
Music Tracks As Used In Order:
- Atlantis - Audionautix
- Icelandic Arpeggios - DivKid
- To Pass Time - Godmode
- Day Sparkles - Geographer
- Light-Gazing - Andrew Langdon
- Absolutely Nothing - Jeremy Blake
- Aioli - Andrew Langdon
Atlantis by Audionautix is licensed under a Creative Commons Attribution 4.0 license. creativecommons.org/licenses/...
Artist: audionautix.com/
#ncommander #retrocomputing #windows - Věda a technologie
CORRECTION: The resource kit was apparently a lot more available than I realized. I knew it was in the book, but I didn't imagine the book was *that* common. I never saw it at any of the shops I worked, but that's a small sample size.
I worked as admin in a small lab, the Resourse Kit seemed to be standard must have at the time. The books filled up half a shelf, and occasionally helped, but that's not why you bought it.
I remember the NT Resource Kit well. I was an NT admin back in the 3.51 / 4.0 days and that kit was indispensable. The book was also riddled with errors - documentation that was flat-out wrong. "It works like this!" No, Microsoft, it does NOT work like that.
This isn't just half-baked, it's god darn frozen.
This beast's still walking in the forest, that's how unbaked it is.
And it wasn't even a dessert.
Its not even there its literally just assorted piles to create a beast to walk around
@@RemnantCultThe beast's name? Windows NT.
it was just a jump for money, so why bother even unfreezing it?
I've always kinda knew this subsystem existed, but wow I didn't realize it was this cursed. Crazy to think that presumably many many hours were put into designing the subsystem, only for it to be basically unused. I wonder what it was like for the Microsoft developers back then.
NT has a native subsystem capability that was leveraged for this. Microsoft simply added another subsystem, although it certainly took a long time to get it going. Maybe there were plans to improve it, maybe there wasn't
Microsoft replaced it with "Windows Services for UNIX", which was SUS compliant (probably not getting away with that half baked POSIX bs anymore), and then WSL1, and then WSL2 (which is literally a linux VM running inside hyper-v with some clever stuff to get it communicating with windows)
Btw, NTFS is also case sensitive, fwiw, windows just...kinda kludges around it
The devs probably not only knew but fully supported developing this "malicious compliance" POSIX subsystem.
Those "many hours" were spent figuring out how they could do as little as possible
I guess for people wanting to use the POSIX subsystem back then, enterprise grade customer support from M$ was basically a necessity.
It's been a long time since I had to fool with it, but iirc "enterprise support" didn't get you far and in our case, lead to migrating to an entire MS shop against the wishes of everybody who didn't have their hands in the piggy bank. Which if you boil it down, that seems to be the intended result anyway.
@@randomtidbits7695 can confirm that this was exactly how it went at the law firm that I used to do IT work for
Enterprises which wanted to use POSIX software on Windows NT were very rare and those that did used one of the variety of 3rd party solutions on the market and not the POSIX subsystem. These 3rd party solutions mostly targeted software companies looking to provide their product to Windows only shops. Most POSIX software running in Windows back then was 3rd party software from these companies and the terms of the POSIX solutions allowed distribution of their software with the POSIX runtimes.
M$ does not support.
There were some major bugs reported to Microsoft in their SNA implementation (another embrace and extinguish, but then again, who really wants SNA).
Bugs were send with fixes and reproduction instructions to their enterprise support.
6 months later a reply came that they found it was a bug and that they will look for a solution.
That's how enterprise grade their support was.
@@bloepje SNA Gateway was a particularly bad product. Many of my employer's customers at the time unfortunately didn't discover this until replacing the reliable Novell SAA with it. It was so bad but also so key to my employer's customers accessing our primary product that eventually my employer was able to strike a deal with Microsoft. They loaned some of their developers to Microsoft to help improve SNA Gateway.
Besides the poor quality of that particular product my experience with Microsoft support was ok, but at that time my customers were also very large enterprises or large government agencies which had large expensive contracts with Microsoft in order to get a high level of fast support.
The POSIX subsystem for Windows was superior to WSL in one very important respect: the name wasn't backwards like "Windows subsystem for Linux" is.
Yeah, WSL sounds like a Wine competitor, not the opposite 😅
Yeah, I was going to say... a Windows subsystem for Linux sounds like a way of running Win32 applications on Linux.
technically i think the grammar is correct, but regardless it does intuitively sound backwards indeed
Windows Subsystem for Linux
the Windows Subsystem (the subsystem that exists within windows) that is used for running Linux
@@shallex5744 yeah, I get that’s what they were going for, but it’s still backwards. Anywhere else, the noun adjunct to “subsystem” describes what the subsystem does, and “for” indicates what subsystem is meant to be a part of/run on. It’s not a Switch emulator for N64, it’s an N64 emulator for Switch, for example. Even if the grammar is technically ambiguous (it probably is, English is not especially precise), WSL is still strictly backwards to the standard convention used in any other similar context.
@@Thanatos2996 yes, it is a very strange name choice indeed
I worked with the NT beta versions and release versions from 3.1 onward, porting a product which ran on various Unixes. We initially thought we'd be able to use the POSIX subsystem, but it was quickly obvious that this was nothing but a box ticking exercise. Our product which ran on NT, on MIPS, alpha and x86, was completely win32 with no use at all of the POSIX subsystem.
I worked on a project in 1995 where management had the vision that we should have a common codebase for Unix and NT. The team actually tried to use the POSIX subsystem whit the same success as showed in the video... Our solution was to write our own libraries as a version of pthread for NT and not to tell management. 😁
Embrace Enhance Extinguish
Let me guess… outlook express?
@@vkvishnu Nothing remotely like that. My work was on a PostScript interpretor for industrial printers (film recorders, A0 sized printers etc). Our product already worked very well on a range of UNIX workstations (SGI, Sun, Digital etc) and porting to other UNIX machines was easy enough because they had a common (POSIX + X11) API, but the Windows port was very much more complicated than expected because the POSIX subsystem was basically completely useless. Therefore we had to write a load of windows specific code.
The UNIX API begun in the early 70s, when the system was ported to C. POSIX provides a standard for this API, as many UNIX systems got further and further apart from the AT&T sources over time. Currently POSIX means at the very least you can run console apps on Linux, *BSDs, and Mac
tbh I'm not surprised this is such a mess, I've never had POSIX accomplish what it was supposed to. Anything beyond basic apps has always required a pile of ifdefs even for stuff like memset/memalign/malloc. They didn't even add endian swapping functions until *2014*... for a standard which is literally centered on standardized IO.
@@ShinyQuagsire it’s way better than nothing though
POSIX is C source code level. Unix API is some kernel calls wrapped by standard library.
@@ShinyQuagsire I think you are conflating Windows adherence to the POSIX standard with the actual POSIX standard.
The reason why WSL exists is because of Windows attempting POSIX compliance.
Yup, posix compat means compiles on both AIX and Linux even though AIX feels like a barren shithole compared to gnu linux
“It just existed so MS could claim POSIX compliance” yeah, pretty much how most people knew at the time as I recall
Back in the 90s I used to work for a company that implemented Windows NT and for a while had the largest NT network in Europe. I was aware of the POSIX subsystem which I understood was present to allow sale of the product to US Government entities that had strict policy on procurement where certain features had to be available regardless of use-case. Another feature was C2 security which Windows NT complied with although security capabiltiy has been enhanced since then. I recall anecodally that some software vendors had made their previously Unix targetted software available on NT using the POSIX subsystem but never saw it myself. I had access to Technet and MSDN so had the resource kit and SDK as well as development tools and there was loads of interesting stuff included. For example the source for the built in screen savers using OpenGL library. At that time Cygwin didn't exist and even doing development on Unix/Linux relied on the runes aligning to setting up Windows for this wasn't vastly more complex!
Yeah, I remember M$ bragging about that C2 compliance from the very start during the presentations of NT 3.1, and how hard work it was to get that certified. Later I found out that this approval was only valid for a certain subset of hardware/software components, and anything with a network (even just LAN) was _not_ C2 compliant. Even back then running without some form of network was pretty useless.
4:56 tar is not a single format and I'm guessing you were trying to unpack a GNU format tar file. With GNU tar, you might have better luck creating archives with one of "--format=posix", "--format=ustar", or even "--format=v7".
I always wondered what the POSIX subsystem option was in Visual Studio's linker options for C++ projects, I guess it still has some support even in Windows 11 and VS 2022, how interesting.
The linker flag specifically sets the subsystem file in the PE header; I don't think it affects anything else besides that, but I'm not 100% certain.
I love the NCommander In Realtime series. Your livestreams are a bit long for my attention span, so I do like these re-narrated shorter versions, since it lets me see what curse-ed things you're doing... and I totally love the mindset. "This thing is available, I wonder if it's usable, it's probably not but they said it worked so we gonna try it". I always knew the POSIX compatability was kinda trash but you've shown it's total trash, thank you for sinking hours into this, great to watch!
My short attention span appreciates these live stream summaries, too.
The literal definition of malicious compliance.
Ironically Windows NT is a fork of OS/2 there joint project with IBM. It's likely it's posix compliance was also this messy
@@JeffreyPiatt NT is not a fork of OS/2, its a nearly 100% independent codebase
POSIX itself is an example of malicious compliance though, since it was all about taking the UNIX API and whitewashing it as a standard for all OSes despite it being obviously intended only for Unixes. They even made the acronym rhyme with Unix, just to rub it in. Which is why POSIX is so limited, once they removed all the bits that were too Unix-y, there wasn't much left. Microsoft made a good call by implementing just the standard to point out the ridiculousness of the standard.
@jankrynicky no, it is literally a fresh reimplementation, the only thing taken directly from OS/2 is the command interpreter
Perhaps *that's* what MSWin's "Malicious Software Removal Tool" is for. Because I install it on a MSWindows system, yet MSWindows is still there afterwards.
POSIX on Windows is so funky, the default C library (msvcrt) fools you into thinking it might be POSIX compliant until you actually try to use it, add the 10 official implementations of POSIX, each for a different version of Windows, third party libraries like Cygwin and MKS, as well as Microsoft simply integrating a Linux kernel into the OS.
How’s POSIX on Mac?
@@MaxOakland _so_ much better. fully functional and works like you would expect it to.
@@adora_was_taken That’s awesome!
@@MaxOakland macOS is weird since it combines Old Macintosh, NeXT, Unix, and Mach operating system heritages. Macintosh gives you the GUI style and file-systems, NeXT gives you the premier programming API, Unix gives you the more standard C API and file-system structure, and Mach gives you some of the kernel and executable file format
Uh, assuming it did pass the NIST test suite, then the problem was with one or more of: the standard, the test suite, or people's expectations. Undocumented, 'soft' standards are a problem.
Agreed. If the government wanted to be able to build rogue on Windows NT, the requirements should have included the pieces needed for it. I can't fault Microsoft for not choosing to go above and beyond what the specified requirements were.
Looked up the WeirdNIX contest in old Usenet posts. One of the examples in Donn Terry's 1986 announcement of the contest pointed out that the existing specification considered a single dot to be a valid return value for getcwd().
Oh awesome, which group? I went looking but couldn't find anything but references to references.
Considering that at that time Microsoft has already have enough experiences with developing XENIX, the half-baked POSIX implementation were deliberate moves by Microsoft.
Reminds me of the time when Java applets were a thing, Microsoft shipped their very own Java Virtual Machine. It was so bad that Sun sued them and eventually won.
This lawsuit eventually brought us C#.
The POSIX subsystem is definitely not "batteries ready" UNIX, but you should consider the amount of heavy lifting Microsoft did to make the NT kernel compatible with UNIX architecture. For instance, process creation is very different in Windows and UNIX because Windows has no notion of forking. NT therefore supports an efficient implementation of forking, using copy-on-write, even though Windows didn't need it. Furthermore, NT had to be designed to permit both a case insensitive (for Windows) and case sensitive (for POSIX) naming conventions.
I'm fairly confident that NT would have a significantly less complicated kernel if POSIX was not a requirement. They probably wouldn't even have gone with the server/client architecture of having different subsystems if not for POSIX. At the time NT was released, the only subsystems were Windows, OS/2 and POSIX. If they dropped POSIX support, I'm pretty sure they could have found an easier way to do just Windows and OS/2.
Even though CYGWIN is a "batteries ready" UNIX, it doesn't take advantage of any of the NT kernel's built-in support for POSIX. This means in principle if you have a UNIX program that does a lot of forking (like a web server), it should run much more efficiently on NT's POSIX subsystem than on CYGWIN.
In essence, Microsoft went to the effort to make the cake (everything difficult about POSIX, they implemented) but they didn't bother to frost it or decorate to make it look appealing. It was of course designed for government contractors who presumably had the ability to code up whatever bells and whistles MS didn't provide. One wonders if some government agencies were making use of this back in the day, and we just never heard about it.
The idea the resource kit was obscure or hard to get a hold of is just incorrect. If you wanted the book along with it every bookstore carried it in their technical book section. It was advertised and well documented in magazines of the era. It was available for free via the Microsoft BBS and the Microsoft FTP site. It was very commonly installed on NT systems.
I worked in IT in this era. I remember the 9x ones. The NT one was rarely, and it didn't note that the POSIX tools were included. The book itself doesn't even list that; I had to check the disc myself. The book just lists traverse checking has to be disabled, and nothing else.
@@NCommander I worked in IT during this era. Your recollection maybe that the NT resource kit was rare but it certainly isn't mine. It was everywhere.
My brother and I had the NT 4resource kit though he had to order it. A few years later I was able to get the 2000 resource kit but that was a lot more common..
Fun fact : Linux isn't actually POSIX-compliant. Stallman thought some of the decisions of the norm were stupid and never actually got the certification. You CAN make it essentially compliant with the variable POSIXLY_CORRECT or POSIX_ME_HARDER, but it's not official, making macOS the only fully POSIX OS outside of "real" Unix like AIX or HP-UX.
"POSIX_ME_HARDER" is the funniest variable name I've ever seen.
@@LonelySpaceDetective For some reason, I want to add DADDY at the end of it.
A couple of Linux distros (specifically, EulerOS from Huawei and Inspur K-UX, both RHEL derivatives) got UNIX 03 certification at one point, proving that Linux indeed _can_ be made "fully compliant" if needed.
There have been 2 UNIX-certified Linux distros tho.
There were few Linux distributions that passed Single UNIX Specification tests and were registered as UNIX systems.
The resource kit was readily available. MSDN subscriptions included it. I do agree with you 100% on the posix mess that MS shipped.
I put a pinned comment about this.
A TechNet subscription (for SysAdmins, not developers) should have also provided the necessary resource kits (but not SDK's).
Here I am, (in Greece)helping with decorating the New Year Cake while taking breaks to watch your contribution to Computer Necromancy. Excellent, fact-packed video.
May 2023 bring more of this. You are a gem.
"Computer Necromancy", I love this term and will be stealing it 👍
No random and srandom, so no proper C Standard Library support or a really broken build environment. With no networking and very limited interop(ooh shared filesystem), They really didn't want anyone using it. Meeting the bare minimum for government contracts, just like any business would. WSL 1 pretty much ended up doing The Posix subsystem the right way, aiming for actual binary compatibility not just poor-quality source compatibility.
WSL1 isn't binary compatible, it is syscall compatible. It also wasn't the first time it had been done either. Circa 2002 out of MS Research a similar project was done which could run Linux ELF binaries on a Subsystem. To demo the ability they were running an Apache webserver and had Firefox accessing it. It was never going to see the light of day with Balmer leading the company. WSL1 is more like WINE in that the syscalls were handled by Windows and translated on the fly to an equivalent set of Windows API calls. WSL2 is more like And Linux or CoLinux where the kernel is actually running in a VM and is isolated in its own Hypervisor. Each have their own strengths and weaknesses, but WSL2 with how it visualized the file system and hardware is more resound and can keep up with Linux kernel changes in a way that WSL1 can't. The magic is how they've bridged WSL and Windows so that they are both working together with only a few seams if you know where to find them. For most of my Linux needs, Windows is the best Linux system I need and it is easy to switch between different Distros.
POSIX on the other hand. It was available for very custom work and not expected for more than supporting those very bespoke customer solutions. The subsystems were initially designed to support more than the Win32 API, but with IBM and DEC usage being significantly anemic, OS/2 and VMS compatibility wasn't a priority. NT 3.1 and 3.5 were intended to offer more, especially on the POSIX side of things seeing as Dave Cutler and many early NT engineers had built VAX for Digital before coming to Microsoft. The bones were there and have been from the beginning -- NT kernel was going to run Win16, Win32, DOS, OS/2, and POSIX applications, each in their own environments -- it was soon clear that Win32 was the Subsystem to design around.
I think it would be interesting for Microsoft to revisit supporting more subsystems. There'd be a great MSR project to make modular environments to host things like a Node/Deno container, one for .Net CLR, Android, or a modern LISP machine, with a Win32 host machine providing a way to keep those other systems running in isolation. It would look a little like Docker Containers in some ways, or a NT-based Kubernetes OS. I think Windows 10X was sort of heading that way with Win32 apps running in their own container, and I think it is still a concept worth going after. WSA and WSL demonstrate that it is possible, but it still needs refinement.
The windows NT networking stack meets POSIX by including the required network diagnostic apps ported directly from BSD very early versions of Windows ipconfig would display the Berkley copyright with the proper switch in DOS prompt so they were compliant with the BSD license.
That Windows NT networking stack was in fact copied directly from the BSD one.
This might be interesting. The subsystem I've heard of existing ever since I learned about NT4 in my teen years, yet never heard about anyone using in the real world.
Because, for all intents and purposes, nobody did.
I saw one extremely edge case once, but that's it.
Also, it was executing a file that was not .exe
3:51 It kind of reminds me of early "compatible" computers that could run MS-DOS applications, but no real software because programs didn't use DOS/BIOS calls for display etc.
Now, I also remember there was a company who made something that rounded out the POSIX subsystem and made it truly useful. MS _did_ provide the critical foundation -- stuff that must be implemented in the OS itself -- of having a 'fork' function and providing for case-sensitive file names.
Windows has the equivalent of fork(), but it is expensive to use.
For example, back in the days when Git was becoming really popular, Mozilla were looking to switch to a DVCS (I think they were using SVN or something). But they had a requirement that whatever it was would work well on Windows. And Git, at that time, made too much use of fork/exec to work nicely on Windows. So Mozilla went with Mercurial instead.
Git has been modified since then, to tone down its appetite for spawning processes, to cater better for Windows installations. On Linux, of course, there was never a problem.
Cygwin has a spawn() function that basically emulates fork/exec with native Win32 calls in the event that that's what you really want to do. But it does provide an implementation of fork that's kind of hilarious in the lengths they had to go to. Obviously it's incredibly slow, but the best part is that because of address space randomization it can sometimes just fail if Windows by chance decided to stick a DLL in the wrong place first. It will retry a finite number of times but there's always a remote possibility that it will just have to give up eventually.
@@lawrencedoliveiro9104 On early Unix systems fork() was also an expensive operation. Since every writable memory page in the parent's address space would be copied to the child.
Hence there was vfork() which would share the address space and suspend scheduling of the parent until the child called exec() (or exit()).
In more modern systems writable pages are shared and marked Copy on Write.
Nice! Wasn't this a big deal at the time? This is crazy stuff! As an economic history researcher, I swear this story deserves a paper, if there isn't one already. Subbing right now! ❤
Microsoft: "See, look how terrible POSIX is. You should just give us a no bid contract to deploy Windows APIs."
Windows NT is still Posix compliant like it's sibling os/2
@@JeffreyPiatt OS/2 was not POSIX compliant.
"it's" is a contraction for "it is". "its" is the pronoun.
@@JeffreyPiatt NT had a os/2 subsystem, BTW
@@adg1355 Only for 1.x text mode programs. Which was actually super useful because OS/2 1.x was a better DOS than DOS, much better for software development. Many dev tools were bound to work under both DOS and OS/2 1.x.
They really did the absolute minimum they had to do to be certified. Shockingly honest.
I'm pretty sure cc.exe and make.exe exist entirely to run the NIST test suite.
It's a miracle those aren't just one massive if-else block that spits out fixed machine code that passes the test depending on the input file name.
@@NCommander running NIST tests suite incidentally is a good candidate for “something to do with it”
On a technical level, it would've been impressive if it was completed and documented properly. The NT Kernel has support for subsystems besides Win32. Would've been fun to actually use it. Ah well.
That was Services for UNIX and Windows Subsystem for Linux v1.
Wsl is a more 21st century solution with containers and virtualization
I was but a starting developer in 2001, but this is an eye opener! Glad M$ came a long way since then - through no choice of their own it seems.
The real punchline is that Microsoft developed and sold a very popular commercial UNIX versions of the 1980s: Xenix. Even considered pushing it as a replacement for MS-DOS... But then AT&T started selling UNIX directly, and Microsoft recognized pretty quickly it couldn't compete with the company it was already paying licensing fees to, and decided to work on a different successor to MS-DOS - OS/2.
As for Xenix? That got sold to a company that was co-developing Xenix with MS: The Santa Cruz Operation, or SCO...
@@watchm4ker I would love to get more history on the NT name, some say it actually came from a version of Windows developed for the Intel i860, code-named N10 ("N-Ten")
@@Rouxenator That, I don't know. I know it used to stand for New Technology, at launch. And that it's... Basically the original version of OS/2 3.0, after Microsoft decided to abandon THAT partnership.
@@watchm4ker There's no relationship between OS/2 Warp and Windows NT.
@@watchm4ker Ah yes, back when getting a Unix box at home was scathingly expensive. Linux really did change everything.
FWIW I dont think there was any malicious intention in "sandboxing" the posix subsystem. In Windows the POSIX subsystem sits above the kernel just like Win32 and OS/2. I dont think this was anything malicious just Dave Cutler drawing a natural boundary around the kernel and implementing what was needed. I worked on this operating system and I can tell you that there was a TON of stuff going on in Windows at the time and it was simply the case that OS/2 and POSIX were very quickly not worth spending developer resources on.
No, that's completely fair. I think it got left out of the actual cut, but in the original notes, I actually did compare it to the OS/2 one, which can access NetBIOS, and IPC structures, and has more integration. Like, you could actually use the OS/2 subsystem as a migration path from OS/2 server to NT, and Microsoft's own pre-NT 3.1 early NT toolchain was OS/2 based and ran through the subsystem.
@@NCommander Totally. Around this time we were building Window 95 with direct X, then merging all of that code into the NT codebase to produce Win2k, figuring out how to service the OS, doing the IE thing, I mean the origin of Windows was a rocket ship. It was a very diff env from today, tbh noone particularly cared about POSIX. Linux was nothing even close to what it is today (a trajectory which changed dramatically when Google invested in Android) and it was definitely not a high priority. The situation has changed 1000% and WSL is a much more committed attempt and IMO absolutely critical for Windows' - survival given the importance of Linux today dev machines need to be good to work with Linux and OSX has an advantage there. But IMO Windows has made great strides here e.g. passthrough GPU virtualization for WSL usage is a major deal for machine learning workloads. Anyway im rambling....
Awesome video, awesome channel, thank you - subscribed!!
I remember using 4DOS at that time. I don´t know how much POSIX-compliant it was, but it extended a lot the scripting functionality of DOS and OS/2.
I just found your channel and watched about 3 years of content in a week. Hope you're doing well and have more fun in the works :)
I remember popping in for a bit during the live stream. Didn't realize you had to use nmake at one point... I'm so sorry. Happy New Year!
Windows terminal stuff has always been atrocious and it was the first thing I thought of when you mentioned Rogue.
Even now it's quite bad, though it has improved significantly. It's still decades behind
SSH wasn’t enabled by default in PowerShell until W10 1809. It was a huge leap forward.
Very cool video...didn't know a posix subsystem existed for windows NT...I only found posix on c functions that looked like yea that might exist on say unix...and it would say depreciated...and I was like ...yea I am using that...for it was something simple I was familiar with...just needed to refresh on the parameters, etc. I guess not knowing about this was a good thing...kept me from attempting it when I was moving from DOS to Windows NT...and didn't know much about how to get there...I found the win32 library interface and I was like ok...winner going with that...worked out ok...well less painful than anything like this.
Amazing stuff and very informative as usual.
Also lol'd @6:50
Hi.
What I remember, but back in win2k world and appropriate MV VS, there was an option to configure a project output to be for Windows POSIX subsystem, as just one setting. I tried it literally once or twice, it worked, but there was absolutely no practical usage.
11:27 Remember that Windows didn’t have a decent terminal emulator anyway (and as far as I know, still doesn’t). Microsoft didn’t like the traditional idea of sending inband escape sequences to trigger terminal functionality like cursor addressing, text attributes etc. So they developed their own terminal API that required special system calls to control all these things.
And, surprise surprise, nobody could be bothered writing software to use these special system calls.
They do not. And haven't had a CLI text editor since 'edit' in the 16 bit days
No wonder WSL is a thing, CLI workflow in the most basic sense is basically non-existent
Strange and incorrect characters in the stream? Sounds like logging into a non-Microsoft IRC server using MS' Comic Chat.
@@RadikAlice Yes they do have a terminal (console host, using pre-11 nomenclature), a bunch of improvements was added to conhost in 2019, and now Windows Terminal is default on W11
Does this have something to do with me not being able to get Python's ctypes library to recognize files that use pthreads?
POSIX subsystem was never mean to be a replacement for Unix but a runtime time environment for applications that had been designed to be posix compatible those applications didn’t make use of all the extended c libraries , Unix tools etc because by design they were meant to be posix compatible. What most commenters and the author should remember is that before nt no other major operating system had ever offered completely different runtimes running on the same kernel out of the box. This allowed specific customers to run these applications side by side with general windows applications.
Remember that every other posix compatible os offered it natively but could not run other runtimes requiring customers to have multiple systems to achieve the same results.
And the windows resource kit visual studio and related documentation was included with every enterprise level msdn subscription which back then was included in major enterprise agreements with Microsoft.
At the first company I you just walked into the it administration office to get these cds.
So a lot of the downsides mentioned in this video are really only applicable if you were a hobbyist to smal enterprise in which case this function wasn’t designed for you.
100 times this ^
POSIX isn't there to run some nerd's *nix nethack, it's there to run POSIX compliant software. That's all.
I think folk confuse _most *nixes are POSIX compliant_ with _all POSIX machines should be same as some *nix_
POSIX is designed to be _Unix-like_ , not _to be_ Unix. It's very basic, and that's the point of it.
[ and aye, as a software dev since the 80s, every company I've worked for from aerospace, thru shipping, to pure IT, has had a full MSDN if they were an enterprise business, it's the way it was - I've got folders and folders of the stuff in my spare room :) ]
Technically for a short while, Windows had: NTVDM - 16 bit, POSIX (lol), Win32 together. Technically also could run DOS apps.
What is the background music called, beninning at 0:04?
You mentioned a book "The Unix Haters Handbook" in a video of yours, one of the solaris ones maybe, anyway, I had to tell you this book is absolutely hilarious. I haven't laughed this hard reading something in years. Thank you.
I bought a copy of Services For Unix back in college where I used Unix and NT workstations in the engineering computer labs and ran Windows on my dorm PC. Writing portable c++ code back then was frustrating, especially not standard Unix filesystem structures for headers/libs.
The Windows subsystems for OS/2 and POSIX were supported up to and including Windows 2000. I used the POSIX subsystem on Windows 2000 back in the day, and was fairly satisfied with it, except that it didn't support fork(2) (the call returned -1 with ENOTSUP / ENOTIMPL). Since I needed an implementation of fork(), I created one in native Win32 code (which was pretty ugly but worked). BTW, most of the tools and documentation for Windows POSIX were available on the Internet by the time I used it. The Windows SDK was downloadable from Microsoft as well.
I really like your channel, it has V I B E S. IDK how else to put it. Something about the music and the calmness of your voice and the highly interesting subject matter. Really great, thank you! :)
And I thought the RS/6000 shipping with reference implementations of core features was bad (which it is)
but this makes that preferable, I bet that if Linux and open source hadn't taken off in the business world
we'd still be dealing with this kind of unsubtle yet subtle enough middle finger towards UNIX as a whole
“Unix” is a trademark that only proprietary products can afford to license. And all those products are either dead or dying right now. What matters nowadays is not “Unix”-compatibility, but “Linux”-compatibility.
@@lawrencedoliveiro9104 While I understand your point, the wording made me roll my eyes and kinda want to say _something_ out of spite
It feels insufferably pedantic, but maybe that's just a gut reaction from a bad first impression
The problem with IE4/Active Desktop is just how resource-hungry it is compared to the original incarnation of Explorer. Plus, on 95 at least, the shipped IE 5.5SP2 runtimes make it leak memory. To each their own, but anywhere the original is an option, I take it.
Back in the day I used active desktop to have a live stream of some kittens playing in a cat shelter stream from a webcam as wallppaer. It used flash and you had to restart explorer everyone once in a while to fix the memory leaks.
Now that I come to think about it, shouldn't WSL be instead called the Linux Subsystem for Windows (LSW)?
UNIX and companies that sold UNIX were their own worse enemy. One of the reasons it was the POSIX standard and not the UNIX standard. X-Windows is another such standard that was crippled from day one. Many UNIX based companies also were not POSIX compliant. Microsoft was hardly alone in that regard.
The world wanted a graphical OS that didn't cost $2000 per year per work station. IBM, HP, Solaris and the others charged ridiculous prices for UNIX and hardware. Linux at the time was a hobbyist toy and wasn't an option. Today people get software almost free but in the 1990's it wasn't cheap and Windows NT cost 1/10th of other OS's. Not a small thing when you have 1 million employees who need computers. POSIX might have been a government specification, but the reality was it was only used by a few people in IT. 95% of computers were used for email, Word, Wordperfect, Lotus and Excel. Managers and IT departments wanted a single type of OS to support, or at least that seemed the vision. The reality was/is that different use-cases required different hardware, different OSs and different IT support.
Even at the time people were aware of the limitations of NT and POSIX. They didn't care as long as the legacy box that only mattered to the old guys running mainframes was ticked. Remember, UNIX was really only run on mainframes or very expensive workstations at the time. Google wasn't even a twinkle in the eye of Silicon valley.
It was only later in the 2000's when Linux was mature enough to save UNIX from obscurity that UNIX compatibility became important. Today things are much different. So it isn't really fair to judge MS from the 1990's. POSIX wasn't all that important and it looked like UNIX was going to die a slow death. If not for Linux, it probably would have.
6:43 in Windows 10 and newer, you can enable or disable case-sensitivity on a per-directory basis now
that is cursed
Sounds like a super convenient way to pull out your hair at some undetermined time in the future! Thanks Microsoft!
@@lucass8119 It's actually really useful for debugging when you are doing iteration on Windows but the target device uses a case-sensitive filesystem. You can set the folders as case-sensitive and catch those bugs early before trying to deploy and debug on the target hardware.
Would love to see a comparison to the later Unix subsystems in NT: SUA/Interix and Windows Services for Unix (SFU). And maybe a side look at Microsoft's own standalone Unix implementation, which also influenced these efforts: Xenix.
It's on my TODO list, although I've covered xenix before I did youtube: casadevall.pro
Windows has another POSIX subsystem called SFU,any idea to try that?
It's on the topic list.
Nowadays, this could be seen as perfect example that a company changes according to their CEO and its decisions.
I dunno. I don’t trust them with Linux. I’m sure they’ll ruin it to promote their own stuff if people get too comfy with Linux Subsystem
I saw mention of the posix system in Windows NT back in 1993. It never even occurred to me to go check it out, since I had access to Unix. What would have been the point?
That's what a lot of people have told me. SFU was really when it was taken more seriously.
@@NCommander I've been in software for 40+ plus years, an I have been burnt many times by emulating other peoples environments. Its always a moving target, and its really amazing how common it is for programs to skirt the rules of an interface and take advantage of undocumented quirks of the interface. This is one reason why I am not a fan of either Wine or the Linux on windows subsystem. If I want the Unix/linux environment, I'll just go use Unix/Linux. If I want a program to port to a different environment, then recompiling it for the new environment is the way to go. For programs that cross multiple platforms, that's why god made cross platform porting kits. Put it this way: a program designed for porting via a kit had portability in mind from day one. A program written for a given interface (Windows/Linux/Mac) was designed to be bedded in cement from day one.
I'm sort of tempted to get a VM setup with the POSIX environment on my end so that I can try and run neofetch in it so my guess is that I would have to get bash working in this environment
Thanks must go out to the FOSS Community for WINE and Bottles. Kudos to Valve for Proton!
Now we just need ReactOS to proceed past pre-Alpha stage...
There also seems to be an OS/2 16bit character mode support in NT up to Win2000
Is this the reason Windows has system32/drivers/etc/hosts? I've always thought this implementation seemed very at odds with everything else in Windows.
The original Windows NT 3.1 TCP/IP stack was a direct port of the one from BSD 4.3 by Spyder Software. I've always assumed it was a holdover from that, although having a hosts file is de facto standard (and might even be in one of the RFCs as required).
@@NCommander The hosts file predates DNS.
On GNU systems, whether you need a hosts file or not actually depends on settings in your nsswitch.conf.
Wanting to say compliant on the box but doing as little as possible to make it so, and making it so damned hard to have any real practicality as a move to make such compliance a thing of the past.
*Babe wake up, new NCommander!*
Trek on NT on DEC Alpha?
Need DEC Alpha hardware. The compilers are all native.
Boo at Microsoft for doing the bare minimum but what'd you expect from them back then ;)
The Resource Kit, however: I don't know anyone that used NT 4 that did not have it installed. If you used NT Workstation at home you were pretty much sysadmin-level to begin with, had a work MSDN subscription, and knew what was out there.
Well in the those days there was a lot of arcane stuff to get a Linux system up and running as well (i still remember modelines for the display driver or wading through long lists of kernel options "fondly"), but at least there was some documentation (read those READMEs) and it didn't have this feeling of malicious intent.
"While I could bore you with more dry, technical mumbo-jumbo" I think the title already weeded anyone that would be bored by that out.
You know, this is a fair point.
Did you try searching old newsgroup archives? That's where I would look for interesting commentary from that time period.
There wasn't a lot, most people complaining they lack parts, or being told it was useless.
Thanks for sharing. 😉👌🏻
Thanks for watching!
I was interested in that thing but never actually had time and motivation to try this out. Now after your video I know that it's not a bad thing that I didn't try. Good thing that it was replaced by SFU which was much more complete and useful. Also I used Cygwin and it was pretty convenient and useful as well. I was even able to run some Unix desktop environments with it and Cygwin/X.
that background music is the exact same as used in the Mountain Beast Mysteries channel (Bigfoor stuff). Weird to hear it on other videos...
It all came from the CZcams Music Library
would be nice to have a stream on SFU to see if it's more sane.
SFU is absolutely much better than this. Wow.
SFU is much more complete and better than this. Not only it provides much more Unix utilities and POSIX API but later versions even contains X11 libraries. SFU is much more close to something made properly than this thing ever was.
What if you strip the NT kernel down to a DOSlike prompt, and try to set it up such that the POSIX subsystem is the only available command set? I don't know exactly how close you can get to this, but it seems like a gloriously cursed idea to me.
In theory it's easy to replace explorer.exe with anything else (for example cmd.exe or some other terminal app, maybe even written by hand to have an illusion of full screen text mode) and stick to that.
Having a "pure" text-mode out of NT would be tricky though and I'm not sure if possible on retail systems, given how system is bound to GUI and various low-level service processes that controls user session & login.
You cannot run POSIX subsystem (PSXSS) without the Win32 subsystem (CSRSS). You can, however, have a pure command line installation of Windows (e.g. the Embedded toolkit allows you to build one), where you can easily replace the command interpreter with sh. Might be an interesting idea to combine with NT Embedded.
@@dr.shuppet5452 It would be probably the most limited "UNIX" system ever 😀But could be interesting with Cygwin (for older Windows) or WSL for newer ones...
@@dr.shuppet5452 You could have text mode Windows installs, if I remember right, all the way until Server 2019, where iirc the option to not include the desktop in the install was removed. Text mode Server 2016 is a very limited experience, as the majority of modern Windows-centric server utilities, or at least the ones I've used, have major GUI components with either minimal or no terminal interface.
Maybe, to commit to the cursed idea as much as possible, the system PATH variables could all be cleared, assuming the system uses hardcoded directories instead of PATH during boot.
@@dr.shuppet5452 Is that a hard dependency, or more of the fact that you wouldn't be able to launch anything? In theory, you probably could create a native mode application (runs before csrss starts) that starts psxss and messages with it, not that it would be a sane endeavor at all, given the absolutely nonexistent documentation on writing native mode applications, and the RE work that would be required (unless you could find relevant bits in the many leaked sources over the years)
(internal details below, probably shouldn't look if you want to contribute to Wine or ReactOS)
Edit: my current findings are the dependencies are mainly on SMSS (which should be already running) and on whatever provides the SAM database, and probably a few others. HOWEVER: smss supports running subsystems under other subsystems, and that's precisely what psxss seems to be. It's not a native mode subsystem, it, as a subsystem, appears to be set up to run under the Windows subsystem. Unlike csrss, smss, which have UMTYPE=ntss, and autochk (which has UMTYPE=nt), psxss has UMTYPE=windows. So you are correct.
I have a feeling that the relevant FIPS standard was written with a close guidance from someone with Microsoft's interest at heart (or in his pocket more likely).
So reaching the bare minimum for compliance was easy, even though the implementation was a complete joke and an insult.
Are you going to do a follow-up covering Interix? That looks like it was more of a serious attempt at providing POSIX compatibility
I will at some point. It's on the topic list.
This video gave me flashbacks when Microsoft decide not only to implement ODF in the most half backed way inimaginable but also publish it's own OOXML format as a standard in the most useless way to make difficult the implementation by the competition 😬😵
Yes, my thoughts exactly. And didn't they send their own people to the standards meeting just to vote for them, and never show up again? I remember that causing a problem because they couldn't get a quorum or something.
IAC, what they shipped was the original version without any of the feedback fixes needed by the eventual standard.
Microsoft itself does not support ISO 29500 (OOXML) properly. Microsoft -Office- 365 still only implements it in “Transitional” mode, since “Strict” mode apparently causes too many compatibility problems.
For a while back in the early days of those standards I remember discovering that Word's implementation of the ODF appeared more true to the official standard than OpenOffice Writer's was. This was because Writer had already progressed to a later version of its native ODF format by the time it became standardized, which had more extended/updated features. I cannot recall that it had any way of saving/exporting to the older but standardized version. Word's way of supporting ODF was rather cumbersome though, as one had to install and deal with it as a separate plugin. Conversely OOXML has always been a mess from its very inception, allowing for including BLOBs in proprietary formats that only Microsoft itself ever has a chance of fully supporting.
Looks a bit like what they did to prevent the netware client from running properly...
Imagine trying to admin this in the pre-Internet days
it's not half-baked, it's fully baked, exactly what they wanted to ship :P
I feel like this could have all been avoid with a generic ‘no, don’t even think about it’ clause.
MS: what if we just….
POSIX: No
MS: But how about…
POSIX: (slaps MS with a wet fish)
its funny you say this because there are actual email exchanges dated 1990 between the posix committee and the NT team in leaked windows source which basically go like this
This may not be the case, but I read that the reason NTFS supports hard links (that is, more than one directory entry pointing to the same file) is that they had to implement this POSIX subsystem. Hard links are _so_ useful. For example, prior to a large scale batch rename, I can clone a directory tree without duplicating the file contents. (This uses cygwin btw.)
I think they were using HPFS at the time
Don't even need cygwin. Lookup "Link Shell Extension". NTFS supports both hard and sym links. And it has for quite some time (NT4!). That plugin has some cool functionality they built on top of the base functionality, so it's definitely worth a look.
@@ccoder4953 hell you don't even need to install anything, the standard command prompt windows ships with has a mklink command for doing this
though having it as a right-click context menu is _very_ handy if you make links a lot
Symlinks were not added to Windows until Vista. Even now, they require special privileges to create.
@@lawrencedoliveiro9104 Actually, we're both sort of wrong. According to Wikipedia:
"Symbolic links to directories or volumes, called junction points and mount points, were introduced with NTFS 3.0 that shipped with Windows 2000. From NTFS 3.1 onwards, symbolic links can be created for any kind of file system object. NTFS 3.1 was introduced together with Windows XP, but the functionality was not made available (through ntfs.sys) to user mode applications. Third-party filter drivers - such as Masatoshi Kimura's opensource senable driver - could however be installed to make the feature available in user mode as well. The ntfs.sys released with Windows Vista made the functionality available to user mode applications by default. "
As far as the special permissions thing:
"The default security settings in Windows Vista/Windows 7 disallow non-elevated administrators and all non-administrators from creating symbolic links but not junctions. This behavior can be changed running "secpol.msc" the Local Security Policy management console (under: Security Settings\Local Policies\User Rights Assignment\Create symbolic links). It can be worked around by starting cmd.exe with Run as administrator option or the runas command. Starting with Windows 10 Insiders build 14972 the requirement for elevated administrator privileges was removed in Windows "Developer Mode", allowing symlinks to be created without needing to elevate the console as administrator. On the API level, a SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE flag is supplied for this purpose"
So, the policy can be changed and it appears Microsoft will soon make the change the default.
I wonder how later editions faired.
Good 'ol malicious compliance. I think I'm preaching the choir here when I mention embrace, extend, and extinguish as MS's modus operandi back then, hell, they coined the term.
As a counterpoint, you should try out OpenSTEP YellowBox NT. It's basically all of OpenSTEP (fka NeXTStep) 4.2, both GUI apps and command-line, but running under NT 4. (I once had it installed under XP, as well.)
It's pretty complete for 1996-97, but good luck building modern stuff. Ironically OpenSTEP 4.2 (including OpenSTEP Mach, the full OS which was derived from NeXTStep and later evolved into OS X) is worse at this than NeXTStep 3.3 in some cases, as the POSIX compatibility is more broken.
We've been having a long discussion on it on Discord, and how it works (which is abusing cygwin). It's on my to cover list as a stream if nothing else.
lol what a complete mess.
Appreciate this kind of dive into such a topic though, it was always something I was vaguely aware of but never researched the true extent of "POSIX compatibility" in Windows NT, never heard of anyone using any such functionality and never used it myself.
I admit though that in modern day Windows it is rather nice to not be constantly thrown off by using Unix commands like I used to be in the old days, I still have no pressing need for things like Linux application compatibility but to not need to make that switch between Unix and old DOS commands is much appreciated, that was quite annoying in the past.
6:04 Being able to traverse a directory (e.g. CD into it or one of its subdirectories) is a separate right from reading (listing directory entries) and writing (creating files) -- I think Unix-likes use the "execute" bit for it, and Windows NT and NTFS do the same in their ACLs. However, looking up this permission for every ancestor directory in addition to the permission of interest for whatever file you're working with takes time, and since the concept didn't even exist on old-technology Windows there was little confidence software would get it right. So all users were by default granted the right to bypass these checks: if they have permission to a file, they have permission to a file, regardless of some execute bit on some parent directory. Needless to say, that's not POSIX-compliant.
While the government would have to revoke that right (compliance!), it's irrelevant to what you're doing so feel free to leave it on.
It actually doesn't work correctly with Traverse Checking left enable. sh.exe doesn't load its prompt properly and there is a bunch of weirdness.
Thank you for sacrificing your sanity so that we may behold this monstrosity. Compiling Star Trek for MIPS Windows sounds like a great idea!
From what I understand, the posix subststem is same mechanism as WSL1 (not wsl2 though, thats just a linux vm)
Indeed. I understand why they went with WSL2 in the end, but WSL1 seems more ... elegant, for want of a better word ...
I hope you don't disappear for 4 months again!
I've been livestreaming, but making video content has been difficult. I've made some headway on a new video, but its been very up and down in real life
@@NCommander It makes sense, don't let my comments build any pressure. I really just wanted to say your videos are awesome!
To court: See bitches, we're POSIX complained
End user compiling hello word: What the f*
Pretty cool stuff, the system architecture seems decently apart from lacking pre-compiled binaries and dependancies. Also pretty amazed you where able to find these resources from decades ago!
Great✌✌
"Tell me you hate standards without telling me you hate standards!"
Microsoft:
"when Microsoft implemented *teh* POSIX standard."
Dave Cutler was a VMS guy, and it shows.
This madness appears to be the inspiration for Gentoo.
hello fellow old nerd.
can you port another one of the bsd-games into a watch for me so i can play it on a watch ? =) i had some help porting it to android but, ,,, its like a game made for a wrist watch with just a few buttons on , i think =) - Norway
I can't believe you actually went through the pains of getting the POSIX stuff to work as far as you did. And knowing the POSIX subsystem was bad I did not know how bad. But that does extend beyond just Windows.
Around '95 or so Unifix was the first Linux distribution to become POSIX certified. Why? Same story as with Microsoft, it opened the market for US government contracts. Did it change Live (for developers like me), the Universe and Everything? I doubt it.
POSIX did provide somwhat of a standard and as such it was somewhat useful. At the same time there were existing de-facto standards such as SVr4, SVr4, BSD and a bazillion of derivates. Just pick the one you least dislike. Talking about dislike. POSIX was not only limited but also in some places weird if not outright broken or even braindead. This was most blatantly expresse in some code by the GNU project which considered POSIX to be too bad to be followed - at least by default. But where desired the variable POSIX_ME_HARDER can be set. Later political correctness struck and that variable became renamed to POSIXLY_CORRECT.
The GNU project apparently is more serious about political correctness than backward compatibility. While even the latest versions still honor the POSIXLY_CORRECT variable support for POSIX_ME_HARDER has been removed at least from df(1).
Meanwhile I think most Linux developers stopped caring about POSIX entirely. Linux is the new standard. Ehh... Family of standards. And being the new 2000 pound goriilla (pardon,penguin) on the market it stomps over the competition.
Porting software to MIPS NT? Sure 🙂In about '95 I receive a Siemens-Nixdorf RM200 R4600-based system for the purpose of porting Linux to it. In that role the system served as one of my workhorse systems for many years. I amused myself for about 10 minutes with MIPS NT before shredding the OS on the system and starting to replace it with Linux/MIPS. Iow I never got to see any MIPS NT software that was not included in a boxed Windows NT package.
Go checkout my "What does it take to port a 16-bit app" video. I actually ported one of the Windows 1.0 sample apps, template, to Windows NT MIPS since you can run it in emulation.
Hey Hey Hey
Just came back for Another ReWatch!
I tnink it's funny that Billy boy was probably the biggest code thief in computer history, and then they have the audacity to claim the TPM standard is to 'make a secure computing environment for the consumer'... what a joke! Just don't look behind the curtain to see how much of your personal data is being uploaded to M$ servers in the background. I think they call that 'telemetry'. Some good folks have written scripts to disable most of that junk, but it seems every time Win 10 has an update, it re-installs all of it, and you have to disable it again. It's pretty scary how many medical facilities STILL use Win XP to this day because they essentially can't upgrade to something newer. We all know how *insecure* XP was, and that's what you're trusting your private medical data to.😳