9 ms·

How the Windows Subsystem for Linux Redirects Syscalls

359 points by jackhammons 10y ago

Animats 10y ago
It's too bad that x86 hardware doesn't do virtualization as well as IBM hardware. You can't stack VMs. That's exactly what's needed here - a non-kernel VM that runs above NT but below the application.
- pmalynin 10y ago
  You can have nested VMs. https://www.kernel.org/doc/Documentation/virtual/kvm/nested-vmx.txt https://www.kernel.org/doc/Documentation/virtual/kvm/nested-...
  - overgryphon 10y ago
    Windows also now supports nested virtualization. https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/user_guide/nesting https://msdn.microsoft.com/en-us/virtualization/hyperv_on_wi...
- pjmlp 10y ago
  Looking at the way mainframes work, with their higher level languages, JIT compilers at kernel level, object databases, type 1 hypervisors, .... It is quite interesting to see mainstream OSes increasingly get adopting all those features.
  - Animats 10y ago
    Very, very slowly. Microprocessors still have DMA instead of mainframe-like "channels", although we're starting to see MMUs on the I/O side. With channels, devices can't blither all over memory and neither driver nor device need be trusted.
- geofft 10y ago
  I thought that a) the conclusion of VMware's "Comparison of techniques" paper [1] was that x86 and possibly everything is Popek-and-Goldberg-virtualizable [2] via binary translation, and b) the last several years of Intel and AMD chips all have hardware virtualization support, including nested virtualization, that made their architectures Popek-and-Goldberg-virtualizable in the obvious way? [1] https://www.vmware.com/pdf/asplos235_adams.pdf https://www.vmware.com/pdf/asplos235_adams.pdf [2] https://en.wikipedia.org/wiki/Popek_and_Goldberg_virtualization_requirements https://en.wikipedia.org/wiki/Popek_and_Goldberg_virtualizat...
deleted 10y ago
[deleted]
smegel 10y ago
Funny they don't mention ioctl.
obnauticus 10y ago
Excellent post, Jack.
l3m0ndr0p 10y ago
Pretty neat stuff. I think that MS should just create their own Linux Distribution & port all MS products. Get rid of the Windows NT Kernel. I believe it's outdated & doesn't have the same update cycle that the Linux Kernel has. Why run a Linux Application/binary on a windows server OS? When you can just run it on Linux OS and get better performance & stability.
- serge2k 10y ago
  > Get rid of the Windows NT Kernel. I believe it's outdated & doesn't have the same update cycle that the Linux Kernel has. Curious why you claim this? What's outdated about the NT Kernel?
  - l3m0ndr0p 10y ago
    Here are some, or maybe this is not part of the NT Kernel... 1. The use of drive letters A-Z for file system access. 2. Creating symbolic links to files and folders, like you can in Unix/Linux. You have to set a setting somewhere to enable this, but there's a security risk. 3. Standard functional/usable non-gui terminal application like Unix/Linux ssh. PowerShell doesn't come close. 4. Ability to SUDO or su Admin like Unix/Linux. Maybe these are not kernel related above, but the OS specific layer.
    - slededit 10y ago
      You are confusing the win32 subsystem with the NT kernel. They are not the same, the win32 layer acts as a translation. Also symbolic and hard links are supported by NTFS, they are just not exposed in the UI. There are utilities to create them if you really want to. The shell itself and the rest of userland has very little to do with the kernel. It seems its the userland you are upset with. Swapping out the kernel won't fix that.
    - asveikau 10y ago
      > 1. The use of drive letters A-Z for file system access. NT has a root directory like Unix does. Drive letters are symbolic links inside a directory called \DosDevices. Granted, this is not user-visible but an implementation detail. The needs of Win32 applications dictate a lot of user-visible behavior. > 2. Creating symbolic links to files and folders, NT supports symbolic links. Open cmd and type "mklink".
dragonbonheur 10y ago
.
- jcoffland 10y ago
  This title would be clearer: Windows subsystem for Linux apps.
- jcoffland 10y ago
  Maybe because the way they stated it it would be a much more attractive technology. Seems like an attempt to regaining ground in the server market. This would be really useful for distributing Windows apps as Linux binaries. It would make it easier to develop from Linux and target Windows. Need the same for OSX.
- coverband 10y ago
  That's the output though, they're not changing anything in Linux. The "Windows subsystem" they're developing will act as the translator to get there.
- pionar 10y ago
  It's a subsystem of Windows for running part of Linux, so Windows Subsystem for Linux. :)
- geofft 10y ago
  "As Brother Francis readily admitted, his mastery of pre-Deluge English was far from masterful yet. The way nouns could sometimes modify other nouns in that tongue had always been one of his weak points. In Latin, as in most simple dialects of the region, a construction like servus puer meant about the same thing as puer servus, and even in English slave boy meant boy slave. But there the similarity ended. He had finally learned that house cat did not mean cat house, and that a dative of purpose or possession, as in mihi amicus, was somehow conveyed by dog food or sentry box even without inflection. But what of a triple appositive like fallout survival shelter? Brother Francis shook his head."
  - stuaxo 10y ago
luchs 10y ago
>As of this article, lxss.sys has ~235 of the Linux syscalls implemented with varying level of support. Is there a list of these syscalls somewhere? It would be cool to check it against the recent Linux API compatibility paper [0, 1]. [0]: http://oscar.cs.stonybrook.edu/api-compat-study/ http://oscar.cs.stonybrook.edu/api-compat-study/ [1]: http://www.oscar.cs.stonybrook.edu/papers/files/syspop16.pdf http://www.oscar.cs.stonybrook.edu/papers/files/syspop16.pdf
- xorblurb 10y ago
  A list of (at least partly) supported syscalls is here: https://msdn.microsoft.com/en-us/commandline/wsl/release_notes https://msdn.microsoft.com/en-us/commandline/wsl/release_not... Not details on which one are fully or partly supported, though.
- besselheim 10y ago
  You piqued my curiosity - just made one by extracting the syscall dispatch table from lxcore.sys and placing it alongside the Linux syscall list: https://goo.gl/QHGe1U https://goo.gl/QHGe1U A lot of coverage there, but interesting to see which ones aren't yet implemented, at least in the recent build 14342. (I used Filippo Valsorda's work from https://filippo.io/linux-syscall-table https://filippo.io/linux-syscall-table as the Linux syscall data source.)
coverband 10y ago
With this feature, if you're a Linux developer, you're automatically a Windows developer as well. Almost like being able to run all Android or iOS apps on Windows phones.[1][2] [1] http://www.pcworld.com/article/3038652/windows/microsoft-kills-project-astoria-the-tool-designed-to-port-android-apps-to-windows-10.html http://www.pcworld.com/article/3038652/windows/microsoft-kil... [2] https://developer.microsoft.com/en-us/windows/bridges/ios https://developer.microsoft.com/en-us/windows/bridges/ios Edit: Now I am puzzled as to why this got downvoted?
- besselheim 10y ago
  If you disassemble lxcore.sys you can still see hints of the Android subsystem project that it grew from: the \Device\adss and /dev/adss devices, the application name Microsoft.Windows.Subsystem.Adss, various function names containing "Adss", and some other textual references to Android.
Maarten88 10y ago
I have installed the current fast ring build and have tried installing several packages on Windows. Some do install and work (compilers, build environment, node, redis server), but packages that use more advanced socket options (such as Ethereum) or that configure a deamon (most databases), still end with an error. Compatibility is improving with every new build, and you can ditch/reset the whole Linux environment on Windows with a single command, which is nice for testing.
- skrowl 10y ago
  They've said the initial intent is for developers to use it, not for running servers / etc (which is why they only target Windows 10 client and not Windows Server OSs).
  - ygjb-dupe 10y ago
    There is "running servers" in production and there is "running servers" in dev. If I can't run the entire stack I use for dev under the subsystem then I will go the other route, which is to continue using VMs. I am excited about the initial release, and the prospect of being able to use Windows for all of the regular things I do, but it's clear that this isn't ready for primetime even as a dev tool.
  - stuaxo 10y ago
    Yup, when I'm developing I need to run pretty much most stuff. I guess, I can install say postgres using the windows native version, but then we are back at square zero.
    - Maarten88 10y ago
      Installing postgres on lxss still ends in a 'syscall not implemented' error.
zxcvcxz 10y ago
I use to run Linux in a VM on windows and use Chocolatey for package management and cygwin and powershell etc, then I realized I was just trying to make Windows into Linux. Seems to be the way things are going and with the addition of the linux subsystem it kind of proves that Windows really isn't a good OS on it's own, especially not for developers. I wish Windows/MS would abandon NT and just create a Linux distro. I don't know anyone who particularly likes NT and jamming multiple systems together seems like an awful idea. Windows services and Linux services likely won't play nice together (think long file paths created by Linux services and other incompatibilities), for them to be 100% backward compatible they need to not only make Windows compatible with the things Linux outputs, but Linux compatible with the things windows services output, and to keep the Linux people from figuring out how to use Windows on Linux systems they'd need to make a lot of what they do closed source. So I don't see a Linux+Windows setup being deployed for production. It's cool for developers, but even then you can't do much real world stuff that utilizes both windows and Linux. If you're only taking advantage of one system then whats the point of having two? I went ahead and made the switch to Linux since I was trying to make Windows behave just like Linux.
- pcwalton 10y ago
  > I wish Windows/MS would abandon NT and just create a Linux distro. I don't know anyone who particularly likes NT and jamming multiple systems together seems like an awful idea. I do. The NT kernel is pretty clean and well architected. (Yes, there are mistakes and cruft in it, but Unix has that in spades.) It's not "jamming multiple systems together"; an explicit design goal of the NT kernel was to support multiple userland APIs in a unified manner. Darwin is a much better example of a messy kernel, with Mach and FreeBSD mashed together in a way that neither was designed for. It's the Win32 API that is the real mess. Having a better officially supported API to talk to the NT kernel can only be a good thing, from my point of view.
  - zxcvcxz 10y ago
    Well I've personally seen Microsoft employees themselves complaining about the state of NT while saying it's "fallen behind Linux". An old HN commenter once wrote (mrb) > There is not much discussion about Windows internals, not only because they are not shared, but also because quite frankly the Windows kernel evolves slower than the Linux kernel in terms of new algorithms implemented. For example it is almost certain that Microsoft never tested I/O schedulers, process schedulers, filesystem optimizations, TCP/IP stack tweaks for wireless networks, etc, as much as the Linux community did. One can tell just by seeing the sheer amount of intense competition and interest amongst Linux kernel developers to research all these areas. >The net result of that is a generally acknowledged fact that Windows is slower than Linux when running complex workloads that push network/disk/cpu scheduling to its limit: https://news.ycombinator.com/item?id=3368771 https://news.ycombinator.com/item?id=3368771 A really concrete and technical example is the network throughput in Windows Vista which is degraded when playing audio! https://blogs.technet.microsoft.com/markrussinovich/2007/08/26/vista-multimedia-playback-and-network-throughput/ https://blogs.technet.microsoft.com/markrussinovich/2007/08/... >Note: my post may sound I am freely bashing Windows, but I am not. This is the cold hard truth. Countless of multi-platform developers will attest to this, me included. I can't even remember the number of times I have written a multi-platform program in C or Java that always runs slower on Windows than on Linux, across dozens of different versions of Windows and Linux. The last time I troubleshooted a Windows performance issue, I found out it was the MFT of an NTFS filesystem was being fragmented; this to say I am generally regarded as the one guy in the company who can troubleshoot any issue, yet I acknowledge I can almost never get Windows to perform as good as, or better than Linux, when there is a performance discrepancy in the first place.
bla2 10y ago
Does anybody know how fork() is implemented? This blog post kind of sounds like fork() would do the slow emulation of it through CreateProcess().
- xorblurb 10y ago
  fork() is properly implemented by the NT kernel. WSL is not layered above Win32.
ataylor284_ 10y ago
> The real NtQueryDirectoryFile API takes 11 parameters Curiosity got the best of me here: I had to look this up in the docs to see how a linux syscall that takes 3 parameters could possibly take 11 parameters. Spoiler alert: they are used for async callbacks, filtering by name, allowing only partial results, and the ability to progressively scan with repeated calls.
- deprave 10y ago
  A lot of Microsoft APIs and subsystems are similarly bloated. There are probably tons of factors at play, but I believe being closed-source and having to support many individual use cases is one fundamental reason. (See for example CreateProcess vs. fork...)
  - bitwize 10y ago
    When it comes to system call interfaces, it's because Dave Cutler has forgotten more than many modern "kernel hackers" will ever know about how to design an OS.
    - deprave 10y ago
      I appreciate the name-dropping. Dave Cutler's skills aside, Unix predates Windows by decades, and to anyone remotely familiar with kernel development it is clear that the sheer quantity and complexity of subsystems stem from the fact that nobody but Microsoft can actually see, modify, and redistribute Windows' source code. Unless you can actually say "here's why Windows is qualitatively better" and point out specific tasks Windows does better, I'll just point you to the fact that the internet infrastructure and most of the servers on it, along with every Apple desktop and pretty much every mobile device, run Unix.
      - ksk 10y ago
        To level the playing field, and if we are to take your opinion seriously, it would be beneficial to know what books or articles or whitepapers you have read to inform yourself about the design of the NT kernel.
_RPM 10y ago
Does Microsoft document all system calls?
- detaro 10y ago
  They document the WinAPI, but how that talks to the kernel is not documented. You can talk to it directly if you want, but there is nothing from Microsoft on how to do that. So if you see those as the true system calls, they are not documented at all.
  - xorblurb 10y ago
    Well, tiny parts of the NT API (callable from userspace) are documented, but then often with the caveat that they are not stable (in practice, even some undocumented ones can be considered stable if used by enough programs in the wild, especially if they are simple and standalone and have no Win32 equivalent) The very precise mechanism, though, is extremely unstable. For example virtually every release of Windows (even sometimes SP) changes the syscall numbers. You have to go through the ntdll, which is kind of a more heavyweight version of the Linux VDSO. (The NTDLL approach was invented way before the VDSO, though)
    - therein 10y ago
      Ntdll is similar to VDSO in the sense that it is loaded into the memory space of every userspace process. Even that I think might have exceptions on the Linux side. Either way, unlike VDSO, Ntdll actually does export functions potentially useful when called from the program. Here is an interesting read. http://undocumented.ntinternals.net/ http://undocumented.ntinternals.net/
      - xorblurb 10y ago
        What do you think the VDSO is used for? It also exports "functions potentially useful when called from the program". The approach is a little different though; ntdll exports all of the NT API, and you need to go through it to reach the NT API in a somehow more stable way than using syscall numbers. OTOH, the VDSO exports only virtual syscalls that gain (or have gained in the past) from being performed in userspace, and even then corresponding syscalls still exist in the kernel, with both stable numbers and even a stable API.
kevincox 10y ago
> the Linux fork syscall has no documented equivalent for Windows Emphasis is mine. I wonder if this is something that cygwin could (ab)use. Also I wonder why they would need this undocumented call.
- bboreham 10y ago
  > Also I wonder why they would need this undocumented call. To implement the first NT Posix subsystem, which was a FIPS requirement.
- xorblurb 10y ago
  Cygwin is layered above Win32. Win32 has no provision to nicely handle forks. So even if there was an NT API fork syscall (I'm don't think there is on Windows 10, WSL does not use the NT API, there is not any more Posix/SFU/{Whatever Unix NT classic subsys of the day} as far as I know), this would not go anywhere.
  - pcwalton 10y ago
    > So even if there was an NT API fork syscall You can do it with NtCreateProcess: https://groups.google.com/d/msg/microsoft.public.win32.programmer.kernel/ejtHCZmdyaI/k0D0Jinx9KwJ https://groups.google.com/d/msg/microsoft.public.win32.progr... (The Win32 userland won't understand what you did, but you can still do it.)
    - xorblurb 10y ago
      Well, you can do it on some versions of Windows. On Windows 10, and even future version of Windows 10, not so sure...
      - lmm 10y ago
        Windows 10 is still windows NT. The NT native API is widely used these days. It would be a huge departure for MS to stop supporting it in future versions of windows.
vegabook 10y ago
Next step is Microsoft basically needs to turn Windows into a flavour of Linux. If they don't, they're under massive pincer threat from Android and Chrome, which are rapidly becoming the consumer endpoints of the future. Windows is about to "do an IBM" and throw away a market that it created. See PS/2 and OS/2. They should probably just buy Canonical. That would put the shivers into Google, properly.
- deleted 10y ago
  [deleted]
- mxuribe 10y ago
  Funny years ago i would have reflexively flabbergasted at the thought of microsoft buying canonical (or any linux distro producer)...but actually thinking on that concept, and seeing recent (perhaps less-than-hostile) approach that microsoft has taken towards open source and linux, that wouldn't be a bad idea. I mean if microsoft could have both offerings - for windows servers and ubuntu-installed servers - i suppose that would be a very smart business move. Assuming they don't actually butcher or deny resources to whatever linux company they would buy, i could see several benefits - not only to microsoft but to developers, system integrators, etc. worldwide. Hey if a side benefit is that it would spur the market (a la google, apple, etc.) a little - to the benefit of us civilians - that's cool too.
  - orionblastar 10y ago
    I think Microsoft should do what Apple did with BSD Unix aka Nextstep and merge it with their old OS. Microsoft should take the Windows GUI and put it over Linux as a desktop manager. Microsoft could sell the Windows GUI for Linux users that want to run Windows apps.
    - vegabook 10y ago
      Could not agree more. Windows WM as an option on Linux is a clear and logical strategy.
  - vegabook 10y ago
emcrazyone 10y ago
I can't think of much that would benefit from this except for, perhaps, headless command line type applications. The one that comes to mind is rsync. Being able to compile the latest version/protocol of rsync on a Linux machine and then running the same binary on a Windows host would be nice but fun seems to end there plus with Cygwin, this is largely a no-brainer without M$ help. What about applications that hook to X Windows or do things like opening the frame buffer device. I've got a messaging application that can be compiled for both Windows and Linux and depending on the OS, I compile a different transport layer. Under Linux heavy use of epoll is used which is very different than how NT handles Async I/O - especially with sockets. So my application's "transport driver" is either compiling an NT code base using WinSock & OVERLAPPED IO or a Linux code base using EPOLL and pthreads. Over all it seems like a nice to have but I'm struggling to extract any real benefit. Can anyone offer up some real good use cases I may be overlooking?
- quux 10y ago
  There are both free and commercial X servers for Windows, and you can get a linux app running under WSL to work with one of those X servers very easily. I played with it a little bit and it worked fine.
quux 10y ago
Interesting, I wonder how much overhead is added to syscalls to look up the process type. Does NT still do this check when no WSL processes are running?
- stuaxo 10y ago
  Pretty sure these are different entry points, so you wouldn't need to do anything different for normal Windows processes whether WSL is running or not.
  - quux 10y ago
    I don't think so... both linux and windows binaries are using the same SYSCALL cpu instruction, and thus must be going to the same handler in the NT kernel.
zxcvcxz 10y ago
You should also mention that you're the author of Windows only software that directly exploits certain features specific to Windows. An implementation of python that according to http://pyparallel.org/ http://pyparallel.org/ out-performs pretty much every other technology. Is that correct? I'm also wondering if anyone has replicated these results? From what I can find not a single person has replicated your tests. According to your own website pretty much every other technology runs better on Linux than it does on windows (http://pyparallel.org/wrk-rps-comparison2.svg http://pyparallel.org/wrk-rps-comparison2.svg). It could be said you're a little biased, seeing as your software only targets Windows.
- Koromix 10y ago
  I only have superficial experience with it, but from what I can tell there is a huge mismatch between IOCP and the UNIX readiness/poll model, and from my experience most server programs are written primarily for the latter. You need to design your server code somewhat differently to take advantage of IOCP. What many UNIX-like softwares do instead is bend IOCP or WaitForMultipleObjects() to behave like Linux. It works, but the performance is not there. Note that I haven't checked the code for any of the softwares in your chart so I could be wrong.
  - zxcvcxz 10y ago
    Here's what I don't understand: If IOCP gives us such a great performance boost why don't I see people using it, even on Windows systems? The first thing that comes to mind is that IOCP can likely only maintain high performance under certain edge-cases that aren't pertinent to the real world, second is that the other I/O models are needed not for unix, but for other aspects of the language its self where IOCP is not appropriate. So IOCP likely causes overhead. I don't know much about it so maybe someone can explain. It sounds revolutionary if it can be applied to the real world and not just edge cases.
    - clevernickname 10y ago
      The simplest explanation is that the performance benefit of using Windows and implementing just about any Windows-specific design is outweighed by the cost of the Windows licensing fees, when compared to a measurably worse-performing Linux or FreeBSD solution that costs nothing. So very few bother to treat Windows versions of "backend" software as anything but an afterthought.
caf 10y ago
Since NT syscalls follow the x64 calling convention, the kernel does not need to save off volatile registers since that was handled by the compiler emitting instructions before the syscall to save off any volatile registers that needed to be preserved. Say what? The NT kernel doesn't restore caller-saved registers at syscall exit? This seems extraordinary, because unless it either restores them or zaps them then it will be in danger of leaking internal kernel values to userspace - and if it zaps them then it might as well save and restore them, so userspace won't need to.
- trentnelson 10y ago
  I think that's referring to the prolog/epilog convention and "homing" of parameter registers, e.g. Frame struct ReturnAddress dq ? HomeRcx dq ? HomeRdx dq ? HomeR8 dq ? HomeR9 dq ? Frame ends NESTED_ENTRY Foo, _TEXT$00 mov Frame.HomeRcx[rsp], rcx mov Frame.HomeRdx[rsp], rcd mov Frame.HomeR8[rsp], r8 mov Frame.HomeR9[rsp], r9 alloc_stack 64 END_PROLOG ; *do stuff* BEGIN_EPILOG add rsp, 64 NESTED_END Foo, _TEXT$00 https://msdn.microsoft.com/en-us/library/tawsa7cb.aspx https://msdn.microsoft.com/en-us/library/tawsa7cb.aspx
negus 10y ago
wtf is "pico process" and "pico driver"?
- wereHamster 10y ago
  https://blogs.msdn.microsoft.com/wsl/2016/05/23/pico-process-overview/ https://blogs.msdn.microsoft.com/wsl/2016/05/23/pico-process...
davidgerard 10y ago
Yes, yes, but can we run Wine on it?
prirun 10y ago
Step 1: embrace

359 points by jackhammons 10y ago

It's too bad that x86 hardware doesn't do virtualization as well as IBM hardware. You can't stack VMs. That's exactly what's needed here - a non-kernel VM that runs above NT but below the application.

pmalynin 10y ago
You can have nested VMs. https://www.kernel.org/doc/Documentation/virtual/kvm/nested-vmx.txt https://www.kernel.org/doc/Documentation/virtual/kvm/nested-...
- overgryphon 10y ago
  Windows also now supports nested virtualization. https://msdn.microsoft.com/en-us/virtualization/hyperv_on_windows/user_guide/nesting https://msdn.microsoft.com/en-us/virtualization/hyperv_on_wi...
pjmlp 10y ago
Looking at the way mainframes work, with their higher level languages, JIT compilers at kernel level, object databases, type 1 hypervisors, .... It is quite interesting to see mainstream OSes increasingly get adopting all those features.
- Animats 10y ago
  Very, very slowly. Microprocessors still have DMA instead of mainframe-like "channels", although we're starting to see MMUs on the I/O side. With channels, devices can't blither all over memory and neither driver nor device need be trusted.
geofft 10y ago
I thought that a) the conclusion of VMware's "Comparison of techniques" paper [1] was that x86 and possibly everything is Popek-and-Goldberg-virtualizable [2] via binary translation, and b) the last several years of Intel and AMD chips all have hardware virtualization support, including nested virtualization, that made their architectures Popek-and-Goldberg-virtualizable in the obvious way? [1] https://www.vmware.com/pdf/asplos235_adams.pdf https://www.vmware.com/pdf/asplos235_adams.pdf [2] https://en.wikipedia.org/wiki/Popek_and_Goldberg_virtualization_requirements https://en.wikipedia.org/wiki/Popek_and_Goldberg_virtualizat...

deleted 10y ago

[deleted]

smegel 10y ago

Funny they don't mention ioctl.

obnauticus 10y ago

Excellent post, Jack.

l3m0ndr0p 10y ago

Pretty neat stuff. I think that MS should just create their own Linux Distribution & port all MS products. Get rid of the Windows NT Kernel. I believe it's outdated & doesn't have the same update cycle that the Linux Kernel has. Why run a Linux Application/binary on a windows server OS? When you can just run it on Linux OS and get better performance & stability.

serge2k 10y ago
> Get rid of the Windows NT Kernel. I believe it's outdated & doesn't have the same update cycle that the Linux Kernel has. Curious why you claim this? What's outdated about the NT Kernel?
- l3m0ndr0p 10y ago
  Here are some, or maybe this is not part of the NT Kernel... 1. The use of drive letters A-Z for file system access. 2. Creating symbolic links to files and folders, like you can in Unix/Linux. You have to set a setting somewhere to enable this, but there's a security risk. 3. Standard functional/usable non-gui terminal application like Unix/Linux ssh. PowerShell doesn't come close. 4. Ability to SUDO or su Admin like Unix/Linux. Maybe these are not kernel related above, but the OS specific layer.
  - slededit 10y ago
    You are confusing the win32 subsystem with the NT kernel. They are not the same, the win32 layer acts as a translation. Also symbolic and hard links are supported by NTFS, they are just not exposed in the UI. There are utilities to create them if you really want to. The shell itself and the rest of userland has very little to do with the kernel. It seems its the userland you are upset with. Swapping out the kernel won't fix that.
  - asveikau 10y ago
    > 1. The use of drive letters A-Z for file system access. NT has a root directory like Unix does. Drive letters are symbolic links inside a directory called \DosDevices. Granted, this is not user-visible but an implementation detail. The needs of Win32 applications dictate a lot of user-visible behavior. > 2. Creating symbolic links to files and folders, NT supports symbolic links. Open cmd and type "mklink".

dragonbonheur 10y ago

jcoffland 10y ago
This title would be clearer: Windows subsystem for Linux apps.
jcoffland 10y ago
Maybe because the way they stated it it would be a much more attractive technology. Seems like an attempt to regaining ground in the server market. This would be really useful for distributing Windows apps as Linux binaries. It would make it easier to develop from Linux and target Windows. Need the same for OSX.
coverband 10y ago
That's the output though, they're not changing anything in Linux. The "Windows subsystem" they're developing will act as the translator to get there.
pionar 10y ago
It's a subsystem of Windows for running part of Linux, so Windows Subsystem for Linux. :)
geofft 10y ago
"As Brother Francis readily admitted, his mastery of pre-Deluge English was far from masterful yet. The way nouns could sometimes modify other nouns in that tongue had always been one of his weak points. In Latin, as in most simple dialects of the region, a construction like servus puer meant about the same thing as puer servus, and even in English slave boy meant boy slave. But there the similarity ended. He had finally learned that house cat did not mean cat house, and that a dative of purpose or possession, as in mihi amicus, was somehow conveyed by dog food or sentry box even without inflection. But what of a triple appositive like fallout survival shelter? Brother Francis shook his head."
- stuaxo 10y ago

luchs 10y ago

>As of this article, lxss.sys has ~235 of the Linux syscalls implemented with varying level of support. Is there a list of these syscalls somewhere? It would be cool to check it against the recent Linux API compatibility paper [0, 1]. [0]: http://oscar.cs.stonybrook.edu/api-compat-study/ http://oscar.cs.stonybrook.edu/api-compat-study/ [1]: http://www.oscar.cs.stonybrook.edu/papers/files/syspop16.pdf http://www.oscar.cs.stonybrook.edu/papers/files/syspop16.pdf

xorblurb 10y ago
A list of (at least partly) supported syscalls is here: https://msdn.microsoft.com/en-us/commandline/wsl/release_notes https://msdn.microsoft.com/en-us/commandline/wsl/release_not... Not details on which one are fully or partly supported, though.
besselheim 10y ago
You piqued my curiosity - just made one by extracting the syscall dispatch table from lxcore.sys and placing it alongside the Linux syscall list: https://goo.gl/QHGe1U https://goo.gl/QHGe1U A lot of coverage there, but interesting to see which ones aren't yet implemented, at least in the recent build 14342. (I used Filippo Valsorda's work from https://filippo.io/linux-syscall-table https://filippo.io/linux-syscall-table as the Linux syscall data source.)

coverband 10y ago

With this feature, if you're a Linux developer, you're automatically a Windows developer as well. Almost like being able to run all Android or iOS apps on Windows phones.[1][2] [1] http://www.pcworld.com/article/3038652/windows/microsoft-kills-project-astoria-the-tool-designed-to-port-android-apps-to-windows-10.html http://www.pcworld.com/article/3038652/windows/microsoft-kil... [2] https://developer.microsoft.com/en-us/windows/bridges/ios https://developer.microsoft.com/en-us/windows/bridges/ios Edit: Now I am puzzled as to why this got downvoted?

besselheim 10y ago
If you disassemble lxcore.sys you can still see hints of the Android subsystem project that it grew from: the \Device\adss and /dev/adss devices, the application name Microsoft.Windows.Subsystem.Adss, various function names containing "Adss", and some other textual references to Android.

Maarten88 10y ago

I have installed the current fast ring build and have tried installing several packages on Windows. Some do install and work (compilers, build environment, node, redis server), but packages that use more advanced socket options (such as Ethereum) or that configure a deamon (most databases), still end with an error. Compatibility is improving with every new build, and you can ditch/reset the whole Linux environment on Windows with a single command, which is nice for testing.

skrowl 10y ago
They've said the initial intent is for developers to use it, not for running servers / etc (which is why they only target Windows 10 client and not Windows Server OSs).
- ygjb-dupe 10y ago
  There is "running servers" in production and there is "running servers" in dev. If I can't run the entire stack I use for dev under the subsystem then I will go the other route, which is to continue using VMs. I am excited about the initial release, and the prospect of being able to use Windows for all of the regular things I do, but it's clear that this isn't ready for primetime even as a dev tool.
- stuaxo 10y ago
  Yup, when I'm developing I need to run pretty much most stuff. I guess, I can install say postgres using the windows native version, but then we are back at square zero.
  - Maarten88 10y ago
    Installing postgres on lxss still ends in a 'syscall not implemented' error.

zxcvcxz 10y ago

I use to run Linux in a VM on windows and use Chocolatey for package management and cygwin and powershell etc, then I realized I was just trying to make Windows into Linux. Seems to be the way things are going and with the addition of the linux subsystem it kind of proves that Windows really isn't a good OS on it's own, especially not for developers. I wish Windows/MS would abandon NT and just create a Linux distro. I don't know anyone who particularly likes NT and jamming multiple systems together seems like an awful idea. Windows services and Linux services likely won't play nice together (think long file paths created by Linux services and other incompatibilities), for them to be 100% backward compatible they need to not only make Windows compatible with the things Linux outputs, but Linux compatible with the things windows services output, and to keep the Linux people from figuring out how to use Windows on Linux systems they'd need to make a lot of what they do closed source. So I don't see a Linux+Windows setup being deployed for production. It's cool for developers, but even then you can't do much real world stuff that utilizes both windows and Linux. If you're only taking advantage of one system then whats the point of having two? I went ahead and made the switch to Linux since I was trying to make Windows behave just like Linux.

pcwalton 10y ago
> I wish Windows/MS would abandon NT and just create a Linux distro. I don't know anyone who particularly likes NT and jamming multiple systems together seems like an awful idea. I do. The NT kernel is pretty clean and well architected. (Yes, there are mistakes and cruft in it, but Unix has that in spades.) It's not "jamming multiple systems together"; an explicit design goal of the NT kernel was to support multiple userland APIs in a unified manner. Darwin is a much better example of a messy kernel, with Mach and FreeBSD mashed together in a way that neither was designed for. It's the Win32 API that is the real mess. Having a better officially supported API to talk to the NT kernel can only be a good thing, from my point of view.
- zxcvcxz 10y ago
  Well I've personally seen Microsoft employees themselves complaining about the state of NT while saying it's "fallen behind Linux". An old HN commenter once wrote (mrb) > There is not much discussion about Windows internals, not only because they are not shared, but also because quite frankly the Windows kernel evolves slower than the Linux kernel in terms of new algorithms implemented. For example it is almost certain that Microsoft never tested I/O schedulers, process schedulers, filesystem optimizations, TCP/IP stack tweaks for wireless networks, etc, as much as the Linux community did. One can tell just by seeing the sheer amount of intense competition and interest amongst Linux kernel developers to research all these areas. >The net result of that is a generally acknowledged fact that Windows is slower than Linux when running complex workloads that push network/disk/cpu scheduling to its limit: https://news.ycombinator.com/item?id=3368771 https://news.ycombinator.com/item?id=3368771 A really concrete and technical example is the network throughput in Windows Vista which is degraded when playing audio! https://blogs.technet.microsoft.com/markrussinovich/2007/08/26/vista-multimedia-playback-and-network-throughput/ https://blogs.technet.microsoft.com/markrussinovich/2007/08/... >Note: my post may sound I am freely bashing Windows, but I am not. This is the cold hard truth. Countless of multi-platform developers will attest to this, me included. I can't even remember the number of times I have written a multi-platform program in C or Java that always runs slower on Windows than on Linux, across dozens of different versions of Windows and Linux. The last time I troubleshooted a Windows performance issue, I found out it was the MFT of an NTFS filesystem was being fragmented; this to say I am generally regarded as the one guy in the company who can troubleshoot any issue, yet I acknowledge I can almost never get Windows to perform as good as, or better than Linux, when there is a performance discrepancy in the first place.

bla2 10y ago

Does anybody know how fork() is implemented? This blog post kind of sounds like fork() would do the slow emulation of it through CreateProcess().

xorblurb 10y ago
fork() is properly implemented by the NT kernel. WSL is not layered above Win32.

ataylor284_ 10y ago

> The real NtQueryDirectoryFile API takes 11 parameters Curiosity got the best of me here: I had to look this up in the docs to see how a linux syscall that takes 3 parameters could possibly take 11 parameters. Spoiler alert: they are used for async callbacks, filtering by name, allowing only partial results, and the ability to progressively scan with repeated calls.

deprave 10y ago
A lot of Microsoft APIs and subsystems are similarly bloated. There are probably tons of factors at play, but I believe being closed-source and having to support many individual use cases is one fundamental reason. (See for example CreateProcess vs. fork...)
- bitwize 10y ago
  When it comes to system call interfaces, it's because Dave Cutler has forgotten more than many modern "kernel hackers" will ever know about how to design an OS.
  - deprave 10y ago
    I appreciate the name-dropping. Dave Cutler's skills aside, Unix predates Windows by decades, and to anyone remotely familiar with kernel development it is clear that the sheer quantity and complexity of subsystems stem from the fact that nobody but Microsoft can actually see, modify, and redistribute Windows' source code. Unless you can actually say "here's why Windows is qualitatively better" and point out specific tasks Windows does better, I'll just point you to the fact that the internet infrastructure and most of the servers on it, along with every Apple desktop and pretty much every mobile device, run Unix.
    - ksk 10y ago
      To level the playing field, and if we are to take your opinion seriously, it would be beneficial to know what books or articles or whitepapers you have read to inform yourself about the design of the NT kernel.

_RPM 10y ago

Does Microsoft document all system calls?

detaro 10y ago
They document the WinAPI, but how that talks to the kernel is not documented. You can talk to it directly if you want, but there is nothing from Microsoft on how to do that. So if you see those as the true system calls, they are not documented at all.
- xorblurb 10y ago
  Well, tiny parts of the NT API (callable from userspace) are documented, but then often with the caveat that they are not stable (in practice, even some undocumented ones can be considered stable if used by enough programs in the wild, especially if they are simple and standalone and have no Win32 equivalent) The very precise mechanism, though, is extremely unstable. For example virtually every release of Windows (even sometimes SP) changes the syscall numbers. You have to go through the ntdll, which is kind of a more heavyweight version of the Linux VDSO. (The NTDLL approach was invented way before the VDSO, though)
  - therein 10y ago
    Ntdll is similar to VDSO in the sense that it is loaded into the memory space of every userspace process. Even that I think might have exceptions on the Linux side. Either way, unlike VDSO, Ntdll actually does export functions potentially useful when called from the program. Here is an interesting read. http://undocumented.ntinternals.net/ http://undocumented.ntinternals.net/
    - xorblurb 10y ago
      What do you think the VDSO is used for? It also exports "functions potentially useful when called from the program". The approach is a little different though; ntdll exports all of the NT API, and you need to go through it to reach the NT API in a somehow more stable way than using syscall numbers. OTOH, the VDSO exports only virtual syscalls that gain (or have gained in the past) from being performed in userspace, and even then corresponding syscalls still exist in the kernel, with both stable numbers and even a stable API.

kevincox 10y ago

> the Linux fork syscall has no documented equivalent for Windows Emphasis is mine. I wonder if this is something that cygwin could (ab)use. Also I wonder why they would need this undocumented call.

bboreham 10y ago
> Also I wonder why they would need this undocumented call. To implement the first NT Posix subsystem, which was a FIPS requirement.
xorblurb 10y ago
Cygwin is layered above Win32. Win32 has no provision to nicely handle forks. So even if there was an NT API fork syscall (I'm don't think there is on Windows 10, WSL does not use the NT API, there is not any more Posix/SFU/{Whatever Unix NT classic subsys of the day} as far as I know), this would not go anywhere.
- pcwalton 10y ago
  > So even if there was an NT API fork syscall You can do it with NtCreateProcess: https://groups.google.com/d/msg/microsoft.public.win32.programmer.kernel/ejtHCZmdyaI/k0D0Jinx9KwJ https://groups.google.com/d/msg/microsoft.public.win32.progr... (The Win32 userland won't understand what you did, but you can still do it.)
  - xorblurb 10y ago
    Well, you can do it on some versions of Windows. On Windows 10, and even future version of Windows 10, not so sure...
    - lmm 10y ago
      Windows 10 is still windows NT. The NT native API is widely used these days. It would be a huge departure for MS to stop supporting it in future versions of windows.

vegabook 10y ago

Next step is Microsoft basically needs to turn Windows into a flavour of Linux. If they don't, they're under massive pincer threat from Android and Chrome, which are rapidly becoming the consumer endpoints of the future. Windows is about to "do an IBM" and throw away a market that it created. See PS/2 and OS/2. They should probably just buy Canonical. That would put the shivers into Google, properly.

deleted 10y ago
[deleted]
mxuribe 10y ago
Funny years ago i would have reflexively flabbergasted at the thought of microsoft buying canonical (or any linux distro producer)...but actually thinking on that concept, and seeing recent (perhaps less-than-hostile) approach that microsoft has taken towards open source and linux, that wouldn't be a bad idea. I mean if microsoft could have both offerings - for windows servers and ubuntu-installed servers - i suppose that would be a very smart business move. Assuming they don't actually butcher or deny resources to whatever linux company they would buy, i could see several benefits - not only to microsoft but to developers, system integrators, etc. worldwide. Hey if a side benefit is that it would spur the market (a la google, apple, etc.) a little - to the benefit of us civilians - that's cool too.
- orionblastar 10y ago
  I think Microsoft should do what Apple did with BSD Unix aka Nextstep and merge it with their old OS. Microsoft should take the Windows GUI and put it over Linux as a desktop manager. Microsoft could sell the Windows GUI for Linux users that want to run Windows apps.
  - vegabook 10y ago
    Could not agree more. Windows WM as an option on Linux is a clear and logical strategy.
- vegabook 10y ago

emcrazyone 10y ago

I can't think of much that would benefit from this except for, perhaps, headless command line type applications. The one that comes to mind is rsync. Being able to compile the latest version/protocol of rsync on a Linux machine and then running the same binary on a Windows host would be nice but fun seems to end there plus with Cygwin, this is largely a no-brainer without M$ help. What about applications that hook to X Windows or do things like opening the frame buffer device. I've got a messaging application that can be compiled for both Windows and Linux and depending on the OS, I compile a different transport layer. Under Linux heavy use of epoll is used which is very different than how NT handles Async I/O - especially with sockets. So my application's "transport driver" is either compiling an NT code base using WinSock & OVERLAPPED IO or a Linux code base using EPOLL and pthreads. Over all it seems like a nice to have but I'm struggling to extract any real benefit. Can anyone offer up some real good use cases I may be overlooking?

quux 10y ago
There are both free and commercial X servers for Windows, and you can get a linux app running under WSL to work with one of those X servers very easily. I played with it a little bit and it worked fine.

quux 10y ago

Interesting, I wonder how much overhead is added to syscalls to look up the process type. Does NT still do this check when no WSL processes are running?

stuaxo 10y ago
Pretty sure these are different entry points, so you wouldn't need to do anything different for normal Windows processes whether WSL is running or not.
- quux 10y ago
  I don't think so... both linux and windows binaries are using the same SYSCALL cpu instruction, and thus must be going to the same handler in the NT kernel.

zxcvcxz 10y ago

You should also mention that you're the author of Windows only software that directly exploits certain features specific to Windows. An implementation of python that according to http://pyparallel.org/ http://pyparallel.org/ out-performs pretty much every other technology. Is that correct? I'm also wondering if anyone has replicated these results? From what I can find not a single person has replicated your tests. According to your own website pretty much every other technology runs better on Linux than it does on windows (http://pyparallel.org/wrk-rps-comparison2.svg http://pyparallel.org/wrk-rps-comparison2.svg). It could be said you're a little biased, seeing as your software only targets Windows.

Koromix 10y ago
I only have superficial experience with it, but from what I can tell there is a huge mismatch between IOCP and the UNIX readiness/poll model, and from my experience most server programs are written primarily for the latter. You need to design your server code somewhat differently to take advantage of IOCP. What many UNIX-like softwares do instead is bend IOCP or WaitForMultipleObjects() to behave like Linux. It works, but the performance is not there. Note that I haven't checked the code for any of the softwares in your chart so I could be wrong.
- zxcvcxz 10y ago
  Here's what I don't understand: If IOCP gives us such a great performance boost why don't I see people using it, even on Windows systems? The first thing that comes to mind is that IOCP can likely only maintain high performance under certain edge-cases that aren't pertinent to the real world, second is that the other I/O models are needed not for unix, but for other aspects of the language its self where IOCP is not appropriate. So IOCP likely causes overhead. I don't know much about it so maybe someone can explain. It sounds revolutionary if it can be applied to the real world and not just edge cases.
  - clevernickname 10y ago
    The simplest explanation is that the performance benefit of using Windows and implementing just about any Windows-specific design is outweighed by the cost of the Windows licensing fees, when compared to a measurably worse-performing Linux or FreeBSD solution that costs nothing. So very few bother to treat Windows versions of "backend" software as anything but an afterthought.

caf 10y ago

Since NT syscalls follow the x64 calling convention, the kernel does not need to save off volatile registers since that was handled by the compiler emitting instructions before the syscall to save off any volatile registers that needed to be preserved. Say what? The NT kernel doesn't restore caller-saved registers at syscall exit? This seems extraordinary, because unless it either restores them or zaps them then it will be in danger of leaking internal kernel values to userspace - and if it zaps them then it might as well save and restore them, so userspace won't need to.

trentnelson 10y ago
I think that's referring to the prolog/epilog convention and "homing" of parameter registers, e.g. Frame struct ReturnAddress dq ? HomeRcx dq ? HomeRdx dq ? HomeR8 dq ? HomeR9 dq ? Frame ends NESTED_ENTRY Foo, _TEXT$00 mov Frame.HomeRcx[rsp], rcx mov Frame.HomeRdx[rsp], rcd mov Frame.HomeR8[rsp], r8 mov Frame.HomeR9[rsp], r9 alloc_stack 64 END_PROLOG ; *do stuff* BEGIN_EPILOG add rsp, 64 NESTED_END Foo, _TEXT$00 https://msdn.microsoft.com/en-us/library/tawsa7cb.aspx https://msdn.microsoft.com/en-us/library/tawsa7cb.aspx

negus 10y ago

wtf is "pico process" and "pico driver"?

wereHamster 10y ago
https://blogs.msdn.microsoft.com/wsl/2016/05/23/pico-process-overview/ https://blogs.msdn.microsoft.com/wsl/2016/05/23/pico-process...

davidgerard 10y ago

Yes, yes, but can we run Wine on it?

prirun 10y ago

Step 1: embrace