Check out some microkernel systems (eg. Hurd). They have their problems, but the core OS is very minimal. The microkernel idea is to move everything that can b

Check out some microkernel systems (eg. Hurd). They have their problems, but the core OS is very minimal. The microkernel idea is to move everything that can be moved, out to user-space. The minimal OS services is typically thread management, address space management, and some kind of IPC. And if you choose not to provide virtual memory or memory protection, you have a very tiny core OS. Everything else, all the "normal" OS services can be provided by application level programs - device handlers, file systems, network stacks, time services, graphics,... almost everything.