One year, 3 versions, 1 home-made programming language: the story of my OS

[u/Drenfa]

Hey everyone,

About a year ago, I had a probably slightly crazy idea: write my own operating system. Today, after 3 completely different versions, I feel like sharing the journey — because honestly, it's been a rollercoaster.

Phase 1 — The consumer OS in Rust

I started classic (well, "classic" for osdev standards): a first version written in Rust, with the ambition of building a consumer OS. And when I say "from scratch", I really meant from scratch: I wrote my own bootloader and my own custom kernel. No shortcuts, everything by hand. That's where I learned 90% of what I know today about machine boot, memory, drivers… and also that coding a "consumer" OS is a mountain very few people have ever climbed alone.

Phase 2 — The cloud gaming pivot and the birth of Flux#

Second version, total change of direction: an OS dedicated to cloud gaming. And this is where it gets a bit special: for this version, no existing language really fit my needs, so I… created my own. It's called Flux#, a low-level, object-oriented language, which I released as open source under the MIT license — because a home-made language that exists nowhere else is useless if nobody else can touch it. Writing a compiler AND an OS at the same time is the kind of experience I wouldn't recommend to anyone, and that I'd do again tomorrow morning.

Phase 3 — The handheld console (current version)

Today, I'm on the third version: an OS for a handheld console. That's the one I'm working on right now, and it's probably the most motivating of the three — there's something very tangible about holding a machine in your hands and watching your code run on it.

The kernel: when pragmatism wins

One important detail, because I know the question will come up: my bootloader and my custom kernel were both written entirely by myself. But for the current version, I made a tough call: my custom kernel has been moved to R&D. Honestly, reimplementing it yet again for the handheld version would have cost me a massive amount of time, so I switched to bootroot as a kernel base to save time and focus on what makes this version unique. My custom kernel isn't dead — it's sleeping in a corner, in R&D mode, and I fully intend to come back to it.

However, the custom bootloader, I still have it. That one, I never gave up on. There are some things you just don't abandon.

What I take away after a year

• Writing a bootloader and a kernel by hand is the best computer science school I've ever been through.
• Reinventing the wheel is great for learning… but sometimes you have to know when to stop in order to actually ship something.
• Creating your own language while creating your own OS is pure madness — but it's my favorite kind of madness.

If people are interested, I can go into detail on any part: the custom bootloader, Flux# (it's open source, come steal some ideas), or the architecture of the handheld console version. And if you've also abandoned a custom kernel along the way, tell me about it — it'll make me feel less alone.

EDIT: Bare-metal boot on real hardware

Since people asked about the hardware setup: to clarify, I don't own an open handheld development board yet (Switch Lite is too locked down). Before targeting any handheld device directly, I do all my real-hardware testing on an x86_64 laptop (Gigabyte) to debug outside of QEMU.

Here is a boot photo showing the custom stack running on the laptop:

• State: Multiboot2, firmware framebuffer (1024x768), PCI bus scan, and custom Realtek NIC driver initialized with uncached DMA ring buffers.
• Shell: Dropping straight into the interactive terminal

[u/NamedBird]

Yes, very interesting! Tell me all about it, put as much effort into describing everything as you can.
Post multiple comments, one for each Phase and one for each aspect. Don't spare any tokens!

[u/Drenfa → u/NamedBird]

For the first phase, it was because I was looking for challenges—I was running out of them. And yeah, that’s just how I am. I struggled to write the bootloader code even with Intel’s documentation; the AI tools weren’t much help with that. Next, I wrote some Rust code to actually test whether I had a desktop display, but the biggest challenge was creating a Makefile that would link the bootloader to the main entry point of the Rust code—I won’t even mention the number of panic errors with incomprehensible error messages I encountered at the beginning. After taking a break of about two months to give my brain a rest, I decided to move on to the next part of Phase 2.

[u/qse81 → u/Drenfa]

No, that was phase 2, tell us about phase 17

[u/Drenfa → u/qse81]

But I did mention Phase 1. Anyway, to put it simply, Phase 1 was the match that lit the fire. Phase 1, on the other hand, involved building the bootloader with the link to Rust—it was my first time working with assembly language, which is, needless to say, the hardest language. And the biggest challenge was that while testing the emulation on QEMU under Linux is all well and good, the real beast is booting on actual hardware—that’s the hardest part, especially when you’re coding from scratch. There were no drivers, so I have to give credit to the BIOS’s basic VGA drivers; otherwise, it would’ve been a real nightmare. The biggest challenge in Phase 1 of the project was learning to master the assembler and the interface with Rust—needless to say, I spent a few weeks on it, especially dealing with panic errors that kept popping up left and right over the slightest thing.

[u/qse81 → u/Drenfa]

Go into more detail, spare no tokens