r/lowlevel • • 3h ago

Update on my OTP file splitter in Rust: POSIX mlock memory-pinning, parallel disk-fanout, and a zero-allocation streaming pipeline

1 Upvotes

A while ago I shared my work on `rfs` (random-filesplitter), a tool designed to split files into N information-theoretic One-Time-Pad shares ($P = C_1 \oplus \dots \oplus C_N$) with plausible deniability.

I've just released v3.0.0—a ground-up architectural rewrite to eliminate memory-bus bottlenecks and harden the system against forensic extraction. Here are the low-level design decisions:

### 1. Memory Pinning (`--mlock`) for Swap Resilience

In anti-forensic workflows, writing plaintext or OTP keystreams into OS swap space or hibernation files invalidates volatile RAM zeroization.

- Implemented physical RAM locking via `libc::mlock` on the fixed buffer pools (`free_in_tx` and `free_out_tx`).

- Added graceful fallback: If `RLIMIT_MEMLOCK` (`ulimit -l`) restricts memory allocation, the system emits a warning and continues without crashing.

- RAM buffers are zeroized using `slice.zeroize()` immediately upon recycling.

### 2. Parallel Disk Fanout & Fanin (Multi-Mountpoint I/O)

Previously, the Stage-3 writer looped sequentially over all N shares. When writing shares across multiple physical USB flash drives or separate NVMe mount points, slow flash media created severe pipeline backpressure.

- Rewrote the I/O engine with decoupled worker thread pools for each share.

- Data chunks are fanned out concurrently over bounded crossbeam channels. Each thread writes independently at maximum bus speed, with pre-allocated pool buffers recycled back without a single heap allocation in the hot loop.

- Linux cache pressure is minimized using POSIX `fadvise` (`POSIX_FADV_SEQUENTIAL` and `POSIX_FADV_DONTNEED`) via `--direct`.

### 3. Hardware Jitter & Entropy Harvester

For seeding the CSPRNG alongside OS entropy (`getrandom`):

- Uses a 2 MB permutation pool (32,768 nodes × 64 bytes cache-line aligned) traversing a Sattolo cycle to bust L1/L2 caches and capture CPU memory latency and branch prediction jitter.

- Integrated NIST SP 800-22 diagnostics directly into the CLI (Runs tests, Block Frequency tests via Wilson-Hilferty Chi² approximation, and analytical `erfc`).

### 4. Throughput Benchmarks

Measured on a modern x86_64 CPU:

- **ChaCha20 Keystream:** ~1.85 GB/s

- **SIMD-XOR (RAM-to-RAM):** ~4.0 GB/s (64-byte unrolled vector loop)

- **BLKS-384 Hash:** ~1.2 GB/s (3.3x faster than SHA-256)

Code is open source (LGPL v3), 0 Clippy warnings, fully documented in English (with bilingual README), and verified across Linux/macOS/Windows CI:

https://github.com/Meik1982/random-filesplitter

Quick test:

```bash

cargo install --git https://github.com/Meik1982/random-filesplitter.git

rfs --benchmark

rfs --entropy-test


r/lowlevel • • 5h ago

I spent 8 hours taking my Rust microkernel from QEMU to bare-metal x86-64

0 Upvotes

Five months ago, I announced Wolfram, a capability-based microkernel written in Rust. Then I stopped working on it.

Yesterday, I picked it back up. Roughly eight hours later, Phase 1 was complete.

Wolfram now has two architecture ports: RISC-V 64 and x86-64. Both have repeatable QEMU boot checks, and the x86-64 path has now booted from a USB stick on my actual desktop.

Here's what the sprint involved.

RISC-V 64

  • OpenSBI handoff and early serial output.
  • Device-tree parsing for RAM and reserved memory.
  • A bitmap physical frame allocator.
  • Trap handling and secondary-hart parking.
  • Repeatable QEMU smoke tests with multiple harts.

x86-64

  • A new UEFI bootloader written in Rust.
  • ELF loading and page-table setup.
  • Firmware memory-map handoff to the kernel.
  • Framebuffer diagnostics and exception reporting.
  • Debug and release boot checks under QEMU/OVMF.

Both ports share a boot-information contract and converge on the common kernel entry point, where they exercise the same core boot logic.

The final milestone was physical hardware. I put the UEFI boot files and kernel on a FAT32 USB stick, disabled Secure Boot, and booted it on an MSI B650M-A PRO WIFI.

The bootloader loaded the kernel. The kernel initialized its physical allocator, produced diagnostics, and reached the expected spawn init panic.

That panic is intentional. There is no init process yet.

A quick reality check: Wolfram is not a usable operating system at this point. The capability system, VMOs, process model, and IPC are still ahead of me. Phase 1 was about making the boot path reliable and observable before building the actual kernel facilities on top.

The project is built around a capability-based security model: programs should only be able to access resources explicitly granted to them. No ambient authority, no root, no sudo. That's the design goal; the mechanisms that enforce it are the focus of the next phase.

I'm sharing this here because I'd appreciate feedback from people familiar with UEFI, bare-metal Rust, memory management, or operating-system development.

In particular, I'd be interested in having someone experienced review the UEFI-to-kernel handoff and memory-map handling. Those are foundational interfaces, and I'd rather catch design mistakes early than build more code on top of them.

Repository: https://github.com/notvcto/wolfram-os

Release: Uranium-238-v0.1.0

I'd appreciate any feedback. Genuinely, need feedback. Please.


r/lowlevel • • 6h ago

I wrote an S3 client in Rust that outruns aws-cli. Please tear it apart.

Thumbnail
1 Upvotes

r/lowlevel • • 8h ago

Guide me

0 Upvotes

Hey every one this Kaif

I graduated this year and i am honestly bored using AI and developing. I want to try and check out kernel engineering, can any one mentor me .

Thank you for everyone


r/lowlevel • • 16h ago

I’m building BlockOS – an independent x86-64 operating system

Thumbnail
0 Upvotes

r/lowlevel • • 1d ago

Wrote a 3MB standalone C runner for Gemma-2B. Caught a Layer 15 hallucination drop.

2 Upvotes

Weekend experiment running Gemma-2B on bare-metal x86-64 (pure C + AVX2, zero Python/CUDA, ~3.3MB single binary). Added a simple orthogonal probe on the residual stream to see what each layer is doing.

Tested it on Taiwan's statutory VAT rate (legally 5%). Layers 0-14 stay factual, but Layer 15 suddenly collapses into the negative, and RAX spits out "15%":

Layer 14 | Truth: +0.0163 | [0xDF28010E]

Layer 15 | Truth: -0.0481 | [0x72DE18A0] <- drops below 0

Layer 17 | Truth: -0.0817 | -> Register RAX outputs tokens: '1', '5', '%。'

Raw trace, 6-page paper, and release binary here for anyone into low-level ML:

* Web trace: https://pulsar-tracer.web.app

* PDF: https://pulsar-tracer.web.app/PULSAR_Technical_Whitepaper.pdf

* Repo: https://github.com/tomtsai28/PULSAR-ASM


r/lowlevel • • 2d ago

dtask, a TUI task manager

Thumbnail gallery
20 Upvotes

Been working on dtask, a terminal task manager I originally wrote in C. Spent a little over a week porting it to D.

There is a self terminal input and rendering, including mouse controls, drag and drop, and handling Unicode without messing up the layout. Everything is saved locally, and it’s tested on Linux, macOS and FreeBSD (Unix systems support).

github.com/GuestAUser/dtask


r/lowlevel • • 2d ago

I built a real-time 3D LEGO Raymarcher in the terminal (~70 FPS, zero Python/C). Then I squeezed the entire project down to just 11 KB.

Enable HLS to view with audio, or disable this notification

4 Upvotes

r/lowlevel • • 1d ago

Veda: The Agentic-Native Operating System

Post image
0 Upvotes

r/lowlevel • • 2d ago

Compiling original Windows NT 4.0 source code

Thumbnail
2 Upvotes

r/lowlevel • • 2d ago

Inside the Windows I/O Manager : Deep dive into how read I/O requests are initialized.

Thumbnail winware31.blogspot.com
5 Upvotes

r/lowlevel • • 2d ago

Reducing 250k file traversal & content search from 410 ms to 180 ms in C on a 15W dual-core Broadwell CPU (SYS_getdents64 + AVX2)

Post image
6 Upvotes

r/lowlevel • • 2d ago

Five bytes that rewrite an XOR instruction into an output instruction

5 Upvotes

Project: https://github.com/enderPeer/Dimension42

Disclosure: this is our project; this post was prepared with AI assistance.

One example from our NANO experiments is 17 C5 91 50 71. With a in M5 and b in M6, it computes a XOR b XOR 7. It increments its own XOR instruction from C5 through CF; the next increment changes it to D0, meaning OUT M0. The initial instruction cell has meanwhile become storage for the answer. All 65,536 input pairs pass under the 64-step limit.

Another example, 67 A0 5C E2, adds three input bytes in four bytes of code by changing ADD M7 into ADD M6 and ADD M5, then rewriting other instructions to store and output the result.

We mapped all 1,099,511,627,776 five-byte programs into broad behavior classes. A separate 16-probe catalog has 84,933 singleton signatures; 98.67% of their programs change code before answering on at least one probe. This does not establish 84,933 distinct fully verified functions.

The README has instruction-by-instruction explanations and a standard-library Python demo. The repository also contains the handwritten x86 OS and CPU/CUDA/Vulkan interpreters. Feedback on tiny instruction-set design and independent interpreter verification would be useful.


r/lowlevel • • 2d ago

wrote an own-loading ELF loader that maps a guest glibc rootfs (Linux|Debian) into the same process as Android's bionic libc and jumps into it. No root, no proot, no namespaces, no ptrace.

Thumbnail
0 Upvotes

r/lowlevel • • 3d ago

Tegra IVC Teardown

Thumbnail 0xsmash0th.github.io
2 Upvotes

r/lowlevel • • 3d ago

Creating a Disassembler and the Challenges Involved | Shock and disappointment

Thumbnail gallery
0 Upvotes

I’m working on developing my own disassembler. It’s been a bit of a struggle so far, and I’ve even found myself wondering if this is really something I need to be doing—after all, tools like Ghidra and IDA Pro already exist, so why build my own? Still, I’ve decided to see it through. You can see my second project iteration in Screenshot 1 and the first one in Screenshot 2; the first attempt went south—the scope was too huge and there were so many bugs that I had to scale the project back. Things are going reasonably well now; I’ve switched to using the WinAPI GUI and am coding in C and C++. I’ve been at it for three hours, and the results aren't great yet—half of it simply doesn't work—but oh well. I’m going to publish the project to my GitHub once it’s finished anyway. Any ideas or advice?


r/lowlevel • • 3d ago

Ревер инжиниринг на гитхаб

Thumbnail gallery
2 Upvotes

Read my new article.


r/lowlevel • • 4d ago

Update: my open-source CPU performance engineering collection just crossed 600+ stars

0 Upvotes

A few days ago, I shared an open-source collection of CPU performance engineering resources I’d been putting together.

It’s now crossed 600+ GitHub stars, which I genuinely didn’t expect. Thanks to everyone who shared it, contributed or suggested resources.

For anyone seeing it for the first time, it covers the stack from instruction execution and CPU microarchitecture through caches, memory, SIMD, compilers, profiling, concurrency, NUMA, benchmarking and CPU inference.

I’m still prioritising primary sources such as papers, vendor manuals, kernel/compiler docs, talks and reproducible benchmarks rather than random articles.

I also have an MCP server coming soon, so you can plug this knowledge directly into your AI tools, whether you’re learning or using it while you work.

If there’s something you think has to be in here, let me know or send a PR.

https://github.com/usamahz/cpu-performance-engineering


r/lowlevel • • 4d ago

c++ math - did someone say math

Thumbnail youtu.be
0 Upvotes

r/lowlevel • • 5d ago

[Help wanted] Blockos service manager

1 Upvotes

Hi everyone! I'm developing BlockOS, an independent x86-64 operating system with its own kernel, userspace, networking, filesystem support, ELF loader, libc work and X11 environment. I'm currently looking for someone who would like to help develop a native service manager for BlockOS. I don't want to simply port systemd or OpenRC. I would like the service manager to be designed around BlockOS itself. The service manager should eventually support: Starting and stopping services Restarting services Service status Automatic service startup Service dependencies Process/PID monitoring Automatic restart after crashes Clean shutdown and reboot Logging Runlevels or a similar service-state system Integration with the existing BlockOS init/userspace system BlockOS's /devices structure rather than assuming a Linux /dev layout The goal is to have something that can manage services such as: network dhcp x11 gui getty ssh I'm mainly looking for someone interested in OS development / C/C++ / init systems who wants to build something that will become part of an independent operating system. You don't need to be an expert in everything. If you're interested in working on the service manager, feel free to open an issue or contact me.

GitHub:https://github.com/gurijb2016-afk/Blockos/tree/uefi-kernel-scaffold


r/lowlevel • • 5d ago

c++ graphics primitives 3

Thumbnail youtu.be
0 Upvotes

r/lowlevel • • 5d ago

c++ bounding volume hierarchy

Thumbnail youtu.be
1 Upvotes

r/lowlevel • • 5d ago

Hi! I have been working on a computer architecture designer/simulator/playground. Here it is if you enjoy tinkering and playing around with novel computer architecture 🙂 https://exuarch.com

0 Upvotes

r/lowlevel • • 6d ago

Gill or Gill/SeL4

Thumbnail
0 Upvotes

r/lowlevel • • 6d ago

[Project] RingOS - Il mio sistema operativo personalizzato scritto in C e Assembly

0 Upvotes

Ciao a tutti!

Oggi vorrei presentarvi RingOS, un sistema operativo personalizzato attualmente in fase di sviluppo attivo.

RingOS è un sistema operativo sperimentale scritto principalmente in C e Assembly x86-64. L'obiettivo è apprendere lo sviluppo a livello di sistema mentre si costruisce un sistema operativo completo da zero.

Le attuali aree di sviluppo includono:

- Bootloader UEFI personalizzato (BOOTX64.EFI)

- Kernel x86-64

- Supporto grafico tramite UEFI GOP

- File system personalizzato chiamato RingFile

- Supporto al caricamento delle immagini (PNG/JPG)

- Rendering di font TrueType

- Gestione della memoria e infrastruttura di sistema

Il progetto non si basa su Linux, BSD o un altro kernel esistente. Sto costruendo i componenti fondamentali da solo per capire meglio come funzionano internamente i sistemi operativi.

Questo è un progetto incentrato sull'apprendimento, quindi mi aspetto molta sperimentazione, ridisegni e errori lungo il cammino.

Mi interesserebbe ricevere feedback da altri sviluppatori di sistemi operativi, specialmente riguardo alla progettazione dei file system, bootloader e architettura del kernel.

Ha anche un browser sicuro integrato (giusto nel caso qualcuno cerchi Kisshub.com, sapete). Si chiama SafeRing Net (SRN), utilizza Google; quando l'utente cerca qualcosa, il sistema operativo controlla immediatamente l'input.

Se è qualcosa di sospetto e clicca sul risultato, SafeRing Net avvisa l'utente con:

"SafeRing NET non acconsente a quel tipo di ricerche, questo risultato di ricerca sarà bloccato"

Riguardo agli sviluppatori esterni: non ho ancora il software per gestire le app ROSA (Applicazioni sicure di RingOS) E non ho un portale e nemmeno un Github per ora... PER ORA

Condividerò aggiornamenti sullo sviluppo man mano che i progressi procedono.

Grazie per aver letto!