[project] README updates

Stub out math.h in order to include c++ headers that include it but
don't actually call any math functions.

.clang-format

@@ -1,21 +0,0 @@
----
-Language:        Cpp
-BasedOnStyle:    LLVM
-
-ColumnLimit:     100
-IndentWidth:     4
-TabWidth:        4
-UseTab:          Always
-
-AccessModifierOffset: -4
-
-AlignEscapedNewlinesLeft: true
-
-AllowShortIfStatementsOnASingleLine: true
-AllowShortLoopsOnASingleLine: true
-
-AlwaysBreakAfterReturnType: AllDefinitions
-AlwaysBreakBeforeMultilineStrings: true
-AlwaysBreakTemplateDeclarations: true
-
-BreakBeforeBraces: Linux

.gitignore

@@ -1,3 +1,4 @@
+.cache
 .lock*
 /build*
 *.bak
@@ -10,3 +11,5 @@ sysroot
 .gdb_history
 .peru
 __pycache__
+/venv
+compile_commands.json

NOTES.md

@@ -1,38 +0,0 @@
-# Design / WIP notes
-
-## TODO
-
-- Paging manager
-  - Copy-on-write pages
-  - Better page-allocation model?
-- Allow for more than one IOAPIC in ACPI module
-  - The objects get created, but GSI lookup only uses the one at index 0
-- mark kernel memory pages global
-- Serial out based on circular/bip biffer and interrupts, not spinning on
-  `write_ready()`
-- Split out more code into kutil for testing
-- AHCI / MSI interrupts on Vbox break?
-- FXSAVE to save XMM registers.
-  - optimization using #NM (0x7) to detect SSE usage
-- Clean up of process memory maps
-- Better stack tracer
-- Bootloader rewrite
-  - C++ and sharing library code for ELF, initrd, etc
-  - Parse initrd and pre-load certain ELF images, eg the process loader process?
-  - Do initial memory bootstrap?
-- Calling global ctors
-- Device Tree
-  - Actual serial driver
-  - Disk driver
-    - File system
-- Multiprocessing
-  - Fast syscalls using syscall/sysret
-
-### Build
-
-- Clean up build generator and its templates
-  - More robust objects to represent modules & targets to pass to templates
-  - Read project setup from a simple JSON/TOML/etc
-  - Better compartmentalizing when doing template inheritance
-- Move to LLD as sysroot linker
-

README.md

@@ -3,13 +3,14 @@
 # The jsix operating system
 
 **jsix** is a custom multi-core x64 operating system that I am building from
-scratch. It's far from finished, or even being usable - see the *Status and
+scratch. It's far from finished, or even being usable (see the *Status and
-Roadmap* section, below.
+Roadmap* section, below) but all currently-planned major kernel features are
+now implemented to at least a passable level.
 
 The design goals of the project are:
 
 * Modernity - I'm not interested in designing for legacy systems, or running on
-  all hardware out there. My target is only 64 bit architecutres, and modern
+  all hardware out there. My target is only 64 bit architectures, and modern
   commodity hardware. Currently that means x64 systems with Nehalem or newer
   CPUs and UEFI firmware. (See [this list][cpu_features] for the currently
   required CPU features.) Eventually I'd like to work on an AArch64 port,
@@ -58,12 +59,18 @@ Remaining to do:
 
 - TLB shootdowns
 - Page swapping
+- Large / huge page support
 
 _Physical page allocation: Sufficient._ The current physical page allocator
-implementation suses a group of block representing up-to-1GiB areas of usable
+implementation uses a group of blocks representing up-to-1GiB areas of usable
 memory as defined by the bootloader. Each block has a three-level bitmap
 denoting free/used pages.
 
+Future work:
+
+- Align blocks so that their first page is 1GiB-aligned, making finding free
+  large/huge pages easier.
+
 #### Multitasking
 
 _Sufficient._ The global scheduler object keeps separate ready/blocked lists
@@ -71,72 +78,104 @@ per core. Cores periodically attempt to balance load via work stealing.
 
 User-space tasks are able to create threads as well as other processes.
 
-Several kernel-only tasks exist, though I'm trying to reduce that. Eventually
-only the timekeeping task should be a separate kernel-only thread.
-
 #### API
 
-_In progress._ User-space tasks are able to make syscalls to the kernel via
+_Syscalls: Sufficient._ User-space tasks are able to make syscalls to the
-fast SYSCALL/SYSRET instructions.
+kernel via fast SYSCALL/SYSRET instructions. Syscalls made via `libj6` look to
+both the callee and the caller like standard SysV ABI function calls. The
+implementations are wrapped in generated wrapper functions which validate the
+request, check capabilities, and find the appropriate kernel objects or handles
+before calling the implementation functions.
 
-Major tasks still to do:
+_IPC: Working, needs optimization._ The current IPC primitives are:
 
-- The process initialization protocol needs to be re-built entirely.
+- _Mailboxes_: endpoints for asynchronously-delivered small messages. Currently
-- Processes' handles to kernel objects need the ability to check capabilities
+  these messages are double-copied - once from the caller into a kernel buffer,
+  and once from the kernel to the receiver. This works and is not a major cause
+  of slowdown, but will need to be optimized in the future.
+- _Channels_: endpoints for asynchronous uni-directional streams of bytes.
+  Currently these also suffer from a double-copy problem, and should probably
+  be replaced eventually by userspace shared memory communication.
+- _Events_: objects that can be signalled to send asynchronous notifications to
+  waiting threads.
 
 #### Hardware Support
 
-  * Framebuffer driver: _In progress._ Currently on machines with a video
+- Framebuffer driver: _In progress._ Currently on machines with a video
-	device accessible by UEFI, jsix starts a user-space framebuffer driver that
+  device accessible by UEFI, jsix starts a user-space framebuffer driver that
-	only prints out kernel logs.
+  only prints out kernel logs.
-  * Serial driver: _To do._ Machines without a video device should have a
+- Serial driver: _In progress._ The current UART currently only exposes COM1
-	user-space log output task like the framebuffer driver, but currently this
+  as an output channel, but no input or other serial ports are exposed.
-	is done inside the kernel.
+- USB driver: _To do_
-  * USB driver: _To do_
+- AHCI (SATA) driver: _To do_
-  * AHCI (SATA) driver: _To do_
 
 ## Building
 
-jsix uses the [Ninja][] build tool, and generates the build files for it with a
+jsix uses the [Ninja][] build tool, and generates the build files for it with
-custom tool called [Bonnibel][]. Bonnibel can be installed with [Cargo][], or
+the `configure` script. The build also relies on a custom toolchain sysroot, which can be
-downloaded as a prebuilt binary from its Github repository.
+downloaded or built using the scripts in [jsix-os/toolchain][].
 
-[Ninja]:    https://ninja-build.org
+[Ninja]:             https://ninja-build.org
-[Bonnibel]: https://github.com/justinian/bonnibel_rs
+[jsix-os/toolchain]: https://github.com/jsix-os/toolchain
-[Cargo]:    https://crates.io/crates/bonnibel
 
-Requrirements:
+Other build dependencies:
 
-* bonnibel
+* [clang][]: the C/C++ compiler
-* ninja
+* [nasm][]: the assembler
-* clang
+* [lld][]: the linker
-* nasm
+* [mtools][]: for creating the FAT image
-* mtools
-* curl for downloading the toolchain
 
-### Setting up the cross toolchain
+[clang]:    https://clang.llvm.org
+[nasm]:     https://www.nasm.us
+[lld]:      https://lld.llvm.org
+[mtools]:   https://www.gnu.org/software/mtools/
 
-Running `pb sync` will download and unpack the toolchain into `sysroot`. 
+The `configure` script has some Python dependencies - these can be installed via
+`pip`, though doing so in a python virtual environment is recommended.
+Installing via `pip` will also install `ninja`.
 
-#### Compiling the toolchain yourself
+A Debian 11 (Bullseye) system can be configured with the necessary build
+dependencies by running the following commands from the jsix repository root:
 
-If you have `clang` and `curl` installed, runing the `scripts/build_sysroot.sh`
+```bash
-script will download and build a LLVM toolchain configured for building jsix
+sudo apt install clang lld nasm mtools python3-pip python3-venv
-host binaries.
+python3 -m venv ./venv
+source venv/bin/activate
+pip install -r requirements.txt
+peru sync
+```
+
+### Setting up the sysroot
+
+Build or download the toolchain sysroot as mentioned above with
+[jsix-os/toolchain][], and symlink the built toolchain directory as `sysroot`
+at the root of this project.
+
+```bash
+# Example if both the toolchain and this project are cloned under ~/src
+ln -s ~/src/toolchain/toolchains/llvm-13 ~/src/jsix/sysroot
+```
 
 ### Building and running jsix
 
-Once the toolchain has been set up, running Bonnibel's `pb init` command will
+Once the toolchain has been set up, running the `./configure` script (see
-set up the build configuration, and `pb build` will actually run the build.  If
+`./configure --help` for available options) will set up the build configuration,
-you have `qemu-system-x86_64` installed, the `qemu.sh` script will to run jsix
+and `ninja -C build` (or wherever you put the build directory) will actually run
-in QEMU `-nographic` mode.
+the build. If you have `qemu-system-x86_64` installed, the `qemu.sh` script will
+to run jsix in QEMU `-nographic` mode.
 
-I personally run this either from a real debian amd64 testing/buster machine or
+I personally run this either from a real debian amd64 bullseye machine or
-a windows WSL debian testing/buster installation. The following should be
+a windows WSL debian bullseye installation. Your mileage may vary with other
-enough to set up such a system to build the kernel:
+setups and distros.
 
-    sudo apt install qemu-system-x86 nasm clang-10 mtools curl ninja-build
+### Running the test suite
-    sudo update-alternatives /usr/bin/clang clang /usr/bin/clang-10 1000
-    sudo update-alternatives /usr/bin/clang++ clang++ /usr/bin/clang++-10 1000
-	curl -L -o pb https://github.com/justinian/bonnibel_rs/releases/download/v2.3.0/pb-linux-amd64 && chmod a+x pb
 
+jsix now has the `test_runner` userspace program that runs various automated
+tests. It is not included in the default build, but if you use the `test.yml`
+manifest it will be built, and can be run with the `test.sh` script or the
+`qemu.sh` script.
+
+```bash
+./configure --manifest=assets/manifests/test.yml
+if ./test.sh; then echo "All tests passed!"; else echo "Failed."; fi
+```

assets/debugging/jsix.elf-gdb.py

@@ -10,6 +10,7 @@ class PrintStackCommand(gdb.Command):
         base = "$rsp"
         if len(args) > 0:
             base = args[0]
+        base = int(gdb.parse_and_eval(base))
 
         depth = 22
         if len(args) > 1:
@@ -18,14 +19,35 @@ class PrintStackCommand(gdb.Command):
         for i in range(depth-1, -1, -1):
             try:
                 offset = i * 8
-                base_addr = gdb.parse_and_eval(base)
+                value = gdb.parse_and_eval(f"*(uint64_t*)({base:#x} + {offset:#x})")
-                value = gdb.parse_and_eval(f"*(uint64_t*)({base} + {offset})")
+                print("{:016x} (+{:04x}): {:016x}".format(base + offset, offset, int(value)))
-                print("{:016x} (+{:04x}): {:016x}".format(int(base_addr) + offset, offset, int(value)))
             except Exception as e:
                 print(e)
                 continue
 
 
+def stack_walk(frame, depth):
+    for i in range(depth-1, -1, -1):
+        ret = gdb.parse_and_eval(f"*(uint64_t*)({frame:#x} + 0x8)")
+
+        name = ""
+        try:
+            block = gdb.block_for_pc(int(ret))
+            if block:
+                name = block.function or ""
+        except RuntimeError:
+            pass
+
+        try:
+            print("{:016x}: {:016x} {}".format(int(frame), int(ret), name))
+            frame = int(gdb.parse_and_eval(f"*(uint64_t*)({frame:#x})"))
+        except gdb.MemoryError:
+            return
+
+        if frame == 0 or ret == 0:
+            return
+
+
 class PrintBacktraceCommand(gdb.Command):
     def __init__(self):
         super().__init__("j6bt", gdb.COMMAND_DATA)
@@ -33,27 +55,17 @@ class PrintBacktraceCommand(gdb.Command):
     def invoke(self, arg, from_tty):
         args = gdb.string_to_argv(arg)
 
-        depth = 30
-        if len(args) > 0:
-            depth = int(args[0])
-
         frame = "$rbp"
+        if len(args) > 0:
+            frame = args[0]
+
+        frame = int(gdb.parse_and_eval(f"{frame}"))
+
+        depth = 30
         if len(args) > 1:
-            frame = args[1]
+            depth = int(gdb.parse_and_eval(args[1]))
 
-        for i in range(depth-1, -1, -1):
+        stack_walk(frame, depth)
-            ret = gdb.parse_and_eval(f"*(uint64_t*)({frame} + 8)")
-            frame = gdb.parse_and_eval(f"*(uint64_t*)({frame})")
-
-            name = ""
-            block = gdb.block_for_pc(int(ret))
-            if block:
-                name = block.function or ""
-
-            print("{:016x} {}".format(int(ret), name))
-
-            if frame == 0 or ret == 0:
-                return
 
 
 class TableWalkCommand(gdb.Command):
@@ -109,7 +121,7 @@ class TableWalkCommand(gdb.Command):
         table = pml4
         entry = 0
         for i in range(len(indices)):
-            entry = int(gdb.parse_and_eval(f'((uint64_t*){table})[{indices[i]}]'))
+            entry = int(gdb.parse_and_eval(f'((uint64_t*)0x{table:x})[0x{indices[i]:x}]'))
             flagset = flagsets[i]
             flag_names = " | ".join([f[1] for f in flagset if (entry & f[0]) == f[0]])
 
@@ -123,10 +135,135 @@ class TableWalkCommand(gdb.Command):
             table = (entry & 0x7ffffffffffffe00) | 0xffffc00000000000
 
 
+class GetThreadsCommand(gdb.Command):
+    FLAGS = {
+        "ready":    0x01,
+        "loading":  0x02,
+        "exited":   0x04,
+        "constant": 0x80,
+    }
+
+    def __init__(self):
+        super().__init__("j6threads", gdb.COMMAND_DATA)
+
+    def get_flags(self, bitset):
+        flags = []
+        for k, v in GetThreadsCommand.FLAGS.items():
+            if bitset & v:
+                flags.append(k)
+        return " ".join(flags)
+
+    def print_thread(self, addr):
+        if addr == 0:
+            print("  <no thread>\n")
+            return
+
+        tcb = f"((TCB*){addr:#x})"
+        thread = f"({tcb}->thread)"
+        stack = int(gdb.parse_and_eval(f"{tcb}->kernel_stack"))
+        rsp = int(gdb.parse_and_eval(f"{tcb}->rsp"))
+        pri = int(gdb.parse_and_eval(f"{tcb}->priority"))
+        flags = int(gdb.parse_and_eval(f"{thread}->m_state"))
+        koid = int(gdb.parse_and_eval(f"{thread}->m_koid"))
+        proc = int(gdb.parse_and_eval(f"{thread}->m_parent.m_koid"))
+
+        creator = int(gdb.parse_and_eval(f"{thread}->m_creator"))
+        if creator != 0:
+            creator_koid = int(gdb.parse_and_eval(f"{thread}->m_creator->m_koid"))
+            creator = f"{creator_koid:x}"
+        else:
+            creator = "<no thread>"
+
+        print(f"  Thread {proc:x}:{koid:x}")
+        print(f"         creator: {creator}")
+        print(f"        priority: {pri}")
+        print(f"           flags: {self.get_flags(flags)}")
+        print(f"          kstack: {stack:#x}")
+        print(f"             rsp: {rsp:#x}")
+        print("------------------------------------")
+
+        if stack != 0:
+            rsp = int(gdb.parse_and_eval(f"{tcb}->rsp"))
+            stack_walk(rsp + 5*8, 5)
+
+        print("")
+
+    def print_thread_list(self, addr, name):
+        if addr == 0:
+            return
+
+        print(f"=== {name} ===")
+
+        while addr != 0:
+            self.print_thread(addr)
+            addr = int(gdb.parse_and_eval(f"((tcb_node*){addr:#x})->m_next"))
+
+
+    def invoke(self, arg, from_tty):
+        args = gdb.string_to_argv(arg)
+        if len(args) > 1:
+            raise RuntimeError("Usage: j6threads [cpu]")
+
+        ncpus = int(gdb.parse_and_eval("g_num_cpus"))
+        cpus = list(range(ncpus))
+        if len(args) == 1:
+            cpus = [int(args[0])]
+
+        for cpu in cpus:
+            runlist = f"scheduler::s_instance->m_run_queues.m_elements[{cpu:#x}]"
+
+            print(f"=== CPU {cpu:2}: CURRENT ===")
+            current = int(gdb.parse_and_eval(f"{runlist}.current"))
+            self.print_thread(current)
+
+            for pri in range(8):
+                ready = int(gdb.parse_and_eval(f"{runlist}.ready[{pri:#x}].m_head"))
+                self.print_thread_list(ready, f"CPU {cpu:2}: PRIORITY {pri}")
+
+            blocked = int(gdb.parse_and_eval(f"{runlist}.blocked.m_head"))
+            self.print_thread_list(blocked, f"CPU {cpu:2}: BLOCKED")
+
+
+class PrintProfilesCommand(gdb.Command):
+    def __init__(self):
+        super().__init__("j6prof", gdb.COMMAND_DATA)
+
+    def invoke(self, arg, from_tty):
+        args = gdb.string_to_argv(arg)
+        if len(args) != 1:
+            raise RuntimeError("Usage: j6prof <profiler class>")
+
+        profclass = args[0]
+        root_type = f"profile_class<{profclass}>"
+
+        try:
+            baseclass = gdb.lookup_type(root_type)
+        except Exception as e:
+            print(e)
+            return
+
+        results = {}
+        max_len = 0
+        count = gdb.parse_and_eval(f"{profclass}::count")
+        for i in range(count):
+            name = gdb.parse_and_eval(f"{root_type}::function_names[{i:#x}]")
+            if name == 0: continue
+
+            call_counts = gdb.parse_and_eval(f"{root_type}::call_counts[{i:#x}]")
+            call_durations = gdb.parse_and_eval(f"{root_type}::call_durations[{i:#x}]")
+            results[name.string()] = float(call_durations) / float(call_counts)
+            max_len = max(max_len, len(name.string()))
+
+        for name, avg in results.items():
+            print(f"{name:>{max_len}}: {avg:15.3f}")
 
 PrintStackCommand()
 PrintBacktraceCommand()
 TableWalkCommand()
+GetThreadsCommand()
+PrintProfilesCommand()
 
-gdb.execute("target remote :1234")
 gdb.execute("display/i $rip")
+if not gdb.selected_inferior().was_attached:
+    gdb.execute("add-symbol-file build/panic.serial.elf")
+    gdb.execute("target remote :1234")

assets/grammars/definitions.g

@@ -0,0 +1,47 @@
+start: import_statement* (object|interface)+
+
+import_statement: "import" PATH
+
+object: description? "object" name options? super? "{" uid cname? capabilities? method* "}"
+
+interface: description? "interface" name options? "{" uid interface_param* "}"
+
+?interface_param: expose | function
+
+expose: "expose" type
+
+uid: "uid" UID
+
+cname: "cname" IDENTIFIER
+
+capabilities: "capabilities" "[" IDENTIFIER+ "]"
+
+super: ":" name
+
+function: description? "function" name options? ("{" param* "}")?
+
+method: description? "method" name options? ("{" param* "}")?
+
+param: "param" name type options? description?
+
+?type: PRIMITIVE | object_name
+
+object_name: "ref" name
+
+id: NUMBER
+name: IDENTIFIER
+options: "[" ( OPTION | IDENTIFIER )+ "]"
+description: COMMENT+
+
+PRIMITIVE: INT_TYPE | "size" | "string" | "buffer" | "address"
+INT_TYPE: /u?int(8|16|32|64)?/
+NUMBER: /(0x)?[0-9a-fA-F]+/
+UID: /[0-9a-fA-F]{16}/
+OPTION.2: IDENTIFIER ":" IDENTIFIER
+COMMENT: /#.*/
+PATH: /"[^"]*"/
+
+%import common.LETTER
+%import common.CNAME -> IDENTIFIER
+%import common.WS
+%ignore WS

assets/initrd.toml

@@ -1,28 +0,0 @@
-# This is the manifest for the initial ramdisk, read by the `makerd` tool.
-# The contents should be a table array of files to add to the ramdistk:
-#
-#  [[files]]
-#  dest = "foo.bar"             # Name of the file in the ramdisk
-#  source = "build/foo/foo.bar" # Location of the file from the project root
-#  executable = true            # Optional, default false. Whether this is an
-#                               # initial application for the kernel to execute
-#                               # on startup
-
-[[files]]
-dest = "screenfont.psf"
-source = "../assets/fonts/tamsyn8x16r.psf"
-
-[[files]]
-dest = "symbol_table.dat"
-source = "symbol_table.dat"
-symbols = true
-
-[[files]]
-dest = "nulldrv1"
-source = "user/nulldrv"
-executable = true
-
-[[files]]
-dest = "nulldrv2"
-source = "user/nulldrv"
-executable = true

assets/manifests/default.yaml

@@ -0,0 +1,8 @@
+---
+init: srv.init
+programs:
+  - name: panic.serial
+    target: kernel
+    flags: panic
+  - name: srv.logger
+  - name: drv.uart

assets/manifests/test.yaml

@@ -0,0 +1,9 @@
+---
+init: srv.init
+flags: ["test"]
+programs:
+  - name: panic.serial
+    target: kernel
+    flags: panic
+  - name: drv.uart
+  - name: test_runner

configs/base.yaml

@@ -0,0 +1,39 @@
+---
+variables:
+  cc: "${source_root}/sysroot/bin/clang"
+  cxx: "${source_root}/sysroot/bin/clang++"
+  ld: "${source_root}/sysroot/bin/ld.lld"
+  ar: ar
+  nasm: nasm
+  objcopy: objcopy
+
+  ccflags: [
+    "-I${source_root}/src/include",
+    "-I${source_root}/src/include/x86_64",
+    "-fcolor-diagnostics",
+    "-U__STDCPP_THREADS__",
+    "-D_LIBCPP_HAS_NO_THREADS",
+    "-DVERSION_MAJOR=${version_major}",
+    "-DVERSION_MINOR=${version_minor}",
+    "-DVERSION_PATCH=${version_patch}",
+    "-DVERSION_GITSHA=0x${version_sha}",
+    '-DGIT_VERSION=\"${version_major}.${version_minor}.${version_patch}+${version_sha}\"',
+    '-DGIT_VERSION_WIDE=L\"${version_major}.${version_minor}.${version_patch}+${version_sha}\"',
+
+    "-Wformat=2", "-Winit-self", "-Wfloat-equal", "-Winline", "-Wmissing-format-attribute",
+    "-Wmissing-include-dirs", "-Wswitch", "-Wundef", "-Wdisabled-optimization",
+    "-Wpointer-arith", "-Wno-attributes", "-Wno-sign-compare", "-Wno-multichar",
+    "-Wno-div-by-zero", "-Wno-endif-labels", "-Wno-pragmas", "-Wno-format-extra-args",
+    "-Wno-unused-result", "-Wno-deprecated-declarations", "-Wno-unused-function",
+    "-Wno-address-of-packed-member", "-Wno-invalid-offsetof", "-Wno-format-nonliteral",
+    "-Werror" ]
+
+  asflags: [
+    "-DVERSION_MAJOR=${version_major}",
+    "-DVERSION_MINOR=${version_minor}",
+    "-DVERSION_PATCH=${version_patch}",
+    "-DVERSION_GITSHA=0x${version_sha}",
+    "-I${source_root}/src/include" ]
+
+  cflags: [ "-std=c11" ]
+  cxxflags: [ "-std=c++17" ]

configs/boot-debug.yaml

@@ -0,0 +1,30 @@
+---
+extends: base
+
+variables:
+  ld: clang++
+
+  ccflags: [
+    "-nostdlib",
+    "-nodefaultlibs",
+    "-fno-builtin",
+
+    "-I${source_root}/external",
+    "--target=x86_64-unknown-windows",
+    "-ffreestanding",
+    "-mno-red-zone",
+    "-fshort-wchar",
+    "-fno-omit-frame-pointer",
+    "-ggdb",
+    "-g3" ]
+
+  cxxflags: [ "-fno-exceptions", "-fno-rtti" ]
+
+  ldflags: [
+    "--target=x86_64-unknown-windows",
+    "-nostdlib",
+    "-Wl,-entry:efi_main",
+    "-Wl,-subsystem:efi_application",
+    "-fuse-ld=lld-link",
+    "-g" ]
+

configs/kernel-debug.yaml

@@ -0,0 +1,55 @@
+---
+extends: base
+
+variables:
+  asflags: [ "-I${source_root}/src/kernel/" ]
+
+  ccflags: [
+    "--target=x86_64-jsix-elf",
+    "-fno-stack-protector",
+
+    "-I${source_root}/external",
+
+    "-nostdinc",
+    "-nostdlib",
+    "-ffreestanding",
+    "-nodefaultlibs",
+    "-fno-builtin",
+
+    "-mno-sse",
+    "-fno-omit-frame-pointer",
+    "-mno-red-zone",
+    "-mcmodel=kernel",
+
+    "-g3",
+    "-ggdb",
+
+    "-D__ELF__",
+    "-D__jsix__",
+    "-D__j6kernel",
+    "-U__linux",
+    "-U__linux__",
+    "-DPRINTF_ALIAS_STANDARD_FUNCTION_NAMES=1",
+    "-DPRINTF_INCLUDE_CONFIG_H=1",
+
+    "-isystem${build_root}/include/libc",
+    "-isystem${source_root}/sysroot/include",
+    "--sysroot='${source_root}/sysroot'" ]
+
+
+  cflags: [ '-nostdinc' ]
+
+  cxxflags: [
+    "-fno-exceptions",
+    "-fno-rtti",
+    "-nostdinc",
+    "-isystem${source_root}/sysroot/include/c++/v1" ]
+
+  ldflags: [
+    "-g",
+    "-nostdlib",
+    "-Bstatic",
+    "--no-eh-frame-hdr",
+    "-z", "norelro",
+    "-z", "separate-code" ]
+

configs/rules.ninja

@@ -0,0 +1,81 @@
+# This file is automatically generated by bonnibel
+
+rule compile.c
+  command = $cc -MMD -MF $out.d $cflags $ccflags -o $out -c $in
+  description = Compiling $name
+  depfile = $out.d
+  deps = gcc
+
+rule dump_c_defs
+  command = echo | $cc $ccflags $cflags -dM -E - > $out
+  description = Dumping C defines for $target
+
+rule dump_c_run
+  command = echo '#!/bin/bash' > $out; echo '$cc $ccflags $cflags $$*' >> $
+      $out; chmod a+x $out
+  description = Dumping C arguments for $target
+
+rule compile.cpp
+  command = $cxx -MMD -MF $out.d $cxxflags $ccflags -o $out -c $in
+  description = Compiling $name
+  depfile = $out.d
+  deps = gcc
+
+rule dump_cpp_defs
+  command = echo | $cxx -x c++ $ccflags $cxxflags -dM -E - > $out
+  description = Dumping C++ defines for $target
+
+rule dump_cpp_run
+  command = echo '#!/bin/bash' > $out; echo '$cxx $ccflags $cxxflags $$*' $
+      >> $out; chmod a+x $out
+  description = Dumping C++ arguments for $target
+
+rule compile.s
+  command = $nasm -o $out -felf64 -MD $out.d $asflags $in
+  description = Assembling $name
+  depfile = $out.d
+  deps = gcc
+
+rule parse.cog
+  command = cog -o $out -d -D target=$target $cogflags $in
+  description = Parsing $name
+
+rule exe
+  command = $ld $ldflags -o $out $in $libs
+  description = Linking $name
+
+rule lib
+  command = $ar qcs $out $in
+  description = Archiving $name
+
+rule cp
+  command = cp $in $out
+  description = Copying $name
+
+rule dump
+  command = objdump -DSC -M intel $in > $out
+  description = Dumping decompiled $name
+
+rule makest
+    description = Making symbol table
+    command = nm -n -S --demangle $in | ${source_root}/scripts/build_symbol_table.py $out
+
+rule makefat
+    description = Creating $name
+    command = $
+        cp $in $out; $
+        mcopy -s -D o -i $out@@1M ${build_root}/fatroot/* ::/
+
+rule strip
+    description = Stripping $name
+    command = $
+        cp $in $out; $
+        objcopy --only-keep-debug $out $debug; $
+        strip -g $out; $
+        objcopy --add-gnu-debuglink=$debug $out
+
+rule touch
+    command = touch $out
+
+rule compdb
+    command = ninja -t compdb > $out

configs/user-debug.yaml

@@ -0,0 +1,39 @@
+---
+extends: base
+
+variables:
+  asflags: [ "-I${source_root}/src/kernel/" ]
+
+  ccflags: [
+    "--target=x86_64-jsix-elf",
+    "-mno-sse",
+    "-fno-omit-frame-pointer",
+    "-fno-stack-protector",
+
+    "-g",
+
+    "-D__ELF__",
+    "-D__jsix__",
+    "-U__linux",
+    "-U__linux__",
+
+    "-isystem${source_root}/sysroot/include",
+    "-isystem${build_root}/include/libc",
+    "--sysroot='${source_root}/sysroot'" ]
+
+
+  cxxflags: [
+    "-fno-exceptions",
+    "-fno-rtti",
+    "-isystem${source_root}/sysroot/include/c++/v1" ]
+
+  ldflags: [
+    "-g",
+    "-Bstatic",
+    "--sysroot='${source_root}/sysroot'",
+    "-L", "${source_root}/sysroot/lib",
+    "-z", "separate-code",
+    "-lc++", "-lc++abi", "-lunwind",
+    "--no-dependent-libraries",
+  ]
+

configure

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+
+def generate(output, config, manifest):
+    from os import makedirs, walk
+    from pathlib import Path
+    from bonnibel.module import Module
+    from bonnibel.project import Project
+
+    root = Path(__file__).parent.resolve()
+    project = Project(root)
+
+    output = root / output
+    sources = root / "src"
+    manifest = root / manifest
+
+    modules = {}
+    for base, dirs, files in walk(sources):
+        path = Path(base) 
+        for f in files:
+            if f.endswith(".module"):
+                fullpath = path / f
+                
+                def module_init(name, **kwargs):
+                    m = Module(name, fullpath, path, **kwargs)
+                    modules[m.name] = m
+                    return m
+
+                glo = {
+                    "module": module_init,
+                    "source_root": root,
+                    "built_root": output,
+                    "module_root": path,
+                }
+                code = compile(open(fullpath, 'r').read(), fullpath, "exec")
+
+                loc = {}
+                exec(code, glo, loc)
+
+    Module.update(modules)
+
+    makedirs(output.resolve(), exist_ok=True)
+    project.generate(root, output, modules, config, manifest)
+    for mod in modules.values():
+        mod.generate(output)
+
+if __name__ == "__main__":
+    import sys
+    from pathlib import Path
+    sys.path.insert(0, str(Path(__file__).parent / "scripts"))
+
+    from argparse import ArgumentParser
+    from bonnibel import BonnibelError
+
+    p = ArgumentParser(description="Generate jsix build files")
+    p.add_argument("--manifest", "-m", metavar="FILE", default="assets/manifests/default.yaml",
+            help="File to use as the system manifest")
+    p.add_argument("--config", "-c", metavar="NAME", default="debug",
+            help="Configuration to build (eg, 'debug' or 'release')")
+    p.add_argument("output", metavar="DIR", default="build", nargs='?',
+            help="Where to create the build root")
+
+    args = p.parse_args()
+
+    try:
+        generate(args.output, args.config, args.manifest)
+
+    except BonnibelError as be:
+        import sys
+        print(f"Error: {be}", file=sys.stderr)
+        sys.exit(1)

definitions/memory_layout.yaml

@@ -0,0 +1,20 @@
+---
+- name: linear
+  size: 64T
+  shared: true
+
+- name: bitmap
+  size: 1T
+  shared: true
+
+- name: heap
+  size: 64G
+
+- name: stacks
+  size: 64G
+
+- name: buffers
+  size: 64G
+
+- name: slabs
+  size: 64G

definitions/objects/channel.def

@@ -0,0 +1,21 @@
+object channel : object {
+    uid 3ea38b96aa0e54c8
+
+    capabilities [
+        send
+        receive
+        close
+    ]
+
+    method create [constructor] 
+    method close [destructor cap:close]
+
+    method send [cap:send] {
+        param data buffer [inout]
+    }
+
+    method receive [cap:receive] {
+        param data buffer [out]
+        param flags uint64
+    }
+}

definitions/objects/event.def

@@ -0,0 +1,22 @@
+object event : object {
+    uid f441e03da5516b1a
+
+    capabilities [
+        signal
+        wait
+    ]
+
+    method create [constructor]
+
+    # Signal events on this object
+    method signal [cap:signal] {
+        param signals uint64        # A bitset of which events to signal
+    }
+
+    # Wait for signaled events on this object
+    method wait [cap:wait] {
+        param signals uint64 [out]  # A bitset of which events were signaled
+        param timeout uint64        # Wait timeout in nanoseconds
+    }
+}
+

definitions/objects/mailbox.def

@@ -0,0 +1,65 @@
+# Mailboxes are objects that enable synchronous or asynchronous
+# IPC via short message-passing of bytes and handles.
+
+object mailbox : object {
+    uid 99934ad04ece1e07
+
+    capabilities [
+        send
+        receive
+        close
+    ]
+
+    method create [constructor]
+    method close [destructor cap:close]
+
+    # Asynchronously send a message to the reciever
+    method send [cap:send handle] {
+        param tag uint64
+        param data buffer [zero_ok]
+        param handles ref object [list]
+    }
+
+    # Receive a pending message, or block waiting for a message to
+    # arrive if block is true.
+    method receive [cap:receive] {
+        param tag uint64 [out]
+        param data buffer [out zero_ok]
+        param handles ref object [out list zero_ok]
+        param reply_tag uint16 [out optional]
+        param badge uint64 [out optional]
+        param flags uint64
+    }
+
+    # Send a message to the reciever, and block until a
+    # response is sent. Note that getting this response
+    # does not require the receive capability.
+    method call [cap:send handle] {
+        param tag uint64 [inout]
+        param data buffer [inout zero_ok]
+        param handles ref object [inout list zero_ok]
+    }
+
+    # Respond to a message sent using call. Note that this
+    # requires the receive capability and not the send capability.
+    method respond [cap:receive] {
+        param tag uint64
+        param data buffer [zero_ok]
+        param handles ref object [list zero_ok]
+        param reply_tag uint16
+    }
+
+    # Respond to a message sent using call, and wait for another
+    # message to arrive. Note that this does not require the send
+    # capability.
+    method respond_receive [cap:receive] {
+        param tag uint64 [inout]
+        param data buffer [inout zero_ok]
+        param data_in size
+        param handles ref object [inout list zero_ok]
+        param handles_in size
+        param reply_tag uint16 [inout]
+        param badge uint64 [out optional]
+        param flags uint64
+    }
+}

definitions/objects/object.def

@@ -0,0 +1,14 @@
+# The base type of all kernel-exposed objects
+object object [virtual] {
+    uid 667f61fb2cd57bb4
+    cname kobject
+
+    capabilities [
+        clone
+    ]
+
+    # Get the internal kernel object id of an object
+    method koid {
+        param koid uint64 [out]
+    }
+}

definitions/objects/process.def

@@ -0,0 +1,31 @@
+import "objects/object.def"
+
+# Processes are a collection of handles and a virtual memory
+# space inside which threads are run.
+
+object process : object {
+    uid 0c69ee0b7502ba31
+
+    capabilities [
+        kill
+        create_thread
+    ]
+
+    # Create a new empty process
+    method create [constructor]
+
+    # Stop all threads and exit the given process
+    method kill [destructor cap:kill]
+
+    # Stop all threads and exit the current process
+    method exit [static noreturn] {
+        param result int32    # The result to retrun to the parent process
+    }
+
+    # Give the given process a handle that points to the same
+    # object as the specified handle.
+    method give_handle {
+        param target ref object [handle]          # A handle in the caller process to send
+        param received ref object [out optional]  # The handle as the recipient will see it
+    }
+}

definitions/objects/system.def

@@ -0,0 +1,40 @@
+# The system object represents a handle to kernel functionality
+# needed by drivers and other priviledged services
+object system : object {
+    uid fa72506a2cf71a30
+
+    capabilities [
+        get_log
+        bind_irq
+        map_phys
+        change_iopl
+    ]
+
+    # Get a log line from the kernel log
+    method get_log [cap:get_log] {
+        param buffer buffer [out zero_ok]   # Buffer for the log message data structure
+    }
+
+    # Ask the kernel to send this process messages whenever
+    # the given IRQ fires
+    method bind_irq [cap:bind_irq] {
+        param dest ref event     # Event object that will receive messages
+        param irq uint           # IRQ number to bind
+        param signal uint        # Signal number on the event to bind to
+    }
+
+    # Create a VMA and map an area of physical memory into it,
+    # also mapping that VMA into the current process
+    method map_phys [cap:map_phys] {
+        param area ref vma [out] # Receives a handle to the VMA created
+        param phys address       # The physical address of the area
+        param size size          # Size of the area, in pages
+        param flags uint32       # Flags to apply to the created VMA
+    }
+
+    # Request the kernel change the IOPL for this process. The only values
+    # that make sense are 0 and 3.
+    method request_iopl [cap:change_iopl] {
+        param iopl uint             # The IOPL to set for this process
+    }
+}

definitions/objects/thread.def

@@ -0,0 +1,23 @@
+object thread : object {
+    uid 11f23e593d5761bd
+
+    capabilities [
+        kill
+    ]
+
+    method create [constructor] {
+        param process ref process [cap:create_thread]
+        param stack_top address
+        param entrypoint address
+    }
+
+    method kill [destructor cap:kill]
+
+    method exit [static] {
+        param status int32
+    }
+
+    method sleep [static] {
+        param duration uint64
+    }
+}

definitions/objects/vma.def

@@ -0,0 +1,36 @@
+import "objects/process.def"
+
+object vma : object {
+    uid d6a12b63b3ed3937
+    cname vm_area
+
+    capabilities [
+        map
+        unmap
+        resize
+    ]
+
+    method create [constructor] {
+        param size size
+        param flags uint32
+    }
+
+    method create_map [constructor cap:map] {
+        param size size
+        param address address
+        param flags uint32
+    }
+
+    method map [cap:map] {
+        param process ref process
+        param address address
+    }
+
+    method unmap [cap:unmap] {
+        param process ref process
+    }
+
+    method resize [cap:resize] {
+        param size size [inout]
+    }
+}

definitions/syscalls.def

@@ -0,0 +1,50 @@
+import "objects/object.def"
+
+import "objects/channel.def"
+import "objects/event.def"
+import "objects/mailbox.def"
+import "objects/process.def"
+import "objects/system.def"
+import "objects/thread.def"
+import "objects/vma.def"
+
+interface syscalls [syscall] {
+    uid 01d9b6a948961097
+
+    expose ref object
+    expose ref system
+    expose ref event
+    expose ref process
+    expose ref thread
+    expose ref mailbox
+    expose ref channel
+    expose ref vma
+
+    # Simple no-op syscall for testing
+    function noop
+
+    # Write a message to the kernel log
+    function log {
+        param message string
+    }
+
+    # Get a list of handles owned by this process. If the
+    # supplied list is not big enough, will set the size
+    # needed in `size` and return j6_err_insufficient
+    function handle_list {
+        param handles ref object [list inout zero_ok] # A list of handles to be filled
+    }
+
+    # Create a clone of an existing handle, possibly with
+    # some capabilities masked out.
+    function handle_clone {
+        param orig ref object [handle cap:clone] # The handle to clone
+        param clone ref object [out]             # The new handle
+        param mask uint32                        # The capability bitmask
+    }
+
+    # Testing mode only: Have the kernel finish and exit QEMU with the given exit code
+    function test_finish [test] {
+        param exit_code uint32
+    }
+}

definitions/sysconf.yaml

@@ -0,0 +1,34 @@
+---
+address: 0x6a360000
+vars:
+  - name: version_major
+    section: kernel
+    type: uint8_t
+
+  - name: version_minor
+    section: kernel
+    type: uint8_t
+
+  - name: version_patch
+    section: kernel
+    type: uint16_t
+
+  - name: version_gitsha
+    section: kernel
+    type: uint32_t
+
+  - name: page_size
+    section: sys
+    type: size_t
+
+  - name: large_page_size
+    section: sys
+    type: size_t
+
+  - name: huge_page_size
+    section: sys
+    type: size_t
+
+  - name: num_cpus
+    section: sys
+    type: uint32_t

modules.yaml

@@ -1,334 +0,0 @@
-name: jsix
-templates: scripts/templates
-modules:
-    kernel:
-        kind: exe
-        output: jsix.elf
-        target: host
-        deps:
-            - cpu
-            - kutil
-        includes:
-            - src/kernel
-        source:
-            - src/kernel/apic.cpp
-            - src/kernel/ap_startup.s
-            - src/kernel/assert.cpp
-            - src/kernel/boot.s
-            - src/kernel/clock.cpp
-            - src/kernel/console.cpp
-            - src/kernel/cpprt.cpp
-            - src/kernel/cpu.cpp
-            - src/kernel/debug.cpp
-            - src/kernel/debug.s
-            - src/kernel/device_manager.cpp
-            - src/kernel/frame_allocator.cpp
-            - src/kernel/fs/gpt.cpp
-            - src/kernel/gdt.cpp
-            - src/kernel/gdtidt.s
-            - src/kernel/hpet.cpp
-            - src/kernel/idt.cpp
-            - src/kernel/interrupts.cpp
-            - src/kernel/interrupts.s
-            - src/kernel/io.cpp
-            - src/kernel/log.cpp
-            - src/kernel/main.cpp
-            - src/kernel/memory_bootstrap.cpp
-            - src/kernel/msr.cpp
-            - src/kernel/objects/channel.cpp
-            - src/kernel/objects/endpoint.cpp
-            - src/kernel/objects/kobject.cpp
-            - src/kernel/objects/thread.cpp
-            - src/kernel/objects/process.cpp
-            - src/kernel/objects/system.cpp
-            - src/kernel/objects/vm_area.cpp
-            - src/kernel/page_table.cpp
-            - src/kernel/page_tree.cpp
-            - src/kernel/pci.cpp
-            - src/kernel/scheduler.cpp
-            - src/kernel/serial.cpp
-            - src/kernel/symbol_table.cpp
-            - src/kernel/syscall.cpp
-            - src/kernel/syscall.s
-            - src/kernel/syscalls/channel.cpp
-            - src/kernel/syscalls/endpoint.cpp
-            - src/kernel/syscalls/object.cpp
-            - src/kernel/syscalls/process.cpp
-            - src/kernel/syscalls/system.cpp
-            - src/kernel/syscalls/thread.cpp
-            - src/kernel/syscalls/vm_area.cpp
-            - src/kernel/task.s
-            - src/kernel/tss.cpp
-            - src/kernel/vm_space.cpp
-
-    boot:
-        kind: exe
-        target: boot
-        output: boot.efi
-        deps:
-            - cpu
-        source:
-            - src/boot/main.cpp
-            - src/boot/console.cpp
-            - src/boot/error.cpp
-            - src/boot/fs.cpp
-            - src/boot/hardware.cpp
-            - src/boot/loader.cpp
-            - src/boot/memory.cpp
-            - src/boot/paging.cpp
-            - src/boot/status.cpp
-            - src/boot/support.cpp
-
-    nulldrv:
-        kind: exe
-        target: user
-        output: nulldrv.elf
-        deps:
-          - libc
-        source:
-            - src/drivers/nulldrv/io.cpp
-            - src/drivers/nulldrv/main.cpp
-            - src/drivers/nulldrv/serial.cpp
-
-    fb:
-        kind: exe
-        target: user
-        output: fb.elf
-        deps:
-          - libc
-        source:
-            - src/drivers/fb/font.cpp
-            - src/drivers/fb/main.cpp
-            - src/drivers/fb/screen.cpp
-            - src/drivers/fb/scrollback.cpp
-
-    kutil:
-        kind: lib
-        output: libkutil.a
-        includes:
-            - src/libraries/kutil/include
-        source:
-            - src/libraries/kutil/assert.cpp
-            - src/libraries/kutil/bip_buffer.cpp
-            - src/libraries/kutil/heap_allocator.cpp
-            - src/libraries/kutil/logger.cpp
-            - src/libraries/kutil/memory.cpp
-            - src/libraries/kutil/printf.c
-            - src/libraries/kutil/spinlock.cpp
-
-    cpu:
-        kind: lib
-        output: libcpu.a
-        includes:
-            - src/libraries/cpu/include
-        source:
-            - src/libraries/cpu/cpu_id.cpp
-
-    j6:
-      kind: lib
-      output: libj6.a
-      includes:
-        - src/libraries/j6/include
-      target: user
-      source:
-        - src/libraries/j6/syscalls.s
-
-    libc:
-        kind: lib
-        output: libc.a
-        includes:
-          - src/libraries/libc/include
-        deps:
-            - j6
-        target: user
-        defines:
-            - DISABLE_SSE
-            - LACKS_UNISTD_H
-            - LACKS_FCNTL_H
-            - LACKS_SYS_PARAM_H
-            - LACKS_SYS_MMAN_H
-            - LACKS_SCHED_H
-            - LACKS_STRINGS_H
-            - HAVE_MMAP=0
-              #- LACKS_STRING_H
-              #- LACKS_ERRNO_H
-              #- LACKS_STDLIB_H
-              #- LACKS_TIME_H
-        source:
-            - src/libraries/libc/arch/x86_64/_Exit.s
-            - src/libraries/libc/arch/x86_64/crt0.s
-            - src/libraries/libc/arch/x86_64/init_libc.c
-            - src/libraries/libc/ctype/isalnum.c
-            - src/libraries/libc/ctype/isalpha.c
-            - src/libraries/libc/ctype/isblank.c
-            - src/libraries/libc/ctype/iscntrl.c
-            - src/libraries/libc/ctype/isdigit.c
-            - src/libraries/libc/ctype/isgraph.c
-            - src/libraries/libc/ctype/islower.c
-            - src/libraries/libc/ctype/isprint.c
-            - src/libraries/libc/ctype/ispunct.c
-            - src/libraries/libc/ctype/isspace.c
-            - src/libraries/libc/ctype/isupper.c
-            - src/libraries/libc/ctype/isxdigit.c
-            - src/libraries/libc/ctype/tolower.c
-            - src/libraries/libc/ctype/toupper.c
-            - src/libraries/libc/inttypes/imaxabs.c
-            - src/libraries/libc/inttypes/imaxdiv.c
-            - src/libraries/libc/inttypes/strtoimax.c
-            - src/libraries/libc/inttypes/strtoumax.c
-            - src/libraries/libc/locale/localeconv.c
-            - src/libraries/libc/locale/setlocale.c
-            - src/libraries/libc/j6libc/assert.c
-            - src/libraries/libc/j6libc/errno.c
-            - src/libraries/libc/j6libc/allocpages.c
-            - src/libraries/libc/j6libc/atomax.c
-            - src/libraries/libc/j6libc/closeall.c
-            - src/libraries/libc/j6libc/close.c
-            - src/libraries/libc/j6libc/digits.c
-            - src/libraries/libc/j6libc/filemode.c
-            - src/libraries/libc/j6libc/fillbuffer.c
-            - src/libraries/libc/j6libc/flushbuffer.c
-            - src/libraries/libc/j6libc/is_leap.c
-            - src/libraries/libc/j6libc/load_lc_collate.c
-            - src/libraries/libc/j6libc/load_lc_ctype.c
-            - src/libraries/libc/j6libc/load_lc_messages.c
-            - src/libraries/libc/j6libc/load_lc_monetary.c
-            - src/libraries/libc/j6libc/load_lc_numeric.c
-            - src/libraries/libc/j6libc/load_lc_time.c
-            - src/libraries/libc/j6libc/load_lines.c
-            - src/libraries/libc/j6libc/open.c
-            - src/libraries/libc/j6libc/prepread.c
-            - src/libraries/libc/j6libc/prepwrite.c
-            - src/libraries/libc/j6libc/print.c
-            - src/libraries/libc/j6libc/rename.c
-            - src/libraries/libc/j6libc/scan.c
-            - src/libraries/libc/j6libc/seed.c
-            - src/libraries/libc/j6libc/seek.c
-            - src/libraries/libc/j6libc/stdinit.c
-            - src/libraries/libc/j6libc/strtox_main.c
-            - src/libraries/libc/j6libc/strtox_prelim.c
-            - src/libraries/libc/j6libc/sbrk.c
-            - src/libraries/libc/signal/raise.c
-            - src/libraries/libc/signal/signal.c
-            - src/libraries/libc/stdio/clearerr.c
-            - src/libraries/libc/stdio/fclose.c
-            - src/libraries/libc/stdio/feof.c
-            - src/libraries/libc/stdio/ferror.c
-            - src/libraries/libc/stdio/fflush.c
-            - src/libraries/libc/stdio/fgetc.c
-            - src/libraries/libc/stdio/fgetpos.c
-            - src/libraries/libc/stdio/fgets.c
-            - src/libraries/libc/stdio/fopen.c
-            - src/libraries/libc/stdio/fprintf.c
-            - src/libraries/libc/stdio/fputc.c
-            - src/libraries/libc/stdio/fputs.c
-            - src/libraries/libc/stdio/fread.c
-            - src/libraries/libc/stdio/freopen.c
-            - src/libraries/libc/stdio/fscanf.c
-            - src/libraries/libc/stdio/fseek.c
-            - src/libraries/libc/stdio/fsetpos.c
-            - src/libraries/libc/stdio/ftell.c
-            - src/libraries/libc/stdio/fwrite.c
-            - src/libraries/libc/stdio/getc.c
-            - src/libraries/libc/stdio/getchar.c
-            - src/libraries/libc/stdio/perror.c
-            - src/libraries/libc/stdio/printf.c
-            - src/libraries/libc/stdio/putc.c
-            - src/libraries/libc/stdio/putchar.c
-            - src/libraries/libc/stdio/puts.c
-            - src/libraries/libc/stdio/remove.c
-            - src/libraries/libc/stdio/rename.c
-            - src/libraries/libc/stdio/rewind.c
-            - src/libraries/libc/stdio/scanf.c
-            - src/libraries/libc/stdio/setbuf.c
-            - src/libraries/libc/stdio/setvbuf.c
-            - src/libraries/libc/stdio/snprintf.c
-            - src/libraries/libc/stdio/sprintf.c
-            - src/libraries/libc/stdio/sscanf.c
-            - src/libraries/libc/stdio/tmpfile.c
-            - src/libraries/libc/stdio/tmpnam.c
-            - src/libraries/libc/stdio/ungetc.c
-            - src/libraries/libc/stdio/vfprintf.c
-            - src/libraries/libc/stdio/vfscanf.c
-            - src/libraries/libc/stdio/vprintf.c
-            - src/libraries/libc/stdio/vscanf.c
-            - src/libraries/libc/stdio/vsnprintf.c
-            - src/libraries/libc/stdio/vsprintf.c
-            - src/libraries/libc/stdio/vsscanf.c
-            - src/libraries/libc/stdlib/abort.c
-            - src/libraries/libc/stdlib/abs.c
-            - src/libraries/libc/stdlib/atexit.c
-            - src/libraries/libc/stdlib/atoi.c
-            - src/libraries/libc/stdlib/atol.c
-            - src/libraries/libc/stdlib/atoll.c
-            - src/libraries/libc/stdlib/bsearch.c
-            - src/libraries/libc/stdlib/div.c
-            - src/libraries/libc/stdlib/exit.c
-            - src/libraries/libc/stdlib/_Exit.c
-            - src/libraries/libc/stdlib/getenv.c
-            - src/libraries/libc/stdlib/labs.c
-            - src/libraries/libc/stdlib/ldiv.c
-            - src/libraries/libc/stdlib/llabs.c
-            - src/libraries/libc/stdlib/lldiv.c
-            - src/libraries/libc/stdlib/malloc.c
-            - src/libraries/libc/stdlib/qsort.c
-            - src/libraries/libc/stdlib/rand.c
-            - src/libraries/libc/stdlib/srand.c
-            - src/libraries/libc/stdlib/strtol.c
-            - src/libraries/libc/stdlib/strtoll.c
-            - src/libraries/libc/stdlib/strtoul.c
-            - src/libraries/libc/stdlib/strtoull.c
-            - src/libraries/libc/stdlib/system.c
-            - src/libraries/libc/string/memchr.c
-            - src/libraries/libc/string/memcmp.c
-            - src/libraries/libc/string/memcpy.c
-            - src/libraries/libc/string/memmove.c
-            - src/libraries/libc/string/memset.c
-            - src/libraries/libc/string/strcat.c
-            - src/libraries/libc/string/strchr.c
-            - src/libraries/libc/string/strcmp.c
-            - src/libraries/libc/string/strcoll.c
-            - src/libraries/libc/string/strcpy.c
-            - src/libraries/libc/string/strcspn.c
-            - src/libraries/libc/string/strerror.c
-            - src/libraries/libc/string/strlen.c
-            - src/libraries/libc/string/strncat.c
-            - src/libraries/libc/string/strncmp.c
-            - src/libraries/libc/string/strncpy.c
-            - src/libraries/libc/string/strpbrk.c
-            - src/libraries/libc/string/strrchr.c
-            - src/libraries/libc/string/strspn.c
-            - src/libraries/libc/string/strstr.c
-            - src/libraries/libc/string/strtok.c
-            - src/libraries/libc/string/strxfrm.c
-            - src/libraries/libc/time/asctime.c
-            - src/libraries/libc/time/clock.c
-            - src/libraries/libc/time/ctime.c
-            - src/libraries/libc/time/difftime.c
-            - src/libraries/libc/time/gmtime.c
-            - src/libraries/libc/time/localtime.c
-            - src/libraries/libc/time/mktime.c
-            - src/libraries/libc/time/strftime.c
-            - src/libraries/libc/time/time.c
-            - src/libraries/libc/time/timespec_get.c
-
-    tests:
-        kind: exe
-        target: native
-        output: tests
-        deps:
-            - kutil
-        source:
-            - src/tests/constexpr_hash.cpp
-            - src/tests/linked_list.cpp
-            - src/tests/logger.cpp
-            - src/tests/heap_allocator.cpp
-            - src/tests/main.cpp
-            - src/tests/map.cpp
-            - src/tests/vector.cpp
-
-overlays:
-    - url: https://f000.backblazeb2.com/file/jsix-os/sysroot-llvm8-20190706.tar.bz2
-      path: sysroot
-    - url: https://f000.backblazeb2.com/file/jsix-os/sysroot-j6libc-8cb8ce7.tar.bz2
-      path: sysroot

other_software.md

@@ -33,42 +33,18 @@ Boost Software license:
 > ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
 > DEALINGS IN THE SOFTWARE.
 
-## cpptoml
-
-jsix uses the [cpptoml][] library for parsing TOML configuration files. cpptoml
-is released under the terms of the MIT license:
-
-[cpptoml]: https://github.com/skystrife/cpptoml
-
-> Copyright (c) 2014 Chase Geigle
->
-> Permission is hereby granted, free of charge, to any person obtaining a copy of
-> this software and associated documentation files (the "Software"), to deal in
-> the Software without restriction, including without limitation the rights to
-> use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
-> the Software, and to permit persons to whom the Software is furnished to do so,
-> subject to the following conditions:
-> 
-> The above copyright notice and this permission notice shall be included in all
-> copies or substantial portions of the Software.
-> 
-> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
-> FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
-> COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
-> IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-> CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
 ## printf
 
-jsix uses Marco Paland's tiny [printf][] library for its `*printf()` functions,
+The implementation of jsix's `*printf()` family of functions is derived from
-which is also released under the terms of the MIT license:
+Marco Paland and Eyal Rozenberg's tiny [printf][] library, which is also
+released under the terms of the MIT license:
 
-[printf]: https://github.com/mpaland/printf
+[printf]: https://github.com/eyalroz/printf
 
 > The MIT License (MIT)
 >
 > Copyright (c) 2014 Marco Paland
+> Copyright (c) 2021 Eyal Rozenberg
 >
 > Permission is hereby granted, free of charge, to any person obtaining a copy
 > of this software and associated documentation files (the "Software"), to deal

qemu.sh

@@ -3,37 +3,55 @@
 build="$(dirname $0)/build"
 assets="$(dirname $0)/assets"
 debug=""
+isaexit='-device isa-debug-exit,iobase=0xf4,iosize=0x04'
 debugtarget="${build}/jsix.elf"
-flash_name="ovmf_vars"
 gfx="-nographic"
 vga="-vga none"
+log=""
 kvm=""
 cpu="Broadwell,+pdpe1gb"
+smp=4
 
-for arg in $@; do
+while true; do
-	case "${arg}" in
+	case "$1" in
-		--debugboot)
+		-b | --debugboot)
 			debug="-s -S"
 			debugtarget="${build}/boot/boot.efi"
-			flash_name="ovmf_vars_d"
+            shift
 			;;
-		--debug)
+		-d | --debug)
 			debug="-s -S"
-			flash_name="ovmf_vars_d"
+            isaexit=""
+            shift
 			;;
-		--gfx)
+		-g | --gfx)
 			gfx=""
 			vga=""
+            shift
 			;;
-		--vga)
+		-v | --vga)
 			vga=""
+            shift
 			;;
-		--kvm)
+		-k | --kvm)
 			kvm="-enable-kvm"
 			cpu="host"
+            shift
 			;;
+        -c | --cpus)
+            smp=$2
+            shift 2
+            ;;
+        -l | --log)
+            log="-d mmu,int,guest_errors -D jsix.log"
+            shift
+            ;;
 		*)
-			build="${arg}"
+            if [ -d "$1" ]; then
+                build="$1"
+                shift
+            fi
+            break
 			;;
 	esac
 done
@@ -56,25 +74,25 @@ if [[ -n $TMUX ]]; then
 	fi
 elif [[ $DESKTOP_SESSION = "i3" ]]; then
 	if [[ -n $debug ]]; then
-		i3-msg exec i3-sensible-terminal -- -e "gdb ${PWD}/${build}/jsix.elf" &
+		i3-msg exec i3-sensible-terminal -- -e "gdb ${debugtarget}" &
 	else
 		i3-msg exec i3-sensible-terminal -- -e 'telnet localhost 45454' &
 	fi
 fi
 
-exec qemu-system-x86_64 \
+qemu-system-x86_64 \
 	-drive "if=pflash,format=raw,readonly,file=${assets}/ovmf/x64/ovmf_code.fd" \
-	-drive "if=pflash,format=raw,file=${build}/${flash_name}.fd" \
+	-drive "if=pflash,format=raw,file=${build}/ovmf_vars.fd" \
 	-drive "format=raw,file=${build}/jsix.img" \
-	-device "isa-debug-exit,iobase=0xf4,iosize=0x04" \
 	-monitor telnet:localhost:45454,server,nowait \
 	-serial stdio \
 	-serial telnet:localhost:45455,server,nowait \
-	-smp 4 \
+	-smp "${smp}" \
-	-m 512 \
+	-m 4096 \
-	-d mmu,int,guest_errors \
-	-D jsix.log \
 	-cpu "${cpu}" \
 	-M q35 \
 	-no-reboot \
-	$gfx $vga $kvm $debug
+	$isaexit $log $gfx $vga $kvm $debug
+
+((result = ($? >> 1) - 1))
+exit $result

requirements.txt

@@ -0,0 +1,5 @@
+cogapp >= 3
+ninja >= 1.10.2
+peru >= 1.2.1
+pyyaml >= 5.4
+lark == 0.12.0

scripts/bonnibel/__init__.py

@@ -0,0 +1,25 @@
+from os.path import join
+
+class BonnibelError(Exception):
+    def __init__(self, message):
+        self.message = message
+
+def load_config(filename):
+    from yaml import safe_load
+    with open(filename, 'r') as infile:
+        return safe_load(infile.read())
+
+def mod_rel(path):
+    return join("${module_dir}", path)
+
+def target_rel(path, module=None):
+    if module:
+        return join("${target_dir}", module + ".dir", path)
+    else:
+        return join("${target_dir}", path)
+
+def include_rel(path, module=None):
+    if module:
+        return join("${build_root}", "include", module, path)
+    else:
+        return join("${build_root}", "include", path)

scripts/bonnibel/manifest.py

@@ -0,0 +1,103 @@
+from . import BonnibelError
+
+class Manifest:
+    from collections import namedtuple
+    Entry = namedtuple("Entry", ("module", "target", "output", "location", "description", "flags"))
+
+    flags = {
+        "graphical": 0x01,
+        "panic":     0x02,
+        "symbols":   0x04,
+    }
+
+    boot_flags = {
+        "debug": 0x01,
+        "test":  0x02,
+    }
+
+    def __init__(self, path, modules):
+        from . import load_config
+
+        config = load_config(path)
+
+        self.kernel = self.__build_entry(modules,
+                config.get("kernel", dict()),
+                name="kernel", target="kernel")
+
+        self.init = self.__build_entry(modules,
+                config.get("init", None))
+
+        self.programs = [self.__build_entry(modules, i)
+                for i in config.get("programs", tuple())]
+
+        self.flags = config.get("flags", tuple())
+
+        self.data = []
+        for d in config.get("data", tuple()):
+            self.add_data(**d)
+
+    def __build_entry(self, modules, config, target="user", name=None):
+        flags = tuple()
+
+        if isinstance(config, str):
+            name = config
+        elif isinstance(config, dict):
+            name = config.get("name", name)
+            target = config.get("target", target)
+            flags = config.get("flags", tuple())
+            if isinstance(flags, str):
+                flags = flags.split()
+
+        mod = modules.get(name)
+        if not mod:
+            raise BonnibelError(f"Manifest specifies unknown module '{name}'")
+
+        for f in flags:
+            if not f in Manifest.flags:
+                raise BonnibelError(f"Manifest specifies unknown flag '{f}'")
+
+        return Manifest.Entry(name, target, mod.output, mod.location, mod.description, flags)
+
+    def add_data(self, output, location, desc, flags=tuple()):
+        e = Manifest.Entry(None, None, output, location, desc, flags)
+        self.data.append(e)
+        return e
+
+    def write_boot_config(self, path):
+        from os.path import join
+        import struct
+
+        with open(path, 'wb') as outfile:
+            magic = "jsixboot".encode("utf-8") # magic string
+            version = 0
+            reserved = 0
+
+            bootflags = sum([Manifest.boot_flags.get(s, 0) for s in self.flags])
+
+            outfile.write(struct.pack("<8sBBHHH",
+                magic, version, reserved,
+                len(self.programs), len(self.data),
+                bootflags))
+
+            def write_str(s):
+                outfile.write(struct.pack("<H", (len(s)+1)*2))
+                outfile.write(s.encode("utf-16le"))
+                outfile.write(b"\0\0")
+
+            def write_ent(ent):
+                flags = 0
+                for f in ent.flags:
+                    flags |= Manifest.flags[f]
+
+                outfile.write(struct.pack("<H", flags))
+                write_str(join(ent.location, ent.output).replace('/','\\'))
+                write_str(ent.description)
+
+            write_ent(self.kernel)
+            write_ent(self.init)
+
+            for p in self.programs:
+                write_ent(p)
+
+            for d in self.data:
+                write_ent(d)

scripts/bonnibel/module.py

@@ -0,0 +1,276 @@
+from . import include_rel, mod_rel, target_rel
+
+def resolve(path):
+    if path.startswith('/') or path.startswith('$'):
+        return path
+    from pathlib import Path
+    return str(Path(path).resolve())
+
+class BuildOptions:
+    def __init__(self, includes=tuple(), libs=tuple(), order_only=tuple()):
+        self.includes = list(includes)
+        self.libs = list(libs)
+        self.order_only = list(order_only)
+
+
+class Module:
+    __fields = {
+        # name: (type, default)
+        "kind": (str, "exe"),
+        "output": (str, None),
+        "targets": (set, ()),
+        "deps":  (set, ()),
+        "public_headers":  (set, ()),
+        "includes": (tuple, ()),
+        "sources": (tuple, ()),
+        "variables": (dict, ()),
+        "default": (bool, False),
+        "location": (str, "jsix"),
+        "description": (str, None),
+        "no_libc": (bool, False),
+    }
+
+    def __init__(self, name, modfile, root, **kwargs):
+        from .source import make_source
+
+        # Required fields
+        self.root = root
+        self.name = name
+        self.modfile = modfile
+
+        for name, data in self.__fields.items():
+            ctor, default = data
+            value = kwargs.get(name, default)
+            if value is not None:
+                value = ctor(value)
+
+            setattr(self, name, value)
+
+        for name in kwargs:
+            if not name in self.__fields:
+                raise AttributeError(f"No attribute named {name} on Module")
+
+        # Turn strings into real Source objects
+        self.sources = [make_source(root, f) for f in self.sources]
+        self.public_headers = [make_source(root, f) for f in self.public_headers]
+
+        # Filled by Module.update
+        self.depmods = set()
+
+    def __str__(self):
+        return "Module {} {}\n\t{}".format(self.kind, self.name, "\n\t".join(map(str, self.sources)))
+
+    @property
+    def output(self):
+        if self.__output is not None:
+            return self.__output
+
+        if self.kind == "lib":
+            return f"lib{self.name}.a"
+        else:
+            return f"{self.name}.elf"
+
+    @output.setter
+    def output(self, value):
+        self.__output = value
+
+    @classmethod
+    def update(cls, mods):
+        from . import BonnibelError
+
+        def resolve(source, modlist):
+            resolved = set()
+            for dep in modlist:
+                if not dep in mods:
+                    raise BonnibelError(f"module '{source.name}' references unknown module '{dep}'")
+                mod = mods[dep]
+                resolved.add(mod)
+            return resolved
+
+        for mod in mods.values():
+            mod.depmods = resolve(mod, mod.deps)
+
+        target_mods = [mod for mod in mods.values() if mod.targets]
+        for mod in target_mods:
+            closed = set()
+            children = set(mod.depmods)
+            while children:
+                child = children.pop()
+                closed.add(child)
+                child.targets |= mod.targets
+                children |= {m for m in child.depmods if not m in closed}
+
+    def generate(self, output):
+        from pathlib import Path
+        from collections import defaultdict
+        from ninja.ninja_syntax import Writer
+
+        def walk_deps(deps):
+            open_set = set(deps)
+            closed_set = set()
+            while open_set:
+                dep = open_set.pop()
+                closed_set.add(dep)
+                open_set |= {m for m in dep.depmods if not m in closed_set}
+            return closed_set
+
+        all_deps = walk_deps(self.depmods)
+
+        def gather_phony(build, deps, child_rel, add_headers=False):
+            phony = ".headers.phony"
+            child_phony = [child_rel(phony, module=c.name)
+                    for c in all_deps]
+
+            header_targets = []
+            if add_headers:
+                if not self.no_libc:
+                    header_targets.append(f"${{build_root}}/include/libc/{phony}")
+                if self.public_headers:
+                    header_targets.append(f"${{build_root}}/include/{self.name}/{phony}")
+
+            build.build(
+                rule = "touch",
+                outputs = [mod_rel(phony)],
+                implicit = child_phony + header_targets,
+                order_only = list(map(mod_rel, deps)),
+            )
+
+        filename = str(output / f"headers.{self.name}.ninja")
+        with open(filename, "w") as buildfile:
+            build = Writer(buildfile)
+
+            build.comment("This file is automatically generated by bonnibel")
+            build.newline()
+
+            build.variable("module_dir", f"${{build_root}}/include/{self.name}")
+
+            header_deps = []
+
+            inputs = []
+            headers = set(self.public_headers)
+            while headers:
+                source = headers.pop()
+                headers.update(source.next)
+
+                if source.action:
+                    build.newline()
+                    build.build(rule=source.action, **source.args)
+
+                if source.gather:
+                    header_deps += list(source.outputs)
+
+                if source.input:
+                    inputs.extend(map(mod_rel, source.outputs))
+
+            build.newline()
+            gather_phony(build, header_deps, include_rel)
+
+        filename = str(output / f"module.{self.name}.ninja")
+        with open(filename, "w") as buildfile:
+            build = Writer(buildfile)
+
+            build.comment("This file is automatically generated by bonnibel")
+            build.newline()
+
+            build.variable("module_dir", target_rel(self.name + ".dir"))
+
+            modopts = BuildOptions(
+                includes = [self.root, "${module_dir}"],
+                )
+            if self.public_headers:
+                modopts.includes += [f"${{build_root}}/include/{self.name}"]
+
+            for key, value in self.variables.items():
+                build.variable(key, value)
+            build.newline()
+
+            for include in self.includes:
+                p = Path(include)
+                if p.is_absolute():
+                    if not p in modopts.includes:
+                        modopts.includes.append(str(p.resolve()))
+                elif include != ".":
+                    incpath = self.root / p
+                    destpath = mod_rel(p)
+                    for header in incpath.rglob("*.h"):
+                        dest_header = f"{destpath}/" + str(header.relative_to(incpath))
+                    modopts.includes.append(str(incpath))
+                    modopts.includes.append(destpath)
+
+            all_deps = walk_deps(self.depmods)
+            for dep in all_deps:
+                if dep.public_headers:
+                    modopts.includes += [f"${{build_root}}/include/{dep.name}"]
+
+                if dep.kind == "lib":
+                    modopts.libs.append(target_rel(dep.output))
+                else:
+                    modopts.order_only.append(target_rel(dep.output))
+
+            if modopts.includes:
+                build.variable("ccflags", ["${ccflags}"] + [f"-I{i}" for i in modopts.includes])
+                build.variable("asflags", ["${asflags}"] + [f"-I{i}" for i in modopts.includes])
+
+            if modopts.libs:
+                build.variable("libs", ["${libs}"] + modopts.libs)
+
+            header_deps = []
+
+            inputs = []
+            sources = set(self.sources)
+            while sources:
+                source = sources.pop()
+                sources.update(source.next)
+
+                if source.action:
+                    build.newline()
+                    build.build(rule=source.action, **source.args)
+
+                if source.gather:
+                    header_deps += list(source.outputs)
+
+                if source.input:
+                    inputs.extend(map(mod_rel, source.outputs))
+
+            build.newline()
+
+            gather_phony(build, header_deps, target_rel, add_headers=True)
+
+            output = target_rel(self.output)
+            dump = output + ".dump"
+            build.newline()
+            build.build(
+                rule = self.kind,
+                outputs = output,
+                inputs = inputs,
+                implicit = modopts.libs,
+                order_only = modopts.order_only,
+            )
+
+            build.newline()
+            build.build(
+                rule = "dump",
+                outputs = dump,
+                inputs = output,
+                variables = {"name": self.name},
+            )
+
+            if self.default:
+                build.newline()
+                build.default(output)
+                build.default(dump)
+
+    def add_input(self, path, **kwargs):
+        from .source import Source
+        s = Source(self.root, path, **kwargs)
+        self.sources.append(s)
+        return s.outputs
+
+    def add_depends(self, paths, deps):
+        for source in self.sources:
+            if source.path in paths:
+                source.add_deps(deps)
+
+        for source in self.public_headers:
+            if source.path in paths:
+                source.add_deps(deps)

scripts/bonnibel/project.py

@@ -0,0 +1,234 @@
+from . import BonnibelError
+
+class Project:
+    def __init__(self, root):
+        from .version import git_version
+
+        self.root = root
+        self.version = git_version(root)
+
+    def __str__(self):
+        return f"{self.name} {self.version.major}.{self.version.minor}.{self.version.patch}-{self.version.sha}"
+
+    def generate(self, root, output, modules, config, manifest_file):
+        import sys
+        import bonnibel
+        from os.path import join
+        from ninja.ninja_syntax import Writer
+        from .target import Target
+
+        targets = set()
+        for mod in modules.values():
+            targets.update({Target.load(root, t, config) for t in mod.targets})
+
+        with open(output / "build.ninja", "w") as buildfile:
+            build = Writer(buildfile)
+
+            build.comment("This file is automatically generated by bonnibel")
+            build.variable("ninja_required_version", "1.3")
+            build.variable("build_root", output)
+            build.variable("source_root", root)
+            build.newline()
+
+            build.include(root / "configs" / "rules.ninja")
+            build.newline()
+
+            build.variable("version_major", self.version.major)
+            build.variable("version_minor", self.version.minor)
+            build.variable("version_patch", self.version.patch)
+            build.variable("version_sha", self.version.sha)
+            build.newline()
+
+            build.variable("cogflags", [
+                "-I", "${source_root}/scripts",
+                "-D", "definitions_path=${source_root}/definitions",
+                ])
+            build.newline()
+
+            for target in targets:
+                build.subninja(output / target.name / "target.ninja")
+            build.newline()
+
+            for mod in modules.values():
+                build.subninja(output / f"headers.{mod.name}.ninja")
+            build.newline()
+
+            build.build(
+                rule = "touch",
+                outputs = "${build_root}/.all_headers",
+                implicit = [f"${{build_root}}/include/{m.name}/.headers.phony"
+                    for m in modules.values() if m.public_headers],
+                )
+            build.build(
+                rule = "phony",
+                outputs = ["all-headers"],
+                inputs = ["${build_root}/.all_headers"])
+
+            debugroot = output / ".debug"
+            debugroot.mkdir(exist_ok=True)
+
+            fatroot = output / "fatroot"
+            fatroot.mkdir(exist_ok=True)
+
+            fatroot_content = []
+
+            def add_fatroot(source, entry):
+                output = join(entry.location, entry.output)
+                fatroot_output = f"${{build_root}}/fatroot/{output}"
+
+                build.build(
+                    rule = "cp",
+                    outputs = [fatroot_output],
+                    inputs = [source],
+                    variables = {
+                        "name": f"Installing {output}",
+                    })
+
+                fatroot_content.append(fatroot_output)
+                build.newline()
+
+            def add_fatroot_exe(entry):
+                input_path = f"${{build_root}}/{entry.target}/{entry.output}"
+                intermediary = f"${{build_root}}/{entry.output}"
+
+                build.build(
+                    rule = "strip",
+                    outputs = [intermediary],
+                    inputs = [input_path],
+                    implicit = [f"{input_path}.dump"],
+                    variables = {
+                        "name": f"Stripping {entry.module}",
+                        "debug": f"${{build_root}}/.debug/{entry.output}.debug",
+                    })
+
+                add_fatroot(intermediary, entry)
+
+            from .manifest import Manifest
+            manifest = Manifest(manifest_file, modules)
+
+            add_fatroot_exe(manifest.kernel)
+            add_fatroot_exe(manifest.init)
+            for program in manifest.programs:
+                add_fatroot_exe(program)
+
+            syms = manifest.add_data("symbol_table.dat",
+                    manifest.kernel.location, "Symbol table", ("symbols",))
+
+            sym_out = f"${{build_root}}/symbol_table.dat"
+            build.build(
+                rule = "makest",
+                outputs = [sym_out],
+                inputs = [f"${{build_root}}/{modules['kernel'].output}"],
+                )
+            add_fatroot(sym_out, syms)
+
+            bootloader = "${build_root}/fatroot/efi/boot/bootx64.efi"
+            build.build(
+                rule = "cp",
+                outputs = [bootloader],
+                inputs = ["${build_root}/boot/boot.efi"],
+                variables = {
+                    "name": "Installing bootloader",
+                })
+            build.newline()
+
+            boot_config = join(fatroot, "jsix_boot.dat")
+            manifest.write_boot_config(boot_config)
+
+            build.build(
+                rule = "makefat",
+                outputs = ["${build_root}/jsix.img"],
+                inputs = ["${source_root}/assets/diskbase.img"],
+                implicit = fatroot_content + [bootloader],
+                variables = {"name": "jsix.img"},
+                )
+            build.newline()
+
+            default_assets = {
+                "UEFI Variables": ("ovmf/x64/ovmf_vars.fd", "ovmf_vars.fd"),
+                "GDB Debug Helpers": ("debugging/jsix.elf-gdb.py", "jsix.elf-gdb.py"),
+            }
+
+            for name, assets in default_assets.items():
+                p = root / "assets" / assets[0]
+                out = f"${{build_root}}/{assets[1]}"
+                build.build(
+                    rule = "cp",
+                    outputs = [out],
+                    inputs = [str(p)],
+                    variables = {"name": name},
+                    )
+                build.default([out])
+                build.newline()
+
+            compdb = "${source_root}/compile_commands.json"
+
+            build.rule("regen",
+                command = " ".join([str(root / 'configure')] + sys.argv[1:]),
+                description = "Regenerate build files",
+                generator = True,
+                )
+            build.newline()
+
+            regen_implicits = \
+                [f"{self.root}/configure", str(manifest_file)] + \
+                [str(mod.modfile) for mod in modules.values()]
+
+            for target in targets:
+                regen_implicits += target.depfiles
+
+            build.build(
+                rule = "compdb",
+                outputs = [compdb],
+                implicit = regen_implicits,
+                )
+            build.default([compdb])
+            build.newline()
+
+            build.build(
+                rule = "regen",
+                outputs = ['build.ninja'],
+                implicit = regen_implicits,
+                implicit_outputs = 
+                    [f"module.{mod.name}.ninja" for mod in modules.values()] +
+                    [f"{target.name}/target.ninja" for target in targets] +
+                    [boot_config],
+                )
+
+            build.newline()
+            build.default(["${build_root}/jsix.img"])
+
+        for target in targets:
+            mods = [m.name for m in modules.values() if target.name in m.targets]
+
+            targetdir = output / target.name
+            targetdir.mkdir(exist_ok=True)
+
+            buildfilename = str(targetdir / "target.ninja")
+            with open(buildfilename, "w") as buildfile:
+                build = Writer(buildfile)
+                build.comment("This file is automatically generated by bonnibel")
+                build.newline()
+
+                build.variable("target", target.name)
+                build.variable("target_dir", output / target.name)
+                build.newline()
+
+                for name, value in target.items():
+                    build.variable(name, value)
+
+                build.newline()
+                for kind in ('defs', 'run'):
+                    for lang in ('c', 'cpp'):
+                        deffile = str(output / target.name / f"{lang}.{kind}")
+
+                        build.build(
+                            rule = f"dump_{lang}_{kind}",
+                            outputs = [deffile],
+                            implicit = [buildfilename],
+                        )
+                        build.default(deffile)
+                        build.newline()
+
+                for mod in mods:
+                    build.subninja(f"module.{mod}.ninja")

scripts/bonnibel/source.py

@@ -0,0 +1,119 @@
+from os.path import join, splitext
+from . import mod_rel
+
+def _resolve(path):
+    if path.startswith('/') or path.startswith('$'):
+        return path
+    from pathlib import Path
+    return str(Path(path).resolve())
+
+def _dynamic_action(name):
+    def prop(self):
+        root, suffix = splitext(self.path)
+        return f"{name}{suffix}"
+    return prop
+
+
+class Source:
+    next = tuple()
+    action = None
+    args = dict()
+    gather = False
+    outputs = tuple()
+    input = False
+
+    def __init__(self, path, root = "${module_dir}", deps=tuple()): 
+        self.path = path
+        self.root = root
+        self.deps = deps
+
+    def add_deps(self, deps):
+        self.deps += tuple(deps)
+
+    @property
+    def fullpath(self):
+        return join(self.root, self.path)
+
+class ParseSource(Source):
+    action = property(_dynamic_action("parse"))
+
+    @property
+    def output(self):
+        root, _ = splitext(self.path)
+        return root
+
+    @property
+    def outputs(self):
+        return (self.output,)
+
+    @property
+    def gather(self):
+        _, suffix = splitext(self.output)
+        return suffix in (".h", ".inc")
+
+    @property
+    def next(self):
+        _, suffix = splitext(self.output)
+        nextType = {
+            ".s": CompileSource,
+            ".cpp": CompileSource,
+            }.get(suffix)
+
+        if nextType:
+            return (nextType(self.output),)
+        return tuple()
+
+    @property
+    def args(self):
+        return dict(
+            outputs = list(map(mod_rel, self.outputs)),
+            inputs = [self.fullpath],
+            implicit = list(map(_resolve, self.deps)),
+            variables = dict(name=self.path),
+            )
+
+class HeaderSource(Source):
+    action = "cp"
+    gather = True
+
+    @property
+    def outputs(self):
+        return (self.path,)
+
+    @property
+    def args(self):
+        return dict(
+            outputs = [mod_rel(self.path)],
+            inputs = [join(self.root, self.path)],
+            implicit = list(map(_resolve, self.deps)),
+            variables = dict(name=self.path),
+            )
+
+class CompileSource(Source):
+    action = property(_dynamic_action("compile"))
+    input = True
+
+    @property
+    def outputs(self):
+        return (self.path + ".o",)
+
+    @property
+    def args(self):
+        return dict(
+            outputs = list(map(mod_rel, self.outputs)),
+            inputs = [join(self.root, self.path)],
+            implicit = list(map(_resolve, self.deps)) + [mod_rel(".headers.phony")],
+            variables = dict(name=self.path),
+            )
+
+def make_source(root, path):
+    _, suffix = splitext(path)
+
+    if suffix in (".s", ".c", ".cpp"):
+        return CompileSource(path, root)
+    elif suffix in (".cog",):
+        return ParseSource(path, root)
+    elif suffix in (".h", ".inc"):
+        return HeaderSource(path, root)
+    else:
+        raise RuntimeError(f"{path} has no Source type")

scripts/bonnibel/target.py

@@ -0,0 +1,50 @@
+class Target(dict):
+    __targets = {}
+
+    @classmethod
+    def load(cls, root, name, config=None):
+        from . import load_config
+
+        if (name, config) in cls.__targets:
+            return cls.__targets[(name, config)]
+
+        configs = root / "configs"
+
+        dicts = []
+        depfiles = []
+        basename = name
+        if config:
+            basename += f"-{config}"
+
+        while basename is not None:
+            filename = str(configs / (basename + ".yaml"))
+            depfiles.append(filename)
+            desc = load_config(filename)
+            basename = desc.get("extends")
+            dicts.append(desc.get("variables", dict()))
+
+        t = Target(name, config, depfiles)
+        for d in reversed(dicts):
+            for k, v in d.items():
+                if isinstance(v, (list, tuple)):
+                    t[k] = t.get(k, list()) + list(v)
+                elif isinstance(v, dict):
+                    t[k] = t.get(k, dict())
+                    t[k].update(v)
+                else:
+                    t[k] = v
+
+        cls.__targets[(name, config)] = t
+        return t
+
+    def __init__(self, name, config, depfiles):
+        self.__name = name
+        self.__config = config
+        self.__depfiles = tuple(depfiles)
+
+    def __hash__(self):
+        return hash((self.__name, self.__config))
+
+    name = property(lambda self: self.__name)
+    config = property(lambda self: self.__config)
+    depfiles = property(lambda self: self.__depfiles)

scripts/bonnibel/version.py

@@ -0,0 +1,41 @@
+from collections import namedtuple as _namedtuple
+
+version = _namedtuple('version', [
+    'major',
+    'minor',
+    'patch',
+    'sha',
+    'dirty',
+    ])
+
+
+def _run_git(root, *args):
+    from subprocess import run
+
+    git = run(['git', '-C', root] + list(args),
+            check=True, capture_output=True)
+    return git.stdout.decode('utf-8').strip()
+
+
+def git_version(root):
+    full_version = _run_git(root, 'describe', '--dirty')
+    full_sha = _run_git(root, 'rev-parse', 'HEAD')
+
+    dirty = False
+    parts1 = full_version.split('-')
+    if parts1[-1] == "dirty":
+        dirty = True
+        parts1 = parts1[:-1]
+
+    if parts1[0][0] == 'v':
+        parts1[0] = parts1[0][1:]
+
+    parts2 = parts1[0].split('.')
+
+    return version(
+        parts2[0],
+        parts2[1],
+        parts2[2],
+        full_sha[:7],
+        dirty)
+

scripts/build_symbol_table.py

@@ -4,37 +4,37 @@
 # is as follows:
 #
 # <num_entires> : 8 bytes
-# <index>       : 16 * N bytes
+# <index>       : 24 * N bytes
 # <name data>   : variable
 #
 # Each index entry has the format
 # <symbol address> : 8 bytes
+# <symbol size>    : 8 bytes
 # <offset of name> : 8 bytes
 #
 # Name offsets are from the start of the symbol table as a whole. (ie,
 # where <num_entries> is located.)
 
+import re
+
+sym_re = re.compile(r'([0-9a-fA-F]{16}) ([0-9a-fA-F]{16}) [tTvVwW] (.*)')
+
 def parse_syms(infile):
     """Take the output of the `nm` command, and parse it into a tuple
     representing the symbols in the text segment of the binary. Returns
     a list of (address, symbol_name)."""
 
-    from cxxfilt import demangle, InvalidName
-
     syms = []
     for line in sys.stdin:
-        addr, t, mangled = line.split()
+        match = sym_re.match(line)
-        if t not in "tTvVwW": continue
+        if not match: continue
 
-        try:
+        addr = int(match.group(1), base=16)
-            name = demangle(mangled)
+        size = int(match.group(2), base=16)
-        except InvalidName:
+        name = match.group(3)
-            continue
+        syms.append([addr, size, name, 0])
 
-        addr = int(addr, base=16)
+    return syms
-        syms.append((addr, name))
-
-    return sorted(syms)
 
 
 def write_table(syms, outfile):
@@ -46,29 +46,26 @@ def write_table(syms, outfile):
     outfile.write(struct.pack("@Q", len(syms)))
     index_pos = outfile.tell()
 
-    outfile.seek(struct.calcsize("@QQ") * len(syms), 1)
+    outfile.seek(struct.calcsize("@QQQ") * len(syms), 1)
     nul = b'\0'
 
-    positions = {}
     for s in syms:
-        addr, name = s
+        s[3] = outfile.tell()
-        positions[addr] = outfile.tell()
+        outfile.write(s[2].encode('utf-8'))
-
-        data = name.encode('utf-8')
-        outfile.write(name.encode('utf-8'))
         outfile.write(nul)
 
     outfile.seek(index_pos)
     for s in syms:
         addr = s[0]
-        pos = positions[addr]
+        size = s[1]
-        outfile.write(struct.pack("@QQ", addr, pos))
+        pos = s[3]
+        outfile.write(struct.pack("@QQQ", addr, size, pos))
 
 
 if __name__ == "__main__":
     import sys
     if len(sys.argv) != 2:
-        print(f"Usage: {sys.argv[0]} <output>")
+        print(f"Usage: nm -n -S --demangle | {sys.argv[0]} <output>")
         sys.exit(1)
 
     outfile = open(sys.argv[1], "wb")

scripts/build_sysroot.sh

@@ -1,131 +0,0 @@
-#!/usr/bin/env bash
-
-TARGET="x86_64-elf"
-LLVM_BRANCH="release_80"
-
-TOOLS="clang lld" # lld libunwind libcxxabi libcxx"
-PROJECTS="compiler-rt libcxxabi libcxx libunwind"
-#RUNTIMES="compiler-rt libcxxabi libcxx libunwind"
-
-set -e
-
-README=$(realpath "$(dirname $0)/readme_for_prebuilt_sysroots.md")
-SYSROOT=$(realpath "$(dirname $0)/../sysroot")
-WORK=$(realpath "$(dirname $0)/sysroot")
-mkdir -p "${SYSROOT}"
-mkdir -p "${WORK}"
-
-export CC=clang
-export CXX=clang++
-
-if [[ ! -d "${WORK}/llvm" ]]; then
-	echo "Downloading LLVM..."
-	git clone -q \
-		--branch "${LLVM_BRANCH}" \
-		--depth 1 \
-		"https://git.llvm.org/git/llvm.git" "${WORK}/llvm"
-fi
-
-for tool in ${TOOLS}; do
-	if [[ ! -d "${WORK}/llvm/tools/${tool}" ]]; then
-		echo "Downloading ${tool}..."
-		git clone -q \
-			--branch "${LLVM_BRANCH}" \
-			--depth 1 \
-			"https://git.llvm.org/git/${tool}.git" "${WORK}/llvm/tools/${tool}"
-	fi
-done
-
-if [[ ! -d "${WORK}/llvm/tools/clang/tools/extra" ]]; then
-	echo "Downloading clang-tools-extra..."
-	git clone -q \
-		--branch "${LLVM_BRANCH}" \
-		--depth 1 \
-		"https://git.llvm.org/git/clang-tools-extra.git" "${WORK}/llvm/tools/clang/tools/extra"
-fi
-
-for proj in ${PROJECTS}; do
-	if [[ ! -d "${WORK}/llvm/projects/${proj}" ]]; then
-		echo "Downloading ${proj}..."
-		git clone -q \
-			--branch "${LLVM_BRANCH}" \
-			--depth 1 \
-			"https://git.llvm.org/git/${proj}.git" "${WORK}/llvm/projects/${proj}"
-	fi
-done
-
-for proj in ${RUNTIMES}; do
-	if [[ ! -d "${WORK}/llvm/runtimes/${proj}" ]]; then
-		echo "Downloading ${proj}..."
-		git clone -q \
-			--branch "${LLVM_BRANCH}" \
-			--depth 1 \
-			"https://git.llvm.org/git/${proj}.git" "${WORK}/llvm/runtime/${proj}"
-	fi
-done
-
-mkdir -p "${WORK}/build/llvm"
-pushd "${WORK}/build/llvm"
-
-echo "Configuring LLVM..."
-
-cmake -G Ninja \
-	-DCLANG_DEFAULT_RTLIB=compiler-rt \
-	-DCLANG_DEFAULT_STD_C=c11 \
-	-DCLANG_DEFAULT_STD_CXX=cxx14 \
-	-DCMAKE_BUILD_TYPE=Release \
-	-DCMAKE_C_COMPILER="clang" \
-	-DCMAKE_CXX_COMPILER="clang++" \
-	-DCMAKE_CXX_FLAGS="-Wno-unused-parameter -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -D_LIBUNWIND_IS_BAREMETAL=1 -U_LIBUNWIND_SUPPORT_DWARF_UNWIND" \
-	-DCMAKE_INSTALL_PREFIX="${SYSROOT}" \
-	-DCMAKE_MAKE_PROGRAM=`which ninja` \
-	-DDEFAULT_SYSROOT="${SYSROOT}" \
-	-DLIBCXX_CXX_ABI=libcxxabi \
-	-DLIBCXX_CXX_ABI_INCLUDE_PATHS="${WORK}/llvm/projects/libcxxabi/include" \
-	-DLIBCXX_CXX_ABI_LIBRARY_PATH=lib \
-	-DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF \
-	-DLIBCXX_ENABLE_NEW_DELETE_DEFINITIONS=ON \
-	-DLIBCXX_ENABLE_SHARED=OFF \
-	-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON \
-	-DLIBCXX_ENABLE_THREADS=OFF \
-	-DLIBCXX_INCLUDE_BENCHMARKS=OFF \
-	-DLIBCXX_USE_COMPILER_RT=ON \
-	-DLIBCXXABI_ENABLE_NEW_DELETE_DEFINITIONS=OFF \
-	-DLIBCXXABI_ENABLE_SHARED=OFF \
-	-DLIBCXXABI_ENABLE_STATIC_UNWINDER=ON \
-	-DLIBCXXABI_ENABLE_THREADS=OFF \
-	-DLIBCXXABI_LIBCXX_PATH="${WORK}/llvm/projects/libcxx" \
-	-DLIBCXXABI_USE_COMPILER_RT=ON \
-	-DLIBCXXABI_USE_LLVM_UNWINDER=ON \
-	-DLIBUNWIND_ENABLE_SHARED=OFF \
-	-DLIBUNWIND_ENABLE_THREADS=OFF \
-	-DLIBUNWIND_USE_COMPILER_RT=ON \
-	-DLLVM_CONFIG_PATH="${SYSROOT}/bin/llvm-config" \
-	-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-elf" \
-	-DLLVM_ENABLE_LIBCXX=ON \
-	-DLLVM_ENABLE_LLD=ON \
-	-DLLVM_ENABLE_PIC=OFF \
-	-DLLVM_ENABLE_THREADS=OFF \
-	-DLLVM_INSTALL_BINUTILS_SYMLINKS=ON \
-	-DLLVM_TARGETS_TO_BUILD="X86" \
-	${WORK}/llvm > cmake_configure.log
-
-#  -DCMAKE_ASM_COMPILER=nasm \
-#  -DCMAKE_LINKER="${SYSROOT}/bin/ld.lld" \
-#  -DCOMPILER_RT_ENABLE_LLD=ON \
-#  -DLIBCXX_ENABLE_LLD=ON \
-#  -DLIBCXX_ENABLE_STATIC_UNWINDER=ON \
-#  -DLIBCXXABI_ENABLE_LLD=ON \
-#  -DLIBUNWIND_ENABLE_LLD=ON \
-#  -DLLVM_ENABLE_PROJECTS="libcxx;libcxxabi;libunwind;compiler-rt" \
-#  -DCOMPILER_RT_BAREMETAL_BUILD=ON \
-#  -DLIBCXXABI_BAREMETAL=ON \
-
-echo "Building LLVM..."
-ninja && ninja install
-ninja cxx cxxabi compiler-rt
-ninja install-compiler-rt install-cxx install-cxxabi
-popd
-
-
-cp "${README}" "${SYSROOT}/README.md"

scripts/definitions/context.py

@@ -0,0 +1,83 @@
+class NotFound(Exception): pass
+
+class Context:
+    def __init__(self, path, verbose=False):
+        if isinstance(path, str):
+            self.__paths = [path]
+        else:
+            self.__paths = path
+
+        self.__closed = set()
+        self.__verbose = verbose
+
+        self.__deps = {}
+
+        self.objects = dict()
+        self.interfaces = dict()
+
+    verbose = property(lambda self: self.__verbose)
+
+    def find(self, filename):
+        from os.path import exists, isabs, join
+
+        if exists(filename) or isabs(filename):
+            return filename
+
+        for path in self.__paths:
+            full = join(path, filename)
+            if exists(full):
+                return full
+
+        raise NotFound(filename)
+
+    def parse(self, filename):
+        pending = set()
+        pending.add(filename)
+
+        from .parser import LarkError
+        from .parser import Lark_StandAlone as Parser
+        from .transformer import DefTransformer
+
+        objects = {}
+        interfaces = {}
+
+        while pending:
+            name = pending.pop()
+            self.__closed.add(name)
+
+            path = self.find(name)
+
+            parser = Parser(transformer=DefTransformer(name))
+
+            try:
+                imps, objs, ints = parser.parse(open(path, "r").read())
+            except LarkError as e:
+                import sys
+                import textwrap
+                print(f"\nError parsing {name}:", file=sys.stderr)
+                print(textwrap.indent(str(e), "    "), file=sys.stderr)
+                sys.exit(1)
+
+            objects.update(objs)
+            interfaces.update(ints)
+
+            self.__deps[name] = imps
+
+            pending.update(imps.difference(self.__closed))
+
+        from .types import ObjectRef
+        ObjectRef.connect(objects)
+
+        for obj in objects.values():
+            for method in obj.methods:
+                caps = method.options.get("cap", list())
+                for cap in caps:
+                    if not cap in obj.caps:
+                        from .errors import UnknownCapError
+                        raise UnknownCapError(f"Unknown capability {cap} on {obj.name}::{method.name}")
+
+        self.objects.update(objects)
+        self.interfaces.update(interfaces)
+
+    def deps(self):
+        return {self.find(k): tuple(map(self.find, v)) for k, v in self.__deps.items()}

scripts/definitions/errors.py

@@ -0,0 +1,3 @@
+class InvalidType(Exception): pass
+class UnknownTypeError(Exception): pass
+class UnknownCapError(Exception): pass

scripts/definitions/templates/syscall/client/private/syscall_trampolines.s

@@ -0,0 +1,23 @@
+%macro SYSCALL 2
+	global __syscall_%1
+	__syscall_%1:
+		push rbp
+		mov rbp, rsp
+
+		; args should already be in rdi, etc, but rcx will
+		; get stomped, so stash it in r10, which isn't a
+		; callee-saved register, but also isn't used in the
+		; function call ABI.
+		mov r10, rcx
+
+		mov rax, %2
+		syscall
+		; result is now already in rax, so just return
+
+		pop rbp
+		ret
+%endmacro
+
+{% for id, scope, method in interface.methods %}
+SYSCALL {% if scope %}{{ scope.name }}_{% endif %}{{ method.name }} {{ id }}
+{% endfor %}

scripts/definitions/templates/syscall/client/public/_object_.hh

@@ -0,0 +1,41 @@
+#pragma once
+/// \file {{filename}}
+{% if object.super %}
+
+#include <j6/{{ object.super.name }}.hh>
+{% endif %}
+
+namespace j6 {
+
+{% if object.desc %}
+{{ object.desc|indent('/// ', first=True) }}
+{% endif %}
+class {{ object.name }}
+{% if object.super %}    : public {{ object.super.name }}
+{% endif %}
+{
+public:
+{% macro argument(type, name, first, options=False) -%}
+    {%- for ctype, suffix in type.c_names(options) -%}
+        {%- if not first or not loop.first %}, {% endif %}{{ ctype }} {{ name }}{{ suffix }}
+    {%- endfor -%}
+{%- endmacro -%}
+
+{% for method in object.methods %}
+{% if method.desc %}    /// {{ method.desc|indent('    /// ') }}{% endif %} 
+{% for param in method.params %}
+{% if param.desc %}    /// \arg {{ "%-10s"|format(param.name) }} {{ param.desc }}
+{% endif %}
+{% endfor %}
+    {%+ if method.static %}static {% endif %}j6_status_t {{ method.name }} (
+{%- for param in method.params %}{{ argument(param.type, param.name, loop.first, options=param.options) }}{% endfor -%});
+
+{% endfor %}
+    ~{{ object.name }}();
+
+private:
+    {{ object.name }}(j6_handle_t handle) : m_handle {handle} {}
+    j6_handle_t m_handle;
+}
+
+} // namespace j6

scripts/definitions/templates/syscall/client/public/syscalls.h

@@ -0,0 +1,25 @@
+#pragma once
+
+#include <j6/types.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+{% macro argument(type, name, first, options=False) -%}
+    {%- for ctype, suffix in type.c_names(options) -%}
+        {%- if not first or not loop.first %}, {% endif %}{{ ctype }} {{ name }}{{ suffix }}
+    {%- endfor -%}
+{%- endmacro %}
+ 
+{% for id, scope, method in interface.methods %}
+j6_status_t __syscall_{% if scope %}{{ scope.name }}_{% endif %}{{ method.name }} (
+    {%- if not method.static -%}{{ argument(scope.reftype, "self", True) }}{% endif -%}
+    {%- set first = method.static -%}
+    {%- for param in method.params %}{{ argument(param.type, param.name, first, options=param.options) }}{% set first = False %}{% endfor -%}
+);
+{% endfor %}
+
+#ifdef __cplusplus
+}
+#endif
+

scripts/definitions/transformer.py

@@ -0,0 +1,156 @@
+from .parser import Transformer, v_args
+
+def get_opts(args):
+    from .types import Caps, CName, Description, Options, Type, UID
+
+    kinds = {
+        Description: "desc",
+        Options: "opts",
+        CName: "cname",
+        Caps: "caps",
+        UID: "uid",
+        Type: "typename",
+    }
+
+    result = dict()
+    outargs = []
+    for a in args:
+        for kind, name in kinds.items():
+            if isinstance(a, kind):
+                result[name] = a
+                break
+        else:
+            outargs.append(a)
+
+    return result, outargs
+
+class DefTransformer(Transformer):
+    def __init__(self, filename):
+        self.filename = filename
+
+    def start(self, args):
+        from .types import Import, Interface, Object
+
+        imports = set()
+        objects = dict()
+        interfaces = dict()
+
+        for o in args:
+            if isinstance(o, Object):
+                objects[o.name] = o
+
+            elif isinstance(o, Interface):
+                interfaces[o.name] = o
+
+            elif isinstance(o, Import):
+                imports.add(o)
+
+        return imports, objects, interfaces
+
+    @v_args(inline=True)
+    def import_statement(self, path):
+        from .types import Import
+        return Import(path)
+
+    def object(self, args):
+        from .types import Object
+        specials, args = get_opts(args)
+        name, args = args[0], args[1:]
+        return Object(name, children=args, **specials)
+
+    def interface(self, args):
+        from .types import Interface
+        specials, args = get_opts(args)
+        name, args = args[0], args[1:]
+        return Interface(name, children=args, **specials)
+
+    def method(self, args):
+        from .types import Method
+        specials, args = get_opts(args)
+        name, args = args[0], args[1:]
+        return Method(name, children=args, **specials)
+
+    def function(self, args):
+        from .types import Function
+        specials, args = get_opts(args)
+        name, args = args[0], args[1:]
+        return Function(name, children=args, **specials)
+
+    def param(self, args):
+        from .types import Param
+        specials, args = get_opts(args)
+        name = args[0]
+        return Param(name, **specials)
+
+    @v_args(inline=True)
+    def expose(self, s):
+        from .types import Expose
+        return Expose(s)
+
+    @v_args(inline=True)
+    def uid(self, s):
+        return s
+
+    @v_args(inline=True)
+    def cname(self, s):
+        from .types import CName
+        return CName(s)
+
+    @v_args(inline=True)
+    def name(self, s):
+        return s
+
+    @v_args(inline=True)
+    def type(self, s):
+        return s
+
+    @v_args(inline=True)
+    def super(self, s):
+        from .types import ObjectRef
+        return ObjectRef(s, self.filename)
+
+    def options(self, args):
+        from .types import Options
+        return Options([str(s) for s in args])
+
+    def capabilities(self, args):
+        from .types import Caps
+        return Caps([str(s) for s in args])
+
+    def description(self, s):
+        from .types import Description
+        return Description("\n".join(s))
+
+    @v_args(inline=True)
+    def object_name(self, n):
+        from .types import ObjectRef
+        return ObjectRef(n, self.filename)
+
+    def PRIMITIVE(self, s):
+        from .types import get_primitive
+        return get_primitive(s)
+
+    def UID(self, s):
+        from .types import UID
+        return UID(int(s, base=16))
+
+    def INT_TYPE(self, s):
+        return s
+
+    def NUMBER(self, s):
+        if s.startswith("0x"):
+            return int(s,16)
+        return int(s)
+
+    def COMMENT(self, s):
+        return s[2:].strip()
+
+    def OPTION(self, s):
+        return str(s)
+
+    def IDENTIFIER(self, s):
+        return str(s)
+
+    def PATH(self, s):
+        return str(s[1:-1])
+

scripts/definitions/types/__init__.py

@@ -0,0 +1,26 @@
+def _indent(x):
+    from textwrap import indent
+    return indent(str(x), '    ')
+
+class CName(str): pass
+class Description(str): pass
+class Import(str): pass
+class Caps(list): pass
+
+class Options(dict):
+    def __init__(self, opts = tuple()):
+        for opt in opts:
+            parts = opt.split(":", 1)
+            self[parts[0]] = self.get(parts[0], []) + ["".join(parts[1:])]
+
+class UID(int):
+    def __str__(self):
+        return f"{self:016x}"
+
+from .object import Object
+from .interface import Interface, Expose
+from .function import Function, Method, Param
+
+from .type import Type
+from .primitive import get_primitive
+from .objref import ObjectRef

scripts/definitions/types/function.py

@@ -0,0 +1,68 @@
+from . import _indent
+from . import Options
+
+def _hasopt(opt):
+    def test(self):
+        return opt in self.options
+    return test
+
+class Function:
+    typename = "function"
+
+    def __init__(self, name, opts=Options(), desc="", children=tuple()):
+        self.name = name
+        self.options = opts
+        self.desc = desc
+        self.params = [c for c in children if isinstance(c, Param)]
+        self.id = -1
+
+    def __str__(self):
+        parts = ["{} {}".format(self.typename, self.name)]
+        if self.desc:
+            parts.append(_indent(self.desc))
+        if self.options:
+            parts.append(f"    Options: {self.options}")
+        parts.extend(map(_indent, self.params))
+        return "\n".join(parts)
+
+    static = property(lambda x: True)
+    constructor = property(lambda x: False)
+
+
+class Method(Function):
+    typename = "method"
+
+    static = property(_hasopt("static"))
+    constructor = property(_hasopt("constructor"))
+
+
+class Param:
+    def __init__(self, name, typename, opts=Options(), desc=""):
+        self.name = name
+        self.type = typename
+        self.options = opts
+        self.desc = desc
+
+        self.caps = set()
+        for key, values in opts.items():
+            if key == "cap":
+                self.caps.update(values)
+
+    def __str__(self):
+        return "param {} {} {}  {}".format(
+                self.name, repr(self.type), self.options, self.desc or "")
+
+    @property
+    def outparam(self):
+        return "out" in self.options or "inout" in self.options
+
+    @property
+    def refparam(self):
+        return self.type.reference or self.outparam
+
+    @property
+    def optional(self):
+        if "optional" in self.options: return "optional"
+        elif "zero_ok" in self.options: return "zero_ok"
+        else: return "required"
+

scripts/definitions/types/interface.py

@@ -0,0 +1,46 @@
+from . import _indent
+from . import Options
+
+class Expose(object):
+    def __init__(self, type):
+        self.type = type
+
+    def __repr__(self):
+        return f'expose {repr(self.type)}'
+
+class Interface:
+    def __init__(self, name, uid, opts=Options(), desc="", children=tuple()):
+        from .function import Function
+
+        self.name = name
+        self.uid = uid
+        self.options = opts
+        self.desc = desc
+
+        self.functions = [c for c in children if isinstance(c, Function)]
+        self.__exposes = [e.type for e in children if isinstance(e, Expose)]
+
+    def __str__(self):
+        parts = [f"interface {self.name}: {self.uid}"]
+        if self.desc:
+            parts.append(_indent(self.desc))
+        if self.options:
+            parts.append(f"    Options: {self.options}")
+        parts.extend(map(_indent, self.exposes))
+        parts.extend(map(_indent, self.functions))
+        return "\n".join(parts)
+
+    @property
+    def methods(self):
+        mm = [(i, None, self.functions[i]) for i in range(len(self.functions))]
+
+        base = len(mm)
+        for o in self.exposes:
+            mm.extend([(base + i, o, o.methods[i]) for i in range(len(o.methods))])
+            base += len(o.methods)
+
+        return mm
+
+    @property
+    def exposes(self):
+        return [ref.object for ref in self.__exposes]

scripts/definitions/types/object.py

@@ -0,0 +1,33 @@
+from . import _indent
+from . import Caps, Options
+
+class Object:
+    def __init__(self, name, uid, typename=None, opts=Options(), caps=Caps(), desc="", children=tuple(), cname=None):
+        self.name = name
+        self.uid = uid
+        self.options = opts
+        self.desc = desc
+        self.methods = children
+        self.cname = cname or name
+        self.caps = caps
+
+        self.__super = typename
+
+        from . import ObjectRef
+        self.__ref = ObjectRef(name)
+
+    def __str__(self):
+        parts = [f"object {self.name}: {self.uid}"]
+        if self.desc:
+            parts.append(_indent(self.desc))
+        if self.options:
+            parts.append(f"    Options: {self.options}")
+        parts.extend(map(_indent, self.methods))
+        return "\n".join(parts)
+
+    reftype = property(lambda self: self.__ref)
+
+    @property
+    def super(self):
+        if self.__super is not None:
+            return self.__super.object

scripts/definitions/types/objref.py

@@ -0,0 +1,49 @@
+from .type import Type
+
+class ObjectRef(Type):
+    all_refs = {}
+
+    def __init__(self, name, filename=None):
+        super().__init__(name)
+        self.__c_type = "j6_handle_t"
+        self.__object = None
+        ObjectRef.all_refs[self] = filename
+
+    def __repr__(self):
+        return f'ObjectRef({self.name})'
+
+    object = property(lambda self: self.__object)
+
+    def c_names(self, options):
+        one = self.__c_type
+        out = bool({"out", "inout"}.intersection(options))
+
+        if "list" in options:
+            two = "size_t"
+            one = f"{one} *"
+
+            if out:
+                two += " *"
+            return ((one, ""), (two, "_count"))
+
+        else:
+            if out:
+                one += " *"
+            return ((one, ""),)
+
+    def cxx_names(self, options):
+        if not self.needs_object(options):
+            return self.c_names(options)
+        return ((f"obj::{self.name} *", ""),)
+
+    def needs_object(self, options):
+        return not bool({"out", "inout", "list", "handle"}.intersection(options))
+
+
+    @classmethod
+    def connect(cls, objects):
+        for ref, filename in cls.all_refs.items():
+            ref.__object = objects.get(ref.name)
+            if ref.__object is None:
+                from ..errors import UnknownTypeError
+                raise UnknownTypeError(ref.name, filename)

scripts/definitions/types/primitive.py

@@ -0,0 +1,72 @@
+from .type import Type
+
+class Primitive(Type):
+    def __init__(self, name, c_type):
+        super().__init__(name)
+        self.c_type = c_type
+
+    def __repr__(self):
+        return f'Primitive({self.name})'
+
+    def c_names(self, options=dict()):
+        one = self.c_type
+        if "out" in options or "inout" in options:
+            one += " *"
+
+        return ((one, ""),)
+
+    def cxx_names(self, options):
+        return self.c_names(options)
+
+class PrimitiveRef(Primitive):
+    def __init__(self, name, c_type, counted=False):
+        super().__init__(name, c_type)
+        self.__counted = counted
+
+    reference = property(lambda self: True)
+
+    def c_names(self, options=dict()):
+        one = f"{self.c_type} *"
+        two = "size_t"
+
+        if "out" in options or "inout" in options:
+            two += " *"
+        else:
+            one = "const " + one
+
+        if self.__counted:
+            return ((one, ""), (two, "_len"))
+        else:
+            return ((one, ""),)
+
+    def cxx_names(self, options):
+        return self.c_names(options)
+
+_primitives = {
+    "string": PrimitiveRef("string", "char"),
+    "buffer": PrimitiveRef("buffer", "void", counted=True),
+
+    "int": Primitive("int", "int"),
+    "uint": Primitive("uint", "unsigned"),
+    "size": Primitive("size", "size_t"),
+    "address": Primitive("address", "uintptr_t"),
+
+    "int8": Primitive("int8", "int8_t"),
+    "uint8": Primitive("uint8", "uint8_t"),
+
+    "int16": Primitive("int16", "int16_t"),
+    "uint16": Primitive("uint16", "uint16_t"),
+
+    "int32": Primitive("int32", "int32_t"),
+    "uint32": Primitive("uint32", "uint32_t"),
+
+    "int64": Primitive("int64", "int64_t"),
+    "uint64": Primitive("uint64", "uint64_t"),
+}
+
+def get_primitive(name):
+    p = _primitives.get(name)
+    if not p:
+        from ..errors import InvalidType
+        raise InvalidType(name)
+    return p

scripts/definitions/types/type.py

@@ -0,0 +1,15 @@
+class Type:
+    def __init__(self, name):
+        self.__name = name
+
+    name = property(lambda self: self.__name)
+    reference = property(lambda self: False)
+
+    def c_names(self, options):
+        raise NotImplemented("Call to base Type.c_names")
+
+    def cxx_names(self, options):
+        raise NotImplemented("Call to base Type.c_names")
+
+    def needs_object(self, options):
+        return False

scripts/idgen

@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+def hashid(s):
+    from hashlib import shake_128 as sh
+    return sh(s.encode('utf-8')).hexdigest(8)
+
+import sys
+
+for arg in sys.argv[1:]:
+    id = hashid(arg)
+    print(f"{arg}: {id}")

scripts/j6libc.py

@@ -0,0 +1,56 @@
+import cog
+
+supported_architectures = {
+    "amd64": "__amd64__",
+}
+
+def arch_includes(header):
+    prefix = "if"
+    for arch, define in supported_architectures.items():
+        cog.outl(f"#{prefix} defined({define})")
+        cog.outl(f"#include <__j6libc/arch/{arch}/{header}>")
+        prefix = "elif"
+    cog.outl("#endif")
+
+
+int_widths = (8, 16, 32, 64)
+int_mods = ("fast", "least")
+
+int_types = {
+    # type: abbrev
+    "char": "char",
+    "short": "short",
+    "int": "int",
+    "long": "long",
+    "long long": "llong",
+}
+
+def definition(kind, name, val, width=24):
+    cog.outl(f"{kind} {name:{width}} {val}")
+
+
+atomic_types = {
+    "_Bool": "bool",
+    "signed char": "schar",
+    "char16_t": "char16_t",
+    "char32_t": "char32_t",
+    "wchar_t": "wchar_t",
+    "wchar_t": "wchar_t",
+    "size_t": "size_t",
+    "ptrdiff_t": "ptrdiff_t",
+    "intptr_t": "intptr_t",
+    "uintptr_t": "uintptr_t",
+    "intmax_t": "intmax_t",
+    "uintmax_t": "uintmax_t",
+    }
+
+for name, abbrev in int_types.items():
+    atomic_types.update({name: abbrev, f"unsigned {name}": f"u{abbrev}"})
+
+for width in int_widths:
+    atomic_types.update({t: t for t in (
+            f"int_least{width}_t",
+            f"uint_least{width}_t",
+            f"int_fast{width}_t",
+            f"uint_fast{width}_t")})
+

scripts/memory.py

@@ -0,0 +1,32 @@
+class Layout:
+    from collections import namedtuple
+    Region = namedtuple("Region", ("name", "start", "size", "shared"))
+
+    sizes = {'G': 1024 ** 3, 'T': 1024 ** 4}
+
+    @staticmethod
+    def get_size(desc):
+        size, mag = int(desc[:-1]), desc[-1]
+
+        try:
+            mult = Layout.sizes[mag]
+        except KeyError:
+            raise RuntimeError(f"No magnitude named '{mag}'.")
+
+        return size * mult
+
+    def __init__(self, path):
+        from yaml import safe_load
+
+        regions = []
+        addr = 1 << 64
+
+        with open(path, 'r') as infile:
+            data = safe_load(infile.read())
+            for r in data:
+                size = Layout.get_size(r["size"])
+                addr -= size
+                regions.append(Layout.Region(r["name"], addr, size,
+                    r.get("shared", False)))
+
+        self.regions = tuple(regions)

scripts/parse_syms.py

@@ -1,10 +0,0 @@
-#!/usr/bin/env python3
-
-def parse_elf(filename):
-    import struct
-    with open(filename, 'rb') as elf:
-
-if __name__ == "__main__":
-    import sys
-    for arg in sys.argv[1:]:
-        parse_elf(arg)

scripts/psf_to_cpp.py

@@ -1,27 +0,0 @@
-#!/usr/bin/env python3
-
-from fontpsf import PSF2
-
-def print_header(filename):
-    font = PSF2.load(filename)
-
-    print("#pragma once")
-    print(f"// This file was autogenerated by psf_to_cpp.py from {font.filename}\n")
-
-    print(f"const uint8_t font_glyph_size = {font.charsize};")
-    print(f"const uint8_t font_glyph_width = {font.dimension[0]};")
-    print(f"const uint8_t font_glyph_height = {font.dimension[1]};")
-    print(f"const uint16_t font_glyph_count = {font.count};\n")
-
-    print('const uint8_t font_glyph_data[] = {')
-
-    for glyph in font:
-        print("    ", "".join([f"0x{b:02x}," for b in glyph.data]), end="")
-        print(" // {}".format(glyph.description()))
-
-    print("};")
-
-if __name__ == "__main__":
-    import sys
-    for filename in sys.argv[1:]:
-        print_header(filename)

scripts/readme_for_prebuilt_sysroots.md

@@ -1,14 +0,0 @@
-# jsix OS sysroot
-
-This is a pre-built sysroot for building the jsix operating system kernel,
-bootloader, and utilities. This package is provided as a convenience, and
-contains software from the following repositories.
-
-## The LLVM toolchain
-
-The LLVM sources as downloaded via git from [llvm.org][llvm] under the terms of
-the [Apache License v2.0][apache2], modified [as described here][llvmlic].
-
-[llvm]: https://llvm.org
-[apache2]: https://www.apache.org/licenses/LICENSE-2.0
-[llvmlic]: https://llvm.org/docs/DeveloperPolicy.html#new-llvm-project-license-framework

scripts/sysconf.py

@@ -0,0 +1,17 @@
+class Sysconf:
+    from collections import namedtuple
+    Var = namedtuple("Var", ("name", "section", "type"))
+
+    def __init__(self, path):
+        from yaml import safe_load
+
+        sys_vars = []
+
+        with open(path, 'r') as infile:
+            data = safe_load(infile.read())
+            self.address = data["address"]
+
+            for v in data["vars"]:
+                sys_vars.append(Sysconf.Var(v["name"], v["section"], v["type"]))
+
+        self.vars = tuple(sys_vars)

scripts/templates/build.ninja.j2

@@ -1,214 +0,0 @@
-ninja_required_version = 1.3
-builddir = {{ buildroot }}
-srcroot = {{ srcroot }}
-modulefile = {{ modulefile }}
-
-{%- for var, value in vars %}
-{{ var }} = {{ value }}
-{%- endfor %}
-
-warnflags = $
-    -Wformat=2 $
-    -Winit-self $
-    -Wfloat-equal $
-    -Winline $
-    -Wmissing-format-attribute $
-    -Wmissing-include-dirs $
-    -Wswitch $
-    -Wundef $
-    -Wdisabled-optimization $
-    -Wpointer-arith $
-    -Wno-attributes $
-    -Wno-sign-compare $
-    -Wno-multichar $
-    -Wno-div-by-zero $
-    -Wno-endif-labels $
-    -Wno-pragmas $
-    -Wno-format-extra-args $
-    -Wno-unused-result $
-    -Wno-deprecated-declarations $
-    -Wno-unused-function $
-    -Wno-address-of-packed-member $
-    -Werror
-
-ccflags = $
-    -I${srcroot}/src/include $
-    -I${srcroot}/src/include/x86_64 $
-    -fcolor-diagnostics $
-    -DVERSION_MAJOR={{ version_major }} $
-    -DVERSION_MINOR={{ version_minor }} $
-    -DVERSION_PATCH={{ version_patch }} $
-    -DVERSION_GITSHA=0x0{{ version_sha }} $
-    -DGIT_VERSION=\"{{ version }}\" $
-    -DGIT_VERSION_WIDE=L\"{{ version }}\" $
-    $warnflags
-
-asflags = $
-    -DVERSION_MAJOR={{ version_major }} $
-    -DVERSION_MINOR={{ version_minor }} $
-    -DVERSION_PATCH={{ version_patch }} $
-    -DVERSION_GITSHA=0x{{ version_sha }} $
-    -I${srcroot}/src/include
-
-cflags = -std=c11
-cxxflags = -std=c++17
-libs =
-
-rule c
-    deps = gcc
-    depfile = $out.d
-    description = Compiling $name
-    command = $cc -MMD -MF $out.d $ccflags $cflags -o $out -c $in
-
-rule dump_c_defs
-    description = Dumping C defines for $target
-    command = echo "" | $cc $ccflags $cflags -dM -E - > $out
-
-rule dump_c_run
-    description = Dumping C arguments for $target
-    command = $
-        echo "#!/bin/bash" > $out; $
-        echo '$cc $ccflags $cflags $$*' >> $out; $
-        chmod a+x $out
-
-rule cpp
-    deps = gcc
-    depfile = $out.d
-    description = Compiling $name
-    command = $cxx -MMD -MF $out.d $cxxflags $ccflags -o $out -c $in
-
-rule dump_cpp_defs
-    description = Dumping C++ defines for $target
-    command = echo "" | $cxx -x c++ $cxxflags $ccflags -dM -E - > $out
-
-rule dump_cpp_run
-    description = Dumping C++ arguments for $target
-    command = $
-        echo "#!/bin/bash" > $out; $
-        echo '$cc $cxxflags $ccflags $$*' >> $out; $
-        chmod a+x $out
-
-rule s
-    deps = gcc
-    depfile = $out.d
-    description = Assembling $name
-    command = $nasm -o $out -felf64 -MD $out.d $asflags $in
-
-rule exe
-    description = Linking $name
-    command = $ld $ldflags -o $out $in $libs
-
-rule lib
-    description = Archiving $name
-    command = $ar qcs $out $in
-
-rule regen
-    generator = true
-    description = Regenrating build files
-    command = $
-        {{ generator }} $
-        --file $modulefile $
-        --dir $builddir $
-        generate
-
-rule cp
-    description = Copying $name
-    command = cp $in $out
-
-rule dump
-    description = Dumping decompiled $name
-    command = objdump -DSC -M intel $in > $out
-
-rule makest
-    description = Making symbol table
-    command = nm $in | ${srcroot}/scripts/build_symbol_table.py $out
-
-rule makeefi
-    description = Converting $name
-    command = objcopy $
-        -j .text $
-        -j .sdata $
-        -j .data $
-        -j .dynamic $
-        -j .dynsym $
-        -j .rel $
-        -j .rela $
-        -j .reloc $
-        --target=efi-app-x86_64 $
-        $in $out
-
-rule makefat
-    description = Creating $name
-    command = $
-        cp $srcroot/assets/diskbase.img $out; $
-        mcopy -s -D o -i $out@@1M $builddir/fatroot/* ::/
-
-rule strip
-    description = Stripping $name
-    command = $
-        cp $in $out; $
-        objcopy --only-keep-debug $out $out.debug; $
-        strip -g $out; $
-        objcopy --add-gnu-debuglink=$out.debug $out
-
-{% for target in targets %}
-subninja {{ target }}/target.ninja
-{% endfor %}
-
-build $
-    {%- for buildfile in buildfiles %}
-    {{ buildfile }} $
-    {%- endfor %}
-    : regen | $
-    {%- for template in templates %}
-    {{ template }} $
-    {%- endfor %}
-    $modulefile $
-    {{ generator }}
-
-build $builddir/ovmf_vars.fd : cp $srcroot/assets/ovmf/x64/ovmf_vars.fd
-    name = ovmf_vars.fd
-
-build $builddir/ovmf_vars_d.fd : cp $srcroot/assets/ovmf/x64/ovmf_vars_d.fd
-    name = ovmf_vars_d.fd
-
-build $builddir/jsix.elf | $builddir/jsix.elf.debug : strip $builddir/host/jsix.elf
-    name = kernel
-
-build $builddir/jsix.dump : dump $builddir/host/jsix.elf
-    name = kernel
-
-build $builddir/jsix.elf-gdb.py : cp ${srcroot}/assets/debugging/jsix.elf-gdb.py
-    name = kernel debug python scripts
-
-build $builddir/fatroot/jsix.elf : cp $builddir/jsix.elf
-    name = kernel to FAT image
-
-build $builddir/fatroot/efi/boot/bootx64.efi : cp $builddir/boot/boot.efi
-    name = bootloader to FAT image
-
-build $builddir/fatroot/nulldrv.elf : cp $builddir/user/nulldrv.elf
-    name = null driver to FAT image
-
-build $builddir/fatroot/fb.elf : cp $builddir/user/fb.elf
-    name = fb driver to FAT image
-
-build ${builddir}/fatroot/symbol_table.dat : makest ${builddir}/jsix.elf
-
-build $builddir/jsix.img : makefat | $
-    $builddir/fatroot/symbol_table.dat $
-    $builddir/fatroot/nulldrv.elf $
-    $builddir/fatroot/fb.elf $
-    $builddir/fatroot/jsix.elf $
-    $builddir/fatroot/efi/boot/bootx64.efi
-    name = jsix.img
-
-default $
-    $builddir/ovmf_vars.fd $
-    $builddir/ovmf_vars_d.fd $
-    $builddir/jsix.dump $
-    $builddir/jsix.elf-gdb.py $
-    $builddir/jsix.img
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.boot.j2

@@ -1,14 +0,0 @@
-{% extends "exe.default.j2" %}
-{% block variables %}
-{{ super() }}
-
-ccflags = $ccflags $
-    -g3 $
-    -DKERNEL_FILENAME=L\"jsix.elf\" $
-    -I${srcroot}/external/include $
-    -I${srcroot}/external/include/X64
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.default.j2

@@ -1,8 +0,0 @@
-{% extends "module.base.j2" %}
-{% block variables %}
-{{ super() }}
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.kernel.j2

@@ -1,13 +0,0 @@
-{% extends "exe.default.j2" %}
-{% block variables %}
-{{ super() }}
-
-asflags = $asflags -I${srcroot}/src/kernel/
-libs = $libs
-ldflags = $ldflags -T ${srcroot}/src/arch/x86_64/kernel.ld
-ccflags = $ccflags -I${srcroot}/external
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.makerd.j2

@@ -1,10 +0,0 @@
-{% extends "exe.default.j2" %}
-{% block variables %}
-{{ super() }}
-
-ccflags = $ccflags -I${srcroot}/external/cpptoml
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.tests.j2

@@ -1,10 +0,0 @@
-{% extends "exe.default.j2" %}
-{% block variables %}
-{{ super() }}
-
-ccflags = $ccflags -ggdb -I${srcroot}/external/catch
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/lib.default.j2

@@ -1,4 +0,0 @@
-{% extends "module.base.j2" %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/module.base.j2

@@ -1,45 +0,0 @@
-moddir = ${builddir}/{{ name }}.dir
-
-{% block variables %}
-ccflags = $ccflags $
-    {%- for dep in depmods %}
-    {%- for inc in dep.includes %}
-    -I${srcroot}/{{ inc }} $
-    {%- endfor %}
-    {%- endfor %}
-    {%- for inc in module.includes %}
-    -I${srcroot}/{{ inc }} $
-    {%- endfor %}
-    {%- for define in module.defines %}
-    -D{{ define }} $
-    {%- endfor %}
-{% endblock %}
-
-{% for source in module.source %}
-build ${moddir}/{{ source.output }} : {{ source.action }} ${srcroot}/{{ source.input }} || {{ buildfile }}
-    name = {{ source.name }}
-{% endfor %}
-
-build ${builddir}/{{ module.output }} : {{ module.kind }} $
-{%- for source in module.source %}
-    ${moddir}/{{ source.output }} $
-{%- endfor -%}
-{%- for dep in deplibs %}
-    ${builddir}/{{ dep.output }} $
-{%- endfor %}
-    | $
-{%- for dep in depexes %}
-    ${builddir}/{{ dep.output }} $
-{%- endfor %}
-    {{ buildfile }}
-    name = {{ name }}
-
-{% if module.default %}
-default ${builddir}/{{ module.output }}
-{% endif %}
-
-{% block extra %}
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.boot.j2

@@ -1,38 +0,0 @@
-{% extends "target.default.j2" %}
-
-{% block binaries %}
-cc = clang
-cxx = clang++
-ld = clang++
-ar = ar
-nasm = nasm
-objcopy = objcopy
-{% endblock %}
-
-{% block variables %}
-
-ccflags = $ccflags $
-    -I $srcroot/external $
-    --target=x86_64-unknown-windows $
-	-ffreestanding $
-	-mno-red-zone $
-	-fshort-wchar $
-	-fno-omit-frame-pointer $
-	-ggdb
-
-cxxflags = $cxxflags $
-    -fno-rtti $
-    -fno-exceptions
-
-ldflags = $ldflags $
-    --target=x86_64-unknown-windows $
-	-nostdlib $
-    -Wl,-entry:efi_main $
-    -Wl,-subsystem:efi_application $
-    -fuse-ld=lld-link $
-	-g
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.default.j2

@@ -1,31 +0,0 @@
-builddir = $builddir/{{ target }}
-target = {{ target }}
-
-{% block variables %}
-{% endblock %}
-
-{% block binaries %}
-cc = clang
-cxx = clang++
-ld = ld
-ar = ar
-nasm = nasm
-objcopy = objcopy
-{% endblock %}
-
-{% for module in modules %}
-subninja {{ module }}.ninja
-{% endfor %}
-
-build ${builddir}/c.defs : dump_c_defs | {{ buildfile }}
-build ${builddir}/cpp.defs : dump_cpp_defs | {{ buildfile }}
-build ${builddir}/c.run : dump_c_run | {{ buildfile }}
-build ${builddir}/cpp.run : dump_cpp_run | {{ buildfile }}
-
-default ${builddir}/c.defs
-default ${builddir}/cpp.defs
-default ${builddir}/c.run
-default ${builddir}/cpp.run
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.host.j2

@@ -1,44 +0,0 @@
-{% extends "target.default.j2" %}
-
-{% block binaries %}
-cc = ${srcroot}/sysroot/bin/clang
-cxx = ${srcroot}/sysroot/bin/clang++
-ld = ${srcroot}/sysroot/bin/ld.lld
-ar = ${srcroot}/sysroot/bin/ar
-nasm = nasm
-objcopy = ${srcroot}/sysroot/bin/objcopy
-{% endblock %}
-
-{% block variables %}
-
-ccflags = $ccflags $
-    -nostdlib $
-    -ffreestanding $
-    -nodefaultlibs $
-    -fno-builtin $
-    -mno-sse $
-    -fno-omit-frame-pointer $
-    -mno-red-zone $
-    -g $
-    -mcmodel=large $
-    -D__ELF__ $
-    -D__JSIX__ $
-    -isystem${srcroot}/sysroot/include $
-    -isystem${srcroot}/src/libraries/libc/include $
-    --sysroot="${srcroot}/sysroot"
-
-cxxflags = $cxxflags $
-    -fno-exceptions $
-    -fno-rtti $
-    -isystem${srcroot}/sysroot/include/c++/v1
-
-ldflags = $ldflags $
-    -g $
-    -nostdlib $
-    -Bsymbolic $
-    -Bstatic
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.native.j2

@@ -1,15 +0,0 @@
-{% extends "target.default.j2" %}
-
-{% block binaries %}
-{{ super() }}
-ld = clang++
-{% endblock %}
-
-{% block variables %}
-
-ccflags = $ccflags -g -ggdb
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.user.j2

@@ -1,45 +0,0 @@
-{% extends "target.default.j2" %}
-
-{% block binaries %}
-cc = ${srcroot}/sysroot/bin/clang
-cxx = ${srcroot}/sysroot/bin/clang++
-ld = ${srcroot}/sysroot/bin/ld.lld
-ar = ${srcroot}/sysroot/bin/ar
-nasm = nasm
-objcopy = ${srcroot}/sysroot/bin/objcopy
-{% endblock %}
-
-{% block variables %}
-
-ccflags = $ccflags $
-    -nostdlib $
-    -nodefaultlibs $
-    -fno-builtin $
-    -mno-sse $
-    -fno-omit-frame-pointer $
-    -mno-red-zone $
-    -g $
-    -mcmodel=large $
-    -D__ELF__ $
-    -D__JSIX__ $
-    -isystem${srcroot}/sysroot/include $
-    -isystem${srcroot}/src/libraries/libc/include $
-    --sysroot="${srcroot}/sysroot"
-
-cxxflags = $cxxflags $
-    -fno-exceptions $
-    -fno-rtti $
-    -isystem${srcroot}/sysroot/include/c++/v1
-
-ldflags = $ldflags $
-    -g $
-    -nostdlib $
-    -Bsymbolic $
-    -Bstatic $
-    --sysroot="${srcroot}/sysroot" $
-	-L "${srcroot}/sysroot/lib" $
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

src/arch/x86_64/config.mk

@@ -1,6 +0,0 @@
-AS := nasm
-ASFLAGS := -felf64
-LDFLAGS := -m elf_x86_64
-CFLAGS := -march=nocona -m64
-
-# vim:ft=make

src/arch/x86_64/details.c

@@ -1 +0,0 @@
-const char *KERNEL_PLATFORM = "x86_64";

src/boot/allocator.cpp

@@ -0,0 +1,173 @@
+#include <uefi/boot_services.h>
+#include <uefi/types.h>
+
+#include <bootproto/init.h>
+#include <bootproto/kernel.h>
+#include <util/no_construct.h>
+#include <util/pointers.h>
+
+#include "allocator.h"
+#include "error.h"
+#include "memory.h"
+
+namespace boot {
+
+util::no_construct<memory::allocator> __g_alloc_storage;
+memory::allocator &g_alloc = __g_alloc_storage.value;
+
+namespace memory {
+
+using bootproto::allocation_register;
+using bootproto::module;
+using bootproto::page_allocation;
+
+static_assert(sizeof(allocation_register) == page_size);
+
+
+void
+allocator::init(
+        allocation_register *&allocs,
+        modules_page *&modules,
+        uefi::boot_services *bs)
+{
+    new (&g_alloc) allocator(*bs);
+    allocs = g_alloc.m_register;
+    modules = g_alloc.m_modules;
+}
+
+
+allocator::allocator(uefi::boot_services &bs) :
+    m_bs(bs),
+    m_register(nullptr),
+    m_modules(nullptr)
+{
+    add_register();
+    add_modules();
+}
+
+void
+allocator::add_register()
+{
+    allocation_register *reg = nullptr;
+
+    try_or_raise(
+        m_bs.allocate_pages(uefi::allocate_type::any_pages,
+            uefi::memory_type::loader_data, 1, reinterpret_cast<void**>(&reg)),
+        L"Failed allocating allocation register page");
+
+    m_bs.set_mem(reg, sizeof(allocation_register), 0);
+
+    if (m_register)
+        m_register->next = reg;
+
+    m_register = reg;
+    return;
+}
+
+void
+allocator::add_modules()
+{
+    modules_page *mods = reinterpret_cast<modules_page*>(
+        allocate_pages(1, alloc_type::init_args, true));
+
+    if (m_modules)
+        m_modules->next = reinterpret_cast<uintptr_t>(mods);
+
+    mods->modules = reinterpret_cast<module*>(mods + 1);
+    m_modules = mods;
+    m_next_mod = mods->modules;
+    return;
+}
+
+void *
+allocator::allocate_pages(size_t count, alloc_type type, bool zero)
+{
+    if (count & ~0xffffffffull) {
+        error::raise(uefi::status::unsupported,
+            L"Cannot allocate more than 16TiB in pages at once.",
+            __LINE__);
+    }
+
+    if (!m_register || m_register->count == 0xff)
+        add_register();
+
+    void *pages = nullptr;
+
+    try_or_raise(
+        m_bs.allocate_pages(uefi::allocate_type::any_pages,
+            uefi::memory_type::loader_data, count, &pages),
+        L"Failed allocating usable pages");
+
+    page_allocation &ent = m_register->entries[m_register->count++];
+    ent.address = reinterpret_cast<uintptr_t>(pages);
+    ent.count = count;
+    ent.type = type;
+
+    if (zero)
+        m_bs.set_mem(pages, count * page_size, 0);
+
+    return pages;
+}
+
+module *
+allocator::allocate_module_untyped(size_t size)
+{
+    size_t remaining =
+        reinterpret_cast<uintptr_t>(m_modules) + page_size
+        - reinterpret_cast<uintptr_t>(m_next_mod);
+
+    if (size > remaining)
+        add_modules();
+
+    ++m_modules->count;
+    module *m = m_next_mod;
+    m_next_mod = util::offset_pointer(m_next_mod, size);
+
+    m->mod_length = size;
+    return m;
+}
+
+void *
+allocator::allocate(size_t size, bool zero)
+{
+    void *p = nullptr;
+    try_or_raise(
+        m_bs.allocate_pool(uefi::memory_type::loader_data, size, &p),
+        L"Could not allocate pool memory");
+
+    if (zero)
+        m_bs.set_mem(p, size, 0);
+
+    return p;
+}
+
+void
+allocator::free(void *p)
+{
+    try_or_raise(
+        m_bs.free_pool(p),
+        L"Freeing pool memory");
+}
+
+void
+allocator::memset(void *start, size_t size, uint8_t value)
+{
+    m_bs.set_mem(start, size, value);
+}
+
+void
+allocator::copy(void *to, void *from, size_t size)
+{
+    m_bs.copy_mem(to, from, size);
+}
+
+} // namespace memory
+} // namespace boot
+
+
+void * operator new (size_t size, void *p) { return p; }
+void * operator new(size_t size)           { return boot::g_alloc.allocate(size); }
+void * operator new [] (size_t size)       { return boot::g_alloc.allocate(size); }
+void operator delete (void *p) noexcept    { return boot::g_alloc.free(p); }
+void operator delete [] (void *p) noexcept { return boot::g_alloc.free(p); }
+

src/boot/allocator.h

@@ -0,0 +1,76 @@
+#pragma once
+/// \file allocator.h
+/// Page allocator class definition
+
+namespace uefi {
+    class boot_services;
+}
+
+namespace bootproto {
+    enum class allocation_type : uint8_t;
+    struct allocation_register;
+    struct module;
+    struct modules_page;
+}
+
+namespace boot {
+namespace memory {
+
+using alloc_type = bootproto::allocation_type;
+
+class allocator
+{
+public:
+    using allocation_register = bootproto::allocation_register;
+    using module = bootproto::module;
+    using modules_page = bootproto::modules_page;
+
+    allocator(uefi::boot_services &bs);
+
+    void * allocate(size_t size, bool zero = false);
+    void free(void *p);
+
+    void * allocate_pages(size_t count, alloc_type type, bool zero = false);
+
+    template <typename M>
+    M * allocate_module(size_t extra = 0) {
+        return static_cast<M*>(allocate_module_untyped(sizeof(M) + extra));
+    }
+
+    void memset(void *start, size_t size, uint8_t value);
+    void copy(void *to, void *from, size_t size);
+
+    inline void zero(void *start, size_t size) { memset(start, size, 0); }
+
+    /// Initialize the global allocator
+    /// \arg allocs   [out] Poiinter to the initial allocation register
+    /// \arg modules  [out] Pointer to the initial modules_page
+    /// \arg bs       UEFI boot services
+    static void init(
+            allocation_register *&allocs,
+            modules_page *&modules,
+            uefi::boot_services *bs);
+
+private:
+    void add_register();
+    void add_modules();
+    module * allocate_module_untyped(size_t size);
+
+    uefi::boot_services &m_bs;
+
+    allocation_register *m_register;
+    modules_page *m_modules;
+    module *m_next_mod;
+};
+
+} // namespace memory
+
+extern memory::allocator &g_alloc;
+
+} // namespace boot
+
+void * operator new (size_t size, void *p);
+void * operator new(size_t size);
+void * operator new [] (size_t size);
+void operator delete (void *p) noexcept;
+void operator delete [] (void *p) noexcept;

src/boot/boot.module

@@ -0,0 +1,23 @@
+# vim: ft=python
+
+boot = module("boot",
+    kind = "exe",
+    output = "boot.efi",
+    targets = [ "boot" ],
+    deps = [ "cpu", "elf", "util", "bootproto" ],
+    sources = [
+        "allocator.cpp",
+        "bootconfig.cpp",
+        "console.cpp",
+        "error.cpp",
+        "fs.cpp",
+        "hardware.cpp",
+        "loader.cpp",
+        "main.cpp",
+        "memory.cpp",
+        "memory_map.cpp",
+        "paging.cpp",
+        "status.cpp",
+        "support.cpp",
+        "video.cpp",
+    ])

src/boot/bootconfig.cpp

@@ -0,0 +1,63 @@
+#include <uefi/boot_services.h>
+#include <uefi/types.h>
+
+#include "bootconfig.h"
+#include "error.h"
+#include "fs.h"
+#include "status.h"
+
+namespace boot {
+
+constexpr uint64_t jsixboot = 0x746f6f627869736a; // "jsixboot"
+
+static const wchar_t *
+read_string(util::buffer &data)
+{
+    uint16_t size = *util::read<uint16_t>(data);
+    const wchar_t *string = reinterpret_cast<const wchar_t*>(data.pointer);
+    data += size;
+    return string;
+}
+
+static void
+read_descriptor(descriptor &e, util::buffer &data)
+{
+    e.flags = static_cast<desc_flags>(*util::read<uint16_t>(data));
+    e.path = read_string(data);
+    e.desc = read_string(data);
+}
+
+bootconfig::bootconfig(util::buffer data, uefi::boot_services *bs)
+{
+    status_line status {L"Loading boot config"};
+
+    if (*util::read<uint64_t>(data) != jsixboot)
+        error::raise(uefi::status::load_error, L"Bad header in jsix_boot.dat");
+
+    const uint8_t version = *util::read<uint8_t>(data);
+    if (version != 0)
+        error::raise(uefi::status::incompatible_version, L"Bad version in jsix_boot.dat");
+
+    data += 1; // reserved byte
+    uint16_t num_programs = *util::read<uint16_t>(data);
+    uint16_t num_data = *util::read<uint16_t>(data);
+
+    m_flags = *util::read<uint16_t>(data);
+
+    read_descriptor(m_kernel, data);
+    read_descriptor(m_init, data);
+
+    m_programs.count = num_programs;
+    m_programs.pointer = new descriptor [num_programs];
+
+    for (unsigned i = 0; i < num_programs; ++i)
+        read_descriptor(m_programs[i], data);
+
+    m_data.count = num_programs;
+    m_data.pointer = new descriptor [num_data];
+
+    for (unsigned i = 0; i < num_data; ++i)
+        read_descriptor(m_data[i], data);
+}
+
+} // namespace boot

src/boot/bootconfig.h

@@ -0,0 +1,45 @@
+/// \file bootconfig.h
+/// Definitions for reading the jsix bootconfig file
+#pragma once
+
+#include <bootproto/bootconfig.h>
+#include <util/counted.h>
+
+namespace uefi {
+    struct boot_services;
+}
+
+namespace boot {
+
+using desc_flags = bootproto::desc_flags;
+
+struct descriptor {
+    desc_flags flags;
+    wchar_t const *path;
+    wchar_t const *desc;
+};
+
+/// A bootconfig is a manifest of potential files.
+class bootconfig
+{
+public:
+    using descriptors = util::counted<descriptor>;
+
+    /// Constructor. Loads bootconfig from the given buffer.
+    bootconfig(util::buffer data, uefi::boot_services *bs);
+
+    inline uint16_t flags() const { return m_flags; }
+    inline const descriptor & kernel() const { return m_kernel; }
+    inline const descriptor & init() const { return m_init; }
+    descriptors programs() { return m_programs; }
+    descriptors data() { return m_data; }
+
+private:
+    uint16_t m_flags;
+    descriptor m_kernel;
+    descriptor m_init;
+    descriptors m_programs;
+    descriptors m_data;
+};
+
+} // namespace boot

src/boot/console.cpp

@@ -1,9 +1,8 @@
 #include <stddef.h>
 #include <stdint.h>
 
+#include <uefi/protos/simple_text_output.h>
 #include <uefi/types.h>
-#include <uefi/graphics.h>
-#include <uefi/protos/graphics_output.h>
 
 #include "console.h"
 #include "error.h"
@@ -20,285 +19,195 @@ size_t COLS = 0;
 console *console::s_console = nullptr;
 
 static const wchar_t digits[] = {u'0', u'1', u'2', u'3', u'4', u'5',
-	u'6', u'7', u'8', u'9', u'a', u'b', u'c', u'd', u'e', u'f'};
+    u'6', u'7', u'8', u'9', u'a', u'b', u'c', u'd', u'e', u'f'};
 
 
-static size_t
+size_t
 wstrlen(const wchar_t *s)
 {
-	size_t count = 0;
+    size_t count = 0;
-	while (s && *s++) count++;
+    while (s && *s++) count++;
-	return count;
+    return count;
 }
 
 
-console::console(uefi::boot_services *bs, uefi::protos::simple_text_output *out) :
+console::console(uefi::protos::simple_text_output *out) :
-	m_rows {0},
+    m_rows {0},
-	m_cols {0},
+    m_cols {0},
-	m_out {out},
+    m_out {out}
-	m_fb {0, 0}
 {
-	pick_mode(bs);
+    try_or_raise(
-
+        m_out->query_mode(m_out->mode->mode, &m_cols, &m_rows),
-	try_or_raise(
+        L"Failed to get text output mode.");
-		m_out->query_mode(m_out->mode->mode, &m_cols, &m_rows),
+    try_or_raise(
-		L"Failed to get text output mode.");
+        m_out->clear_screen(),
-
+        L"Failed to clear screen");
-	try_or_raise(
+    s_console = this;
-		m_out->clear_screen(),
-		L"Failed to clear screen");
-
-	m_out->set_attribute(uefi::attribute::light_cyan);
-	m_out->output_string(L"jsix loader ");
-
-	m_out->set_attribute(uefi::attribute::light_magenta);
-	m_out->output_string(GIT_VERSION_WIDE);
-
-	m_out->set_attribute(uefi::attribute::light_gray);
-	m_out->output_string(L" booting...\r\n");
-
-	if (m_fb.type != kernel::args::fb_type::none) {
-		wchar_t const * type = nullptr;
-		switch (m_fb.type) {
-		case kernel::args::fb_type::rgb8:
-			type = L"rgb8";
-			break;
-		case kernel::args::fb_type::bgr8:
-			type = L"bgr8";
-			break;
-		default:
-			type = L"unknown";
-		}
-
-		printf(L"Found framebuffer: %dx%d[%d] type %s @0x%x\r\n",
-			m_fb.horizontal, m_fb.vertical, m_fb.scanline, type, m_fb.phys_addr);
-	} else {
-		printf(L"No framebuffer found.\r\n");
-	}
-
-	s_console = this;
 }
 
 void
-console::pick_mode(uefi::boot_services *bs)
+console::announce()
 {
-	uefi::status status;
+    m_out->set_attribute(uefi::attribute::light_cyan);
-	uefi::protos::graphics_output *gfx_out_proto;
+    m_out->output_string(L"jsix loader ");
-	uefi::guid guid = uefi::protos::graphics_output::guid;
 
-	m_fb.type = kernel::args::fb_type::none;
+    m_out->set_attribute(uefi::attribute::light_magenta);
+    m_out->output_string(GIT_VERSION_WIDE);
 
-	uefi::status has_gop = bs->locate_protocol(&guid, nullptr,
+    m_out->set_attribute(uefi::attribute::light_gray);
-		(void **)&gfx_out_proto);
+    m_out->output_string(L" booting...\r\n");
-
-	if (has_gop != uefi::status::success)
-		// No video output found, skip it
-		return;
-
-	const uint32_t modes = gfx_out_proto->mode->max_mode;
-	uint32_t best = gfx_out_proto->mode->mode;
-
-	uefi::graphics_output_mode_info *info =
-		(uefi::graphics_output_mode_info *)gfx_out_proto->mode;
-
-	uint32_t res = info->horizontal_resolution * info->vertical_resolution;
-	int pixmode = static_cast<int>(info->pixel_format);
-
-	for (uint32_t i = 0; i < modes; ++i) {
-		size_t size = 0;
-
-		try_or_raise(
-			gfx_out_proto->query_mode(i, &size, &info),
-			L"Failed to find a graphics mode the driver claimed to support");
-
-#ifdef MAX_HRES
-		if (info->horizontal_resolution > MAX_HRES) continue;
-#endif
-
-		const uint32_t new_res = info->horizontal_resolution * info->vertical_resolution;
-		int new_pixmode = static_cast<int>(info->pixel_format);
-
-		if (new_pixmode <= pixmode && new_res >= res) {
-			best = i;
-			res = new_res;
-			pixmode = new_pixmode;
-		}
-	}
-
-	try_or_raise(
-		gfx_out_proto->set_mode(best),
-		L"Failed to set graphics mode");
-
-	if (pixmode <= static_cast<int>(uefi::pixel_format::bgr8)) {
-		m_fb.phys_addr = gfx_out_proto->mode->frame_buffer_base;
-		m_fb.size = gfx_out_proto->mode->frame_buffer_size;
-		m_fb.vertical = gfx_out_proto->mode->info->vertical_resolution;
-		m_fb.horizontal = gfx_out_proto->mode->info->horizontal_resolution;
-		m_fb.scanline = gfx_out_proto->mode->info->pixels_per_scanline;
-
-		switch (gfx_out_proto->mode->info->pixel_format) {
-		case uefi::pixel_format::rgb8:
-			m_fb.type = kernel::args::fb_type::rgb8;
-			break;
-		case uefi::pixel_format::bgr8:
-			m_fb.type = kernel::args::fb_type::bgr8;
-			break;
-		default:
-			m_fb.type = kernel::args::fb_type::none;
-		}
-	}
 }
 
 size_t
 console::print_hex(uint32_t n) const
 {
-	wchar_t buffer[9];
+    wchar_t buffer[9];
-	wchar_t *p = buffer;
+    wchar_t *p = buffer;
-	for (int i = 7; i >= 0; --i) {
+    for (int i = 7; i >= 0; --i) {
-		uint8_t nibble = (n >> (i*4)) & 0xf;
+        uint8_t nibble = (n >> (i*4)) & 0xf;
-		*p++ = digits[nibble];
+        *p++ = digits[nibble];
-	}
+    }
-	*p = 0;
+    *p = 0;
-	m_out->output_string(buffer);
+    m_out->output_string(buffer);
-	return 8;
+    return 8;
 }
 
 size_t
 console::print_long_hex(uint64_t n) const
 {
-	wchar_t buffer[17];
+    wchar_t buffer[17];
-	wchar_t *p = buffer;
+    wchar_t *p = buffer;
-	for (int i = 15; i >= 0; --i) {
+    for (int i = 15; i >= 0; --i) {
-		uint8_t nibble = (n >> (i*4)) & 0xf;
+        uint8_t nibble = (n >> (i*4)) & 0xf;
-		*p++ = digits[nibble];
+        *p++ = digits[nibble];
-	}
+    }
-	*p = 0;
+    *p = 0;
-	m_out->output_string(buffer);
+    m_out->output_string(buffer);
-	return 16;
+    return 16;
 }
 
 size_t
 console::print_dec(uint32_t n) const
 {
-	wchar_t buffer[11];
+    wchar_t buffer[11];
-	wchar_t *p = buffer + 10;
+    wchar_t *p = buffer + 10;
-	*p-- = 0;
+    *p-- = 0;
-	do {
+    do {
-		*p-- = digits[n % 10];
+        *p-- = digits[n % 10];
-		n /= 10;
+        n /= 10;
-	} while (n != 0);
+    } while (n != 0);
 
-	m_out->output_string(++p);
+    m_out->output_string(++p);
-	return 10 - (p - buffer);
+    return 10 - (p - buffer);
 }
 
 size_t
 console::print_long_dec(uint64_t n) const
 {
-	wchar_t buffer[21];
+    wchar_t buffer[21];
-	wchar_t *p = buffer + 20;
+    wchar_t *p = buffer + 20;
-	*p-- = 0;
+    *p-- = 0;
-	do {
+    do {
-		*p-- = digits[n % 10];
+        *p-- = digits[n % 10];
-		n /= 10;
+        n /= 10;
-	} while (n != 0);
+    } while (n != 0);
 
-	m_out->output_string(++p);
+    m_out->output_string(++p);
-	return 20 - (p - buffer);
+    return 20 - (p - buffer);
 }
 
 size_t
 console::vprintf(const wchar_t *fmt, va_list args) const
 {
-	wchar_t buffer[256];
+    wchar_t buffer[256];
-	const wchar_t *r = fmt;
+    const wchar_t *r = fmt;
-	wchar_t *w = buffer;
+    wchar_t *w = buffer;
-	size_t count = 0;
+    size_t count = 0;
 
-	while (r && *r) {
+    while (r && *r) {
-		if (*r != L'%') {
+        if (*r != L'%') {
-			count++;
+            count++;
-			*w++ = *r++;
+            *w++ = *r++;
-			continue;
+            continue;
-		}
+        }
 
-		*w = 0;
+        *w = 0;
-		m_out->output_string(buffer);
+        m_out->output_string(buffer);
-		w = buffer;
+        w = buffer;
 
-		r++; // chomp the %
+        r++; // chomp the %
 
-		switch (*r++) {
+        switch (*r++) {
-			case L'%':
+            case L'%':
-				m_out->output_string(const_cast<wchar_t*>(L"%"));
+                m_out->output_string(const_cast<wchar_t*>(L"%"));
-				count++;
+                count++;
-				break;
+                break;
 
-			case L'x':
+            case L'x':
-				count += print_hex(va_arg(args, uint32_t));
+                count += print_hex(va_arg(args, uint32_t));
-				break;
+                break;
 
-			case L'd':
+            case L'd':
-			case L'u':
+            case L'u':
-				count += print_dec(va_arg(args, uint32_t));
+                count += print_dec(va_arg(args, uint32_t));
-				break;
+                break;
 
-			case L's':
+            case L's':
-				{
+                {
-					wchar_t *s = va_arg(args, wchar_t*);
+                    wchar_t *s = va_arg(args, wchar_t*);
-					count += wstrlen(s);
+                    count += wstrlen(s);
-					m_out->output_string(s);
+                    m_out->output_string(s);
-				}
+                }
-				break;
+                break;
 
-			case L'l':
+            case L'l':
-				switch (*r++) {
+                switch (*r++) {
-					case L'x':
+                    case L'x':
-						count += print_long_hex(va_arg(args, uint64_t));
+                        count += print_long_hex(va_arg(args, uint64_t));
-						break;
+                        break;
 
-					case L'd':
+                    case L'd':
-					case L'u':
+                    case L'u':
-						count += print_long_dec(va_arg(args, uint64_t));
+                        count += print_long_dec(va_arg(args, uint64_t));
-						break;
+                        break;
 
-					default:
+                    default:
-						break;
+                        break;
-				}
+                }
-				break;
+                break;
 
-			default:
+            default:
-				break;
+                break;
-		}
+        }
-	}
+    }
 
-	*w = 0;
+    *w = 0;
-	m_out->output_string(buffer);
+    m_out->output_string(buffer);
-	return count;
+    return count;
 }
 
 size_t
 console::printf(const wchar_t *fmt, ...) const
 {
-	va_list args;
+    va_list args;
-	va_start(args, fmt);
+    va_start(args, fmt);
 
-	size_t result = vprintf(fmt, args);
+    size_t result = vprintf(fmt, args);
 
-	va_end(args);
+    va_end(args);
-	return result;
+    return result;
 }
 
 size_t
 console::print(const wchar_t *fmt, ...)
 {
-	va_list args;
+    va_list args;
-	va_start(args, fmt);
+    va_start(args, fmt);
 
-	size_t result = get().vprintf(fmt, args);
+    size_t result = get().vprintf(fmt, args);
 
-	va_end(args);
+    va_end(args);
-	return result;
+    return result;
 }
 
 } // namespace boot

src/boot/console.h

@@ -1,12 +1,15 @@
+#pragma once
 /// \file console.h
 /// Text output handler
-#pragma once
+
 #include <stdarg.h>
 #include <stddef.h>
-#include <uefi/boot_services.h>
+#include <stdint.h>
-#include <uefi/protos/simple_text_output.h>
+
-#include "kernel_args.h"
+namespace uefi {
-#include "types.h"
+namespace protos {
+    struct simple_text_output;
+}}
 
 namespace boot {
 
@@ -14,32 +17,30 @@ namespace boot {
 class console
 {
 public:
-	using framebuffer = kernel::args::framebuffer;
+    console(uefi::protos::simple_text_output *out);
 
-	console(uefi::boot_services *bs, uefi::protos::simple_text_output *out);
+    void announce();
 
-	size_t print_hex(uint32_t n) const;
+    size_t print_hex(uint32_t n) const;
-	size_t print_dec(uint32_t n) const;
+    size_t print_dec(uint32_t n) const;
-	size_t print_long_hex(uint64_t n) const;
+    size_t print_long_hex(uint64_t n) const;
-	size_t print_long_dec(uint64_t n) const;
+    size_t print_long_dec(uint64_t n) const;
-	size_t printf(const wchar_t *fmt, ...) const;
+    size_t printf(const wchar_t *fmt, ...) const;
 
-	const framebuffer & fb() const { return m_fb; };
+    static console & get() { return *s_console; }
-
+    static size_t print(const wchar_t *fmt, ...);
-	static console & get() { return *s_console; }
-	static size_t print(const wchar_t *fmt, ...);
 
 private:
-	friend class status_line;
+    friend class status_line;
 
-	void pick_mode(uefi::boot_services *bs);
+    size_t vprintf(const wchar_t *fmt, va_list args) const;
-	size_t vprintf(const wchar_t *fmt, va_list args) const;
 
-	size_t m_rows, m_cols;
+    size_t m_rows, m_cols;
-	uefi::protos::simple_text_output *m_out;
+    uefi::protos::simple_text_output *m_out;
-	framebuffer m_fb;
 
-	static console *s_console;
+    static console *s_console;
 };
 
+size_t wstrlen(const wchar_t *s);
+
 } // namespace boot

src/boot/elf.h

@@ -1,83 +0,0 @@
-/// \file elf.h
-/// Definitions and related constants for ELF64 structures
-#pragma once
-#include <stdint.h>
-
-namespace boot {
-namespace elf {
-
-constexpr uint8_t version = 1;
-constexpr uint8_t word_size = 2;
-constexpr uint8_t endianness = 1;
-constexpr uint8_t os_abi = 0;
-constexpr uint16_t machine = 0x3e;
-
-const unsigned PT_LOAD = 1;
-const unsigned ST_PROGBITS = 1;
-const unsigned ST_NOBITS = 8;
-const unsigned long SHF_ALLOC = 0x2;
-
-struct header
-{
-	char magic[4];
-
-	uint8_t word_size;
-	uint8_t endianness;
-	uint8_t header_version;
-	uint8_t os_abi;
-
-	uint64_t reserved;
-
-	uint16_t type;
-	uint16_t machine;
-
-	uint32_t version;
-
-	uint64_t entrypoint;
-	uint64_t ph_offset;
-	uint64_t sh_offset;
-
-	uint32_t flags;
-
-	uint16_t eh_size;
-
-	uint16_t ph_entsize;
-	uint16_t ph_num;
-
-	uint16_t sh_entsize;
-	uint16_t sh_num;
-
-	uint16_t sh_str_idx;
-} __attribute__ ((packed));
-
-struct program_header
-{
-	uint32_t type;
-	uint32_t flags;
-	uint64_t offset;
-
-	uint64_t vaddr;
-	uint64_t paddr;
-
-	uint64_t file_size;
-	uint64_t mem_size;
-
-	uint64_t align;
-} __attribute__ ((packed));
-
-struct section_header
-{
-	uint32_t name;
-	uint32_t type;
-	uint64_t flags;
-	uint64_t addr;
-	uint64_t offset;
-	uint64_t size;
-	uint32_t link;
-	uint32_t info;
-	uint64_t align;
-	uint64_t entry_size;
-} __attribute__ ((packed));
-
-} // namespace elf
-} // namespace boot

src/boot/error.cpp

@@ -1,14 +1,13 @@
 #include "error.h"
 #include "console.h"
-#include "kernel_args.h"
 #include "status.h"
 
 namespace boot {
 namespace error {
 
 struct error_code_desc {
-	uefi::status code;
+    uefi::status code;
-	const wchar_t *name;
+    const wchar_t *name;
 };
 
 struct error_code_desc error_table[] = {
@@ -17,46 +16,46 @@ struct error_code_desc error_table[] = {
 #include "uefi/errors.inc"
 #undef STATUS_ERROR
 #undef STATUS_WARNING
-	{ uefi::status::success, nullptr }
+    { uefi::status::success, nullptr }
 };
 
 const wchar_t *
 message(uefi::status status)
 {
-	int32_t i = -1;
+    int32_t i = -1;
-	while (error_table[++i].name != nullptr) {
+    while (error_table[++i].name != nullptr) {
-		if (error_table[i].code == status) return error_table[i].name;
+        if (error_table[i].code == status) return error_table[i].name;
-	}
+    }
 
-	if (uefi::is_error(status))
+    if (uefi::is_error(status))
-		return L"Unknown Error";
+        return L"Unknown Error";
-	else
+    else
-		return L"Unknown Warning";
+        return L"Unknown Warning";
 }
 
 [[ noreturn ]] static void
 cpu_assert(uefi::status s, const wchar_t *message, size_t line)
 {
-	asm volatile (
+    asm volatile (
-		"movq $0xeeeeeeebadbadbad, %%r8;"
+        "movq $0xeeeeeeebadbadbad, %%r8;"
-		"movq %0, %%r9;"
+        "movq %0, %%r9;"
-		"movq %1, %%r10;"
+        "movq %1, %%r10;"
-		"movq %2, %%r11;"
+        "movq %2, %%r11;"
-		"movq $0, %%rdx;"
+        "movq $0, %%rdx;"
-		"divq %%rdx;"
+        "divq %%rdx;"
-		:
+        :
-		: "r"((uint64_t)s), "r"(message), "r"(line)
+        : "r"((uint64_t)s), "r"(message), "r"(line)
-		: "rax", "rdx", "r8", "r9", "r10");
+        : "rax", "rdx", "r8", "r9", "r10");
-	while (1) asm("hlt");
+    while (1) asm("hlt");
 }
 
 [[ noreturn ]] void
 raise(uefi::status status, const wchar_t *message, size_t line)
 {
-	if(status_line::fail(message, status))
+    if(status_line::fail(message, status))
-		while (1) asm("hlt");
+        while (1) asm("hlt");
-	else
+    else
-		cpu_assert(status, message, line);
+        cpu_assert(status, message, line);
 }
 
 
@@ -65,6 +64,6 @@ raise(uefi::status status, const wchar_t *message, size_t line)
 
 void debug_break()
 {
-	volatile int go = 0;
+    volatile int go = 0;
-	while (!go);
+    while (!go);
 }

src/boot/error.h

@@ -26,8 +26,8 @@ void debug_break();
 /// \arg s  An expression evaluating to a UEFI status
 /// \arg m  The error message to use on failure
 #define try_or_raise(s, m) \
-	do { \
+    do { \
-		uefi::status _s = (s); \
+        uefi::status _s = (s); \
-		if (uefi::is_error(_s)) ::boot::error::raise(_s, (m), __LINE__); \
+        if (uefi::is_error(_s)) ::boot::error::raise(_s, (m), __LINE__); \
-	} while(0)
+    } while(0)
 

src/boot/fs.cpp

@@ -1,112 +1,110 @@
+#include <uefi/boot_services.h>
 #include <uefi/types.h>
 #include <uefi/protos/file.h>
 #include <uefi/protos/file_info.h>
 #include <uefi/protos/loaded_image.h>
 #include <uefi/protos/simple_file_system.h>
 
-#include "fs.h"
+#include <bootproto/kernel.h>
+
+#include "allocator.h"
 #include "console.h"
 #include "error.h"
+#include "fs.h"
 #include "memory.h"
 #include "status.h"
 
 namespace boot {
 namespace fs {
 
-file::file(uefi::protos::file *f, uefi::boot_services *bs) :
+using memory::alloc_type;
-	m_file(f),
+
-	m_bs(bs)
+file::file(uefi::protos::file *f) :
+    m_file(f)
 {
 }
 
 file::file(file &o) :
-	m_file(o.m_file),
+    m_file(o.m_file)
-	m_bs(o.m_bs)
 {
-	o.m_file = nullptr;
+    o.m_file = nullptr;
 }
 
 file::file(file &&o) :
-	m_file(o.m_file),
+    m_file(o.m_file)
-	m_bs(o.m_bs)
 {
-	o.m_file = nullptr;
+    o.m_file = nullptr;
 }
 
 file::~file()
 {
-	if (m_file)
+    if (m_file)
-		m_file->close();
+        m_file->close();
 }
 
 file
 file::open(const wchar_t *path)
 {
-	uefi::protos::file *fh = nullptr;
+    uefi::protos::file *fh = nullptr;
 
-	try_or_raise(
+    try_or_raise(
-		m_file->open(&fh, path, uefi::file_mode::read, uefi::file_attr::none),
+        m_file->open(&fh, path, uefi::file_mode::read, uefi::file_attr::none),
-		L"Could not open relative path to file");
+        L"Could not open relative path to file");
 
-	return file(fh, m_bs);
+    return file(fh);
 }
 
-buffer
+util::buffer
-file::load(uefi::memory_type mem_type)
+file::load()
 {
-	uint8_t info_buf[sizeof(uefi::protos::file_info) + 100];
+    uint8_t info_buf[sizeof(uefi::protos::file_info) + 100];
-	size_t size = sizeof(info_buf);
+    size_t size = sizeof(info_buf);
-	uefi::guid info_guid = uefi::protos::file_info::guid;
+    uefi::guid info_guid = uefi::protos::file_info::guid;
 
-	try_or_raise(
+    try_or_raise(
-		m_file->get_info(&info_guid, &size, &info_buf),
+        m_file->get_info(&info_guid, &size, &info_buf),
-		L"Could not get file info");
+        L"Could not get file info");
 
-	uefi::protos::file_info *info =
+    uefi::protos::file_info *info =
-		reinterpret_cast<uefi::protos::file_info*>(&info_buf);
+        reinterpret_cast<uefi::protos::file_info*>(&info_buf);
 
-	size_t pages = memory::bytes_to_pages(info->file_size);
+    size_t pages = memory::bytes_to_pages(info->file_size);
-	void *data = nullptr;
+    void *data = g_alloc.allocate_pages(pages, alloc_type::file);
-	try_or_raise(
-		m_bs->allocate_pages(
-			uefi::allocate_type::any_pages,
-			mem_type, pages, &data),
-		L"Could not allocate pages to load file");
 
-	size = info->file_size;
+    size = info->file_size;
-	try_or_raise(
+    try_or_raise(
-		m_file->read(&size, data),
+        m_file->read(&size, data),
-		L"Could not read from file");
+        L"Could not read from file");
 
-	return { .size = size,  .data = data };
+    return { .pointer = data, .count = size };
 }
 
 file
 get_boot_volume(uefi::handle image, uefi::boot_services *bs)
 {
-	status_line status(L"Looking up boot volume");
+    status_line status(L"Looking up boot volume");
 
-	const uefi::guid le_guid = uefi::protos::loaded_image::guid;
+    const uefi::guid le_guid = uefi::protos::loaded_image::guid;
-	uefi::protos::loaded_image *loaded_image = nullptr;
+    uefi::protos::loaded_image *loaded_image = nullptr;
 
-	try_or_raise(
+    try_or_raise(
-		bs->handle_protocol(image, &le_guid,
+        bs->handle_protocol(image, &le_guid,
-			reinterpret_cast<void**>(&loaded_image)),
+            reinterpret_cast<void**>(&loaded_image)),
-		L"Could not find currently running UEFI loaded image");
+        L"Could not find currently running UEFI loaded image");
 
-	const uefi::guid sfs_guid = uefi::protos::simple_file_system::guid;
+    const uefi::guid sfs_guid = uefi::protos::simple_file_system::guid;
-	uefi::protos::simple_file_system *fs;
+    uefi::protos::simple_file_system *fs;
-	try_or_raise(
+    try_or_raise(
-		bs->handle_protocol(loaded_image->device_handle, &sfs_guid,
+        bs->handle_protocol(loaded_image->device_handle, &sfs_guid,
-			reinterpret_cast<void**>(&fs)),
+            reinterpret_cast<void**>(&fs)),
-		L"Could not find filesystem protocol for boot volume");
+        L"Could not find filesystem protocol for boot volume");
 
-	uefi::protos::file *f;
+    uefi::protos::file *f;
-	try_or_raise(
+    try_or_raise(
-		fs->open_volume(&f),
+        fs->open_volume(&f),
-		L"Could not open the boot volume");
+        L"Could not open the boot volume");
 
-	return file(f, bs);
+    return file(f);
 }
 
 } // namespace fs

src/boot/fs.h

@@ -3,9 +3,13 @@
 #pragma once
 
 #include <uefi/types.h>
-#include <uefi/boot_services.h>
+#include <util/counted.h>
-#include <uefi/protos/file.h>
+
-#include "types.h"
+namespace uefi {
+    struct boot_services;
+namespace protos {
+    struct file;
+}}
 
 namespace boot {
 namespace fs {
@@ -14,27 +18,26 @@ namespace fs {
 class file
 {
 public:
-	file(file &&o);
+    file(file &&o);
-	file(file &o);
+    file(file &o);
-	~file();
+    ~file();
 
-	/// Open another file or directory, relative to this one.
+    /// Open another file or directory, relative to this one.
-	/// \arg path  Relative path to the target file from this one
+    /// \arg path  Relative path to the target file from this one
-	file open(const wchar_t *path);
+    file open(const wchar_t *path);
 
-	/// Load the contents of this file into memory.
+    /// Load the contents of this file into memory.
-	/// \arg mem_type  The UEFI memory type to use for allocation
+    /// \returns       A buffer describing the loaded memory. The
-	/// \returns       A buffer describing the loaded memory. The
+    ///                memory will be page-aligned.
-	///                memory will be page-aligned.
+    util::buffer load();
-	buffer load(uefi::memory_type mem_type = uefi::memory_type::loader_data);
 
 private:
-	friend file get_boot_volume(uefi::handle, uefi::boot_services*);
+    friend file get_boot_volume(uefi::handle, uefi::boot_services*);
 
-	file(uefi::protos::file *f, uefi::boot_services *bs);
+    file(uefi::protos::file *f);
 
-	uefi::protos::file *m_file;
+    uefi::protos::file *m_file;
-	uefi::boot_services *m_bs;
+    uefi::boot_services *m_bs;
 };
 
 /// Get the filesystem this loader was loaded from.

src/boot/hardware.cpp

@@ -1,6 +1,7 @@
-#include "hardware.h"
 #include "console.h"
+#include "cpu/cpu_id.h"
 #include "error.h"
+#include "hardware.h"
 #include "status.h"
 
 namespace boot {
@@ -9,73 +10,94 @@ namespace hw {
 void *
 find_acpi_table(uefi::system_table *st)
 {
-	status_line status(L"Searching for ACPI table");
+    status_line status(L"Searching for ACPI table");
 
-	// Find ACPI tables. Ignore ACPI 1.0 if a 2.0 table is found.
+    // Find ACPI tables. Ignore ACPI 1.0 if a 2.0 table is found.
-	uintptr_t acpi1_table = 0;
+    uintptr_t acpi1_table = 0;
 
-	for (size_t i = 0; i < st->number_of_table_entries; ++i) {
+    for (size_t i = 0; i < st->number_of_table_entries; ++i) {
-		uefi::configuration_table *table = &st->configuration_table[i];
+        uefi::configuration_table *table = &st->configuration_table[i];
 
-		// If we find an ACPI 2.0 table, return it immediately
+        // If we find an ACPI 2.0 table, return it immediately
-		if (table->vendor_guid == uefi::vendor_guids::acpi2)
+        if (table->vendor_guid == uefi::vendor_guids::acpi2)
-			return table->vendor_table;
+            return table->vendor_table;
 
-		if (table->vendor_guid == uefi::vendor_guids::acpi1) {
+        if (table->vendor_guid == uefi::vendor_guids::acpi1) {
-			// Mark a v1 table with the LSB high
+            // Mark a v1 table with the LSB high
-			acpi1_table = reinterpret_cast<uintptr_t>(table->vendor_table);
+            acpi1_table = reinterpret_cast<uintptr_t>(table->vendor_table);
-			acpi1_table |= 1;
+            acpi1_table |= 1;
-		}
+        }
-	}
+    }
 
-	if (!acpi1_table) {
+    if (!acpi1_table) {
-		error::raise(uefi::status::not_found, L"Could not find ACPI table");
+        error::raise(uefi::status::not_found, L"Could not find ACPI table");
-	} else if (acpi1_table & 1) {
+    } else if (acpi1_table & 1) {
-		status_line::warn(L"Only found ACPI 1.0 table");
+        status_line::warn(L"Only found ACPI 1.0 table");
-	}
+    }
 
-	return reinterpret_cast<void*>(acpi1_table);
+    return reinterpret_cast<void*>(acpi1_table);
 }
 
 static uint64_t
 rdmsr(uint32_t addr)
 {
-	uint32_t low, high;
+    uint32_t low, high;
-	__asm__ __volatile__ ("rdmsr" : "=a"(low), "=d"(high) : "c"(addr));
+    __asm__ __volatile__ ("rdmsr" : "=a"(low), "=d"(high) : "c"(addr));
-	return (static_cast<uint64_t>(high) << 32) | low;
+    return (static_cast<uint64_t>(high) << 32) | low;
 }
 
 static void
 wrmsr(uint32_t addr, uint64_t value)
 {
-	uint32_t low = value & 0xffffffff;
+    uint32_t low = value & 0xffffffff;
-	uint32_t high = value >> 32;
+    uint32_t high = value >> 32;
-	__asm__ __volatile__ ("wrmsr" :: "c"(addr), "a"(low), "d"(high));
+    __asm__ __volatile__ ("wrmsr" :: "c"(addr), "a"(low), "d"(high));
 }
 
-
 void
 setup_control_regs()
 {
-	uint64_t cr4 = 0;
+    uint64_t cr4 = 0;
-	asm volatile ( "mov %%cr4, %0" : "=r" (cr4) );
+    asm volatile ( "mov %%cr4, %0" : "=r" (cr4) );
-	cr4 |=
+    cr4 |=
-		0x000080 | // Enable global pages
+        0x000080 | // Enable global pages
-		0x000200 | // Enable FXSAVE/FXRSTOR
+        0x000200 | // Enable FXSAVE/FXRSTOR
-		0x010000 | // Enable FSGSBASE
+        0x010000 | // Enable FSGSBASE
-		0x020000 | // Enable PCIDs
+        0x020000 | // Enable PCIDs
-		0;
+        0;
-	asm volatile ( "mov %0, %%cr4" :: "r" (cr4) );
+    asm volatile ( "mov %0, %%cr4" :: "r" (cr4) );
 
-	// Set up IA32_EFER
+    // Set up IA32_EFER
-	constexpr uint32_t IA32_EFER = 0xC0000080;
+    constexpr uint32_t IA32_EFER = 0xC0000080;
-	uint64_t efer = rdmsr(IA32_EFER);
+    uint64_t efer = rdmsr(IA32_EFER);
-	efer |=
+    efer |=
-		0x0001 | // Enable SYSCALL
+        0x0001 | // Enable SYSCALL
-		0x0800 | // Enable NX bit
+        0x0800 | // Enable NX bit
-		0;
+        0;
-	wrmsr(IA32_EFER, efer);
+    wrmsr(IA32_EFER, efer);
 }
 
+void
+check_cpu_supported()
+{
+    status_line status {L"Checking CPU features"};
+
+    cpu::cpu_id cpu;
+    uint64_t missing = cpu.missing();
+    if (missing) {
+#define CPU_FEATURE_OPT(...)
+#define CPU_FEATURE_REQ(name, ...) \
+        if (!cpu.has_feature(cpu::feature::name)) { \
+            status::fail(L"CPU required feature " L ## #name, uefi::status::unsupported); \
+        }
+#include "cpu/features.inc"
+#undef CPU_FEATURE_REQ
+#undef CPU_FEATURE_OPT
+
+        error::raise(uefi::status::unsupported, L"CPU not supported");
+    }
+}
+
+
 } // namespace hw
 } // namespace boot

src/boot/hardware.h

@@ -16,5 +16,8 @@ void * find_acpi_table(uefi::system_table *st);
 /// Enable CPU options in CR4 etc for the kernel starting state.
 void setup_control_regs();
 
+/// Check that all required cpu features are supported
+void check_cpu_supported();
+
 } // namespace hw
 } // namespace boot

src/boot/loader.cpp

@@ -1,116 +1,155 @@
 #include <uefi/boot_services.h>
 #include <uefi/types.h>
 
-#include "loader.h"
+#include <bootproto/init.h>
+#include <elf/file.h>
+#include <elf/headers.h>
+#include <util/pointers.h>
+
+#include "allocator.h"
+#include "bootconfig.h"
 #include "console.h"
-#include "elf.h"
 #include "error.h"
 #include "fs.h"
+#include "loader.h"
 #include "memory.h"
 #include "paging.h"
 #include "status.h"
 
-namespace args = kernel::args;
-
 namespace boot {
 namespace loader {
 
-buffer
+using memory::alloc_type;
+
+util::buffer
 load_file(
-	fs::file &disk,
+    fs::file &disk,
-	const wchar_t *name,
+    const descriptor &desc)
-	const wchar_t *path,
-	uefi::memory_type type)
 {
-	status_line status(L"Loading file", name);
+    status_line status(L"Loading file", desc.path);
 
-	fs::file file = disk.open(path);
+    fs::file file = disk.open(desc.path);
-	buffer b = file.load(type);
+    util::buffer b = file.load();
 
-	//console::print(L"    Loaded at: 0x%lx, %d bytes\r\n", b.data, b.size);
+    //console::print(L"    Loaded at: 0x%lx, %d bytes\r\n", b.data, b.size);
-	return b;
+    return b;
 }
 
 
-static bool
+static void
-is_elfheader_valid(const elf::header *header)
+create_module(util::buffer data, const descriptor &desc, bool loaded)
 {
-	return
+    size_t path_len = wstrlen(desc.path);
-		header->magic[0] == 0x7f &&
+    bootproto::module_program *mod = g_alloc.allocate_module<bootproto::module_program>(path_len);
-		header->magic[1] == 'E' &&
+    mod->mod_type = bootproto::module_type::program;
-		header->magic[2] == 'L' &&
+    mod->base_address = reinterpret_cast<uintptr_t>(data.pointer);
-		header->magic[3] == 'F' &&
+    mod->size = data.count;
-		header->word_size == elf::word_size &&
+    if (loaded)
-		header->endianness == elf::endianness &&
+        mod->mod_flags = static_cast<bootproto::module_flags>(
-		header->os_abi == elf::os_abi &&
+            static_cast<uint8_t>(mod->mod_flags) |
-		header->machine == elf::machine &&
+            static_cast<uint8_t>(bootproto::module_flags::no_load));
-		header->header_version == elf::version;
+
+    // TODO: support non-ascii path characters and do real utf-16 to utf-8
+    // conversion
+    for (int i = 0; i < path_len; ++i) {
+        char c = desc.path[i];
+        mod->filename[i] = c == '\\' ? '/' : c;
+    }
+    mod->filename[path_len] = 0;
+}
+
+bootproto::program *
+load_program(
+    fs::file &disk,
+    const descriptor &desc,
+    bool add_module)
+{
+    status_line status(L"Loading program", desc.desc);
+
+    util::buffer data = load_file(disk, desc);
+
+    if (add_module)
+        create_module(data, desc, true);
+
+    elf::file program(data.pointer, data.count);
+    if (!program.valid())
+        error::raise(uefi::status::load_error, L"ELF file not valid");
+
+    size_t num_sections = 0;
+    for (auto &seg : program.programs()) {
+        if (seg.type == elf::segment_type::load)
+            ++num_sections;
+    }
+
+    bootproto::program_section *sections = new bootproto::program_section [num_sections];
+
+    size_t next_section = 0;
+    for (auto &seg : program.programs()) {
+        if (seg.type != elf::segment_type::load)
+            continue;
+
+        bootproto::program_section &section = sections[next_section++];
+
+        size_t page_count = memory::bytes_to_pages(seg.mem_size);
+
+        if (seg.mem_size > seg.file_size) {
+            void *pages = g_alloc.allocate_pages(page_count, alloc_type::program, true);
+            void *source = util::offset_pointer(data.pointer, seg.offset);
+            g_alloc.copy(pages, source, seg.file_size);
+            section.phys_addr = reinterpret_cast<uintptr_t>(pages);
+        } else {
+            section.phys_addr = program.base() + seg.offset;
+        }
+
+        section.virt_addr = seg.vaddr;
+        section.size = seg.mem_size;
+        section.type = static_cast<bootproto::section_flags>(seg.flags);
+    }
+
+    bootproto::program *prog = new bootproto::program;
+    prog->sections = { .pointer = sections, .count = num_sections };
+    prog->phys_base = program.base();
+    prog->entrypoint = program.entrypoint();
+    return prog;
 }
 
 void
-load_program(
+load_module(
-	args::program &program,
+    fs::file &disk,
-	const wchar_t *name,
+    const descriptor &desc)
-	buffer data,
-	uefi::boot_services *bs)
 {
-	status_line status(L"Loading program:", name);
+    status_line status(L"Loading module", desc.desc);
-	const elf::header *header = reinterpret_cast<const elf::header*>(data.data);
 
-	if (data.size < sizeof(elf::header) || !is_elfheader_valid(header))
+    util::buffer data = load_file(disk, desc);
-		error::raise(uefi::status::load_error, L"ELF file not valid");
+    create_module(data, desc, false);
+}
 
-	uintptr_t prog_base = uintptr_t(-1);
+void
-	uintptr_t prog_end = 0;
+verify_kernel_header(bootproto::program &program)
+{
+    status_line status(L"Verifying kernel header");
 
-	for (int i = 0; i < header->ph_num; ++i) {
+    const bootproto::header *header =
-		ptrdiff_t offset = header->ph_offset + i * header->ph_entsize;
+        reinterpret_cast<const bootproto::header *>(program.sections[0].phys_addr);
-		const elf::program_header *pheader =
-			offset_ptr<elf::program_header>(data.data, offset);
 
-		if (pheader->type != elf::PT_LOAD)
+    if (header->magic != bootproto::header_magic)
-			continue;
+        error::raise(uefi::status::load_error, L"Bad kernel magic number");
 
-		uintptr_t end = pheader->vaddr + pheader->mem_size;
+    if (header->length < sizeof(bootproto::header))
-		if (pheader->vaddr < prog_base) prog_base = pheader->vaddr;
+        error::raise(uefi::status::load_error, L"Bad kernel header length");
-		if (end > prog_end) prog_end = end;
-	}
 
-	size_t total_size = prog_end - prog_base;
+    if (header->version < bootproto::min_header_version)
-	size_t num_pages = memory::bytes_to_pages(total_size);
+        error::raise(uefi::status::unsupported, L"Kernel header version not supported");
-	void *pages = nullptr;
 
-	try_or_raise(
+    console::print(L"    Loaded kernel vserion: %d.%d.%d %x\r\n",
-		bs->allocate_pages(uefi::allocate_type::any_pages,
+            header->version_major, header->version_minor, header->version_patch,
-			uefi::memory_type::loader_data, num_pages, &pages),
+            header->version_gitsha);
-		L"Failed allocating space for program");
 
-	bs->set_mem(pages, total_size, 0);
+    /*
-
+    for (auto &section : program.sections)
-	program.base = prog_base;
+        console::print(L"    Section: p:0x%lx v:0x%lx fs:0x%x ms:0x%x\r\n",
-	program.total_size = total_size;
+            section.phys_addr, section.virt_addr, section.file_size, section.mem_size);
-	program.num_sections = 0;
+    */
-	for (int i = 0; i < header->ph_num; ++i) {
-		ptrdiff_t offset = header->ph_offset + i * header->ph_entsize;
-		const elf::program_header *pheader =
-			offset_ptr<elf::program_header>(data.data, offset);
-
-		if (pheader->type != elf::PT_LOAD)
-			continue;
-
-		args::program_section &section = program.sections[program.num_sections++];
-
-		void *src_start = offset_ptr<void>(data.data, pheader->offset);
-		void *dest_start = offset_ptr<void>(pages, pheader->vaddr - prog_base);
-
-		bs->copy_mem(dest_start, src_start, pheader->file_size);
-		section.phys_addr = reinterpret_cast<uintptr_t>(dest_start);
-		section.virt_addr = pheader->vaddr;
-		section.size = pheader->mem_size;
-		section.type = static_cast<args::section_flags>(pheader->flags);
-	}
-
-	program.entrypoint = header->entrypoint;
 }
 
 } // namespace loader

src/boot/loader.h

@@ -2,40 +2,53 @@
 /// Definitions for loading the kernel into memory
 #pragma once
 
-#include <uefi/boot_services.h>
+#include <util/counted.h>
 
-#include "kernel_args.h"
+namespace bootproto {
-#include "memory.h"
+    struct program;
-#include "types.h"
+    struct module;
+}
 
 namespace boot {
 
-namespace fs { class file; }
+class descriptor;
+
+namespace fs {
+    class file;
+}
 
 namespace loader {
 
 /// Load a file from disk into memory.
 /// \arg disk  The opened UEFI filesystem to load from
-/// \arg name  Name of the module (informational only)
+/// \arg desc  The program descriptor identifying the file
-/// \arg path  Path on `disk` of the file to load
+util::buffer
-/// \arg type  Memory type to use for allocation
-buffer
 load_file(
-	fs::file &disk,
+    fs::file &disk,
-	const wchar_t *name,
+    const descriptor &desc);
-	const wchar_t *path,
-	uefi::memory_type type = uefi::memory_type::loader_data);
 
 /// Parse and load an ELF file in memory into a loaded image.
-/// \arg program  The program structure to fill
+/// \arg disk       The opened UEFI filesystem to load from
-/// \arg data     Buffer of the ELF file in memory
+/// \arg desc       The descriptor identifying the program
-/// \arg bs       Boot services
+/// \arg add_module Also create a module for this loaded program
-void
+bootproto::program *
 load_program(
-	kernel::args::program &program,
+    fs::file &disk,
-	const wchar_t *name,
+    const descriptor &desc,
-	buffer data,
+    bool add_module = false);
-	uefi::boot_services *bs);
+
+/// Load a file from disk into memory, creating an init args module
+/// \arg disk  The opened UEFI filesystem to load from
+/// \arg desc  The program descriptor identifying the file
+void
+load_module(
+    fs::file &disk,
+    const descriptor &desc);
+
+/// Verify that a loaded ELF has the j6 kernel header
+/// \arg program  The program to check for a header
+void
+verify_kernel_header(bootproto::program &program);
 
 } // namespace loader
 } // namespace boot

src/boot/main.cpp

@@ -7,169 +7,140 @@
 #include <stddef.h>
 #include <stdint.h>
 
+#include <bootproto/bootconfig.h>
+#include <bootproto/kernel.h>
+#include <bootproto/memory.h>
+#include <util/counted.h>
+#include <util/pointers.h>
+
+#include "allocator.h"
+#include "bootconfig.h"
 #include "console.h"
-#include "cpu/cpu_id.h"
 #include "error.h"
 #include "fs.h"
 #include "hardware.h"
 #include "loader.h"
 #include "memory.h"
+#include "memory_map.h"
 #include "paging.h"
 #include "status.h"
+#include "video.h"
 
-#include "kernel_args.h"
-
-namespace kernel {
-#include "kernel_memory.h"
-}
-
-namespace args = kernel::args;
 
 namespace boot {
 
-constexpr int max_modules = 5; // Max modules to allocate room for
-constexpr int max_programs = 5; // Max programs to allocate room for
-
-struct program_desc
-{
-	const wchar_t *name;
-	const wchar_t *path;
-};
-
-const program_desc program_list[] = {
-	{L"kernel", L"jsix.elf"},
-	{L"null driver", L"nulldrv.elf"},
-	{L"fb driver", L"fb.elf"},
-};
-
 /// Change a pointer to point to the higher-half linear-offset version
 /// of the address it points to.
 template <typename T>
 void change_pointer(T *&pointer)
 {
-	pointer = offset_ptr<T>(pointer, kernel::memory::page_offset);
+    pointer = util::offset_pointer(pointer, bootproto::mem::linear_offset);
-}
-
-/// Allocate space for kernel args. Allocates enough space from pool
-/// memory for the args header and the module and program headers.
-args::header *
-allocate_args_structure(
-	uefi::boot_services *bs,
-	size_t max_modules,
-	size_t max_programs)
-{
-	status_line status {L"Setting up kernel args memory"};
-
-	args::header *args = nullptr;
-
-	size_t args_size =
-		sizeof(args::header) + // The header itself
-		max_modules * sizeof(args::module) +  // The module structures
-		max_programs * sizeof(args::program); // The program structures
-
-	try_or_raise(
-		bs->allocate_pool(uefi::memory_type::loader_data, args_size,
-			reinterpret_cast<void**>(&args)),
-		L"Could not allocate argument memory");
-
-	bs->set_mem(args, args_size, 0);
-
-	args->modules =
-		reinterpret_cast<args::module*>(args + 1);
-	args->num_modules = 0;
-
-	args->programs =
-		reinterpret_cast<args::program*>(args->modules + max_modules);
-	args->num_programs = 0;
-
-	return args;
-}
-
-/// Add a module to the kernel args list
-inline void
-add_module(args::header *args, args::mod_type type, buffer &data)
-{
-	args::module &m = args->modules[args->num_modules++];
-	m.type = type;
-	m.location = data.data;
-	m.size = data.size;
-
-	change_pointer(m.location);
-}
-
-/// Check that all required cpu features are supported
-void
-check_cpu_supported()
-{
-	status_line status {L"Checking CPU features"};
-
-	cpu::cpu_id cpu;
-	uint64_t missing = cpu.missing();
-	if (missing) {
-#define CPU_FEATURE_OPT(...)
-#define CPU_FEATURE_REQ(name, ...) \
-		if (!cpu.has_feature(cpu::feature::name)) { \
-			status::fail(L"CPU required feature " L ## #name, uefi::status::unsupported); \
-		}
-#include "cpu/features.inc"
-#undef CPU_FEATURE_REQ
-#undef CPU_FEATURE_OPT
-
-		error::raise(uefi::status::unsupported, L"CPU not supported");
-	}
 }
 
 /// The main procedure for the portion of the loader that runs while
 /// UEFI is still in control of the machine. (ie, while the loader still
-/// has access to boot services.
+/// has access to boot services.)
-args::header *
+bootproto::args *
 uefi_preboot(uefi::handle image, uefi::system_table *st)
 {
-	status_line status {L"Performing UEFI pre-boot"};
+    uefi::boot_services *bs = st->boot_services;
+    uefi::runtime_services *rs = st->runtime_services;
 
-	uefi::boot_services *bs = st->boot_services;
+    status_line status {L"Performing UEFI pre-boot"};
-	uefi::runtime_services *rs = st->runtime_services;
-	memory::init_pointer_fixup(bs, rs);
 
-	args::header *args =
+    hw::check_cpu_supported();
-		allocate_args_structure(bs, max_modules, max_programs);
+    memory::init_pointer_fixup(bs, rs);
 
-	args->magic = args::magic;
+    bootproto::args *args = new bootproto::args;
-	args->version = args::version;
+    g_alloc.zero(args, sizeof(bootproto::args));
-	args->runtime_services = rs;
-	args->acpi_table = hw::find_acpi_table(st);
-	paging::allocate_tables(args, bs);
 
-	memory::mark_pointer_fixup(&args->runtime_services);
+    args->magic = bootproto::args_magic;
+    args->version = bootproto::args_version;
+    args->runtime_services = rs;
+    args->acpi_table = hw::find_acpi_table(st);
+    memory::mark_pointer_fixup(&args->runtime_services);
 
-	fs::file disk = fs::get_boot_volume(image, bs);
+    paging::allocate_tables(args);
 
-	buffer symbols = loader::load_file(disk, L"symbol table", L"symbol_table.dat",
+    return args;
-				uefi::memory_type::loader_data);
+}
-	add_module(args, args::mod_type::symbol_table, symbols);
 
-	for (auto &desc : program_list) {
+/// Load the kernel and other programs from disk
-		buffer buf = loader::load_file(disk, desc.name, desc.path);
+void
-		args::program &program = args->programs[args->num_programs++];
+load_resources(bootproto::args *args, video::screen *screen, uefi::handle image, uefi::boot_services *bs)
-		loader::load_program(program, desc.name, buf, bs);
+{
-	}
+    status_line status {L"Loading programs"};
 
-	return args;
+    fs::file disk = fs::get_boot_volume(image, bs);
+    fs::file bc_data = disk.open(L"jsix_boot.dat");
+    bootconfig bc {bc_data.load(), bs};
+
+    args->kernel = loader::load_program(disk, bc.kernel(), true);
+    args->init = loader::load_program(disk, bc.init());
+    args->flags = static_cast<bootproto::boot_flags>(bc.flags());
+
+    namespace bits = util::bits;
+    using bootproto::desc_flags;
+
+    if (screen) {
+        video::make_module(screen);
+
+        // Go through the screen-specific descriptors first to
+        // give them priority
+        for (const descriptor &d : bc.programs()) {
+            if (!bits::has(d.flags, desc_flags::graphical))
+                continue;
+
+            if (bits::has(d.flags, desc_flags::panic))
+                args->panic = loader::load_program(disk, d);
+            else
+                loader::load_module(disk, d);
+        }
+    }
+
+    // Load the non-graphical descriptors
+    for (const descriptor &d : bc.programs()) {
+        if (bits::has(d.flags, desc_flags::graphical))
+            continue;
+
+        if (bits::has(d.flags, desc_flags::panic) && !args->panic)
+            args->panic = loader::load_program(disk, d);
+        else
+            loader::load_module(disk, d);
+    }
+
+    // For now the only data we load is the symbol table
+    for (const descriptor &d : bc.data()) {
+        if (!bits::has(d.flags, desc_flags::symbols))
+            continue;
+
+        util::buffer symbol_table = loader::load_file(disk, d);
+        args->symbol_table = symbol_table.pointer;
+        break;
+    }
+
+    loader::verify_kernel_header(*args->kernel);
 }
 
 memory::efi_mem_map
-uefi_exit(args::header *args, uefi::handle image, uefi::boot_services *bs)
+uefi_exit(bootproto::args *args, uefi::handle image, uefi::boot_services *bs)
 {
-	status_line status {L"Exiting UEFI", nullptr, false};
+    status_line status {L"Exiting UEFI", nullptr, false};
 
-	memory::efi_mem_map map =
+    memory::efi_mem_map map;
-		memory::build_kernel_mem_map(args, bs);
+    map.update(*bs);
 
-	try_or_raise(
+    args->mem_map = memory::build_kernel_map(map);
-		bs->exit_boot_services(image, map.key),
+    args->frame_blocks = memory::build_frame_blocks(args->mem_map);
-		L"Failed to exit boot services");
 
-	return map;
+    map.update(*bs);
+    status.do_blank();
+
+    try_or_raise(
+        bs->exit_boot_services(image, map.key),
+        L"Failed to exit boot services");
+
+    return map;
 }
 
 } // namespace boot
@@ -178,41 +149,53 @@ uefi_exit(args::header *args, uefi::handle image, uefi::boot_services *bs)
 extern "C" uefi::status
 efi_main(uefi::handle image, uefi::system_table *st)
 {
-	using namespace boot;
+    using namespace boot;
-	console con(st->boot_services, st->con_out);
-	check_cpu_supported();
 
-	args::header *args = uefi_preboot(image, st);
+    uefi::boot_services *bs = st->boot_services;
-	memory::efi_mem_map map = uefi_exit(args, image, st->boot_services);
+    console con(st->con_out);
 
-	args->video = con.fb();
+    bootproto::allocation_register *allocs = nullptr;
-	status_bar status {con.fb()}; // Switch to fb status display
+    bootproto::modules_page *modules = nullptr;
+    memory::allocator::init(allocs, modules, bs);
 
-	// Map the kernel to the appropriate address
+    video::screen *screen = video::pick_mode(bs);
-	args::program &kernel = args->programs[0];
+    con.announce();
-	for (auto &section : kernel.sections)
-		if (section.size)
-			paging::map_section(args, section);
 
-	memory::fix_frame_blocks(args);
+    bootproto::args *args = uefi_preboot(image, st);
+    load_resources(args, screen, image, bs);
+    memory::efi_mem_map map = uefi_exit(args, image, st->boot_services);
 
-	kernel::entrypoint kentry =
+    args->allocations = allocs;
-		reinterpret_cast<kernel::entrypoint>(kernel.entrypoint);
+    args->modules = reinterpret_cast<uintptr_t>(modules);
-	status.next();
 
-	hw::setup_control_regs();
+    status_bar status {screen}; // Switch to fb status display
-	memory::virtualize(args->pml4, map, st->runtime_services);
-	status.next();
 
-	change_pointer(args);
+    // Map the kernel and panic handler to the appropriate addresses
-	change_pointer(args->pml4);
+    paging::map_program(args, *args->kernel);
-	change_pointer(args->modules);
+    paging::map_program(args, *args->panic);
-	change_pointer(args->programs);
 
-	status.next();
+    memory::fix_frame_blocks(args);
 
-	kentry(args);
+    bootproto::entrypoint kentry =
-	debug_break();
+        reinterpret_cast<bootproto::entrypoint>(args->kernel->entrypoint);
-	return uefi::status::unsupported;
+    //status.next();
+
+    hw::setup_control_regs();
+    memory::virtualize(args->pml4, map, st->runtime_services);
+    //status.next();
+
+    change_pointer(args);
+    change_pointer(args->pml4);
+
+    change_pointer(args->kernel);
+    change_pointer(args->kernel->sections.pointer);
+    change_pointer(args->init);
+    change_pointer(args->init->sections.pointer);
+
+    //status.next();
+
+    kentry(args);
+    debug_break();
+    return uefi::status::unsupported;
 }
 

src/boot/memory.cpp

@@ -1,382 +1,77 @@
 #include <stddef.h>
+
+#include <uefi/boot_services.h>
+#include <uefi/runtime_services.h>
 #include <uefi/types.h>
 
-#include "kernel_memory.h"
+#include <bootproto/memory.h>
 
 #include "console.h"
 #include "error.h"
 #include "memory.h"
+#include "memory_map.h"
 #include "paging.h"
 #include "status.h"
 
 namespace boot {
 namespace memory {
 
-using mem_entry = kernel::args::mem_entry;
-using mem_type = kernel::args::mem_type;
-using frame_block = kernel::args::frame_block;
-using kernel::args::frames_per_block;
-
 size_t fixup_pointer_index = 0;
 void **fixup_pointers[64];
 
-static const wchar_t *memory_type_names[] = {
-	L"reserved memory type",
-	L"loader code",
-	L"loader data",
-	L"boot services code",
-	L"boot services data",
-	L"runtime services code",
-	L"runtime services data",
-	L"conventional memory",
-	L"unusable memory",
-	L"acpi reclaim memory",
-	L"acpi memory nvs",
-	L"memory mapped io",
-	L"memory mapped io port space",
-	L"pal code",
-	L"persistent memory"
-};
-
-static const wchar_t *kernel_memory_type_names[] = {
-	L"free",
-	L"pending",
-	L"acpi",
-	L"uefi_runtime",
-	L"mmio",
-	L"persistent"
-};
-
-static const wchar_t *
-memory_type_name(uefi::memory_type t)
-{
-	if (t < uefi::memory_type::max_memory_type)
-		return memory_type_names[static_cast<uint32_t>(t)];
-
-	return L"Bad Type Value";
-}
-
-static const wchar_t *
-kernel_memory_type_name(kernel::args::mem_type t)
-{
-	return kernel_memory_type_names[static_cast<uint32_t>(t)];
-}
-
 void
 update_marked_addresses(uefi::event, void *context)
 {
-	uefi::runtime_services *rs =
+    uefi::runtime_services *rs =
-		reinterpret_cast<uefi::runtime_services*>(context);
+        reinterpret_cast<uefi::runtime_services*>(context);
 
-	for (size_t i = 0; i < fixup_pointer_index; ++i) {
+    for (size_t i = 0; i < fixup_pointer_index; ++i) {
-		if (fixup_pointers[i])
+        if (fixup_pointers[i])
-			rs->convert_pointer(0, fixup_pointers[i]);
+            rs->convert_pointer(0, fixup_pointers[i]);
-	}
+    }
 }
 
 void
 init_pointer_fixup(uefi::boot_services *bs, uefi::runtime_services *rs)
 {
-	status_line status(L"Initializing pointer virtualization event");
+    status_line status(L"Initializing pointer virtualization event");
 
-	uefi::event event;
+    uefi::event event;
-	bs->set_mem(&fixup_pointers, sizeof(fixup_pointers), 0);
+    bs->set_mem(&fixup_pointers, sizeof(fixup_pointers), 0);
-	fixup_pointer_index = 0;
+    fixup_pointer_index = 0;
 
-	try_or_raise(
+    try_or_raise(
-		bs->create_event(
+        bs->create_event(
-			uefi::evt::signal_virtual_address_change,
+            uefi::evt::signal_virtual_address_change,
-			uefi::tpl::callback,
+            uefi::tpl::callback,
-			(uefi::event_notify)&update_marked_addresses,
+            (uefi::event_notify)&update_marked_addresses,
-			rs,
+            rs,
-			&event),
+            &event),
-		L"Error creating memory virtualization event");
+        L"Error creating memory virtualization event");
 }
 
 void
 mark_pointer_fixup(void **p)
 {
-	fixup_pointers[fixup_pointer_index++] = p;
+    fixup_pointers[fixup_pointer_index++] = p;
-}
-
-bool
-can_merge(mem_entry &prev, mem_type type, uefi::memory_descriptor *next)
-{
-	return
-		prev.type == type &&
-		prev.start + (page_size * prev.pages) == next->physical_start &&
-		prev.attr == (next->attribute & 0xffffffff);
-}
-
-void
-get_uefi_mappings(efi_mem_map &map, uefi::boot_services *bs)
-{
-	size_t length = map.total;
-	uefi::status status = bs->get_memory_map(
-		&length, map.entries, &map.key, &map.size, &map.version);
-	map.length = length;
-
-	if (status == uefi::status::success)
-		return;
-
-	if (status != uefi::status::buffer_too_small)
-		error::raise(status, L"Error getting memory map size");
-
-	if (map.entries) {
-		try_or_raise(
-			bs->free_pool(reinterpret_cast<void*>(map.entries)),
-			L"Freeing previous memory map space");
-	}
-
-	map.total = length + 10*map.size;
-
-	try_or_raise(
-		bs->allocate_pool(
-			uefi::memory_type::loader_data, map.total,
-			reinterpret_cast<void**>(&map.entries)),
-		L"Allocating space for memory map");
-
-	map.length = map.total;
-	try_or_raise(
-		bs->get_memory_map(&map.length, map.entries, &map.key, &map.size, &map.version),
-		L"Getting UEFI memory map");
-}
-
-inline size_t bitmap_size(size_t frames) { return (frames + 63) / 64; }
-inline size_t num_blocks(size_t frames) { return (frames + (frames_per_block-1)) / frames_per_block; }
-
-void
-build_kernel_frame_blocks(const mem_entry *map, size_t nent, kernel::args::header *args, uefi::boot_services *bs)
-{
-	status_line status {L"Creating kernel frame accounting map"};
-
-	size_t block_count = 0;
-	size_t total_bitmap_size = 0;
-	for (size_t i = 0; i < nent; ++i) {
-		const mem_entry &ent = map[i];
-		if (ent.type != mem_type::free)
-			continue;
-
-		block_count += num_blocks(ent.pages);
-		total_bitmap_size += bitmap_size(ent.pages) * sizeof(uint64_t);
-	}
-
-	size_t total_size = block_count * sizeof(frame_block) + total_bitmap_size;
-
-	frame_block *blocks = nullptr;
-	try_or_raise(
-		bs->allocate_pages(
-			uefi::allocate_type::any_pages,
-			uefi::memory_type::loader_data,
-			bytes_to_pages(total_size),
-			reinterpret_cast<void**>(&blocks)),
-		L"Error allocating kernel frame block space");
-
-	frame_block *next_block = blocks;
-	for (size_t i = 0; i < nent; ++i) {
-		const mem_entry &ent = map[i];
-		if (ent.type != mem_type::free)
-			continue;
-
-		size_t page_count = ent.pages;
-		uintptr_t base_addr = ent.start;
-		while (page_count) {
-			frame_block *blk = next_block++;
-			bs->set_mem(blk, sizeof(frame_block), 0);
-
-			blk->attrs = ent.attr;
-			blk->base = base_addr;
-			base_addr += frames_per_block * page_size;
-
-			if (page_count >= frames_per_block) {
-				page_count -= frames_per_block;
-				blk->count = frames_per_block;
-				blk->map1 = ~0ull;
-				bs->set_mem(blk->map2, sizeof(blk->map2), 0xff);
-			} else {
-				blk->count = page_count;
-				unsigned i = 0;
-
-				uint64_t b1 = (page_count + 4095) / 4096;
-				blk->map1 = (1 << b1) - 1;
-
-				uint64_t b2 = (page_count + 63) / 64;
-				uint64_t b2q = b2 / 64;
-				uint64_t b2r = b2 % 64;
-				bs->set_mem(blk->map2, b2q, 0xff);
-				blk->map2[b2q] = (1 << b2r) - 1;
-				break;
-			}
-		}
-	}
-
-	uint64_t *bitmap = reinterpret_cast<uint64_t*>(next_block);
-	bs->set_mem(bitmap, total_bitmap_size, 0);
-	for (unsigned i = 0; i < block_count; ++i) {
-		frame_block &blk = blocks[i];
-		blk.bitmap = bitmap;
-
-		size_t b = blk.count / 64;
-		size_t r = blk.count % 64;
-		bs->set_mem(blk.bitmap, b*8, 0xff);
-		blk.bitmap[b] = (1 << r) - 1;
-
-		bitmap += bitmap_size(blk.count);
-	}
-
-	args->frame_block_count = block_count;
-	args->frame_block_pages = bytes_to_pages(total_size);
-	args->frame_blocks = blocks;
-}
-
-void
-fix_frame_blocks(kernel::args::header *args)
-{
-	// Map the frame blocks to the appropriate address
-	paging::map_pages(args,
-		reinterpret_cast<uintptr_t>(args->frame_blocks),
-		::memory::bitmap_start,
-		args->frame_block_pages,
-		true, false);
-
-	uintptr_t offset = ::memory::bitmap_start -
-		reinterpret_cast<uintptr_t>(args->frame_blocks);
-
-	for (unsigned i = 0; i < args->frame_block_count; ++i) {
-		frame_block &blk = args->frame_blocks[i];
-		blk.bitmap = reinterpret_cast<uint64_t*>(
-			reinterpret_cast<uintptr_t>(blk.bitmap) + offset);
-	}
-}
-
-efi_mem_map
-build_kernel_mem_map(kernel::args::header *args, uefi::boot_services *bs)
-{
-	status_line status {L"Creating kernel memory map"};
-
-	efi_mem_map map;
-	get_uefi_mappings(map, bs);
-
-	size_t map_size = map.num_entries() * sizeof(mem_entry);
-
-	kernel::args::mem_entry *kernel_map = nullptr;
-	try_or_raise(
-		bs->allocate_pages(
-			uefi::allocate_type::any_pages,
-			uefi::memory_type::loader_data,
-			bytes_to_pages(map_size),
-			reinterpret_cast<void**>(&kernel_map)),
-		L"Error allocating kernel memory map module space");
-
-	bs->set_mem(kernel_map, map_size, 0);
-	get_uefi_mappings(map, bs);
-
-	size_t nent = 0;
-	bool first = true;
-	for (auto desc : map) {
-		/*
-		// EFI map dump
-		console::print(L"   eRange %lx (%lx) %x(%s) [%lu]\r\n",
-			desc->physical_start, desc->attribute, desc->type, memory_type_name(desc->type), desc->number_of_pages);
-		*/
-
-		mem_type type;
-		switch (desc->type) {
-			case uefi::memory_type::reserved:
-			case uefi::memory_type::unusable_memory:
-			case uefi::memory_type::acpi_memory_nvs:
-			case uefi::memory_type::pal_code:
-				continue;
-
-			case uefi::memory_type::loader_code:
-			case uefi::memory_type::boot_services_code:
-			case uefi::memory_type::boot_services_data:
-			case uefi::memory_type::conventional_memory:
-			case uefi::memory_type::loader_data:
-				type = mem_type::free;
-				break;
-
-			case uefi::memory_type::runtime_services_code:
-			case uefi::memory_type::runtime_services_data:
-				type = mem_type::uefi_runtime;
-				break;
-
-			case uefi::memory_type::acpi_reclaim_memory:
-				type = mem_type::acpi;
-				break;
-
-			case uefi::memory_type::memory_mapped_io:
-			case uefi::memory_type::memory_mapped_io_port_space:
-				type = mem_type::mmio;
-				break;
-
-			case uefi::memory_type::persistent_memory:
-				type = mem_type::persistent;
-				break;
-
-			default:
-				error::raise(
-					uefi::status::invalid_parameter,
-					L"Got an unexpected memory type from UEFI memory map");
-		}
-
-		// TODO: validate uefi's map is sorted
-		if (first) {
-			first = false;
-			mem_entry &ent = kernel_map[nent++];
-			ent.start = desc->physical_start;
-			ent.pages = desc->number_of_pages;
-			ent.type = type;
-			ent.attr = (desc->attribute & 0xffffffff);
-			continue;
-		}
-
-		mem_entry &prev = kernel_map[nent - 1];
-		if (can_merge(prev, type, desc)) {
-			prev.pages += desc->number_of_pages;
-		} else {
-			mem_entry &next = kernel_map[nent++];
-			next.start = desc->physical_start;
-			next.pages = desc->number_of_pages;
-			next.type = type;
-			next.attr = (desc->attribute & 0xffffffff);
-		}
-	}
-
-	// Give just the actually-set entries in the header
-	args->mem_map = kernel_map;
-	args->map_count = nent;
-
-	/*
-	// kernel map dump
-	for (unsigned i = 0; i < nent; ++i) {
-		const kernel::args::mem_entry &e = kernel_map[i];
-		console::print(L"   kRange %lx (%lx) %x(%s) [%lu]\r\n",
-			e.start, e.attr, e.type, kernel_memory_type_name(e.type), e.pages);
-	}
-	*/
-
-	build_kernel_frame_blocks(kernel_map, nent, args, bs);
-	get_uefi_mappings(map, bs);
-	return map;
 }
 
 void
 virtualize(void *pml4, efi_mem_map &map, uefi::runtime_services *rs)
 {
-	paging::add_current_mappings(reinterpret_cast<paging::page_table*>(pml4));
+    paging::add_current_mappings(reinterpret_cast<paging::page_table*>(pml4));
 
-	for (auto desc : map)
+    for (auto &desc : map)
-		desc->virtual_start = desc->physical_start + ::memory::page_offset;
+        desc.virtual_start = desc.physical_start + bootproto::mem::linear_offset;
 
-	// Write our new PML4 pointer to CR3
+    // Write our new PML4 pointer to CR3
-	asm volatile ( "mov %0, %%cr3" :: "r" (pml4) );
+    asm volatile ( "mov %0, %%cr3" :: "r" (pml4) );
-	__sync_synchronize();
+    __sync_synchronize();
 
-	try_or_raise(
+    try_or_raise(
-		rs->set_virtual_address_map(
+        rs->set_virtual_address_map(
-			map.length, map.size, map.version, map.entries),
+            map.length, map.size, map.version, map.entries),
-		L"Error setting virtual address map");
+        L"Error setting virtual address map");
 }
 
 

src/boot/memory.h

@@ -1,21 +1,25 @@
+#pragma once
 /// \file memory.h
 /// Memory-related constants and functions.
-#pragma once
+
-#include <uefi/boot_services.h>
-#include <uefi/runtime_services.h>
 #include <stdint.h>
-#include "kernel_args.h"
+
-#include "pointer_manipulation.h"
+namespace uefi {
+    struct boot_services;
+    struct runtime_services;
+}
 
 namespace boot {
 namespace memory {
 
+class efi_mem_map;
+
 /// UEFI specifies that pages are always 4 KiB.
 constexpr size_t page_size = 0x1000;
 
 /// Get the number of pages needed to hold `bytes` bytes
 inline constexpr size_t bytes_to_pages(size_t bytes) {
-	return ((bytes - 1) / page_size) + 1;
+    return ((bytes - 1) / page_size) + 1;
 }
 
 /// \defgroup pointer_fixup
@@ -33,52 +37,14 @@ void mark_pointer_fixup(void **p);
 
 /// @}
 
-/// Struct that represents UEFI's memory map. Contains a pointer to the map data
-/// as well as the data on how to read it.
-struct efi_mem_map
-{
-	using desc = uefi::memory_descriptor;
-	using iterator = offset_iterator<desc>;
-
-	size_t length;     ///< Total length of the map data
-	size_t total;      ///< Total allocated space for map data
-	size_t size;       ///< Size of an entry in the array
-	size_t key;        ///< Key for detecting changes
-	uint32_t version;  ///< Version of the `memory_descriptor` struct
-	desc *entries;     ///< The array of UEFI descriptors
-
-	efi_mem_map() : length(0), total(0), size(0), key(0), version(0), entries(nullptr) {}
-
-	/// Get the count of entries in the array
-	inline size_t num_entries() const { return length / size; }
-
-	/// Return an iterator to the beginning of the array
-	iterator begin() { return iterator(entries, size); }
-
-	/// Return an iterator to the end of the array
-	iterator end() { return offset_ptr<desc>(entries, length); }
-};
-
-/// Add the kernel's memory map as a module to the kernel args.
-/// \returns  The uefi memory map used to build the kernel map
-efi_mem_map build_kernel_mem_map(kernel::args::header *args, uefi::boot_services *bs);
-
-/// Create the kernel frame allocation maps
-void build_kernel_frame_blocks(
-		const kernel::args::mem_entry *map, size_t nent,
-		kernel::args::header *args, uefi::boot_services *bs);
-
-/// Map the frame allocation maps to the right spot and fix up pointers
-void fix_frame_blocks(kernel::args::header *args);
-
 /// Activate the given memory mappings. Sets the given page tables live as well
 /// as informs UEFI runtime services of the new mappings.
 /// \arg pml4  The root page table for the new mappings
 /// \arg map   The UEFI memory map, used to update runtime services
 void virtualize(
-	void *pml4,
+    void *pml4,
-	efi_mem_map &map,
+    efi_mem_map &map,
-	uefi::runtime_services *rs);
+    uefi::runtime_services *rs);
 
 } // namespace boot
 } // namespace memory

src/boot/memory_map.cpp

@@ -0,0 +1,306 @@
+#include <uefi/boot_services.h>
+#include <uefi/types.h>
+
+#include <bootproto/kernel.h>
+#include <bootproto/memory.h>
+#include <util/pointers.h>
+
+#include "allocator.h"
+#include "error.h"
+#include "memory.h"
+#include "memory_map.h"
+#include "paging.h"
+#include "status.h"
+
+namespace boot {
+namespace memory {
+
+using bootproto::frame_block;
+using bootproto::frames_per_block;
+using bootproto::mem_entry;
+using bootproto::mem_type;
+
+
+void
+efi_mem_map::update(uefi::boot_services &bs)
+{
+    size_t l = total;
+    uefi::status status = bs.get_memory_map(
+        &l, entries, &key, &size, &version);
+    length = l;
+
+    if (status == uefi::status::success)
+        return;
+
+    if (status != uefi::status::buffer_too_small)
+        error::raise(status, L"Error getting memory map size");
+
+    if (entries) {
+        try_or_raise(
+            bs.free_pool(reinterpret_cast<void*>(entries)),
+            L"Freeing previous memory map space");
+    }
+
+    total = length + 10 * size;
+
+    try_or_raise(
+        bs.allocate_pool(
+            uefi::memory_type::loader_data, total,
+            reinterpret_cast<void**>(&entries)),
+        L"Allocating space for memory map");
+
+    length = total;
+    try_or_raise(
+        bs.get_memory_map(&length, entries, &key, &size, &version),
+        L"Getting UEFI memory map");
+}
+
+static const wchar_t *memory_type_names[] = {
+    L"reserved memory type",
+    L"loader code",
+    L"loader data",
+    L"boot services code",
+    L"boot services data",
+    L"runtime services code",
+    L"runtime services data",
+    L"conventional memory",
+    L"unusable memory",
+    L"acpi reclaim memory",
+    L"acpi memory nvs",
+    L"memory mapped io",
+    L"memory mapped io port space",
+    L"pal code",
+    L"persistent memory"
+};
+
+static const wchar_t *kernel_memory_type_names[] = {
+    L"free",
+    L"pending",
+    L"acpi",
+    L"uefi_runtime",
+    L"mmio",
+    L"persistent"
+};
+
+static const wchar_t *
+memory_type_name(uefi::memory_type t)
+{
+    if (t < uefi::memory_type::max_memory_type)
+        return memory_type_names[static_cast<uint32_t>(t)];
+
+    return L"Bad Type Value";
+}
+
+static const wchar_t *
+kernel_memory_type_name(bootproto::mem_type t)
+{
+    return kernel_memory_type_names[static_cast<uint32_t>(t)];
+}
+
+inline bool
+can_merge(mem_entry &prev, mem_type type, uefi::memory_descriptor &next)
+{
+    return
+        prev.type == type &&
+        prev.start + (page_size * prev.pages) == next.physical_start &&
+        prev.attr == (next.attribute & 0xffffffff);
+}
+
+util::counted<mem_entry>
+build_kernel_map(efi_mem_map &map)
+{
+    status_line status {L"Creating kernel memory map"};
+
+    size_t map_size = map.num_entries() * sizeof(mem_entry);
+    size_t num_pages = bytes_to_pages(map_size);
+    mem_entry *kernel_map = reinterpret_cast<mem_entry*>(
+        g_alloc.allocate_pages(num_pages, alloc_type::mem_map, true));
+
+    size_t nent = 0;
+    bool first = true;
+    for (auto &desc : map) {
+        /*
+        // EFI map dump
+        console::print(L"   eRange %lx (%lx) %x(%s) [%lu]\r\n",
+            desc.physical_start, desc.attribute, desc.type, memory_type_name(desc.type), desc.number_of_pages);
+        */
+
+        mem_type type;
+        switch (desc.type) {
+            case uefi::memory_type::reserved:
+            case uefi::memory_type::unusable_memory:
+            case uefi::memory_type::acpi_memory_nvs:
+            case uefi::memory_type::pal_code:
+                continue;
+
+            case uefi::memory_type::loader_code:
+            case uefi::memory_type::boot_services_code:
+            case uefi::memory_type::boot_services_data:
+            case uefi::memory_type::conventional_memory:
+            case uefi::memory_type::loader_data:
+                type = mem_type::free;
+                break;
+
+            case uefi::memory_type::runtime_services_code:
+            case uefi::memory_type::runtime_services_data:
+                type = mem_type::uefi_runtime;
+                break;
+
+            case uefi::memory_type::acpi_reclaim_memory:
+                type = mem_type::acpi;
+                break;
+
+            case uefi::memory_type::memory_mapped_io:
+            case uefi::memory_type::memory_mapped_io_port_space:
+                type = mem_type::mmio;
+                break;
+
+            case uefi::memory_type::persistent_memory:
+                type = mem_type::persistent;
+                break;
+
+            default:
+                error::raise(
+                    uefi::status::invalid_parameter,
+                    L"Got an unexpected memory type from UEFI memory map");
+        }
+
+        // TODO: validate uefi's map is sorted
+        if (first) {
+            first = false;
+            mem_entry &ent = kernel_map[nent++];
+            ent.start = desc.physical_start;
+            ent.pages = desc.number_of_pages;
+            ent.type = type;
+            ent.attr = (desc.attribute & 0xffffffff);
+            continue;
+        }
+
+        mem_entry &prev = kernel_map[nent - 1];
+        if (can_merge(prev, type, desc)) {
+            prev.pages += desc.number_of_pages;
+        } else {
+            mem_entry &next = kernel_map[nent++];
+            next.start = desc.physical_start;
+            next.pages = desc.number_of_pages;
+            next.type = type;
+            next.attr = (desc.attribute & 0xffffffff);
+        }
+    }
+
+    /*
+    // kernel map dump
+    for (unsigned i = 0; i < nent; ++i) {
+        const mem_entry &e = kernel_map[i];
+        console::print(L"   kRange %lx (%lx) %x(%s) [%lu]\r\n",
+            e.start, e.attr, e.type, kernel_memory_type_name(e.type), e.pages);
+    }
+    */
+
+    return { .pointer = kernel_map, .count = nent };
+}
+
+inline size_t bitmap_size(size_t frames) { return (frames + 63) / 64; }
+inline size_t num_blocks(size_t frames) { return (frames + (frames_per_block-1)) / frames_per_block; }
+
+util::counted<bootproto::frame_block>
+build_frame_blocks(const util::counted<bootproto::mem_entry> &kmap)
+{
+    status_line status {L"Creating kernel frame accounting map"};
+
+    size_t block_count = 0;
+    size_t total_bitmap_size = 0;
+    for (size_t i = 0; i < kmap.count; ++i) {
+        const mem_entry &ent = kmap[i];
+        if (ent.type != mem_type::free)
+            continue;
+
+        block_count += num_blocks(ent.pages);
+        total_bitmap_size += bitmap_size(ent.pages) * sizeof(uint64_t);
+    }
+
+    size_t total_size = block_count * sizeof(frame_block) + total_bitmap_size;
+
+    frame_block *blocks = reinterpret_cast<frame_block*>(
+        g_alloc.allocate_pages(bytes_to_pages(total_size), alloc_type::frame_map, true));
+
+    frame_block *next_block = blocks;
+    for (size_t i = 0; i < kmap.count; ++i) {
+        const mem_entry &ent = kmap[i];
+        if (ent.type != mem_type::free)
+            continue;
+
+        size_t page_count = ent.pages;
+        uintptr_t base_addr = ent.start;
+        while (page_count) {
+            frame_block *blk = next_block++;
+
+            blk->flags = static_cast<bootproto::frame_flags>(ent.attr);
+            blk->base = base_addr;
+            base_addr += frames_per_block * page_size;
+
+            if (page_count >= frames_per_block) {
+                page_count -= frames_per_block;
+                blk->count = frames_per_block;
+                blk->map1 = ~0ull;
+                g_alloc.memset(blk->map2, sizeof(blk->map2), 0xff);
+            } else {
+                blk->count = page_count;
+                unsigned i = 0;
+
+                uint64_t b1 = (page_count + 4095) / 4096;
+                blk->map1 = (1 << b1) - 1;
+
+                uint64_t b2 = (page_count + 63) / 64;
+                uint64_t b2q = b2 / 64;
+                uint64_t b2r = b2 % 64;
+                g_alloc.memset(blk->map2, b2q, 0xff);
+                blk->map2[b2q] = (1 << b2r) - 1;
+                break;
+            }
+        }
+    }
+
+    uint64_t *bitmap = reinterpret_cast<uint64_t*>(next_block);
+    for (unsigned i = 0; i < block_count; ++i) {
+        frame_block &blk = blocks[i];
+        blk.bitmap = bitmap;
+
+        size_t b = blk.count / 64;
+        size_t r = blk.count % 64;
+        g_alloc.memset(blk.bitmap, b*8, 0xff);
+        blk.bitmap[b] = (1 << r) - 1;
+
+        bitmap += bitmap_size(blk.count);
+    }
+
+    return { .pointer = blocks, .count = block_count };
+}
+
+void
+fix_frame_blocks(bootproto::args *args)
+{
+    util::counted<frame_block> &blocks = args->frame_blocks;
+
+    size_t size = blocks.count * sizeof(frame_block);
+    for (unsigned i = 0; i < blocks.count; ++i)
+        size += bitmap_size(blocks[i].count) * sizeof(uint64_t);
+
+    size_t pages = bytes_to_pages(size);
+    uintptr_t addr = reinterpret_cast<uintptr_t>(blocks.pointer);
+
+    // Map the frame blocks to the appropriate address
+    paging::map_pages(args, addr,
+        bootproto::mem::bitmap_offset, pages, true, false);
+
+    uintptr_t offset = bootproto::mem::bitmap_offset - addr;
+
+    for (unsigned i = 0; i < blocks.count; ++i) {
+        frame_block &blk = blocks[i];
+        blk.bitmap = util::offset_pointer(blk.bitmap, offset);
+    }
+}
+
+
+} // namespace memory
+} // namespace boot

src/boot/memory_map.h

@@ -0,0 +1,62 @@
+#pragma once
+/// \file memory_map.h
+/// Memory-map related types and functions
+
+#include <util/counted.h>
+#include <util/pointers.h>
+
+namespace uefi {
+    struct boot_services;
+    struct memory_descriptor;
+}
+
+namespace bootproto {
+    struct args;
+    struct frame_block;
+    struct mem_entry;
+}
+
+namespace boot {
+namespace memory {
+
+/// Struct that represents UEFI's memory map. Contains a pointer to the map data
+/// as well as the data on how to read it.
+struct efi_mem_map
+{
+    using desc = uefi::memory_descriptor;
+    using iterator = util::offset_iterator<desc>;
+
+    size_t length;     ///< Total length of the map data
+    size_t total;      ///< Total allocated space for map data
+    size_t size;       ///< Size of an entry in the array
+    size_t key;        ///< Key for detecting changes
+    uint32_t version;  ///< Version of the `memory_descriptor` struct
+    desc *entries;     ///< The array of UEFI descriptors
+
+    efi_mem_map() : length(0), total(0), size(0), key(0), version(0), entries(nullptr) {}
+
+    /// Update the map from UEFI
+    void update(uefi::boot_services &bs);
+
+    /// Get the count of entries in the array
+    inline size_t num_entries() const { return length / size; }
+
+    /// Return an iterator to the beginning of the array
+    inline iterator begin() { return iterator(entries, size); }
+
+    /// Return an iterator to the end of the array
+    inline iterator end() { return util::offset_pointer(entries, length); }
+};
+
+/// Add the kernel's memory map as a module to the kernel args.
+/// \returns  The uefi memory map used to build the kernel map
+util::counted<bootproto::mem_entry> build_kernel_map(efi_mem_map &map);
+
+/// Create the kernel frame allocation maps
+util::counted<bootproto::frame_block> build_frame_blocks(const util::counted<bootproto::mem_entry> &kmap);
+
+/// Map the frame allocation maps to the right spot and fix up pointers
+void fix_frame_blocks(bootproto::args *args);
+
+} // namespace boot
+} // namespace memory

src/boot/paging.cpp

@@ -1,19 +1,21 @@
-#include "kernel_memory.h"
+#include <arch/memory.h>
+#include <bootproto/memory.h>
+#include <util/counted.h>
+#include <util/pointers.h>
 
+#include "allocator.h"
 #include "console.h"
 #include "error.h"
 #include "loader.h"
 #include "memory.h"
 #include "paging.h"
-#include "pointer_manipulation.h"
 #include "status.h"
 
 namespace boot {
 namespace paging {
 
+using memory::alloc_type;
 using memory::page_size;
-using ::memory::pml4e_kernel;
-using ::memory::table_entries;
 
 // Flags: 0 0 0 1  0 0 0 0  0 0 0 1 = 0x0101
 //         IGN  |  | | | |  | | | +- Present
@@ -54,234 +56,227 @@ constexpr uint64_t huge_page_flags = 0x18b;
 /// Page table entry flags for entries pointing at another table
 constexpr uint64_t table_flags = 0x003;
 
+
+inline void *
+pop_pages(util::counted<void> &pages, size_t count)
+{
+    if (count > pages.count)
+        error::raise(uefi::status::out_of_resources, L"Page table cache empty", 0x7ab1e5);
+
+    void *next = pages.pointer;
+    pages.pointer = util::offset_pointer(pages.pointer, count*page_size);
+    pages.count -= count;
+    return next;
+}
+
 /// Iterator over page table entries.
 template <unsigned D = 4>
 class page_entry_iterator
 {
 public:
-	/// Constructor.
+    /// Constructor.
-	/// \arg virt       Virtual address this iterator is starting at
+    /// \arg virt    Virtual address this iterator is starting at
-	/// \arg pml4       Root of the page tables to iterate
+    /// \arg pml4    Root of the page tables to iterate
-	/// \arg page_cache Pointer to pages that can be used for page tables
+    /// \arg pages   Cache of usable table pages
-	/// \arg page_count Number of pages pointed to by `page_cache`
+    page_entry_iterator(
-	page_entry_iterator(
+            uintptr_t virt,
-			uintptr_t virt,
+            page_table *pml4,
-			page_table *pml4,
+            util::counted<void> &pages) :
-			void *&page_cache,
+        m_pages(pages)
-			size_t &cache_count) :
+    {
-		m_page_cache(page_cache),
+        m_table[0] = pml4;
-		m_cache_count(cache_count)
+        for (unsigned i = 0; i < D; ++i) {
-	{
+            m_index[i] = static_cast<uint16_t>((virt >> (12 + 9*(3-i))) & 0x1ff);
-		m_table[0] = pml4;
+            ensure_table(i);
-		for (unsigned i = 0; i < D; ++i) {
+        }
-			m_index[i] = static_cast<uint16_t>((virt >> (12 + 9*(3-i))) & 0x1ff);
+    }
-			ensure_table(i);
-		}
-	}
 
-	uintptr_t vaddress() const {
+    uintptr_t vaddress() const {
-		uintptr_t address = 0;
+        uintptr_t address = 0;
-		for (unsigned i = 0; i < D; ++i)
+        for (unsigned i = 0; i < D; ++i)
-			address |= static_cast<uintptr_t>(m_index[i]) << (12 + 9*(3-i));
+            address |= static_cast<uintptr_t>(m_index[i]) << (12 + 9*(3-i));
-		if (address & (1ull<<47)) // canonicalize the address
+        if (address & (1ull<<47)) // canonicalize the address
-			address |= (0xffffull<<48);
+            address |= (0xffffull<<48);
-		return address;
+        return address;
-	}
+    }
 
-	void increment()
+    void increment()
-	{
+    {
-		for (unsigned i = D - 1; i >= 0; --i) {
+        for (unsigned i = D - 1; i >= 0; --i) {
-			if (++m_index[i] <= 511) {
+            if (++m_index[i] <= 511) {
-				for (unsigned j = i + 1; j < D; ++j)
+                for (unsigned j = i + 1; j < D; ++j)
-					ensure_table(j);
+                    ensure_table(j);
-				return;
+                return;
-			}
+            }
 
-			m_index[i] = 0;
+            m_index[i] = 0;
-		}
+        }
-	}
+    }
 
-	uint64_t & operator*() { return entry(D-1); }
+    uint64_t & operator*() { return entry(D-1); }
 
 private:
-	inline uint64_t & entry(unsigned level) { return m_table[level]->entries[m_index[level]]; }
+    inline uint64_t & entry(unsigned level) { return m_table[level]->entries[m_index[level]]; }
 
-	void ensure_table(unsigned level)
+    void ensure_table(unsigned level)
-	{
+    {
-		// We're only dealing with D levels of paging, and
+        // We're only dealing with D levels of paging, and
-		// there must always be a PML4.
+        // there must always be a PML4.
-		if (level < 1 || level >= D)
+        if (level < 1 || level >= D)
-			return;
+            return;
 
-		// Entry in the parent that points to the table we want
+        // Entry in the parent that points to the table we want
-		uint64_t & parent_ent = entry(level - 1);
+        uint64_t & parent_ent = entry(level - 1);
 
-		if (!(parent_ent & 1)) {
+        if (!(parent_ent & 1)) {
-			if (!m_cache_count--)
+            page_table *table = reinterpret_cast<page_table*>(pop_pages(m_pages, 1));
-				error::raise(uefi::status::out_of_resources, L"Page table cache empty", 0x7ab1e5);
+            parent_ent = (reinterpret_cast<uintptr_t>(table) & ~0xfffull) | table_flags;
+            m_table[level] = table;
+        } else {
+            m_table[level] = reinterpret_cast<page_table*>(parent_ent & ~0xfffull);
+        }
+    }
 
-			page_table *table = reinterpret_cast<page_table*>(m_page_cache);
+    util::counted<void> &m_pages;
-			m_page_cache = offset_ptr<void>(m_page_cache, page_size);
+    page_table *m_table[D];
-
+    uint16_t m_index[D];
-			parent_ent = (reinterpret_cast<uintptr_t>(table) & ~0xfffull) | table_flags;
-			m_table[level] = table;
-		} else {
-			m_table[level] = reinterpret_cast<page_table*>(parent_ent & ~0xfffull);
-		}
-	}
-
-	void *&m_page_cache;
-	size_t &m_cache_count;
-	page_table *m_table[D];
-	uint16_t m_index[D];
 };
 
 
 static void
-add_offset_mappings(page_table *pml4, void *&page_cache, size_t &num_pages)
+add_offset_mappings(page_table *pml4, util::counted<void> &pages)
 {
-	uintptr_t phys = 0;
+    uintptr_t phys = 0;
-	uintptr_t virt = ::memory::page_offset; // Start of offset-mapped area
+    uintptr_t virt = bootproto::mem::linear_offset; // Start of offset-mapped area
-	size_t pages = 64 * 1024; // 64 TiB of 1 GiB pages
+    size_t page_count = 64 * 1024; // 64 TiB of 1 GiB pages
-	constexpr size_t GiB = 0x40000000ull;
+    constexpr size_t GiB = 0x40000000ull;
 
-	page_entry_iterator<2> iterator{
+    page_entry_iterator<2> iterator{virt, pml4, pages};
-		virt, pml4,
-		page_cache,
-		num_pages};
 
-	while (true) {
+    while (true) {
-		*iterator = phys | huge_page_flags;
+        *iterator = phys | huge_page_flags;
-		if (--pages == 0)
+        if (--page_count == 0)
-			break;
+            break;
 
-		iterator.increment();
+        iterator.increment();
-		phys += GiB;
+        phys += GiB;
-	}
+    }
 }
 
 static void
-add_kernel_pds(page_table *pml4, void *&page_cache, size_t &num_pages)
+add_kernel_pds(page_table *pml4, util::counted<void> &pages)
 {
-	for (unsigned i = pml4e_kernel; i < table_entries; ++i) {
+    constexpr unsigned start = arch::kernel_root_index; 
-		pml4->set(i, page_cache, table_flags);
+    constexpr unsigned end = arch::table_entries; 
-		page_cache = offset_ptr<void>(page_cache, page_size);
+
-		num_pages--;
+    for (unsigned i = start; i < end; ++i)
-	}
+        pml4->set(i, pop_pages(pages, 1), table_flags);
 }
 
 void
 add_current_mappings(page_table *new_pml4)
 {
-	// Get the pointer to the current PML4
+    // Get the pointer to the current PML4
-	page_table *old_pml4 = 0;
+    page_table *old_pml4 = 0;
-	asm volatile ( "mov %%cr3, %0" : "=r" (old_pml4) );
+    asm volatile ( "mov %%cr3, %0" : "=r" (old_pml4) );
 
-	// Only copy mappings in the lower half
+    // Only copy mappings in the lower half
-	for (int i = 0; i < ::memory::pml4e_kernel; ++i) {
+    constexpr unsigned halfway = arch::kernel_root_index; 
-		uint64_t entry = old_pml4->entries[i];
+    for (int i = 0; i < halfway; ++i) {
-		if (entry & 1)
+        uint64_t entry = old_pml4->entries[i];
-			new_pml4->entries[i] = entry;
+        if (entry & 1)
-	}
+            new_pml4->entries[i] = entry;
+    }
 }
 
 void
-allocate_tables(kernel::args::header *args, uefi::boot_services *bs)
+allocate_tables(bootproto::args *args)
 {
-	status_line status(L"Allocating initial page tables");
+    status_line status(L"Allocating initial page tables");
 
-	static constexpr size_t pd_tables = 256;   // number of pages for kernelspace PDs
+    static constexpr size_t pd_tables = 256;   // number of pages for kernelspace PDs
-	static constexpr size_t extra_tables = 64; // number of extra pages
+    static constexpr size_t extra_tables = 64; // number of extra pages
 
-	// number of pages for kernelspace PDs + PML4
+    // number of pages for kernelspace PDs + PML4
-	static constexpr size_t kernel_tables = pd_tables + 1;
+    static constexpr size_t kernel_tables = pd_tables + 1;
 
-	static constexpr size_t tables_needed = kernel_tables + extra_tables;
+    static constexpr size_t tables_needed = kernel_tables + extra_tables;
 
-	void *addr = nullptr;
+    void *addr = g_alloc.allocate_pages(tables_needed, alloc_type::page_table, true);
-	try_or_raise(
+    page_table *pml4 = reinterpret_cast<page_table*>(addr);
-		bs->allocate_pages(
-			uefi::allocate_type::any_pages,
-			uefi::memory_type::loader_data,
-			tables_needed,
-			&addr),
-		L"Error allocating page table pages.");
 
-	bs->set_mem(addr, tables_needed*page_size, 0);
+    args->pml4 = pml4;
+    args->page_tables = { .pointer = pml4 + 1, .count = tables_needed - 1 };
 
-	page_table *pml4 = reinterpret_cast<page_table*>(addr);
+    console::print(L"    First page (pml4) at: 0x%lx\r\n", pml4);
 
-	args->pml4 = pml4;
+    add_kernel_pds(pml4, args->page_tables);
-	args->table_pages = tables_needed;
+    add_offset_mappings(pml4, args->page_tables);
-	args->table_count = tables_needed - 1;
-	args->page_tables = offset_ptr<void>(addr, page_size);
 
-	console::print(L"    First page (pml4) at: 0x%lx\r\n", pml4);
+    //console::print(L"    Set up initial mappings, %d spare tables.\r\n", args->table_count);
-
-	add_kernel_pds(pml4, args->page_tables, args->table_count);
-	add_offset_mappings(pml4, args->page_tables, args->table_count);
-
-	//console::print(L"    Set up initial mappings, %d spare tables.\r\n", args->table_count);
 }
 
 template <typename E>
 constexpr bool has_flag(E set, E flag) {
-	return
+    return
-		(static_cast<uint64_t>(set) & static_cast<uint64_t>(flag)) ==
+        (static_cast<uint64_t>(set) & static_cast<uint64_t>(flag)) ==
-		 static_cast<uint64_t>(flag);
+         static_cast<uint64_t>(flag);
 }
 
 void
 map_pages(
-	kernel::args::header *args,
+    bootproto::args *args,
-	uintptr_t phys, uintptr_t virt,
+    uintptr_t phys, uintptr_t virt,
-	size_t count, bool write_flag, bool exe_flag)
+    size_t count, bool write_flag, bool exe_flag)
 {
-	if (!count)
+    if (!count)
-		return;
+        return;
 
-	paging::page_table *pml4 =
+    paging::page_table *pml4 =
-		reinterpret_cast<paging::page_table*>(args->pml4);
+        reinterpret_cast<paging::page_table*>(args->pml4);
 
-	page_entry_iterator<4> iterator{
+    page_entry_iterator<4> iterator{virt, pml4, args->page_tables};
-		virt, pml4,
-		args->page_tables,
-		args->table_count};
 
-	uint64_t flags = page_flags;
+    uint64_t flags = page_flags;
-	if (!exe_flag)
+    if (!exe_flag)
-		flags |= (1ull << 63); // set NX bit
+        flags |= (1ull << 63); // set NX bit
-	if (write_flag)
+    if (write_flag)
-		flags |= 2;
+        flags |= 2;
 
-	while (true) {
+    while (true) {
-		*iterator = phys | flags;
+        *iterator = phys | flags;
-		if (--count == 0)
+        if (--count == 0)
-			break;
+            break;
 
-		iterator.increment();
+        iterator.increment();
-		phys += page_size;
+        phys += page_size;
-	}
+    }
 }
 
 void
 map_section(
-	kernel::args::header *args,
+    bootproto::args *args,
-	const kernel::args::program_section &section)
+    const bootproto::program_section &section)
 {
-	using kernel::args::section_flags;
+    using bootproto::section_flags;
 
-	size_t pages = memory::bytes_to_pages(section.size);
+    map_pages(
-
+        args,
-	map_pages(
+        section.phys_addr,
-		args,
+        section.virt_addr,
-		section.phys_addr,
+        memory::bytes_to_pages(section.size),
-		section.virt_addr,
+        has_flag(section.type, section_flags::write),
-		pages,
+        has_flag(section.type, section_flags::execute));
-		has_flag(section.type, section_flags::write),
-		has_flag(section.type, section_flags::execute));
 }
 
+void
+map_program(
+    bootproto::args *args,
+    bootproto::program &program)
+{
+    for (auto &section : program.sections)
+        paging::map_section(args, section);
+}
 
 } // namespace paging
 } // namespace boot