[project] README updates

Stub out math.h in order to include c++ headers that include it but
don't actually call any math functions.

.clang-format

@@ -1,21 +0,0 @@
----
-Language:        Cpp
-BasedOnStyle:    LLVM
-
-ColumnLimit:     100
-IndentWidth:     4
-TabWidth:        4
-UseTab:          Always
-
-AccessModifierOffset: -4
-
-AlignEscapedNewlinesLeft: true
-
-AllowShortIfStatementsOnASingleLine: true
-AllowShortLoopsOnASingleLine: true
-
-AlwaysBreakAfterReturnType: AllDefinitions
-AlwaysBreakBeforeMultilineStrings: true
-AlwaysBreakTemplateDeclarations: true
-
-BreakBeforeBraces: Linux

.gitignore

@@ -1,10 +1,15 @@
+.cache
 .lock*
 /build*
 *.bak
 tags
 jsix.log
+*.out
 *.o
 *.a
 sysroot
 .gdb_history
 .peru
+__pycache__
+/venv
+compile_commands.json

NOTES.md

@@ -1,38 +0,0 @@
-# Design / WIP notes
-
-## TODO
-
-- Paging manager
-  - Copy-on-write pages
-  - Better page-allocation model?
-- Allow for more than one IOAPIC in ACPI module
-  - The objects get created, but GSI lookup only uses the one at index 0
-- mark kernel memory pages global
-- Serial out based on circular/bip biffer and interrupts, not spinning on
-  `write_ready()`
-- Split out more code into kutil for testing
-- AHCI / MSI interrupts on Vbox break?
-- FXSAVE to save XMM registers.
-  - optimization using #NM (0x7) to detect SSE usage
-- Clean up of process memory maps
-- Better stack tracer
-- Bootloader rewrite
-  - C++ and sharing library code for ELF, initrd, etc
-  - Parse initrd and pre-load certain ELF images, eg the process loader process?
-  - Do initial memory bootstrap?
-- Calling global ctors
-- Device Tree
-  - Actual serial driver
-  - Disk driver
-    - File system
-- Multiprocessing
-  - Fast syscalls using syscall/sysret
-
-### Build
-
-- Clean up build generator and its templates
-  - More robust objects to represent modules & targets to pass to templates
-  - Read project setup from a simple JSON/TOML/etc
-  - Better compartmentalizing when doing template inheritance
-- Move to LLD as sysroot linker
-

README.md

@@ -1,15 +1,19 @@
-# jsix: A toy OS kernel
+![jsix](assets/jsix.svg)
 
-**jsix** is the kernel for the hobby OS that I am currently building. It's
+# The jsix operating system
-far from finished, or even being usable. Instead, it's a sandbox for me to play
+
-with kernel-level code and explore architectures.
+**jsix** is a custom multi-core x64 operating system that I am building from
+scratch. It's far from finished, or even being usable (see the *Status and
+Roadmap* section, below) but all currently-planned major kernel features are
+now implemented to at least a passable level.
 
 The design goals of the project are:
 
 * Modernity - I'm not interested in designing for legacy systems, or running on
-  all hardware out there. My target is only 64 bit architecutres, and modern
+  all hardware out there. My target is only 64 bit architectures, and modern
   commodity hardware. Currently that means x64 systems with Nehalem or newer
-  CPUs and UEFI firmware. Eventually I'd like to work on an AArch64 port,
+  CPUs and UEFI firmware. (See [this list][cpu_features] for the currently
+  required CPU features.) Eventually I'd like to work on an AArch64 port,
   partly to force myself to factor out the architecture-dependent pieces of the
   code base.
 
@@ -17,13 +21,11 @@ The design goals of the project are:
   processes as possible, in the microkernel fashion. A sub-goal of this is to
   explore where the bottlenecks of such a microkernel are now, and whether
   eschewing legacy hardware will let me design a system that's less bogged down
-  by the traditional microkernel problems. Given that there are no processes
+  by the traditional microkernel problems.
-  yet, the kernel is monolithic by default.
 
 * Exploration - I'm really mostly doing this to have fun learning and exploring
-  modern OS development. Modular design may be tossed out (hopefully
+  modern OS development. Initial feature implementations may temporarily throw
-  temporarily) in some places to allow me to play around with the related
+  away modular design to allow for exploration of the related hardware.
-  hardware.
 
 A note on the name: This kernel was originally named Popcorn, but I have since
 discovered that the Popcorn Linux project is also developing a kernel with that
@@ -31,48 +33,149 @@ name, started around the same time as this project. So I've renamed this kernel
 jsix (Always styled _jsix_ or `j6`, never capitalized) as an homage to L4, xv6,
 and my wonderful wife.
 
+[cpu_features]: https://github.com/justinian/jsix/blob/master/src/libraries/cpu/include/cpu/features.inc
+
+## Status and Roadmap
+
+The following major feature areas are targets for jsix development:
+
+#### UEFI boot loader
+
+_Done._ The bootloader loads the kernel and initial userspace programs, and
+sets up necessary kernel arguments about the memory map and EFI GOP
+framebuffer. Possible future ideas:
+
+- take over more init-time functions from the kernel
+- rewrite it in Zig
+
+#### Memory
+
+_Virtual memory: Sufficient._ The kernel manages virtual memory with a number
+of kinds of `vm_area` objects representing mapped areas, which can belong to
+one or more `vm_space` objects which represent a whole virtual memory space.
+(Each process has a `vm_space`, and so does the kernel itself.)
+
+Remaining to do:
+
+- TLB shootdowns
+- Page swapping
+- Large / huge page support
+
+_Physical page allocation: Sufficient._ The current physical page allocator
+implementation uses a group of blocks representing up-to-1GiB areas of usable
+memory as defined by the bootloader. Each block has a three-level bitmap
+denoting free/used pages.
+
+Future work:
+
+- Align blocks so that their first page is 1GiB-aligned, making finding free
+  large/huge pages easier.
+
+#### Multitasking
+
+_Sufficient._ The global scheduler object keeps separate ready/blocked lists
+per core. Cores periodically attempt to balance load via work stealing.
+
+User-space tasks are able to create threads as well as other processes.
+
+#### API
+
+_Syscalls: Sufficient._ User-space tasks are able to make syscalls to the
+kernel via fast SYSCALL/SYSRET instructions. Syscalls made via `libj6` look to
+both the callee and the caller like standard SysV ABI function calls. The
+implementations are wrapped in generated wrapper functions which validate the
+request, check capabilities, and find the appropriate kernel objects or handles
+before calling the implementation functions.
+
+_IPC: Working, needs optimization._ The current IPC primitives are:
+
+- _Mailboxes_: endpoints for asynchronously-delivered small messages. Currently
+  these messages are double-copied - once from the caller into a kernel buffer,
+  and once from the kernel to the receiver. This works and is not a major cause
+  of slowdown, but will need to be optimized in the future.
+- _Channels_: endpoints for asynchronous uni-directional streams of bytes.
+  Currently these also suffer from a double-copy problem, and should probably
+  be replaced eventually by userspace shared memory communication.
+- _Events_: objects that can be signalled to send asynchronous notifications to
+  waiting threads.
+
+#### Hardware Support
+
+- Framebuffer driver: _In progress._ Currently on machines with a video
+  device accessible by UEFI, jsix starts a user-space framebuffer driver that
+  only prints out kernel logs.
+- Serial driver: _In progress._ The current UART currently only exposes COM1
+  as an output channel, but no input or other serial ports are exposed.
+- USB driver: _To do_
+- AHCI (SATA) driver: _To do_
+
 ## Building
 
-jsix uses the [Ninja][] build tool, and generates the build files for it with a
+jsix uses the [Ninja][] build tool, and generates the build files for it with
-custom tool called [Bonnibel][]. Bonnibel can be installed with [Cargo][], or
+the `configure` script. The build also relies on a custom toolchain sysroot, which can be
-downloaded as a prebuilt binary from its Github repository.
+downloaded or built using the scripts in [jsix-os/toolchain][].
 
-[Ninja]:    https://ninja-build.org
+[Ninja]:             https://ninja-build.org
-[Bonnibel]: https://github.com/justinian/bonnibel
+[jsix-os/toolchain]: https://github.com/jsix-os/toolchain
-[Cargo]:    https://crates.io/crates/bonnibel
 
-Requrirements:
+Other build dependencies:
 
-* bonnibel
+* [clang][]: the C/C++ compiler
-* ninja
+* [nasm][]: the assembler
-* clang
+* [lld][]: the linker
-* nasm
+* [mtools][]: for creating the FAT image
-* mtools
-* curl for downloading the toolchain
 
-### Setting up the cross toolchain
+[clang]:    https://clang.llvm.org
+[nasm]:     https://www.nasm.us
+[lld]:      https://lld.llvm.org
+[mtools]:   https://www.gnu.org/software/mtools/
 
-Running `pb sync` will download and unpack the toolchain into `sysroot`. 
+The `configure` script has some Python dependencies - these can be installed via
+`pip`, though doing so in a python virtual environment is recommended.
+Installing via `pip` will also install `ninja`.
 
-#### Compiling the toolchain yourself
+A Debian 11 (Bullseye) system can be configured with the necessary build
+dependencies by running the following commands from the jsix repository root:
 
-If you have `clang` and `curl` installed, runing the `scripts/build_sysroot.sh`
+```bash
-script will download and build a LLVM toolchain configured for building jsix
+sudo apt install clang lld nasm mtools python3-pip python3-venv
-host binaries.
+python3 -m venv ./venv
+source venv/bin/activate
+pip install -r requirements.txt
+peru sync
+```
+
+### Setting up the sysroot
+
+Build or download the toolchain sysroot as mentioned above with
+[jsix-os/toolchain][], and symlink the built toolchain directory as `sysroot`
+at the root of this project.
+
+```bash
+# Example if both the toolchain and this project are cloned under ~/src
+ln -s ~/src/toolchain/toolchains/llvm-13 ~/src/jsix/sysroot
+```
 
 ### Building and running jsix
 
-Once the toolchain has been set up, running Bonnibel's `pb init` command will
+Once the toolchain has been set up, running the `./configure` script (see
-set up the build configuration, and `pb build` will actually run the build.  If
+`./configure --help` for available options) will set up the build configuration,
-you have `qemu-system-x86_64` installed, the `qemu.sh` script will to run jsix
+and `ninja -C build` (or wherever you put the build directory) will actually run
-in QEMU `-nographic` mode.
+the build. If you have `qemu-system-x86_64` installed, the `qemu.sh` script will
+to run jsix in QEMU `-nographic` mode.
 
-I personally run this either from a real debian amd64 testing/buster machine or
+I personally run this either from a real debian amd64 bullseye machine or
-a windows WSL debian testing/buster installation. The following should be
+a windows WSL debian bullseye installation. Your mileage may vary with other
-enough to set up such a system to build the kernel:
+setups and distros.
 
-    sudo apt install qemu-system-x86 nasm clang-6.0 mtools curl
+### Running the test suite
-    sudo update-alternatives /usr/bin/clang clang /usr/bin/clang-6.0 1000
-    sudo update-alternatives /usr/bin/clang++ clang++ /usr/bin/clang++-6.0 1000
-	curl -L -o pb https://github.com/justinian/bonnibel/releases/download/2.0.0/pb_linux_amd64 && chmod a+x pb
 
+jsix now has the `test_runner` userspace program that runs various automated
+tests. It is not included in the default build, but if you use the `test.yml`
+manifest it will be built, and can be run with the `test.sh` script or the
+`qemu.sh` script.
+
+```bash
+./configure --manifest=assets/manifests/test.yml
+if ./test.sh; then echo "All tests passed!"; else echo "Failed."; fi
+```

assets/debugging/jsix.elf-gdb.py

@@ -10,16 +10,42 @@ class PrintStackCommand(gdb.Command):
         base = "$rsp"
         if len(args) > 0:
             base = args[0]
+        base = int(gdb.parse_and_eval(base))
 
         depth = 22
         if len(args) > 1:
             depth = int(args[1])
 
         for i in range(depth-1, -1, -1):
-            offset = i * 8
+            try:
-            base_addr = gdb.parse_and_eval(base)
+                offset = i * 8
-            value = gdb.parse_and_eval(f"*(uint64_t*)({base} + {offset})")
+                value = gdb.parse_and_eval(f"*(uint64_t*)({base:#x} + {offset:#x})")
-            print("{:016x} (+{:04x}): {:016x}".format(int(base_addr) + offset, offset, int(value)))
+                print("{:016x} (+{:04x}): {:016x}".format(base + offset, offset, int(value)))
+            except Exception as e:
+                print(e)
+                continue
+
+
+def stack_walk(frame, depth):
+    for i in range(depth-1, -1, -1):
+        ret = gdb.parse_and_eval(f"*(uint64_t*)({frame:#x} + 0x8)")
+
+        name = ""
+        try:
+            block = gdb.block_for_pc(int(ret))
+            if block:
+                name = block.function or ""
+        except RuntimeError:
+            pass
+
+        try:
+            print("{:016x}: {:016x} {}".format(int(frame), int(ret), name))
+            frame = int(gdb.parse_and_eval(f"*(uint64_t*)({frame:#x})"))
+        except gdb.MemoryError:
+            return
+
+        if frame == 0 or ret == 0:
+            return
 
 
 class PrintBacktraceCommand(gdb.Command):
@@ -33,27 +59,211 @@ class PrintBacktraceCommand(gdb.Command):
         if len(args) > 0:
             frame = args[0]
 
+        frame = int(gdb.parse_and_eval(f"{frame}"))
+
         depth = 30
         if len(args) > 1:
-            depth = int(args[1])
+            depth = int(gdb.parse_and_eval(args[1]))
 
-        for i in range(depth-1, -1, -1):
+        stack_walk(frame, depth)
-            ret = gdb.parse_and_eval(f"*(uint64_t*)({frame} + 8)")
-            frame = gdb.parse_and_eval(f"*(uint64_t*)({frame})")
 
-            name = ""
-            block = gdb.block_for_pc(int(ret))
-            if block:
-                name = block.function or ""
 
-            print("{:016x} {}".format(int(ret), name))
+class TableWalkCommand(gdb.Command):
+    def __init__(self):
+        super().__init__("j6tw", gdb.COMMAND_DATA)
 
-            if frame == 0 or ret == 0:
+    def invoke(self, arg, from_tty):
-                return
+        args = gdb.string_to_argv(arg)
+        if len(args) < 2:
+            raise Exception("Must be: j6tw <pml4> <addr>")
 
+        pml4 = int(gdb.parse_and_eval(args[0]))
+        addr = int(gdb.parse_and_eval(args[1]))
+
+        indices = [
+            (addr >> 39) & 0x1ff,
+            (addr >> 30) & 0x1ff,
+            (addr >> 21) & 0x1ff,
+            (addr >> 12) & 0x1ff,
+            ]
+
+        names = ["PML4", "PDP", "PD", "PT"]
+
+        table_flags = [
+            (0x0001, "present"),
+            (0x0002, "write"),
+            (0x0004, "user"),
+            (0x0008, "pwt"),
+            (0x0010, "pcd"),
+            (0x0020, "accessed"),
+            (0x0040, "dirty"),
+            (0x0080, "largepage"),
+            (0x0100, "global"),
+            (0x1080, "pat"),
+            ((1<<63), "xd"),
+        ]
+
+        page_flags = [
+            (0x0001, "present"),
+            (0x0002, "write"),
+            (0x0004, "user"),
+            (0x0008, "pwt"),
+            (0x0010, "pcd"),
+            (0x0020, "accessed"),
+            (0x0040, "dirty"),
+            (0x0080, "pat"),
+            (0x0100, "global"),
+            ((1<<63), "xd"),
+        ]
+
+        flagsets = [table_flags, table_flags, table_flags, page_flags]
+
+        table = pml4
+        entry = 0
+        for i in range(len(indices)):
+            entry = int(gdb.parse_and_eval(f'((uint64_t*)0x{table:x})[0x{indices[i]:x}]'))
+            flagset = flagsets[i]
+            flag_names = " | ".join([f[1] for f in flagset if (entry & f[0]) == f[0]])
+
+            print(f"{names[i]:>4}: {table:016x}")
+            print(f"      index: {indices[i]:3} {entry:016x}")
+            print(f"      flags: {flag_names}")
+
+            if (entry & 1) == 0 or (i < 3 and (entry & 0x80)):
+                break
+
+            table = (entry & 0x7ffffffffffffe00) | 0xffffc00000000000
+
+
+class GetThreadsCommand(gdb.Command):
+    FLAGS = {
+        "ready":    0x01,
+        "loading":  0x02,
+        "exited":   0x04,
+        "constant": 0x80,
+    }
+
+    def __init__(self):
+        super().__init__("j6threads", gdb.COMMAND_DATA)
+
+    def get_flags(self, bitset):
+        flags = []
+        for k, v in GetThreadsCommand.FLAGS.items():
+            if bitset & v:
+                flags.append(k)
+        return " ".join(flags)
+
+    def print_thread(self, addr):
+        if addr == 0:
+            print("  <no thread>\n")
+            return
+
+        tcb = f"((TCB*){addr:#x})"
+        thread = f"({tcb}->thread)"
+        stack = int(gdb.parse_and_eval(f"{tcb}->kernel_stack"))
+        rsp = int(gdb.parse_and_eval(f"{tcb}->rsp"))
+        pri = int(gdb.parse_and_eval(f"{tcb}->priority"))
+        flags = int(gdb.parse_and_eval(f"{thread}->m_state"))
+        koid = int(gdb.parse_and_eval(f"{thread}->m_koid"))
+        proc = int(gdb.parse_and_eval(f"{thread}->m_parent.m_koid"))
+
+        creator = int(gdb.parse_and_eval(f"{thread}->m_creator"))
+        if creator != 0:
+            creator_koid = int(gdb.parse_and_eval(f"{thread}->m_creator->m_koid"))
+            creator = f"{creator_koid:x}"
+        else:
+            creator = "<no thread>"
+
+        print(f"  Thread {proc:x}:{koid:x}")
+        print(f"         creator: {creator}")
+        print(f"        priority: {pri}")
+        print(f"           flags: {self.get_flags(flags)}")
+        print(f"          kstack: {stack:#x}")
+        print(f"             rsp: {rsp:#x}")
+        print("------------------------------------")
+
+        if stack != 0:
+            rsp = int(gdb.parse_and_eval(f"{tcb}->rsp"))
+            stack_walk(rsp + 5*8, 5)
+
+        print("")
+
+    def print_thread_list(self, addr, name):
+        if addr == 0:
+            return
+
+        print(f"=== {name} ===")
+
+        while addr != 0:
+            self.print_thread(addr)
+            addr = int(gdb.parse_and_eval(f"((tcb_node*){addr:#x})->m_next"))
+
+
+    def invoke(self, arg, from_tty):
+        args = gdb.string_to_argv(arg)
+        if len(args) > 1:
+            raise RuntimeError("Usage: j6threads [cpu]")
+
+        ncpus = int(gdb.parse_and_eval("g_num_cpus"))
+        cpus = list(range(ncpus))
+        if len(args) == 1:
+            cpus = [int(args[0])]
+
+        for cpu in cpus:
+            runlist = f"scheduler::s_instance->m_run_queues.m_elements[{cpu:#x}]"
+
+            print(f"=== CPU {cpu:2}: CURRENT ===")
+            current = int(gdb.parse_and_eval(f"{runlist}.current"))
+            self.print_thread(current)
+
+            for pri in range(8):
+                ready = int(gdb.parse_and_eval(f"{runlist}.ready[{pri:#x}].m_head"))
+                self.print_thread_list(ready, f"CPU {cpu:2}: PRIORITY {pri}")
+
+            blocked = int(gdb.parse_and_eval(f"{runlist}.blocked.m_head"))
+            self.print_thread_list(blocked, f"CPU {cpu:2}: BLOCKED")
+
+
+class PrintProfilesCommand(gdb.Command):
+    def __init__(self):
+        super().__init__("j6prof", gdb.COMMAND_DATA)
+
+    def invoke(self, arg, from_tty):
+        args = gdb.string_to_argv(arg)
+        if len(args) != 1:
+            raise RuntimeError("Usage: j6prof <profiler class>")
+
+        profclass = args[0]
+        root_type = f"profile_class<{profclass}>"
+
+        try:
+            baseclass = gdb.lookup_type(root_type)
+        except Exception as e:
+            print(e)
+            return
+
+        results = {}
+        max_len = 0
+        count = gdb.parse_and_eval(f"{profclass}::count")
+        for i in range(count):
+            name = gdb.parse_and_eval(f"{root_type}::function_names[{i:#x}]")
+            if name == 0: continue
+
+            call_counts = gdb.parse_and_eval(f"{root_type}::call_counts[{i:#x}]")
+            call_durations = gdb.parse_and_eval(f"{root_type}::call_durations[{i:#x}]")
+            results[name.string()] = float(call_durations) / float(call_counts)
+            max_len = max(max_len, len(name.string()))
+
+        for name, avg in results.items():
+            print(f"{name:>{max_len}}: {avg:15.3f}")
 
 PrintStackCommand()
 PrintBacktraceCommand()
+TableWalkCommand()
+GetThreadsCommand()
+PrintProfilesCommand()
 
-gdb.execute("target remote :1234")
 gdb.execute("display/i $rip")
+if not gdb.selected_inferior().was_attached:
+    gdb.execute("add-symbol-file build/panic.serial.elf")
+    gdb.execute("target remote :1234")

assets/grammars/definitions.g

@@ -0,0 +1,47 @@
+start: import_statement* (object|interface)+
+
+import_statement: "import" PATH
+
+object: description? "object" name options? super? "{" uid cname? capabilities? method* "}"
+
+interface: description? "interface" name options? "{" uid interface_param* "}"
+
+?interface_param: expose | function
+
+expose: "expose" type
+
+uid: "uid" UID
+
+cname: "cname" IDENTIFIER
+
+capabilities: "capabilities" "[" IDENTIFIER+ "]"
+
+super: ":" name
+
+function: description? "function" name options? ("{" param* "}")?
+
+method: description? "method" name options? ("{" param* "}")?
+
+param: "param" name type options? description?
+
+?type: PRIMITIVE | object_name
+
+object_name: "ref" name
+
+id: NUMBER
+name: IDENTIFIER
+options: "[" ( OPTION | IDENTIFIER )+ "]"
+description: COMMENT+
+
+PRIMITIVE: INT_TYPE | "size" | "string" | "buffer" | "address"
+INT_TYPE: /u?int(8|16|32|64)?/
+NUMBER: /(0x)?[0-9a-fA-F]+/
+UID: /[0-9a-fA-F]{16}/
+OPTION.2: IDENTIFIER ":" IDENTIFIER
+COMMENT: /#.*/
+PATH: /"[^"]*"/
+
+%import common.LETTER
+%import common.CNAME -> IDENTIFIER
+%import common.WS
+%ignore WS

assets/initrd.toml

@@ -1,28 +0,0 @@
-# This is the manifest for the initial ramdisk, read by the `makerd` tool.
-# The contents should be a table array of files to add to the ramdistk:
-#
-#  [[files]]
-#  dest = "foo.bar"             # Name of the file in the ramdisk
-#  source = "build/foo/foo.bar" # Location of the file from the project root
-#  executable = true            # Optional, default false. Whether this is an
-#                               # initial application for the kernel to execute
-#                               # on startup
-
-[[files]]
-dest = "screenfont.psf"
-source = "../assets/fonts/tamsyn8x16r.psf"
-
-[[files]]
-dest = "symbol_table.dat"
-source = "symbol_table.dat"
-symbols = true
-
-[[files]]
-dest = "nulldrv1"
-source = "user/nulldrv"
-executable = true
-
-[[files]]
-dest = "nulldrv2"
-source = "user/nulldrv"
-executable = true

assets/jsix.svg

@@ -0,0 +1 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?><!-- Generator: Gravit.io --><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" style="isolation:isolate" viewBox="176.562 356.069 211.11 113" width="211.11pt" height="113pt"><g><g><rect x="176.562" y="356.069" width="211.11" height="113" transform="matrix(1,0,0,1,0,0)" fill="rgb(255,255,255)"/><g><path d=" M 212.981 372.36 L 219.564 376.16 L 226.147 379.961 L 226.147 387.563 L 226.147 395.164 L 219.564 398.965 L 212.981 402.766 L 206.398 398.965 L 199.815 395.164 L 199.815 387.563 L 199.815 379.961 L 206.398 376.16 L 212.981 372.36 L 212.981 372.36 L 212.981 372.36 Z  M 256.292 397.366 L 262.875 401.166 L 269.458 404.967 L 269.458 412.569 L 269.458 420.17 L 262.875 423.971 L 256.292 427.772 L 249.709 423.971 L 243.126 420.17 L 243.126 412.569 L 243.126 404.967 L 249.709 401.166 L 256.292 397.366 L 256.292 397.366 Z  M 183.622 387.283 L 205.52 374.64 L 227.418 361.997 L 249.316 374.64 L 271.214 387.283 L 271.214 412.569 L 271.214 437.854 L 249.316 450.497 L 227.418 463.14 L 205.52 450.497 L 183.622 437.854 L 183.622 412.569 L 183.622 387.283 L 183.622 387.283 L 183.622 387.283 Z  M 241.855 372.36 L 248.438 376.16 L 255.021 379.961 L 255.021 387.563 L 255.021 395.164 L 248.438 398.965 L 241.855 402.766 L 235.272 398.965 L 228.689 395.164 L 228.689 387.563 L 228.689 379.961 L 235.272 376.16 L 241.855 372.36 Z " fill-rule="evenodd" fill="rgb(49,79,128)"/><path d=" M 298.642 379.579 L 291.621 379.579 L 291.621 372.558 L 298.642 372.558 L 298.642 379.579 Z  M 285.214 446.718 L 285.214 441.452 L 287.32 441.452 L 287.32 441.452 Q 289.339 441.452 290.524 440.092 L 290.524 440.092 L 290.524 440.092 Q 291.708 438.731 291.708 436.625 L 291.708 436.625 L 291.708 387.039 L 298.729 387.039 L 298.729 436.011 L 298.729 436.011 Q 298.729 440.925 295.921 443.822 L 295.921 443.822 L 295.921 443.822 Q 293.113 446.718 288.286 446.718 L 288.286 446.718 L 285.214 446.718 Z  M 306.628 432.676 L 306.628 427.41 L 314.088 427.41 L 314.088 427.41 Q 317.862 427.41 319.573 425.347 L 319.573 425.347 L 319.573 425.347 Q 321.285 423.285 321.285 419.95 L 321.285 419.95 L 321.285 419.95 Q 321.285 417.317 319.705 415.474 L 319.705 415.474 L 319.705 415.474 Q 318.125 413.631 314.966 411.174 L 314.966 411.174 L 314.966 411.174 Q 312.245 408.98 310.621 407.356 L 310.621 407.356 L 310.621 407.356 Q 308.998 405.732 307.813 403.319 L 307.813 403.319 L 307.813 403.319 Q 306.628 400.905 306.628 397.746 L 306.628 397.746 L 306.628 397.746 Q 306.628 393.095 309.744 390.067 L 309.744 390.067 L 309.744 390.067 Q 312.859 387.039 318.125 387.039 L 318.125 387.039 L 325.76 387.039 L 325.76 392.305 L 319.441 392.305 L 319.441 392.305 Q 313.21 392.305 313.21 398.185 L 313.21 398.185 L 313.21 398.185 Q 313.21 400.467 314.615 402.134 L 314.615 402.134 L 314.615 402.134 Q 316.019 403.802 319.003 406.083 L 319.003 406.083 L 319.003 406.083 Q 321.723 408.19 323.479 409.901 L 323.479 409.901 L 323.479 409.901 Q 325.234 411.613 326.463 414.202 L 326.463 414.202 L 326.463 414.202 Q 327.691 416.791 327.691 420.301 L 327.691 420.301 L 327.691 420.301 Q 327.691 426.532 324.4 429.604 L 324.4 429.604 L 324.4 429.604 Q 321.109 432.676 315.141 432.676 L 315.141 432.676 L 306.628 432.676 Z  M 342.611 379.579 L 335.59 379.579 L 335.59 372.558 L 342.611 372.558 L 342.611 379.579 Z  M 342.611 432.676 L 335.59 432.676 L 335.59 387.039 L 342.611 387.039 L 342.611 432.676 Z  M 356.126 432.676 L 348.754 432.676 L 361.392 409.77 L 349.632 387.039 L 356.39 387.039 L 364.639 403.187 L 372.977 387.039 L 379.735 387.039 L 367.974 409.77 L 380.612 432.676 L 373.24 432.676 L 364.639 416.001 L 356.126 432.676 Z " fill="rgb(49,79,128)"/></g></g></g></svg>

assets/manifests/default.yaml

@@ -0,0 +1,8 @@
+---
+init: srv.init
+programs:
+  - name: panic.serial
+    target: kernel
+    flags: panic
+  - name: srv.logger
+  - name: drv.uart

assets/manifests/test.yaml

@@ -0,0 +1,9 @@
+---
+init: srv.init
+flags: ["test"]
+programs:
+  - name: panic.serial
+    target: kernel
+    flags: panic
+  - name: drv.uart
+  - name: test_runner

configs/base.yaml

@@ -0,0 +1,39 @@
+---
+variables:
+  cc: "${source_root}/sysroot/bin/clang"
+  cxx: "${source_root}/sysroot/bin/clang++"
+  ld: "${source_root}/sysroot/bin/ld.lld"
+  ar: ar
+  nasm: nasm
+  objcopy: objcopy
+
+  ccflags: [
+    "-I${source_root}/src/include",
+    "-I${source_root}/src/include/x86_64",
+    "-fcolor-diagnostics",
+    "-U__STDCPP_THREADS__",
+    "-D_LIBCPP_HAS_NO_THREADS",
+    "-DVERSION_MAJOR=${version_major}",
+    "-DVERSION_MINOR=${version_minor}",
+    "-DVERSION_PATCH=${version_patch}",
+    "-DVERSION_GITSHA=0x${version_sha}",
+    '-DGIT_VERSION=\"${version_major}.${version_minor}.${version_patch}+${version_sha}\"',
+    '-DGIT_VERSION_WIDE=L\"${version_major}.${version_minor}.${version_patch}+${version_sha}\"',
+
+    "-Wformat=2", "-Winit-self", "-Wfloat-equal", "-Winline", "-Wmissing-format-attribute",
+    "-Wmissing-include-dirs", "-Wswitch", "-Wundef", "-Wdisabled-optimization",
+    "-Wpointer-arith", "-Wno-attributes", "-Wno-sign-compare", "-Wno-multichar",
+    "-Wno-div-by-zero", "-Wno-endif-labels", "-Wno-pragmas", "-Wno-format-extra-args",
+    "-Wno-unused-result", "-Wno-deprecated-declarations", "-Wno-unused-function",
+    "-Wno-address-of-packed-member", "-Wno-invalid-offsetof", "-Wno-format-nonliteral",
+    "-Werror" ]
+
+  asflags: [
+    "-DVERSION_MAJOR=${version_major}",
+    "-DVERSION_MINOR=${version_minor}",
+    "-DVERSION_PATCH=${version_patch}",
+    "-DVERSION_GITSHA=0x${version_sha}",
+    "-I${source_root}/src/include" ]
+
+  cflags: [ "-std=c11" ]
+  cxxflags: [ "-std=c++17" ]

configs/boot-debug.yaml

@@ -0,0 +1,30 @@
+---
+extends: base
+
+variables:
+  ld: clang++
+
+  ccflags: [
+    "-nostdlib",
+    "-nodefaultlibs",
+    "-fno-builtin",
+
+    "-I${source_root}/external",
+    "--target=x86_64-unknown-windows",
+    "-ffreestanding",
+    "-mno-red-zone",
+    "-fshort-wchar",
+    "-fno-omit-frame-pointer",
+    "-ggdb",
+    "-g3" ]
+
+  cxxflags: [ "-fno-exceptions", "-fno-rtti" ]
+
+  ldflags: [
+    "--target=x86_64-unknown-windows",
+    "-nostdlib",
+    "-Wl,-entry:efi_main",
+    "-Wl,-subsystem:efi_application",
+    "-fuse-ld=lld-link",
+    "-g" ]
+

configs/kernel-debug.yaml

@@ -0,0 +1,55 @@
+---
+extends: base
+
+variables:
+  asflags: [ "-I${source_root}/src/kernel/" ]
+
+  ccflags: [
+    "--target=x86_64-jsix-elf",
+    "-fno-stack-protector",
+
+    "-I${source_root}/external",
+
+    "-nostdinc",
+    "-nostdlib",
+    "-ffreestanding",
+    "-nodefaultlibs",
+    "-fno-builtin",
+
+    "-mno-sse",
+    "-fno-omit-frame-pointer",
+    "-mno-red-zone",
+    "-mcmodel=kernel",
+
+    "-g3",
+    "-ggdb",
+
+    "-D__ELF__",
+    "-D__jsix__",
+    "-D__j6kernel",
+    "-U__linux",
+    "-U__linux__",
+    "-DPRINTF_ALIAS_STANDARD_FUNCTION_NAMES=1",
+    "-DPRINTF_INCLUDE_CONFIG_H=1",
+
+    "-isystem${build_root}/include/libc",
+    "-isystem${source_root}/sysroot/include",
+    "--sysroot='${source_root}/sysroot'" ]
+
+
+  cflags: [ '-nostdinc' ]
+
+  cxxflags: [
+    "-fno-exceptions",
+    "-fno-rtti",
+    "-nostdinc",
+    "-isystem${source_root}/sysroot/include/c++/v1" ]
+
+  ldflags: [
+    "-g",
+    "-nostdlib",
+    "-Bstatic",
+    "--no-eh-frame-hdr",
+    "-z", "norelro",
+    "-z", "separate-code" ]
+

configs/rules.ninja

@@ -0,0 +1,81 @@
+# This file is automatically generated by bonnibel
+
+rule compile.c
+  command = $cc -MMD -MF $out.d $cflags $ccflags -o $out -c $in
+  description = Compiling $name
+  depfile = $out.d
+  deps = gcc
+
+rule dump_c_defs
+  command = echo | $cc $ccflags $cflags -dM -E - > $out
+  description = Dumping C defines for $target
+
+rule dump_c_run
+  command = echo '#!/bin/bash' > $out; echo '$cc $ccflags $cflags $$*' >> $
+      $out; chmod a+x $out
+  description = Dumping C arguments for $target
+
+rule compile.cpp
+  command = $cxx -MMD -MF $out.d $cxxflags $ccflags -o $out -c $in
+  description = Compiling $name
+  depfile = $out.d
+  deps = gcc
+
+rule dump_cpp_defs
+  command = echo | $cxx -x c++ $ccflags $cxxflags -dM -E - > $out
+  description = Dumping C++ defines for $target
+
+rule dump_cpp_run
+  command = echo '#!/bin/bash' > $out; echo '$cxx $ccflags $cxxflags $$*' $
+      >> $out; chmod a+x $out
+  description = Dumping C++ arguments for $target
+
+rule compile.s
+  command = $nasm -o $out -felf64 -MD $out.d $asflags $in
+  description = Assembling $name
+  depfile = $out.d
+  deps = gcc
+
+rule parse.cog
+  command = cog -o $out -d -D target=$target $cogflags $in
+  description = Parsing $name
+
+rule exe
+  command = $ld $ldflags -o $out $in $libs
+  description = Linking $name
+
+rule lib
+  command = $ar qcs $out $in
+  description = Archiving $name
+
+rule cp
+  command = cp $in $out
+  description = Copying $name
+
+rule dump
+  command = objdump -DSC -M intel $in > $out
+  description = Dumping decompiled $name
+
+rule makest
+    description = Making symbol table
+    command = nm -n -S --demangle $in | ${source_root}/scripts/build_symbol_table.py $out
+
+rule makefat
+    description = Creating $name
+    command = $
+        cp $in $out; $
+        mcopy -s -D o -i $out@@1M ${build_root}/fatroot/* ::/
+
+rule strip
+    description = Stripping $name
+    command = $
+        cp $in $out; $
+        objcopy --only-keep-debug $out $debug; $
+        strip -g $out; $
+        objcopy --add-gnu-debuglink=$debug $out
+
+rule touch
+    command = touch $out
+
+rule compdb
+    command = ninja -t compdb > $out

configs/user-debug.yaml

@@ -0,0 +1,39 @@
+---
+extends: base
+
+variables:
+  asflags: [ "-I${source_root}/src/kernel/" ]
+
+  ccflags: [
+    "--target=x86_64-jsix-elf",
+    "-mno-sse",
+    "-fno-omit-frame-pointer",
+    "-fno-stack-protector",
+
+    "-g",
+
+    "-D__ELF__",
+    "-D__jsix__",
+    "-U__linux",
+    "-U__linux__",
+
+    "-isystem${source_root}/sysroot/include",
+    "-isystem${build_root}/include/libc",
+    "--sysroot='${source_root}/sysroot'" ]
+
+
+  cxxflags: [
+    "-fno-exceptions",
+    "-fno-rtti",
+    "-isystem${source_root}/sysroot/include/c++/v1" ]
+
+  ldflags: [
+    "-g",
+    "-Bstatic",
+    "--sysroot='${source_root}/sysroot'",
+    "-L", "${source_root}/sysroot/lib",
+    "-z", "separate-code",
+    "-lc++", "-lc++abi", "-lunwind",
+    "--no-dependent-libraries",
+  ]
+

configure

@@ -0,0 +1,70 @@
+#!/usr/bin/env python3
+
+def generate(output, config, manifest):
+    from os import makedirs, walk
+    from pathlib import Path
+    from bonnibel.module import Module
+    from bonnibel.project import Project
+
+    root = Path(__file__).parent.resolve()
+    project = Project(root)
+
+    output = root / output
+    sources = root / "src"
+    manifest = root / manifest
+
+    modules = {}
+    for base, dirs, files in walk(sources):
+        path = Path(base) 
+        for f in files:
+            if f.endswith(".module"):
+                fullpath = path / f
+                
+                def module_init(name, **kwargs):
+                    m = Module(name, fullpath, path, **kwargs)
+                    modules[m.name] = m
+                    return m
+
+                glo = {
+                    "module": module_init,
+                    "source_root": root,
+                    "built_root": output,
+                    "module_root": path,
+                }
+                code = compile(open(fullpath, 'r').read(), fullpath, "exec")
+
+                loc = {}
+                exec(code, glo, loc)
+
+    Module.update(modules)
+
+    makedirs(output.resolve(), exist_ok=True)
+    project.generate(root, output, modules, config, manifest)
+    for mod in modules.values():
+        mod.generate(output)
+
+if __name__ == "__main__":
+    import sys
+    from pathlib import Path
+    sys.path.insert(0, str(Path(__file__).parent / "scripts"))
+
+    from argparse import ArgumentParser
+    from bonnibel import BonnibelError
+
+    p = ArgumentParser(description="Generate jsix build files")
+    p.add_argument("--manifest", "-m", metavar="FILE", default="assets/manifests/default.yaml",
+            help="File to use as the system manifest")
+    p.add_argument("--config", "-c", metavar="NAME", default="debug",
+            help="Configuration to build (eg, 'debug' or 'release')")
+    p.add_argument("output", metavar="DIR", default="build", nargs='?',
+            help="Where to create the build root")
+
+    args = p.parse_args()
+
+    try:
+        generate(args.output, args.config, args.manifest)
+
+    except BonnibelError as be:
+        import sys
+        print(f"Error: {be}", file=sys.stderr)
+        sys.exit(1)

definitions/memory_layout.yaml

@@ -0,0 +1,20 @@
+---
+- name: linear
+  size: 64T
+  shared: true
+
+- name: bitmap
+  size: 1T
+  shared: true
+
+- name: heap
+  size: 64G
+
+- name: stacks
+  size: 64G
+
+- name: buffers
+  size: 64G
+
+- name: slabs
+  size: 64G

definitions/objects/channel.def

@@ -0,0 +1,21 @@
+object channel : object {
+    uid 3ea38b96aa0e54c8
+
+    capabilities [
+        send
+        receive
+        close
+    ]
+
+    method create [constructor] 
+    method close [destructor cap:close]
+
+    method send [cap:send] {
+        param data buffer [inout]
+    }
+
+    method receive [cap:receive] {
+        param data buffer [out]
+        param flags uint64
+    }
+}

definitions/objects/event.def

@@ -0,0 +1,22 @@
+object event : object {
+    uid f441e03da5516b1a
+
+    capabilities [
+        signal
+        wait
+    ]
+
+    method create [constructor]
+
+    # Signal events on this object
+    method signal [cap:signal] {
+        param signals uint64        # A bitset of which events to signal
+    }
+
+    # Wait for signaled events on this object
+    method wait [cap:wait] {
+        param signals uint64 [out]  # A bitset of which events were signaled
+        param timeout uint64        # Wait timeout in nanoseconds
+    }
+}
+

definitions/objects/mailbox.def

@@ -0,0 +1,65 @@
+# Mailboxes are objects that enable synchronous or asynchronous
+# IPC via short message-passing of bytes and handles.
+
+object mailbox : object {
+    uid 99934ad04ece1e07
+
+    capabilities [
+        send
+        receive
+        close
+    ]
+
+    method create [constructor]
+    method close [destructor cap:close]
+
+    # Asynchronously send a message to the reciever
+    method send [cap:send handle] {
+        param tag uint64
+        param data buffer [zero_ok]
+        param handles ref object [list]
+    }
+
+    # Receive a pending message, or block waiting for a message to
+    # arrive if block is true.
+    method receive [cap:receive] {
+        param tag uint64 [out]
+        param data buffer [out zero_ok]
+        param handles ref object [out list zero_ok]
+        param reply_tag uint16 [out optional]
+        param badge uint64 [out optional]
+        param flags uint64
+    }
+
+    # Send a message to the reciever, and block until a
+    # response is sent. Note that getting this response
+    # does not require the receive capability.
+    method call [cap:send handle] {
+        param tag uint64 [inout]
+        param data buffer [inout zero_ok]
+        param handles ref object [inout list zero_ok]
+    }
+
+    # Respond to a message sent using call. Note that this
+    # requires the receive capability and not the send capability.
+    method respond [cap:receive] {
+        param tag uint64
+        param data buffer [zero_ok]
+        param handles ref object [list zero_ok]
+        param reply_tag uint16
+    }
+
+    # Respond to a message sent using call, and wait for another
+    # message to arrive. Note that this does not require the send
+    # capability.
+    method respond_receive [cap:receive] {
+        param tag uint64 [inout]
+        param data buffer [inout zero_ok]
+        param data_in size
+        param handles ref object [inout list zero_ok]
+        param handles_in size
+        param reply_tag uint16 [inout]
+        param badge uint64 [out optional]
+        param flags uint64
+    }
+}

definitions/objects/object.def

@@ -0,0 +1,14 @@
+# The base type of all kernel-exposed objects
+object object [virtual] {
+    uid 667f61fb2cd57bb4
+    cname kobject
+
+    capabilities [
+        clone
+    ]
+
+    # Get the internal kernel object id of an object
+    method koid {
+        param koid uint64 [out]
+    }
+}

definitions/objects/process.def

@@ -0,0 +1,31 @@
+import "objects/object.def"
+
+# Processes are a collection of handles and a virtual memory
+# space inside which threads are run.
+
+object process : object {
+    uid 0c69ee0b7502ba31
+
+    capabilities [
+        kill
+        create_thread
+    ]
+
+    # Create a new empty process
+    method create [constructor]
+
+    # Stop all threads and exit the given process
+    method kill [destructor cap:kill]
+
+    # Stop all threads and exit the current process
+    method exit [static noreturn] {
+        param result int32    # The result to retrun to the parent process
+    }
+
+    # Give the given process a handle that points to the same
+    # object as the specified handle.
+    method give_handle {
+        param target ref object [handle]          # A handle in the caller process to send
+        param received ref object [out optional]  # The handle as the recipient will see it
+    }
+}

definitions/objects/system.def

@@ -0,0 +1,40 @@
+# The system object represents a handle to kernel functionality
+# needed by drivers and other priviledged services
+object system : object {
+    uid fa72506a2cf71a30
+
+    capabilities [
+        get_log
+        bind_irq
+        map_phys
+        change_iopl
+    ]
+
+    # Get a log line from the kernel log
+    method get_log [cap:get_log] {
+        param buffer buffer [out zero_ok]   # Buffer for the log message data structure
+    }
+
+    # Ask the kernel to send this process messages whenever
+    # the given IRQ fires
+    method bind_irq [cap:bind_irq] {
+        param dest ref event     # Event object that will receive messages
+        param irq uint           # IRQ number to bind
+        param signal uint        # Signal number on the event to bind to
+    }
+
+    # Create a VMA and map an area of physical memory into it,
+    # also mapping that VMA into the current process
+    method map_phys [cap:map_phys] {
+        param area ref vma [out] # Receives a handle to the VMA created
+        param phys address       # The physical address of the area
+        param size size          # Size of the area, in pages
+        param flags uint32       # Flags to apply to the created VMA
+    }
+
+    # Request the kernel change the IOPL for this process. The only values
+    # that make sense are 0 and 3.
+    method request_iopl [cap:change_iopl] {
+        param iopl uint             # The IOPL to set for this process
+    }
+}

definitions/objects/thread.def

@@ -0,0 +1,23 @@
+object thread : object {
+    uid 11f23e593d5761bd
+
+    capabilities [
+        kill
+    ]
+
+    method create [constructor] {
+        param process ref process [cap:create_thread]
+        param stack_top address
+        param entrypoint address
+    }
+
+    method kill [destructor cap:kill]
+
+    method exit [static] {
+        param status int32
+    }
+
+    method sleep [static] {
+        param duration uint64
+    }
+}

definitions/objects/vma.def

@@ -0,0 +1,36 @@
+import "objects/process.def"
+
+object vma : object {
+    uid d6a12b63b3ed3937
+    cname vm_area
+
+    capabilities [
+        map
+        unmap
+        resize
+    ]
+
+    method create [constructor] {
+        param size size
+        param flags uint32
+    }
+
+    method create_map [constructor cap:map] {
+        param size size
+        param address address
+        param flags uint32
+    }
+
+    method map [cap:map] {
+        param process ref process
+        param address address
+    }
+
+    method unmap [cap:unmap] {
+        param process ref process
+    }
+
+    method resize [cap:resize] {
+        param size size [inout]
+    }
+}

definitions/syscalls.def

@@ -0,0 +1,50 @@
+import "objects/object.def"
+
+import "objects/channel.def"
+import "objects/event.def"
+import "objects/mailbox.def"
+import "objects/process.def"
+import "objects/system.def"
+import "objects/thread.def"
+import "objects/vma.def"
+
+interface syscalls [syscall] {
+    uid 01d9b6a948961097
+
+    expose ref object
+    expose ref system
+    expose ref event
+    expose ref process
+    expose ref thread
+    expose ref mailbox
+    expose ref channel
+    expose ref vma
+
+    # Simple no-op syscall for testing
+    function noop
+
+    # Write a message to the kernel log
+    function log {
+        param message string
+    }
+
+    # Get a list of handles owned by this process. If the
+    # supplied list is not big enough, will set the size
+    # needed in `size` and return j6_err_insufficient
+    function handle_list {
+        param handles ref object [list inout zero_ok] # A list of handles to be filled
+    }
+
+    # Create a clone of an existing handle, possibly with
+    # some capabilities masked out.
+    function handle_clone {
+        param orig ref object [handle cap:clone] # The handle to clone
+        param clone ref object [out]             # The new handle
+        param mask uint32                        # The capability bitmask
+    }
+
+    # Testing mode only: Have the kernel finish and exit QEMU with the given exit code
+    function test_finish [test] {
+        param exit_code uint32
+    }
+}

definitions/sysconf.yaml

@@ -0,0 +1,34 @@
+---
+address: 0x6a360000
+vars:
+  - name: version_major
+    section: kernel
+    type: uint8_t
+
+  - name: version_minor
+    section: kernel
+    type: uint8_t
+
+  - name: version_patch
+    section: kernel
+    type: uint16_t
+
+  - name: version_gitsha
+    section: kernel
+    type: uint32_t
+
+  - name: page_size
+    section: sys
+    type: size_t
+
+  - name: large_page_size
+    section: sys
+    type: size_t
+
+  - name: huge_page_size
+    section: sys
+    type: size_t
+
+  - name: num_cpus
+    section: sys
+    type: uint32_t

external/uefi/boot_services.h

@@ -14,8 +14,10 @@
 namespace uefi {
 namespace bs_impl {
 	using allocate_pages = status (*)(allocate_type, memory_type, size_t, void**);
+	using free_pages = status (*)(void*, size_t);
 	using get_memory_map = status (*)(size_t*, memory_descriptor*, size_t*, size_t*, uint32_t*);
 	using allocate_pool = status (*)(memory_type, uint64_t, void**);
+	using free_pool = status (*)(void*);
 	using handle_protocol = status (*)(handle, const guid*, void**);
 	using create_event = status (*)(evt, tpl, event_notify, void*, event*);
 	using exit_boot_services = status (*)(handle, size_t);
@@ -35,10 +37,10 @@ struct boot_services {
 
 	// Memory Services
 	bs_impl::allocate_pages allocate_pages;
-	void *free_pages;
+	bs_impl::free_pages free_pages;
 	bs_impl::get_memory_map get_memory_map;
 	bs_impl::allocate_pool allocate_pool;
-	void *free_pool;
+	bs_impl::free_pool free_pool;
 
 	// Event & Timer Services
 	bs_impl::create_event create_event;

external/uefi/networking.h

@@ -0,0 +1,390 @@
+#pragma once
+#ifndef _uefi_networking_h_
+#define _uefi_networking_h_
+
+// This Source Code Form is part of the j6-uefi-headers project and is subject
+// to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was
+// not distributed with this file, You can obtain one at
+// http://mozilla.org/MPL/2.0/.
+
+#include <stdint.h>
+#include <uefi/types.h>
+
+namespace uefi {
+
+//
+// IPv4 definitions
+//
+struct ipv4_address
+{
+	uint8_t addr[4];
+};
+
+//
+// IPv6 definitions
+//
+struct ipv6_address
+{
+	uint8_t addr[16];
+};
+
+struct ip6_address_info
+{
+    ipv6_address address;
+    uint8_t prefix_length;
+};
+
+struct ip6_route_table
+{
+    ipv6_address gateway;
+    ipv6_address destination;
+    uint8_t prefix_length;
+};
+
+enum class ip6_neighbor_state : int {
+    incomplete,
+    reachable,
+    stale,
+    delay,
+    probe,
+}
+
+struct ip6_neighbor_cache
+{
+    ipv6_address neighbor;
+    mac_address link_address;
+    ip6_neighbor_state state;
+};
+
+enum class icmpv6_type : uint8_t
+{
+    dest_unreachable = 0x1,
+    packet_too_big = 0x2,
+    time_exceeded = 0x3,
+    parameter_problem = 0x4,
+    echo_request = 0x80,
+    echo_reply = 0x81,
+    listener_query = 0x82,
+    listener_report = 0x83,
+    listener_done = 0x84,
+    router_solicit = 0x85,
+    router_advertise = 0x86,
+    neighbor_solicit = 0x87,
+    neighbor_advertise = 0x88,
+    redirect = 0x89,
+    listener_report_2 = 0x8f,
+};
+
+enum class icmpv6_code : uint8_t
+{
+    // codes for icmpv6_type::dest_unreachable
+    no_route_to_dest = 0x0,
+    comm_prohibited = 0x1,
+    beyond_scope = 0x2,
+    addr_unreachable = 0x3,
+    port_unreachable = 0x4,
+    source_addr_failed = 0x5,
+    route_rejected = 0x6,
+
+    // codes for icmpv6_type::time_exceeded
+    timeout_hop_limit = 0x0,
+    timeout_reassemble = 0x1,
+
+    // codes for icmpv6_type::parameter_problem
+    erroneous_header = 0x0,
+    unrecognize_next_hdr = 0x1,
+    unrecognize_option = 0x2,
+};
+
+struct ip6_icmp_type
+{
+    icmpv6_type type;
+    icmpv6_code code;
+};
+
+struct ip6_config_data
+{
+    uint8_t default_protocol;
+    bool accept_any_protocol;
+    bool accept_icmp_errors;
+    bool accept_promiscuous;
+    ipv6_address destination_address;
+    ipv6_address station_address;
+    uint8_t traffic_class;
+    uint8_t hop_limit;
+    uint32_t flow_label;
+    uint32_t receive_timeout;
+    uint32_t transmit_timeout;
+};
+
+struct ip6_mode_data
+{
+    bool is_started;
+    uint32_t max_packet_size;
+    ip6_config_data config_data;
+    bool is_configured;
+    uint32_t address_count;
+    ip6_address_info * address_list;
+    uint32_t group_count;
+    ipv6_address * group_table;
+    uint32_t route_count;
+    ip6_route_table * route_table;
+    uint32_t neighbor_count;
+    ip6_neighbor_cache * neighbor_cache;
+    uint32_t prefix_count;
+    ip6_address_info * prefix_table;
+    uint32_t icmp_type_count;
+     * icmp_type_list;
+};
+
+struct ip6_header
+{
+    uint8_t traffic_class_h : 4;
+    uint8_t version : 4;
+    uint8_t flow_label_h : 4;
+    uint8_t traffic_class_l : 4;
+    uint16_t flow_label_l;
+    uint16_t payload_length;
+    uint8_t next_header;
+    uint8_t hop_limit;
+    ipv6_address source_address;
+    ipv6_address destination_address;
+} __attribute__ ((packed));
+
+struct ip6_fragment_data
+{
+    uint32_t fragment_length;
+    void *fragment_buffer;
+};
+
+struct ip6_override_data
+{
+    uint8_t protocol;
+    uint8_t hop_limit;
+    uint32_t flow_label;
+};
+
+struct ip6_receive_data
+{
+    time time_stamp;
+    event recycle_signal;
+    uint32_t header_length;
+    ip6_header *header;
+    uint32_t data_length;
+    uint32_t fragment_count;
+    ip6_fragment_data fragment_table[1];
+};
+
+struct ip6_transmit_data
+{
+    ipv6_address destination_address;
+    ip6_override_data *override_data;
+
+    uint32_t ext_hdrs_length;
+    void *ext_hdrs;
+    uint8_t next_header;
+    uint32_t data_length;
+    uint32_t fragment_count;
+    ip6_fragment_data fragment_table[1];
+};
+
+struct ip6_completion_token
+{
+    event event;
+    status status;
+    union {
+        ip6_receive_data *rx_data;
+        ip6_transmit_data *tx_data;
+    } packet;
+};
+
+enum class ip6_config_data_type : int
+{
+    interface_info,
+    alt_interface_id,
+    policy,
+    dup_addr_detect_transmits,
+    manual_address,
+    gateway,
+    dns_server,
+    maximum
+};
+
+struct ip6_config_interface_info
+{
+    wchar_t name[32];
+    uint8_t if_type;
+    uint32_t hw_address_size;
+    mac_address hw_address;
+    uint32_t address_info_count;
+    ip6_address_info *address_info;
+    uint32_t route_count;
+    ip6_route_table *route_table;
+};
+
+struct ip6_config_interface_id
+{
+    uint8_t id[8];
+};
+
+enum class ip6_config_policy : int
+{
+    manual,
+    automatic
+};
+
+struct ip6_config_dup_addr_detect_transmits
+{
+    uint32_t dup_addr_detect_transmits;
+};
+
+struct ip6_config_manual_address
+{
+    ipv6_address address;
+    bool is_anycast;
+    uint8_t prefix_length;
+};
+
+//
+// IP definitions
+//
+union ip_address
+{
+	uint8_t addr[4];
+	ipv4_address v4;
+	ipv6_address v6;
+};
+
+//
+// HTTP definitions
+//
+struct httpv4_access_point
+{
+	bool use_default_address;
+	ipv4_address local_address;
+	ipv4_address local_subnet;
+	uint16_t local_port;
+};
+
+struct httpv6_access_point
+{
+	ipv6_address local_address;
+	uint16_t local_port;
+};
+
+enum class http_version : int {
+	v10,
+	v11,
+	unsupported,
+};
+
+struct http_config_data
+{
+	http_version http_version;
+	uint32_t time_out_millisec;
+	bool local_address_is_ipv6;
+	union {
+		httpv4_access_point *ipv4_node;
+		httpv6_access_point *ipv6_node;
+	} access_point;
+};
+
+enum class http_method : int {
+	get,
+	post,
+	patch,
+	options,
+	connect,
+	head,
+	put,
+	delete_,
+	trace,
+};
+
+struct http_request_data
+{
+	http_method method;
+	wchar_t *url;
+};
+
+enum class http_status_code : int {
+	unsupported,
+	continue_,
+	switching_protocols,
+	ok,
+	created,
+	accepted,
+	non_authoritative_information,
+	no_content,
+	reset_content,
+	partial_content,
+	multiple_choices,
+	moved_permanently,
+	found,
+	see_other,
+	not_modified,
+	use_proxy,
+	temporary_redirect,
+	bad_request,
+	unauthorized,
+	payment_required,
+	forbidden,
+	not_found,
+	method_not_allowed,
+	not_acceptable,
+	proxy_authentication_required,
+	request_time_out,
+	conflict,
+	gone,
+	length_required,
+	precondition_failed,
+	request_entity_too_large,
+	request_uri_too_large,
+	unsupported_media_type,
+	requested_range_not_satisfied,
+	expectation_failed,
+	internal_server_error,
+	not_implemented,
+	bad_gateway,
+	service_unavailable,
+	gateway_timeout,
+	http_version_not_supported,
+	permanent_redirect,  // I hate your decisions, uefi.
+};
+
+struct http_response_data
+{
+	http_status_code status_code;
+};
+
+struct http_header
+{
+	char *field_name;
+	char *field_value;
+};
+
+struct http_message
+{
+	union {
+		http_request_data *request;
+		http_response_data *response;
+	} data;
+
+	size_t header_count;
+	http_header *headers;
+
+	size_t body_length;
+	void *body;
+};
+
+struct http_token
+{
+	event event;
+	status status;
+	http_message *message;
+};
+
+
+} // namespace uefi
+
+#endif

external/uefi/protos/device_path.h

@@ -16,6 +16,7 @@ struct device_path
 {
     static constexpr uefi::guid guid{ 0x09576e91,0x6d3f,0x11d2,{0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     uint8_t type;
     uint8_t sub_type;
     uint16_t length;

external/uefi/protos/file_info.h

@@ -16,6 +16,7 @@ struct file_info
 {
     static constexpr uefi::guid guid{ 0x09576e92,0x6d3f,0x11d2,{0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     uint64_t size;
     uint64_t file_size;
     uint64_t physical_size;

external/uefi/protos/graphics_output.h

@@ -17,6 +17,7 @@ struct graphics_output
 {
     static constexpr uefi::guid guid{ 0x9042a9de,0x23dc,0x4a38,{0x96,0xfb,0x7a,0xde,0xd0,0x80,0x51,0x6a} };
 
+
     inline uefi::status query_mode(uint32_t mode_number, uint64_t * size_of_info, uefi::graphics_output_mode_info ** info) {
         return _query_mode(this, mode_number, size_of_info, info);
     }

external/uefi/protos/http.h

@@ -0,0 +1,72 @@
+#pragma once
+#ifndef _uefi_protos_http_h_
+#define _uefi_protos_http_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+#include <uefi/networking.h>
+
+namespace uefi {
+namespace protos {
+struct http;
+
+struct http
+{
+    static constexpr uefi::guid guid{ 0x7a59b29b,0x910b,0x4171,{0x82,0x42,0xa8,0x5a,0x0d,0xf2,0x5b,0x5b} };
+    static constexpr uefi::guid service_binding{ 0xbdc8e6af,0xd9bc,0x4379,{0xa7,0x2a,0xe0,0xc4,0xe7,0x5d,0xae,0x1c} };
+
+
+    inline uefi::status get_mode_data(uefi::http_config_data * http_config_data) {
+        return _get_mode_data(this, http_config_data);
+    }
+
+    inline uefi::status configure(uefi::http_config_data * http_config_data) {
+        return _configure(this, http_config_data);
+    }
+
+    inline uefi::status request(uefi::http_token * token) {
+        return _request(this, token);
+    }
+
+    inline uefi::status cancel(uefi::http_token * token) {
+        return _cancel(this, token);
+    }
+
+    inline uefi::status response(uefi::http_token * token) {
+        return _response(this, token);
+    }
+
+    inline uefi::status poll() {
+        return _poll(this);
+    }
+
+
+protected:
+    using _get_mode_data_def = uefi::status (*)(uefi::protos::http *, uefi::http_config_data *);
+    _get_mode_data_def _get_mode_data;
+
+    using _configure_def = uefi::status (*)(uefi::protos::http *, uefi::http_config_data *);
+    _configure_def _configure;
+
+    using _request_def = uefi::status (*)(uefi::protos::http *, uefi::http_token *);
+    _request_def _request;
+
+    using _cancel_def = uefi::status (*)(uefi::protos::http *, uefi::http_token *);
+    _cancel_def _cancel;
+
+    using _response_def = uefi::status (*)(uefi::protos::http *, uefi::http_token *);
+    _response_def _response;
+
+    using _poll_def = uefi::status (*)(uefi::protos::http *);
+    _poll_def _poll;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_http_h_

external/uefi/protos/ip6.h

@@ -0,0 +1,93 @@
+#pragma once
+#ifndef _uefi_protos_ip6_h_
+#define _uefi_protos_ip6_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+#include <uefi/networking.h>
+
+namespace uefi {
+namespace protos {
+struct ip6;
+
+struct ip6
+{
+    static constexpr uefi::guid guid{ 0x2c8759d5,0x5c2d,0x66ef,{0x92,0x5f,0xb6,0x6c,0x10,0x19,0x57,0xe2} };
+    static constexpr uefi::guid service_binding{ 0xec835dd3,0xfe0f,0x617b,{0xa6,0x21,0xb3,0x50,0xc3,0xe1,0x33,0x88} };
+
+
+    inline uefi::status get_mode_data(uefi::ip6_mode_data * ip6_mode_data, uefi::managed_network_config_data * mnp_config_data, uefi::simple_network_mode * snp_config_data) {
+        return _get_mode_data(this, ip6_mode_data, mnp_config_data, snp_config_data);
+    }
+
+    inline uefi::status configure(uefi::ip6_config_data * ip6_config_data) {
+        return _configure(this, ip6_config_data);
+    }
+
+    inline uefi::status groups(bool join_flag, uefi::ipv6_address * group_address) {
+        return _groups(this, join_flag, group_address);
+    }
+
+    inline uefi::status routes(bool delete_route, uefi::ipv6_address * destination, uint8_t prefix_length, uefi::ipv6_address * gateway_address) {
+        return _routes(this, delete_route, destination, prefix_length, gateway_address);
+    }
+
+    inline uefi::status neighbors(bool delete_flag, uefi::ipv6_address * target_ip6_address, uefi::mac_address * target_link_address, uint32_t timeout, bool override) {
+        return _neighbors(this, delete_flag, target_ip6_address, target_link_address, timeout, override);
+    }
+
+    inline uefi::status transmit(uefi::ip6_completion_token * token) {
+        return _transmit(this, token);
+    }
+
+    inline uefi::status receive(uefi::ip6_completion_token * token) {
+        return _receive(this, token);
+    }
+
+    inline uefi::status cancel(uefi::ip6_completion_token * token) {
+        return _cancel(this, token);
+    }
+
+    inline uefi::status poll() {
+        return _poll(this);
+    }
+
+
+protected:
+    using _get_mode_data_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_mode_data *, uefi::managed_network_config_data *, uefi::simple_network_mode *);
+    _get_mode_data_def _get_mode_data;
+
+    using _configure_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_config_data *);
+    _configure_def _configure;
+
+    using _groups_def = uefi::status (*)(uefi::protos::ip6 *, bool, uefi::ipv6_address *);
+    _groups_def _groups;
+
+    using _routes_def = uefi::status (*)(uefi::protos::ip6 *, bool, uefi::ipv6_address *, uint8_t, uefi::ipv6_address *);
+    _routes_def _routes;
+
+    using _neighbors_def = uefi::status (*)(uefi::protos::ip6 *, bool, uefi::ipv6_address *, uefi::mac_address *, uint32_t, bool);
+    _neighbors_def _neighbors;
+
+    using _transmit_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_completion_token *);
+    _transmit_def _transmit;
+
+    using _receive_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_completion_token *);
+    _receive_def _receive;
+
+    using _cancel_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_completion_token *);
+    _cancel_def _cancel;
+
+    using _poll_def = uefi::status (*)(uefi::protos::ip6 *);
+    _poll_def _poll;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_ip6_h_

external/uefi/protos/ip6_config.h

@@ -0,0 +1,57 @@
+#pragma once
+#ifndef _uefi_protos_ip6_config_h_
+#define _uefi_protos_ip6_config_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+#include <uefi/networking.h>
+
+namespace uefi {
+namespace protos {
+struct ip6_config;
+
+struct ip6_config
+{
+    static constexpr uefi::guid guid{ 0x937fe521,0x95ae,0x4d1a,{0x89,0x29,0x48,0xbc,0xd9,0x0a,0xd3,0x1a} };
+
+
+    inline uefi::status set_data(uefi::ip6_config_data_type data_type, size_t data_size, void * data) {
+        return _set_data(this, data_type, data_size, data);
+    }
+
+    inline uefi::status get_data(uefi::ip6_config_data_type data_type, size_t data_size, void * data) {
+        return _get_data(this, data_type, data_size, data);
+    }
+
+    inline uefi::status register_data_notify(uefi::ip6_config_data_type data_type, uefi::event event) {
+        return _register_data_notify(this, data_type, event);
+    }
+
+    inline uefi::status unregister_data_notify(uefi::ip6_config_data_type data_type, uefi::event event) {
+        return _unregister_data_notify(this, data_type, event);
+    }
+
+
+protected:
+    using _set_data_def = uefi::status (*)(uefi::protos::ip6_config *, uefi::ip6_config_data_type, size_t, void *);
+    _set_data_def _set_data;
+
+    using _get_data_def = uefi::status (*)(uefi::protos::ip6_config *, uefi::ip6_config_data_type, size_t, void *);
+    _get_data_def _get_data;
+
+    using _register_data_notify_def = uefi::status (*)(uefi::protos::ip6_config *, uefi::ip6_config_data_type, uefi::event);
+    _register_data_notify_def _register_data_notify;
+
+    using _unregister_data_notify_def = uefi::status (*)(uefi::protos::ip6_config *, uefi::ip6_config_data_type, uefi::event);
+    _unregister_data_notify_def _unregister_data_notify;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_ip6_config_h_

external/uefi/protos/load_file.h

@@ -0,0 +1,36 @@
+#pragma once
+#ifndef _uefi_protos_load_file_h_
+#define _uefi_protos_load_file_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+#include <uefi/protos/device_path.h>
+
+namespace uefi {
+namespace protos {
+struct load_file;
+
+struct load_file
+{
+    static constexpr uefi::guid guid{ {0x56ec3091,0x954c,0x11d2,{0x8e,0x3f,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
+
+
+    inline uefi::status load_file(uefi::protos::device_path * file_path, bool boot_policy, size_t * buffer_size, void * buffer) {
+        return _load_file(this, file_path, boot_policy, buffer_size, buffer);
+    }
+
+
+protected:
+    using _load_file_def = uefi::status (*)(uefi::protos::load_file *, uefi::protos::device_path *, bool, size_t *, void *);
+    _load_file_def _load_file;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_load_file_h_

external/uefi/protos/loaded_image.h

@@ -18,6 +18,7 @@ struct loaded_image
 {
     static constexpr uefi::guid guid{ 0x5b1b31a1,0x9562,0x11d2,{0x8e,0x3f,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     inline uefi::status unload(uefi::handle image_handle) {
         return _unload(image_handle);
     }

external/uefi/protos/service_binding.h

@@ -0,0 +1,41 @@
+#pragma once
+#ifndef _uefi_protos_service_binding_h_
+#define _uefi_protos_service_binding_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+
+namespace uefi {
+namespace protos {
+struct service_binding;
+
+struct service_binding
+{
+
+
+    inline uefi::status create_child(uefi::handle * child_handle) {
+        return _create_child(this, child_handle);
+    }
+
+    inline uefi::status destroy_child(uefi::handle child_handle) {
+        return _destroy_child(this, child_handle);
+    }
+
+
+protected:
+    using _create_child_def = uefi::status (*)(uefi::protos::service_binding *, uefi::handle *);
+    _create_child_def _create_child;
+
+    using _destroy_child_def = uefi::status (*)(uefi::protos::service_binding *, uefi::handle);
+    _destroy_child_def _destroy_child;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_service_binding_h_

external/uefi/protos/simple_file_system.h

@@ -17,6 +17,7 @@ struct simple_file_system
 {
     static constexpr uefi::guid guid{ 0x0964e5b22,0x6459,0x11d2,{0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     inline uefi::status open_volume(uefi::protos::file ** root) {
         return _open_volume(this, root);
     }

external/uefi/protos/simple_text_output.h

@@ -17,6 +17,7 @@ struct simple_text_output
 {
     static constexpr uefi::guid guid{ 0x387477c2,0x69c7,0x11d2,{0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     inline uefi::status reset(bool extended_verification) {
         return _reset(this, extended_verification);
     }

modules.yaml

@@ -1,298 +0,0 @@
-name: jsix
-templates: scripts/templates
-modules:
-    kernel:
-        kind: exe
-        output: jsix.elf
-        target: host
-        deps:
-            - kutil
-        includes:
-            - src/kernel
-        source:
-            - src/kernel/apic.cpp
-            - src/kernel/assert.cpp
-            - src/kernel/boot.s
-            - src/kernel/clock.cpp
-            - src/kernel/console.cpp
-            - src/kernel/cpprt.cpp
-            - src/kernel/cpu.cpp
-            - src/kernel/debug.cpp
-            - src/kernel/debug.s
-            - src/kernel/device_manager.cpp
-            - src/kernel/font.cpp
-            - src/kernel/frame_allocator.cpp
-            - src/kernel/fs/gpt.cpp
-            - src/kernel/gdt.cpp
-            - src/kernel/gdt.s
-            - src/kernel/hpet.cpp
-            - src/kernel/interrupts.cpp
-            - src/kernel/interrupts.s
-            - src/kernel/io.cpp
-            - src/kernel/loader.s
-            - src/kernel/log.cpp
-            - src/kernel/main.cpp
-            - src/kernel/memory_bootstrap.cpp
-            - src/kernel/msr.cpp
-            - src/kernel/objects/channel.cpp
-            - src/kernel/objects/endpoint.cpp
-            - src/kernel/objects/kobject.cpp
-            - src/kernel/objects/thread.cpp
-            - src/kernel/objects/process.cpp
-            - src/kernel/objects/system.cpp
-            - src/kernel/objects/vm_area.cpp
-            - src/kernel/page_table.cpp
-            - src/kernel/pci.cpp
-            - src/kernel/scheduler.cpp
-            - src/kernel/screen.cpp
-            - src/kernel/serial.cpp
-            - src/kernel/symbol_table.cpp
-            - src/kernel/syscall.cpp
-            - src/kernel/syscall.s
-            - src/kernel/syscalls/channel.cpp
-            - src/kernel/syscalls/endpoint.cpp
-            - src/kernel/syscalls/object.cpp
-            - src/kernel/syscalls/process.cpp
-            - src/kernel/syscalls/system.cpp
-            - src/kernel/syscalls/thread.cpp
-            - src/kernel/syscalls/vm_area.cpp
-            - src/kernel/task.s
-            - src/kernel/vm_mapper.cpp
-            - src/kernel/vm_space.cpp
-
-    boot:
-        kind: exe
-        target: boot
-        output: boot.efi
-        source:
-            - src/boot/main.cpp
-            - src/boot/console.cpp
-            - src/boot/error.cpp
-            - src/boot/fs.cpp
-            - src/boot/hardware.cpp
-            - src/boot/loader.cpp
-            - src/boot/memory.cpp
-            - src/boot/paging.cpp
-            - src/boot/support.cpp
-
-    nulldrv:
-        kind: exe
-        target: user
-        output: nulldrv.elf
-        deps:
-          - libc
-        source:
-            - src/drivers/nulldrv/io.cpp
-            - src/drivers/nulldrv/main.cpp
-            - src/drivers/nulldrv/main.s
-            - src/drivers/nulldrv/serial.cpp
-
-    kutil:
-        kind: lib
-        output: libkutil.a
-        includes:
-            - src/libraries/kutil/include
-        source:
-            - src/libraries/kutil/assert.cpp
-            - src/libraries/kutil/bip_buffer.cpp
-            - src/libraries/kutil/heap_allocator.cpp
-            - src/libraries/kutil/logger.cpp
-            - src/libraries/kutil/memory.cpp
-            - src/libraries/kutil/printf.c
-
-
-    libc:
-        kind: lib
-        output: libc.a
-        includes:
-          - src/libraries/libc/include
-        target: user
-        defines:
-            - DISABLE_SSE
-            - LACKS_UNISTD_H
-            - LACKS_FCNTL_H
-            - LACKS_SYS_PARAM_H
-            - LACKS_SYS_MMAN_H
-            - LACKS_SCHED_H
-            - LACKS_STRINGS_H
-            - HAVE_MMAP=0
-              #- LACKS_STRING_H
-              #- LACKS_ERRNO_H
-              #- LACKS_STDLIB_H
-              #- LACKS_TIME_H
-        source:
-            - src/libraries/libc/arch/x86_64/_Exit.s
-            - src/libraries/libc/arch/x86_64/syscalls.s
-            - src/libraries/libc/ctype/isalnum.c
-            - src/libraries/libc/ctype/isalpha.c
-            - src/libraries/libc/ctype/isblank.c
-            - src/libraries/libc/ctype/iscntrl.c
-            - src/libraries/libc/ctype/isdigit.c
-            - src/libraries/libc/ctype/isgraph.c
-            - src/libraries/libc/ctype/islower.c
-            - src/libraries/libc/ctype/isprint.c
-            - src/libraries/libc/ctype/ispunct.c
-            - src/libraries/libc/ctype/isspace.c
-            - src/libraries/libc/ctype/isupper.c
-            - src/libraries/libc/ctype/isxdigit.c
-            - src/libraries/libc/ctype/tolower.c
-            - src/libraries/libc/ctype/toupper.c
-            - src/libraries/libc/inttypes/imaxabs.c
-            - src/libraries/libc/inttypes/imaxdiv.c
-            - src/libraries/libc/inttypes/strtoimax.c
-            - src/libraries/libc/inttypes/strtoumax.c
-            - src/libraries/libc/locale/localeconv.c
-            - src/libraries/libc/locale/setlocale.c
-            - src/libraries/libc/j6libc/assert.c
-            - src/libraries/libc/j6libc/errno.c
-            - src/libraries/libc/j6libc/allocpages.c
-            - src/libraries/libc/j6libc/atomax.c
-            - src/libraries/libc/j6libc/closeall.c
-            - src/libraries/libc/j6libc/close.c
-            - src/libraries/libc/j6libc/digits.c
-            - src/libraries/libc/j6libc/filemode.c
-            - src/libraries/libc/j6libc/fillbuffer.c
-            - src/libraries/libc/j6libc/flushbuffer.c
-            - src/libraries/libc/j6libc/is_leap.c
-            - src/libraries/libc/j6libc/load_lc_collate.c
-            - src/libraries/libc/j6libc/load_lc_ctype.c
-            - src/libraries/libc/j6libc/load_lc_messages.c
-            - src/libraries/libc/j6libc/load_lc_monetary.c
-            - src/libraries/libc/j6libc/load_lc_numeric.c
-            - src/libraries/libc/j6libc/load_lc_time.c
-            - src/libraries/libc/j6libc/load_lines.c
-            - src/libraries/libc/j6libc/open.c
-            - src/libraries/libc/j6libc/prepread.c
-            - src/libraries/libc/j6libc/prepwrite.c
-            - src/libraries/libc/j6libc/print.c
-            - src/libraries/libc/j6libc/rename.c
-            - src/libraries/libc/j6libc/scan.c
-            - src/libraries/libc/j6libc/seed.c
-            - src/libraries/libc/j6libc/seek.c
-            - src/libraries/libc/j6libc/stdinit.c
-            - src/libraries/libc/j6libc/strtox_main.c
-            - src/libraries/libc/j6libc/strtox_prelim.c
-            - src/libraries/libc/j6libc/sbrk.c
-            - src/libraries/libc/signal/raise.c
-            - src/libraries/libc/signal/signal.c
-            - src/libraries/libc/stdio/clearerr.c
-            - src/libraries/libc/stdio/fclose.c
-            - src/libraries/libc/stdio/feof.c
-            - src/libraries/libc/stdio/ferror.c
-            - src/libraries/libc/stdio/fflush.c
-            - src/libraries/libc/stdio/fgetc.c
-            - src/libraries/libc/stdio/fgetpos.c
-            - src/libraries/libc/stdio/fgets.c
-            - src/libraries/libc/stdio/fopen.c
-            - src/libraries/libc/stdio/fprintf.c
-            - src/libraries/libc/stdio/fputc.c
-            - src/libraries/libc/stdio/fputs.c
-            - src/libraries/libc/stdio/fread.c
-            - src/libraries/libc/stdio/freopen.c
-            - src/libraries/libc/stdio/fscanf.c
-            - src/libraries/libc/stdio/fseek.c
-            - src/libraries/libc/stdio/fsetpos.c
-            - src/libraries/libc/stdio/ftell.c
-            - src/libraries/libc/stdio/fwrite.c
-            - src/libraries/libc/stdio/getc.c
-            - src/libraries/libc/stdio/getchar.c
-            - src/libraries/libc/stdio/perror.c
-            - src/libraries/libc/stdio/printf.c
-            - src/libraries/libc/stdio/putc.c
-            - src/libraries/libc/stdio/putchar.c
-            - src/libraries/libc/stdio/puts.c
-            - src/libraries/libc/stdio/remove.c
-            - src/libraries/libc/stdio/rename.c
-            - src/libraries/libc/stdio/rewind.c
-            - src/libraries/libc/stdio/scanf.c
-            - src/libraries/libc/stdio/setbuf.c
-            - src/libraries/libc/stdio/setvbuf.c
-            - src/libraries/libc/stdio/snprintf.c
-            - src/libraries/libc/stdio/sprintf.c
-            - src/libraries/libc/stdio/sscanf.c
-            - src/libraries/libc/stdio/tmpfile.c
-            - src/libraries/libc/stdio/tmpnam.c
-            - src/libraries/libc/stdio/ungetc.c
-            - src/libraries/libc/stdio/vfprintf.c
-            - src/libraries/libc/stdio/vfscanf.c
-            - src/libraries/libc/stdio/vprintf.c
-            - src/libraries/libc/stdio/vscanf.c
-            - src/libraries/libc/stdio/vsnprintf.c
-            - src/libraries/libc/stdio/vsprintf.c
-            - src/libraries/libc/stdio/vsscanf.c
-            - src/libraries/libc/stdlib/abort.c
-            - src/libraries/libc/stdlib/abs.c
-            - src/libraries/libc/stdlib/atexit.c
-            - src/libraries/libc/stdlib/atoi.c
-            - src/libraries/libc/stdlib/atol.c
-            - src/libraries/libc/stdlib/atoll.c
-            - src/libraries/libc/stdlib/bsearch.c
-            - src/libraries/libc/stdlib/div.c
-            - src/libraries/libc/stdlib/exit.c
-            - src/libraries/libc/stdlib/_Exit.c
-            - src/libraries/libc/stdlib/getenv.c
-            - src/libraries/libc/stdlib/labs.c
-            - src/libraries/libc/stdlib/ldiv.c
-            - src/libraries/libc/stdlib/llabs.c
-            - src/libraries/libc/stdlib/lldiv.c
-            - src/libraries/libc/stdlib/malloc.c
-            - src/libraries/libc/stdlib/qsort.c
-            - src/libraries/libc/stdlib/rand.c
-            - src/libraries/libc/stdlib/srand.c
-            - src/libraries/libc/stdlib/strtol.c
-            - src/libraries/libc/stdlib/strtoll.c
-            - src/libraries/libc/stdlib/strtoul.c
-            - src/libraries/libc/stdlib/strtoull.c
-            - src/libraries/libc/stdlib/system.c
-            - src/libraries/libc/string/memchr.c
-            - src/libraries/libc/string/memcmp.c
-            - src/libraries/libc/string/memcpy.c
-            - src/libraries/libc/string/memmove.c
-            - src/libraries/libc/string/memset.c
-            - src/libraries/libc/string/strcat.c
-            - src/libraries/libc/string/strchr.c
-            - src/libraries/libc/string/strcmp.c
-            - src/libraries/libc/string/strcoll.c
-            - src/libraries/libc/string/strcpy.c
-            - src/libraries/libc/string/strcspn.c
-            - src/libraries/libc/string/strerror.c
-            - src/libraries/libc/string/strlen.c
-            - src/libraries/libc/string/strncat.c
-            - src/libraries/libc/string/strncmp.c
-            - src/libraries/libc/string/strncpy.c
-            - src/libraries/libc/string/strpbrk.c
-            - src/libraries/libc/string/strrchr.c
-            - src/libraries/libc/string/strspn.c
-            - src/libraries/libc/string/strstr.c
-            - src/libraries/libc/string/strtok.c
-            - src/libraries/libc/string/strxfrm.c
-            - src/libraries/libc/time/asctime.c
-            - src/libraries/libc/time/clock.c
-            - src/libraries/libc/time/ctime.c
-            - src/libraries/libc/time/difftime.c
-            - src/libraries/libc/time/gmtime.c
-            - src/libraries/libc/time/localtime.c
-            - src/libraries/libc/time/mktime.c
-            - src/libraries/libc/time/strftime.c
-            - src/libraries/libc/time/time.c
-            - src/libraries/libc/time/timespec_get.c
-
-    tests:
-        kind: exe
-        target: native
-        output: tests
-        deps:
-            - kutil
-        source:
-            - src/tests/constexpr_hash.cpp
-            - src/tests/linked_list.cpp
-            - src/tests/logger.cpp
-            - src/tests/heap_allocator.cpp
-            - src/tests/main.cpp
-            - src/tests/map.cpp
-            - src/tests/vector.cpp
-overlays:
-    - url: https://f000.backblazeb2.com/file/jsix-os/sysroot-llvm8-20190706.tar.bz2
-      path: sysroot
-    - url: https://f000.backblazeb2.com/file/jsix-os/sysroot-j6libc-8cb8ce7.tar.bz2
-      path: sysroot

other_software.md

@@ -33,42 +33,18 @@ Boost Software license:
 > ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
 > DEALINGS IN THE SOFTWARE.
 
-## cpptoml
-
-jsix uses the [cpptoml][] library for parsing TOML configuration files. cpptoml
-is released under the terms of the MIT license:
-
-[cpptoml]: https://github.com/skystrife/cpptoml
-
-> Copyright (c) 2014 Chase Geigle
->
-> Permission is hereby granted, free of charge, to any person obtaining a copy of
-> this software and associated documentation files (the "Software"), to deal in
-> the Software without restriction, including without limitation the rights to
-> use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
-> the Software, and to permit persons to whom the Software is furnished to do so,
-> subject to the following conditions:
-> 
-> The above copyright notice and this permission notice shall be included in all
-> copies or substantial portions of the Software.
-> 
-> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
-> FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
-> COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
-> IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
-> CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
 ## printf
 
-jsix uses Marco Paland's tiny [printf][] library for its `*printf()` functions,
+The implementation of jsix's `*printf()` family of functions is derived from
-which is also released under the terms of the MIT license:
+Marco Paland and Eyal Rozenberg's tiny [printf][] library, which is also
+released under the terms of the MIT license:
 
-[printf]: https://github.com/mpaland/printf
+[printf]: https://github.com/eyalroz/printf
 
 > The MIT License (MIT)
 >
 > Copyright (c) 2014 Marco Paland
+> Copyright (c) 2021 Eyal Rozenberg
 >
 > Permission is hereby granted, free of charge, to any person obtaining a copy
 > of this software and associated documentation files (the "Software"), to deal

qemu.sh

@@ -3,32 +3,55 @@
 build="$(dirname $0)/build"
 assets="$(dirname $0)/assets"
 debug=""
+isaexit='-device isa-debug-exit,iobase=0xf4,iosize=0x04'
 debugtarget="${build}/jsix.elf"
-flash_name="ovmf_vars"
 gfx="-nographic"
+vga="-vga none"
+log=""
 kvm=""
 cpu="Broadwell,+pdpe1gb"
+smp=4
 
-for arg in $@; do
+while true; do
-	case "${arg}" in
+	case "$1" in
-		--debugboot)
+		-b | --debugboot)
 			debug="-s -S"
 			debugtarget="${build}/boot/boot.efi"
-			flash_name="ovmf_vars_d"
+            shift
 			;;
-		--debug)
+		-d | --debug)
 			debug="-s -S"
-			flash_name="ovmf_vars_d"
+            isaexit=""
+            shift
 			;;
-		--gfx)
+		-g | --gfx)
 			gfx=""
+			vga=""
+            shift
 			;;
-		--kvm)
+		-v | --vga)
+			vga=""
+            shift
+			;;
+		-k | --kvm)
 			kvm="-enable-kvm"
 			cpu="host"
+            shift
 			;;
+        -c | --cpus)
+            smp=$2
+            shift 2
+            ;;
+        -l | --log)
+            log="-d mmu,int,guest_errors -D jsix.log"
+            shift
+            ;;
 		*)
-			build="${arg}"
+            if [ -d "$1" ]; then
+                build="$1"
+                shift
+            fi
+            break
 			;;
 	esac
 done
@@ -51,25 +74,25 @@ if [[ -n $TMUX ]]; then
 	fi
 elif [[ $DESKTOP_SESSION = "i3" ]]; then
 	if [[ -n $debug ]]; then
-		i3-msg exec i3-sensible-terminal -- -e "gdb ${PWD}/${build}/jsix.elf" &
+		i3-msg exec i3-sensible-terminal -- -e "gdb ${debugtarget}" &
 	else
 		i3-msg exec i3-sensible-terminal -- -e 'telnet localhost 45454' &
 	fi
 fi
 
-exec qemu-system-x86_64 \
+qemu-system-x86_64 \
 	-drive "if=pflash,format=raw,readonly,file=${assets}/ovmf/x64/ovmf_code.fd" \
-	-drive "if=pflash,format=raw,file=${build}/${flash_name}.fd" \
+	-drive "if=pflash,format=raw,file=${build}/ovmf_vars.fd" \
 	-drive "format=raw,file=${build}/jsix.img" \
-	-device "isa-debug-exit,iobase=0xf4,iosize=0x04" \
 	-monitor telnet:localhost:45454,server,nowait \
 	-serial stdio \
 	-serial telnet:localhost:45455,server,nowait \
-	-smp 4 \
+	-smp "${smp}" \
-	-m 512 \
+	-m 4096 \
-	-d mmu,int,guest_errors \
-	-D jsix.log \
 	-cpu "${cpu}" \
 	-M q35 \
 	-no-reboot \
-	$gfx $kvm $debug
+	$isaexit $log $gfx $vga $kvm $debug
+
+((result = ($? >> 1) - 1))
+exit $result

requirements.txt

@@ -0,0 +1,5 @@
+cogapp >= 3
+ninja >= 1.10.2
+peru >= 1.2.1
+pyyaml >= 5.4
+lark == 0.12.0

scripts/bonnibel/__init__.py

@@ -0,0 +1,25 @@
+from os.path import join
+
+class BonnibelError(Exception):
+    def __init__(self, message):
+        self.message = message
+
+def load_config(filename):
+    from yaml import safe_load
+    with open(filename, 'r') as infile:
+        return safe_load(infile.read())
+
+def mod_rel(path):
+    return join("${module_dir}", path)
+
+def target_rel(path, module=None):
+    if module:
+        return join("${target_dir}", module + ".dir", path)
+    else:
+        return join("${target_dir}", path)
+
+def include_rel(path, module=None):
+    if module:
+        return join("${build_root}", "include", module, path)
+    else:
+        return join("${build_root}", "include", path)

scripts/bonnibel/manifest.py

@@ -0,0 +1,103 @@
+from . import BonnibelError
+
+class Manifest:
+    from collections import namedtuple
+    Entry = namedtuple("Entry", ("module", "target", "output", "location", "description", "flags"))
+
+    flags = {
+        "graphical": 0x01,
+        "panic":     0x02,
+        "symbols":   0x04,
+    }
+
+    boot_flags = {
+        "debug": 0x01,
+        "test":  0x02,
+    }
+
+    def __init__(self, path, modules):
+        from . import load_config
+
+        config = load_config(path)
+
+        self.kernel = self.__build_entry(modules,
+                config.get("kernel", dict()),
+                name="kernel", target="kernel")
+
+        self.init = self.__build_entry(modules,
+                config.get("init", None))
+
+        self.programs = [self.__build_entry(modules, i)
+                for i in config.get("programs", tuple())]
+
+        self.flags = config.get("flags", tuple())
+
+        self.data = []
+        for d in config.get("data", tuple()):
+            self.add_data(**d)
+
+    def __build_entry(self, modules, config, target="user", name=None):
+        flags = tuple()
+
+        if isinstance(config, str):
+            name = config
+        elif isinstance(config, dict):
+            name = config.get("name", name)
+            target = config.get("target", target)
+            flags = config.get("flags", tuple())
+            if isinstance(flags, str):
+                flags = flags.split()
+
+        mod = modules.get(name)
+        if not mod:
+            raise BonnibelError(f"Manifest specifies unknown module '{name}'")
+
+        for f in flags:
+            if not f in Manifest.flags:
+                raise BonnibelError(f"Manifest specifies unknown flag '{f}'")
+
+        return Manifest.Entry(name, target, mod.output, mod.location, mod.description, flags)
+
+    def add_data(self, output, location, desc, flags=tuple()):
+        e = Manifest.Entry(None, None, output, location, desc, flags)
+        self.data.append(e)
+        return e
+
+    def write_boot_config(self, path):
+        from os.path import join
+        import struct
+
+        with open(path, 'wb') as outfile:
+            magic = "jsixboot".encode("utf-8") # magic string
+            version = 0
+            reserved = 0
+
+            bootflags = sum([Manifest.boot_flags.get(s, 0) for s in self.flags])
+
+            outfile.write(struct.pack("<8sBBHHH",
+                magic, version, reserved,
+                len(self.programs), len(self.data),
+                bootflags))
+
+            def write_str(s):
+                outfile.write(struct.pack("<H", (len(s)+1)*2))
+                outfile.write(s.encode("utf-16le"))
+                outfile.write(b"\0\0")
+
+            def write_ent(ent):
+                flags = 0
+                for f in ent.flags:
+                    flags |= Manifest.flags[f]
+
+                outfile.write(struct.pack("<H", flags))
+                write_str(join(ent.location, ent.output).replace('/','\\'))
+                write_str(ent.description)
+
+            write_ent(self.kernel)
+            write_ent(self.init)
+
+            for p in self.programs:
+                write_ent(p)
+
+            for d in self.data:
+                write_ent(d)

scripts/bonnibel/module.py

@@ -0,0 +1,276 @@
+from . import include_rel, mod_rel, target_rel
+
+def resolve(path):
+    if path.startswith('/') or path.startswith('$'):
+        return path
+    from pathlib import Path
+    return str(Path(path).resolve())
+
+class BuildOptions:
+    def __init__(self, includes=tuple(), libs=tuple(), order_only=tuple()):
+        self.includes = list(includes)
+        self.libs = list(libs)
+        self.order_only = list(order_only)
+
+
+class Module:
+    __fields = {
+        # name: (type, default)
+        "kind": (str, "exe"),
+        "output": (str, None),
+        "targets": (set, ()),
+        "deps":  (set, ()),
+        "public_headers":  (set, ()),
+        "includes": (tuple, ()),
+        "sources": (tuple, ()),
+        "variables": (dict, ()),
+        "default": (bool, False),
+        "location": (str, "jsix"),
+        "description": (str, None),
+        "no_libc": (bool, False),
+    }
+
+    def __init__(self, name, modfile, root, **kwargs):
+        from .source import make_source
+
+        # Required fields
+        self.root = root
+        self.name = name
+        self.modfile = modfile
+
+        for name, data in self.__fields.items():
+            ctor, default = data
+            value = kwargs.get(name, default)
+            if value is not None:
+                value = ctor(value)
+
+            setattr(self, name, value)
+
+        for name in kwargs:
+            if not name in self.__fields:
+                raise AttributeError(f"No attribute named {name} on Module")
+
+        # Turn strings into real Source objects
+        self.sources = [make_source(root, f) for f in self.sources]
+        self.public_headers = [make_source(root, f) for f in self.public_headers]
+
+        # Filled by Module.update
+        self.depmods = set()
+
+    def __str__(self):
+        return "Module {} {}\n\t{}".format(self.kind, self.name, "\n\t".join(map(str, self.sources)))
+
+    @property
+    def output(self):
+        if self.__output is not None:
+            return self.__output
+
+        if self.kind == "lib":
+            return f"lib{self.name}.a"
+        else:
+            return f"{self.name}.elf"
+
+    @output.setter
+    def output(self, value):
+        self.__output = value
+
+    @classmethod
+    def update(cls, mods):
+        from . import BonnibelError
+
+        def resolve(source, modlist):
+            resolved = set()
+            for dep in modlist:
+                if not dep in mods:
+                    raise BonnibelError(f"module '{source.name}' references unknown module '{dep}'")
+                mod = mods[dep]
+                resolved.add(mod)
+            return resolved
+
+        for mod in mods.values():
+            mod.depmods = resolve(mod, mod.deps)
+
+        target_mods = [mod for mod in mods.values() if mod.targets]
+        for mod in target_mods:
+            closed = set()
+            children = set(mod.depmods)
+            while children:
+                child = children.pop()
+                closed.add(child)
+                child.targets |= mod.targets
+                children |= {m for m in child.depmods if not m in closed}
+
+    def generate(self, output):
+        from pathlib import Path
+        from collections import defaultdict
+        from ninja.ninja_syntax import Writer
+
+        def walk_deps(deps):
+            open_set = set(deps)
+            closed_set = set()
+            while open_set:
+                dep = open_set.pop()
+                closed_set.add(dep)
+                open_set |= {m for m in dep.depmods if not m in closed_set}
+            return closed_set
+
+        all_deps = walk_deps(self.depmods)
+
+        def gather_phony(build, deps, child_rel, add_headers=False):
+            phony = ".headers.phony"
+            child_phony = [child_rel(phony, module=c.name)
+                    for c in all_deps]
+
+            header_targets = []
+            if add_headers:
+                if not self.no_libc:
+                    header_targets.append(f"${{build_root}}/include/libc/{phony}")
+                if self.public_headers:
+                    header_targets.append(f"${{build_root}}/include/{self.name}/{phony}")
+
+            build.build(
+                rule = "touch",
+                outputs = [mod_rel(phony)],
+                implicit = child_phony + header_targets,
+                order_only = list(map(mod_rel, deps)),
+            )
+
+        filename = str(output / f"headers.{self.name}.ninja")
+        with open(filename, "w") as buildfile:
+            build = Writer(buildfile)
+
+            build.comment("This file is automatically generated by bonnibel")
+            build.newline()
+
+            build.variable("module_dir", f"${{build_root}}/include/{self.name}")
+
+            header_deps = []
+
+            inputs = []
+            headers = set(self.public_headers)
+            while headers:
+                source = headers.pop()
+                headers.update(source.next)
+
+                if source.action:
+                    build.newline()
+                    build.build(rule=source.action, **source.args)
+
+                if source.gather:
+                    header_deps += list(source.outputs)
+
+                if source.input:
+                    inputs.extend(map(mod_rel, source.outputs))
+
+            build.newline()
+            gather_phony(build, header_deps, include_rel)
+
+        filename = str(output / f"module.{self.name}.ninja")
+        with open(filename, "w") as buildfile:
+            build = Writer(buildfile)
+
+            build.comment("This file is automatically generated by bonnibel")
+            build.newline()
+
+            build.variable("module_dir", target_rel(self.name + ".dir"))
+
+            modopts = BuildOptions(
+                includes = [self.root, "${module_dir}"],
+                )
+            if self.public_headers:
+                modopts.includes += [f"${{build_root}}/include/{self.name}"]
+
+            for key, value in self.variables.items():
+                build.variable(key, value)
+            build.newline()
+
+            for include in self.includes:
+                p = Path(include)
+                if p.is_absolute():
+                    if not p in modopts.includes:
+                        modopts.includes.append(str(p.resolve()))
+                elif include != ".":
+                    incpath = self.root / p
+                    destpath = mod_rel(p)
+                    for header in incpath.rglob("*.h"):
+                        dest_header = f"{destpath}/" + str(header.relative_to(incpath))
+                    modopts.includes.append(str(incpath))
+                    modopts.includes.append(destpath)
+
+            all_deps = walk_deps(self.depmods)
+            for dep in all_deps:
+                if dep.public_headers:
+                    modopts.includes += [f"${{build_root}}/include/{dep.name}"]
+
+                if dep.kind == "lib":
+                    modopts.libs.append(target_rel(dep.output))
+                else:
+                    modopts.order_only.append(target_rel(dep.output))
+
+            if modopts.includes:
+                build.variable("ccflags", ["${ccflags}"] + [f"-I{i}" for i in modopts.includes])
+                build.variable("asflags", ["${asflags}"] + [f"-I{i}" for i in modopts.includes])
+
+            if modopts.libs:
+                build.variable("libs", ["${libs}"] + modopts.libs)
+
+            header_deps = []
+
+            inputs = []
+            sources = set(self.sources)
+            while sources:
+                source = sources.pop()
+                sources.update(source.next)
+
+                if source.action:
+                    build.newline()
+                    build.build(rule=source.action, **source.args)
+
+                if source.gather:
+                    header_deps += list(source.outputs)
+
+                if source.input:
+                    inputs.extend(map(mod_rel, source.outputs))
+
+            build.newline()
+
+            gather_phony(build, header_deps, target_rel, add_headers=True)
+
+            output = target_rel(self.output)
+            dump = output + ".dump"
+            build.newline()
+            build.build(
+                rule = self.kind,
+                outputs = output,
+                inputs = inputs,
+                implicit = modopts.libs,
+                order_only = modopts.order_only,
+            )
+
+            build.newline()
+            build.build(
+                rule = "dump",
+                outputs = dump,
+                inputs = output,
+                variables = {"name": self.name},
+            )
+
+            if self.default:
+                build.newline()
+                build.default(output)
+                build.default(dump)
+
+    def add_input(self, path, **kwargs):
+        from .source import Source
+        s = Source(self.root, path, **kwargs)
+        self.sources.append(s)
+        return s.outputs
+
+    def add_depends(self, paths, deps):
+        for source in self.sources:
+            if source.path in paths:
+                source.add_deps(deps)
+
+        for source in self.public_headers:
+            if source.path in paths:
+                source.add_deps(deps)

scripts/bonnibel/project.py

@@ -0,0 +1,234 @@
+from . import BonnibelError
+
+class Project:
+    def __init__(self, root):
+        from .version import git_version
+
+        self.root = root
+        self.version = git_version(root)
+
+    def __str__(self):
+        return f"{self.name} {self.version.major}.{self.version.minor}.{self.version.patch}-{self.version.sha}"
+
+    def generate(self, root, output, modules, config, manifest_file):
+        import sys
+        import bonnibel
+        from os.path import join
+        from ninja.ninja_syntax import Writer
+        from .target import Target
+
+        targets = set()
+        for mod in modules.values():
+            targets.update({Target.load(root, t, config) for t in mod.targets})
+
+        with open(output / "build.ninja", "w") as buildfile:
+            build = Writer(buildfile)
+
+            build.comment("This file is automatically generated by bonnibel")
+            build.variable("ninja_required_version", "1.3")
+            build.variable("build_root", output)
+            build.variable("source_root", root)
+            build.newline()
+
+            build.include(root / "configs" / "rules.ninja")
+            build.newline()
+
+            build.variable("version_major", self.version.major)
+            build.variable("version_minor", self.version.minor)
+            build.variable("version_patch", self.version.patch)
+            build.variable("version_sha", self.version.sha)
+            build.newline()
+
+            build.variable("cogflags", [
+                "-I", "${source_root}/scripts",
+                "-D", "definitions_path=${source_root}/definitions",
+                ])
+            build.newline()
+
+            for target in targets:
+                build.subninja(output / target.name / "target.ninja")
+            build.newline()
+
+            for mod in modules.values():
+                build.subninja(output / f"headers.{mod.name}.ninja")
+            build.newline()
+
+            build.build(
+                rule = "touch",
+                outputs = "${build_root}/.all_headers",
+                implicit = [f"${{build_root}}/include/{m.name}/.headers.phony"
+                    for m in modules.values() if m.public_headers],
+                )
+            build.build(
+                rule = "phony",
+                outputs = ["all-headers"],
+                inputs = ["${build_root}/.all_headers"])
+
+            debugroot = output / ".debug"
+            debugroot.mkdir(exist_ok=True)
+
+            fatroot = output / "fatroot"
+            fatroot.mkdir(exist_ok=True)
+
+            fatroot_content = []
+
+            def add_fatroot(source, entry):
+                output = join(entry.location, entry.output)
+                fatroot_output = f"${{build_root}}/fatroot/{output}"
+
+                build.build(
+                    rule = "cp",
+                    outputs = [fatroot_output],
+                    inputs = [source],
+                    variables = {
+                        "name": f"Installing {output}",
+                    })
+
+                fatroot_content.append(fatroot_output)
+                build.newline()
+
+            def add_fatroot_exe(entry):
+                input_path = f"${{build_root}}/{entry.target}/{entry.output}"
+                intermediary = f"${{build_root}}/{entry.output}"
+
+                build.build(
+                    rule = "strip",
+                    outputs = [intermediary],
+                    inputs = [input_path],
+                    implicit = [f"{input_path}.dump"],
+                    variables = {
+                        "name": f"Stripping {entry.module}",
+                        "debug": f"${{build_root}}/.debug/{entry.output}.debug",
+                    })
+
+                add_fatroot(intermediary, entry)
+
+            from .manifest import Manifest
+            manifest = Manifest(manifest_file, modules)
+
+            add_fatroot_exe(manifest.kernel)
+            add_fatroot_exe(manifest.init)
+            for program in manifest.programs:
+                add_fatroot_exe(program)
+
+            syms = manifest.add_data("symbol_table.dat",
+                    manifest.kernel.location, "Symbol table", ("symbols",))
+
+            sym_out = f"${{build_root}}/symbol_table.dat"
+            build.build(
+                rule = "makest",
+                outputs = [sym_out],
+                inputs = [f"${{build_root}}/{modules['kernel'].output}"],
+                )
+            add_fatroot(sym_out, syms)
+
+            bootloader = "${build_root}/fatroot/efi/boot/bootx64.efi"
+            build.build(
+                rule = "cp",
+                outputs = [bootloader],
+                inputs = ["${build_root}/boot/boot.efi"],
+                variables = {
+                    "name": "Installing bootloader",
+                })
+            build.newline()
+
+            boot_config = join(fatroot, "jsix_boot.dat")
+            manifest.write_boot_config(boot_config)
+
+            build.build(
+                rule = "makefat",
+                outputs = ["${build_root}/jsix.img"],
+                inputs = ["${source_root}/assets/diskbase.img"],
+                implicit = fatroot_content + [bootloader],
+                variables = {"name": "jsix.img"},
+                )
+            build.newline()
+
+            default_assets = {
+                "UEFI Variables": ("ovmf/x64/ovmf_vars.fd", "ovmf_vars.fd"),
+                "GDB Debug Helpers": ("debugging/jsix.elf-gdb.py", "jsix.elf-gdb.py"),
+            }
+
+            for name, assets in default_assets.items():
+                p = root / "assets" / assets[0]
+                out = f"${{build_root}}/{assets[1]}"
+                build.build(
+                    rule = "cp",
+                    outputs = [out],
+                    inputs = [str(p)],
+                    variables = {"name": name},
+                    )
+                build.default([out])
+                build.newline()
+
+            compdb = "${source_root}/compile_commands.json"
+
+            build.rule("regen",
+                command = " ".join([str(root / 'configure')] + sys.argv[1:]),
+                description = "Regenerate build files",
+                generator = True,
+                )
+            build.newline()
+
+            regen_implicits = \
+                [f"{self.root}/configure", str(manifest_file)] + \
+                [str(mod.modfile) for mod in modules.values()]
+
+            for target in targets:
+                regen_implicits += target.depfiles
+
+            build.build(
+                rule = "compdb",
+                outputs = [compdb],
+                implicit = regen_implicits,
+                )
+            build.default([compdb])
+            build.newline()
+
+            build.build(
+                rule = "regen",
+                outputs = ['build.ninja'],
+                implicit = regen_implicits,
+                implicit_outputs = 
+                    [f"module.{mod.name}.ninja" for mod in modules.values()] +
+                    [f"{target.name}/target.ninja" for target in targets] +
+                    [boot_config],
+                )
+
+            build.newline()
+            build.default(["${build_root}/jsix.img"])
+
+        for target in targets:
+            mods = [m.name for m in modules.values() if target.name in m.targets]
+
+            targetdir = output / target.name
+            targetdir.mkdir(exist_ok=True)
+
+            buildfilename = str(targetdir / "target.ninja")
+            with open(buildfilename, "w") as buildfile:
+                build = Writer(buildfile)
+                build.comment("This file is automatically generated by bonnibel")
+                build.newline()
+
+                build.variable("target", target.name)
+                build.variable("target_dir", output / target.name)
+                build.newline()
+
+                for name, value in target.items():
+                    build.variable(name, value)
+
+                build.newline()
+                for kind in ('defs', 'run'):
+                    for lang in ('c', 'cpp'):
+                        deffile = str(output / target.name / f"{lang}.{kind}")
+
+                        build.build(
+                            rule = f"dump_{lang}_{kind}",
+                            outputs = [deffile],
+                            implicit = [buildfilename],
+                        )
+                        build.default(deffile)
+                        build.newline()
+
+                for mod in mods:
+                    build.subninja(f"module.{mod}.ninja")

scripts/bonnibel/source.py

@@ -0,0 +1,119 @@
+from os.path import join, splitext
+from . import mod_rel
+
+def _resolve(path):
+    if path.startswith('/') or path.startswith('$'):
+        return path
+    from pathlib import Path
+    return str(Path(path).resolve())
+
+def _dynamic_action(name):
+    def prop(self):
+        root, suffix = splitext(self.path)
+        return f"{name}{suffix}"
+    return prop
+
+
+class Source:
+    next = tuple()
+    action = None
+    args = dict()
+    gather = False
+    outputs = tuple()
+    input = False
+
+    def __init__(self, path, root = "${module_dir}", deps=tuple()): 
+        self.path = path
+        self.root = root
+        self.deps = deps
+
+    def add_deps(self, deps):
+        self.deps += tuple(deps)
+
+    @property
+    def fullpath(self):
+        return join(self.root, self.path)
+
+class ParseSource(Source):
+    action = property(_dynamic_action("parse"))
+
+    @property
+    def output(self):
+        root, _ = splitext(self.path)
+        return root
+
+    @property
+    def outputs(self):
+        return (self.output,)
+
+    @property
+    def gather(self):
+        _, suffix = splitext(self.output)
+        return suffix in (".h", ".inc")
+
+    @property
+    def next(self):
+        _, suffix = splitext(self.output)
+        nextType = {
+            ".s": CompileSource,
+            ".cpp": CompileSource,
+            }.get(suffix)
+
+        if nextType:
+            return (nextType(self.output),)
+        return tuple()
+
+    @property
+    def args(self):
+        return dict(
+            outputs = list(map(mod_rel, self.outputs)),
+            inputs = [self.fullpath],
+            implicit = list(map(_resolve, self.deps)),
+            variables = dict(name=self.path),
+            )
+
+class HeaderSource(Source):
+    action = "cp"
+    gather = True
+
+    @property
+    def outputs(self):
+        return (self.path,)
+
+    @property
+    def args(self):
+        return dict(
+            outputs = [mod_rel(self.path)],
+            inputs = [join(self.root, self.path)],
+            implicit = list(map(_resolve, self.deps)),
+            variables = dict(name=self.path),
+            )
+
+class CompileSource(Source):
+    action = property(_dynamic_action("compile"))
+    input = True
+
+    @property
+    def outputs(self):
+        return (self.path + ".o",)
+
+    @property
+    def args(self):
+        return dict(
+            outputs = list(map(mod_rel, self.outputs)),
+            inputs = [join(self.root, self.path)],
+            implicit = list(map(_resolve, self.deps)) + [mod_rel(".headers.phony")],
+            variables = dict(name=self.path),
+            )
+
+def make_source(root, path):
+    _, suffix = splitext(path)
+
+    if suffix in (".s", ".c", ".cpp"):
+        return CompileSource(path, root)
+    elif suffix in (".cog",):
+        return ParseSource(path, root)
+    elif suffix in (".h", ".inc"):
+        return HeaderSource(path, root)
+    else:
+        raise RuntimeError(f"{path} has no Source type")

scripts/bonnibel/target.py

@@ -0,0 +1,50 @@
+class Target(dict):
+    __targets = {}
+
+    @classmethod
+    def load(cls, root, name, config=None):
+        from . import load_config
+
+        if (name, config) in cls.__targets:
+            return cls.__targets[(name, config)]
+
+        configs = root / "configs"
+
+        dicts = []
+        depfiles = []
+        basename = name
+        if config:
+            basename += f"-{config}"
+
+        while basename is not None:
+            filename = str(configs / (basename + ".yaml"))
+            depfiles.append(filename)
+            desc = load_config(filename)
+            basename = desc.get("extends")
+            dicts.append(desc.get("variables", dict()))
+
+        t = Target(name, config, depfiles)
+        for d in reversed(dicts):
+            for k, v in d.items():
+                if isinstance(v, (list, tuple)):
+                    t[k] = t.get(k, list()) + list(v)
+                elif isinstance(v, dict):
+                    t[k] = t.get(k, dict())
+                    t[k].update(v)
+                else:
+                    t[k] = v
+
+        cls.__targets[(name, config)] = t
+        return t
+
+    def __init__(self, name, config, depfiles):
+        self.__name = name
+        self.__config = config
+        self.__depfiles = tuple(depfiles)
+
+    def __hash__(self):
+        return hash((self.__name, self.__config))
+
+    name = property(lambda self: self.__name)
+    config = property(lambda self: self.__config)
+    depfiles = property(lambda self: self.__depfiles)

scripts/bonnibel/version.py

@@ -0,0 +1,41 @@
+from collections import namedtuple as _namedtuple
+
+version = _namedtuple('version', [
+    'major',
+    'minor',
+    'patch',
+    'sha',
+    'dirty',
+    ])
+
+
+def _run_git(root, *args):
+    from subprocess import run
+
+    git = run(['git', '-C', root] + list(args),
+            check=True, capture_output=True)
+    return git.stdout.decode('utf-8').strip()
+
+
+def git_version(root):
+    full_version = _run_git(root, 'describe', '--dirty')
+    full_sha = _run_git(root, 'rev-parse', 'HEAD')
+
+    dirty = False
+    parts1 = full_version.split('-')
+    if parts1[-1] == "dirty":
+        dirty = True
+        parts1 = parts1[:-1]
+
+    if parts1[0][0] == 'v':
+        parts1[0] = parts1[0][1:]
+
+    parts2 = parts1[0].split('.')
+
+    return version(
+        parts2[0],
+        parts2[1],
+        parts2[2],
+        full_sha[:7],
+        dirty)
+

scripts/build_symbol_table.py

@@ -4,33 +4,37 @@
 # is as follows:
 #
 # <num_entires> : 8 bytes
-# <index>       : 16 * N bytes
+# <index>       : 24 * N bytes
 # <name data>   : variable
 #
 # Each index entry has the format
 # <symbol address> : 8 bytes
+# <symbol size>    : 8 bytes
 # <offset of name> : 8 bytes
 #
 # Name offsets are from the start of the symbol table as a whole. (ie,
 # where <num_entries> is located.)
 
+import re
+
+sym_re = re.compile(r'([0-9a-fA-F]{16}) ([0-9a-fA-F]{16}) [tTvVwW] (.*)')
+
 def parse_syms(infile):
     """Take the output of the `nm` command, and parse it into a tuple
     representing the symbols in the text segment of the binary. Returns
     a list of (address, symbol_name)."""
 
-    from cxxfilt import demangle
-
     syms = []
     for line in sys.stdin:
-        addr, t, mangled = line.split()
+        match = sym_re.match(line)
-        if t not in "tTvVwW": continue
+        if not match: continue
 
-        addr = int(addr, base=16)
+        addr = int(match.group(1), base=16)
-        name = demangle(mangled)
+        size = int(match.group(2), base=16)
-        syms.append((addr, name))
+        name = match.group(3)
+        syms.append([addr, size, name, 0])
 
-    return sorted(syms)
+    return syms
 
 
 def write_table(syms, outfile):
@@ -42,29 +46,26 @@ def write_table(syms, outfile):
     outfile.write(struct.pack("@Q", len(syms)))
     index_pos = outfile.tell()
 
-    outfile.seek(struct.calcsize("@QQ") * len(syms), 1)
+    outfile.seek(struct.calcsize("@QQQ") * len(syms), 1)
     nul = b'\0'
 
-    positions = {}
     for s in syms:
-        addr, name = s
+        s[3] = outfile.tell()
-        positions[addr] = outfile.tell()
+        outfile.write(s[2].encode('utf-8'))
-
-        data = name.encode('utf-8')
-        outfile.write(name.encode('utf-8'))
         outfile.write(nul)
 
     outfile.seek(index_pos)
     for s in syms:
         addr = s[0]
-        pos = positions[addr]
+        size = s[1]
-        outfile.write(struct.pack("@QQ", addr, pos))
+        pos = s[3]
+        outfile.write(struct.pack("@QQQ", addr, size, pos))
 
 
 if __name__ == "__main__":
     import sys
     if len(sys.argv) != 2:
-        print(f"Usage: {sys.argv[0]} <output>")
+        print(f"Usage: nm -n -S --demangle | {sys.argv[0]} <output>")
         sys.exit(1)
 
     outfile = open(sys.argv[1], "wb")

scripts/build_sysroot.sh

@@ -1,131 +0,0 @@
-#!/usr/bin/env bash
-
-TARGET="x86_64-elf"
-LLVM_BRANCH="release_80"
-
-TOOLS="clang lld" # lld libunwind libcxxabi libcxx"
-PROJECTS="compiler-rt libcxxabi libcxx libunwind"
-#RUNTIMES="compiler-rt libcxxabi libcxx libunwind"
-
-set -e
-
-README=$(realpath "$(dirname $0)/readme_for_prebuilt_sysroots.md")
-SYSROOT=$(realpath "$(dirname $0)/../sysroot")
-WORK=$(realpath "$(dirname $0)/sysroot")
-mkdir -p "${SYSROOT}"
-mkdir -p "${WORK}"
-
-export CC=clang
-export CXX=clang++
-
-if [[ ! -d "${WORK}/llvm" ]]; then
-	echo "Downloading LLVM..."
-	git clone -q \
-		--branch "${LLVM_BRANCH}" \
-		--depth 1 \
-		"https://git.llvm.org/git/llvm.git" "${WORK}/llvm"
-fi
-
-for tool in ${TOOLS}; do
-	if [[ ! -d "${WORK}/llvm/tools/${tool}" ]]; then
-		echo "Downloading ${tool}..."
-		git clone -q \
-			--branch "${LLVM_BRANCH}" \
-			--depth 1 \
-			"https://git.llvm.org/git/${tool}.git" "${WORK}/llvm/tools/${tool}"
-	fi
-done
-
-if [[ ! -d "${WORK}/llvm/tools/clang/tools/extra" ]]; then
-	echo "Downloading clang-tools-extra..."
-	git clone -q \
-		--branch "${LLVM_BRANCH}" \
-		--depth 1 \
-		"https://git.llvm.org/git/clang-tools-extra.git" "${WORK}/llvm/tools/clang/tools/extra"
-fi
-
-for proj in ${PROJECTS}; do
-	if [[ ! -d "${WORK}/llvm/projects/${proj}" ]]; then
-		echo "Downloading ${proj}..."
-		git clone -q \
-			--branch "${LLVM_BRANCH}" \
-			--depth 1 \
-			"https://git.llvm.org/git/${proj}.git" "${WORK}/llvm/projects/${proj}"
-	fi
-done
-
-for proj in ${RUNTIMES}; do
-	if [[ ! -d "${WORK}/llvm/runtimes/${proj}" ]]; then
-		echo "Downloading ${proj}..."
-		git clone -q \
-			--branch "${LLVM_BRANCH}" \
-			--depth 1 \
-			"https://git.llvm.org/git/${proj}.git" "${WORK}/llvm/runtime/${proj}"
-	fi
-done
-
-mkdir -p "${WORK}/build/llvm"
-pushd "${WORK}/build/llvm"
-
-echo "Configuring LLVM..."
-
-cmake -G Ninja \
-	-DCLANG_DEFAULT_RTLIB=compiler-rt \
-	-DCLANG_DEFAULT_STD_C=c11 \
-	-DCLANG_DEFAULT_STD_CXX=cxx14 \
-	-DCMAKE_BUILD_TYPE=Release \
-	-DCMAKE_C_COMPILER="clang" \
-	-DCMAKE_CXX_COMPILER="clang++" \
-	-DCMAKE_CXX_FLAGS="-Wno-unused-parameter -D_LIBCPP_HAS_NO_ALIGNED_ALLOCATION -D_LIBUNWIND_IS_BAREMETAL=1 -U_LIBUNWIND_SUPPORT_DWARF_UNWIND" \
-	-DCMAKE_INSTALL_PREFIX="${SYSROOT}" \
-	-DCMAKE_MAKE_PROGRAM=`which ninja` \
-	-DDEFAULT_SYSROOT="${SYSROOT}" \
-	-DLIBCXX_CXX_ABI=libcxxabi \
-	-DLIBCXX_CXX_ABI_INCLUDE_PATHS="${WORK}/llvm/projects/libcxxabi/include" \
-	-DLIBCXX_CXX_ABI_LIBRARY_PATH=lib \
-	-DLIBCXX_ENABLE_EXPERIMENTAL_LIBRARY=OFF \
-	-DLIBCXX_ENABLE_NEW_DELETE_DEFINITIONS=ON \
-	-DLIBCXX_ENABLE_SHARED=OFF \
-	-DLIBCXX_ENABLE_STATIC_ABI_LIBRARY=ON \
-	-DLIBCXX_ENABLE_THREADS=OFF \
-	-DLIBCXX_INCLUDE_BENCHMARKS=OFF \
-	-DLIBCXX_USE_COMPILER_RT=ON \
-	-DLIBCXXABI_ENABLE_NEW_DELETE_DEFINITIONS=OFF \
-	-DLIBCXXABI_ENABLE_SHARED=OFF \
-	-DLIBCXXABI_ENABLE_STATIC_UNWINDER=ON \
-	-DLIBCXXABI_ENABLE_THREADS=OFF \
-	-DLIBCXXABI_LIBCXX_PATH="${WORK}/llvm/projects/libcxx" \
-	-DLIBCXXABI_USE_COMPILER_RT=ON \
-	-DLIBCXXABI_USE_LLVM_UNWINDER=ON \
-	-DLIBUNWIND_ENABLE_SHARED=OFF \
-	-DLIBUNWIND_ENABLE_THREADS=OFF \
-	-DLIBUNWIND_USE_COMPILER_RT=ON \
-	-DLLVM_CONFIG_PATH="${SYSROOT}/bin/llvm-config" \
-	-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-elf" \
-	-DLLVM_ENABLE_LIBCXX=ON \
-	-DLLVM_ENABLE_LLD=ON \
-	-DLLVM_ENABLE_PIC=OFF \
-	-DLLVM_ENABLE_THREADS=OFF \
-	-DLLVM_INSTALL_BINUTILS_SYMLINKS=ON \
-	-DLLVM_TARGETS_TO_BUILD="X86" \
-	${WORK}/llvm > cmake_configure.log
-
-#  -DCMAKE_ASM_COMPILER=nasm \
-#  -DCMAKE_LINKER="${SYSROOT}/bin/ld.lld" \
-#  -DCOMPILER_RT_ENABLE_LLD=ON \
-#  -DLIBCXX_ENABLE_LLD=ON \
-#  -DLIBCXX_ENABLE_STATIC_UNWINDER=ON \
-#  -DLIBCXXABI_ENABLE_LLD=ON \
-#  -DLIBUNWIND_ENABLE_LLD=ON \
-#  -DLLVM_ENABLE_PROJECTS="libcxx;libcxxabi;libunwind;compiler-rt" \
-#  -DCOMPILER_RT_BAREMETAL_BUILD=ON \
-#  -DLIBCXXABI_BAREMETAL=ON \
-
-echo "Building LLVM..."
-ninja && ninja install
-ninja cxx cxxabi compiler-rt
-ninja install-compiler-rt install-cxx install-cxxabi
-popd
-
-
-cp "${README}" "${SYSROOT}/README.md"

scripts/definitions/context.py

@@ -0,0 +1,83 @@
+class NotFound(Exception): pass
+
+class Context:
+    def __init__(self, path, verbose=False):
+        if isinstance(path, str):
+            self.__paths = [path]
+        else:
+            self.__paths = path
+
+        self.__closed = set()
+        self.__verbose = verbose
+
+        self.__deps = {}
+
+        self.objects = dict()
+        self.interfaces = dict()
+
+    verbose = property(lambda self: self.__verbose)
+
+    def find(self, filename):
+        from os.path import exists, isabs, join
+
+        if exists(filename) or isabs(filename):
+            return filename
+
+        for path in self.__paths:
+            full = join(path, filename)
+            if exists(full):
+                return full
+
+        raise NotFound(filename)
+
+    def parse(self, filename):
+        pending = set()
+        pending.add(filename)
+
+        from .parser import LarkError
+        from .parser import Lark_StandAlone as Parser
+        from .transformer import DefTransformer
+
+        objects = {}
+        interfaces = {}
+
+        while pending:
+            name = pending.pop()
+            self.__closed.add(name)
+
+            path = self.find(name)
+
+            parser = Parser(transformer=DefTransformer(name))
+
+            try:
+                imps, objs, ints = parser.parse(open(path, "r").read())
+            except LarkError as e:
+                import sys
+                import textwrap
+                print(f"\nError parsing {name}:", file=sys.stderr)
+                print(textwrap.indent(str(e), "    "), file=sys.stderr)
+                sys.exit(1)
+
+            objects.update(objs)
+            interfaces.update(ints)
+
+            self.__deps[name] = imps
+
+            pending.update(imps.difference(self.__closed))
+
+        from .types import ObjectRef
+        ObjectRef.connect(objects)
+
+        for obj in objects.values():
+            for method in obj.methods:
+                caps = method.options.get("cap", list())
+                for cap in caps:
+                    if not cap in obj.caps:
+                        from .errors import UnknownCapError
+                        raise UnknownCapError(f"Unknown capability {cap} on {obj.name}::{method.name}")
+
+        self.objects.update(objects)
+        self.interfaces.update(interfaces)
+
+    def deps(self):
+        return {self.find(k): tuple(map(self.find, v)) for k, v in self.__deps.items()}

scripts/definitions/errors.py

@@ -0,0 +1,3 @@
+class InvalidType(Exception): pass
+class UnknownTypeError(Exception): pass
+class UnknownCapError(Exception): pass

scripts/definitions/templates/syscall/client/private/syscall_trampolines.s

@@ -0,0 +1,23 @@
+%macro SYSCALL 2
+	global __syscall_%1
+	__syscall_%1:
+		push rbp
+		mov rbp, rsp
+
+		; args should already be in rdi, etc, but rcx will
+		; get stomped, so stash it in r10, which isn't a
+		; callee-saved register, but also isn't used in the
+		; function call ABI.
+		mov r10, rcx
+
+		mov rax, %2
+		syscall
+		; result is now already in rax, so just return
+
+		pop rbp
+		ret
+%endmacro
+
+{% for id, scope, method in interface.methods %}
+SYSCALL {% if scope %}{{ scope.name }}_{% endif %}{{ method.name }} {{ id }}
+{% endfor %}

scripts/definitions/templates/syscall/client/public/_object_.hh

@@ -0,0 +1,41 @@
+#pragma once
+/// \file {{filename}}
+{% if object.super %}
+
+#include <j6/{{ object.super.name }}.hh>
+{% endif %}
+
+namespace j6 {
+
+{% if object.desc %}
+{{ object.desc|indent('/// ', first=True) }}
+{% endif %}
+class {{ object.name }}
+{% if object.super %}    : public {{ object.super.name }}
+{% endif %}
+{
+public:
+{% macro argument(type, name, first, options=False) -%}
+    {%- for ctype, suffix in type.c_names(options) -%}
+        {%- if not first or not loop.first %}, {% endif %}{{ ctype }} {{ name }}{{ suffix }}
+    {%- endfor -%}
+{%- endmacro -%}
+
+{% for method in object.methods %}
+{% if method.desc %}    /// {{ method.desc|indent('    /// ') }}{% endif %} 
+{% for param in method.params %}
+{% if param.desc %}    /// \arg {{ "%-10s"|format(param.name) }} {{ param.desc }}
+{% endif %}
+{% endfor %}
+    {%+ if method.static %}static {% endif %}j6_status_t {{ method.name }} (
+{%- for param in method.params %}{{ argument(param.type, param.name, loop.first, options=param.options) }}{% endfor -%});
+
+{% endfor %}
+    ~{{ object.name }}();
+
+private:
+    {{ object.name }}(j6_handle_t handle) : m_handle {handle} {}
+    j6_handle_t m_handle;
+}
+
+} // namespace j6

scripts/definitions/templates/syscall/client/public/syscalls.h

@@ -0,0 +1,25 @@
+#pragma once
+
+#include <j6/types.h>
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+{% macro argument(type, name, first, options=False) -%}
+    {%- for ctype, suffix in type.c_names(options) -%}
+        {%- if not first or not loop.first %}, {% endif %}{{ ctype }} {{ name }}{{ suffix }}
+    {%- endfor -%}
+{%- endmacro %}
+ 
+{% for id, scope, method in interface.methods %}
+j6_status_t __syscall_{% if scope %}{{ scope.name }}_{% endif %}{{ method.name }} (
+    {%- if not method.static -%}{{ argument(scope.reftype, "self", True) }}{% endif -%}
+    {%- set first = method.static -%}
+    {%- for param in method.params %}{{ argument(param.type, param.name, first, options=param.options) }}{% set first = False %}{% endfor -%}
+);
+{% endfor %}
+
+#ifdef __cplusplus
+}
+#endif
+

scripts/definitions/transformer.py

@@ -0,0 +1,156 @@
+from .parser import Transformer, v_args
+
+def get_opts(args):
+    from .types import Caps, CName, Description, Options, Type, UID
+
+    kinds = {
+        Description: "desc",
+        Options: "opts",
+        CName: "cname",
+        Caps: "caps",
+        UID: "uid",
+        Type: "typename",
+    }
+
+    result = dict()
+    outargs = []
+    for a in args:
+        for kind, name in kinds.items():
+            if isinstance(a, kind):
+                result[name] = a
+                break
+        else:
+            outargs.append(a)
+
+    return result, outargs
+
+class DefTransformer(Transformer):
+    def __init__(self, filename):
+        self.filename = filename
+
+    def start(self, args):
+        from .types import Import, Interface, Object
+
+        imports = set()
+        objects = dict()
+        interfaces = dict()
+
+        for o in args:
+            if isinstance(o, Object):
+                objects[o.name] = o
+
+            elif isinstance(o, Interface):
+                interfaces[o.name] = o
+
+            elif isinstance(o, Import):
+                imports.add(o)
+
+        return imports, objects, interfaces
+
+    @v_args(inline=True)
+    def import_statement(self, path):
+        from .types import Import
+        return Import(path)
+
+    def object(self, args):
+        from .types import Object
+        specials, args = get_opts(args)
+        name, args = args[0], args[1:]
+        return Object(name, children=args, **specials)
+
+    def interface(self, args):
+        from .types import Interface
+        specials, args = get_opts(args)
+        name, args = args[0], args[1:]
+        return Interface(name, children=args, **specials)
+
+    def method(self, args):
+        from .types import Method
+        specials, args = get_opts(args)
+        name, args = args[0], args[1:]
+        return Method(name, children=args, **specials)
+
+    def function(self, args):
+        from .types import Function
+        specials, args = get_opts(args)
+        name, args = args[0], args[1:]
+        return Function(name, children=args, **specials)
+
+    def param(self, args):
+        from .types import Param
+        specials, args = get_opts(args)
+        name = args[0]
+        return Param(name, **specials)
+
+    @v_args(inline=True)
+    def expose(self, s):
+        from .types import Expose
+        return Expose(s)
+
+    @v_args(inline=True)
+    def uid(self, s):
+        return s
+
+    @v_args(inline=True)
+    def cname(self, s):
+        from .types import CName
+        return CName(s)
+
+    @v_args(inline=True)
+    def name(self, s):
+        return s
+
+    @v_args(inline=True)
+    def type(self, s):
+        return s
+
+    @v_args(inline=True)
+    def super(self, s):
+        from .types import ObjectRef
+        return ObjectRef(s, self.filename)
+
+    def options(self, args):
+        from .types import Options
+        return Options([str(s) for s in args])
+
+    def capabilities(self, args):
+        from .types import Caps
+        return Caps([str(s) for s in args])
+
+    def description(self, s):
+        from .types import Description
+        return Description("\n".join(s))
+
+    @v_args(inline=True)
+    def object_name(self, n):
+        from .types import ObjectRef
+        return ObjectRef(n, self.filename)
+
+    def PRIMITIVE(self, s):
+        from .types import get_primitive
+        return get_primitive(s)
+
+    def UID(self, s):
+        from .types import UID
+        return UID(int(s, base=16))
+
+    def INT_TYPE(self, s):
+        return s
+
+    def NUMBER(self, s):
+        if s.startswith("0x"):
+            return int(s,16)
+        return int(s)
+
+    def COMMENT(self, s):
+        return s[2:].strip()
+
+    def OPTION(self, s):
+        return str(s)
+
+    def IDENTIFIER(self, s):
+        return str(s)
+
+    def PATH(self, s):
+        return str(s[1:-1])
+

scripts/definitions/types/__init__.py

@@ -0,0 +1,26 @@
+def _indent(x):
+    from textwrap import indent
+    return indent(str(x), '    ')
+
+class CName(str): pass
+class Description(str): pass
+class Import(str): pass
+class Caps(list): pass
+
+class Options(dict):
+    def __init__(self, opts = tuple()):
+        for opt in opts:
+            parts = opt.split(":", 1)
+            self[parts[0]] = self.get(parts[0], []) + ["".join(parts[1:])]
+
+class UID(int):
+    def __str__(self):
+        return f"{self:016x}"
+
+from .object import Object
+from .interface import Interface, Expose
+from .function import Function, Method, Param
+
+from .type import Type
+from .primitive import get_primitive
+from .objref import ObjectRef

scripts/definitions/types/function.py

@@ -0,0 +1,68 @@
+from . import _indent
+from . import Options
+
+def _hasopt(opt):
+    def test(self):
+        return opt in self.options
+    return test
+
+class Function:
+    typename = "function"
+
+    def __init__(self, name, opts=Options(), desc="", children=tuple()):
+        self.name = name
+        self.options = opts
+        self.desc = desc
+        self.params = [c for c in children if isinstance(c, Param)]
+        self.id = -1
+
+    def __str__(self):
+        parts = ["{} {}".format(self.typename, self.name)]
+        if self.desc:
+            parts.append(_indent(self.desc))
+        if self.options:
+            parts.append(f"    Options: {self.options}")
+        parts.extend(map(_indent, self.params))
+        return "\n".join(parts)
+
+    static = property(lambda x: True)
+    constructor = property(lambda x: False)
+
+
+class Method(Function):
+    typename = "method"
+
+    static = property(_hasopt("static"))
+    constructor = property(_hasopt("constructor"))
+
+
+class Param:
+    def __init__(self, name, typename, opts=Options(), desc=""):
+        self.name = name
+        self.type = typename
+        self.options = opts
+        self.desc = desc
+
+        self.caps = set()
+        for key, values in opts.items():
+            if key == "cap":
+                self.caps.update(values)
+
+    def __str__(self):
+        return "param {} {} {}  {}".format(
+                self.name, repr(self.type), self.options, self.desc or "")
+
+    @property
+    def outparam(self):
+        return "out" in self.options or "inout" in self.options
+
+    @property
+    def refparam(self):
+        return self.type.reference or self.outparam
+
+    @property
+    def optional(self):
+        if "optional" in self.options: return "optional"
+        elif "zero_ok" in self.options: return "zero_ok"
+        else: return "required"
+

scripts/definitions/types/interface.py

@@ -0,0 +1,46 @@
+from . import _indent
+from . import Options
+
+class Expose(object):
+    def __init__(self, type):
+        self.type = type
+
+    def __repr__(self):
+        return f'expose {repr(self.type)}'
+
+class Interface:
+    def __init__(self, name, uid, opts=Options(), desc="", children=tuple()):
+        from .function import Function
+
+        self.name = name
+        self.uid = uid
+        self.options = opts
+        self.desc = desc
+
+        self.functions = [c for c in children if isinstance(c, Function)]
+        self.__exposes = [e.type for e in children if isinstance(e, Expose)]
+
+    def __str__(self):
+        parts = [f"interface {self.name}: {self.uid}"]
+        if self.desc:
+            parts.append(_indent(self.desc))
+        if self.options:
+            parts.append(f"    Options: {self.options}")
+        parts.extend(map(_indent, self.exposes))
+        parts.extend(map(_indent, self.functions))
+        return "\n".join(parts)
+
+    @property
+    def methods(self):
+        mm = [(i, None, self.functions[i]) for i in range(len(self.functions))]
+
+        base = len(mm)
+        for o in self.exposes:
+            mm.extend([(base + i, o, o.methods[i]) for i in range(len(o.methods))])
+            base += len(o.methods)
+
+        return mm
+
+    @property
+    def exposes(self):
+        return [ref.object for ref in self.__exposes]

scripts/definitions/types/object.py

@@ -0,0 +1,33 @@
+from . import _indent
+from . import Caps, Options
+
+class Object:
+    def __init__(self, name, uid, typename=None, opts=Options(), caps=Caps(), desc="", children=tuple(), cname=None):
+        self.name = name
+        self.uid = uid
+        self.options = opts
+        self.desc = desc
+        self.methods = children
+        self.cname = cname or name
+        self.caps = caps
+
+        self.__super = typename
+
+        from . import ObjectRef
+        self.__ref = ObjectRef(name)
+
+    def __str__(self):
+        parts = [f"object {self.name}: {self.uid}"]
+        if self.desc:
+            parts.append(_indent(self.desc))
+        if self.options:
+            parts.append(f"    Options: {self.options}")
+        parts.extend(map(_indent, self.methods))
+        return "\n".join(parts)
+
+    reftype = property(lambda self: self.__ref)
+
+    @property
+    def super(self):
+        if self.__super is not None:
+            return self.__super.object

scripts/definitions/types/objref.py

@@ -0,0 +1,49 @@
+from .type import Type
+
+class ObjectRef(Type):
+    all_refs = {}
+
+    def __init__(self, name, filename=None):
+        super().__init__(name)
+        self.__c_type = "j6_handle_t"
+        self.__object = None
+        ObjectRef.all_refs[self] = filename
+
+    def __repr__(self):
+        return f'ObjectRef({self.name})'
+
+    object = property(lambda self: self.__object)
+
+    def c_names(self, options):
+        one = self.__c_type
+        out = bool({"out", "inout"}.intersection(options))
+
+        if "list" in options:
+            two = "size_t"
+            one = f"{one} *"
+
+            if out:
+                two += " *"
+            return ((one, ""), (two, "_count"))
+
+        else:
+            if out:
+                one += " *"
+            return ((one, ""),)
+
+    def cxx_names(self, options):
+        if not self.needs_object(options):
+            return self.c_names(options)
+        return ((f"obj::{self.name} *", ""),)
+
+    def needs_object(self, options):
+        return not bool({"out", "inout", "list", "handle"}.intersection(options))
+
+
+    @classmethod
+    def connect(cls, objects):
+        for ref, filename in cls.all_refs.items():
+            ref.__object = objects.get(ref.name)
+            if ref.__object is None:
+                from ..errors import UnknownTypeError
+                raise UnknownTypeError(ref.name, filename)

scripts/definitions/types/primitive.py

@@ -0,0 +1,72 @@
+from .type import Type
+
+class Primitive(Type):
+    def __init__(self, name, c_type):
+        super().__init__(name)
+        self.c_type = c_type
+
+    def __repr__(self):
+        return f'Primitive({self.name})'
+
+    def c_names(self, options=dict()):
+        one = self.c_type
+        if "out" in options or "inout" in options:
+            one += " *"
+
+        return ((one, ""),)
+
+    def cxx_names(self, options):
+        return self.c_names(options)
+
+class PrimitiveRef(Primitive):
+    def __init__(self, name, c_type, counted=False):
+        super().__init__(name, c_type)
+        self.__counted = counted
+
+    reference = property(lambda self: True)
+
+    def c_names(self, options=dict()):
+        one = f"{self.c_type} *"
+        two = "size_t"
+
+        if "out" in options or "inout" in options:
+            two += " *"
+        else:
+            one = "const " + one
+
+        if self.__counted:
+            return ((one, ""), (two, "_len"))
+        else:
+            return ((one, ""),)
+
+    def cxx_names(self, options):
+        return self.c_names(options)
+
+_primitives = {
+    "string": PrimitiveRef("string", "char"),
+    "buffer": PrimitiveRef("buffer", "void", counted=True),
+
+    "int": Primitive("int", "int"),
+    "uint": Primitive("uint", "unsigned"),
+    "size": Primitive("size", "size_t"),
+    "address": Primitive("address", "uintptr_t"),
+
+    "int8": Primitive("int8", "int8_t"),
+    "uint8": Primitive("uint8", "uint8_t"),
+
+    "int16": Primitive("int16", "int16_t"),
+    "uint16": Primitive("uint16", "uint16_t"),
+
+    "int32": Primitive("int32", "int32_t"),
+    "uint32": Primitive("uint32", "uint32_t"),
+
+    "int64": Primitive("int64", "int64_t"),
+    "uint64": Primitive("uint64", "uint64_t"),
+}
+
+def get_primitive(name):
+    p = _primitives.get(name)
+    if not p:
+        from ..errors import InvalidType
+        raise InvalidType(name)
+    return p

scripts/definitions/types/type.py

@@ -0,0 +1,15 @@
+class Type:
+    def __init__(self, name):
+        self.__name = name
+
+    name = property(lambda self: self.__name)
+    reference = property(lambda self: False)
+
+    def c_names(self, options):
+        raise NotImplemented("Call to base Type.c_names")
+
+    def cxx_names(self, options):
+        raise NotImplemented("Call to base Type.c_names")
+
+    def needs_object(self, options):
+        return False

scripts/fontpsf.py

@@ -0,0 +1,75 @@
+_MAGIC = (0x72, 0xb5, 0x4a, 0x86)
+_VERSION = 0
+
+class PSF2:
+    from collections import namedtuple
+    Header = namedtuple("PSF2Header",
+        ["version", "offset", "flags", "count", "charsize", "height", "width"])
+
+    def __init__(self, filename, header, data):
+        self.__filename = filename
+        self.__header = header
+        self.__data = data
+
+    data = property(lambda self: self.__data)
+    header = property(lambda self: self.__header)
+    count = property(lambda self: self.__header.count)
+    charsize = property(lambda self: self.__header.charsize)
+    dimension = property(lambda self: (self.__header.width, self.__header.height))
+    filename = property(lambda self: self.__filename)
+
+    @classmethod
+    def load(cls, filename):
+        from os.path import basename
+        from struct import unpack_from
+
+        data = open(filename, 'rb').read()
+
+        fmt = "BBBBIIIIIII"
+        values = unpack_from(fmt, data)
+
+        if values[:len(_MAGIC)] != _MAGIC:
+            raise Exception("Bad magic number in header")
+
+        header = PSF2.Header(*values[len(_MAGIC):])
+        if header.version != _VERSION:
+            raise Exception(f"Bad version {header.version} in header")
+
+        return cls(basename(filename), header, data)
+
+    class Glyph:
+        __slots__ = ['index', 'data']
+        def __init__(self, i, data):
+            self.index = i
+            self.data = data
+        def __index__(self):
+            return self.index
+        def empty(self):
+            return not bool([b for b in self.data if b != 0])
+        def description(self):
+            c = chr(self.index)
+            if c.isprintable():
+                return "Glyph {:02x}: '{}'".format(self.index, c)
+            else:
+                return "Glyph {:02x}".format(self.index)
+
+    def __getitem__(self, i):
+        c = self.__header.charsize
+        n = i * c + self.__header.offset
+        return PSF2.Glyph(i, self.__data[n:n+c])
+
+    class __iter:
+        __slots__ = ['font', 'n']
+        def __init__(self, font):
+            self.font = font
+            self.n = 0
+        def __next__(self):
+            if self.n < self.font.count:
+                glyph = self.font[self.n]
+                self.n += 1
+                return glyph
+            else:
+                raise StopIteration
+
+    def __iter__(self):
+        return PSF2.__iter(self)

scripts/idgen

@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+
+def hashid(s):
+    from hashlib import shake_128 as sh
+    return sh(s.encode('utf-8')).hexdigest(8)
+
+import sys
+
+for arg in sys.argv[1:]:
+    id = hashid(arg)
+    print(f"{arg}: {id}")

scripts/j6libc.py

@@ -0,0 +1,56 @@
+import cog
+
+supported_architectures = {
+    "amd64": "__amd64__",
+}
+
+def arch_includes(header):
+    prefix = "if"
+    for arch, define in supported_architectures.items():
+        cog.outl(f"#{prefix} defined({define})")
+        cog.outl(f"#include <__j6libc/arch/{arch}/{header}>")
+        prefix = "elif"
+    cog.outl("#endif")
+
+
+int_widths = (8, 16, 32, 64)
+int_mods = ("fast", "least")
+
+int_types = {
+    # type: abbrev
+    "char": "char",
+    "short": "short",
+    "int": "int",
+    "long": "long",
+    "long long": "llong",
+}
+
+def definition(kind, name, val, width=24):
+    cog.outl(f"{kind} {name:{width}} {val}")
+
+
+atomic_types = {
+    "_Bool": "bool",
+    "signed char": "schar",
+    "char16_t": "char16_t",
+    "char32_t": "char32_t",
+    "wchar_t": "wchar_t",
+    "wchar_t": "wchar_t",
+    "size_t": "size_t",
+    "ptrdiff_t": "ptrdiff_t",
+    "intptr_t": "intptr_t",
+    "uintptr_t": "uintptr_t",
+    "intmax_t": "intmax_t",
+    "uintmax_t": "uintmax_t",
+    }
+
+for name, abbrev in int_types.items():
+    atomic_types.update({name: abbrev, f"unsigned {name}": f"u{abbrev}"})
+
+for width in int_widths:
+    atomic_types.update({t: t for t in (
+            f"int_least{width}_t",
+            f"uint_least{width}_t",
+            f"int_fast{width}_t",
+            f"uint_fast{width}_t")})
+

scripts/memory.py

@@ -0,0 +1,32 @@
+class Layout:
+    from collections import namedtuple
+    Region = namedtuple("Region", ("name", "start", "size", "shared"))
+
+    sizes = {'G': 1024 ** 3, 'T': 1024 ** 4}
+
+    @staticmethod
+    def get_size(desc):
+        size, mag = int(desc[:-1]), desc[-1]
+
+        try:
+            mult = Layout.sizes[mag]
+        except KeyError:
+            raise RuntimeError(f"No magnitude named '{mag}'.")
+
+        return size * mult
+
+    def __init__(self, path):
+        from yaml import safe_load
+
+        regions = []
+        addr = 1 << 64
+
+        with open(path, 'r') as infile:
+            data = safe_load(infile.read())
+            for r in data:
+                size = Layout.get_size(r["size"])
+                addr -= size
+                regions.append(Layout.Region(r["name"], addr, size,
+                    r.get("shared", False)))
+
+        self.regions = tuple(regions)

scripts/parse_font.py

@@ -1,21 +1,6 @@
 #!/usr/bin/env python3
 
-MAGIC = (0x72, 0xb5, 0x4a, 0x86)
+from fontpsf import PSF2
-
-from collections import namedtuple
-PSF2 = namedtuple("PSF2", ["version", "offset", "flags", "count", "charsize", "height", "width"])
-
-def read_header(data):
-    from struct import unpack_from, calcsize
-
-    fmt = "BBBBIIIIIII"
-
-    values = unpack_from(fmt, data)
-    if values[:len(MAGIC)] != MAGIC:
-        raise Exception("Bad magic number in header")
-
-    return PSF2(*values[len(MAGIC):])
-
 
 def print_glyph(header, data):
     bw = (header.width + 7) // 8
@@ -28,16 +13,15 @@ def print_glyph(header, data):
 
 
 def display_font(filename):
-    data = open(filename, 'rb').read()
+    font = PSF2.load(filename)
+    print(font.header)
 
-    header = read_header(data)
+    for glyph in font:
-    print(header)
+        if glyph.empty():
-
+            print("{}: BLANK".format(glyph.description()))
-    c = header.charsize
+        else:
-    for i in range(0, header.count):
+            print("{}:".format(glyph.description()))
-        n = i * c + header.offset
+            print_glyph(font.header, glyph.data)
-        print("Glyph {}:".format(i))
-        print_glyph(header, data[n:n+c])
 
 
 if __name__ == "__main__":

scripts/parse_syms.py

@@ -1,10 +0,0 @@
-#!/usr/bin/env python3
-
-def parse_elf(filename):
-    import struct
-    with open(filename, 'rb') as elf:
-
-if __name__ == "__main__":
-    import sys
-    for arg in sys.argv[1:]:
-        parse_elf(arg)

scripts/readme_for_prebuilt_sysroots.md

@@ -1,14 +0,0 @@
-# jsix OS sysroot
-
-This is a pre-built sysroot for building the jsix operating system kernel,
-bootloader, and utilities. This package is provided as a convenience, and
-contains software from the following repositories.
-
-## The LLVM toolchain
-
-The LLVM sources as downloaded via git from [llvm.org][llvm] under the terms of
-the [Apache License v2.0][apache2], modified [as described here][llvmlic].
-
-[llvm]: https://llvm.org
-[apache2]: https://www.apache.org/licenses/LICENSE-2.0
-[llvmlic]: https://llvm.org/docs/DeveloperPolicy.html#new-llvm-project-license-framework

scripts/sysconf.py

@@ -0,0 +1,17 @@
+class Sysconf:
+    from collections import namedtuple
+    Var = namedtuple("Var", ("name", "section", "type"))
+
+    def __init__(self, path):
+        from yaml import safe_load
+
+        sys_vars = []
+
+        with open(path, 'r') as infile:
+            data = safe_load(infile.read())
+            self.address = data["address"]
+
+            for v in data["vars"]:
+                sys_vars.append(Sysconf.Var(v["name"], v["section"], v["type"]))
+
+        self.vars = tuple(sys_vars)

scripts/templates/build.ninja.j2

@@ -1,214 +0,0 @@
-ninja_required_version = 1.3
-builddir = {{ buildroot }}
-srcroot = {{ srcroot }}
-modulefile = {{ modulefile }}
-
-{%- for var, value in vars %}
-{{ var }} = {{ value }}
-{%- endfor %}
-
-warnflags = $
-    -Wformat=2 $
-    -Winit-self $
-    -Wfloat-equal $
-    -Winline $
-    -Wmissing-format-attribute $
-    -Wmissing-include-dirs $
-    -Wswitch $
-    -Wundef $
-    -Wdisabled-optimization $
-    -Wpointer-arith $
-    -Wno-attributes $
-    -Wno-sign-compare $
-    -Wno-multichar $
-    -Wno-div-by-zero $
-    -Wno-endif-labels $
-    -Wno-pragmas $
-    -Wno-format-extra-args $
-    -Wno-unused-result $
-    -Wno-deprecated-declarations $
-    -Wno-unused-function $
-    -Wno-address-of-packed-member $
-    -Werror
-
-ccflags = $
-    -I${srcroot}/src/include $
-    -I${srcroot}/src/include/x86_64 $
-    -fcolor-diagnostics $
-    -DVERSION_MAJOR={{ version_major }} $
-    -DVERSION_MINOR={{ version_minor }} $
-    -DVERSION_PATCH={{ version_patch }} $
-    -DVERSION_GITSHA=0x0{{ version_sha }} $
-    -DGIT_VERSION=\"{{ version }}\" $
-    -DGIT_VERSION_WIDE=L\"{{ version }}\" $
-    $warnflags
-
-asflags = $
-    -DVERSION_MAJOR={{ version_major }} $
-    -DVERSION_MINOR={{ version_minor }} $
-    -DVERSION_PATCH={{ version_patch }} $
-    -DVERSION_GITSHA=0x{{ version_sha }} $
-    -I${srcroot}/src/include
-
-cflags = -std=c11
-cxxflags = -std=c++17
-libs =
-
-rule c
-    deps = gcc
-    depfile = $out.d
-    description = Compiling $name
-    command = $cc -MMD -MF $out.d $ccflags $cflags -o $out -c $in
-
-rule dump_c_defs
-    description = Dumping C defines for $target
-    command = echo "" | $cc $ccflags $cflags -dM -E - > $out
-
-rule dump_c_run
-    description = Dumping C arguments for $target
-    command = $
-        echo "#!/bin/bash" > $out; $
-        echo '$cc $ccflags $cflags $$*' >> $out; $
-        chmod a+x $out
-
-rule cpp
-    deps = gcc
-    depfile = $out.d
-    description = Compiling $name
-    command = $cxx -MMD -MF $out.d $cxxflags $ccflags -o $out -c $in
-
-rule dump_cpp_defs
-    description = Dumping C++ defines for $target
-    command = echo "" | $cxx -x c++ $cxxflags $ccflags -dM -E - > $out
-
-rule dump_cpp_run
-    description = Dumping C++ arguments for $target
-    command = $
-        echo "#!/bin/bash" > $out; $
-        echo '$cc $cxxflags $ccflags $$*' >> $out; $
-        chmod a+x $out
-
-rule s
-    deps = gcc
-    depfile = $out.d
-    description = Assembling $name
-    command = $nasm -o $out -felf64 -MD $out.d $asflags $in
-
-rule exe
-    description = Linking $name
-    command = $ld $ldflags -o $out $in $libs
-
-rule lib
-    description = Archiving $name
-    command = $ar qcs $out $in
-
-rule regen
-    generator = true
-    description = Regenrating build files
-    command = $
-        {{ generator }} $
-        --file $modulefile $
-        --dir $builddir $
-        generate
-
-rule cp
-    description = Copying $name
-    command = cp $in $out
-
-rule dump
-    description = Dumping decompiled $name
-    command = objdump -DSC -M intel $in > $out
-
-rule makest
-    description = Making symbol table
-    command = nm $in | ${srcroot}/scripts/build_symbol_table.py $out
-
-rule makeefi
-    description = Converting $name
-    command = objcopy $
-        -j .text $
-        -j .sdata $
-        -j .data $
-        -j .dynamic $
-        -j .dynsym $
-        -j .rel $
-        -j .rela $
-        -j .reloc $
-        --target=efi-app-x86_64 $
-        $in $out
-
-rule makefat
-    description = Creating $name
-    command = $
-        cp $srcroot/assets/diskbase.img $out; $
-        mcopy -s -D o -i $out@@1M $builddir/fatroot/* ::/
-
-rule strip
-    description = Stripping $name
-    command = $
-        cp $in $out; $
-        objcopy --only-keep-debug $out $out.debug; $
-        strip -g $out; $
-        objcopy --add-gnu-debuglink=$out.debug $out
-
-{% for target in targets %}
-subninja {{ target }}/target.ninja
-{% endfor %}
-
-build $
-    {%- for buildfile in buildfiles %}
-    {{ buildfile }} $
-    {%- endfor %}
-    : regen | $
-    {%- for template in templates %}
-    {{ template }} $
-    {%- endfor %}
-    $modulefile $
-    {{ generator }}
-
-build $builddir/ovmf_vars.fd : cp $srcroot/assets/ovmf/x64/ovmf_vars.fd
-    name = ovmf_vars.fd
-
-build $builddir/ovmf_vars_d.fd : cp $srcroot/assets/ovmf/x64/ovmf_vars_d.fd
-    name = ovmf_vars_d.fd
-
-build $builddir/jsix.elf | $builddir/jsix.elf.debug : strip $builddir/host/jsix.elf
-    name = kernel
-
-build $builddir/jsix.dump : dump $builddir/host/jsix.elf
-    name = kernel
-
-build $builddir/jsix.elf-gdb.py : cp ${srcroot}/assets/debugging/jsix.elf-gdb.py
-    name = kernel debug python scripts
-
-build $builddir/fatroot/jsix.elf : cp $builddir/jsix.elf
-    name = kernel to FAT image
-
-build $builddir/fatroot/efi/boot/bootx64.efi : cp $builddir/boot/boot.efi
-    name = bootloader to FAT image
-
-build $builddir/fatroot/nulldrv.elf : cp $builddir/user/nulldrv.elf
-    name = null driver to FAT image
-
-build $builddir/fatroot/terminal.elf : cp $builddir/user/nulldrv.elf
-    name = terminal driver to FAT image
-
-build ${builddir}/fatroot/symbol_table.dat : makest ${builddir}/jsix.elf
-
-build $builddir/jsix.img : makefat | $
-    $builddir/fatroot/symbol_table.dat $
-    $builddir/fatroot/nulldrv.elf $
-    $builddir/fatroot/terminal.elf $
-    $builddir/fatroot/jsix.elf $
-    $builddir/fatroot/efi/boot/bootx64.efi
-    name = jsix.img
-
-default $
-    $builddir/ovmf_vars.fd $
-    $builddir/ovmf_vars_d.fd $
-    $builddir/jsix.dump $
-    $builddir/jsix.elf-gdb.py $
-    $builddir/jsix.img
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.boot.j2

@@ -1,14 +0,0 @@
-{% extends "exe.default.j2" %}
-{% block variables %}
-{{ super() }}
-
-ccflags = $ccflags $
-    -g3 $
-    -DKERNEL_FILENAME=L\"jsix.elf\" $
-    -I${srcroot}/external/include $
-    -I${srcroot}/external/include/X64
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.default.j2

@@ -1,8 +0,0 @@
-{% extends "module.base.j2" %}
-{% block variables %}
-{{ super() }}
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.kernel.j2

@@ -1,13 +0,0 @@
-{% extends "exe.default.j2" %}
-{% block variables %}
-{{ super() }}
-
-asflags = $asflags -I${srcroot}/src/kernel/
-libs = $libs
-ldflags = $ldflags -T ${srcroot}/src/arch/x86_64/kernel.ld
-ccflags = $ccflags -I${srcroot}/external
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.makerd.j2

@@ -1,10 +0,0 @@
-{% extends "exe.default.j2" %}
-{% block variables %}
-{{ super() }}
-
-ccflags = $ccflags -I${srcroot}/external/cpptoml
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/exe.tests.j2

@@ -1,10 +0,0 @@
-{% extends "exe.default.j2" %}
-{% block variables %}
-{{ super() }}
-
-ccflags = $ccflags -ggdb -I${srcroot}/external/catch
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/lib.default.j2

@@ -1,4 +0,0 @@
-{% extends "module.base.j2" %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/module.base.j2

@@ -1,45 +0,0 @@
-moddir = ${builddir}/{{ name }}.dir
-
-{% block variables %}
-ccflags = $ccflags $
-    {%- for dep in depmods %}
-    {%- for inc in dep.includes %}
-    -I${srcroot}/{{ inc }} $
-    {%- endfor %}
-    {%- endfor %}
-    {%- for inc in module.includes %}
-    -I${srcroot}/{{ inc }} $
-    {%- endfor %}
-    {%- for define in module.defines %}
-    -D{{ define }} $
-    {%- endfor %}
-{% endblock %}
-
-{% for source in module.source %}
-build ${moddir}/{{ source.output }} : {{ source.action }} ${srcroot}/{{ source.input }} || {{ buildfile }}
-    name = {{ source.name }}
-{% endfor %}
-
-build ${builddir}/{{ module.output }} : {{ module.kind }} $
-{%- for source in module.source %}
-    ${moddir}/{{ source.output }} $
-{%- endfor -%}
-{%- for dep in deplibs %}
-    ${builddir}/{{ dep.output }} $
-{%- endfor %}
-    | $
-{%- for dep in depexes %}
-    ${builddir}/{{ dep.output }} $
-{%- endfor %}
-    {{ buildfile }}
-    name = {{ name }}
-
-{% if module.default %}
-default ${builddir}/{{ module.output }}
-{% endif %}
-
-{% block extra %}
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.boot.j2

@@ -1,38 +0,0 @@
-{% extends "target.default.j2" %}
-
-{% block binaries %}
-cc = clang
-cxx = clang++
-ld = clang++
-ar = ar
-nasm = nasm
-objcopy = objcopy
-{% endblock %}
-
-{% block variables %}
-
-ccflags = $ccflags $
-    -I $srcroot/external $
-    --target=x86_64-unknown-windows $
-	-ffreestanding $
-	-mno-red-zone $
-	-fshort-wchar $
-	-fno-omit-frame-pointer $
-	-ggdb
-
-cxxflags = $cxxflags $
-    -fno-rtti $
-    -fno-exceptions
-
-ldflags = $ldflags $
-    --target=x86_64-unknown-windows $
-	-nostdlib $
-    -Wl,-entry:efi_main $
-    -Wl,-subsystem:efi_application $
-    -fuse-ld=lld-link $
-	-g
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.default.j2

@@ -1,31 +0,0 @@
-builddir = $builddir/{{ target }}
-target = {{ target }}
-
-{% block variables %}
-{% endblock %}
-
-{% block binaries %}
-cc = clang
-cxx = clang++
-ld = ld
-ar = ar
-nasm = nasm
-objcopy = objcopy
-{% endblock %}
-
-{% for module in modules %}
-subninja {{ module }}.ninja
-{% endfor %}
-
-build ${builddir}/c.defs : dump_c_defs | {{ buildfile }}
-build ${builddir}/cpp.defs : dump_cpp_defs | {{ buildfile }}
-build ${builddir}/c.run : dump_c_run | {{ buildfile }}
-build ${builddir}/cpp.run : dump_cpp_run | {{ buildfile }}
-
-default ${builddir}/c.defs
-default ${builddir}/cpp.defs
-default ${builddir}/c.run
-default ${builddir}/cpp.run
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.host.j2

@@ -1,44 +0,0 @@
-{% extends "target.default.j2" %}
-
-{% block binaries %}
-cc = ${srcroot}/sysroot/bin/clang
-cxx = ${srcroot}/sysroot/bin/clang++
-ld = ${srcroot}/sysroot/bin/ld.lld
-ar = ${srcroot}/sysroot/bin/ar
-nasm = nasm
-objcopy = ${srcroot}/sysroot/bin/objcopy
-{% endblock %}
-
-{% block variables %}
-
-ccflags = $ccflags $
-    -nostdlib $
-    -ffreestanding $
-    -nodefaultlibs $
-    -fno-builtin $
-    -mno-sse $
-    -fno-omit-frame-pointer $
-    -mno-red-zone $
-    -g $
-    -mcmodel=large $
-    -D__ELF__ $
-    -D__JSIX__ $
-    -isystem${srcroot}/sysroot/include $
-    -isystem${srcroot}/src/libraries/libc/include $
-    --sysroot="${srcroot}/sysroot"
-
-cxxflags = $cxxflags $
-    -fno-exceptions $
-    -fno-rtti $
-    -isystem${srcroot}/sysroot/include/c++/v1
-
-ldflags = $ldflags $
-    -g $
-    -nostdlib $
-    -Bsymbolic $
-    -Bstatic
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.native.j2

@@ -1,15 +0,0 @@
-{% extends "target.default.j2" %}
-
-{% block binaries %}
-{{ super() }}
-ld = clang++
-{% endblock %}
-
-{% block variables %}
-
-ccflags = $ccflags -g -ggdb
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/target.user.j2

@@ -1,45 +0,0 @@
-{% extends "target.default.j2" %}
-
-{% block binaries %}
-cc = ${srcroot}/sysroot/bin/clang
-cxx = ${srcroot}/sysroot/bin/clang++
-ld = ${srcroot}/sysroot/bin/ld.lld
-ar = ${srcroot}/sysroot/bin/ar
-nasm = nasm
-objcopy = ${srcroot}/sysroot/bin/objcopy
-{% endblock %}
-
-{% block variables %}
-
-ccflags = $ccflags $
-    -nostdlib $
-    -nodefaultlibs $
-    -fno-builtin $
-    -mno-sse $
-    -fno-omit-frame-pointer $
-    -mno-red-zone $
-    -g $
-    -mcmodel=large $
-    -D__ELF__ $
-    -D__JSIX__ $
-    -isystem${srcroot}/sysroot/include $
-    -isystem${srcroot}/src/libraries/libc/include $
-    --sysroot="${srcroot}/sysroot"
-
-cxxflags = $cxxflags $
-    -fno-exceptions $
-    -fno-rtti $
-    -isystem${srcroot}/sysroot/include/c++/v1
-
-ldflags = $ldflags $
-    -g $
-    -nostdlib $
-    -Bsymbolic $
-    -Bstatic $
-    --sysroot="${srcroot}/sysroot" $
-	-L "${srcroot}/sysroot/lib" $
-
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

src/arch/x86_64/config.mk

@@ -1,6 +0,0 @@
-AS := nasm
-ASFLAGS := -felf64
-LDFLAGS := -m elf_x86_64
-CFLAGS := -march=nocona -m64
-
-# vim:ft=make

src/arch/x86_64/details.c

@@ -1 +0,0 @@
-const char *KERNEL_PLATFORM = "x86_64";

src/boot/allocator.cpp

@@ -0,0 +1,173 @@
+#include <uefi/boot_services.h>
+#include <uefi/types.h>
+
+#include <bootproto/init.h>
+#include <bootproto/kernel.h>
+#include <util/no_construct.h>
+#include <util/pointers.h>
+
+#include "allocator.h"
+#include "error.h"
+#include "memory.h"
+
+namespace boot {
+
+util::no_construct<memory::allocator> __g_alloc_storage;
+memory::allocator &g_alloc = __g_alloc_storage.value;
+
+namespace memory {
+
+using bootproto::allocation_register;
+using bootproto::module;
+using bootproto::page_allocation;
+
+static_assert(sizeof(allocation_register) == page_size);
+
+
+void
+allocator::init(
+        allocation_register *&allocs,
+        modules_page *&modules,
+        uefi::boot_services *bs)
+{
+    new (&g_alloc) allocator(*bs);
+    allocs = g_alloc.m_register;
+    modules = g_alloc.m_modules;
+}
+
+
+allocator::allocator(uefi::boot_services &bs) :
+    m_bs(bs),
+    m_register(nullptr),
+    m_modules(nullptr)
+{
+    add_register();
+    add_modules();
+}
+
+void
+allocator::add_register()
+{
+    allocation_register *reg = nullptr;
+
+    try_or_raise(
+        m_bs.allocate_pages(uefi::allocate_type::any_pages,
+            uefi::memory_type::loader_data, 1, reinterpret_cast<void**>(&reg)),
+        L"Failed allocating allocation register page");
+
+    m_bs.set_mem(reg, sizeof(allocation_register), 0);
+
+    if (m_register)
+        m_register->next = reg;
+
+    m_register = reg;
+    return;
+}
+
+void
+allocator::add_modules()
+{
+    modules_page *mods = reinterpret_cast<modules_page*>(
+        allocate_pages(1, alloc_type::init_args, true));
+
+    if (m_modules)
+        m_modules->next = reinterpret_cast<uintptr_t>(mods);
+
+    mods->modules = reinterpret_cast<module*>(mods + 1);
+    m_modules = mods;
+    m_next_mod = mods->modules;
+    return;
+}
+
+void *
+allocator::allocate_pages(size_t count, alloc_type type, bool zero)
+{
+    if (count & ~0xffffffffull) {
+        error::raise(uefi::status::unsupported,
+            L"Cannot allocate more than 16TiB in pages at once.",
+            __LINE__);
+    }
+
+    if (!m_register || m_register->count == 0xff)
+        add_register();
+
+    void *pages = nullptr;
+
+    try_or_raise(
+        m_bs.allocate_pages(uefi::allocate_type::any_pages,
+            uefi::memory_type::loader_data, count, &pages),
+        L"Failed allocating usable pages");
+
+    page_allocation &ent = m_register->entries[m_register->count++];
+    ent.address = reinterpret_cast<uintptr_t>(pages);
+    ent.count = count;
+    ent.type = type;
+
+    if (zero)
+        m_bs.set_mem(pages, count * page_size, 0);
+
+    return pages;
+}
+
+module *
+allocator::allocate_module_untyped(size_t size)
+{
+    size_t remaining =
+        reinterpret_cast<uintptr_t>(m_modules) + page_size
+        - reinterpret_cast<uintptr_t>(m_next_mod);
+
+    if (size > remaining)
+        add_modules();
+
+    ++m_modules->count;
+    module *m = m_next_mod;
+    m_next_mod = util::offset_pointer(m_next_mod, size);
+
+    m->mod_length = size;
+    return m;
+}
+
+void *
+allocator::allocate(size_t size, bool zero)
+{
+    void *p = nullptr;
+    try_or_raise(
+        m_bs.allocate_pool(uefi::memory_type::loader_data, size, &p),
+        L"Could not allocate pool memory");
+
+    if (zero)
+        m_bs.set_mem(p, size, 0);
+
+    return p;
+}
+
+void
+allocator::free(void *p)
+{
+    try_or_raise(
+        m_bs.free_pool(p),
+        L"Freeing pool memory");
+}
+
+void
+allocator::memset(void *start, size_t size, uint8_t value)
+{
+    m_bs.set_mem(start, size, value);
+}
+
+void
+allocator::copy(void *to, void *from, size_t size)
+{
+    m_bs.copy_mem(to, from, size);
+}
+
+} // namespace memory
+} // namespace boot
+
+
+void * operator new (size_t size, void *p) { return p; }
+void * operator new(size_t size)           { return boot::g_alloc.allocate(size); }
+void * operator new [] (size_t size)       { return boot::g_alloc.allocate(size); }
+void operator delete (void *p) noexcept    { return boot::g_alloc.free(p); }
+void operator delete [] (void *p) noexcept { return boot::g_alloc.free(p); }
+

src/boot/allocator.h

@@ -0,0 +1,76 @@
+#pragma once
+/// \file allocator.h
+/// Page allocator class definition
+
+namespace uefi {
+    class boot_services;
+}
+
+namespace bootproto {
+    enum class allocation_type : uint8_t;
+    struct allocation_register;
+    struct module;
+    struct modules_page;
+}
+
+namespace boot {
+namespace memory {
+
+using alloc_type = bootproto::allocation_type;
+
+class allocator
+{
+public:
+    using allocation_register = bootproto::allocation_register;
+    using module = bootproto::module;
+    using modules_page = bootproto::modules_page;
+
+    allocator(uefi::boot_services &bs);
+
+    void * allocate(size_t size, bool zero = false);
+    void free(void *p);
+
+    void * allocate_pages(size_t count, alloc_type type, bool zero = false);
+
+    template <typename M>
+    M * allocate_module(size_t extra = 0) {
+        return static_cast<M*>(allocate_module_untyped(sizeof(M) + extra));
+    }
+
+    void memset(void *start, size_t size, uint8_t value);
+    void copy(void *to, void *from, size_t size);
+
+    inline void zero(void *start, size_t size) { memset(start, size, 0); }
+
+    /// Initialize the global allocator
+    /// \arg allocs   [out] Poiinter to the initial allocation register
+    /// \arg modules  [out] Pointer to the initial modules_page
+    /// \arg bs       UEFI boot services
+    static void init(
+            allocation_register *&allocs,
+            modules_page *&modules,
+            uefi::boot_services *bs);
+
+private:
+    void add_register();
+    void add_modules();
+    module * allocate_module_untyped(size_t size);
+
+    uefi::boot_services &m_bs;
+
+    allocation_register *m_register;
+    modules_page *m_modules;
+    module *m_next_mod;
+};
+
+} // namespace memory
+
+extern memory::allocator &g_alloc;
+
+} // namespace boot
+
+void * operator new (size_t size, void *p);
+void * operator new(size_t size);
+void * operator new [] (size_t size);
+void operator delete (void *p) noexcept;
+void operator delete [] (void *p) noexcept;

src/boot/boot.module

@@ -0,0 +1,23 @@
+# vim: ft=python
+
+boot = module("boot",
+    kind = "exe",
+    output = "boot.efi",
+    targets = [ "boot" ],
+    deps = [ "cpu", "elf", "util", "bootproto" ],
+    sources = [
+        "allocator.cpp",
+        "bootconfig.cpp",
+        "console.cpp",
+        "error.cpp",
+        "fs.cpp",
+        "hardware.cpp",
+        "loader.cpp",
+        "main.cpp",
+        "memory.cpp",
+        "memory_map.cpp",
+        "paging.cpp",
+        "status.cpp",
+        "support.cpp",
+        "video.cpp",
+    ])

src/boot/bootconfig.cpp

@@ -0,0 +1,63 @@
+#include <uefi/boot_services.h>
+#include <uefi/types.h>
+
+#include "bootconfig.h"
+#include "error.h"
+#include "fs.h"
+#include "status.h"
+
+namespace boot {
+
+constexpr uint64_t jsixboot = 0x746f6f627869736a; // "jsixboot"
+
+static const wchar_t *
+read_string(util::buffer &data)
+{
+    uint16_t size = *util::read<uint16_t>(data);
+    const wchar_t *string = reinterpret_cast<const wchar_t*>(data.pointer);
+    data += size;
+    return string;
+}
+
+static void
+read_descriptor(descriptor &e, util::buffer &data)
+{
+    e.flags = static_cast<desc_flags>(*util::read<uint16_t>(data));
+    e.path = read_string(data);
+    e.desc = read_string(data);
+}
+
+bootconfig::bootconfig(util::buffer data, uefi::boot_services *bs)
+{
+    status_line status {L"Loading boot config"};
+
+    if (*util::read<uint64_t>(data) != jsixboot)
+        error::raise(uefi::status::load_error, L"Bad header in jsix_boot.dat");
+
+    const uint8_t version = *util::read<uint8_t>(data);
+    if (version != 0)
+        error::raise(uefi::status::incompatible_version, L"Bad version in jsix_boot.dat");
+
+    data += 1; // reserved byte
+    uint16_t num_programs = *util::read<uint16_t>(data);
+    uint16_t num_data = *util::read<uint16_t>(data);
+
+    m_flags = *util::read<uint16_t>(data);
+
+    read_descriptor(m_kernel, data);
+    read_descriptor(m_init, data);
+
+    m_programs.count = num_programs;
+    m_programs.pointer = new descriptor [num_programs];
+
+    for (unsigned i = 0; i < num_programs; ++i)
+        read_descriptor(m_programs[i], data);
+
+    m_data.count = num_programs;
+    m_data.pointer = new descriptor [num_data];
+
+    for (unsigned i = 0; i < num_data; ++i)
+        read_descriptor(m_data[i], data);
+}
+
+} // namespace boot

src/boot/bootconfig.h

@@ -0,0 +1,45 @@
+/// \file bootconfig.h
+/// Definitions for reading the jsix bootconfig file
+#pragma once
+
+#include <bootproto/bootconfig.h>
+#include <util/counted.h>
+
+namespace uefi {
+    struct boot_services;
+}
+
+namespace boot {
+
+using desc_flags = bootproto::desc_flags;
+
+struct descriptor {
+    desc_flags flags;
+    wchar_t const *path;
+    wchar_t const *desc;
+};
+
+/// A bootconfig is a manifest of potential files.
+class bootconfig
+{
+public:
+    using descriptors = util::counted<descriptor>;
+
+    /// Constructor. Loads bootconfig from the given buffer.
+    bootconfig(util::buffer data, uefi::boot_services *bs);
+
+    inline uint16_t flags() const { return m_flags; }
+    inline const descriptor & kernel() const { return m_kernel; }
+    inline const descriptor & init() const { return m_init; }
+    descriptors programs() { return m_programs; }
+    descriptors data() { return m_data; }
+
+private:
+    uint16_t m_flags;
+    descriptor m_kernel;
+    descriptor m_init;
+    descriptors m_programs;
+    descriptors m_data;
+};
+
+} // namespace boot

src/boot/console.cpp

@@ -1,9 +1,8 @@
 #include <stddef.h>
 #include <stdint.h>
 
+#include <uefi/protos/simple_text_output.h>
 #include <uefi/types.h>
-#include <uefi/graphics.h>
-#include <uefi/protos/graphics_output.h>
 
 #include "console.h"
 #include "error.h"
@@ -17,431 +16,198 @@ namespace boot {
 size_t ROWS = 0;
 size_t COLS = 0;
 
-static constexpr int level_ok = 0;
-static constexpr int level_warn = 1;
-static constexpr int level_fail = 2;
-
-static const wchar_t *level_tags[] = {
-	L"  ok  ",
-	L" warn ",
-	L"failed"
-};
-static const uefi::attribute level_colors[] = {
-	uefi::attribute::green,
-	uefi::attribute::brown,
-	uefi::attribute::light_red
-};
-
 console *console::s_console = nullptr;
-status_line *status_line::s_current = nullptr;
 
 static const wchar_t digits[] = {u'0', u'1', u'2', u'3', u'4', u'5',
-	u'6', u'7', u'8', u'9', u'a', u'b', u'c', u'd', u'e', u'f'};
+    u'6', u'7', u'8', u'9', u'a', u'b', u'c', u'd', u'e', u'f'};
 
 
-static size_t
+size_t
 wstrlen(const wchar_t *s)
 {
-	size_t count = 0;
+    size_t count = 0;
-	while (s && *s++) count++;
+    while (s && *s++) count++;
-	return count;
+    return count;
 }
 
 
-console::console(uefi::boot_services *bs, uefi::protos::simple_text_output *out) :
+console::console(uefi::protos::simple_text_output *out) :
-	m_rows(0),
+    m_rows {0},
-	m_cols(0),
+    m_cols {0},
-	m_out(out)
+    m_out {out}
 {
-	pick_mode(bs);
+    try_or_raise(
-
+        m_out->query_mode(m_out->mode->mode, &m_cols, &m_rows),
-	try_or_raise(
+        L"Failed to get text output mode.");
-		m_out->query_mode(m_out->mode->mode, &m_cols, &m_rows),
+    try_or_raise(
-		L"Failed to get text output mode.");
+        m_out->clear_screen(),
-
+        L"Failed to clear screen");
-	try_or_raise(
+    s_console = this;
-		m_out->clear_screen(),
-		L"Failed to clear screen");
-
-	m_out->set_attribute(uefi::attribute::light_cyan);
-	m_out->output_string(L"jsix loader ");
-
-	m_out->set_attribute(uefi::attribute::light_magenta);
-	m_out->output_string(GIT_VERSION_WIDE);
-
-	m_out->set_attribute(uefi::attribute::light_gray);
-	m_out->output_string(L" booting...\r\n\n");
-
-	s_console = this;
 }
 
 void
-console::pick_mode(uefi::boot_services *bs)
+console::announce()
 {
-	uefi::status status;
+    m_out->set_attribute(uefi::attribute::light_cyan);
-	uefi::protos::graphics_output *gfx_out_proto;
+    m_out->output_string(L"jsix loader ");
-	uefi::guid guid = uefi::protos::graphics_output::guid;
 
-	try_or_raise(
+    m_out->set_attribute(uefi::attribute::light_magenta);
-		bs->locate_protocol(&guid, nullptr, (void **)&gfx_out_proto),
+    m_out->output_string(GIT_VERSION_WIDE);
-		L"Failed to find a Graphics Output Protocol handle");
 
-	const uint32_t modes = gfx_out_proto->mode->max_mode;
+    m_out->set_attribute(uefi::attribute::light_gray);
-	uint32_t best = gfx_out_proto->mode->mode;
+    m_out->output_string(L" booting...\r\n");
-
-	uefi::graphics_output_mode_info *info =
-		(uefi::graphics_output_mode_info *)gfx_out_proto->mode;
-
-	uint32_t res = info->horizontal_resolution * info->vertical_resolution;
-	int is_fb = info->pixel_format != uefi::pixel_format::blt_only;
-
-	for (uint32_t i = 0; i < modes; ++i) {
-		size_t size = 0;
-
-		try_or_raise(
-			gfx_out_proto->query_mode(i, &size, &info),
-			L"Failed to find a graphics mode the driver claimed to support");
-
-#ifdef MAX_HRES
-		if (info->horizontal_resolution > MAX_HRES) continue;
-#endif
-
-		const uint32_t new_res = info->horizontal_resolution * info->vertical_resolution;
-		const int new_is_fb = info->pixel_format != uefi::pixel_format::blt_only;
-
-		if (new_is_fb > is_fb && new_res >= res) {
-			best = i;
-			res = new_res;
-		}
-	}
-
-	try_or_raise(
-		gfx_out_proto->set_mode(best),
-		L"Failed to set graphics mode");
 }
 
 size_t
 console::print_hex(uint32_t n) const
 {
-	wchar_t buffer[9];
+    wchar_t buffer[9];
-	wchar_t *p = buffer;
+    wchar_t *p = buffer;
-	for (int i = 7; i >= 0; --i) {
+    for (int i = 7; i >= 0; --i) {
-		uint8_t nibble = (n >> (i*4)) & 0xf;
+        uint8_t nibble = (n >> (i*4)) & 0xf;
-		*p++ = digits[nibble];
+        *p++ = digits[nibble];
-	}
+    }
-	*p = 0;
+    *p = 0;
-	m_out->output_string(buffer);
+    m_out->output_string(buffer);
-	return 8;
+    return 8;
 }
 
 size_t
 console::print_long_hex(uint64_t n) const
 {
-	wchar_t buffer[17];
+    wchar_t buffer[17];
-	wchar_t *p = buffer;
+    wchar_t *p = buffer;
-	for (int i = 15; i >= 0; --i) {
+    for (int i = 15; i >= 0; --i) {
-		uint8_t nibble = (n >> (i*4)) & 0xf;
+        uint8_t nibble = (n >> (i*4)) & 0xf;
-		*p++ = digits[nibble];
+        *p++ = digits[nibble];
-	}
+    }
-	*p = 0;
+    *p = 0;
-	m_out->output_string(buffer);
+    m_out->output_string(buffer);
-	return 16;
+    return 16;
 }
 
 size_t
 console::print_dec(uint32_t n) const
 {
-	wchar_t buffer[11];
+    wchar_t buffer[11];
-	wchar_t *p = buffer + 10;
+    wchar_t *p = buffer + 10;
-	*p-- = 0;
+    *p-- = 0;
-	do {
+    do {
-		*p-- = digits[n % 10];
+        *p-- = digits[n % 10];
-		n /= 10;
+        n /= 10;
-	} while (n != 0);
+    } while (n != 0);
 
-	m_out->output_string(++p);
+    m_out->output_string(++p);
-	return 10 - (p - buffer);
+    return 10 - (p - buffer);
 }
 
 size_t
 console::print_long_dec(uint64_t n) const
 {
-	wchar_t buffer[21];
+    wchar_t buffer[21];
-	wchar_t *p = buffer + 20;
+    wchar_t *p = buffer + 20;
-	*p-- = 0;
+    *p-- = 0;
-	do {
+    do {
-		*p-- = digits[n % 10];
+        *p-- = digits[n % 10];
-		n /= 10;
+        n /= 10;
-	} while (n != 0);
+    } while (n != 0);
 
-	m_out->output_string(++p);
+    m_out->output_string(++p);
-	return 20 - (p - buffer);
+    return 20 - (p - buffer);
 }
 
 size_t
 console::vprintf(const wchar_t *fmt, va_list args) const
 {
-	wchar_t buffer[256];
+    wchar_t buffer[256];
-	const wchar_t *r = fmt;
+    const wchar_t *r = fmt;
-	wchar_t *w = buffer;
+    wchar_t *w = buffer;
-	size_t count = 0;
+    size_t count = 0;
 
-	while (r && *r) {
+    while (r && *r) {
-		if (*r != L'%') {
+        if (*r != L'%') {
-			count++;
+            count++;
-			*w++ = *r++;
+            *w++ = *r++;
-			continue;
+            continue;
-		}
+        }
 
-		*w = 0;
+        *w = 0;
-		m_out->output_string(buffer);
+        m_out->output_string(buffer);
-		w = buffer;
+        w = buffer;
 
-		r++; // chomp the %
+        r++; // chomp the %
 
-		switch (*r++) {
+        switch (*r++) {
-			case L'%':
+            case L'%':
-				m_out->output_string(const_cast<wchar_t*>(L"%"));
+                m_out->output_string(const_cast<wchar_t*>(L"%"));
-				count++;
+                count++;
-				break;
+                break;
 
-			case L'x':
+            case L'x':
-				count += print_hex(va_arg(args, uint32_t));
+                count += print_hex(va_arg(args, uint32_t));
-				break;
+                break;
 
-			case L'd':
+            case L'd':
-			case L'u':
+            case L'u':
-				count += print_dec(va_arg(args, uint32_t));
+                count += print_dec(va_arg(args, uint32_t));
-				break;
+                break;
 
-			case L's':
+            case L's':
-				{
+                {
-					wchar_t *s = va_arg(args, wchar_t*);
+                    wchar_t *s = va_arg(args, wchar_t*);
-					count += wstrlen(s);
+                    count += wstrlen(s);
-					m_out->output_string(s);
+                    m_out->output_string(s);
-				}
+                }
-				break;
+                break;
 
-			case L'l':
+            case L'l':
-				switch (*r++) {
+                switch (*r++) {
-					case L'x':
+                    case L'x':
-						count += print_long_hex(va_arg(args, uint64_t));
+                        count += print_long_hex(va_arg(args, uint64_t));
-						break;
+                        break;
 
-					case L'd':
+                    case L'd':
-					case L'u':
+                    case L'u':
-						count += print_long_dec(va_arg(args, uint64_t));
+                        count += print_long_dec(va_arg(args, uint64_t));
-						break;
+                        break;
 
-					default:
+                    default:
-						break;
+                        break;
-				}
+                }
-				break;
+                break;
 
-			default:
+            default:
-				break;
+                break;
-		}
+        }
-	}
+    }
 
-	*w = 0;
+    *w = 0;
-	m_out->output_string(buffer);
+    m_out->output_string(buffer);
-	return count;
+    return count;
 }
 
 size_t
 console::printf(const wchar_t *fmt, ...) const
 {
-	va_list args;
+    va_list args;
-	va_start(args, fmt);
+    va_start(args, fmt);
 
-	size_t result = vprintf(fmt, args);
+    size_t result = vprintf(fmt, args);
 
-	va_end(args);
+    va_end(args);
-	return result;
+    return result;
 }
 
 size_t
 console::print(const wchar_t *fmt, ...)
 {
-	va_list args;
+    va_list args;
-	va_start(args, fmt);
+    va_start(args, fmt);
 
-	size_t result = get().vprintf(fmt, args);
+    size_t result = get().vprintf(fmt, args);
 
-	va_end(args);
+    va_end(args);
-	return result;
+    return result;
 }
 
-status_line::status_line(const wchar_t *message, const wchar_t *context) :
-	m_level(level_ok)
-{
-	auto out = console::get().m_out;
-	m_line = out->mode->cursor_row;
-	m_depth = (s_current ? 1 + s_current->m_depth : 0);
-
-	int indent = 2 * m_depth;
-	out->set_cursor_position(indent, m_line);
-	out->set_attribute(uefi::attribute::light_gray);
-	out->output_string(message);
-
-	if (context) {
-		out->output_string(L": ");
-		out->output_string(context);
-	}
-
-	out->output_string(L"\r\n");
-
-	m_next = s_current;
-	s_current = this;
-}
-
-status_line::~status_line()
-{
-	if (s_current != this)
-		error::raise(uefi::status::unsupported, L"Destroying non-current status_line");
-
-	finish();
-	if (m_next && m_level > m_next->m_level) {
-		m_next->m_level = m_level;
-		m_next->print_status_tag();
-	}
-	s_current = m_next;
-}
-
-void
-status_line::print_status_tag()
-{
-	auto out = console::get().m_out;
-	int row = out->mode->cursor_row;
-	int col = out->mode->cursor_column;
-
-	uefi::attribute color = level_colors[m_level];
-	const wchar_t *tag = level_tags[m_level];
-
-	out->set_cursor_position(50, m_line);
-
-	out->set_attribute(uefi::attribute::light_gray);
-	out->output_string(L"[");
-	out->set_attribute(color);
-	out->output_string(tag);
-	out->set_attribute(uefi::attribute::light_gray);
-	out->output_string(L"]\r\n");
-
-	out->set_cursor_position(col, row);
-}
-
-void
-status_line::do_warn(const wchar_t *message, const wchar_t *error)
-{
-	auto out = console::get().m_out;
-	int row = out->mode->cursor_row;
-
-	if (m_level < level_warn) {
-		m_level = level_warn;
-		print_status_tag();
-	}
-
-	int indent = 2 + 2 * m_depth;
-	out->set_cursor_position(indent, row);
-	out->set_attribute(uefi::attribute::yellow);
-	out->output_string(message);
-
-	if (error) {
-		out->output_string(L": ");
-		out->output_string(error);
-	}
-
-	out->set_attribute(uefi::attribute::light_gray);
-	out->output_string(L"\r\n");
-}
-
-void
-status_line::do_fail(const wchar_t *message, const wchar_t *error)
-{
-	auto out = console::get().m_out;
-	int row = out->mode->cursor_row;
-
-	if (s_current->m_level < level_fail) {
-		m_level = level_fail;
-		print_status_tag();
-	}
-
-	int indent = 2 + 2 * m_depth;
-	out->set_cursor_position(indent, row);
-	out->set_attribute(uefi::attribute::red);
-	out->output_string(message);
-
-	if (error) {
-		out->output_string(L": ");
-		out->output_string(error);
-	}
-
-	out->set_attribute(uefi::attribute::light_gray);
-	out->output_string(L"\r\n");
-}
-
-void
-status_line::finish()
-{
-	if (m_level <= level_ok)
-		print_status_tag();
-}
-
-/*
-uefi::status
-con_get_framebuffer(
-	EFI_BOOT_SERVICES *bootsvc,
-	void **buffer,
-	size_t *buffer_size,
-	uint32_t *hres,
-	uint32_t *vres,
-	uint32_t *rmask,
-	uint32_t *gmask,
-	uint32_t *bmask)
-{
-	uefi::status status;
-
-	uefi::protos::graphics_output *gop;
-	status = bootsvc->LocateProtocol(&guid_gfx_out, NULL, (void **)&gop);
-	if (status != EFI_NOT_FOUND) {
-		CHECK_EFI_STATUS_OR_RETURN(status, "LocateProtocol gfx");
-
-		*buffer = (void *)gop->Mode->FrameBufferBase;
-		*buffer_size = gop->Mode->FrameBufferSize;
-		*hres = gop->Mode->Info->horizontal_resolution;
-		*vres = gop->Mode->Info->vertical_resolution;
-
-		switch (gop->Mode->Info->PixelFormat) {
-			case PixelRedGreenBlueReserved8BitPerColor:
-				*rmask = 0x0000ff;
-				*gmask = 0x00ff00;
-				*bmask = 0xff0000;
-				return EFI_SUCCESS;
-
-			case PixelBlueGreenRedReserved8BitPerColor:
-				*bmask = 0x0000ff;
-				*gmask = 0x00ff00;
-				*rmask = 0xff0000;
-				return EFI_SUCCESS;
-
-			case PixelBitMask:
-				*rmask = gop->Mode->Info->PixelInformation.RedMask;
-				*gmask = gop->Mode->Info->PixelInformation.GreenMask;
-				*bmask = gop->Mode->Info->PixelInformation.BlueMask;
-				return EFI_SUCCESS;
-
-			default:
-				// Not a framebuffer, fall through to zeroing out
-				// values below.
-				break;
-		}
-	}
-
-	*buffer = NULL;
-	*buffer_size = *hres = *vres = 0;
-	*rmask = *gmask = *bmask = 0;
-	return EFI_SUCCESS;
-}
-*/
-
 } // namespace boot

src/boot/console.h

@@ -1,10 +1,15 @@
-/// \file console.h
-/// Text output and status message handling
 #pragma once
+/// \file console.h
+/// Text output handler
+
 #include <stdarg.h>
 #include <stddef.h>
-#include <uefi/boot_services.h>
+#include <stdint.h>
-#include <uefi/protos/simple_text_output.h>
+
+namespace uefi {
+namespace protos {
+    struct simple_text_output;
+}}
 
 namespace boot {
 
@@ -12,69 +17,30 @@ namespace boot {
 class console
 {
 public:
-	console(uefi::boot_services *bs, uefi::protos::simple_text_output *out);
+    console(uefi::protos::simple_text_output *out);
 
-	size_t print_hex(uint32_t n) const;
+    void announce();
-	size_t print_dec(uint32_t n) const;
-	size_t print_long_hex(uint64_t n) const;
-	size_t print_long_dec(uint64_t n) const;
-	size_t printf(const wchar_t *fmt, ...) const;
 
-	static console & get() { return *s_console; }
+    size_t print_hex(uint32_t n) const;
-	static size_t print(const wchar_t *fmt, ...);
+    size_t print_dec(uint32_t n) const;
+    size_t print_long_hex(uint64_t n) const;
+    size_t print_long_dec(uint64_t n) const;
+    size_t printf(const wchar_t *fmt, ...) const;
+
+    static console & get() { return *s_console; }
+    static size_t print(const wchar_t *fmt, ...);
 
 private:
-	friend class status_line;
+    friend class status_line;
 
-	void pick_mode(uefi::boot_services *bs);
+    size_t vprintf(const wchar_t *fmt, va_list args) const;
-	size_t vprintf(const wchar_t *fmt, va_list args) const;
 
-	size_t m_rows, m_cols;
+    size_t m_rows, m_cols;
-	uefi::protos::simple_text_output *m_out;
+    uefi::protos::simple_text_output *m_out;
 
-	static console *s_console;
+    static console *s_console;
 };
 
-/// Scoped status line reporter. Prints a message and an "OK" if no errors
+size_t wstrlen(const wchar_t *s);
-/// or warnings were reported before destruction, otherwise reports the
-/// error or warning.
-class status_line
-{
-public:
-	/// Constructor.
-	/// \arg message  Description of the operation in progress
-	/// \arg context  If non-null, printed after `message` and a colon
-	status_line(const wchar_t *message, const wchar_t *context = nullptr);
-	~status_line();
-
-	/// Set the state to warning, and print a message. If the state is already at
-	/// warning or error, the state is unchanged but the message is still printed.
-	/// \arg message  The warning message to print
-	/// \arg error    If non-null, printed after `message`
-	inline static void warn(const wchar_t *message, const wchar_t *error = nullptr) {
-		if (s_current) s_current->do_warn(message, error);
-	}
-
-	/// Set the state to error, and print a message. If the state is already at
-	/// error, the state is unchanged but the message is still printed.
-	/// \arg message  The error message to print
-	/// \arg error    If non-null, printed after `message`
-	inline static void fail(const wchar_t *message, const wchar_t *error = nullptr) {
-		if (s_current) s_current->do_fail(message, error);
-	}
-
-private:
-	void print_status_tag();
-	void do_warn(const wchar_t *message, const wchar_t *error);
-	void do_fail(const wchar_t *message, const wchar_t *error);
-	void finish();
-
-	size_t m_line;
-	int m_level;
-	int m_depth;
-
-	status_line *m_next;
-	static status_line *s_current;
-};
 
 } // namespace boot