Overhaul memory allocation model

This commit makes several fundamental changes to memory handling:

- the frame allocator is now only an allocator for free frames, and does
  not track used frames.
- the frame allocator now stores its free list inside the free frames
  themselves, as a hybrid stack/span model.
  - This has the implication that all frames must currently fit within
    the offset area.
- kutil has a new allocator interface, which is the only allowed way for
  any code outside of src/kernel to allocate. Code under src/kernel
  _may_ use new/delete, but should prefer the allocator interface.
- the heap manager has become heap_allocator, which is merely an
  implementation of kutil::allocator which doles out sections of a given
  address range.
- the heap manager now only writes block headers when necessary,
  avoiding page faults until they're actually needed
- page_manager now has a page fault handler, which checks with the
  address_manager to see if the address is known, and provides a frame
  mapping if it is, allowing heap manager to work with its entire
  address size from the start. (Currently 32GiB.)

.gitignore

@@ -2,7 +2,6 @@
 /build*
 *.bak
 tags
-.gdbinit
 popcorn.log
 *.o
 *.a

LICENSE.md

@@ -81,6 +81,33 @@ MIT license:
 > IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
 > CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
+## printf
+
+Popcorn uses 's tiny [printf](https://github.com/mpaland/printf) library for its
+`*printf()` functions, which is also released under the terms of the MIT license:
+
+> The MIT License (MIT)
+>
+> Copyright (c) 2014 Marco Paland
+>
+> Permission is hereby granted, free of charge, to any person obtaining a copy
+> of this software and associated documentation files (the "Software"), to deal
+> in the Software without restriction, including without limitation the rights
+> to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+> copies of the Software, and to permit persons to whom the Software is
+> furnished to do so, subject to the following conditions:
+>
+> The above copyright notice and this permission notice shall be included in all
+> copies or substantial portions of the Software.
+>
+> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+> IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+> FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+> AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+> LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+> OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+> SOFTWARE.
+
 ## GNU-EFI
 
 Popcorn's UEFI bootloader initially used

NOTES.md

@@ -2,7 +2,9 @@
 
 ## TODO
 
-- Better page-allocation model
+- Paging manager
+  - Copy-on-write pages
+  - Better page-allocation model?
 - Allow for more than one IOAPIC in ACPI module
   - The objects get created, but GSI lookup only uses the one at index 0
 - mark kernel memory pages global
@@ -19,12 +21,18 @@
   - Parse initrd and pre-load certain ELF images, eg the process loader process?
   - Do initial memory bootstrap?
 - Calling global ctors
-
-
 - Device Tree
   - Actual serial driver
   - Disk driver
     - File system
 - Multiprocessing
-  - Syscalls
+  - Fast syscalls using syscall/sysret
+
+### Build
+
+- Clean up build generator and its templates
+  - More robust objects to represent modules & targets to pass to templates
+  - Read project setup from a simple JSON/TOML/etc
+  - Better compartmentalizing when doing template inheritance
+- Move to LLD as sysroot linker
 

README.md

@@ -27,11 +27,21 @@ The design goals of the project are:
 
 ## Building
 
-Popcorn uses the `ninja` build tool, and generates the build files for it with
-the `generate_build.py` script.  The other requirements are:
+Popcorn uses the [Ninja][] build tool, and generates the build files for it
+with a custom tool called [Bonnibel][]. Bonnibel requires [Python 3][] and can
+be downloaded with `pip`:
 
-* python 3 for generating the build config
-  * The Jinja2 package is also required
+```
+pip3 install bonnibel
+```
+
+[Ninja]:    https://ninja-build.org
+[Bonnibel]: https://github.com/justinian/bonnibel
+[Python 3]: https://python.org
+
+Requrirements:
+
+* python 3 (for installing and running Bonnibel)
 * clang
 * mtools
 * ninja
@@ -45,7 +55,7 @@ Popcorn host binaries.
 
 ### Building and running Popcorn
 
-Once the toolchain has been set up, running `generate_build.py` will set up the
+Once the toolchain has been set up, running Bonnibel's `pb` command will set up the
 build configuration, and `ninja -C build` will actually run the build.  If you
 have `qemu-system-x86_64` installed, the `qemu.sh` script will to run Popcorn
 in QEMU `-nographic` mode.
@@ -54,7 +64,8 @@ I personally run this either from a real debian amd64 testing/buster machine or
 a windows WSL debian testing/buster installation. The following should be
 enough to set up such a system to build the kernel:
 
-    sudo apt install qemu-system-x86 nasm clang-6.0 mtools
-	sudo update-alternatives /usr/bin/clang clang /usr/bin/clang-6.0 1000
-	sudo update-alternatives /usr/bin/clang++ clang++ /usr/bin/clang++-6.0 1000
+    sudo apt install qemu-system-x86 nasm clang-6.0 mtools python3-pip curl
+    sudo update-alternatives /usr/bin/clang clang /usr/bin/clang-6.0 1000
+    sudo update-alternatives /usr/bin/clang++ clang++ /usr/bin/clang++-6.0 1000
+    sudo pip3 install bonnibel
 

assets/debugging/popcorn.elf-gdb.py

@@ -0,0 +1,31 @@
+import gdb
+
+class PrintStackCommand(gdb.Command):
+    def __init__(self):
+        super().__init__("popc_stack", gdb.COMMAND_DATA)
+
+    def invoke(self, arg, from_tty):
+        args = gdb.string_to_argv(arg)
+
+        base = "$rsp"
+        if len(args) > 0:
+            base = args[0]
+
+        depth = 22
+        if len(args) > 1:
+            depth = int(args[1])
+
+        for i in range(depth-1, -1, -1):
+            offset = i * 8
+            base_addr = gdb.parse_and_eval(base)
+            value = gdb.parse_and_eval(f"*(uint64_t*)({base} + {offset})")
+            print("{:016x} (+{:04x}): {:016x}".format(int(base_addr) + offset, offset, int(value)))
+
+
+PrintStackCommand()
+
+import time
+time.sleep(3.5)
+gdb.execute("target remote :1234")
+gdb.execute("set waiting = false")
+gdb.execute("display/i $rip")

assets/initrd.toml

@@ -14,10 +14,10 @@ source = "../assets/fonts/tamsyn8x16r.psf"
 
 [[files]]
 dest = "nulldrv1"
-source = "host/nulldrv"
+source = "user/nulldrv"
 executable = true
 
 [[files]]
 dest = "nulldrv2"
-source = "host/nulldrv"
+source = "user/nulldrv"
 executable = true

generate_build.py

@@ -1,165 +0,0 @@
-#!/usr/bin/env python3
-
-from collections import namedtuple
-
-library = namedtuple('library', ['path', 'deps'])
-program = namedtuple('program', ['path', 'deps', 'output', 'targets'])
-source = namedtuple('source', ['name', 'input', 'output', 'action'])
-version = namedtuple('version', ['major', 'minor', 'patch', 'sha', 'dirty'])
-
-MODULES = {
-    "elf":       library('src/libraries/elf', ['kutil']),
-    "initrd":    library('src/libraries/initrd', ["kutil"]),
-    "kutil":     library('src/libraries/kutil', []),
-
-    "makerd":    program('src/tools/makerd', ["initrd", "kutil"], "makerd", ["native"]),
-
-    "boot":      program('src/boot', [], "boot.elf", ["boot"]),
-
-    "nulldrv":   program('src/drivers/nulldrv', [], "nulldrv", ["host"]),
-    "kernel":    program('src/kernel', ["elf", "initrd", "kutil"], "popcorn.elf", ["host"]),
-}
-
-
-def get_template(env, typename, name):
-    from jinja2.exceptions import TemplateNotFound
-    try:
-        return env.get_template("{}.{}.ninja.j2".format(typename, name))
-    except TemplateNotFound:
-        return env.get_template("{}.default.ninja.j2".format(typename))
-
-
-def get_sources(path, srcroot):
-    import os
-    from os.path import abspath, join, relpath, splitext
-
-    actions = {'.c': 'cc', '.cpp': 'cxx', '.s': 'nasm'}
-
-    sources = []
-    for root, dirs, files in os.walk(path):
-        for f in files:
-            base, ext = splitext(f)
-            if not ext in actions: continue
-            name = join(root, f)
-            sources.append(
-                source(
-                    relpath(name, srcroot),
-                    abspath(name),
-                    relpath(abspath(name), path) + ".o",
-                    actions[ext]))
-
-    return sources
-
-
-def get_git_version():
-    from subprocess import run
-    cp = run(['git', 'describe', '--dirty', '--abbrev=7'],
-            check=True, capture_output=True)
-    full_version = cp.stdout.decode('utf-8').strip()
-
-    dirty = False
-    parts1 = full_version.split('-')
-    if parts1[-1] == "dirty":
-        dirty = True
-        parts1 = parts1[:-1]
-
-    parts2 = parts1[0].split('.')
-
-    return version(
-        parts2[0],
-        parts2[1],
-        parts2[2],
-        parts1[-1][1:],
-        dirty)
-
-
-def main(buildroot):
-    import os
-    from os.path import abspath, dirname, isdir, join
-
-    generator = abspath(__file__)
-    srcroot = dirname(generator)
-
-    if buildroot is None:
-        buildroot = join(srcroot, "build")
-
-    if not isdir(buildroot):
-        os.mkdir(buildroot)
-
-    git_version = get_git_version()
-    print("Generating build files for Popcorn {}.{}.{}-{}...".format(
-        git_version.major, git_version.minor, git_version.patch, git_version.sha))
-
-    from jinja2 import Environment, FileSystemLoader
-    env = Environment(loader=FileSystemLoader(join(srcroot, "scripts", "templates")))
-
-    targets = {}
-    templates = set()
-    buildfiles = []
-    for name, mod in MODULES.items():
-        if isinstance(mod, program):
-            for target in mod.targets:
-                if not target in targets:
-                    targets[target] = set()
-
-                open_list = [name]
-                while open_list:
-                    depname = open_list.pop()
-                    dep = MODULES[depname]
-                    open_list.extend(dep.deps)
-                    targets[target].add(depname)
-
-        sources = get_sources(join(srcroot, mod.path), join(srcroot, "src"))
-        buildfile = join(buildroot, name + ".ninja")
-        buildfiles.append(buildfile)
-        with open(buildfile, 'w') as out:
-            #print("Generating module", name)
-            template = get_template(env, type(mod).__name__, name)
-            templates.add(template.filename)
-            out.write(template.render(
-                name=name,
-                module=mod,
-                sources=sources,
-                buildfile=buildfile,
-                version=git_version))
-
-    for target, mods in targets.items():
-        root = join(buildroot, target)
-        if not isdir(root):
-            os.mkdir(root)
-
-        buildfile = join(root, "target.ninja")
-        buildfiles.append(buildfile)
-        with open(buildfile, 'w') as out:
-            #print("Generating target", target)
-            template = get_template(env, "target", target)
-            templates.add(template.filename)
-            out.write(template.render(
-                target=target,
-                modules=mods,
-                buildfile=buildfile,
-                version=git_version))
-
-    buildfile = join(buildroot, "build.ninja")
-    buildfiles.append(buildfile)
-    with open(buildfile, 'w') as out:
-        #print("Generating main build.ninja")
-        template = env.get_template('build.ninja.j2')
-        templates.add(template.filename)
-
-        out.write(template.render(
-            targets=targets,
-            buildroot=buildroot,
-            srcroot=srcroot,
-            buildfile=buildfile,
-            buildfiles=buildfiles,
-            templates=[abspath(f) for f in templates],
-            generator=generator,
-            version=git_version))
-
-if __name__ == "__main__":
-    import sys
-    buildroot = None
-    if len(sys.argv) > 1:
-        buildroot = sys.argv[1]
-    main(buildroot)

modules.yaml

@@ -0,0 +1,132 @@
+name: Popcorn
+templates: scripts/templates
+modules:
+    kernel:
+        output: popcorn.elf
+        target: host
+        deps:
+            - elf
+            - initrd
+            - kutil
+        includes:
+            - src/kernel
+        source:
+            - src/kernel/crti.s
+            - src/kernel/apic.cpp
+            - src/kernel/assert.cpp
+            - src/kernel/boot.s
+            - src/kernel/console.cpp
+            - src/kernel/cpprt.cpp
+            - src/kernel/cpu.cpp
+            - src/kernel/debug.cpp
+            - src/kernel/debug.s
+            - src/kernel/device_manager.cpp
+            - src/kernel/font.cpp
+            - src/kernel/frame_allocator.cpp
+            - src/kernel/fs/gpt.cpp
+            - src/kernel/gdt.cpp
+            - src/kernel/gdt.s
+            - src/kernel/interrupts.cpp
+            - src/kernel/interrupts.s
+            - src/kernel/io.cpp
+            - src/kernel/loader.s
+            - src/kernel/log.cpp
+            - src/kernel/main.cpp
+            - src/kernel/memory_bootstrap.cpp
+            - src/kernel/msr.cpp
+            - src/kernel/page_manager.cpp
+            - src/kernel/pci.cpp
+            - src/kernel/process.cpp
+            - src/kernel/scheduler.cpp
+            - src/kernel/screen.cpp
+            - src/kernel/serial.cpp
+            - src/kernel/syscall.cpp
+            - src/kernel/syscall.s
+            - src/kernel/syscalls/exit.cpp
+            - src/kernel/syscalls/fork.cpp
+            - src/kernel/syscalls/getpid.cpp
+            - src/kernel/syscalls/message.cpp
+            - src/kernel/syscalls/noop.cpp
+            - src/kernel/syscalls/pause.cpp
+            - src/kernel/syscalls/sleep.cpp
+            - src/kernel/task.s
+            - src/kernel/crtn.s
+
+    boot:
+        kind: exe
+        target: boot
+        output: boot.elf
+        source:
+            - src/boot/crt0.s
+            - src/boot/console.cpp
+            - src/boot/guids.cpp
+            - src/boot/loader.cpp
+            - src/boot/main.cpp
+            - src/boot/memory.cpp
+            - src/boot/reloc.cpp
+            - src/boot/utility.cpp
+
+    nulldrv:
+        kind: exe
+        target: user
+        output: nulldrv
+        source:
+            - src/drivers/nulldrv/main.cpp
+            - src/drivers/nulldrv/main.s
+
+    elf:
+        kind: lib
+        output: libelf.a
+        deps:
+            - kutil
+        includes:
+            - src/libraries/elf/include
+        source:
+            - src/libraries/elf/elf.cpp
+
+    initrd:
+        kind: lib
+        output: libinitrd.a
+        deps:
+            - kutil
+        includes:
+            - src/libraries/initrd/include
+        source:
+            - src/libraries/initrd/initrd.cpp
+
+    kutil:
+        kind: lib
+        output: libkutil.a
+        includes:
+            - src/libraries/kutil/include
+        source:
+            - src/libraries/kutil/assert.cpp
+            - src/libraries/kutil/bip_buffer.cpp
+            - src/libraries/kutil/heap_allocator.cpp
+            - src/libraries/kutil/logger.cpp
+            - src/libraries/kutil/memory.cpp
+            - src/libraries/kutil/printf.c
+
+    makerd:
+        kind: exe
+        target: native
+        output: makerd
+        deps:
+            - initrd
+        source:
+            - src/tools/makerd/entry.cpp
+            - src/tools/makerd/main.cpp
+
+    tests:
+        kind: exe
+        target: native
+        output: tests
+        deps:
+            - kutil
+        source:
+            - src/tests/address_manager.cpp
+            - src/tests/constexpr_hash.cpp
+            - src/tests/linked_list.cpp
+            - src/tests/logger.cpp
+            - src/tests/heap_allocator.cpp
+            - src/tests/main.cpp

qemu.sh

@@ -1,22 +1,56 @@
 #!/usr/bin/env bash
 
-build=${1:-"$(dirname $0)/build"}
-ninja -C $build
-
+build="$(dirname $0)/build"
+assets="$(dirname $0)/assets"
+debug=""
+flash_name="ovmf_vars"
+gfx="-nographic"
 kvm=""
-if [[ -f /dev/kvm ]]; then
-	kvm="--enable-kvm"
+
+for arg in $@; do
+	case "${arg}" in
+		--debug)
+			debug="-s"
+			flash_name="ovmf_vars_d"
+			;;
+		--gfx)
+			gfx=""
+			;;
+		--kvm)
+			kvm="-enable-kvm"
+			;;
+		*)
+			build="${arg}"
+			;;
+	esac
+done
+
+if [[ ! -c /dev/kvm ]]; then
+	kvm=""
+fi
+
+if ! ninja -C "${build}"; then
+	exit 1
+fi
+
+if [[ -n $TMUX ]]; then
+	if [[ -n $debug ]]; then
+		tmux split-window "gdb ${build}/popcorn.elf" &
+	else
+		tmux split-window -l 10 "sleep 1; telnet localhost 45454" &
+	fi
 fi
 
 exec qemu-system-x86_64 \
-	-drive "if=pflash,format=raw,file=${build}/flash.img" \
+	-drive "if=pflash,format=raw,readonly,file=${assets}/ovmf/x64/ovmf_code.fd" \
+	-drive "if=pflash,format=raw,file=${build}/${flash_name}.fd" \
 	-drive "format=raw,file=${build}/popcorn.img" \
-	-smp 1 \
+	-monitor telnet:localhost:45454,server,nowait \
+	-smp 4 \
 	-m 512 \
 	-d mmu,int,guest_errors \
 	-D popcorn.log \
 	-cpu Broadwell \
 	-M q35 \
 	-no-reboot \
-	-nographic \
-	$kvm
+	$gfx $kvm $debug

scripts/build_sysroot_clang.sh

@@ -3,6 +3,7 @@
 TARGET="x86_64-elf"
 NASM_VERSION="2.13.03"
 BINUTILS_VERSION="2.31.1"
+LLVM_BRANCH="release_80"
 
 TOOLS="clang" # lld libunwind libcxxabi libcxx"
 PROJECTS="compiler-rt libcxxabi libcxx libunwind"
@@ -77,7 +78,7 @@ function build_llvm() {
 	if [[ ! -d "${WORK}/llvm" ]]; then
 		echo "Downloading LLVM..."
 		git clone -q \
-			--branch release_70 \
+			--branch "${LLVM_BRANCH}" \
 			--depth 1 \
 			"https://git.llvm.org/git/llvm.git" "${WORK}/llvm"
 	fi
@@ -86,7 +87,7 @@ function build_llvm() {
 		if [[ ! -d "${WORK}/llvm/tools/${tool}" ]]; then
 			echo "Downloading ${tool}..."
 			git clone -q \
-				--branch release_70 \
+				--branch "${LLVM_BRANCH}" \
 				--depth 1 \
 				"https://git.llvm.org/git/${tool}.git" "${WORK}/llvm/tools/${tool}"
 		fi
@@ -95,7 +96,7 @@ function build_llvm() {
 	if [[ ! -d "${WORK}/llvm/tools/clang/tools/extra" ]]; then
 		echo "Downloading clang-tools-extra..."
 		git clone -q \
-			--branch release_70 \
+			--branch "${LLVM_BRANCH}" \
 			--depth 1 \
 			"https://git.llvm.org/git/clang-tools-extra.git" "${WORK}/llvm/tools/clang/tools/extra"
 	fi
@@ -104,7 +105,7 @@ function build_llvm() {
 		if [[ ! -d "${WORK}/llvm/projects/${proj}" ]]; then
 			echo "Downloading ${proj}..."
 			git clone -q \
-				--branch release_70 \
+				--branch "${LLVM_BRANCH}" \
 				--depth 1 \
 				"https://git.llvm.org/git/${proj}.git" "${WORK}/llvm/projects/${proj}"
 		fi
@@ -114,7 +115,7 @@ function build_llvm() {
 		if [[ ! -d "${WORK}/llvm/runtimes/${proj}" ]]; then
 			echo "Downloading ${proj}..."
 			git clone -q \
-				--branch release_70 \
+				--branch "${LLVM_BRANCH}" \
 				--depth 1 \
 				"https://git.llvm.org/git/${proj}.git" "${WORK}/llvm/runtime/${proj}"
 		fi
@@ -216,5 +217,5 @@ update_links
 
 export CC="${SYSROOT}/bin/clang"
 export CXX="${SYSROOT}/bin/clang++"
-export LD="${SYSROOT}/bin/ld.lld"
+export LD="${SYSROOT}/bin/ld"
 build_libc

scripts/templates/build.ninja.j2

@@ -1,86 +1,14 @@
-ninja_required_version = 1.3
-builddir = {{ buildroot }}
-srcroot = {{ srcroot }}
+{% extends "build.base.j2" %}
 
-warnflags = $
-    -Wformat=2 $
-    -Winit-self $
-    -Wfloat-equal $
-    -Winline $
-    -Wmissing-format-attribute $
-    -Wmissing-include-dirs $
-    -Wswitch $
-    -Wundef $
-    -Wdisabled-optimization $
-    -Wpointer-arith $
-    -Wno-attributes $
-    -Wno-sign-compare $
-    -Wno-multichar $
-    -Wno-div-by-zero $
-    -Wno-endif-labels $
-    -Wno-pragmas $
-    -Wno-format-extra-args $
-    -Wno-unused-result $
-    -Wno-deprecated-declarations $
-    -Wno-unused-function $
-    -Werror
-
-ccflags = $
+{% block variables %}
+{{ super() }}
+ccflags = $ccflags $
     -I${srcroot}/src/include $
-    -I${srcroot}/src/include/x86_64 $
-    -DVERSION_MAJOR={{ version.major }} $
-    -DVERSION_MINOR={{ version.minor }} $
-    -DVERSION_PATCH={{ version.patch }} $
-    -DVERSION_GITSHA=0x{% if version.dirty %}1{% else %}0{% endif %}{{ version.sha }} $
-    -DGIT_VERSION=\"{{ version.major }}.{{ version.minor }}.{{ version.patch }}-{{ version.sha }}\" $
-    -DGIT_VERSION_WIDE=L\"{{ version.major }}.{{ version.minor }}.{{ version.patch }}-{{ version.sha }}\" $
-    $warnflags
-
-asflags = $
-    -DVERSION_MAJOR={{ version.major }} $
-    -DVERSION_MINOR={{ version.minor }} $
-    -DVERSION_PATCH={{ version.patch }} $
-    -DVERSION_GITSHA=0x{% if version.dirty %}1{% else %}0{% endif %}{{ version.sha }}
-
-cflags = -std=c11
-cxxflags = -std=c++14
-libs =
-
-rule cc
-    deps = gcc
-    depfile = $out.d
-    description = Compiling $name
-    command = $cc -MMD -MF $out.d $ccflags $cflags -o $out -c $in
-
-rule cxx
-    deps = gcc
-    depfile = $out.d
-    description = Compiling $name
-    command = $cxx -MMD -MF $out.d $cxxflags $ccflags -o $out -c $in
-
-rule nasm
-    deps = gcc
-    depfile = $out.d
-    description = Assembling $name
-    command = $nasm -o $out -felf64 -MD $out.d $asflags $in
-
-rule exe
-    description = Linking $name
-    command = $ld $ldflags -o $out $in $libs
-
-rule lib
-    description = Archiving $name
-    command = $ar qcs $out $in
-
-rule regen
-    generator = true
-    description = Regenrating build files
-    command = {{ generator }} $builddir
-
-rule cp
-    description = Copying $name
-    command = cp $in $out
+    -I${srcroot}/src/include/x86_64 
+{% endblock %}
 
+{% block baserules %}
+{{ super() }}
 rule makerd
     description = Making init ramdisk
     command = $builddir/native/makerd $in $out
@@ -103,26 +31,34 @@ rule makefat
     description = Creating $name
     command = $
         cp $srcroot/assets/diskbase.img $out; $
-        mcopy -s -D o -i $out@@1024K $builddir/fatroot/* ::/
+        mcopy -s -D o -i $out@@1M $builddir/fatroot/* ::/
 
-{% for target in targets %}
-subninja {{ target }}/target.ninja
-{% endfor %}
+rule strip
+    description = Stripping $name
+    command = $
+        cp $in $out; $
+        objcopy --only-keep-debug $out $out.debug; $
+        strip -g $out; $
+        objcopy --add-gnu-debuglink=$out.debug $out
+{% endblock %}
 
-build $
-    {%- for buildfile in buildfiles %}
-    {{ buildfile }} $
-    {%- endfor %}
-    : regen | $
-    {%- for template in templates %}
-    {{ template }} $
-    {%- endfor %}
-    {{ generator }}
+{% block extra %}
+build $builddir/ovmf_vars.fd : cp $srcroot/assets/ovmf/x64/ovmf_vars.fd
+    name = ovmf_vars.fd
 
-build $builddir/flash.img : cp $srcroot/assets/ovmf/x64/OVMF.fd
-    name = flash.img
+build $builddir/ovmf_vars_d.fd : cp $srcroot/assets/ovmf/x64/ovmf_vars_d.fd
+    name = ovmf_vars_d.fd
 
-build $builddir/fatroot/popcorn.elf : cp $builddir/host/popcorn.elf
+build $builddir/popcorn.elf | $builddir/popcorn.elf.debug : strip $builddir/host/popcorn.elf
+    name = kernel
+
+build $builddir/popcorn.dump : dump $builddir/host/popcorn.elf
+    name = kernel
+
+build $builddir/popcorn.elf-gdb.py : cp ${srcroot}/assets/debugging/popcorn.elf-gdb.py
+    name = kernel debug python scripts
+
+build $builddir/fatroot/popcorn.elf : cp $builddir/popcorn.elf
     name = kernel to FAT image
 
 build $builddir/fatroot/efi/boot/bootx64.efi : cp $builddir/boot/boot.efi
@@ -130,7 +66,7 @@ build $builddir/fatroot/efi/boot/bootx64.efi : cp $builddir/boot/boot.efi
 
 build $builddir/fatroot/initrd.img : makerd ${srcroot}/assets/initrd.toml | $
     ${builddir}/native/makerd $
-    ${builddir}/host/nulldrv
+    ${builddir}/user/nulldrv
 
 build $builddir/popcorn.img : makefat | $
     $builddir/fatroot/initrd.img $
@@ -138,5 +74,13 @@ build $builddir/popcorn.img : makefat | $
     $builddir/fatroot/efi/boot/bootx64.efi
     name = popcorn.img
 
-# vim: et ts=4 sts=4 sw=4
+default $
+    $builddir/ovmf_vars.fd $
+    $builddir/ovmf_vars_d.fd $
+    $builddir/popcorn.dump $
+    $builddir/popcorn.elf-gdb.py $
+    $builddir/popcorn.img
+{% endblock %}
+
+# vim: ft=ninja et ts=4 sts=4 sw=4
 

scripts/templates/exe.boot.j2

@@ -1,19 +1,14 @@
-{% extends "program.default.ninja.j2" %}
+{% extends "exe.default.j2" %}
 {% block variables %}
 {{ super() }}
 
-ld = ld
-cc = clang
-cxx = clang++
-
 ccflags = $ccflags $
     -DKERNEL_FILENAME=L\"popcorn.elf\" $
     -DGNU_EFI_USE_MS_ABI $
     -DHAVE_USE_MS_ABI $
     -DEFI_DEBUG=0 $
     -DEFI_DEBUG_CLEAR_MEMORY=0 $
-    -fPIC $
-    -fshort-wchar
+    -DBOOTLOADER_DEBUG
 
 ldflags = $ldflags $
     -T ${srcroot}/src/arch/x86_64/boot.ld $
@@ -27,3 +22,6 @@ build $builddir/boot.efi : makeefi ${builddir}/{{ module.output }}
     name = boot.efi
 
 {% endblock %}
+
+# vim: ft=ninja et ts=4 sts=4 sw=4
+

scripts/templates/exe.kernel.j2

@@ -1,4 +1,4 @@
-{% extends "program.default.ninja.j2" %}
+{% extends "exe.default.j2" %}
 {% block variables %}
 {{ super() }}
 
@@ -7,3 +7,6 @@ libs = $libs
 ldflags = $ldflags -T ${srcroot}/src/arch/x86_64/kernel.ld
 
 {% endblock %}
+
+# vim: ft=ninja et ts=4 sts=4 sw=4
+

scripts/templates/exe.tests.j2

@@ -0,0 +1,10 @@
+{% extends "exe.default.j2" %}
+{% block variables %}
+{{ super() }}
+
+ccflags = $ccflags -ggdb
+
+{% endblock %}
+
+# vim: ft=ninja et ts=4 sts=4 sw=4
+

scripts/templates/library.default.ninja.j2

@@ -1 +0,0 @@
-{% extends "module.base.ninja.j2" %}

scripts/templates/module.base.ninja.j2

@@ -1,29 +0,0 @@
-moddir = ${builddir}/{{ name }}.dir
-
-{% block variables %}
-ccflags = $ccflags $
-    {%- for dep in module.deps %}
-    -I${srcroot}/src/libraries/{{ dep }}/include $
-    {%- endfor %}
-    -I${srcroot}/{{ module.path }} $
-    -I${srcroot}/{{ module.path }}/include
-{% endblock %}
-
-{% for source in sources %}
-build ${moddir}/{{ source.output }} : {{ source.action }} {{ source.input }} || {{ buildfile }}
-    name = {{ source.name }}
-{% endfor %}
-
-build {% block artifact %} ${builddir}/lib{{ name }}.a : lib {% endblock %} $
-{%- block extrasources %}{% endblock -%}
-{%- for source in sources %}
-    ${moddir}/{{ source.output }}{% if not loop.last %} ${% endif %}
-{%- endfor -%}
-{%- if module.deps %}| {% for dep in module.deps %} ${builddir}/lib{{ dep }}.a {% endfor %}{% endif %}
-    name = {{ name }}
-
-{% block extra %}
-{% endblock %}
-
-# vim: ft=ninja et ts=4 sts=4 sw=4
-

scripts/templates/program.default.ninja.j2

@@ -1,13 +0,0 @@
-{% extends "module.base.ninja.j2" %}
-{% block variables %}
-{{ super() }}
-
-libs = $
-    -L${builddir} $
-    {%- for dep in module.deps %}
-    -l{{dep}} $
-    {%- endfor %}
-    $libs
-
-{% endblock %}
-{% block artifact %}${builddir}/{{ module.output }} : exe{% endblock %}

scripts/templates/target.boot.j2

@@ -0,0 +1,41 @@
+{% extends "target.default.j2" %}
+
+{% block binaries %}
+cc = clang
+cxx = clang++
+ld = ld
+ar = ar
+nasm = nasm
+objcopy = objcopy
+{% endblock %}
+
+{% block variables %}
+
+ccflags = $ccflags $
+	-ggdb $
+	-nostdlib $
+	-ffreestanding $
+	-nodefaultlibs $
+	-fno-builtin $
+	-mno-sse $
+	-fno-omit-frame-pointer $
+	-mno-red-zone $
+	-fshort-wchar $
+	-D__ELF__ $
+	-fPIC
+
+cxxflags = $cxxflags $
+	-fno-exceptions $
+	-fno-rtti $
+
+ldflags = $ldflags $
+	-g $
+	-nostdlib $
+	-znocombreloc $
+	-Bsymbolic $
+	-nostartfiles
+
+{% endblock %}
+
+# vim: ft=ninja et ts=4 sts=4 sw=4
+

scripts/templates/target.boot.ninja.j2

@@ -1,35 +0,0 @@
-{% extends "target.default.ninja.j2" %}
-
-{% block binaries %}
-ld = ld
-cc = clang
-cxx = clang++
-nasm = nasm
-{% endblock %}
-
-{% block variables %}
-
-ccflags = $ccflags $
-    -ggdb $
-    -nostdlib $
-    -ffreestanding $
-    -nodefaultlibs $
-    -fno-builtin $
-    -mno-sse $
-    -fno-omit-frame-pointer $
-    -mno-red-zone
-
-cxxflags = $cxxflags $
-    -nostdlibinc $
-    -isystem${srcroot}/sysroot/include/c++/v1 $
-    -fno-exceptions $
-    -fno-rtti
-
-ldflags = $ldflags $
-    -g $
-    -nostdlib $
-    -znocombreloc $
-	-Bsymbolic $
-    -nostartfiles
-
-{% endblock %}

scripts/templates/target.default.ninja.j2

@@ -1,17 +0,0 @@
-builddir = $builddir/{{ target }}
-
-{% block variables %}
-{% endblock %}
-
-{% block binaries %}
-cc = clang
-cxx = clang++
-ld = clang++
-ar = ar
-nasm = nasm
-objcopy = objcopy
-{% endblock %}
-
-{% for module in modules %}
-subninja {{ module }}.ninja
-{% endfor %}

scripts/templates/target.host.j2

@@ -1,4 +1,5 @@
-{% extends "target.default.ninja.j2" %}
+{% extends "target.default.j2" %}
+
 {% block binaries %}
 cc = ${srcroot}/sysroot/bin/clang
 cxx = ${srcroot}/sysroot/bin/clang++
@@ -39,4 +40,6 @@ ldflags = $ldflags $
     -Bstatic
 
 {% endblock %}
-# vim: et ts=4 sts=4 sw=4
+
+# vim: ft=ninja et ts=4 sts=4 sw=4
+

scripts/templates/target.native.j2

@@ -0,0 +1,15 @@
+{% extends "target.default.j2" %}
+
+{% block binaries %}
+{{ super() }}
+ld = clang++
+{% endblock %}
+
+{% block variables %}
+
+ccflags = $ccflags -g -ggdb
+
+{% endblock %}
+
+# vim: ft=ninja et ts=4 sts=4 sw=4
+

scripts/templates/target.native.ninja.j2

@@ -1 +0,0 @@
-{% extends "target.default.ninja.j2" %}

scripts/templates/target.user.j2

@@ -0,0 +1,49 @@
+{% extends "target.default.j2" %}
+
+{% block binaries %}
+cc = ${srcroot}/sysroot/bin/clang
+cxx = ${srcroot}/sysroot/bin/clang++
+ld = ${srcroot}/sysroot/bin/x86_64-elf-ld
+ar = ${srcroot}/sysroot/bin/x86_64-elf-ar
+nasm = ${srcroot}/sysroot/bin/nasm
+objcopy = ${srcroot}/sysroot/bin/x86_64-elf-objcopy
+{% endblock %}
+
+{% block variables %}
+
+ccflags = $ccflags $
+    -nostdlib $
+    -nodefaultlibs $
+    -fno-builtin $
+    -mno-sse $
+    -fno-omit-frame-pointer $
+    -mno-red-zone $
+    -g $
+    -mcmodel=large $
+    -D__ELF__ $
+    -D__POPCORN__ $
+    -isystem${srcroot}/sysroot/include $
+    --sysroot="${srcroot}/sysroot"
+
+cxxflags = $cxxflags $
+    -fno-exceptions $
+    -fno-rtti $
+    -isystem${srcroot}/sysroot/include/c++/v1
+
+ldflags = $ldflags $
+    -g $
+    -nostdlib $
+    -znocombreloc $
+    -Bsymbolic $
+    -nostartfiles $
+    -Bstatic $
+    --sysroot="${srcroot}/sysroot" $
+	-L "${srcroot}/sysroot/lib" $
+
+libs = $libs $
+	-lc
+
+{% endblock %}
+
+# vim: ft=ninja et ts=4 sts=4 sw=4
+

src/boot/console.cpp

@@ -1,4 +1,3 @@
-#include <stdarg.h>
 #include <stddef.h>
 #include <stdint.h>
 
@@ -9,16 +8,55 @@
 size_t ROWS = 0;
 size_t COLS = 0;
 
-static EFI_SIMPLE_TEXT_OUT_PROTOCOL *con_out = 0;
+static EFI_SIMPLE_TEXT_OUT_PROTOCOL *m_out = 0;
 
-const CHAR16 digits[] = {u'0', u'1', u'2', u'3', u'4', u'5', u'6', u'7', u'8', u'9', u'a', u'b', u'c', u'd', u'e', u'f'};
+static const wchar_t digits[] = {u'0', u'1', u'2', u'3', u'4', u'5',
+	u'6', u'7', u'8', u'9', u'a', u'b', u'c', u'd', u'e', u'f'};
+
+console::console(EFI_SYSTEM_TABLE *system_table) :
+	m_rows(0),
+	m_cols(0),
+	m_out(nullptr)
+{
+	s_console = this;
+	m_boot = system_table->BootServices;
+	m_out = system_table->ConOut;
+}
 
 EFI_STATUS
-con_pick_mode(EFI_BOOT_SERVICES *bootsvc)
+console::initialize(const wchar_t *version)
+{
+	EFI_STATUS status;
+
+	// Might not find a video device at all, so ignore not found errors
+	status = pick_mode();
+	if (status != EFI_NOT_FOUND)
+		CHECK_EFI_STATUS_OR_FAIL(status);
+
+	status = m_out->QueryMode(m_out, m_out->Mode->Mode, &m_cols, &m_rows);
+	CHECK_EFI_STATUS_OR_RETURN(status, "QueryMode");
+
+	status = m_out->ClearScreen(m_out);
+	CHECK_EFI_STATUS_OR_RETURN(status, "ClearScreen");
+
+	m_out->SetAttribute(m_out, EFI_LIGHTCYAN);
+	m_out->OutputString(m_out, (wchar_t *)L"Popcorn loader ");
+
+	m_out->SetAttribute(m_out, EFI_LIGHTMAGENTA);
+	m_out->OutputString(m_out, (wchar_t *)version);
+
+	m_out->SetAttribute(m_out, EFI_LIGHTGRAY);
+	m_out->OutputString(m_out, (wchar_t *)L" booting...\r\n\n");
+
+	return status;
+}
+
+EFI_STATUS
+console::pick_mode()
 {
 	EFI_STATUS status;
 	EFI_GRAPHICS_OUTPUT_PROTOCOL *gfx_out_proto;
-	status = bootsvc->LocateProtocol(&guid_gfx_out, NULL, (void **)&gfx_out_proto);
+	status = m_boot->LocateProtocol(&guid_gfx_out, NULL, (void **)&gfx_out_proto);
 	CHECK_EFI_STATUS_OR_RETURN(status, "LocateProtocol gfx");
 
 	const uint32_t modes = gfx_out_proto->Mode->MaxMode;
@@ -53,106 +91,72 @@ con_pick_mode(EFI_BOOT_SERVICES *bootsvc)
 	return EFI_SUCCESS;
 }
 
-EFI_STATUS
-con_initialize(EFI_SYSTEM_TABLE *system_table, const CHAR16 *version)
-{
-	EFI_STATUS status;
-
-	EFI_BOOT_SERVICES *bootsvc = system_table->BootServices;
-	con_out = system_table->ConOut;
-
-	// Might not find a video device at all, so ignore not found errors
-	status = con_pick_mode(bootsvc);
-	if (status != EFI_NOT_FOUND)
-		CHECK_EFI_STATUS_OR_RETURN(status, "con_pick_mode");
-
-	status = con_out->QueryMode(con_out, con_out->Mode->Mode, &COLS, &ROWS);
-	CHECK_EFI_STATUS_OR_RETURN(status, "QueryMode");
-
-	status = con_out->ClearScreen(con_out);
-	CHECK_EFI_STATUS_OR_RETURN(status, "ClearScreen");
-
-	con_out->SetAttribute(con_out, EFI_LIGHTCYAN);
-	con_out->OutputString(con_out, (CHAR16 *)L"Popcorn loader ");
-
-	con_out->SetAttribute(con_out, EFI_LIGHTMAGENTA);
-	con_out->OutputString(con_out, (CHAR16 *)version);
-
-	con_out->SetAttribute(con_out, EFI_LIGHTGRAY);
-	con_out->OutputString(con_out, (CHAR16 *)L" booting...\r\n\n");
-
-	return status;
-}
-
 size_t
-con_print_hex(uint32_t n)
+console::print_hex(uint32_t n) const
 {
-	CHAR16 buffer[9];
-	CHAR16 *p = buffer;
+	wchar_t buffer[9];
+	wchar_t *p = buffer;
 	for (int i = 7; i >= 0; --i) {
 		uint8_t nibble = (n & (0xf << (i*4))) >> (i*4);
 		*p++ = digits[nibble];
 	}
 	*p = 0;
-	con_out->OutputString(con_out, buffer);
+	m_out->OutputString(m_out, buffer);
 	return 8;
 }
 
 size_t
-con_print_long_hex(uint64_t n)
+console::print_long_hex(uint64_t n) const
 {
-	CHAR16 buffer[17];
-	CHAR16 *p = buffer;
+	wchar_t buffer[17];
+	wchar_t *p = buffer;
 	for (int i = 15; i >= 0; --i) {
 		uint8_t nibble = (n & (0xf << (i*4))) >> (i*4);
 		*p++ = digits[nibble];
 	}
 	*p = 0;
-	con_out->OutputString(con_out, buffer);
+	m_out->OutputString(m_out, buffer);
 	return 16;
 }
 
 size_t
-con_print_dec(uint32_t n)
+console::print_dec(uint32_t n) const
 {
-	CHAR16 buffer[11];
-	CHAR16 *p = buffer + 10;
+	wchar_t buffer[11];
+	wchar_t *p = buffer + 10;
 	*p-- = 0;
 	do {
 		*p-- = digits[n % 10];
 		n /= 10;
 	} while (n != 0);
 
-	con_out->OutputString(con_out, ++p);
+	m_out->OutputString(m_out, ++p);
 	return 10 - (p - buffer);
 }
 
 size_t
-con_print_long_dec(uint64_t n)
+console::print_long_dec(uint64_t n) const
 {
-	CHAR16 buffer[21];
-	CHAR16 *p = buffer + 20;
+	wchar_t buffer[21];
+	wchar_t *p = buffer + 20;
 	*p-- = 0;
 	do {
 		*p-- = digits[n % 10];
 		n /= 10;
 	} while (n != 0);
 
-	con_out->OutputString(con_out, ++p);
+	m_out->OutputString(m_out, ++p);
 	return 20 - (p - buffer);
 }
 
 size_t
-con_printf(const CHAR16 *fmt, ...)
+console::vprintf(const wchar_t *fmt, va_list args) const
 {
-	CHAR16 buffer[256];
-	const CHAR16 *r = fmt;
-	CHAR16 *w = buffer;
-	va_list args;
+	wchar_t buffer[256];
+	const wchar_t *r = fmt;
+	wchar_t *w = buffer;
 	size_t count = 0;
 
-	va_start(args, fmt);
-
 	while (r && *r) {
 		if (*r != L'%') {
 			count++;
@@ -161,43 +165,43 @@ con_printf(const CHAR16 *fmt, ...)
 		}
 
 		*w = 0;
-		con_out->OutputString(con_out, buffer);
+		m_out->OutputString(m_out, buffer);
 		w = buffer;
 
 		r++; // chomp the %
 
 		switch (*r++) {
 			case L'%':
-				con_out->OutputString(con_out, L"%");
+				m_out->OutputString(m_out, const_cast<wchar_t*>(L"%"));
 				count++;
 				break;
 
 			case L'x':
-				count += con_print_hex(va_arg(args, uint32_t));
+				count += print_hex(va_arg(args, uint32_t));
 				break;
 
 			case L'd':
 			case L'u':
-				count += con_print_dec(va_arg(args, uint32_t));
+				count += print_dec(va_arg(args, uint32_t));
 				break;
 
 			case L's':
 				{
-					CHAR16 *s = va_arg(args, CHAR16*);
+					wchar_t *s = va_arg(args, wchar_t*);
 					count += wstrlen(s);
-					con_out->OutputString(con_out, s);
+					m_out->OutputString(m_out, s);
 				}
 				break;
 
 			case L'l':
 				switch (*r++) {
 					case L'x':
-						count += con_print_long_hex(va_arg(args, uint64_t));
+						count += print_long_hex(va_arg(args, uint64_t));
 						break;
 
 					case L'd':
 					case L'u':
-						count += con_print_long_dec(va_arg(args, uint64_t));
+						count += print_long_dec(va_arg(args, uint64_t));
 						break;
 
 					default:
@@ -211,44 +215,66 @@ con_printf(const CHAR16 *fmt, ...)
 	}
 
 	*w = 0;
-	con_out->OutputString(con_out, buffer);
-
-	va_end(args);
+	m_out->OutputString(m_out, buffer);
 	return count;
 }
 
-void
-con_status_begin(const CHAR16 *message)
+size_t
+console::printf(const wchar_t *fmt, ...) const
 {
-	con_out->SetAttribute(con_out, EFI_LIGHTGRAY);
-	con_out->OutputString(con_out, (CHAR16 *)message);
+	va_list args;
+	va_start(args, fmt);
+
+	size_t result = vprintf(fmt, args);
+
+	va_end(args);
+	return result;
+}
+
+size_t
+console::print(const wchar_t *fmt, ...)
+{
+	va_list args;
+	va_start(args, fmt);
+
+	size_t result = get().vprintf(fmt, args);
+
+	va_end(args);
+	return result;
 }
 
 void
-con_status_ok()
+console::status_begin(const wchar_t *message) const
 {
-	con_out->SetAttribute(con_out, EFI_LIGHTGRAY);
-	con_out->OutputString(con_out, (CHAR16 *)L"[");
-	con_out->SetAttribute(con_out, EFI_GREEN);
-	con_out->OutputString(con_out, (CHAR16 *)L"  ok  ");
-	con_out->SetAttribute(con_out, EFI_LIGHTGRAY);
-	con_out->OutputString(con_out, (CHAR16 *)L"]\r\n");
+	m_out->SetAttribute(m_out, EFI_LIGHTGRAY);
+	m_out->OutputString(m_out, (wchar_t *)message);
 }
 
 void
-con_status_fail(const CHAR16 *error)
+console::status_ok() const
 {
-	con_out->SetAttribute(con_out, EFI_LIGHTGRAY);
-	con_out->OutputString(con_out, (CHAR16 *)L"[");
-	con_out->SetAttribute(con_out, EFI_LIGHTRED);
-	con_out->OutputString(con_out, (CHAR16 *)L"failed");
-	con_out->SetAttribute(con_out, EFI_LIGHTGRAY);
-	con_out->OutputString(con_out, (CHAR16 *)L"]\r\n");
+	m_out->SetAttribute(m_out, EFI_LIGHTGRAY);
+	m_out->OutputString(m_out, (wchar_t *)L"[");
+	m_out->SetAttribute(m_out, EFI_GREEN);
+	m_out->OutputString(m_out, (wchar_t *)L"  ok  ");
+	m_out->SetAttribute(m_out, EFI_LIGHTGRAY);
+	m_out->OutputString(m_out, (wchar_t *)L"]\r\n");
+}
 
-	con_out->SetAttribute(con_out, EFI_RED);
-	con_out->OutputString(con_out, (CHAR16 *)error);
-	con_out->SetAttribute(con_out, EFI_LIGHTGRAY);
-	con_out->OutputString(con_out, (CHAR16 *)L"\r\n");
+void
+console::status_fail(const wchar_t *error) const
+{
+	m_out->SetAttribute(m_out, EFI_LIGHTGRAY);
+	m_out->OutputString(m_out, (wchar_t *)L"[");
+	m_out->SetAttribute(m_out, EFI_LIGHTRED);
+	m_out->OutputString(m_out, (wchar_t *)L"failed");
+	m_out->SetAttribute(m_out, EFI_LIGHTGRAY);
+	m_out->OutputString(m_out, (wchar_t *)L"]\r\n");
+
+	m_out->SetAttribute(m_out, EFI_RED);
+	m_out->OutputString(m_out, (wchar_t *)error);
+	m_out->SetAttribute(m_out, EFI_LIGHTGRAY);
+	m_out->OutputString(m_out, (wchar_t *)L"\r\n");
 }
 
 EFI_STATUS

src/boot/console.h

@@ -1,12 +1,38 @@
 #pragma once
+#include <stdarg.h>
 #include <stddef.h>
 #include <efi/efi.h>
 
-EFI_STATUS con_initialize(EFI_SYSTEM_TABLE *system_table, const CHAR16 *version);
-void con_status_begin(const CHAR16 *message);
-void con_status_ok();
-void con_status_fail(const CHAR16 *error);
-size_t con_printf(const CHAR16 *fmt, ...);
+class console
+{
+public:
+	console(EFI_SYSTEM_TABLE *system_table);
+
+	EFI_STATUS initialize(const wchar_t *version);
+
+	void status_begin(const wchar_t *message) const;
+	void status_fail(const wchar_t *error) const;
+	void status_ok() const;
+
+	size_t print_hex(uint32_t n) const;
+	size_t print_dec(uint32_t n) const;
+	size_t print_long_hex(uint64_t n) const;
+	size_t print_long_dec(uint64_t n) const;
+	size_t printf(const wchar_t *fmt, ...) const;
+
+	static const console & get() { return *s_console; }
+	static size_t print(const wchar_t *fmt, ...);
+
+private:
+	EFI_STATUS pick_mode();
+	size_t vprintf(const wchar_t *fmt, va_list args) const;
+
+	size_t m_rows, m_cols;
+	EFI_BOOT_SERVICES *m_boot;
+	EFI_SIMPLE_TEXT_OUT_PROTOCOL *m_out;
+
+	static console *s_console;
+};
 
 EFI_STATUS
 con_get_framebuffer(

src/boot/guids.inc

@@ -3,5 +3,9 @@ GUID(0x8868e871,0xe4f1,0x11d3,0xbc,0x22,0x00,0x80,0xc7,0x3c,0x88,0x81, guid_acpi
 GUID(0x09576e92,0x6d3f,0x11d2,0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b, guid_file_info);
 GUID(0x9042a9de,0x23dc,0x4a38,0x96,0xfb,0x7a,0xde,0xd0,0x80,0x51,0x6a, guid_gfx_out);
 GUID(0x964e5b22,0x6459,0x11d2,0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b, guid_simple_filesystem);
+GUID(0x09576e91,0x6d3f,0x11d2,0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b, guid_device_path);
+GUID(0x8b843e20,0x8132,0x4852,0x90,0xcc,0x55,0x1a,0x4e,0x4a,0x7f,0x1c, guid_device_path_to_text);
+
+GUID(0x10d0669c,0x9ec6,0x4268,0xbc,0x48,0xff,0x74,0x75,0x21,0xfe,0x07, guid_popcorn_vendor);
 
 // vim: ft=c

src/boot/loader.cpp

@@ -6,8 +6,59 @@
 
 #define PAGE_SIZE 0x1000
 
-static CHAR16 kernel_name[] = KERNEL_FILENAME;
-static CHAR16 initrd_name[] = INITRD_FILENAME;
+static wchar_t kernel_name[] = KERNEL_FILENAME;
+static wchar_t initrd_name[] = INITRD_FILENAME;
+
+EFI_STATUS
+loader_alloc_aligned(
+	EFI_BOOT_SERVICES *bootsvc,
+	EFI_MEMORY_TYPE mem_type,
+	size_t *length,
+	void **pages)
+{
+	EFI_STATUS status;
+	EFI_PHYSICAL_ADDRESS addr;
+
+	size_t alignment = PAGE_SIZE;
+	while (alignment < *length)
+		alignment *= 2;
+
+	size_t page_count = alignment / PAGE_SIZE;
+	*length = alignment;
+
+	con_debug(L"Trying to find %d aligned pages for %x\n", page_count, mem_type);
+
+	status = bootsvc->AllocatePages(AllocateAnyPages, mem_type, page_count * 2, &addr);
+	CHECK_EFI_STATUS_OR_RETURN(status, L"Allocating %d pages for alignment", page_count * 2);
+	con_debug(L"    Found %d pages at %lx\n", page_count * 2, addr);
+
+	EFI_PHYSICAL_ADDRESS aligned = addr;
+	aligned = ((aligned - 1) & ~(alignment - 1)) + alignment;
+	con_debug(L"    Aligning %lx to %lx\n", addr, aligned);
+
+	size_t before = 
+		(reinterpret_cast<uint64_t>(aligned) -
+		reinterpret_cast<uint64_t>(addr)) /
+		PAGE_SIZE;
+
+	if (before) {
+		con_debug(L"    Freeing %d initial pages\n", before);
+		bootsvc->FreePages(addr, before);
+	}
+
+	size_t after = page_count - before;
+	if (after) {
+		EFI_PHYSICAL_ADDRESS end = 
+			reinterpret_cast<EFI_PHYSICAL_ADDRESS>(
+				reinterpret_cast<uint64_t>(aligned) +
+				page_count * PAGE_SIZE);
+		con_debug(L"    Freeing %d remaining pages\n", after);
+		bootsvc->FreePages(end, after);
+	}
+
+	*pages = (void *)aligned;
+	return EFI_SUCCESS;
+}
 
 EFI_STATUS
 loader_alloc_pages(
@@ -21,12 +72,10 @@ loader_alloc_pages(
 	size_t page_count = ((*length - 1) / PAGE_SIZE) + 1;
 	EFI_PHYSICAL_ADDRESS addr = (EFI_PHYSICAL_ADDRESS)*pages;
 
+	con_debug(L"Trying to find %d non-aligned pages for %x at %lx\n",
+			page_count, mem_type, addr);
+
 	status = bootsvc->AllocatePages(AllocateAddress, mem_type, page_count, &addr);
-	if (status == EFI_NOT_FOUND || status == EFI_OUT_OF_RESOURCES) {
-		// couldn't get the address we wanted, try loading the kernel anywhere
-		status =
-			bootsvc->AllocatePages(AllocateAnyPages, mem_type, page_count, &addr);
-	}
 	CHECK_EFI_STATUS_OR_RETURN(status,
 		L"Allocating %d kernel pages type %x",
 		page_count, mem_type);
@@ -46,7 +95,7 @@ loader_load_initrd(
 	EFI_STATUS status;
 
 	EFI_FILE_PROTOCOL *file = NULL;
-	status = root->Open(root, &file, (CHAR16 *)initrd_name, EFI_FILE_MODE_READ,
+	status = root->Open(root, &file, (wchar_t *)initrd_name, EFI_FILE_MODE_READ,
 						EFI_FILE_READ_ONLY | EFI_FILE_HIDDEN | EFI_FILE_SYSTEM);
 
 	if (status == EFI_NOT_FOUND)
@@ -62,9 +111,9 @@ loader_load_initrd(
 
 	data->initrd_length = ((EFI_FILE_INFO *)info)->FileSize;
 
-	status = loader_alloc_pages(
+	status = loader_alloc_aligned(
 			bootsvc,
-			INITRD_MEMTYPE,
+			memtype_initrd,
 			&data->initrd_length,
 			&data->initrd);
 	CHECK_EFI_STATUS_OR_RETURN(status, L"Allocating pages");
@@ -87,10 +136,10 @@ loader_load_elf(
 {
 	EFI_STATUS status;
 
-	con_debug(L"Opening kernel file %s\r\n", (CHAR16 *)kernel_name);
+	con_debug(L"Opening kernel file %s\r\n", (wchar_t *)kernel_name);
 
 	EFI_FILE_PROTOCOL *file = NULL;
-	status = root->Open(root, &file, (CHAR16 *)kernel_name, EFI_FILE_MODE_READ,
+	status = root->Open(root, &file, (wchar_t *)kernel_name, EFI_FILE_MODE_READ,
 						EFI_FILE_READ_ONLY | EFI_FILE_HIDDEN | EFI_FILE_SYSTEM);
 
 	if (status == EFI_NOT_FOUND)
@@ -135,13 +184,12 @@ loader_load_elf(
 		header.machine != 0x3e)
 		CHECK_EFI_STATUS_OR_RETURN(EFI_LOAD_ERROR, L"ELF load error: wrong machine architecture");
 
-	con_debug(L"ELF is valid, entrypoint %lu\r\n", header.entrypoint);
+	con_debug(L"ELF is valid, entrypoint %lx\r\n", header.entrypoint);
 
 	data->kernel_entry = (void *)header.entrypoint;
 
 	struct elf_program_header prog_header;
 	for (int i = 0; i < header.ph_num; ++i) {
-		con_debug(L"Reading ELF program header %d\r\n", i);
 
 		status = file->SetPosition(file, header.ph_offset + i * header.ph_entsize);
 		CHECK_EFI_STATUS_OR_RETURN(status, L"Setting ELF file position");
@@ -154,18 +202,17 @@ loader_load_elf(
 
 		length = prog_header.mem_size;
 		void *addr = (void *)(prog_header.vaddr - KERNEL_VIRT_ADDRESS);
-		status = loader_alloc_pages(bootsvc, KERNEL_MEMTYPE, &length, &addr);
+		status = loader_alloc_pages(bootsvc, memtype_kernel, &length, &addr);
 		CHECK_EFI_STATUS_OR_RETURN(status, L"Allocating kernel pages");
 
 		if (data->kernel == 0)
 			data->kernel = addr;
 		data->kernel_length = (uint64_t)addr + length - (uint64_t)data->kernel;
 	}
+	con_debug(L"Read %d ELF program headers\r\n", header.ph_num);
 
 	struct elf_section_header sec_header;
 	for (int i = 0; i < header.sh_num; ++i) {
-		con_debug(L"Reading ELF section header %d ", i);
-
 		status = file->SetPosition(file, header.sh_offset + i * header.sh_entsize);
 		CHECK_EFI_STATUS_OR_RETURN(status, L"Setting ELF file position");
 
@@ -174,14 +221,12 @@ loader_load_elf(
 		CHECK_EFI_STATUS_OR_RETURN(status, L"Reading ELF section header");
 
 		if ((sec_header.flags & ELF_SHF_ALLOC) == 0) {
-			con_debug(L"SHF_ALLOC section\r\n");
 			continue;
 		}
 
 		void *addr = (void *)(sec_header.addr - KERNEL_VIRT_ADDRESS);
 
 		if (sec_header.type == ELF_ST_PROGBITS) {
-			con_debug(L"PROGBITS section\r\n");
 			status = file->SetPosition(file, sec_header.offset);
 			CHECK_EFI_STATUS_OR_RETURN(status, L"Setting ELF file position");
 
@@ -189,12 +234,10 @@ loader_load_elf(
 			status = file->Read(file, &length, addr);
 			CHECK_EFI_STATUS_OR_RETURN(status, L"Reading file");
 		} else if (sec_header.type == ELF_ST_NOBITS) {
-			con_debug(L"NOBITS section\r\n");
 			bootsvc->SetMem(addr, sec_header.size, 0);
-		} else {
-			con_debug(L"other section\r\n");
 		}
 	}
+	con_debug(L"Read %d ELF section headers\r\n", header.ph_num);
 
 	status = file->Close(file);
 	CHECK_EFI_STATUS_OR_RETURN(status, L"Closing file handle");
@@ -234,19 +277,19 @@ loader_load_kernel(
 
 		CHECK_EFI_STATUS_OR_RETURN(status, L"loader_load_elf: %s", kernel_name);
 
-		data->initrd = (void *)((uint64_t)data->kernel + data->kernel_length);
-		status = loader_load_initrd(bootsvc, root, data);
-
-		CHECK_EFI_STATUS_OR_RETURN(status, L"loader_load_file: %s", initrd_name);
-
-		data->data = (void *)((uint64_t)data->initrd + data->initrd_length);
+		data->data = (void *)((uint64_t)data->kernel + data->kernel_length);
 		data->data_length += PAGE_SIZE; // extra page for map growth
-		status = loader_alloc_pages(
+
+		status = loader_alloc_aligned(
 				bootsvc,
-				KERNEL_DATA_MEMTYPE,
+				memtype_data,
 				&data->data_length,
 				&data->data);
-		CHECK_EFI_STATUS_OR_RETURN(status, L"loader_alloc_pages: kernel data");
+		CHECK_EFI_STATUS_OR_RETURN(status, L"loader_alloc_aligned: kernel data");
+
+		data->initrd = (void *)((uint64_t)data->data + data->data_length);
+		status = loader_load_initrd(bootsvc, root, data);
+		CHECK_EFI_STATUS_OR_RETURN(status, L"loader_load_file: %s", initrd_name);
 
 		return EFI_SUCCESS;
 	}

src/boot/loader.h

@@ -12,18 +12,6 @@
 #define KERNEL_VIRT_ADDRESS 0xFFFFFF0000000000
 #endif
 
-#ifndef KERNEL_MEMTYPE
-#define KERNEL_MEMTYPE 0x80000000
-#endif
-
-#ifndef INITRD_MEMTYPE
-#define INITRD_MEMTYPE 0x80000001
-#endif
-
-#ifndef KERNEL_DATA_MEMTYPE
-#define KERNEL_DATA_MEMTYPE 0x80000002
-#endif
-
 #ifndef KERNEL_FILENAME
 #define KERNEL_FILENAME L"kernel.elf"
 #endif

src/boot/main.cpp

@@ -5,11 +5,15 @@
 
 #include "console.h"
 #include "guids.h"
-#include "kernel_data.h"
+#include "kernel_args.h"
 #include "loader.h"
 #include "memory.h"
 #include "utility.h"
 
+#ifndef SCRATCH_PAGES
+#define SCRATCH_PAGES 64
+#endif
+
 #ifndef GIT_VERSION_WIDE
 #define GIT_VERSION_WIDE L"no version"
 #endif
@@ -30,20 +34,68 @@ struct kernel_header {
 };
 #pragma pack(pop)
 
+using kernel_entry = void (*)(kernel_args *);
+
+static void
+type_to_wchar(wchar_t *into, uint32_t type)
+{
+	for (int j=0; j<4; ++j)
+		into[j] = static_cast<wchar_t>(reinterpret_cast<char *>(&type)[j]);
+}
+
 EFI_STATUS
+detect_debug_mode(EFI_RUNTIME_SERVICES *run, kernel_args *header) {
+	wchar_t var_name[] = L"debug";
+
+	EFI_STATUS status;
+	uint8_t debug = 0;
+	UINTN var_size = sizeof(debug);
+
+#ifdef __POPCORN_SET_DEBUG_UEFI_VAR__
+	debug = __POPCORN_SET_DEBUG_UEFI_VAR__;
+	uint32_t attrs =
+		EFI_VARIABLE_NON_VOLATILE |
+		EFI_VARIABLE_BOOTSERVICE_ACCESS |
+		EFI_VARIABLE_RUNTIME_ACCESS;
+	status = run->SetVariable(
+			var_name,
+			&guid_popcorn_vendor,
+			attrs,
+			var_size,
+			&debug);
+	CHECK_EFI_STATUS_OR_RETURN(status, "detect_debug_mode::SetVariable");
+#endif
+
+	status = run->GetVariable(
+			var_name,
+			&guid_popcorn_vendor,
+			nullptr,
+			&var_size,
+			&debug);
+	CHECK_EFI_STATUS_OR_RETURN(status, "detect_debug_mode::GetVariable");
+
+	if (debug)
+		header->flags |= POPCORN_FLAG_DEBUG;
+
+	return EFI_SUCCESS;
+}
+
+extern "C" EFI_STATUS
 efi_main(EFI_HANDLE image_handle, EFI_SYSTEM_TABLE *system_table)
 {
 	EFI_STATUS status;
 	EFI_BOOT_SERVICES *bootsvc = system_table->BootServices;
 	EFI_RUNTIME_SERVICES *runsvc = system_table->RuntimeServices;
 
+	console con(system_table);
+
 	// When checking console initialization, use CHECK_EFI_STATUS_OR_RETURN
 	// because we can't be sure if the console was fully set up
-	status = con_initialize(system_table, GIT_VERSION_WIDE);
-	CHECK_EFI_STATUS_OR_RETURN(status, "con_initialize");
+	status = con.initialize(GIT_VERSION_WIDE);
+	CHECK_EFI_STATUS_OR_RETURN(status, "console::initialize");
 	// From here on out, we can use CHECK_EFI_STATUS_OR_FAIL instead
 
-	memory_init_pointer_fixup(bootsvc, runsvc);
+	memory_init_pointer_fixup(bootsvc, runsvc, SCRATCH_PAGES);
 
 	// Find ACPI tables. Ignore ACPI 1.0 if a 2.0 table is found.
 	//
@@ -65,50 +117,51 @@ efi_main(EFI_HANDLE image_handle, EFI_SYSTEM_TABLE *system_table)
 	status = memory_get_map_length(bootsvc, &data_length);
 	CHECK_EFI_STATUS_OR_FAIL(status);
 
-	size_t header_size = sizeof(struct popcorn_data);
-	const size_t header_align = alignof(struct popcorn_data);
+	size_t header_size = sizeof(kernel_args);
+	const size_t header_align = alignof(kernel_args);
 	if (header_size % header_align)
 		header_size += header_align - (header_size % header_align);
 
 	data_length += header_size;
 
-
 	// Load the kernel image from disk and check it
 	//
-	con_printf(L"Loading kernel into memory...\r\n");
+	console::print(L"Loading kernel into memory...\r\n");
 
 	struct loader_data load;
 	load.data_length = data_length;
 	status = loader_load_kernel(bootsvc, &load);
 	CHECK_EFI_STATUS_OR_FAIL(status);
 
-	con_printf(L"    %u image bytes at 0x%x\r\n", load.kernel_length, load.kernel);
-	con_printf(L"    %u initrd bytes at 0x%x\r\n", load.initrd_length, load.initrd);
-	con_printf(L"    %u data bytes at 0x%x\r\n", load.data_length, load.data);
+	console::print(L"    %x image bytes  at 0x%x\r\n", load.kernel_length, load.kernel);
+	console::print(L"    %x data bytes   at 0x%x\r\n", load.data_length, load.data);
+	console::print(L"    %x initrd bytes at 0x%x\r\n", load.initrd_length, load.initrd);
 
 	struct kernel_header *version = (struct kernel_header *)load.kernel;
 	if (version->magic != KERNEL_HEADER_MAGIC) {
-		con_printf(L"    bad magic %x\r\n", version->magic);
+		console::print(L"    bad magic %x\r\n", version->magic);
 		CHECK_EFI_STATUS_OR_FAIL(EFI_CRC_ERROR);
 	}
 
-	con_printf(L"    Kernel version %d.%d.%d %x%s\r\n",
+	console::print(L"    Kernel version %d.%d.%d %x%s\r\n",
 			version->major, version->minor, version->patch, version->gitsha & 0x0fffffff,
-			version->gitsha & 0xf0000000 ? "*" : "");
-	con_printf(L"    Entrypoint 0x%x\r\n", load.kernel_entry);
+			version->gitsha & 0xf0000000 ? L"*" : L"");
+	console::print(L"    Entrypoint 0x%x\r\n", load.kernel_entry);
 
-	void (*kernel_main)() = load.kernel_entry;
+	kernel_entry kernel_main =
+		reinterpret_cast<kernel_entry>(load.kernel_entry);
 	memory_mark_pointer_fixup((void **)&kernel_main);
 
 	// Set up the kernel data pages to pass to the kernel
 	//
-	struct popcorn_data *data_header = (struct popcorn_data *)load.data; 
+	struct kernel_args *data_header = (struct kernel_args *)load.data;
 	memory_mark_pointer_fixup((void **)&data_header);
 
 	data_header->magic = DATA_HEADER_MAGIC;
 	data_header->version = DATA_HEADER_VERSION;
-	data_header->length = sizeof(struct popcorn_data);
+	data_header->length = sizeof(struct kernel_args);
 
+	data_header->scratch_pages = SCRATCH_PAGES;
 	data_header->flags = 0;
 
 	data_header->initrd = load.initrd;
@@ -148,8 +201,9 @@ efi_main(EFI_HANDLE image_handle, EFI_SYSTEM_TABLE *system_table)
 	// Save the memory map and tell the firmware we're taking control.
 	//
 	struct memory_map map;
-	map.entries = data_header->memory_map;
 	map.length = (load.data_length - header_size);
+	map.entries =
+		reinterpret_cast<EFI_MEMORY_DESCRIPTOR *>(data_header->memory_map);
 
 	status = memory_get_map(bootsvc, &map);
 	CHECK_EFI_STATUS_OR_FAIL(status);
@@ -157,6 +211,8 @@ efi_main(EFI_HANDLE image_handle, EFI_SYSTEM_TABLE *system_table)
 	data_header->memory_map_length = map.length;
 	data_header->memory_map_desc_size = map.size;
 
+	detect_debug_mode(runsvc, data_header);
+
 	// bootsvc->Stall(5000000);
 
 	status = bootsvc->ExitBootServices(image_handle, map.key);

src/boot/memory.cpp

@@ -4,13 +4,18 @@
 #include "memory.h"
 #include "utility.h"
 
+const EFI_MEMORY_TYPE memtype_kernel  = static_cast<EFI_MEMORY_TYPE>(0x80000000);
+const EFI_MEMORY_TYPE memtype_data    = static_cast<EFI_MEMORY_TYPE>(0x80000001);
+const EFI_MEMORY_TYPE memtype_initrd  = static_cast<EFI_MEMORY_TYPE>(0x80000002);
+const EFI_MEMORY_TYPE memtype_scratch = static_cast<EFI_MEMORY_TYPE>(0x80000003);
+
 #define INCREMENT_DESC(p, b)  (EFI_MEMORY_DESCRIPTOR*)(((uint8_t*)(p))+(b))
 
 size_t fixup_pointer_index = 0;
 void **fixup_pointers[64];
 uint64_t *new_pml4 = 0;
 
-const CHAR16 *memory_type_names[] = {
+const wchar_t *memory_type_names[] = {
 	L"EfiReservedMemoryType",
 	L"EfiLoaderCode",
 	L"EfiLoaderData",
@@ -28,13 +33,17 @@ const CHAR16 *memory_type_names[] = {
 	L"EfiPersistentMemory",
 };
 
-static const CHAR16 *
+static const wchar_t *
 memory_type_name(UINT32 value)
 {
-	if (value >= (sizeof(memory_type_names) / sizeof(CHAR16 *))) {
-		if (value == KERNEL_DATA_MEMTYPE) return L"Kernel Data";
-		else if (value == KERNEL_MEMTYPE) return L"Kernel Image";
-		else return L"Bad Type Value";
+	if (value >= (sizeof(memory_type_names) / sizeof(wchar_t *))) {
+		switch (value) {
+			case memtype_kernel:  return L"Kernel Data";
+			case memtype_data:    return L"Kernel Data";
+			case memtype_initrd:  return L"Initial Ramdisk";
+			case memtype_scratch: return L"Kernel Scratch Space";
+			default: return L"Bad Type Value";
+		}
 	}
 	return memory_type_names[value];
 }
@@ -50,7 +59,7 @@ memory_update_marked_addresses(EFI_EVENT UNUSED *event, void *context)
 }
 
 EFI_STATUS
-memory_init_pointer_fixup(EFI_BOOT_SERVICES *bootsvc, EFI_RUNTIME_SERVICES *runsvc)
+memory_init_pointer_fixup(EFI_BOOT_SERVICES *bootsvc, EFI_RUNTIME_SERVICES *runsvc, unsigned scratch_pages)
 {
 	EFI_STATUS status;
 	EFI_EVENT event;
@@ -67,7 +76,7 @@ memory_init_pointer_fixup(EFI_BOOT_SERVICES *bootsvc, EFI_RUNTIME_SERVICES *runs
 	// Reserve a page for our replacement PML4, plus some pages for the kernel to use
 	// as page tables while it gets started.
 	EFI_PHYSICAL_ADDRESS addr = 0;
-	status = bootsvc->AllocatePages(AllocateAnyPages, EfiLoaderData, 16, &addr);
+	status = bootsvc->AllocatePages(AllocateAnyPages, memtype_scratch, scratch_pages, &addr);
 	CHECK_EFI_STATUS_OR_RETURN(status, "Failed to allocate page table pages.");
 
 	new_pml4 = (uint64_t *)addr;
@@ -139,19 +148,19 @@ memory_dump_map(struct memory_map *map)
 
 	const size_t count = map->length / map->size;
 
-	con_printf(L"Memory map:\n");
-	con_printf(L"\t	Descriptor Count: %d (%d bytes)\n", count, map->length);
-	con_printf(L"\t  Descriptor Size: %d bytes\n", map->size);
-	con_printf(L"\t      Type offset: %d\n\n", offsetof(EFI_MEMORY_DESCRIPTOR, Type));
+	console::print(L"Memory map:\n");
+	console::print(L"\t	Descriptor Count: %d (%d bytes)\n", count, map->length);
+	console::print(L"\t  Descriptor Size: %d bytes\n", map->size);
+	console::print(L"\t      Type offset: %d\n\n", offsetof(EFI_MEMORY_DESCRIPTOR, Type));
 
 	EFI_MEMORY_DESCRIPTOR *end = INCREMENT_DESC(map->entries, map->length);
 	EFI_MEMORY_DESCRIPTOR *d = map->entries;
 	while (d < end) {
 		int runtime = (d->Attribute & EFI_MEMORY_RUNTIME) == EFI_MEMORY_RUNTIME;
-		con_printf(L"%s%s ", memory_type_name(d->Type), runtime ? L"*" : L" ");
-		con_printf(L"%lx ", d->PhysicalStart);
-		con_printf(L"%lx ", d->VirtualStart);
-		con_printf(L"[%4d]\n", d->NumberOfPages);
+		console::print(L"%s%s ", memory_type_name(d->Type), runtime ? L"*" : L" ");
+		console::print(L"%lx ", d->PhysicalStart);
+		console::print(L"%lx ", d->VirtualStart);
+		console::print(L"[%4d]\n", d->NumberOfPages);
 
 		d = INCREMENT_DESC(d, map->size);
 	}
@@ -187,9 +196,10 @@ memory_virtualize(EFI_RUNTIME_SERVICES *runsvc, struct memory_map *map)
 	EFI_MEMORY_DESCRIPTOR *d = map->entries;
 	while (d < end) {
 		switch (d->Type) {
-		case KERNEL_MEMTYPE:
-		case INITRD_MEMTYPE:
-		case KERNEL_DATA_MEMTYPE:
+		case memtype_kernel:
+		case memtype_data:
+		case memtype_initrd:
+		case memtype_scratch:
 			d->Attribute |= EFI_MEMORY_RUNTIME;
 			d->VirtualStart = d->PhysicalStart + KERNEL_VIRT_ADDRESS;
 

src/boot/memory.h

@@ -1,6 +1,11 @@
 #pragma once
 #include <efi/efi.h>
 
+extern const EFI_MEMORY_TYPE memtype_kernel;
+extern const EFI_MEMORY_TYPE memtype_data;
+extern const EFI_MEMORY_TYPE memtype_initrd;
+extern const EFI_MEMORY_TYPE memtype_scratch;
+
 struct memory_map {
 	size_t length;
 	size_t size;
@@ -9,7 +14,10 @@ struct memory_map {
 	EFI_MEMORY_DESCRIPTOR *entries;
 };
 
-EFI_STATUS memory_init_pointer_fixup(EFI_BOOT_SERVICES *bootsvc, EFI_RUNTIME_SERVICES *runsvc);
+EFI_STATUS memory_init_pointer_fixup(
+		EFI_BOOT_SERVICES *bootsvc,
+		EFI_RUNTIME_SERVICES *runsvc,
+		unsigned scratch_pages);
 void memory_mark_pointer_fixup(void **p);
 
 EFI_STATUS memory_get_map_length(EFI_BOOT_SERVICES *bootsvc, size_t *size);

src/boot/reloc.cpp

@@ -38,6 +38,7 @@
 #include <efi/efi.h>
 #include <elf.h>
 
+extern "C"
 EFI_STATUS _relocate (long ldbase, Elf64_Dyn *dyn,
 		      EFI_HANDLE image EFI_UNUSED,
 		      EFI_SYSTEM_TABLE *systab EFI_UNUSED)

src/boot/utility.cpp

@@ -2,7 +2,7 @@
 
 struct error_code_desc {
 	EFI_STATUS code;
-	CHAR16 *name;
+	const wchar_t *name;
 };
 
 // Based off the gnu-efi table
@@ -47,7 +47,7 @@ struct error_code_desc error_table[] = {
 	{ 0, NULL }
 };
 
-const CHAR16 *
+const wchar_t *
 util_error_message(EFI_STATUS status)
 {
 	int32_t i = -1;
@@ -62,7 +62,7 @@ util_error_message(EFI_STATUS status)
 }
 
 size_t
-wstrlen(const CHAR16 *s)
+wstrlen(const wchar_t *s)
 {
 	size_t count = 0;
 	while (s && *s++) count++;

src/boot/utility.h

@@ -5,18 +5,18 @@
 
 #define UNUSED __attribute__((unused))
 
-size_t wstrlen(const CHAR16 *s);
-const CHAR16 *util_error_message(EFI_STATUS status);
+size_t wstrlen(const wchar_t *s);
+const wchar_t *util_error_message(EFI_STATUS status);
 
 #define CHECK_EFI_STATUS_OR_RETURN(s, msg, ...) \
 	if (EFI_ERROR((s))) { \
-		con_printf(L"ERROR: " msg L": %s\r\n", ##__VA_ARGS__, util_error_message(s)); \
+		console::print(L"ERROR: " msg L": %s\r\n", ##__VA_ARGS__, util_error_message(s)); \
 		return (s); \
 	}
 
 #define CHECK_EFI_STATUS_OR_FAIL(s) \
 	if (EFI_ERROR((s))) { \
-		con_status_fail(util_error_message(s)); \
+		console::get().status_fail(util_error_message(s)); \
 		while (1) __asm__("hlt"); \
 	}
 
@@ -34,7 +34,7 @@ const CHAR16 *util_error_message(EFI_STATUS status);
 	}
 
 #ifdef BOOTLOADER_DEBUG
-#define con_debug(...) con_printf(L"DEBUG: " __VA_ARGS__)
+#define con_debug(...) console::print(L"DEBUG: " __VA_ARGS__)
 #else
 #define con_debug(...)
 #endif

src/drivers/nulldrv/main.cpp

@@ -1 +1,24 @@
-int main(int argc, const char **argv) { return 0; }
+#include <stdint.h>
+#include <stdlib.h>
+
+extern "C" {
+	int32_t getpid();
+	int32_t fork();
+	void sleep(uint64_t til);
+	void debug();
+	void message(const char *msg);
+
+	int main(int, const char **);
+}
+
+
+int
+main(int argc, const char **argv)
+{
+	int32_t pid = getpid();
+	int32_t child = fork();
+	message("hello from nulldrv!");
+	for (int i = 1; i < 5; ++i)
+		sleep(i*10);
+	return 0;
+}

src/drivers/nulldrv/main.s

@@ -1,25 +1,79 @@
+section .bss
+mymessage:
+	resq 1024
+
+extern main
+extern exit
+
+section .text
+global getpid
+getpid:
+	push rbp
+	mov rbp, rsp
+
+	mov rax, 0x02  ; getpid syscall
+	syscall        ; pid is now already in rax, so just return
+
+	pop rbp
+	ret
+
+global debug
+debug:
+	push rbp
+	mov rbp, rsp
+
+	mov rax, 0x00  ; debug syscall
+	syscall
+
+	pop rbp
+	ret
+
+global sleep
+sleep:
+	push rbp
+	mov rbp, rsp
+
+	mov rax, 0x21  ; sleep syscall
+	syscall
+
+	pop rbp
+	ret
+
+global fork
+fork:
+	push rbp
+	mov rbp, rsp
+
+	mov rax, 0x03
+	syscall    ; pid left in rax
+
+	pop rbp
+	ret
+	
+
+global message
+message:
+	push rbp
+	mov rbp, rsp
+
+	; message should already be in rdi
+	mov rax, 0x10
+	syscall
+
+	pop rbp
+	ret
+
+
 global _start
 _start:
 	xor rbp, rbp	; Sentinel rbp
-	mov r11, 0		; counter
-	mov rbx, 20		; sleep timeout
+	push rbp
+	push rbp
+	mov rbp, rsp
 
-.loop:
-	mov rax, 1		; DEBUG syscall
-	;mov rax, 0		; NOOP syscall
-	;syscall
-	int 0xee
+	mov rdi, 0
+	mov rsi, 0
+	call main
 
-	inc r11
-	cmp r11, 2
-
-	jle .loop
-
-	mov rax, 4		; SLEEP syscall
-	; syscall
-	int 0xee
-
-	add rbx, 20
-
-	mov r11, 0
-	jmp .loop
+	mov rdi, rax
+	call exit

src/include/efi/efidef.h

@@ -20,9 +20,9 @@ Revision History
 
 --*/
 
-typedef UINT16          CHAR16;
-typedef UINT8           CHAR8;
-typedef UINT8           BOOLEAN;
+typedef wchar_t         CHAR16;
+typedef char            CHAR8;
+typedef uint8_t         BOOLEAN;
 #ifndef CONST
    #define CONST const
 #endif

src/include/kernel_args.h

@@ -7,13 +7,16 @@
 #define DATA_HEADER_MAGIC     0x600dda7a
 #define DATA_HEADER_VERSION   1
 
+#define POPCORN_FLAG_DEBUG    0x00000001
+
 #pragma pack(push, 1)
-struct popcorn_data {
+struct kernel_args {
 	uint32_t magic;
 	uint16_t version;
 	uint16_t length;
 
-	uint32_t _reserved0;
+	uint16_t _reserved0;
+	uint16_t scratch_pages;
 	uint32_t flags;
 
 	void *initrd;

src/include/kernel_memory.h

@@ -0,0 +1,32 @@
+#pragma once
+/// \file kernel_memory.h
+/// Constants related to the kernel's memory layout
+
+#include <stddef.h>
+#include <stdint.h>
+
+namespace memory {
+
+	/// Size of a single page frame.
+	static const size_t frame_size = 0x1000;
+
+	/// Start of kernel memory.
+	static const uintptr_t kernel_offset = 0xffffff0000000000;
+
+	/// Offset from physical where page tables are mapped.
+	static const uintptr_t page_offset = 0xffffff8000000000;
+
+	/// Initial process thread's stack address
+	static const uintptr_t initial_stack = 0x0000800000000000;
+
+	/// Initial process thread's stack size, in pages
+	static const unsigned initial_stack_pages = 1;
+
+	/// Max size of the kernel heap
+	static const size_t kernel_max_heap = 0x800000000; // 32GiB
+
+	/// Helper to determine if a physical address can be accessed
+	/// through the page_offset area.
+	inline bool page_mappable(uintptr_t a) { return (a & page_offset) == 0; }
+
+} // namespace memory

src/include/log_areas.inc

@@ -0,0 +1,9 @@
+LOG(apic,   info);
+LOG(device, info);
+LOG(paging, warn);
+LOG(driver, info);
+LOG(memory, info);
+LOG(fs,     info);
+LOG(task,   info);
+LOG(boot,   info);
+LOG(syscall,info);

src/kernel/apic.cpp

@@ -46,6 +46,7 @@ apic::apic(uint32_t *base) :
 lapic::lapic(uint32_t *base, isr spurious) :
 	apic(base)
 {
+	// TODO: This causes a "reserved" page fault under KVM
 	apic_write(m_base, 0xf0, static_cast<uint32_t>(spurious));
 	log::info(logs::apic, "LAPIC created, base %lx", m_base);
 }

src/kernel/assert.cpp

@@ -8,19 +8,14 @@ __kernel_assert(const char *file, unsigned line, const char *message)
 	if (cons) {
 		cons->set_color(9 , 0);
 		cons->puts("\n\n       ERROR: ");
+		cons->puts(message);
+		cons->puts("\n              ");
 		cons->puts(file);
 		cons->puts(":");
 		cons->put_dec(line);
-		cons->puts(":  ");
-		cons->puts(message);
+		cons->puts("\n");
 	}
 
-	__asm__ ( "int $0e7h" );
 	while (1) __asm__ ("hlt");
-}
-
-extern "C" [[noreturn]] void
-__assert_fail(const char *message, const char *file, unsigned int line, const char *function)
-{
-	__kernel_assert(file, line, message);
+	__asm__ ( "int $0xe4" );
 }

src/kernel/boot.s

@@ -18,11 +18,11 @@ global _start:function (_start.end - _start)
 _start:
 	cli
 
-	mov rsp, stack_end
-	push 0 ; signal end of stack with 0 return address
-	push 0 ; and a few extra entries in case of stack
-	push 0 ; problems
-	push 0
+	mov rsp, idle_stack_end
+	mov qword [rsp + 0x00], 0 ; signal end of stack with 0 return address
+	mov qword [rsp + 0x08], 0 ; and a few extra entries in case of stack
+	mov qword [rsp + 0x10], 0 ; problems
+	mov qword [rsp + 0x18], 0
 
 	mov rbp, rsp
 	extern kernel_main
@@ -47,6 +47,9 @@ interrupts_disable:
 
 section .bss
 align 0x100
-stack_begin:
-	resb 0x4000 ; 16KiB stack space
-stack_end:
+idle_stack_begin:
+	resb 0x1000 ; 4KiB stack space
+
+global idle_stack_end
+idle_stack_end:
+	resq 4

src/kernel/console.cpp

@@ -1,6 +1,7 @@
 #include "kutil/coord.h"
 #include "kutil/guid.h"
 #include "kutil/memory.h"
+#include "kutil/printf.h"
 #include "console.h"
 #include "font.h"
 #include "screen.h"
@@ -278,100 +279,9 @@ console::putc(char c)
 void console::vprintf(const char *fmt, va_list args)
 {
 	static const unsigned buf_size = 256;
-	char buffer[256];
-
-	const char *r = fmt;
-	char *w = buffer;
-	char *wend = buffer + buf_size;
-
-#define flush() do { *w = 0; puts(buffer); w = buffer; } while(0)
-
-	while (r && *r) {
-		if (w == wend) flush();
-
-		if (*r != '%') {
-			*w++ = *r++;
-			continue;
-		}
-
-		r++; // chomp the %
-		flush();
-
-		bool done = false;
-		bool right = true;
-		int width = 0;
-		char pad = ' ';
-
-		while (!done) {
-			char c = *r++;
-			switch (c) {
-				case '%': *w = '%'; done = true; break;
-
-				case '0':
-					if (width == 0) pad = '0';
-					// else fall through
-				case '1':
-				case '2':
-				case '3':
-				case '4':
-				case '5':
-				case '6':
-				case '7':
-				case '8':
-				case '9': width = width * 10 + (c - '0'); break;
-
-				case '-': right = !right; break;
-
-				case 'x': put_hex<uint32_t>(va_arg(args, uint32_t), right ? width : -width, pad); done = true; break;
-
-				case 'd':
-				case 'u': put_dec<uint32_t>(va_arg(args, uint32_t), right ? width : -width, pad); done = true; break;
-
-				case 's': {
-						const char *s = va_arg(args, const char*);
-						if (s) puts(s);
-					}
-					done = true;
-					break;
-
-				case 'G': {
-						// Special: GUID type
-						kutil::guid g = va_arg(args, kutil::guid);
-						put_hex<uint32_t>(g.a, 8, '0');
-						putc('-');
-						put_hex<uint16_t>((g.a >> 32) & 0xffff, 4, '0');
-						putc('-');
-						put_hex<uint16_t>((g.a >> 48) & 0xffff, 4, '0');
-						putc('-');
-						put_hex<uint16_t>((kutil::byteswap(g.b) >> 16) & 0xffff, 4, '0');
-						putc('-');
-						put_hex<uint16_t>(kutil::byteswap(g.b) & 0xffff, 4, '0');
-						put_hex<uint32_t>(kutil::byteswap(g.b >> 32), 8, '0');
-					}
-					done = true;
-					break;
-
-				case 'l':
-					switch (*r++) {
-						case 'x': put_hex<uint64_t>(va_arg(args, uint64_t), right ? width : -width, pad); done = true; break;
-
-						case 'd':
-						case 'u': put_dec<uint32_t>(va_arg(args, uint64_t), right ? width : -width, pad); done = true; break;
-
-						default:
-							done = true;
-							break;
-					}
-					break;
-
-				default:
-					done = true;
-					break;
-			}
-		}
-	}
-
-	flush();
+	char buffer[buf_size];
+	vsnprintf_(buffer, buf_size, fmt, args);
+	puts(buffer);
 }
 
 void

src/kernel/cpu.cpp

@@ -3,6 +3,8 @@
 #include "cpu.h"
 #include "log.h"
 
+cpu_data bsp_cpu_data;
+
 inline static void
 __cpuid(
 	uint32_t leaf,

src/kernel/cpu.h

@@ -1,14 +1,28 @@
 #pragma once
 
+#include <stdint.h>
+
+struct process;
+
 struct cpu_state
 {
-	uint64_t ds;
 	uint64_t r15, r14, r13, r12, r11, r10, r9,  r8;
 	uint64_t rdi, rsi, rbp, rbx, rdx, rcx, rax;
 	uint64_t interrupt, errorcode;
 	uint64_t rip, cs, rflags, user_rsp, ss;
 };
 
+/// Per-cpu state data. If you change this, remember to update the assembly
+/// version in 'tasking.inc'
+struct cpu_data
+{
+	uintptr_t rsp0;
+	uintptr_t rsp3;
+	process *tcb;
+};
+
+extern cpu_data bsp_cpu_data;
+
 class cpu_id
 {
 public:

src/kernel/crti.s

@@ -0,0 +1,14 @@
+section .init
+global _init:function
+_init:
+	push rbp
+	mov rbp, rsp
+	; Control flow falls through to other .init sections
+
+section .fini
+global _fini:function
+_fini:
+	push rbp
+	mov rbp, rsp
+	; Control flow falls through to other .fini sections
+

src/kernel/crtn.s

@@ -0,0 +1,10 @@
+section .init
+	; Control flow falls through to here from other .init sections
+	pop rbp
+	ret
+
+section .fini
+	; Control flow falls through to here from other .fini sections
+	pop rbp
+	ret
+

src/kernel/debug.cpp

@@ -2,54 +2,67 @@
 #include "cpu.h"
 #include "debug.h"
 #include "gdt.h"
+#include "page_manager.h"
 
-#define print_reg(name, value) cons->printf("         %s: %016lx\n", name, (value));
+size_t __counter_syscall_enter = 0;
+size_t __counter_syscall_sysret = 0;
 
 void
 print_regs(const cpu_state &regs)
 {
 	console *cons = console::get();
 
-	print_reg("rax", regs.rax);
-	print_reg("rbx", regs.rbx);
-	print_reg("rcx", regs.rcx);
-	print_reg("rdx", regs.rdx);
-	print_reg("rdi", regs.rdi);
-	print_reg("rsi", regs.rsi);
+	uint64_t cr2 = 0;
+	__asm__ __volatile__ ("mov %%cr2, %0" : "=r"(cr2));
+
+	print_regL("rax", regs.rax);
+	print_regM("rbx", regs.rbx);
+	print_regR("rcx", regs.rcx);
+	print_regL("rdx", regs.rdx);
+	print_regM("rdi", regs.rdi);
+	print_regR("rsi", regs.rsi);
 
 	cons->puts("\n");
-	print_reg(" r8", regs.r8);
-	print_reg(" r9", regs.r9);
-	print_reg("r10", regs.r10);
-	print_reg("r11", regs.r11);
-	print_reg("r12", regs.r12);
-	print_reg("r13", regs.r13);
-	print_reg("r14", regs.r14);
-	print_reg("r15", regs.r15);
+	print_regL(" r8", regs.r8);
+	print_regM(" r9", regs.r9);
+	print_regR("r10", regs.r10);
+	print_regL("r11", regs.r11);
+	print_regM("r12", regs.r12);
+	print_regR("r13", regs.r13);
+	print_regL("r14", regs.r14);
+	print_regM("r15", regs.r15);
+
+	cons->puts("\n\n");
+	print_regL("rbp", regs.rbp);
+	print_regM("rsp", regs.user_rsp);
+	print_regR("sp0", bsp_cpu_data.rsp0);
+
+	print_regL("rip", regs.rip);
+	print_regM("cr3", page_manager::get()->get_pml4());
+	print_regR("cr2", cr2);
 
 	cons->puts("\n");
-	print_reg("rbp", regs.rbp);
-	print_reg("rsp", regs.user_rsp);
-	print_reg("sp0", tss_get_stack(0));
-
-	cons->puts("\n");
-	print_reg(" ds", regs.ds);
-	print_reg(" cs", regs.cs);
-	print_reg(" ss", regs.ss);
-
-	cons->puts("\n");
-	print_reg("rip", regs.rip);
 }
 
+struct frame
+{
+	frame *prev;
+	uintptr_t return_addr;
+};
+
 void
 print_stacktrace(int skip)
 {
 	console *cons = console::get();
-	int frame = 0;
-	uint64_t bp = get_frame(skip);
-	while (bp) {
-		cons->printf("    frame %2d: %lx\n", frame, bp);
-		bp = get_frame(++frame + skip);
+
+	frame *fp = nullptr;
+	int fi = -skip;
+	__asm__ __volatile__ ( "mov %%rbp, %0" : "=r" (fp) );
+
+	while (fp && fp->return_addr) {
+		if (fi++ >= 0)
+			cons->printf("  frame %2d: %lx\n", fi-1, fp->return_addr);
+		fp = fp->prev;
 	}
 }
 

src/kernel/debug.h

@@ -8,12 +8,18 @@ extern "C" {
 	uintptr_t get_rsp();
 	uintptr_t get_rip();
 	uintptr_t get_frame(int frame);
-
+	uintptr_t get_gsbase();
+	void _halt();
 }
 
+extern size_t __counter_syscall_enter;
+extern size_t __counter_syscall_sysret;
+
 void print_regs(const cpu_state &regs);
 void print_stack(const cpu_state &regs);
-void print_stacktrace(int skip = 0);
+void print_stacktrace(int skip);
 
-#define print_reg(name, value) cons->printf("         %s: %016lx\n", name, (value));
+#define print_regL(name, value) cons->printf("       %s: %016lx", name, (value));
+#define print_regM(name, value) cons->printf("    %s: %016lx", name, (value));
+#define print_regR(name, value) cons->printf("    %s: %016lx\n", name, (value));
 

src/kernel/debug.s

@@ -8,24 +8,13 @@ get_rip:
 	pop rax ; do the same thing as 'ret', except with 'jmp'
 	jmp rax ; with the return address still in rax
 
+global get_gsbase
+get_gsbase:
+	rdgsbase rax
+	ret
+
 global _halt
 _halt:
 	hlt
 	jmp _halt
 
-
-global get_frame
-get_frame:
-	mov rcx, rbp
-
-.loop:
-	mov rax, [rcx + 8]
-	mov rcx, [rcx]
-	cmp rdi, 0
-	je .done
-
-	sub rdi, 1
-	jmp .loop
-
-.done:
-	ret

src/kernel/device_manager.cpp

@@ -14,7 +14,7 @@
 
 static const char expected_signature[] = "RSD PTR ";
 
-device_manager device_manager::s_instance(nullptr);
+device_manager device_manager::s_instance(nullptr, kutil::allocator::invalid);
 
 struct acpi1_rsdp
 {
@@ -59,8 +59,13 @@ void irq4_callback(void *)
 }
 
 
-device_manager::device_manager(const void *root_table) :
-	m_lapic(nullptr)
+device_manager::device_manager(const void *root_table, kutil::allocator &alloc) :
+	m_lapic(nullptr),
+	m_ioapics(alloc),
+	m_pci(alloc),
+	m_devices(alloc),
+	m_irqs(alloc),
+	m_blockdevs(alloc)
 {
 	kassert(root_table != 0, "ACPI root table pointer is null.");
 
@@ -93,7 +98,7 @@ device_manager::device_manager(const void *root_table) :
 ioapic *
 device_manager::get_ioapic(int i)
 {
-	return (i < m_ioapics.count()) ? m_ioapics[i] : nullptr;
+	return (i < m_ioapics.count()) ? &m_ioapics[i] : nullptr;
 }
 
 static void
@@ -148,26 +153,44 @@ device_manager::load_apic(const acpi_apic *apic)
 	uint8_t const *p = apic->controller_data;
 	uint8_t const *end = p + count;
 
-	// Pass one: set up IOAPIC objcts
+	// Pass one: count IOAPIC objcts
+	int num_ioapics = 0;
+	while (p < end) {
+		const uint8_t type = p[0];
+		const uint8_t length = p[1];
+		if (type == 1) num_ioapics++;
+		p += length;
+	}
+
+	m_ioapics.set_capacity(num_ioapics);
+
+	// Pass two: set up IOAPIC objcts
+	p = apic->controller_data;
 	while (p < end) {
 		const uint8_t type = p[0];
 		const uint8_t length = p[1];
 		if (type == 1) {
 			uint32_t *base = reinterpret_cast<uint32_t *>(kutil::read_from<uint32_t>(p+4));
 			uint32_t base_gsr = kutil::read_from<uint32_t>(p+8);
-			m_ioapics.append(new ioapic(base, base_gsr));
+			m_ioapics.emplace(base, base_gsr);
 		}
 		p += length;
 	}
 
-	// Pass two: configure APIC objects
+	// Pass three: configure APIC objects
 	p = apic->controller_data;
 	while (p < end) {
 		const uint8_t type = p[0];
 		const uint8_t length = p[1];
 
 		switch (type) {
-		case 0: // Local APIC
+		case 0: { // Local APIC
+				uint8_t uid = kutil::read_from<uint8_t>(p+2);
+				uint8_t id = kutil::read_from<uint8_t>(p+3);
+				log::debug(logs::device, "    Local APIC uid %x id %x", id);
+			}
+			break;
+
 		case 1: // I/O APIC
 			break;
 
@@ -180,7 +203,7 @@ device_manager::load_apic(const acpi_apic *apic)
 						source, gsi, (flags & 0x3), ((flags >> 2) & 0x3));
 
 				// TODO: in a multiple-IOAPIC system this might be elsewhere
-				m_ioapics[0]->redirect(source, static_cast<isr>(gsi), flags, true);
+				m_ioapics[0].redirect(source, static_cast<isr>(gsi), flags, true);
 			}
 			break;
 
@@ -206,10 +229,10 @@ device_manager::load_apic(const acpi_apic *apic)
 		p += length;
 	}
 
-	for (uint8_t i = 0; i < m_ioapics[0]->get_num_gsi(); ++i) {
+	for (uint8_t i = 0; i < m_ioapics[0].get_num_gsi(); ++i) {
 		switch (i) {
 			case 2: break;
-			default: m_ioapics[0]->mask(i, false);
+			default: m_ioapics[0].mask(i, false);
 		}
 	}
 

src/kernel/device_manager.h

@@ -2,14 +2,13 @@
 /// \file device_manager.h
 /// The device manager definition
 #include "kutil/vector.h"
+#include "apic.h"
 #include "pci.h"
 
 struct acpi_xsdt;
 struct acpi_apic;
 struct acpi_mcfg;
 class block_device;
-class lapic;
-class ioapic;
 
 using irq_callback = void (*)(void *);
 
@@ -19,8 +18,9 @@ class device_manager
 {
 public:
 	/// Constructor.
-	/// \arg root_table   Pointer to the ACPI RSDP
-	device_manager(const void *root_table);
+	/// \arg root_table  Pointer to the ACPI RSDP
+	/// \arg alloc       Allocator for device arrays
+	device_manager(const void *root_table, kutil::allocator &alloc);
 
 	/// Get the system global device manager.
 	/// \returns  A reference to the system device manager
@@ -105,7 +105,7 @@ private:
 	void bad_irq(uint8_t irq);
 
 	lapic *m_lapic;
-	kutil::vector<ioapic *> m_ioapics;
+	kutil::vector<ioapic> m_ioapics;
 
 	kutil::vector<pci_group> m_pci;
 	kutil::vector<pci_device> m_devices;

src/kernel/frame_allocator.cpp

@@ -0,0 +1,93 @@
+#include "kutil/assert.h"
+#include "kutil/memory.h"
+#include "frame_allocator.h"
+
+using memory::frame_size;
+using memory::page_offset;
+using frame_block_node = kutil::list_node<frame_block>;
+
+frame_allocator g_frame_allocator;
+
+int
+frame_block::compare(const frame_block *rhs) const
+{
+	if (address < rhs->address)
+		return -1;
+	else if (address > rhs->address)
+		return 1;
+	return 0;
+}
+
+
+frame_allocator::raw_alloc::raw_alloc(frame_allocator &fa) : m_fa(fa) {}
+
+void *
+frame_allocator::raw_alloc::allocate(size_t size)
+{
+	kassert(size <= frame_size, "Raw allocator only allocates a single page");
+
+	uintptr_t addr = 0;
+	if (size <= frame_size)
+		m_fa.allocate(1, &addr);
+	return reinterpret_cast<void*>(addr + page_offset);
+}
+
+void
+frame_allocator::raw_alloc::free(void *p)
+{
+	m_fa.free(reinterpret_cast<uintptr_t>(p), 1);
+}
+
+
+frame_allocator::frame_allocator() :
+	m_raw_alloc(*this)
+{
+}
+
+size_t
+frame_allocator::allocate(size_t count, uintptr_t *address)
+{
+	kassert(!m_free.empty(), "frame_allocator::pop_frames ran out of free frames!");
+	if (m_free.empty())
+		return 0;
+
+	auto *first = m_free.front();
+
+	if (count >= first->count) {
+		*address = first->address;
+		m_free.remove(first);
+		return first->count;
+	} else {
+		first->count -= count;
+		*address = first->address + (first->count * frame_size);
+		return count;
+	}
+}
+
+inline uintptr_t end(frame_block *node) { return node->address + node->count * frame_size; }
+
+void
+frame_allocator::free(uintptr_t address, size_t count)
+{
+	frame_block_node *node =
+		reinterpret_cast<frame_block_node*>(address + page_offset);
+
+	kutil::memset(node, 0, sizeof(frame_block_node));
+	node->address = address;
+	node->count = count;
+
+	m_free.sorted_insert(node);
+
+	frame_block_node *next = node->next();
+	if (next && end(node) == next->address) {
+		node->count += next->count;
+		m_free.remove(next);
+	}
+
+	frame_block_node *prev = node->prev();
+	if (prev && end(prev) == address) {
+		prev->count += node->count;
+		m_free.remove(node);
+	}
+}
+

src/kernel/frame_allocator.h

@@ -0,0 +1,70 @@
+#pragma once
+/// \file frame_allocator.h
+/// Allocator for physical memory frames
+
+#include <stdint.h>
+
+#include "kutil/allocator.h"
+#include "kutil/linked_list.h"
+
+struct frame_block;
+using frame_block_list = kutil::linked_list<frame_block>;
+
+/// Allocator for physical memory frames
+class frame_allocator
+{
+public:
+	/// Default constructor
+	frame_allocator();
+
+	/// Get free frames from the free list. Only frames from the first free block
+	/// are returned, so the number may be less than requested, but they will
+	/// be contiguous.
+	/// \arg count    The maximum number of frames to get
+	/// \arg address  [out] The physical address of the first frame
+	/// \returns      The number of frames retrieved
+	size_t allocate(size_t count, uintptr_t *address);
+
+	/// Free previously allocated frames.
+	/// \arg address  The physical address of the first frame to free
+	/// \arg count    The number of frames to be freed
+	void free(uintptr_t address, size_t count);
+
+	/// Get a memory allocator that allocates raw pages
+	/// \returns  The allocator ojbect
+	kutil::allocator & raw_allocator() { return m_raw_alloc; }
+
+private:
+	class raw_alloc :
+		public kutil::allocator
+	{
+	public:
+		raw_alloc(frame_allocator &fa);
+		virtual void * allocate(size_t size) override;
+		virtual void free(void *p) override;
+	private:
+		frame_allocator &m_fa;
+	};
+
+	raw_alloc m_raw_alloc;
+	frame_block_list m_free; ///< Free frames list
+
+	frame_allocator(const frame_allocator &) = delete;
+};
+
+
+/// A block of contiguous frames. Each `frame_block` represents contiguous
+/// physical frames with the same attributes.
+struct frame_block
+{
+	uintptr_t address;
+	uint32_t count;
+
+	/// Compare two blocks by address.
+	/// \arg rhs   The right-hand comparator
+	/// \returns   <0 if this is sorts earlier, >0 if this sorts later, 0 for equal
+	int compare(const frame_block *rhs) const;
+};
+
+
+extern frame_allocator g_frame_allocator;

src/kernel/gdt.cpp

@@ -184,18 +184,25 @@ gdt_init()
 }
 
 void
-gdt_dump()
+gdt_dump(int index)
 {
 	const table_ptr &table = g_gdtr;
 
 	console *cons = console::get();
-	cons->printf("         GDT: loc:%lx size:%d\n", table.base, table.limit+1);
 
+	int start = 0;
 	int count = (table.limit + 1) / sizeof(gdt_descriptor);
+	if (index != -1) {
+		start = index;
+		count = 1;
+	} else {
+		cons->printf("         GDT: loc:%lx size:%d\n", table.base, table.limit+1);
+	}
+
 	const gdt_descriptor *gdt =
 		reinterpret_cast<const gdt_descriptor *>(table.base);
 
-	for (int i = 0; i < count; ++i) {
+	for (int i = start; i < start+count; ++i) {
 		uint32_t base =
 			(gdt[i].base_high << 24) |
 			(gdt[i].base_mid << 16) |
@@ -233,17 +240,25 @@ gdt_dump()
 }
 
 void
-idt_dump()
+idt_dump(int index)
 {
 	const table_ptr &table = g_idtr;
 
-	log::info(logs::boot, "Loaded IDT at: %lx size: %d bytes", table.base, table.limit+1);
 
+	int start = 0;
 	int count = (table.limit + 1) / sizeof(idt_descriptor);
+	if (index != -1) {
+		start = index;
+		count = 1;
+		log::info(logs::boot, "IDT FOR INDEX %02x", index);
+	} else {
+		log::info(logs::boot, "Loaded IDT at: %lx size: %d bytes", table.base, table.limit+1);
+	}
+
 	const idt_descriptor *idt =
 		reinterpret_cast<const idt_descriptor *>(table.base);
 
-	for (int i = 0; i < count; ++i) {
+	for (int i = start; i < start+count; ++i) {
 		uint64_t base =
 			(static_cast<uint64_t>(idt[i].base_high) << 32) |
 			(static_cast<uint64_t>(idt[i].base_mid)  << 16) |

src/kernel/gdt.h

@@ -25,7 +25,9 @@ void tss_set_stack(int ring, uintptr_t rsp);
 uintptr_t tss_get_stack(int ring);
 
 /// Dump information about the current GDT to the screen
-void gdt_dump();
+/// \arg index  Which entry to print, or -1 for all entries
+void gdt_dump(int index = -1);
 
 /// Dump information about the current IDT to the screen
-void idt_dump();
+/// \arg index  Which entry to print, or -1 for all entries
+void idt_dump(int index = -1);

src/kernel/gdt.s

@@ -25,7 +25,7 @@ gdt_write:
 	push rax
 	o64 retf
 .next:
-	ltr dx
+	ltr dx ; third arg is the TSS
 	ret
 
 global gdt_load

src/kernel/interrupts.cpp

@@ -18,9 +18,8 @@ static const uint16_t PIC2 = 0xa0;
 extern "C" {
 	void _halt();
 
-	uintptr_t isr_handler(uintptr_t, cpu_state);
-	uintptr_t irq_handler(uintptr_t, cpu_state);
-	uintptr_t syscall_handler(uintptr_t, cpu_state);
+	void isr_handler(cpu_state*);
+	void irq_handler(cpu_state*);
 
 #define ISR(i, name)     extern void name ();
 #define EISR(i, name)    extern void name ();
@@ -104,33 +103,65 @@ interrupts_init()
 	log::info(logs::boot, "Interrupts enabled.");
 }
 
-uintptr_t
-isr_handler(uintptr_t return_rsp, cpu_state regs)
+void
+isr_handler(cpu_state *regs)
 {
 	console *cons = console::get();
 
-	switch (static_cast<isr>(regs.interrupt & 0xff)) {
+	switch (static_cast<isr>(regs->interrupt & 0xff)) {
+
+	case isr::isrDebug: {
+			cons->set_color(11);
+			cons->puts("\nDebug Exception:\n");
+			cons->set_color();
+
+			uint64_t dr = 0;
+
+			__asm__ __volatile__ ("mov %%dr0, %0" : "=r"(dr));
+			print_regL("dr0", dr);
+
+			__asm__ __volatile__ ("mov %%dr1, %0" : "=r"(dr));
+			print_regM("dr1", dr);
+
+			__asm__ __volatile__ ("mov %%dr2, %0" : "=r"(dr));
+			print_regM("dr2", dr);
+
+			__asm__ __volatile__ ("mov %%dr3, %0" : "=r"(dr));
+			print_regR("dr3", dr);
+
+			__asm__ __volatile__ ("mov %%dr6, %0" : "=r"(dr));
+			print_regL("dr6", dr);
+
+			__asm__ __volatile__ ("mov %%dr7, %0" : "=r"(dr));
+			print_regR("dr7", dr);
+
+			print_regL("rip", regs->rip);
+			print_regM("rsp", regs->user_rsp);
+			print_regM("fla", regs->rflags);
+			_halt();
+		}
+		break;
 
 	case isr::isrGPFault: {
 			cons->set_color(9);
 			cons->puts("\nGeneral Protection Fault:\n");
 			cons->set_color();
 
-			cons->printf("   errorcode: %lx", regs.errorcode);
-			if (regs.errorcode & 0x01) cons->puts(" external");
+			cons->printf("   errorcode: %lx", regs->errorcode);
+			if (regs->errorcode & 0x01) cons->puts(" external");
 
-			int index = (regs.errorcode & 0xffff) >> 4;
+			int index = (regs->errorcode & 0xffff) >> 4;
 			if (index) {
-				switch ((regs.errorcode & 0x07) >> 1) {
+				switch ((regs->errorcode & 0x07) >> 1) {
 				case 0:
 					cons->printf(" GDT[%x]\n", index);
-					gdt_dump();
+					gdt_dump(index);
 					break;
 
 				case 1:
 				case 3:
 					cons->printf(" IDT[%x]\n", index);
-					idt_dump();
+					idt_dump(index);
 					break;
 
 				default:
@@ -140,46 +171,40 @@ isr_handler(uintptr_t return_rsp, cpu_state regs)
 			} else {
 				cons->putc('\n');
 			}
-			print_regs(regs);
+			print_regs(*regs);
 			/*
 			print_stacktrace(2);
-			print_stack(regs);
+			print_stack(*regs);
 			*/
-
 		}
 		_halt();
 		break;
 
 	case isr::isrPageFault: {
-			cons->set_color(11);
-			cons->puts("\nPage Fault:\n");
-			cons->set_color();
-
-			cons->puts("       flags:");
-			if (regs.errorcode & 0x01) cons->puts(" present");
-			if (regs.errorcode & 0x02) cons->puts(" write");
-			if (regs.errorcode & 0x04) cons->puts(" user");
-			if (regs.errorcode & 0x08) cons->puts(" reserved");
-			if (regs.errorcode & 0x10) cons->puts(" ip");
-			cons->puts("\n");
-
-			uint64_t cr2 = 0;
+			uintptr_t cr2 = 0;
 			__asm__ __volatile__ ("mov %%cr2, %0" : "=r"(cr2));
-			print_reg("cr2", cr2);
 
-			print_reg("rsp", regs.user_rsp);
-			print_reg("rip", regs.rip);
+			if (!page_manager::get()->fault_handler(cr2)) {
+				cons->set_color(11);
+				cons->puts("\nPage Fault:\n");
+				cons->set_color();
 
-			cons->puts("\n");
-			print_stacktrace(2);
+				cons->puts("       flags:");
+				if (regs->errorcode & 0x01) cons->puts(" present");
+				if (regs->errorcode & 0x02) cons->puts(" write");
+				if (regs->errorcode & 0x04) cons->puts(" user");
+				if (regs->errorcode & 0x08) cons->puts(" reserved");
+				if (regs->errorcode & 0x10) cons->puts(" ip");
+				cons->puts("\n");
+				print_regs(*regs);
+				print_stacktrace(2);
+				_halt();
+			}
 		}
-		_halt();
 		break;
 
-	case isr::isrTimer: {
-			scheduler &s = scheduler::get();
-			return_rsp = s.tick(return_rsp);
-		}
+	case isr::isrTimer:
+		scheduler::get().tick();
 		break;
 
 	case isr::isrLINT0:
@@ -192,20 +217,21 @@ isr_handler(uintptr_t return_rsp, cpu_state regs)
 
 	case isr::isrAssert: {
 			cons->set_color();
-			print_regs(regs);
+			print_regs(*regs);
 			print_stacktrace(2);
 		}
 		_halt();
 		break;
 
-	case isr::isrSyscall: {
-			return_rsp = syscall_dispatch(return_rsp, regs);
-		}
+	/*
+	case isr::isrSyscall:
+		syscall_dispatch(regs);
 		break;
+	*/
 
 	case isr::isrSpurious:
 		// No EOI for the spurious interrupt
-		return return_rsp;
+		return;
 
 	case isr::isrIgnore0:
 	case isr::isrIgnore1:
@@ -215,7 +241,7 @@ isr_handler(uintptr_t return_rsp, cpu_state regs)
 	case isr::isrIgnore5:
 	case isr::isrIgnore6:
 	case isr::isrIgnore7:
-		//cons->printf("\nIGNORED: %02x\n", regs.interrupt);
+		//cons->printf("\nIGNORED: %02x\n", regs->interrupt);
 		outb(PIC1, 0x20);
 		break;
 
@@ -227,7 +253,7 @@ isr_handler(uintptr_t return_rsp, cpu_state regs)
 	case isr::isrIgnoreD:
 	case isr::isrIgnoreE:
 	case isr::isrIgnoreF:
-		//cons->printf("\nIGNORED: %02x\n", regs.interrupt);
+		//cons->printf("\nIGNORED: %02x\n", regs->interrupt);
 		outb(PIC1, 0x20);
 		outb(PIC2, 0x20);
 		break;
@@ -235,41 +261,32 @@ isr_handler(uintptr_t return_rsp, cpu_state regs)
 	default:
 		cons->set_color(9);
 		cons->printf("\nReceived %02x interrupt:\n",
-				(static_cast<isr>(regs.interrupt)));
+				(static_cast<isr>(regs->interrupt)));
 
 		cons->set_color();
 		cons->printf("         ISR: %02lx  ERR: %lx\n\n",
-				regs.interrupt, regs.errorcode);
+				regs->interrupt, regs->errorcode);
 
-		print_regs(regs);
-		//print_stacktrace(2);
+		print_regs(*regs);
+		print_stacktrace(2);
 		_halt();
 	}
 	*reinterpret_cast<uint32_t *>(0xffffff80fee000b0) = 0;
-
-	return return_rsp;
 }
 
-uintptr_t
-irq_handler(uintptr_t return_rsp, cpu_state regs)
+void
+irq_handler(cpu_state *regs)
 {
 	console *cons = console::get();
-	uint8_t irq = get_irq(regs.interrupt);
+	uint8_t irq = get_irq(regs->interrupt);
 	if (! device_manager::get().dispatch_irq(irq)) {
 		cons->set_color(11);
 		cons->printf("\nReceived unknown IRQ: %d (vec %d)\n",
-				irq, regs.interrupt);
+				irq, regs->interrupt);
 		cons->set_color();
-		print_regs(regs);
+		print_regs(*regs);
 		_halt();
 	}
 
 	*reinterpret_cast<uint32_t *>(0xffffff80fee000b0) = 0;
-	return return_rsp;
-}
-
-uintptr_t
-syscall_handler(uintptr_t return_rsp, cpu_state regs)
-{
-	return syscall_dispatch(return_rsp, regs);
 }

src/kernel/interrupts.s

@@ -3,37 +3,36 @@
 extern isr_handler
 global isr_handler_prelude
 isr_handler_prelude:
-	push_all_and_segments
+	push_all
+	check_swap_gs
 
 	mov rdi, rsp
+	mov rsi, rsp
 	call isr_handler
-	mov rsp, rax
-
-	pop_all_and_segments
-
-	add rsp, 16		; because the ISRs added err/num
-	sti
-	iretq
+	jmp isr_handler_return
 
 extern irq_handler
 global irq_handler_prelude
 irq_handler_prelude:
-	push_all_and_segments
+	push_all
+	check_swap_gs
 
 	mov rdi, rsp
+	mov rsi, rsp
 	call irq_handler
-	mov rsp, rax
+	; fall through to isr_handler_return
 
-	pop_all_and_segments
+global isr_handler_return
+isr_handler_return:
+	check_swap_gs
+	pop_all
 
 	add rsp, 16		; because the ISRs added err/num
-	sti
 	iretq
 
 %macro EMIT_ISR 2
 	global %1
 	%1:
-		cli
 		push 0
 		push %2
 		jmp isr_handler_prelude
@@ -42,7 +41,6 @@ irq_handler_prelude:
 %macro EMIT_EISR 2
 	global %1
 	%1:
-		cli
 		push %2
 		jmp isr_handler_prelude
 %endmacro
@@ -50,14 +48,13 @@ irq_handler_prelude:
 %macro EMIT_IRQ 2
 	global %1
 	%1:
-		cli
 		push 0
 		push %2
 		jmp irq_handler_prelude
 %endmacro
 
-%define EISR(i, name)     EMIT_EISR name, i
-%define UISR(i, name)     EMIT_ISR name, i
+%define EISR(i, name)     EMIT_EISR name, i   ; ISR with error code
+%define UISR(i, name)     EMIT_ISR name, i    ; ISR callable from user space
 %define  ISR(i, name)     EMIT_ISR name, i
 %define  IRQ(i, q, name)  EMIT_IRQ name, i
 

src/kernel/loader.s

@@ -1,25 +1,22 @@
 %include "push_all.inc"
 
-extern load_process
+extern load_process_image
 
 global ramdisk_process_loader
 ramdisk_process_loader:
 
 	; create_process already pushed a cpu_state onto the stack for us, this
-	; acts both as the cpu_state parameter to load_process, and the saved
-	; state for the following iretq
-	;
-	; Additional parameters:
-	;  rax - the address of the program image
-	;  rbx - the size of the program image
-	;  rcx - the address of this process' process structure
-	mov rdi, rax
-	mov rsi, rbx
-	mov rdx, rcx
-	call load_process
+	; acts both as the cpu_state parameter to load_process_image, and the
+	; saved state for the following iretq
 
-	pop_all_and_segments
-	add rsp, 16		; because the ISRs add err/num
+	pop rdi ; the address of the program image
+	pop rsi ; the size of the program image
+	pop rdx ; the address of this process' process structure
+
+	call load_process_image
+
+	push rax ; load_process_image returns the process entrypoint
+
+	swapgs
 	iretq
 
-

src/kernel/log.cpp

@@ -1,106 +1,55 @@
-#include <type_traits>
-#include "kutil/assert.h"
 #include "kutil/memory.h"
 #include "console.h"
 #include "log.h"
+#include "scheduler.h"
 
 
-static const uint64_t default_enabled[] = {0x00, 0xff, 0xff, 0xff};
-static const uint8_t level_colors[] = {0x07, 0x0f, 0x0b, 0x09};
+static uint8_t log_buffer[0x10000];
+static log::logger g_logger(log_buffer, sizeof(log_buffer));
 
-static const char *levels[] = {"debug", " info", " warn", "error", "fatal"};
-static const char *areas[] = {
-	"boot  ",
-	"memory",
-	"apic  ",
-	"device",
-	"driver",
-	"file  ",
-	"task  ",
+static const uint8_t level_colors[] = {0x07, 0x07, 0x0f, 0x0b, 0x09};
 
-	nullptr
-};
-
-
-log log::s_log;
-
-log::log() : m_cons(nullptr)
+static void
+output_log(log::area_t area, log::level severity, const char *message)
 {
-	kassert(0, "Invalid log constructor");
-}
-
-log::log(console *cons) :
-	m_cons(cons)
-{
-	const int num_levels = static_cast<int>(level::max) - 1;
-	for (int i = 0; i < num_levels; ++i)
-		m_enabled[i] = default_enabled[i];
+	auto *cons = console::get();
+	cons->set_color(level_colors[static_cast<int>(severity)]);
+	cons->printf("%7s %5s: %s\n",
+			g_logger.area_name(area),
+			g_logger.level_name(severity),
+			message);
+	cons->set_color();
 }
 
 void
-log::init(console *cons)
+logger_task()
 {
-	new (&s_log) log(cons);
-	log::info(logs::boot, "Logging system initialized.");
-}
+	uint8_t buffer[257];
+	auto *ent = reinterpret_cast<log::logger::entry *>(buffer);
+	auto *cons = console::get();
 
-static inline uint64_t
-bit_mask(logs area) { return 1 << static_cast<uint64_t>(area); }
+	g_logger.set_immediate(nullptr);
+	log::info(logs::task, "Starting kernel logger task");
 
-void
-log::enable(logs type, level at_level)
-{
-	using under_t = std::underlying_type<level>::type;
-	under_t at = static_cast<under_t>(at_level);
-	under_t max = sizeof(m_enabled) / sizeof(m_enabled[0]);
+	scheduler &s = scheduler::get();
 
-	for (under_t i = 0; i < max; ++i) {
-		if (i >= at)
-			s_log.m_enabled[i] |= bit_mask(type);
-		else
-			s_log.m_enabled[i] &= ~bit_mask(type);
+	while (true) {
+		if(g_logger.get_entry(buffer, sizeof(buffer))) {
+			buffer[ent->bytes] = 0;
+			cons->set_color(level_colors[static_cast<int>(ent->severity)]);
+			cons->printf("%7s %5s: %s\n",
+				g_logger.area_name(ent->area),
+				g_logger.level_name(ent->severity),
+				ent->message);
+			cons->set_color();
+		} else {
+			s.schedule();
+		}
 	}
 }
 
-template <>
-void
-log::trylog<log::level::fatal>(logs area, const char *fmt, ...)
+void logger_init()
 {
-	va_list args;
-	va_start(args, fmt);
-	s_log.output(level::fatal, area, fmt, args);
-	va_end(args);
-	while(1) __asm__ ("hlt");
+	new (&g_logger) log::logger(log_buffer, sizeof(log_buffer));
+	g_logger.set_immediate(output_log);
 }
-
-template <log::level L>
-void
-log::trylog(logs area, const char *fmt, ...)
-{
-	auto i = static_cast<std::underlying_type<level>::type>(L);
-	if ((s_log.m_enabled[i] & bit_mask(area)) == 0) return;
-
-	va_list args;
-	va_start(args, fmt);
-	s_log.output(L, area, fmt, args);
-	va_end(args);
-}
-
-void
-log::output(level severity, logs area, const char *fmt, va_list args)
-{
-	m_cons->set_color(level_colors[static_cast<int>(severity)]);
-
-	m_cons->printf("%s %s: ",
-			areas[static_cast<int>(area)],
-			levels[static_cast<int>(severity)]);
-	m_cons->vprintf(fmt, args);
-	m_cons->set_color();
-	m_cons->puts("\n");
-}
-
-const log::trylog_p log::debug = &trylog<level::debug>;
-const log::trylog_p log::info = &trylog<level::info>;
-const log::trylog_p log::warn = &trylog<level::warn>;
-const log::trylog_p log::error = &trylog<level::error>;
-const log::trylog_p log::fatal = &trylog<level::fatal>;

src/kernel/log.h

@@ -1,51 +1,9 @@
 #pragma once
-#include <stdarg.h>
-#include <stdint.h>
 
-class console;
+#include "kutil/logger.h"
 
+namespace log = kutil::log;
+namespace logs = kutil::logs;
 
-enum class logs
-{
-	boot,
-	memory,
-	apic,
-	device,
-	driver,
-	fs,
-	task,
-
-	max
-};
-
-
-class log
-{
-public:
-	enum class level {debug, info, warn, error, fatal, max};
-
-	static void init(console *cons);
-	static void enable(logs type, level at_level);
-
-	template <level L>
-	static void trylog(logs area, const char *fmt, ...);
-	using trylog_p = void (*)(logs area, const char *fmt, ...);
-
-	static const trylog_p debug;
-	static const trylog_p info;
-	static const trylog_p warn;
-	static const trylog_p error;
-	static const trylog_p fatal;
-
-private:
-	void output(level severity, logs area, const char *fmt, va_list args);
-
-	/// Bitmasks for what categories are enabled. fatal is
-	/// always enabled, so leave it out.
-	uint64_t m_enabled[static_cast<uint64_t>(level::max) - 1];
-	console *m_cons;
-
-	log();
-	log(console *cons);
-	static log s_log;
-};
+void logger_init();
+void logger_task();

src/kernel/main.cpp

@@ -11,16 +11,15 @@
 #include "gdt.h"
 #include "interrupts.h"
 #include "io.h"
-#include "kernel_data.h"
+#include "kernel_args.h"
 #include "log.h"
 #include "page_manager.h"
 #include "scheduler.h"
-#include "screen.h"
 #include "serial.h"
 #include "syscall.h"
 
 extern "C" {
-	void kernel_main(popcorn_data *header);
+	void kernel_main(kernel_args *header);
 	void *__bss_start, *__bss_end;
 }
 
@@ -37,43 +36,31 @@ init_console()
 	cons->set_color(0x08, 0x00);
 	cons->puts(GIT_VERSION " booting...\n");
 
-	log::init(cons);
-	log::enable(logs::apic, log::level::debug);
-	log::enable(logs::device, log::level::info);
-	log::enable(logs::driver, log::level::debug);
-	log::enable(logs::memory, log::level::info);
-	log::enable(logs::fs, log::level::debug);
-	log::enable(logs::task, log::level::debug);
+	logger_init();
 }
 
 void
-kernel_main(popcorn_data *header)
+kernel_main(kernel_args *header)
 {
-#ifdef DEBUG
-	// Run `waf configure --debug` to enable compiling with DEBUG turned on.
-	// Then attach to QEMU's gdb server and `set waiting = false` to start
-	// the kernel. This compensates for GDB's poor handling of QEMU going
-	// through the x86 PC startup and switching to 64 bit mode when you
-	// attach to qemu with the -S option.
-	bool waiting = true;
+	bool waiting = header && (header->flags && POPCORN_FLAG_DEBUG);
 	while (waiting);
-#endif
 
 	kutil::assert_set_callback(__kernel_assert);
 
 	gdt_init();
 	interrupts_init();
 
-	page_manager *pager = new (&g_page_manager) page_manager;
-
-	memory_initialize(
+	kutil::allocator &heap = memory_initialize(
+			header->scratch_pages,
 			header->memory_map,
 			header->memory_map_length,
 			header->memory_map_desc_size);
 
-	pager->map_offset_pointer(
-			&header->frame_buffer,
-			header->frame_buffer_length);
+	if (header->frame_buffer && header->frame_buffer_length) {
+		page_manager::get()->map_offset_pointer(
+				&header->frame_buffer,
+				header->frame_buffer_length);
+	}
 
 	init_console();
 
@@ -84,18 +71,18 @@ kernel_main(popcorn_data *header)
 	log::debug(logs::boot, "ACPI root table is at: %016lx", header->acpi_table);
 	log::debug(logs::boot, "Runtime service is at: %016lx", header->runtime);
 
-	initrd::disk ird(header->initrd);
+	initrd::disk ird(header->initrd, heap);
 	log::info(logs::boot, "initrd loaded with %d files.", ird.files().count());
 	for (auto &f : ird.files())
 		log::info(logs::boot, "  %s%s (%d bytes).", f.executable() ? "*" : "", f.name(), f.size());
 
 	/*
-	pager->dump_pml4(nullptr, 0);
-	pager->dump_blocks(true);
+	   page_manager::get()->dump_pml4(nullptr, 0);
+	   page_manager::get()->dump_blocks(true);
 	*/
 
 	device_manager *devices =
-		new (&device_manager::get()) device_manager(header->acpi_table);
+		new (&device_manager::get()) device_manager(header->acpi_table, heap);
 
 	interrupts_enable();
 
@@ -142,14 +129,14 @@ kernel_main(popcorn_data *header)
 	devices->get_lapic()->calibrate_timer();
 
 	syscall_enable();
-	scheduler *sched = new (&scheduler::get()) scheduler(devices->get_lapic());
+	scheduler *sched = new (&scheduler::get()) scheduler(devices->get_lapic(), heap);
+
+	sched->create_kernel_task(-1, logger_task);
 
-	/*
 	for (auto &f : ird.files()) {
 		if (f.executable())
-			sched->create_process(f.name(), f.data(), f.size());
+			sched->load_process(f.name(), f.data(), f.size());
 	}
-	*/
 
 	sched->start();
 }

src/kernel/memory_bootstrap.cpp

@@ -1,32 +1,27 @@
+#include <algorithm>
 #include <utility>
+#include "kutil/address_manager.h"
 #include "kutil/assert.h"
-#include "kutil/linked_list.h"
-#include "kutil/slab_allocator.h"
+#include "kutil/heap_allocator.h"
+#include "frame_allocator.h"
 #include "io.h"
+#include "log.h"
 #include "page_manager.h"
 
-const unsigned efi_page_size = 0x1000;
-const unsigned ident_page_flags = 0xb;
+using memory::frame_size;
+using memory::kernel_max_heap;
+using memory::kernel_offset;
+using memory::page_offset;
 
-namespace {
-	// Page-by-page initial allocator for the initial page_block allocator
-	struct page_consumer
-	{
-		page_consumer(uintptr_t start) : current(start) {}
+static const unsigned ident_page_flags = 0xb;
 
-		void * operator()(size_t size) {
-			kassert(size == page_manager::page_size, "page_consumer used with non-page size!");
-			void *retval = reinterpret_cast<void *>(current);
-			current += size;
-			return retval;
-		}
+kutil::address_manager g_kernel_address_manager;
+kutil::heap_allocator g_kernel_heap;
 
-		uintptr_t current;
-	};
-}
-
-using block_list = kutil::linked_list<page_block>;
-using block_allocator = kutil::slab_allocator<page_block, page_consumer &>;
+void * operator new(size_t size)           { return g_kernel_heap.allocate(size); }
+void * operator new [] (size_t size)       { return g_kernel_heap.allocate(size); }
+void operator delete (void *p) noexcept    { return g_kernel_heap.free(p); }
+void operator delete [] (void *p) noexcept { return g_kernel_heap.free(p); }
 
 enum class efi_memory_type : uint32_t
 {
@@ -49,66 +44,13 @@ enum class efi_memory_type : uint32_t
 	efi_max,
 
 	popcorn_kernel = 0x80000000,
-	popcorn_font,
 	popcorn_data,
-	popcorn_log,
-	popcorn_pml4,
+	popcorn_initrd,
+	popcorn_scratch,
 
 	popcorn_max
 };
 
-const char *efi_memory_type_names[] = {
-	"             reserved",
-	"          loader_code",
-	"          loader_data",
-	"   boot_services_code",
-	"   boot_services_data",
-	"runtime_services_code",
-	"runtime_services_data",
-	"            available",
-	"             unusable",
-	"         acpi_reclaim",
-	"             acpi_nvs",
-	"                 mmio",
-	"            mmio_port",
-	"             pal_code",
-
-	"       popcorn_kernel",
-	"         popcorn_font",
-	"         popcorn_data",
-	"          popcorn_log",
-	"         popcorn_pml4",
-};
-
-static const char *
-get_efi_name(efi_memory_type t)
-{
-	static const unsigned offset =
-		(unsigned)efi_memory_type::popcorn_kernel - (unsigned)efi_memory_type::efi_max;
-
-	return t >= efi_memory_type::popcorn_kernel ?
-		efi_memory_type_names[(unsigned)t - offset] :
-		efi_memory_type_names[(unsigned)t];
-}
-
-enum class efi_memory_flag : uint64_t
-{
-	can_mark_uc  = 0x0000000000000001, // uc = un-cacheable
-	can_mark_wc  = 0x0000000000000002, // wc = write-combining
-	can_mark_wt  = 0x0000000000000004, // wt = write through
-	can_mark_wb  = 0x0000000000000008, // wb = write back
-	can_mark_uce = 0x0000000000000010, // uce = un-cacheable exported
-	can_mark_wp  = 0x0000000000001000, // wp = write protected
-	can_mark_rp  = 0x0000000000002000, // rp = read protected
-	can_mark_xp  = 0x0000000000004000, // xp = exceute protected
-	can_mark_ro  = 0x0000000000020000, // ro = read only
-
-	non_volatile  = 0x0000000000008000,
-	more_reliable = 0x0000000000010000,
-	runtime       = 0x8000000000000000
-};
-IS_BITFIELD(efi_memory_flag);
-
 struct efi_memory_descriptor
 {
 	efi_memory_type type;
@@ -116,388 +58,185 @@ struct efi_memory_descriptor
 	uint64_t physical_start;
 	uint64_t virtual_start;
 	uint64_t pages;
-	efi_memory_flag flags;
+	uint64_t flags;
 };
 
-static const efi_memory_descriptor *
-desc_incr(const efi_memory_descriptor *d, size_t desc_length)
+struct memory_map
 {
-	return reinterpret_cast<const efi_memory_descriptor *>(
-			reinterpret_cast<const uint8_t *>(d) + desc_length);
-}
+	memory_map(const void *efi_map, size_t map_length, size_t desc_length) :
+		efi_map(efi_map), map_length(map_length), desc_length(desc_length) {}
 
-page_block_list::item_type *
-remove_block_for(page_block_list &list, uintptr_t phys_start, size_t pages, page_block_list &cache)
-{
-	// This is basically just the removal portion of page_manager::unmap_pages,
-	// but with physical addresses, and only ever removing a single block.
+	class iterator
+	{
+	public:
+		iterator(const memory_map &map, efi_memory_descriptor const *item) :
+			map(map), item(item) {}
 
-	for (auto *item : list) {
-		if (!item->contains_physical(phys_start))
-			continue;
-
-		uint64_t size = page_manager::page_size * pages;
-		uint64_t end = phys_start + size;
-		uint64_t leading = phys_start - item->physical_address;
-		uint64_t trailing = item->physical_end() - end;
-
-		if (leading) {
-			uint64_t pages = leading / page_manager::page_size;
-
-			page_block_list::item_type *lead_block = cache.pop_front();
-
-			lead_block->copy(item);
-			lead_block->count = pages;
-
-			item->count -= pages;
-			item->physical_address += leading;
-
-			if (item->virtual_address)
-				item->virtual_address += leading;
-
-			list.insert_before(item, lead_block);
+		inline efi_memory_descriptor const * operator*() const { return item; }
+		inline bool operator!=(const iterator &other) { return item != other.item; }
+		inline iterator & operator++() {
+			item = kutil::offset_pointer(item, map.desc_length);
+			return *this;
 		}
 
-		if (trailing) {
-			uint64_t pages = trailing / page_manager::page_size;
+	private:
+		const memory_map &map;
+		efi_memory_descriptor const *item;
+	};
 
-			page_block_list::item_type *trail_block = cache.pop_front();
-
-			trail_block->copy(item);
-			trail_block->count = pages;
-			trail_block->physical_address += size;
-
-			item->count -= pages;
-
-			if (item->virtual_address)
-				trail_block->virtual_address += size;
-
-			list.insert_before(item, trail_block);
-		}
-
-		list.remove(item);
-		return item;
+	iterator begin() const {
+		return iterator(*this, reinterpret_cast<efi_memory_descriptor const *>(efi_map));
 	}
 
-	kassert(false, "Couldn't find block to remove");
-	return nullptr;
-}
-
-void
-gather_block_lists(
-		block_allocator &allocator,
-		block_list &used,
-		block_list &free,
-		const void *memory_map,
-		size_t map_length,
-		size_t desc_length)
-{
-	efi_memory_descriptor const *desc = reinterpret_cast<efi_memory_descriptor const *>(memory_map);
-	efi_memory_descriptor const *end = desc_incr(desc, map_length);
-
-	while (desc < end) {
-		auto *block = allocator.pop();
-		block->physical_address = desc->physical_start;
-		block->virtual_address = desc->virtual_start;
-		block->count = desc->pages;
-
-		switch (desc->type) {
-		case efi_memory_type::loader_code:
-		case efi_memory_type::loader_data:
-			block->flags = page_block_flags::used | page_block_flags::pending_free;
-			break;
-
-		case efi_memory_type::boot_services_code:
-		case efi_memory_type::boot_services_data:
-		case efi_memory_type::available:
-			block->flags = page_block_flags::free;
-			break;
-
-		case efi_memory_type::acpi_reclaim:
-			block->flags =
-				page_block_flags::used |
-				page_block_flags::mapped |
-				page_block_flags::acpi_wait;
-
-			block->virtual_address = block->physical_address;
-			break;
-
-		case efi_memory_type::persistent:
-			block->flags = page_block_flags::nonvolatile;
-			break;
-
-		default:
-			block->flags = page_block_flags::used | page_block_flags::permanent;
-			break;
-		}
-
-		if (block->has_flag(page_block_flags::used)) {
-			if (block->virtual_address || !block->physical_address)
-				block->flags |= page_block_flags::mapped;
-
-			used.push_back(block);
-		} else {
-			free.push_back(block);
-		}
-
-		desc = desc_incr(desc, desc_length);
-	}
-}
-
-void
-copy_new_table(page_table *base, unsigned index, page_table *new_table)
-{
-	uint64_t entry = base->entries[index];
-
-	// If this is a large page and not a a table, bail out.
-	if(entry & 0x80) return;
-
-	if (entry & 0x1) {
-		page_table *old_next = reinterpret_cast<page_table *>(
-				base->entries[index] & ~0xffful);
-		for (int i = 0; i < 512; ++i) new_table->entries[i] = old_next->entries[i];
-	} else {
-		for (int i = 0; i < 512; ++i) new_table->entries[i] = 0;
+	iterator end() const {
+		const void *end = kutil::offset_pointer(efi_map, map_length);
+		return iterator(*this, reinterpret_cast<efi_memory_descriptor const *>(end));
 	}
 
-	base->entries[index] = reinterpret_cast<uint64_t>(new_table) | ident_page_flags;
-}
+	const void *efi_map;
+	size_t map_length;
+	size_t desc_length;
+};
 
-static uint64_t
-find_efi_free_aligned_pages(const void *memory_map, size_t map_length, size_t desc_length, unsigned pages)
+class memory_bootstrap
 {
-	efi_memory_descriptor const *desc =
-		reinterpret_cast<efi_memory_descriptor const *>(memory_map);
-	efi_memory_descriptor const *end = desc_incr(desc, map_length);
+public:
+	memory_bootstrap(const void *memory_map, size_t map_length, size_t desc_length) :
+		map(memory_map, map_length, desc_length) {}
 
-	const unsigned want_space = pages * page_manager::page_size;
-	uint64_t start_phys = 0;
-
-	for (; desc < end; desc = desc_incr(desc, desc_length)) {
-		if (desc->type != efi_memory_type::available)
-			continue;
-
-		// See if the first wanted pages fit in one page table. If we
-		// find free memory at zero, skip ahead because we're not ready
-		// to deal with 0 being a valid pointer yet.
-		start_phys = desc->physical_start;
-		if (start_phys == 0)
-			start_phys += efi_page_size;
-
-		const uint64_t desc_end =
-			desc->physical_start + desc->pages * efi_page_size;
-
-		uint64_t end = start_phys + want_space;
-		if (end < desc_end) {
-			page_table_indices start_idx{start_phys};
-			page_table_indices end_idx{end};
-			if (start_idx[0] == end_idx[0] &&
-				start_idx[1] == end_idx[1] &&
-				start_idx[2] == end_idx[2])
-				break;
-
-			// Try seeing if the page-table-aligned version fits
-			start_phys = page_table_align(start_phys);
-			end = start_phys + want_space;
-			if (end < desc_end)
-				break;
-		}
-	}
-
-	kassert(desc < end, "Couldn't find wanted pages of aligned scratch space.");
-	return start_phys;
-}
-
-static unsigned
-check_needs_page_ident(page_table *table, unsigned index, page_table **free_pages)
-{
-	if ((table->entries[index] & 0x1) == 1) return 0;
-
-	kassert(*free_pages, "check_needs_page_ident needed to allocate but had no free pages");
-
-	page_table *new_table = (*free_pages)++;
-	for (int i=0; i<512; ++i) new_table->entries[i] = 0;
-	table->entries[index] = reinterpret_cast<uint64_t>(new_table) | ident_page_flags;
-	return 1;
-}
-
-static unsigned
-page_in_ident(
-		page_table *pml4,
-		uint64_t phys_addr,
-		uint64_t virt_addr,
-		uint64_t count,
-		page_table *free_pages)
-{
-	page_table_indices idx{virt_addr};
-	page_table *tables[4] = {pml4, nullptr, nullptr, nullptr};
-
-	unsigned pages_consumed = 0;
-	for (; idx[0] < 512; idx[0] += 1) {
-		pages_consumed += check_needs_page_ident(tables[0], idx[0], &free_pages);
-		tables[1] = reinterpret_cast<page_table *>(
-				tables[0]->entries[idx[0]] & ~0xfffull);
-
-		for (; idx[1] < 512; idx[1] += 1, idx[2] = 0, idx[3] = 0) {
-			pages_consumed += check_needs_page_ident(tables[1], idx[1], &free_pages);
-			tables[2] = reinterpret_cast<page_table *>(
-					tables[1]->entries[idx[1]] & ~0xfffull);
-
-			for (; idx[2] < 512; idx[2] += 1, idx[3] = 0) {
-				if (idx[3] == 0 &&
-					count >= 512 &&
-					tables[2]->get(idx[2]) == nullptr) {
-					// Do a 2MiB page instead
-					tables[2]->entries[idx[2]] = phys_addr | 0x80 | ident_page_flags;
-
-					phys_addr += page_manager::page_size * 512;
-					count -= 512;
-					if (count == 0) return pages_consumed;
-					continue;
-				}
-
-				pages_consumed += check_needs_page_ident(tables[2], idx[2], &free_pages);
-				tables[3] = reinterpret_cast<page_table *>(
-						tables[2]->entries[idx[2]] & ~0xfffull);
-
-				for (; idx[3] < 512; idx[3] += 1) {
-					tables[3]->entries[idx[3]] = phys_addr | ident_page_flags;
-					phys_addr += page_manager::page_size;
-					if (--count == 0) return pages_consumed;
-				}
+	void add_free_frames(frame_allocator &fa) {
+		for (auto *desc : map) {
+			if (desc->type == efi_memory_type::loader_code ||
+				desc->type == efi_memory_type::loader_data ||
+				desc->type == efi_memory_type::boot_services_code ||
+				desc->type == efi_memory_type::boot_services_data ||
+				desc->type == efi_memory_type::available)
+			{
+				fa.free(desc->physical_start, desc->pages);
 			}
 		}
 	}
 
-	kassert(0, "Ran to end of page_in_ident");
-	return 0; // Cannot reach
-}
+	void add_used_frames(kutil::address_manager &am) {
+		for (auto *desc : map) {
+			if (desc->type == efi_memory_type::popcorn_data ||
+				desc->type == efi_memory_type::popcorn_initrd)
+			{
+				uintptr_t virt_addr = desc->physical_start + kernel_offset;
+				am.mark(virt_addr, desc->pages * frame_size);
+			}
+			else if (desc->type == efi_memory_type::popcorn_kernel)
+			{
+				uintptr_t virt_addr = desc->physical_start + kernel_offset;
+				am.mark_permanent(virt_addr, desc->pages * frame_size);
+			}
+		}
+	}
 
-void
-memory_initialize(const void *memory_map, size_t map_length, size_t desc_length)
+	void page_in_kernel(page_manager &pm, page_table *pml4) {
+		for (auto *desc : map) {
+			if (desc->type == efi_memory_type::popcorn_kernel ||
+				desc->type == efi_memory_type::popcorn_data ||
+				desc->type == efi_memory_type::popcorn_initrd)
+			{
+				uintptr_t virt_addr = desc->physical_start + kernel_offset;
+				pm.page_in(pml4, desc->physical_start, virt_addr, desc->pages);
+			}
+
+			if (desc->type == efi_memory_type::acpi_reclaim) {
+				pm.page_in(pml4, desc->physical_start, desc->physical_start, desc->pages);
+			}
+		}
+
+		// Put our new PML4 into CR3 to start using it
+		page_manager::set_pml4(pml4);
+		pm.m_kernel_pml4 = pml4;
+	}
+
+private:
+	const memory_map map;
+};
+
+kutil::allocator &
+memory_initialize(uint16_t scratch_pages, const void *memory_map, size_t map_length, size_t desc_length)
 {
-	// The bootloader reserved 16 pages for page tables, which we'll use to bootstrap.
-	// The first one is the already-installed PML4, so grab it from CR3.
-	uint64_t cr3;
-	__asm__ __volatile__ ( "mov %%cr3, %0" : "=r" (cr3) );
-	page_table *tables = reinterpret_cast<page_table *>(cr3 & ~0xfffull);
-
-	// We'll need to make sure the options we want in CR4 are set
+	// make sure the options we want in CR4 are set
 	uint64_t cr4;
 	__asm__ __volatile__ ( "mov %%cr4, %0" : "=r" (cr4) );
-	cr4 |= 0x00080; // Enable global pages
-	cr4 |= 0x00200; // Enable FXSAVE/FXRSTOR
-	cr4 |= 0x20000; // Enable PCIDs
+	cr4 |=
+		0x000080 | // Enable global pages
+		0x000200 | // Enable FXSAVE/FXRSTOR
+		0x010000 | // Enable FSGSBASE
+		0x020000 | // Enable PCIDs
+		0;
 	__asm__ __volatile__ ( "mov %0, %%cr4" :: "r" (cr4) );
 
-	// Now go through EFi's memory map and find a region of scratch space.
-	const unsigned want_pages = 32;
-	uint64_t free_region_start_phys =
-		find_efi_free_aligned_pages(memory_map, map_length, desc_length, want_pages);
+	// The bootloader reserved "scratch_pages" pages for page tables and
+	// scratch space, which we'll use to bootstrap.  The first one is the
+	// already-installed PML4, so grab it from CR3.
+	uint64_t scratch_phys;
+	__asm__ __volatile__ ( "mov %%cr3, %0" : "=r" (scratch_phys) );
+	scratch_phys &= ~0xfffull;
 
-	// Offset-map this region into the higher half.
-	uint64_t free_region_start_virt =
-		free_region_start_phys + page_manager::page_offset;
+	// The tables are ident-mapped currently, so the cr3 physical address works. But let's
+	// get them into the offset-mapped area asap.
+	page_table *tables = reinterpret_cast<page_table *>(scratch_phys);
 
-	uint64_t free_next = free_region_start_virt;
-
-	// We'll need to copy any existing tables (except the PML4 which the
-	// bootloader gave us) into our reserved pages so we can edit them.
-	page_table_indices fr_idx{free_region_start_virt};
-
-	copy_new_table(&tables[0], fr_idx[0], &tables[1]);
-	copy_new_table(&tables[1], fr_idx[1], &tables[2]);
-	copy_new_table(&tables[2], fr_idx[2], &tables[3]);
-	page_in_ident(&tables[0], free_region_start_phys, free_region_start_virt, want_pages, nullptr);
-
-	// We now have pages starting at "free_next" to bootstrap ourselves. Start by
-	// taking inventory of free pages.
-	page_consumer allocator(free_next);
-	block_allocator block_slab(page_manager::page_size, allocator);
-	block_list used;
-	block_list free;
-
-	gather_block_lists(block_slab, used, free, memory_map, map_length, desc_length);
-	block_slab.allocate(); // Make sure we have extra
-
-	free_next = allocator.current;
-
-	// Now go back through these lists and consolidate
-	block_slab.append(page_block::consolidate(free));
-	block_slab.append(page_block::consolidate(used));
-
-	// Pull out the block that represents the bootstrap pages we've used
-	uint64_t used_bytes = free_next - free_region_start_virt;
-	uint64_t used_pages = used_bytes / page_manager::page_size;
-	uint64_t remaining_pages = want_pages - used_pages;
-
-	auto *removed = remove_block_for(free, free_region_start_phys,
-			used_pages, block_slab);
-
-	kassert(removed, "remove_block_for didn't find the bootstrap region.");
-	kassert(removed->physical_address == free_region_start_phys,
-			"remove_block_for found the wrong region.");
-
-	// Add it to the used list
-	removed->virtual_address = free_region_start_virt;
-	removed->flags = page_block_flags::used | page_block_flags::mapped;
-	used.sorted_insert(removed);
-
-	// Pull out the block that represents the rest
-	uint64_t free_next_phys = free_region_start_phys + used_bytes;
-
-	removed = remove_block_for(free, free_next_phys,
-			remaining_pages, block_slab);
-
-	kassert(removed, "remove_block_for didn't find the page table region.");
-	kassert(removed->physical_address == free_next_phys,
-			"remove_block_for found the wrong region.");
-
-	uint64_t pt_start_phys = removed->physical_address;
-	uint64_t pt_start_virt = removed->physical_address + page_manager::page_offset;
-
-	// Record that we're about to remap it into the page table address space
-	removed->virtual_address = pt_start_virt;
-	removed->flags = page_block_flags::used | page_block_flags::mapped;
-	used.sorted_insert(removed);
-
-	page_manager *pm = &g_page_manager;
-
-	// Actually remap them into page table space
-	pm->page_out(&tables[0], free_next, remaining_pages);
-
-	page_table_indices pg_idx{pt_start_virt};
-	copy_new_table(&tables[0], pg_idx[0], &tables[4]);
-	copy_new_table(&tables[4], pg_idx[1], &tables[5]);
-	copy_new_table(&tables[5], pg_idx[2], &tables[6]);
-
-	page_in_ident(&tables[0], pt_start_phys, pt_start_virt, remaining_pages, tables + 4);
+	page_table *id_pml4 = &tables[0];
+	page_table *id_pdp = &tables[1];
+	for (int i=0; i<512; ++i)
+		id_pdp->entries[i] = (static_cast<uintptr_t>(i) << 30) | 0x18b;
+	id_pml4->entries[511] = reinterpret_cast<uintptr_t>(id_pdp) | 0x10b;
 
 	// Make sure the page table is finished updating before we write to memory
 	__sync_synchronize();
 	io_wait();
 
-	// Finally, build an acutal set of kernel page tables that just contains
+	uintptr_t scratch_virt = scratch_phys + page_offset;
+	memory_bootstrap bootstrap {memory_map, map_length, desc_length};
+
+	// Now tell the frame allocator what's free
+	frame_allocator *fa = new (&g_frame_allocator) frame_allocator;
+	bootstrap.add_free_frames(*fa);
+
+	// Build an initial address manager that we'll copy into the real
+	// address manager later (so that we can use a raw allocator now)
+	kutil::allocator &alloc = fa->raw_allocator();
+	kutil::address_manager init_am(alloc);
+	init_am.add_regions(kernel_offset, page_offset - kernel_offset);
+	bootstrap.add_used_frames(init_am);
+
+	// Add the heap into the address manager
+	uintptr_t heap_start = page_offset - kernel_max_heap;
+	init_am.mark(heap_start, kernel_max_heap);
+
+	kutil::allocator *heap_alloc =
+		new (&g_kernel_heap) kutil::heap_allocator(heap_start, kernel_max_heap);
+
+	// Copy everything into the real address manager
+	kutil::address_manager *am =
+		new (&g_kernel_address_manager) kutil::address_manager(
+				std::move(init_am), *heap_alloc);
+
+	// Create the page manager
+	page_manager *pm = new (&g_page_manager) page_manager(*fa, *am);
+
+	// Give the frame_allocator back the rest of the scratch pages
+	fa->free(scratch_phys + (3 * frame_size), scratch_pages - 3);
+
+	// Finally, build an acutal set of kernel page tables where we'll only add
 	// what the kernel actually has mapped, but making everything writable
 	// (especially the page tables themselves)
-	page_table *pml4 = reinterpret_cast<page_table *>(pt_start_virt);
-	for (int i=0; i<512; ++i) pml4->entries[i] = 0;
+	page_table *pml4 = &tables[2];
+	pml4 = kutil::offset_pointer(pml4, page_offset);
 
-	// Give the rest to the page_manager's cache for use in page_in
-	pm->free_table_pages(pml4 + 1, remaining_pages - 1);
+	kutil::memset(pml4, 0, sizeof(page_table));
+	pml4->entries[511] = reinterpret_cast<uintptr_t>(id_pdp) | 0x10b;
 
-	for (auto *block : used) {
-		if (!block->has_flag(page_block_flags::mapped)) continue;
-		pm->page_in(pml4, block->physical_address, block->virtual_address, block->count);
-	}
+	bootstrap.page_in_kernel(*pm, pml4);
 
-	// Put our new PML4 into CR3 to start using it
-	page_manager::set_pml4(pml4);
+	// Reclaim the old PML4
+	fa->free(scratch_phys, 1);
 
-	// We now have all used memory mapped ourselves. Let the page_manager take
-	// over from here.
-	g_page_manager.init(
-			std::move(free),
-			std::move(used),
-			std::move(block_slab));
+	return *heap_alloc;
 }

src/kernel/page_manager.cpp

@@ -1,26 +1,33 @@
 #include <algorithm>
 
 #include "kutil/assert.h"
-#include "kutil/memory_manager.h"
 #include "console.h"
+#include "io.h"
 #include "log.h"
 #include "page_manager.h"
 
-page_manager g_page_manager;
-kutil::memory_manager g_kernel_memory_manager;
+using memory::frame_size;
+using memory::kernel_offset;
+using memory::page_offset;
+using memory::page_mappable;
+
+extern kutil::address_manager g_kernel_address_manager;
+page_manager g_page_manager(
+	g_frame_allocator,
+	g_kernel_address_manager);
 
 
 static uintptr_t
 pt_to_phys(page_table *pt)
 {
-	return reinterpret_cast<uintptr_t>(pt) - page_manager::page_offset;
+	return reinterpret_cast<uintptr_t>(pt) - page_offset;
 }
 
 
 static page_table *
 pt_from_phys(uintptr_t p)
 {
-	return reinterpret_cast<page_table *>((p + page_manager::page_offset) & ~0xfffull);
+	return reinterpret_cast<page_table *>((p + page_offset) & ~0xfffull);
 }
 
 
@@ -31,142 +38,13 @@ struct free_page_header
 };
 
 
-void mm_grow_callback(void *next, size_t length)
+page_manager::page_manager(
+		frame_allocator &frames,
+		kutil::address_manager &addrs) :
+	m_page_cache(nullptr),
+	m_frames(frames),
+	m_addrs(addrs)
 {
-	kassert(length % page_manager::page_size == 0,
-			"Heap manager requested a fractional page.");
-
-	size_t pages = length / page_manager::page_size;
-	log::info(logs::memory, "Heap manager growing heap by %d pages.", pages);
-	g_page_manager.map_pages(reinterpret_cast<uintptr_t>(next), pages);
-}
-
-
-int
-page_block::compare(const page_block *rhs) const
-{
-	if (virtual_address < rhs->virtual_address)
-		return -1;
-	else if (virtual_address > rhs->virtual_address)
-		return 1;
-
-	if (physical_address < rhs->physical_address)
-		return -1;
-	else if (physical_address > rhs->physical_address)
-		return 1;
-
-	return 0;
-}
-
-page_block_list
-page_block::consolidate(page_block_list &list)
-{
-	page_block_list freed;
-
-	for (auto *cur : list) {
-		auto *next = cur->next();
-
-		while (next &&
-			cur->flags == next->flags &&
-			cur->physical_end() == next->physical_address &&
-			(!cur->has_flag(page_block_flags::mapped) ||
-			cur->virtual_end() == next->virtual_address)) {
-
-			cur->count += next->count;
-			list.remove(next);
-			freed.push_back(next);
-		}
-	}
-
-	return freed;
-}
-
-void
-page_block::dump(const page_block_list &list, const char *name, bool show_unmapped)
-{
-	log::info(logs::memory, "Block list %s:", name);
-
-	int count = 0;
-	for (auto *cur : list) {
-		count += 1;
-		if (!(show_unmapped || cur->has_flag(page_block_flags::mapped)))
-			continue;
-
-		if (cur->virtual_address) {
-			page_table_indices start{cur->virtual_address};
-			log::info(logs::memory, "  %016lx %08x [%6d] %016lx (%d,%d,%d,%d)",
-					cur->physical_address,
-					cur->flags,
-					cur->count,
-					cur->virtual_address,
-					start[0], start[1], start[2], start[3]);
-		} else {
-			page_table_indices start{cur->virtual_address};
-			log::info(logs::memory, "  %016lx %08x [%6d]",
-					cur->physical_address,
-					cur->flags,
-					cur->count);
-		}
-	}
-
-	log::info(logs::memory, "  Total: %d", count);
-}
-
-void
-page_block::zero()
-{
-	physical_address = 0;
-	virtual_address = 0;
-	count = 0;
-	flags = page_block_flags::free;
-}
-
-void
-page_block::copy(page_block *other)
-{
-	physical_address = other->physical_address;
-	virtual_address = other->virtual_address;
-	count = other->count;
-	flags = other->flags;
-}
-
-
-page_manager::page_manager() :
-	m_block_slab(page_size),
-	m_page_cache(nullptr)
-{
-	kassert(this == &g_page_manager, "Attempt to create another page_manager.");
-}
-
-void
-page_manager::init(
-	page_block_list free,
-	page_block_list used,
-	page_block_list cache)
-{
-	m_free.append(free);
-	m_used.append(used);
-	m_block_slab.append(cache);
-
-	consolidate_blocks();
-
-	// Initialize the kernel memory manager
-	uintptr_t end = 0;
-	for (auto *block : m_used) {
-		if (block->virtual_address &&
-			block->virtual_address < page_offset) {
-			end = block->virtual_end();
-		} else {
-			break;
-		}
-	}
-
-	new (&g_kernel_memory_manager) kutil::memory_manager(
-			reinterpret_cast<void *>(end),
-			mm_grow_callback);
-	kutil::setup::set_heap(&g_kernel_memory_manager);
-
-	m_kernel_pml4 = get_pml4();
 }
 
 page_table *
@@ -174,67 +52,154 @@ page_manager::create_process_map()
 {
 	page_table *table = get_table_page();
 
-	kutil::memset(table, 0, page_size);
+	kutil::memset(table, 0, frame_size);
 	table->entries[510] = m_kernel_pml4->entries[510];
 	table->entries[511] = m_kernel_pml4->entries[511];
 
 	// Create the initial user stack
 	map_pages(
-		initial_stack - (initial_stack_pages * page_size),
-		initial_stack_pages,
+		memory::initial_stack - (memory::initial_stack_pages * frame_size),
+		memory::initial_stack_pages,
 		true, // This is the ring3 stack, user = true
 		table);
 
 	return table;
 }
 
-void
-page_manager::delete_process_map(page_table *table)
+uintptr_t
+page_manager::copy_page(uintptr_t orig)
 {
-	// TODO: recurse table entries and mark them free
-	unmap_pages(table, 1);
+	bool paged_orig = false;
+	bool paged_copy = false;
+
+	uintptr_t orig_virt;
+
+	if (page_mappable(orig)) {
+		orig_virt = orig + page_offset;
+	} else {
+		orig_virt = m_addrs.allocate(frame_size);
+		page_in(get_pml4(), orig, orig_virt, 1);
+		paged_orig = true;
+	}
+
+	uintptr_t copy = 0;
+	uintptr_t copy_virt;
+	size_t n = m_frames.allocate(1, &copy);
+	kassert(n, "copy_page could not allocate page");
+
+	if (page_mappable(copy)) {
+		copy_virt = copy + page_offset;
+	} else {
+		copy_virt = m_addrs.allocate(frame_size);
+		page_in(get_pml4(), copy, copy_virt, 1);
+		paged_copy = true;
+	}
+
+	// TODO: multiple page copies at a time, so that we don't have to keep
+	// paying this mapping penalty
+	if (paged_orig || paged_copy) {
+		set_pml4(get_pml4());
+		__sync_synchronize();
+		io_wait();
+	}
+
+	kutil::memcpy(
+			reinterpret_cast<void *>(copy_virt),
+			reinterpret_cast<void *>(orig_virt),
+			frame_size);
+
+	if (paged_orig) {
+		page_out(get_pml4(), orig_virt, 1);
+		m_addrs.free(orig_virt);
+	}
+
+	if (paged_copy) {
+		page_out(get_pml4(), copy_virt, 1);
+		m_addrs.free(copy_virt);
+	}
+
+	return copy;
+}
+
+page_table *
+page_manager::copy_table(page_table *from, page_table::level lvl, page_table_indices index)
+{
+	page_table *to = get_table_page();
+	log::debug(logs::paging, "Page manager copying level %d table at %016lx to %016lx.", lvl, from, to);
+
+	if (lvl == page_table::level::pml4) {
+		to->entries[510] = m_kernel_pml4->entries[510];
+		to->entries[511] = m_kernel_pml4->entries[511];
+	}
+
+	const int max =
+		lvl == page_table::level::pml4 ?
+			510 :
+			512;
+
+	unsigned pages_copied = 0;
+	uintptr_t from_addr = 0;
+	uintptr_t to_addr = 0;
+
+	for (int i = 0; i < max; ++i) {
+		if (!from->is_present(i)) {
+			to->entries[i] = 0;
+			continue;
+		}
+
+		index[lvl] = i;
+
+		bool is_page =
+			lvl == page_table::level::pt ||
+			from->is_large_page(lvl, i);
+
+		if (is_page) {
+			uint16_t flags = from->entries[i] &  0xfffull;
+			uintptr_t orig = from->entries[i] & ~0xfffull;
+			to->entries[i] = copy_page(orig) | flags;
+			if (!pages_copied++)
+				from_addr = index.addr();
+			to_addr = index.addr();
+		} else {
+			uint16_t flags = 0;
+			page_table *next_from = from->get(i, &flags);
+			page_table *next_to = copy_table(next_from, page_table::deeper(lvl), index);
+			to->set(i, next_to, flags);
+		}
+	}
+
+	if (pages_copied)
+		log::debug(logs::paging, "   copied %3u pages %016lx - %016lx",
+			pages_copied, from_addr, to_addr + frame_size);
+
+	return to;
+}
+
+void
+page_manager::delete_process_map(page_table *pml4)
+{
+	bool was_pml4 = (pml4 == get_pml4());
+	if (was_pml4)
+		set_pml4(m_kernel_pml4);
+
+	log::info(logs::paging, "Deleting process pml4 at %016lx%s",
+			pml4, was_pml4 ? " (was current)" : "");
+
+	unmap_table(pml4, page_table::level::pml4, true);
 }
 
 void
 page_manager::map_offset_pointer(void **pointer, size_t length)
 {
-	uintptr_t *p = reinterpret_cast<uintptr_t *>(pointer);
-	uintptr_t v = *p + page_offset;
-	uintptr_t c = ((length - 1) / page_size) + 1;
-
-	// TODO: cleanly search/split this as a block out of used/free if possible
-	auto *block = m_block_slab.pop();
-
-	// TODO: page-align
-	block->physical_address = *p;
-	block->virtual_address = v;
-	block->count = c;
-	block->flags =
-		page_block_flags::used |
-		page_block_flags::mapped |
-		page_block_flags::mmio;
-
-	m_used.sorted_insert(block);
-
-	page_table *pml4 = get_pml4();
-	page_in(pml4, *p, v, c);
-	*p = v;
+	log::debug(logs::paging, "Mapping offset pointer region at %016lx size 0x%lx", *pointer, length);
+	*pointer = kutil::offset_pointer(*pointer, page_offset);
 }
 
 void
-page_manager::dump_blocks(bool used_only)
+page_manager::dump_pml4(page_table *pml4, bool recurse)
 {
-	page_block::dump(m_used, "used", true);
-	if (!used_only)
-		page_block::dump(m_free, "free", true);
-}
-
-void
-page_manager::dump_pml4(page_table *pml4, int max_index)
-{
-	if (pml4 == nullptr)
-		pml4 = get_pml4();
-	pml4->dump(4, max_index);
+	if (pml4 == nullptr) pml4 = get_pml4();
+	pml4->dump(page_table::level::pml4, recurse);
 }
 
 page_table *
@@ -242,35 +207,26 @@ page_manager::get_table_page()
 {
 	if (!m_page_cache) {
 		uintptr_t phys = 0;
-		size_t n = pop_pages(32, &phys);
+		size_t n = m_frames.allocate(32, &phys); // TODO: indicate frames must be offset-mappable
 		uintptr_t virt = phys + page_offset;
 
-		auto *block = m_block_slab.pop();
-
-		block->physical_address = phys;
-		block->virtual_address = virt;
-		block->count = n;
-
-		m_used.sorted_insert(block);
-
-		page_in(get_pml4(), phys, virt, n);
-
 		m_page_cache = reinterpret_cast<free_page_header *>(virt);
 
 		// The last one needs to be null, so do n-1
-		uintptr_t end = virt + (n-1) * page_size;
+		uintptr_t end = virt + (n-1) * frame_size;
 		while (virt < end) {
 			reinterpret_cast<free_page_header *>(virt)->next =
-				reinterpret_cast<free_page_header *>(virt + page_size);
-			virt += page_size;
+				reinterpret_cast<free_page_header *>(virt + frame_size);
+			virt += frame_size;
 		}
 		reinterpret_cast<free_page_header *>(virt)->next = nullptr;
 
-		log::debug(logs::memory, "Mappd %d new page table pages at %lx", n, phys);
+		log::info(logs::paging, "Mappd %d new page table pages at %lx", n, phys);
 	}
 
 	free_page_header *page = m_page_cache;
 	m_page_cache = page->next;
+
 	return reinterpret_cast<page_table *>(page);
 }
 
@@ -279,7 +235,7 @@ page_manager::free_table_pages(void *pages, size_t count)
 {
 	uintptr_t start = reinterpret_cast<uintptr_t>(pages);
 	for (size_t i = 0; i < count; ++i) {
-		uintptr_t addr = start + (i * page_size);
+		uintptr_t addr = start + (i * frame_size);
 		free_page_header *header = reinterpret_cast<free_page_header *>(addr);
 		header->count = 1;
 		header->next = m_page_cache;
@@ -287,143 +243,126 @@ page_manager::free_table_pages(void *pages, size_t count)
 	}
 }
 
-void
-page_manager::consolidate_blocks()
-{
-	m_block_slab.append(page_block::consolidate(m_free));
-	m_block_slab.append(page_block::consolidate(m_used));
-}
-
 void *
 page_manager::map_pages(uintptr_t address, size_t count, bool user, page_table *pml4)
 {
+	if (!address) {
+		kassert(!user, "Cannot call map_pages with 0 address for user mapping");
+		address = m_addrs.allocate(count * frame_size);
+	}
+
 	void *ret = reinterpret_cast<void *>(address);
 	if (!pml4) pml4 = get_pml4();
 
 	while (count) {
-		kassert(!m_free.empty(), "page_manager::map_pages ran out of free pages!");
-
 		uintptr_t phys = 0;
-		size_t n = pop_pages(count, &phys);
+		size_t n = m_frames.allocate(count, &phys);
 
-		auto *block = m_block_slab.pop();
-
-		block->physical_address = phys;
-		block->virtual_address = address;
-		block->count = n;
-		block->flags =
-				page_block_flags::used |
-				page_block_flags::mapped;
-
-		m_used.sorted_insert(block);
-
-		log::debug(logs::memory, "Paging in %d pages at p:%016lx to v:%016lx into %016lx table",
+		log::info(logs::paging, "Paging in %d pages at p:%016lx to v:%016lx into %016lx table",
 				n, phys, address, pml4);
 
 		page_in(pml4, phys, address, n, user);
 
-		address += n * page_size;
+		address += n * frame_size;
 		count -= n;
 	}
 
 	return ret;
 }
 
-void *
-page_manager::map_offset_pages(size_t count)
+void
+page_manager::unmap_table(page_table *table, page_table::level lvl, bool free, page_table_indices index)
 {
-	page_table *pml4 = get_pml4();
+	const int max =
+		lvl == page_table::level::pml4 ?
+			510 :
+			512;
 
-	for (auto *free : m_free) {
-		if (free->count < count) continue;
+	uintptr_t free_start = 0;
+	uintptr_t free_start_virt = 0;
+	uintptr_t free_count = 0;
 
-		auto *used = m_block_slab.pop();
+	size_t size =
+		lvl == page_table::level::pdp ? (1<<30) :
+		lvl == page_table::level::pd  ? (1<<21) :
+		lvl == page_table::level::pt  ? (1<<12) :
+		0;
 
-		used->count = count;
-		used->physical_address = free->physical_address;
-		used->virtual_address = used->physical_address + page_offset;
-		used->flags =
-			page_block_flags::used |
-			page_block_flags::mapped;
+	for (int i = 0; i < max; ++i) {
+		if (!table->is_present(i)) continue;
 
-		m_used.sorted_insert(used);
+		index[lvl] = i;
 
-		free->physical_address += count * page_size;
-		free->count -= count;
+		bool is_page =
+			lvl == page_table::level::pt ||
+			table->is_large_page(lvl, i);
 
-		if (free->count == 0) {
-			m_free.remove(free);
-			free->zero();
-			m_block_slab.push(free);
+		if (is_page) {
+			uintptr_t frame = table->entries[i] & ~0xfffull;
+			if (!free_count || frame != free_start + free_count * size) {
+				if (free_count && free) {
+					log::debug(logs::paging,
+							"  freeing v:%016lx-%016lx p:%016lx-%016lx",
+							free_start_virt, free_start_virt + free_count * frame_size,
+							free_start, free_start + free_count * frame_size);
+
+					m_frames.free(free_start, (free_count * size) / frame_size);
+					free_count = 0;
+				}
+
+				if (!free_count) {
+					free_start = frame;
+					free_start_virt = index.addr();
+				}
+			}
+
+			free_count += 1;
+		} else {
+			page_table *next = table->get(i);
+			unmap_table(next, page_table::deeper(lvl), free, index);
 		}
-
-		log::debug(logs::memory, "Got request for offset map %016lx [%d]", used->virtual_address, count);
-		page_in(pml4, used->physical_address, used->virtual_address, count);
-		return reinterpret_cast<void *>(used->virtual_address);
 	}
 
-	return nullptr;
+	if (free_count && free) {
+		log::debug(logs::paging,
+				"  freeing v:%016lx-%016lx p:%016lx-%016lx",
+				free_start_virt, free_start_virt + free_count * frame_size,
+				free_start, free_start + free_count * frame_size);
+
+		m_frames.free(free_start, (free_count * size) / frame_size);
+	}
+	free_table_pages(table, 1);
+
+	log::debug(logs::paging, "Unmapped%s lv %d table at %016lx",
+			free ? " (and freed)" : "", lvl, table);
 }
 
 void
-page_manager::unmap_pages(void* address, size_t count)
+page_manager::unmap_pages(void* address, size_t count, page_table *pml4)
 {
-	uintptr_t addr = reinterpret_cast<uintptr_t>(address);
-	size_t block_count = 0;
+	if (!pml4)
+		pml4 = get_pml4();
 
-	for (auto *block : m_used) {
-		if (!block->contains(addr)) continue;
+	uintptr_t iaddr = reinterpret_cast<uintptr_t>(address);
 
-		size_t size = page_size * count;
-		uintptr_t end = addr + size;
-
-		size_t leading = addr - block->virtual_address;
-		size_t trailing =
-			end > block->virtual_end() ?
-			0 : (block->virtual_end() - end);
-
-		if (leading) {
-			size_t pages = leading / page_size;
-
-			auto *lead_block = m_block_slab.pop();
-
-			lead_block->copy(block);
-			lead_block->count = pages;
-
-			block->count -= pages;
-			block->physical_address += leading;
-			block->virtual_address += leading;
-
-			m_used.insert_before(block, lead_block);
-		}
-
-		if (trailing) {
-			size_t pages = trailing / page_size;
-
-			auto *trail_block = m_block_slab.pop();
-
-			trail_block->copy(block);
-			trail_block->count = pages;
-			trail_block->physical_address += size;
-			trail_block->virtual_address += size;
-
-			block->count -= pages;
-
-			m_used.insert_after(block, trail_block);
-		}
-
-		addr += block->count * page_size;
-
-		block->virtual_address = 0;
-		block->flags = block->flags &
-			~(page_block_flags::used | page_block_flags::mapped);
-
-		m_used.remove(block);
-		m_free.sorted_insert(block);
-		++block_count;
+	page_out(pml4, iaddr, count, true);
+	if (iaddr >= kernel_offset) {
+		// TODO
+		// m_addrs.free(address, count);
 	}
+}
 
-	kassert(block_count, "Couldn't find existing mapped pages to unmap");
+bool
+page_manager::fault_handler(uintptr_t addr)
+{
+	if (!m_addrs.contains(addr))
+		return false;
+
+	uintptr_t page = addr & ~0xfffull;
+	bool user = addr < kernel_offset;
+	map_pages(page, 1, user);
+
+	return true;
 }
 
 void
@@ -437,8 +376,13 @@ page_manager::check_needs_page(page_table *table, unsigned index, bool user)
 }
 
 void
-page_manager::page_in(page_table *pml4, uintptr_t phys_addr, uintptr_t virt_addr, size_t count, bool user)
+page_manager::page_in(page_table *pml4, uintptr_t phys_addr, uintptr_t virt_addr, size_t count, bool user, bool large)
 {
+	/*
+	log::debug(logs::paging, "page_in for table %016lx p:%016lx v:%016lx c:%4d u:%d l:%d",
+			pml4, phys_addr, virt_addr, count, user, large);
+	*/
+
 	page_table_indices idx{virt_addr};
 	page_table *tables[4] = {pml4, nullptr, nullptr, nullptr};
 
@@ -455,12 +399,13 @@ page_manager::page_in(page_table *pml4, uintptr_t phys_addr, uintptr_t virt_addr
 			tables[2] = tables[1]->get(idx[1]);
 
 			for (; idx[2] < 512; idx[2] += 1, idx[3] = 0) {
-				if (idx[3] == 0 &&
+				if (large &&
+					idx[3] == 0 &&
 					count >= 512 &&
 					tables[2]->get(idx[2]) == nullptr) {
 					// Do a 2MiB page instead
 					tables[2]->entries[idx[2]] = phys_addr | flags | 0x80;
-					phys_addr += page_size * 512;
+					phys_addr += frame_size * 512;
 					count -= 512;
 					if (count == 0) return;
 					continue;
@@ -471,7 +416,7 @@ page_manager::page_in(page_table *pml4, uintptr_t phys_addr, uintptr_t virt_addr
 
 				for (; idx[3] < 512; idx[3] += 1) {
 					tables[3]->entries[idx[3]] = phys_addr | flags;
-					phys_addr += page_size;
+					phys_addr += frame_size;
 					if (--count == 0) return;
 				}
 			}
@@ -482,26 +427,41 @@ page_manager::page_in(page_table *pml4, uintptr_t phys_addr, uintptr_t virt_addr
 }
 
 void
-page_manager::page_out(page_table *pml4, uintptr_t virt_addr, size_t count)
+page_manager::page_out(page_table *pml4, uintptr_t virt_addr, size_t count, bool free)
 {
 	page_table_indices idx{virt_addr};
 	page_table *tables[4] = {pml4, nullptr, nullptr, nullptr};
 
+	bool found = false;
+	uintptr_t free_start = 0;
+	unsigned free_count = 0;
+
 	for (; idx[0] < 512; idx[0] += 1) {
-		tables[1] = reinterpret_cast<page_table *>(
-				tables[0]->entries[idx[0]] & ~0xfffull);
+		tables[1] = tables[0]->get(idx[0]);
 
 		for (; idx[1] < 512; idx[1] += 1) {
-			tables[2] = reinterpret_cast<page_table *>(
-					tables[1]->entries[idx[1]] & ~0xfffull);
+			tables[2] = tables[1]->get(idx[1]);
 
 			for (; idx[2] < 512; idx[2] += 1) {
-				tables[3] = reinterpret_cast<page_table *>(
-						tables[2]->entries[idx[2]] & ~0xfffull);
+				tables[3] = tables[2]->get(idx[2]);
 
 				for (; idx[3] < 512; idx[3] += 1) {
+					uintptr_t entry = tables[3]->entries[idx[3]] & ~0xfffull;
+					if (!found || entry != free_start + free_count * frame_size) {
+						if (found && free) m_frames.free(free_start, free_count);
+						free_start = tables[3]->entries[idx[3]] & ~0xfffull;
+						free_count = 1;
+						found = true;
+					} else {
+						free_count++;
+					}
+
 					tables[3]->entries[idx[3]] = 0;
-					if (--count == 0) return;
+
+					if (--count == 0) {
+						if (free) m_frames.free(free_start, free_count);
+						return;
+					}
 				}
 			}
 		}
@@ -510,62 +470,60 @@ page_manager::page_out(page_table *pml4, uintptr_t virt_addr, size_t count)
 	kassert(0, "Ran to end of page_out");
 }
 
-size_t
-page_manager::pop_pages(size_t count, uintptr_t *address)
-{
-	kassert(!m_free.empty(), "page_manager::pop_pages ran out of free pages!");
-
-	auto *first = m_free.front();
-
-	unsigned n = std::min(count, static_cast<size_t>(first->count));
-	*address = first->physical_address;
-
-	first->physical_address += n * page_size;
-	first->count -= n;
-	if (first->count == 0)
-		m_block_slab.push(m_free.pop_front());
-
-	return n;
-}
-
-
 void
-page_table::dump(int level, int max_index, uint64_t offset)
+page_table::dump(page_table::level lvl, bool recurse)
 {
 	console *cons = console::get();
 
-	cons->printf("\nLevel %d page table @ %lx (off %lx):\n", level, this, offset);
+	cons->printf("\nLevel %d page table @ %lx:\n", lvl, this);
 	for (int i=0; i<512; ++i) {
 		uint64_t ent = entries[i];
-		if (ent == 0) continue;
 
-		if ((ent & 0x1) == 0) {
+		if ((ent & 0x1) == 0)
 			cons->printf("  %3d: %016lx   NOT PRESENT\n", i, ent);
-			continue;
-		}
 
-		if ((level == 2 || level == 3) && (ent & 0x80) == 0x80) {
+		else if ((lvl == level::pdp || lvl == level::pd) && (ent & 0x80) == 0x80)
 			cons->printf("  %3d: %016lx -> Large page at    %016lx\n", i, ent, ent & ~0xfffull);
-			continue;
-		} else if (level == 1) {
+
+		else if (lvl == level::pt)
 			cons->printf("  %3d: %016lx -> Page at          %016lx\n", i, ent, ent & ~0xfffull);
-		} else {
+
+		else
 			cons->printf("  %3d: %016lx -> Level %d table at %016lx\n",
-					i, ent, level - 1, (ent & ~0xfffull) + offset);
-			continue;
-		}
+					i, ent, deeper(lvl), (ent & ~0xfffull) + page_offset);
 	}
 
-	if (--level > 0) {
-		for (int i=0; i<=max_index; ++i) {
-			uint64_t ent = entries[i];
-			if ((ent & 0x1) == 0) continue;
-			if ((ent & 0x80)) continue;
+	if (lvl != level::pt && recurse) {
+		for (int i=0; i<=512; ++i) {
+			if (is_large_page(lvl, i))
+				continue;
 
-			page_table *next = reinterpret_cast<page_table *>((ent & ~0xffful) + offset); 
-			next->dump(level, 511, offset);
+			page_table *next = get(i);
+			if (next)
+				next->dump(deeper(lvl), true);
 		}
 	}
 }
 
+page_table_indices::page_table_indices(uint64_t v) :
+	index{
+		(v >> 39) & 0x1ff,
+		(v >> 30) & 0x1ff,
+		(v >> 21) & 0x1ff,
+		(v >> 12) & 0x1ff }
+{}
 
+uintptr_t
+page_table_indices::addr() const
+{
+	return
+		(index[0] << 39) |
+		(index[1] << 30) |
+		(index[2] << 21) |
+		(index[3] << 12);
+}
+
+bool operator==(const page_table_indices &l, const page_table_indices &r)
+{
+	return l[0] == r[0] && l[1] == r[1] && l[2] == r[2] && l[3] == r[3];
+}

src/kernel/page_manager.h

@@ -5,44 +5,30 @@
 #include <stddef.h>
 #include <stdint.h>
 
+#include "kutil/address_manager.h"
 #include "kutil/enum_bitfields.h"
 #include "kutil/linked_list.h"
 #include "kutil/slab_allocator.h"
+#include "frame_allocator.h"
+#include "kernel_memory.h"
+#include "page_table.h"
 
-struct page_block;
-struct page_table;
 struct free_page_header;
 
-using page_block_list = kutil::linked_list<page_block>;
-using page_block_slab = kutil::slab_allocator<page_block>;
-
-/// Manager for allocation of physical pages.
+/// Manager for allocation and mapping of pages
 class page_manager
 {
 public:
-	/// Size of a single page.
-	static const size_t page_size = 0x1000;
-
-	/// Start of the higher half.
-	static const uintptr_t high_offset = 0xffffff0000000000;
-
-	/// Offset from physical where page tables are mapped.
-	static const uintptr_t page_offset = 0xffffff8000000000;
-
-	/// Initial process thread's stack address
-	static const uintptr_t initial_stack = 0x0000800000000000;
-
-	/// Initial process thread's stack size, in pages
-	static const unsigned initial_stack_pages = 1;
-
-	page_manager();
+	page_manager(
+		frame_allocator &frames,
+		kutil::address_manager &addrs);
 
 	/// Helper to get the number of pages needed for a given number of bytes.
 	/// \arg bytes  The number of bytes desired
 	/// \returns    The number of pages needed to contain the desired bytes
 	static inline size_t page_count(size_t bytes)
 	{
-		return (bytes - 1) / page_size + 1;
+		return (bytes - 1) / memory::frame_size + 1;
 	}
 
 	/// Helper to read the PML4 table from CR3.
@@ -51,15 +37,16 @@ public:
 	{
 		uintptr_t pml4 = 0;
 		__asm__ __volatile__ ( "mov %%cr3, %0" : "=r" (pml4) );
-		return reinterpret_cast<page_table *>((pml4 & ~0xfffull) + page_offset);
+		return reinterpret_cast<page_table *>((pml4 & ~0xfffull) + memory::page_offset);
 	}
 
 	/// Helper to set the PML4 table pointer in CR3.
 	/// \arg pml4  A pointer to the PML4 table to install.
 	static inline void set_pml4(page_table *pml4)
 	{
-		uintptr_t p = reinterpret_cast<uintptr_t>(pml4) - page_offset;
-		__asm__ __volatile__ ( "mov %0, %%cr3" :: "r" (p & ~0xfffull) );
+		constexpr uint64_t phys_mask = ~memory::page_offset & ~0xfffull;
+		uintptr_t p = reinterpret_cast<uintptr_t>(pml4) & phys_mask;
+		__asm__ __volatile__ ( "mov %0, %%cr3" :: "r" (p) );
 	}
 
 	/// Allocate but don't switch to a new PML4 table. This table
@@ -67,28 +54,32 @@ public:
 	/// \returns      A pointer to the PML4 table
 	page_table * create_process_map();
 
-	/// Deallocate a process' PML4 table.
-	void delete_process_map(page_table *table);
+	/// Deallocate a process' PML4 table and entries.
+	/// \arg pml4  The process' PML4 table
+	void delete_process_map(page_table *pml4);
+
+	/// Copy a process' memory mappings (and memory pages).
+	/// \arg from  Page table to copy from
+	/// \arg lvl   Level of the given tables (default is PML4)
+	/// \returns   The new page table
+	page_table * copy_table(page_table *from,
+			page_table::level lvl = page_table::level::pml4,
+			page_table_indices index = {});
 
 	/// Allocate and map pages into virtual memory.
-	/// \arg address  The virtual address at which to map the pages
+	/// \arg address  The virtual address at which to map the pages, or zero
+	///               for any free kernel space.
 	/// \arg count    The number of pages to map
 	/// \arg user     True is this memory is user-accessible
 	/// \arg pml4     The pml4 to map into - null for the current one
 	/// \returns      A pointer to the start of the mapped region
 	void * map_pages(uintptr_t address, size_t count, bool user = false, page_table *pml4 = nullptr);
 
-	/// Allocate and map contiguous pages into virtual memory, with
-	/// a constant offset from their physical address.
-	/// \arg count    The number of pages to map
-	/// \returns      A pointer to the start of the mapped region, or
-	/// nullptr if no region could be found to fit the request.
-	void * map_offset_pages(size_t count);
-
-	/// Unmap existing pages from memory.
+	/// Unmap and free existing pages from memory.
 	/// \arg address  The virtual address of the memory to unmap
 	/// \arg count    The number of pages to unmap
-	void unmap_pages(void *address, size_t count);
+	/// \arg pml4     The pml4 to unmap from - null for the current one
+	void unmap_pages(void *address, size_t count, page_table *pml4 = nullptr);
 
 	/// Offset-map a pointer. No physical pages will be mapped.
 	/// \arg pointer  Pointer to a pointer to the memory area to be mapped
@@ -100,7 +91,7 @@ public:
 	/// \returns Physical address of the memory pointed to by p
 	inline uintptr_t offset_phys(void *p) const
 	{
-		return reinterpret_cast<uintptr_t>(kutil::offset_pointer(p, -page_offset));
+		return reinterpret_cast<uintptr_t>(kutil::offset_pointer(p, -memory::page_offset));
 	}
 
 	/// Get the virtual address of an offset-mapped physical address
@@ -108,39 +99,31 @@ public:
 	/// \returns Virtual address of the memory at address a
 	inline void * offset_virt(uintptr_t a) const
 	{
-		return kutil::offset_pointer(reinterpret_cast<void *>(a), page_offset);
+		return kutil::offset_pointer(reinterpret_cast<void *>(a), memory::page_offset);
 	}
 
-	/// Log the current free/used block lists.
-	/// \arg used_only  If true, skip printing free list. Default false.
-	void dump_blocks(bool used_only = false);
-
 	/// Dump the given or current PML4 to the console
-	/// \arg pml4       The page table to use, null for the current one
-	/// \arg max_index  The max index of pml4 to print
-	void dump_pml4(page_table *pml4 = nullptr, int max_index = 511);
+	/// \arg pml4     The page table to use, null for the current one
+	/// \arg recurse  Whether to print sub-tables
+	void dump_pml4(page_table *pml4 = nullptr, bool recurse = true);
 
 	/// Get the system page manager.
 	/// \returns  A pointer to the system page manager
 	static page_manager * get();
 
+	/// Get a pointer to the kernel's PML4
+	inline page_table * get_kernel_pml4() { return m_kernel_pml4; }
+
+	/// Attempt to handle a page fault.
+	/// \arg addr  Address that triggered the fault
+	/// \returns   True if the fault was handled
+	bool fault_handler(uintptr_t addr);
+
 private:
-	/// Set up the memory manager from bootstraped memory
-	void init(
-			page_block_list free,
-			page_block_list used,
-			page_block_list cache);
-
-	/// Initialize the virtual memory manager based on this object's state
-	void init_memory_manager();
-
-	/// Create a `page_block` struct or pull one from the cache.
-	/// \returns  An empty `page_block` struct
-	page_block * get_block();
-
-	/// Return a list of `page_block` structs to the cache.
-	/// \arg block   A list of `page_block` structs
-	void free_blocks(page_block *block);
+	/// Copy a physical page
+	/// \arg orig  Physical address of the page to copy
+	/// \returns   Physical address of the new page
+	uintptr_t copy_page(uintptr_t orig);
 
 	/// Allocate a page for a page table, or pull one from the cache
 	/// \returns  An empty page mapped in page space
@@ -151,10 +134,6 @@ private:
 	/// \arg count  Number of pages in the range
 	void free_table_pages(void *pages, size_t count);
 
-	/// Consolidate the free and used block lists. Return freed blocks
-	/// to the cache.
-	void consolidate_blocks();
-
 	/// Helper function to allocate a new page table. If table entry `i` in
 	/// table `base` is empty, allocate a new page table and point `base[i]` at
 	/// it.
@@ -168,40 +147,38 @@ private:
 	/// \arg phys_addr  The starting physical address of the pages to be mapped
 	/// \arg virt_addr  The starting virtual address ot the memory to be mapped
 	/// \arg count      The number of pages to map
-	/// \art user  True if this is a userspace mapping
+	/// \arg user       True if this is a userspace mapping
+	/// \arg large      Whether to allow large pages
 	void page_in(
 			page_table *pml4,
 			uintptr_t phys_addr,
 			uintptr_t virt_addr,
 			size_t count,
-			bool user = false);
+			bool user = false,
+			bool large = false);
 
 	/// Low-level routine for unmapping a number of pages from the given page table.
 	/// \arg pml4       The root page table for this mapping
 	/// \arg virt_addr  The starting virtual address ot the memory to be unmapped
 	/// \arg count      The number of pages to unmap
+	/// \arg free       Whether to return the pages to the frame allocator
 	void page_out(
 			page_table *pml4,
 			uintptr_t virt_addr,
-			size_t count);
+			size_t count,
+			bool free = false);
 
-	/// Get free pages from the free list. Only pages from the first free block
-	/// are returned, so the number may be less than requested, but they will
-	/// be contiguous. Pages will not be mapped into virtual memory.
-	/// \arg count    The maximum number of pages to get
-	/// \arg address  [out] The address of the first page
-	/// \returns      The number of pages retrieved
-	size_t pop_pages(size_t count, uintptr_t *address);
+	/// Low-level routine for unmapping an entire table of memory at once
+	void unmap_table(page_table *table, page_table::level lvl, bool free,
+			page_table_indices index = {});
 
 	page_table *m_kernel_pml4; ///< The PML4 of just kernel pages
-
-	page_block_list m_free; ///< Free pages list
-	page_block_list m_used; ///< In-use pages list
-	page_block_slab m_block_slab; ///< page_block slab allocator
-
 	free_page_header *m_page_cache; ///< Cache of free pages to use for tables
 
-	friend void memory_initialize(const void *, size_t, size_t);
+	frame_allocator &m_frames;
+	kutil::address_manager &m_addrs;
+
+	friend class memory_bootstrap;
 	page_manager(const page_manager &) = delete;
 };
 
@@ -210,105 +187,6 @@ extern page_manager g_page_manager;
 
 inline page_manager * page_manager::get() { return &g_page_manager; }
 
-/// Flags used by `page_block`.
-enum class page_block_flags : uint32_t
-{
-	free         = 0x00000000,  ///< Not a flag, value for free memory
-	used         = 0x00000001,  ///< Memory is in use
-	mapped       = 0x00000002,  ///< Memory is mapped to virtual address
-
-	mmio         = 0x00000010,  ///< Memory is a MMIO region
-	nonvolatile  = 0x00000020,  ///< Memory is non-volatile storage
-
-	pending_free = 0x10000000,  ///< Memory should be freed
-	acpi_wait    = 0x40000000,  ///< Memory should be freed after ACPI init
-	permanent    = 0x80000000,  ///< Memory is permanently unusable
-
-	max_flags
-};
-IS_BITFIELD(page_block_flags);
-
-
-/// A block of contiguous pages. Each `page_block` represents contiguous
-/// physical pages with the same attributes. A `page_block *` is also a
-/// linked list of such structures.
-struct page_block
-{
-	uintptr_t physical_address;
-	uintptr_t virtual_address;
-	uint32_t count;
-	page_block_flags flags;
-
-	inline bool has_flag(page_block_flags f) const { return bitfield_has(flags, f); }
-	inline uintptr_t physical_end() const { return physical_address + (count * page_manager::page_size); }
-	inline uintptr_t virtual_end() const { return virtual_address + (count * page_manager::page_size); }
-
-	inline bool contains(uintptr_t vaddr) const { return vaddr >= virtual_address && vaddr < virtual_end(); }
-	inline bool contains_physical(uintptr_t addr) const { return addr >= physical_address && addr < physical_end(); }
-
-	/// Helper to zero out a block and optionally set the next pointer.
-	void zero();
-
-	/// Helper to copy a bock from another block
-	/// \arg other  The block to copy from
-	void copy(page_block *other);
-
-	/// Compare two blocks by address.
-	/// \arg rhs   The right-hand comparator
-	/// \returns   <0 if this is sorts earlier, >0 if this sorts later, 0 for equal
-	int compare(const page_block *rhs) const;
-
-	/// Traverse the list, joining adjacent blocks where possible.
-	/// \arg list  The list to consolidate
-	/// \returns   A linked list of freed page_block structures.
-	static page_block_list consolidate(page_block_list &list);
-
-	/// Traverse the list, printing debug info on this list.
-	/// \arg list  The list to print
-	/// \arg name  [optional] String to print as the name of this list
-	/// \arg show_permanent [optional] If false, hide unmapped blocks
-	static void dump(const page_block_list &list, const char *name = nullptr, bool show_unmapped = false);
-};
-
-
-
-/// Struct to allow easy accessing of a memory page being used as a page table.
-struct page_table
-{
-	using pm = page_manager;
-
-	uint64_t entries[512];
-
-	inline page_table * get(int i) const {
-		uint64_t entry = entries[i];
-		if ((entry & 0x1) == 0) return nullptr;
-		return reinterpret_cast<page_table *>((entry & ~0xfffull) + pm::page_offset);
-	}
-
-	inline void set(int i, page_table *p, uint16_t flags) {
-		entries[i] = (reinterpret_cast<uint64_t>(p) - pm::page_offset) | (flags & 0xfff);
-	}
-
-	void dump(int level = 4, int max_index = 511, uint64_t offset = page_manager::page_offset);
-};
-
-
-/// Helper struct for computing page table indices of a given address.
-struct page_table_indices
-{
-	page_table_indices(uint64_t v = 0) :
-		index{
-			(v >> 39) & 0x1ff,
-			(v >> 30) & 0x1ff,
-			(v >> 21) & 0x1ff,
-			(v >> 12) & 0x1ff }
-	{}
-
-	/// Get the index for a given level of page table.
-	uint64_t & operator[](size_t i) { return index[i]; }
-	uint64_t index[4]; ///< Indices for each level of tables.
-};
-
 
 /// Calculate a page-aligned address.
 /// \arg p    The address to align.
@@ -317,8 +195,8 @@ template <typename T> inline T
 page_align(T p)
 {
 	return reinterpret_cast<T>(
-		((reinterpret_cast<uintptr_t>(p) - 1) & ~(page_manager::page_size - 1))
-		+ page_manager::page_size);
+		((reinterpret_cast<uintptr_t>(p) - 1) & ~(memory::frame_size - 1))
+		+ memory::frame_size);
 }
 
 /// Calculate a page-table-aligned address. That is, an address that is
@@ -332,11 +210,9 @@ page_table_align(T p)
 }
 
 
-/// Calculate the number of pages needed for the give number of bytes.
-/// \arg n   Number of bytes
-/// \returns Number of pages
-inline size_t page_count(size_t n) { return ((n - 1) / page_manager::page_size) + 1; }
-
-
 /// Bootstrap the memory managers.
-void memory_initialize(const void *memory_map, size_t map_length, size_t desc_length);
+kutil::allocator & memory_initialize(
+		uint16_t scratch_pages,
+		const void *memory_map,
+		size_t map_length,
+		size_t desc_length);

src/kernel/page_table.h

@@ -0,0 +1,63 @@
+#pragma once
+/// \file page_table.h
+/// Helper structures for dealing with page tables.
+
+#include <stdint.h>
+#include "kernel_memory.h"
+
+class page_manager;
+
+/// Struct to allow easy accessing of a memory page being used as a page table.
+struct page_table
+{
+	enum class level : unsigned { pml4, pdp, pd, pt };
+	inline static level deeper(level l) {
+		return static_cast<level>(static_cast<unsigned>(l) + 1);
+	}
+
+	uint64_t entries[512];
+
+	inline page_table * get(int i, uint16_t *flags = nullptr) const {
+		uint64_t entry = entries[i];
+		if ((entry & 0x1) == 0) return nullptr;
+		if (flags) *flags = entry & 0xfffull;
+		return reinterpret_cast<page_table *>((entry & ~0xfffull) + memory::page_offset);
+	}
+
+	inline void set(int i, page_table *p, uint16_t flags) {
+		entries[i] = (reinterpret_cast<uint64_t>(p) - memory::page_offset) | (flags & 0xfff);
+	}
+
+	inline bool is_present(int i) const { return (entries[i] & 0x1) == 0x1; }
+
+	inline bool is_large_page(level l, int i) const {
+		return
+			(l == level::pdp || l == level::pd) &&
+			(entries[i] & 0x80) == 0x80;
+	}
+
+	void dump(
+		level lvl = level::pml4,
+		bool recurse = true);
+};
+
+
+/// Helper struct for computing page table indices of a given address.
+struct page_table_indices
+{
+	page_table_indices(uint64_t v = 0);
+
+	uintptr_t addr() const;
+
+	inline operator uintptr_t() const { return addr(); }
+
+	/// Get the index for a given level of page table.
+	uint64_t & operator[](int i) { return index[i]; }
+	uint64_t operator[](int i) const { return index[i]; }
+	uint64_t & operator[](page_table::level i) { return index[static_cast<unsigned>(i)]; }
+	uint64_t operator[](page_table::level i) const { return index[static_cast<unsigned>(i)]; }
+	uint64_t index[4]; ///< Indices for each level of tables.
+};
+
+bool operator==(const page_table_indices &l, const page_table_indices &r);
+

src/kernel/process.cpp

@@ -1,28 +1,168 @@
+#include "kutil/heap_allocator.h"
+#include "cpu.h"
+#include "debug.h"
+#include "log.h"
 #include "process.h"
+#include "scheduler.h"
 
+extern "C" void task_fork_return_thunk();
+extern kutil::heap_allocator g_kernel_heap; // TODO: this is a bad hack to get access to the heap
 
 void
+process::exit(uint32_t code)
+{
+	return_code = code;
+	flags -= process_flags::running;
+	page_manager::get()->delete_process_map(pml4);
+}
+
+pid_t
+process::fork()
+{
+	auto &sched = scheduler::get();
+	auto *child = sched.create_process();
+	kassert(child, "process::fork() got null child");
+
+	child->ppid = pid;
+	child->flags =
+		process_flags::running |
+		process_flags::ready;
+
+	sched.m_runlists[child->priority].push_back(child);
+
+	child->pml4 = page_manager::get()->copy_table(pml4);
+	kassert(child->pml4, "process::fork() got null pml4");
+
+	child->rsp3 = bsp_cpu_data.rsp3;
+	child->setup_kernel_stack();
+
+	log::debug(logs::task, "Copied process %d to %d",
+			pid, child->pid);
+
+	log::debug(logs::task, "    PML4 %016lx", child->pml4);
+	log::debug(logs::task, "    RSP3 %016lx", child->rsp3);
+	log::debug(logs::task, "    RSP0 %016lx", child->rsp0);
+
+	// Initialize a new empty stack with a fake saved state
+	// for returning out of syscall_handler_prelude
+	size_t ret_seg_size = sizeof(uintptr_t) * 8;
+	child->rsp -= ret_seg_size;
+
+	void *this_ret_seg =
+		reinterpret_cast<void*>(rsp0 - ret_seg_size);
+	void *child_ret_seg =
+		reinterpret_cast<void*>(child->rsp);
+	kutil::memcpy(child_ret_seg, this_ret_seg, ret_seg_size);
+
+	child->add_fake_task_return(
+		reinterpret_cast<uintptr_t>(task_fork_return_thunk));
+
+	log::debug(logs::task, "    RSP  %016lx", child->rsp);
+
+	return child->pid;
+}
+
+void *
+process::setup_kernel_stack()
+{
+	constexpr unsigned null_frame_entries = 2;
+	constexpr size_t null_frame_size = null_frame_entries * sizeof(uint64_t);
+
+	void *stack_bottom = g_kernel_heap.allocate(initial_stack_size);
+	kutil::memset(stack_bottom, 0, initial_stack_size);
+
+	log::debug(logs::memory, "Created kernel stack at %016lx size 0x%lx",
+			stack_bottom, initial_stack_size);
+
+	void *stack_top =
+		kutil::offset_pointer(stack_bottom,
+				initial_stack_size - null_frame_size);
+
+	uint64_t *null_frame = reinterpret_cast<uint64_t*>(stack_top);
+	for (unsigned i = 0; i < null_frame_entries; ++i)
+		null_frame[i] = 0;
+
+	kernel_stack_size = initial_stack_size;
+	kernel_stack = reinterpret_cast<uintptr_t>(stack_bottom);
+	rsp0 = reinterpret_cast<uintptr_t>(stack_top);
+	rsp = rsp0;
+
+	return stack_top;
+}
+
+void
+process::add_fake_task_return(uintptr_t rip)
+{
+	rsp -= sizeof(uintptr_t) * 7;
+	uintptr_t *stack = reinterpret_cast<uintptr_t*>(rsp);
+
+	stack[6] = rip;        // return rip
+	stack[5] = rsp0;       // rbp
+	stack[4] = 0xbbbbbbbb; // rbx
+	stack[3] = 0x12121212; // r12
+	stack[2] = 0x13131313; // r13
+	stack[1] = 0x14141414; // r14
+	stack[0] = 0x15151515; // r15
+}
+
+bool
 process::wait_on_signal(uint64_t sigmask)
 {
 	waiting = process_wait::signal;
 	waiting_info = sigmask;
 	flags -= process_flags::ready;
+	return true;
 }
 
-void
+bool
 process::wait_on_child(uint32_t pid)
 {
 	waiting = process_wait::child;
 	waiting_info = pid;
 	flags -= process_flags::ready;
+	return true;
 }
 
-void
+bool
 process::wait_on_time(uint64_t time)
 {
 	waiting = process_wait::time;
 	waiting_info = time;
 	flags -= process_flags::ready;
+	return true;
+}
+
+bool
+process::wait_on_send(uint32_t target_id)
+{
+	scheduler &s = scheduler::get();
+	process *target = s.get_process_by_id(target_id);
+	if (!target) return false;
+
+	if (!target->wake_on_receive(this)) {
+		waiting = process_wait::send;
+		waiting_info = target_id;
+		flags -= process_flags::ready;
+	}
+
+	return true;
+}
+
+bool
+process::wait_on_receive(uint32_t source_id)
+{
+	scheduler &s = scheduler::get();
+	process *source = s.get_process_by_id(source_id);
+	if (!source) return false;
+
+	if (!source->wake_on_send(this)) {
+		waiting = process_wait::receive;
+		waiting_info = source_id;
+		flags -= process_flags::ready;
+		return true;
+	}
+
+	return false;
 }
 
 bool
@@ -49,7 +189,6 @@ process::wake_on_child(process *child)
 	return true;
 }
 
-
 bool
 process::wake_on_time(uint64_t now)
 {
@@ -62,3 +201,28 @@ process::wake_on_time(uint64_t now)
 	return true;
 }
 
+bool
+process::wake_on_send(process *target)
+{
+	if (waiting != process_wait::send ||
+		waiting_info != target->pid)
+		return false;
+
+	waiting = process_wait::none;
+	flags += process_flags::ready;
+	return true;
+}
+
+
+bool
+process::wake_on_receive(process *source)
+{
+	if (waiting != process_wait::receive ||
+		waiting_info != source->pid)
+		return false;
+
+	waiting = process_wait::none;
+	flags += process_flags::ready;
+	return true;
+}
+

src/kernel/process.h

@@ -7,6 +7,9 @@
 #include "kutil/linked_list.h"
 #include "page_manager.h"
 
+typedef int32_t pid_t;
+struct cpu_state;
+
 
 enum class process_flags : uint32_t
 {
@@ -25,14 +28,27 @@ enum class process_wait : uint8_t
 	none,
 	signal,
 	child,
-	time
+	time,
+	send,
+	receive
 };
 
-/// A process
+/// A process.
+/// 
 struct process
 {
-	uint32_t pid;
-	uint32_t ppid;
+	static const size_t initial_stack_size = 0x1000;
+
+	// Fields used by assembly routines go first.  If you change any of these,
+	// be sure to change the assembly definitions in 'tasking.inc'
+	uintptr_t rsp;
+	uintptr_t rsp0;
+	uintptr_t rsp3;
+	page_table *pml4;
+	// End of assembly fields
+
+	pid_t pid;
+	pid_t ppid;
 
 	process_flags flags;
 
@@ -47,20 +63,44 @@ struct process
 
 	uint32_t reserved1;
 
-	uintptr_t rsp;
-	page_table *pml4;
+	uintptr_t kernel_stack;
+	size_t kernel_stack_size;
+
+	/// Terminate this process.
+	/// \arg code   The return code to exit with.
+	void exit(unsigned code);
+
+	/// Copy this process.
+	/// \returns    Returns the child's pid to the parent, and
+	///             0 to the child.
+	pid_t fork();
 
 	/// Unready this process until it gets a signal
 	/// \arg sigmask  A bitfield of signals to wake on
-	void wait_on_signal(uint64_t sigmask);
+	/// \returns  Whether the process should be rescheduled
+	bool wait_on_signal(uint64_t sigmask);
 
 	/// Unready this process until a child exits
 	/// \arg pid  PID of the child to wait for, or 0 for any
-	void wait_on_child(uint32_t pid);
+	/// \returns  Whether the process should be rescheduled
+	bool wait_on_child(uint32_t pid);
 
 	/// Unready this process until after the given time
 	/// \arg time  The time after which to wake
-	void wait_on_time(uint64_t time);
+	/// \returns  Whether the process should be rescheduled
+	bool wait_on_time(uint64_t time);
+
+	/// Try to send to the target process, becoming unready if it
+	/// is not waiting on receive.
+	/// \arg target_id  The process to send to
+	/// \returns  Whether the process should be rescheduled
+	bool wait_on_send(uint32_t target_id);
+
+	/// Try to receive from one or more processes, becoming unready
+	/// if none of them are waiting on a send to this process.
+	/// \arg source_id  The process to receive from
+	/// \returns  Whether the process should be rescheduled
+	bool wait_on_receive(uint32_t source_id);
 
 	/// If this process is waiting on the given signal, wake it
 	/// \argument signal  The signal sent to the process
@@ -76,6 +116,29 @@ struct process
 	/// \argument now  The current time
 	/// \returns  True if this wake was handled
 	bool wake_on_time(uint64_t now);
+
+	/// If this process is waiting to send to this target, wake it
+	/// \argument target  The target process
+	/// \returns  True if this wake was handled
+	bool wake_on_send(process *target);
+
+	/// If this process is waiting to receieve from this source, wake it
+	/// \argument source  The process that is sending
+	/// \returns  True if this wake was handled
+	bool wake_on_receive(process *source);
+
+private:
+	friend class scheduler;
+
+	/// Set up a new empty kernel stack for this process. Sets rsp0 on this
+	/// process object, but also returns it.
+	/// \returns   The new rsp0 as a pointer
+	void * setup_kernel_stack();
+
+	/// Initialize this process' kenrel stack with a fake return segment for
+	/// returning out of task_switch.
+	/// \arg rip  The rip to return to
+	void add_fake_task_return(uintptr_t rip);
 };
 
 using process_list = kutil::linked_list<process>;

src/kernel/push_all.inc

@@ -1,48 +1,82 @@
-%macro push_all_and_segments 0
-	push rax
-	push rcx
-	push rdx
-	push rbx
-	push rbp
-	push rsi
-	push rdi
+struc REGS
+.r15           resq 1  ; 0x00
+.r14           resq 1  ; 0x08
+.r13           resq 1  ; 0x10
+.r12           resq 1  ; 0x18
+.r11           resq 1  ; 0x20
+.r10           resq 1  ; 0x28
+.r9            resq 1  ; 0x30
+.r8            resq 1  ; 0x38
 
-	push r8
-	push r9
-	push r10
-	push r11
-	push r12
-	push r13
-	push r14
-	push r15
+.rdi           resq 1  ; 0x40
+.rsi           resq 1  ; 0x48
+.rbp           resq 1  ; 0x50
+.rbx           resq 1  ; 0x58
+.rdx           resq 1  ; 0x60
+.rcx           resq 1  ; 0x68
+.rax           resq 1  ; 0x70
 
-	mov ax, ds
-	push rax
+.int           resq 1  ; 0x78
+.err           resq 1  ; 0x80
+.rip           resq 1  ; 0x88
+.cs3           resq 1  ; 0x90
+.rflags        resq 1  ; 0x98
+.rsp3          resq 1  ; 0xa0
+.ss3           resq 1  ; 0xa8
+endstruc
+
+regs_total_size equ 0xb0
+regs_extra_size equ 0x78
+
+%macro push_all 0
+	sub rsp, regs_extra_size
+
+	mov [rsp + REGS.rax], rax
+	mov [rsp + REGS.rcx], rcx
+	mov [rsp + REGS.rdx], rdx
+	mov [rsp + REGS.rbx], rbx
+	mov [rsp + REGS.rbp], rbp
+	mov [rsp + REGS.rsi], rsi
+	mov [rsp + REGS.rdi], rdi
+
+	mov [rsp + REGS.r8 ], r8
+	mov [rsp + REGS.r9 ], r9
+	mov [rsp + REGS.r10], r10
+	mov [rsp + REGS.r11], r11
+	mov [rsp + REGS.r12], r12
+	mov [rsp + REGS.r13], r13
+	mov [rsp + REGS.r14], r14
+	mov [rsp + REGS.r15], r15
 %endmacro
 
-%macro pop_all_and_segments 0
-	pop rax
-	mov ds, ax
-	mov es, ax
-	mov fs, ax
-	mov gs, ax
+%macro pop_all 0
+	mov rax, [rsp + REGS.rax]
+	mov rcx, [rsp + REGS.rcx]
+	mov rdx, [rsp + REGS.rdx]
+	mov rbx, [rsp + REGS.rbx]
+	mov rbp, [rsp + REGS.rbp]
+	mov rsi, [rsp + REGS.rsi]
+	mov rdi, [rsp + REGS.rdi]
 
-	pop r15
-	pop r14
-	pop r13
-	pop r12
-	pop r11
-	pop r10
-	pop r9
-	pop r8
+	mov r8,  [rsp + REGS.r8 ]
+	mov r9,  [rsp + REGS.r9 ]
+	mov r10, [rsp + REGS.r10]
+	mov r11, [rsp + REGS.r11]
+	mov r12, [rsp + REGS.r12]
+	mov r13, [rsp + REGS.r13]
+	mov r14, [rsp + REGS.r14]
+	mov r15, [rsp + REGS.r15]
 
-	pop rdi
-	pop rsi
-	pop rbp
-	pop rbx
-	pop rdx
-	pop rcx
-	pop rax
+	add rsp, regs_extra_size
+%endmacro
+
+%macro check_swap_gs 0
+	mov rax, [rsp+0x90]
+	and rax, 0x03 ; mask out the RPL
+	cmp rax, 0x03
+	jne %%noswapgs
+	swapgs
+%%noswapgs:
 %endmacro
 
 ; vim: ft=asm

src/kernel/scheduler.cpp

@@ -1,9 +1,11 @@
 #include "apic.h"
 #include "console.h"
 #include "cpu.h"
+#include "debug.h"
 #include "gdt.h"
 #include "interrupts.h"
 #include "io.h"
+#include "kernel_memory.h"
 #include "log.h"
 #include "msr.h"
 #include "page_manager.h"
@@ -12,30 +14,35 @@
 #include "elf/elf.h"
 #include "kutil/assert.h"
 
-scheduler scheduler::s_instance(nullptr);
+using memory::initial_stack;
+
+scheduler scheduler::s_instance(nullptr, kutil::allocator::invalid);
 
-const int stack_size = 0x1000;
 const uint64_t rflags_noint = 0x002;
 const uint64_t rflags_int = 0x202;
 
 extern "C" {
 	void ramdisk_process_loader();
-	void load_process(const void *image_start, size_t bytes, process *proc, cpu_state state);
+	uintptr_t load_process_image(const void *image_start, size_t bytes, process *proc);
 };
 
-scheduler::scheduler(lapic *apic) :
+extern uint64_t idle_stack_end;
+
+scheduler::scheduler(lapic *apic, kutil::allocator &alloc) :
 	m_apic(apic),
-	m_next_pid(1)
+	m_next_pid(1),
+	m_process_allocator(alloc)
 {
 	auto *idle = m_process_allocator.pop();
-
 	uint8_t last_pri = num_priorities - 1;
 
 	// The kernel idle task, also the thread we're in now
 	idle->pid = 0;
 	idle->ppid = 0;
 	idle->priority = last_pri;
-	idle->rsp = 0; // This will get set when we switch away
+	idle->rsp = 0;  // This will get set when we switch away
+	idle->rsp3 = 0; // Never used for the idle task
+	idle->rsp0 = reinterpret_cast<uintptr_t>(&idle_stack_end);
 	idle->pml4 = page_manager::get_pml4();
 	idle->quanta = process_quanta;
 	idle->flags =
@@ -45,10 +52,13 @@ scheduler::scheduler(lapic *apic) :
 
 	m_runlists[last_pri].push_back(idle);
 	m_current = idle;
+
+	bsp_cpu_data.rsp0 = idle->rsp0;
+	bsp_cpu_data.tcb = idle;
 }
 
-void
-load_process(const void *image_start, size_t bytes, process *proc, cpu_state state)
+uintptr_t
+load_process_image(const void *image_start, size_t bytes, process *proc)
 {
 	// We're now in the process space for this process, allocate memory for the
 	// process code and load it
@@ -58,7 +68,7 @@ load_process(const void *image_start, size_t bytes, process *proc, cpu_state sta
 
 	// TODO: Handle bad images gracefully
 	elf::elf image(image_start, bytes);
-	kassert(image.valid(), "Invalid ELF passed to load_process");
+	kassert(image.valid(), "Invalid ELF passed to load_process_image");
 
 	const unsigned program_count = image.program_count();
 	for (unsigned i = 0; i < program_count; ++i) {
@@ -67,7 +77,7 @@ load_process(const void *image_start, size_t bytes, process *proc, cpu_state sta
 		if (header->type != elf::segment_type::load)
 			continue;
 
-		uintptr_t aligned = header->vaddr & ~(page_manager::page_size - 1);
+		uintptr_t aligned = header->vaddr & ~(memory::frame_size - 1);
 		size_t size = (header->vaddr + header->mem_size) - aligned;
 		size_t pages = page_manager::page_count(size);
 
@@ -80,7 +90,7 @@ load_process(const void *image_start, size_t bytes, process *proc, cpu_state sta
 		void *mapped = pager->map_pages(aligned, pages, true);
 		kassert(mapped, "Tried to map userspace pages and failed!");
 
-		kutil::memset(mapped, 0, pages * page_manager::page_size);
+		kutil::memset(mapped, 0, pages * memory::frame_size);
 	}
 
 	const unsigned section_count = image.section_count();
@@ -99,15 +109,29 @@ load_process(const void *image_start, size_t bytes, process *proc, cpu_state sta
 		kutil::memcpy(dest, src, header->size);
 	}
 
-	state.rip = image.entrypoint();
 	proc->flags &= ~process_flags::loading;
 
-	log::debug(logs::task, "  Loaded! New process rip: %016lx", state.rip);
+	uintptr_t entrypoint = image.entrypoint();
+	log::debug(logs::task, "  Loaded! New process rip: %016lx", entrypoint);
+	return entrypoint;
+}
+
+process_node *
+scheduler::create_process(pid_t pid)
+{
+	kassert(pid <= 0, "Cannot specify a positive pid in create_process");
+
+	auto *proc = m_process_allocator.pop();
+	proc->pid = pid ? pid : m_next_pid++;
+	proc->priority = default_priority;
+	return proc;
 }
 
 void
-scheduler::create_process(const char *name, const void *data, size_t size)
+scheduler::load_process(const char *name, const void *data, size_t size)
 {
+	auto *proc = create_process();
+
 	uint16_t kcs = (1 << 3) | 0; // Kernel CS is GDT entry 1, ring 0
 	uint16_t cs = (5 << 3) | 3;  // User CS is GDT entry 5, ring 3
 
@@ -115,46 +139,28 @@ scheduler::create_process(const char *name, const void *data, size_t size)
 	uint16_t ss = (4 << 3) | 3;  // User SS is GDT entry 4, ring 3
 
 	// Set up the page tables - this also allocates an initial user stack
-	page_table *pml4 = page_manager::get()->create_process_map();
+	proc->pml4 = page_manager::get()->create_process_map();
 
-	// Create a one-page kernel stack space
-	void *stack0 = kutil::malloc(stack_size);
-	kutil::memset(stack0, 0, stack_size);
+	// Create an initial kernel stack space
+	void *sp0 = proc->setup_kernel_stack();
+	uintptr_t *stack = reinterpret_cast<uintptr_t *>(sp0) - 7;
 
-	// Stack grows down, point to the end
-	void *sp0 = kutil::offset_pointer(stack0, stack_size);
+	// Pass args to ramdisk_process_loader on the stack
+	stack[0] = reinterpret_cast<uintptr_t>(data);
+	stack[1] = reinterpret_cast<uintptr_t>(size);
+	stack[2] = reinterpret_cast<uintptr_t>(proc);
 
-	cpu_state *state = reinterpret_cast<cpu_state *>(sp0) - 1;
+	proc->rsp = reinterpret_cast<uintptr_t>(stack);
+	proc->add_fake_task_return(
+			reinterpret_cast<uintptr_t>(ramdisk_process_loader));
 
-	// Highest state in the stack is the process' kernel stack for the loader
-	// to iret to:
-	state->ds = state->ss = ss;
-	state->cs = cs;
-	state->rflags = rflags_int;
-	state->rip = 0; // to be filled by the loader
-	state->user_rsp = page_manager::initial_stack;
+	// Arguments for iret - rip will be pushed on before these
+	stack[3] = cs;
+	stack[4] = rflags_int;
+	stack[5] = initial_stack;
+	stack[6] = ss;
 
-	// Next state in the stack is the loader's kernel stack. The scheduler will
-	// iret to this which will kick off the loading:
-	cpu_state *loader_state = reinterpret_cast<cpu_state *>(sp0) - 2;
-
-	loader_state->ds = loader_state->ss = kss;
-	loader_state->cs = kcs;
-	loader_state->rflags = rflags_noint;
-	loader_state->rip = reinterpret_cast<uint64_t>(ramdisk_process_loader);
-	loader_state->user_rsp = reinterpret_cast<uint64_t>(state);
-
-	loader_state->rax = reinterpret_cast<uint64_t>(data);
-	loader_state->rbx = size;
-
-	uint16_t pid = m_next_pid++;
-	auto *proc = m_process_allocator.pop();
-
-	proc->pid = pid;
-	proc->ppid = 0; // TODO
-	proc->priority = default_priority;
-	proc->rsp = reinterpret_cast<uintptr_t>(loader_state);
-	proc->pml4 = pml4;
+	proc->rsp3 = initial_stack;
 	proc->quanta = process_quanta;
 	proc->flags =
 		process_flags::running |
@@ -163,17 +169,44 @@ scheduler::create_process(const char *name, const void *data, size_t size)
 
 	m_runlists[default_priority].push_back(proc);
 
-	loader_state->rcx = reinterpret_cast<uint64_t>(proc);
-
 	log::debug(logs::task, "Creating process %s: pid %d  pri %d", name, proc->pid, proc->priority);
-	log::debug(logs::task, "     RSP0 %016lx", state);
-	log::debug(logs::task, "     PML4 %016lx", pml4);
+	log::debug(logs::task, "     RSP  %016lx", proc->rsp);
+	log::debug(logs::task, "     RSP0 %016lx", proc->rsp0);
+	log::debug(logs::task, "     PML4 %016lx", proc->pml4);
+}
+
+void
+scheduler::create_kernel_task(pid_t pid, void (*task)())
+{
+	auto *proc = create_process(pid);
+
+	uint16_t kcs = (1 << 3) | 0; // Kernel CS is GDT entry 1, ring 0
+	uint16_t kss = (2 << 3) | 0; // Kernel SS is GDT entry 2, ring 0
+
+	// Create an initial kernel stack space
+	proc->setup_kernel_stack();
+	proc->add_fake_task_return(
+			reinterpret_cast<uintptr_t>(task));
+
+	proc->pml4 = page_manager::get()->get_kernel_pml4();
+	proc->quanta = process_quanta;
+	proc->flags =
+		process_flags::running |
+		process_flags::ready;
+
+	m_runlists[default_priority].push_back(proc);
+
+	log::debug(logs::task, "Creating kernel task: pid %d  pri %d", proc->pid, proc->priority);
+	log::debug(logs::task, "    RSP0 %016lx", proc->rsp0);
+	log::debug(logs::task, "     RSP %016lx", proc->rsp);
+	log::debug(logs::task, "    PML4 %016lx", proc->pml4);
 }
 
 void
 scheduler::start()
 {
 	log::info(logs::task, "Starting scheduler.");
+	wrmsr(msr::ia32_gs_base, reinterpret_cast<uintptr_t>(&bsp_cpu_data));
 	m_tick_count = m_apic->enable_timer(isr::isrTimer, quantum_micros, false);
 }
 
@@ -216,33 +249,32 @@ void scheduler::prune(uint64_t now)
 	while (proc) {
 		bool ready = proc->flags && process_flags::ready;
 		ready |= proc->wake_on_time(now);
-		if (!ready) {
-			proc = proc->next();
-			continue;
-		}
 
 		auto *remove = proc;
 		proc = proc->next();
+		if (!ready) continue;
+
 		m_blocked.remove(remove);
 		m_runlists[remove->priority].push_front(remove);
 	}
 }
 
-uintptr_t
-scheduler::schedule(uintptr_t rsp0)
+void
+scheduler::schedule()
 {
-
 	// TODO: lol a real clock
 	static uint64_t now = 0;
-	prune(++now);
 
-	m_current->rsp = rsp0;
+	pid_t lastpid = m_current->pid;
+
 	m_runlists[m_current->priority].remove(m_current);
-
-	if (m_current->flags && process_flags::ready)
+	if (m_current->flags && process_flags::ready) {
 		m_runlists[m_current->priority].push_back(m_current);
-	else
+	} else {
 		m_blocked.push_back(m_current);
+	}
+
+	prune(++now);
 
 	uint8_t pri = 0;
 	while (m_runlists[pri].empty()) {
@@ -251,32 +283,24 @@ scheduler::schedule(uintptr_t rsp0)
 	}
 
 	m_current = m_runlists[pri].pop_front();
-	rsp0 = m_current->rsp;
 
-	// Set rsp0 to after the end of the about-to-be-popped cpu state
-	tss_set_stack(0, rsp0 + sizeof(cpu_state));
-	wrmsr(msr::ia32_kernel_gs_base, rsp0);
+	if (lastpid != m_current->pid) {
+		task_switch(m_current);
 
-	// Swap page tables
-	page_table *pml4 = m_current->pml4;
-	page_manager::set_pml4(pml4);
-
-	bool loading = m_current->flags && process_flags::loading;
-	log::debug(logs::task, "Scheduler switched to process %d, priority %d%s.",
-			m_current->pid, m_current->priority, loading ? " (loading)" : "");
-
-	return rsp0;
+		bool loading = m_current->flags && process_flags::loading;
+		log::debug(logs::task, "Scheduler switched to process %d, priority %d%s.",
+				m_current->pid, m_current->priority, loading ? " (loading)" : "");
+	}
 }
 
-uintptr_t
-scheduler::tick(uintptr_t rsp0)
+void
+scheduler::tick()
 {
 	if (--m_current->quanta == 0) {
 		m_current->quanta = process_quanta;
-		rsp0 = schedule(rsp0);
+		schedule();
 	}
 	m_apic->reset_timer(m_tick_count);
-	return rsp0;
 }
 
 process_node *

src/kernel/scheduler.h

@@ -3,6 +3,7 @@
 /// The task scheduler and related definitions
 
 #include <stdint.h>
+#include "kutil/allocator.h"
 #include "kutil/slab_allocator.h"
 #include "process.h"
 
@@ -10,7 +11,9 @@ class lapic;
 struct page_table;
 struct cpu_state;
 
-extern "C" uintptr_t isr_handler(uintptr_t, cpu_state);
+extern "C" void isr_handler(cpu_state*);
+extern "C" void task_switch(process *next);
+extern "C" void task_fork(process *child);
 
 
 /// The task scheduler
@@ -21,29 +24,33 @@ public:
 	static const uint8_t default_priority = num_priorities / 2;
 
 	/// How long the timer quantum is
-	static const uint64_t quantum_micros = 100000;
+	static const uint64_t quantum_micros = 1000;
 
 	/// How many quantums a process gets before being rescheduled
-	static const uint16_t process_quanta = 10;
+	static const uint16_t process_quanta = 100;
 
 	/// Constructor.
 	/// \arg apic  Pointer to the local APIC object
-	scheduler(lapic *apic);
+	/// \arg alloc Allocator to use for TCBs
+	scheduler(lapic *apic, kutil::allocator &alloc);
 
 	/// Create a new process from a program image in memory.
 	/// \arg name  Name of the program image
 	/// \arg data  Pointer to the image data
 	/// \arg size  Size of the program image, in bytes
-	void create_process(const char *name, const void *data, size_t size);
+	void load_process(const char *name, const void *data, size_t size);
+
+	/// Create a new kernel task
+	/// \arg pid   Pid to use for this task, must be negative
+	/// \arg proc  Function to run as a kernel task
+	void create_kernel_task(pid_t pid, void (*task)());
 
 	/// Start the scheduler working. This may involve starting
 	/// timer interrupts or other preemption methods.
 	void start();
 
 	/// Run the scheduler, possibly switching to a new task
-	/// \arg rsp0  The stack pointer of the current interrupt handler
-	/// \returns   The stack pointer to switch to
-	uintptr_t schedule(uintptr_t rsp0);
+	void schedule();
 
 	/// Get the current process.
 	/// \returns  A pointer to the current process' process struct
@@ -59,13 +66,18 @@ public:
 	static scheduler & get() { return s_instance; }
 
 private:
-	friend uintptr_t syscall_dispatch(uintptr_t, const cpu_state &);
-	friend uintptr_t isr_handler(uintptr_t, cpu_state);
+	friend uintptr_t syscall_dispatch(uintptr_t, cpu_state &);
+	friend void isr_handler(cpu_state*);
+	friend class process;
+
+	/// Create a new process object. This process will have its pid
+	/// set but nothing else.
+	/// \arg pid  The pid to give the process (0 for automatic)
+	/// \returns  The new process object
+	process_node * create_process(pid_t pid = 0);
 
 	/// Handle a timer tick
-	/// \arg rsp0  The stack pointer of the current interrupt handler
-	/// \returns   The stack pointer to switch to
-	uintptr_t tick(uintptr_t rsp0);
+	void tick();
 
 	void prune(uint64_t now);
 

src/kernel/syscall.cpp

@@ -1,16 +1,125 @@
 #include "console.h"
 #include "cpu.h"
 #include "debug.h"
+#include "log.h"
 #include "msr.h"
 #include "process.h"
 #include "scheduler.h"
 #include "syscall.h"
 
 extern "C" {
-	void _halt();
+	void syscall_invalid(uint64_t call);
 	void syscall_handler_prelude();
 }
 
+namespace syscalls {
+
+
+void send() {}
+void receive() {}
+
+} // namespace syscalls
+
+uintptr_t syscall_registry[static_cast<unsigned>(syscall::MAX)];
+const char * syscall_names[static_cast<unsigned>(syscall::MAX)];
+
+void
+syscall_invalid(uint64_t call)
+{
+	console *cons = console::get();
+	cons->set_color(9);
+	cons->printf("\nReceived unknown syscall: %02x\n", call);
+
+	const unsigned num_calls =
+		static_cast<unsigned>(syscall::MAX);
+
+	cons->printf("  Known syscalls:\n");
+		cons->printf("          invalid %016lx\n", syscall_invalid);
+
+	for (unsigned i = 0; i < num_calls; ++i) {
+		const char *name = syscall_names[i];
+		uintptr_t handler = syscall_registry[i];
+		if (name)
+			cons->printf("    %02x %10s %016lx\n", i, name, handler);
+	}
+
+	cons->set_color();
+	_halt();
+}
+
+/*
+void
+syscall_dispatch(cpu_state *regs)
+{
+	console *cons = console::get();
+	syscall call = static_cast<syscall>(regs->rax);
+
+	auto &s = scheduler::get();
+	auto *p = s.current();
+
+	switch (call) {
+	case syscall::noop:
+		break;
+
+	case syscall::debug:
+		cons->set_color(11);
+		cons->printf("\nProcess %d: Received DEBUG syscall\n", p->pid);
+		cons->set_color();
+		print_regs(*regs);
+		cons->printf("\n         Syscall enters: %8d\n", __counter_syscall_enter);
+		cons->printf("         Syscall sysret: %8d\n", __counter_syscall_sysret);
+		break;
+
+	case syscall::send:
+		{
+			pid_t target = regs->rdi;
+			uintptr_t data = regs->rsi;
+
+			cons->set_color(11);
+			cons->printf("\nProcess %d: Received SEND syscall, target %d, data %016lx\n", p->pid, target, data);
+			cons->set_color();
+
+			if (p->wait_on_send(target))
+				s.schedule();
+		}
+		break;
+
+	case syscall::receive:
+		{
+			pid_t source = regs->rdi;
+			uintptr_t data = regs->rsi;
+
+			cons->set_color(11);
+			cons->printf("\nProcess %d: Received RECEIVE syscall, source %d, dat %016lx\n", p->pid, source, data);
+			cons->set_color();
+
+			if (p->wait_on_receive(source))
+				s.schedule();
+		}
+		break;
+
+	case syscall::fork:
+		{
+			cons->set_color(11);
+			cons->printf("\nProcess %d: Received FORK syscall\n", p->pid);
+			cons->set_color();
+
+			pid_t pid = p->fork(regs);
+			cons->printf("\n   fork returning %d\n", pid);
+			regs->rax = pid;
+		}
+		break;
+
+	default:
+		cons->set_color(9);
+		cons->printf("\nReceived unknown syscall: %02x\n", call);
+		cons->set_color();
+		_halt();
+		break;
+	}
+}
+*/
+
 void
 syscall_enable()
 {
@@ -32,65 +141,20 @@ syscall_enable()
 
 	// IA32_FMASK - FLAGS mask inside syscall
 	wrmsr(msr::ia32_fmask, 0x200);
-}
 
-uintptr_t
-syscall_dispatch(uintptr_t return_rsp, const cpu_state &regs)
-{
-	console *cons = console::get();
-	syscall call = static_cast<syscall>(regs.rax);
+	static constexpr unsigned num_calls =
+		static_cast<unsigned>(syscall::MAX);
 
-	switch (call) {
-	case syscall::noop:
-		break;
-
-	case syscall::debug:
-		cons->set_color(11);
-		cons->printf("\nReceived DEBUG syscall\n");
-		cons->set_color();
-		print_regs(regs);
-		break;
-
-	case syscall::message:
-		cons->set_color(11);
-		cons->printf("\nReceived MESSAGE syscall\n");
-		cons->set_color();
-		break;
-
-	case syscall::pause:
-		{
-			cons->set_color(11);
-
-			auto &s = scheduler::get();
-			auto *p = s.current();
-			p->wait_on_signal(-1ull);
-			cons->printf("\nReceived PAUSE syscall\n");
-			return_rsp = s.tick(return_rsp);
-			cons->set_color();
-		}
-		break;
-
-	case syscall::sleep:
-		{
-			cons->set_color(11);
-
-			auto &s = scheduler::get();
-			auto *p = s.current();
-			p->wait_on_time(regs.rbx);
-			cons->printf("\nReceived SLEEP syscall\n");
-			return_rsp = s.tick(return_rsp);
-			cons->set_color();
-		}
-		break;
-
-	default:
-		cons->set_color(9);
-		cons->printf("\nReceived unknown syscall: %02x\n", call);
-		cons->set_color();
-		_halt();
-		break;
+	for (unsigned i = 0; i < num_calls; ++i) {
+		syscall_registry[i] = reinterpret_cast<uintptr_t>(syscall_invalid);
+		syscall_names[i] = nullptr;
 	}
 
-	return return_rsp;
+#define SYSCALL(id, name, result, ...) \
+	syscall_registry[id] = reinterpret_cast<uintptr_t>(syscalls::name); \
+	syscall_names[id] = #name; \
+	static_assert( id <= num_calls, "Syscall " #name " has id > syscall::MAX" );
+#include "syscalls.inc"
+#undef SYSCALL
 }
 

src/kernel/syscall.h

@@ -6,15 +6,20 @@ struct cpu_state;
 
 enum class syscall : uint64_t
 {
-	noop,
-	debug,
-	message,
-	pause,
-	sleep,
+#define SYSCALL(id, name, result, ...) name = id,
+#include "syscalls.inc"
+#undef SYSCALL
 
-	last_syscall
+	// Maximum syscall id. If you change this, also change
+	// MAX_SYSCALLS in syscall.s
+	MAX = 0x40
 };
 
 void syscall_enable();
-uintptr_t syscall_dispatch(uintptr_t, const cpu_state &);
 
+namespace syscalls
+{
+#define SYSCALL(id, name, result, ...) result name (__VA_ARGS__);
+#include "syscalls.inc"
+#undef SYSCALL
+}

src/kernel/syscall.s

@@ -1,24 +1,62 @@
 %include "push_all.inc"
+%include "tasking.inc"
+
+; Make sure to keep MAX_SYSCALLS in sync with
+; syscall::MAX in syscall.h
+MAX_SYSCALLS equ 0x40
+
+extern __counter_syscall_enter
+extern __counter_syscall_sysret
+
+extern syscall_registry
+extern syscall_invalid
 
-extern syscall_handler
 global syscall_handler_prelude
+global syscall_handler_prelude.return
 syscall_handler_prelude:
-	push 0		; ss, doesn't matter here
-	push rsp
-	pushf
-	push 0		; cs, doesn't matter here
-	push rcx	; user rip
-	push 0		; bogus interrupt
-	push 0		; bogus errorcode
-	push_all_and_segments
+	swapgs
+	mov [gs:CPU_DATA.rsp3], rsp
+	mov rsp, [gs:CPU_DATA.rsp0]
 
-	mov rdi, rsp
-	call syscall_handler
-	mov rsp, rax
+	push rcx
+	push rbp
+	mov rbp, rsp
 
-	pop_all_and_segments
-	add rsp, 16	; ignore bogus interrupt / error
-	pop rcx		; user rip
-	add rsp, 32 ; ignore cs, flags, rsp, ss
+	push rbx
+	push r11
+	push r12
+	push r13
+	push r14
+	push r15
 
+	inc qword [rel __counter_syscall_enter]
+
+	cmp rax, MAX_SYSCALLS
+	jle .ok_syscall
+
+	mov rdi, rax
+	call syscall_invalid
+
+.ok_syscall:
+	lea r11, [rel syscall_registry]
+	mov r11, [r11 + rax * 8]
+	call r11
+
+	inc qword [rel __counter_syscall_sysret]
+
+.return:
+	pop r15
+	pop r14
+	pop r13
+	pop r12
+	pop r11
+	pop rbx
+
+	pop rbp
+	pop rcx
+
+	mov [gs:CPU_DATA.rsp0], rsp
+	mov rsp, [gs:CPU_DATA.rsp3]
+
+	swapgs
 	o64 sysret

src/kernel/syscalls.inc

@@ -0,0 +1,12 @@
+SYSCALL(0x00, noop,     void)
+SYSCALL(0x01, exit,     void, int64_t)
+SYSCALL(0x02, getpid,   pid_t)
+SYSCALL(0x03, fork,     pid_t)
+
+SYSCALL(0x10, message,  void, const char *)
+
+SYSCALL(0x20, pause,    void)
+SYSCALL(0x21, sleep,    void, uint64_t)
+
+SYSCALL(0x30, send,     void)
+SYSCALL(0x31, receive,  void)

src/kernel/syscalls/exit.cpp

@@ -0,0 +1,17 @@
+#include "log.h"
+#include "scheduler.h"
+
+namespace syscalls {
+
+void
+exit(int64_t status)
+{
+	auto &s = scheduler::get();
+	auto *p = s.current();
+	log::debug(logs::syscall, "Process %d exiting with code %d", p->pid, status);
+
+	p->exit(status);
+	s.schedule();
+}
+
+} // namespace syscalls

src/kernel/syscalls/fork.cpp

@@ -0,0 +1,24 @@
+
+#include "log.h"
+#include "scheduler.h"
+
+namespace syscalls {
+
+pid_t
+fork()
+{
+	auto &s = scheduler::get();
+	auto *p = s.current();
+	pid_t ppid = p->pid;
+
+	log::debug(logs::syscall, "Process %d calling fork()", ppid);
+
+	pid_t pid = p->fork();
+
+	p = s.current();
+	log::debug(logs::syscall, "Process %d's fork: returning %d from process %d", ppid, pid, p->pid);
+
+	return pid;
+}
+
+} // namespace syscalls

src/kernel/syscalls/getpid.cpp

@@ -0,0 +1,14 @@
+#include "log.h"
+#include "scheduler.h"
+
+namespace syscalls {
+
+pid_t
+getpid()
+{
+	auto &s = scheduler::get();
+	auto *p = s.current();
+	return p->pid;
+}
+
+} // namespace syscalls

src/kernel/syscalls/message.cpp

@@ -0,0 +1,14 @@
+#include "log.h"
+#include "scheduler.h"
+
+namespace syscalls {
+
+void
+message(const char *message)
+{
+	auto &s = scheduler::get();
+	auto *p = s.current();
+	log::info(logs::syscall, "Message[%d]: %s", p->pid, message);
+}
+
+} // namespace syscalls

src/kernel/syscalls/noop.cpp

@@ -0,0 +1,14 @@
+#include "log.h"
+#include "scheduler.h"
+
+namespace syscalls {
+
+void
+noop()
+{
+	auto &s = scheduler::get();
+	auto *p = s.current();
+	log::debug(logs::syscall, "Process %d called noop syscall.", p->pid);
+}
+
+} // namespace syscalls

src/kernel/syscalls/pause.cpp

@@ -0,0 +1,15 @@
+#include "log.h"
+#include "scheduler.h"
+
+namespace syscalls {
+
+void
+pause()
+{
+	auto &s = scheduler::get();
+	auto *p = s.current();
+	p->wait_on_signal(-1ull);
+	s.schedule();
+}
+
+} // namespace syscalls

src/kernel/syscalls/sleep.cpp

@@ -0,0 +1,17 @@
+#include "log.h"
+#include "scheduler.h"
+
+namespace syscalls {
+
+void
+sleep(uint64_t til)
+{
+	auto &s = scheduler::get();
+	auto *p = s.current();
+	log::debug(logs::syscall, "Process %d sleeping until %d", p->pid, til);
+
+	p->wait_on_time(til);
+	s.schedule();
+}
+
+} // namespace syscalls

src/kernel/task.s

@@ -0,0 +1,63 @@
+%include "tasking.inc"
+
+extern g_tss
+global task_switch
+task_switch:
+	push rbp
+	mov rbp, rsp
+
+	; Save the rest of the callee-saved regs
+	push rbx
+	push r12
+	push r13
+	push r14
+	push r15
+
+	; Update previous task's TCB
+	mov rax, [gs:CPU_DATA.tcb]     ; rax: current task TCB
+	mov [rax + TCB.rsp], rsp
+
+	; Copy off saved user rsp
+	mov rcx, [gs:CPU_DATA.rsp3]    ; rcx: curretn task's saved user rsp
+	mov [rax + TCB.rsp3], rcx
+
+	; Install next task's TCB
+	mov [gs:CPU_DATA.tcb], rdi     ; rdi: next TCB (function param)
+	mov rsp, [rdi + TCB.rsp]       ; next task's stack pointer
+	mov rax, 0x0000007fffffffff
+	and rax, [rdi + TCB.pml4]      ; rax: next task's pml4 (phys portion of address)
+
+	; Update syscall/interrupt rsp
+	mov rcx, [rdi + TCB.rsp0]      ; rcx: top of next task's kernel stack
+	mov [gs:CPU_DATA.rsp0], rcx
+
+	; Update saved user rsp
+	mov rcx, [rdi + TCB.rsp3]      ; rcx: new task's saved user rsp
+	mov [gs:CPU_DATA.rsp3], rcx
+
+	lea rdx, [rel g_tss]           ; rdx: address of TSS
+	mov [rdx + TSS.rsp0], rcx
+
+	; check if we need to update CR3
+	mov rdx, cr3                   ; rdx: old CR3
+	cmp rax, rdx
+	je .no_cr3
+	mov cr3, rax
+.no_cr3:
+
+	pop r15
+	pop r14
+	pop r13
+	pop r12
+	pop rbx
+
+	pop rbp
+	ret
+
+
+extern syscall_handler_prelude.return
+global task_fork_return_thunk
+task_fork_return_thunk:
+	mov rax, 0
+	jmp syscall_handler_prelude.return
+

src/kernel/tasking.inc

@@ -0,0 +1,33 @@
+struc TCB
+.rsp:          resq 1
+.rsp0:		   resq 1
+.rsp3:		   resq 1
+.pml4:         resq 1
+endstruc
+
+struc CPU_DATA
+.rsp0:         resq 1
+.rsp3:         resq 1
+.tcb:          resq 1
+endstruc
+
+struc TSS
+.res0:         resd 1
+.rsp0:         resq 1
+.rsp1:         resq 1
+.rsp2:         resq 1
+.ist0:         resq 1
+.ist1:         resq 1
+.ist2:         resq 1
+.ist3:         resq 1
+.ist4:         resq 1
+.ist5:         resq 1
+.ist6:         resq 1
+.ist7:         resq 1
+.res1:         resq 1
+.res2:         resw 1
+.iomap:        resw 1
+endstruc
+
+
+; vim: ft=asm

src/libraries/initrd/include/initrd/initrd.h

@@ -49,7 +49,7 @@ class disk
 public:
 	/// Constructor.
 	/// \arg start  The start of the initrd in memory
-	disk(const void *start);
+	disk(const void *start, kutil::allocator &alloc);
 
 	/// Get the vector of files on the disk
 	const kutil::vector<file> & files() const { return m_files; }

src/libraries/initrd/initrd.cpp

@@ -23,7 +23,8 @@ file::executable() const {
 }
 
 
-disk::disk(const void *start)
+disk::disk(const void *start, kutil::allocator &alloc) :
+	m_files(alloc)
 {
 	auto *header = reinterpret_cast<const disk_header *>(start);
 	size_t length = header->length;

src/libraries/kutil/bip_buffer.cpp

@@ -0,0 +1,95 @@
+#include "kutil/assert.h"
+#include "kutil/bip_buffer.h"
+
+namespace kutil {
+
+bip_buffer::bip_buffer() :
+		m_start_a(0),
+		m_start_b(0),
+		m_size_a(0),
+		m_size_b(0),
+		m_buffer_size(0),
+		m_buffer(nullptr)
+	{}
+
+bip_buffer::bip_buffer(uint8_t *buffer, size_t size) :
+		m_start_a(0),
+		m_start_b(0),
+		m_size_a(0),
+		m_size_b(0),
+		m_buffer_size(size),
+		m_buffer(buffer)
+	{}
+
+size_t bip_buffer::reserve(size_t size, void **area)
+{
+	if (m_size_r) {
+		*area = nullptr;
+		return 0;
+	}
+
+	size_t remaining = 0;
+	if (m_size_b) {
+		// If B exists, we're appending there. Get space between
+		// the end of B and start of A.
+		remaining = m_start_a - m_start_b - m_size_b;
+		m_start_r = m_start_b + m_size_b;
+	} else {
+		// B doesn't exist, check the space both before and after A.
+		// If the end of A has enough room for this write, put it there.
+		remaining = m_buffer_size - m_start_a - m_size_a;
+		m_start_r = m_start_a + m_size_a;
+
+		// Otherwise use the bigger of the areas in front of and after A
+		if (remaining < size && m_start_a > remaining) {
+			remaining = m_start_a;
+			m_start_r = 0;
+		}
+	}
+
+	if (!remaining) {
+		*area = nullptr;
+		return 0;
+	}
+
+	m_size_r = (remaining < size) ? remaining : size;
+	*area = &m_buffer[m_start_r];
+	return m_size_r;
+}
+
+void bip_buffer::commit(size_t size)
+{
+	kassert(size <= m_size_r, "Tried to commit more than reserved");
+
+	if (m_start_r == m_start_a + m_size_a) {
+		// We were adding to A
+		m_size_a += size;
+	} else {
+		// We were adding to B
+		kassert(m_start_r == m_start_b + m_size_b, "Bad m_start_r!");
+		m_size_b += size;
+	}
+
+	m_start_r = m_size_r = 0;
+}
+
+size_t bip_buffer::get_block(void **area) const
+{
+	*area = m_size_a ? &m_buffer[m_start_a] : nullptr;
+	return m_size_a;
+}
+
+void bip_buffer::consume(size_t size)
+{
+	kassert(size <= m_size_a, "Consumed more bytes than exist in A");
+	if (size >= m_size_a) {
+		m_size_a = m_size_b;
+		m_start_a = m_start_b;
+		m_size_b = m_start_b = 0;
+	} else {
+		m_size_a -= size;
+		m_start_a += size;
+	}
+}
+
+} // namespace kutil

src/libraries/kutil/heap_allocator.cpp

@@ -1,12 +1,11 @@
 #include <stdint.h>
 #include "kutil/assert.h"
 #include "kutil/memory.h"
-#include "kutil/memory_manager.h"
+#include "kutil/heap_allocator.h"
 
 namespace kutil {
 
-
-struct memory_manager::mem_header
+struct heap_allocator::mem_header
 {
 	mem_header(mem_header *prev, mem_header *next, uint8_t size) :
 		m_prev(prev), m_next(next)
@@ -14,34 +13,29 @@ struct memory_manager::mem_header
 		set_size(size);
 	}
 
-	inline void set_size(uint8_t size)
-	{
+	inline void set_size(uint8_t size) {
 		m_prev = reinterpret_cast<mem_header *>(
 			reinterpret_cast<uintptr_t>(prev()) | (size & 0x3f));
 	}
 
-	inline void set_used(bool used)
-	{
+	inline void set_used(bool used) {
 		m_next = reinterpret_cast<mem_header *>(
 			reinterpret_cast<uintptr_t>(next()) | (used ? 1 : 0));
 	}
 
-	inline void set_next(mem_header *next)
-	{
+	inline void set_next(mem_header *next) {
 		bool u = used();
 		m_next = next;
 		set_used(u);
 	}
 
-	inline void set_prev(mem_header *prev)
-	{
+	inline void set_prev(mem_header *prev) {
 		uint8_t s = size();
 		m_prev = prev;
 		set_size(s);
 	}
 
-	void remove()
-	{
+	void remove() {
 		if (next()) next()->set_prev(prev());
 		if (prev()) prev()->set_next(next());
 		set_prev(nullptr);
@@ -67,30 +61,24 @@ private:
 };
 
 
-memory_manager::memory_manager() :
-	m_start(nullptr),
-	m_length(0),
-	m_grow(nullptr)
-{
-	kutil::memset(m_free, 0, sizeof(m_free));
-}
+heap_allocator::heap_allocator() : m_next(0), m_size(0) {}
 
-memory_manager::memory_manager(void *start, grow_callback grow_cb) :
-	m_start(start),
-	m_length(0),
-	m_grow(grow_cb)
+heap_allocator::heap_allocator(uintptr_t start, size_t size) :
+	m_next(start), m_size(size)
 {
 	kutil::memset(m_free, 0, sizeof(m_free));
-	grow_memory();
 }
 
 void *
-memory_manager::allocate(size_t length)
+heap_allocator::allocate(size_t length)
 {
 	size_t total = length + sizeof(mem_header);
 	unsigned size = min_size;
 	while (total > (1 << size)) size++;
+
 	kassert(size <= max_size, "Tried to allocate a block bigger than max_size");
+	if (size > max_size)
+		return nullptr;
 
 	mem_header *header = pop_free(size);
 	header->set_used(true);
@@ -98,18 +86,28 @@ memory_manager::allocate(size_t length)
 }
 
 void
-memory_manager::free(void *p)
+heap_allocator::free(void *p)
 {
+	if (!p) return;
+
 	mem_header *header = reinterpret_cast<mem_header *>(p);
 	header -= 1; // p points after the header
 	header->set_used(false);
 
-	while (true) {
+	while (header->size() != max_size) {
+		auto size = header->size();
+
 		mem_header *buddy = header->buddy();
-		if (buddy->used() || buddy->size() != header->size()) break;
+		if (buddy->used() || buddy->size() != size)
+			break;
+
+		if (get_free(size) == buddy)
+			get_free(size) = buddy->next();
+
 		buddy->remove();
+
 		header = header->eldest() ? header : buddy;
-		header->set_size(header->size() + 1);
+		header->set_size(size + 1);
 	}
 
 	uint8_t size = header->size();
@@ -120,49 +118,60 @@ memory_manager::free(void *p)
 }
 
 void
-memory_manager::grow_memory()
+heap_allocator::ensure_block(unsigned size)
 {
-	size_t length = (1 << max_size);
-
-	void *next = kutil::offset_pointer(m_start, m_length);
-	kassert(m_grow, "Tried to grow heap without a growth callback");
-	m_grow(next, length);
-
-	mem_header *block = new (next) mem_header(nullptr, get_free(max_size), max_size);
-	get_free(max_size) = block;
-	if (block->next())
-		block->next()->set_prev(block);
-	m_length += length;
-}
-
-void
-memory_manager::ensure_block(unsigned size)
-{
-	if (get_free(size) != nullptr) return;
-	else if (size == max_size) {
-		grow_memory();
+	if (get_free(size) != nullptr)
 		return;
+
+	if (size == max_size) {
+		size_t bytes = (1 << max_size);
+		if (bytes <= m_size) {
+			mem_header *next = reinterpret_cast<mem_header *>(m_next);
+			new (next) mem_header(nullptr, nullptr, size);
+			get_free(size) = next;
+			m_next += bytes;
+			m_size -= bytes;
+		}
+	} else {
+		mem_header *orig = pop_free(size + 1);
+		if (orig) {
+			mem_header *next = kutil::offset_pointer(orig, 1 << size);
+			new (next) mem_header(orig, nullptr, size);
+
+			orig->set_next(next);
+			orig->set_size(size);
+			get_free(size) = orig;
+		}
 	}
-
-	mem_header *orig = pop_free(size + 1);
-
-	mem_header *next = kutil::offset_pointer(orig, 1 << size);
-	new (next) mem_header(orig, nullptr, size);
-
-	orig->set_next(next);
-	orig->set_size(size);
-	get_free(size) = orig;
 }
 
-memory_manager::mem_header *
-memory_manager::pop_free(unsigned size)
+heap_allocator::mem_header *
+heap_allocator::pop_free(unsigned size)
 {
 	ensure_block(size);
 	mem_header *block = get_free(size);
-	get_free(size) = block->next();
-
-	block->remove();
+	if (block) {
+		get_free(size) = block->next();
+		block->remove();
+	}
 	return block;
 }
 
+
+class invalid_allocator :
+	public allocator
+{
+public:
+	virtual void * allocate(size_t) override {
+		kassert(false, "Attempting to allocate from allocator::invalid");
+		return nullptr;
+	}
+
+	virtual void free(void *) override {
+		kassert(false, "Attempting to free from allocator::invalid");
+	}
+} _invalid_allocator;
+
+allocator &allocator::invalid = _invalid_allocator;
+
 } // namespace kutil

src/libraries/kutil/include/kutil/address_manager.h

@@ -0,0 +1,15 @@
+#pragma once
+/// \file address_manager.h
+/// The virtual memory address space manager
+
+#include "kutil/buddy_allocator.h"
+
+namespace kutil {
+
+using address_manager =
+	buddy_allocator<
+		12,  // Min allocation:  4KiB
+		36   // Max allocation: 64GiB
+	>;
+
+} //namespace kutil

src/libraries/kutil/include/kutil/allocator.h

@@ -0,0 +1,31 @@
+#pragma once
+/// \file allocator.h
+/// Allocator interface
+
+#include <stdint.h>
+#include "kernel_memory.h"
+
+namespace kutil {
+
+class allocator
+{
+public:
+	/// Allocate memory.
+	/// \arg length  The amount of memory to allocate, in bytes
+	/// \returns     A pointer to the allocated memory, or nullptr if
+	///              allocation failed.
+	virtual void * allocate(size_t size) = 0;
+
+	/// Free a previous allocation.
+	/// \arg p  A pointer previously retuned by allocate()
+	virtual void free(void *p) = 0;
+
+	template <typename T>
+	inline T * allocate(unsigned count) {
+		return reinterpret_cast<T*>(allocate(count * sizeof(T)));
+	}
+
+	static allocator &invalid;
+};
+
+} // namespace kutil

src/libraries/kutil/include/kutil/bip_buffer.h

@@ -0,0 +1,59 @@
+#pragma once
+/// \file bip_buffer.h
+/// A Bip Buffer (bipartite circular buffer). For more on the Bip Buffer structure, see
+/// https://www.codeproject.com/Articles/3479/The-Bip-Buffer-The-Circular-Buffer-with-a-Twist
+
+#include <stddef.h>
+#include <stdint.h>
+
+namespace kutil {
+
+class bip_buffer
+{
+public:
+	/// Default constructor. Creates a zero-size buffer.
+	bip_buffer();
+
+	/// Constructor.
+	bip_buffer(uint8_t *buffer, size_t size);
+
+	/// Reserve an area of buffer for a write.
+	/// \arg size  Requested size, in bytes
+	/// \arg area  [out] Pointer to returned area
+	/// \returns   Size of returned area, in bytes, or 0 on failure
+	size_t reserve(size_t size, void **area);
+
+	/// Commit a pending write started by reserve()
+	/// \arg size  Amount of data used, in bytes
+	void commit(size_t size);
+
+	/// Get a pointer to a block of data in the buffer.
+	/// \arg area  [out] Pointer to the retuned area
+	/// \returns   Size of the returned area, in bytes
+	size_t get_block(void **area) const;
+
+	/// Mark a number of bytes as consumed, freeing buffer space
+	/// \arg size  Number of bytes to consume
+	void consume(size_t size);
+
+	/// Get total amount of data in the buffer.
+	/// \returns  Number of bytes committed to the buffer
+	inline size_t size() const { return m_size_a + m_size_b; }
+
+	/// Get total amount of free buffer remaining
+	/// \returns  Number of bytes of buffer that are free
+	inline size_t free_space() const { return m_buffer_size - size(); }
+
+private:
+	size_t m_start_a;
+	size_t m_start_b;
+	size_t m_start_r;
+	size_t m_size_a;
+	size_t m_size_b;
+	size_t m_size_r;
+
+	const size_t m_buffer_size;
+	uint8_t * const m_buffer;
+};
+
+} // namespace kutil

src/libraries/kutil/include/kutil/buddy_allocator.h

@@ -0,0 +1,302 @@
+#pragma once
+/// \file buddy_allocator.h
+/// Helper base class for buddy allocators with external node storage.
+
+#include <stdint.h>
+#include <utility>
+#include "kutil/assert.h"
+#include "kutil/linked_list.h"
+#include "kutil/slab_allocator.h"
+
+namespace kutil {
+
+struct buddy_region;
+
+
+template<
+	unsigned size_min,
+	unsigned size_max,
+	typename region_type = buddy_region>
+class buddy_allocator
+{
+public:
+	using region_node = list_node<region_type>;
+	using region_list = linked_list<region_type>;
+
+	static const size_t min_alloc = (1 << size_min);
+	static const size_t max_alloc = (1 << size_max);
+
+	/// Default constructor creates an invalid object.
+	buddy_allocator() : m_alloc(allocator::invalid) {}
+
+	/// Constructor.
+	/// \arg alloc  Allocator to use for region nodes
+	buddy_allocator(allocator &alloc) : m_alloc(alloc) {}
+
+	/// Move-like constructor. Takes ownership of existing regions.
+	buddy_allocator(buddy_allocator &&other, allocator &alloc) :
+		m_alloc(alloc)
+	{
+		for (unsigned i = 0; i < buckets; ++i) {
+			m_free[i] = std::move(other.m_free[i]);
+			m_used[i] = std::move(other.m_used[i]);
+		}
+	}
+
+	/// Add address space to be managed.
+	/// \arg start   Initial address in the managed range
+	/// \arg length  Size of the managed range, in bytes
+	void add_regions(uintptr_t start, size_t length)
+	{
+		uintptr_t p = start;
+		unsigned size = size_max;
+
+		while (size >= size_min) {
+			size_t chunk_size = 1ull << size;
+
+			while (p + chunk_size <= start + length) {
+				region_node *r = m_alloc.pop();
+				r->size = size_max;
+				r->address = p;
+
+				free_bucket(size).sorted_insert(r);
+				p += chunk_size;
+			}
+
+			size--;
+		}
+	}
+
+	/// Allocate address space from the managed area.
+	/// \arg length  The amount of memory to allocate, in bytes
+	/// \returns     The address of the start of the allocated area, or 0 on
+	///              failure
+	uintptr_t allocate(size_t length)
+	{
+		unsigned size = size_min;
+		while ((1ull << size) < length)
+			if (size++ == size_max)
+				return 0;
+
+		unsigned request = size;
+		while (free_bucket(request).empty())
+			if (request++ == size_max)
+				return 0;
+
+		region_node *r = nullptr;
+		while (request > size) {
+			r = free_bucket(request).pop_front();
+			region_node *n = split(r);
+			request--;
+
+			free_bucket(request).sorted_insert(n);
+			if (request != size)
+				free_bucket(request).sorted_insert(r);
+		}
+
+		if (r == nullptr) r = free_bucket(size).pop_front();
+		used_bucket(size).sorted_insert(r);
+
+		return r->address;
+	}
+
+	/// Mark a region as allocated.
+	/// \arg start   The start of the region
+	/// \arg length  The size of the region, in bytes
+	/// \returns     The address of the start of the allocated area, or 0 on
+	///              failure. This may be less than `start`.
+	uintptr_t mark(uintptr_t start, size_t length)
+	{
+		uintptr_t end = start + length;
+		region_node *found = nullptr;
+
+		for (unsigned i = size_max; i >= size_min && !found; --i) {
+			for (auto *r : free_bucket(i)) {
+				if (start >= r->address && end <= r->end()) {
+					free_bucket(i).remove(r);
+					found = r;
+					break;
+				}
+			}
+		}
+
+		kassert(found, "buddy_allocator::mark called for unknown region");
+		if (!found)
+			return 0;
+
+		found = maybe_split(found, start, end);
+		used_bucket(found->size).sorted_insert(found);
+		return found->address;
+	}
+
+	/// Mark a region as permanently allocated. The region is not returned,
+	/// as the block can never be freed. This may remove several smaller
+	/// regions in order to more closely fit the region described.
+	/// \arg start   The start of the region
+	/// \arg length  The size of the region, in bytes
+	/// \returns     The address of the start of the allocated area, or 0 on
+	///              failure. This may be less than `start`.
+	void mark_permanent(uintptr_t start, size_t length)
+	{
+		uintptr_t end = start + length;
+		for (unsigned i = size_max; i >= size_min; --i) {
+			for (auto *r : free_bucket(i)) {
+				if (start >= r->address && end <= r->end()) {
+					free_bucket(i).remove(r);
+					delete_region(r, start, end);
+					return;
+				}
+			}
+		}
+
+		kassert(false, "buddy_allocator::mark_permanent called for unknown region");
+	}
+
+	/// Free a previous allocation.
+	/// \arg p  An address previously retuned by allocate()
+	void free(uintptr_t p)
+	{
+		region_node *found = find(p, true);
+
+		kassert(found, "buddy_allocator::free called for unknown region");
+		if (!found)
+			return;
+
+		used_bucket(found->size).remove(found);
+		free_bucket(found->size).sorted_insert(found);
+
+		while (auto *bud = get_buddy(found)) {
+			region_node *eld = found->elder() ? found : bud;
+			region_node *other = found->elder() ? bud : found;
+
+			free_bucket(other->size).remove(other);
+			m_alloc.push(other);
+
+			free_bucket(eld->size).remove(eld);
+			eld->size++;
+			free_bucket(eld->size).sorted_insert(eld);
+
+			found = eld;
+		}
+	}
+
+	/// Check if an allocation exists
+	/// \arg addr  Address within the managed space
+	/// \returns   True if the address is in a region currently allocated
+	bool contains(uintptr_t addr)
+	{
+		for (unsigned i = size_max; i >= size_min; --i) {
+			for (auto *r : used_bucket(i)) {
+				if (r->contains(addr))
+					return true;
+				else if (r->address < addr)
+					break;
+			}
+		}
+
+		return false;
+	}
+
+protected:
+	/// Split a region of the given size into two smaller regions, returning
+	/// the new latter half
+	region_node * split(region_node *reg)
+	{
+		region_node *other = m_alloc.pop();
+
+		other->size = --reg->size;
+		other->address = reg->address + (1ull << reg->size);
+		return other;
+	}
+
+	/// Find a node with the given address
+	region_node * find(uintptr_t p, bool used = true)
+	{
+		for (unsigned size = size_max; size >= size_min; --size) {
+			auto &list = used ? used_bucket(size) : free_bucket(size);
+			for (auto *f : list) {
+				if (f->address < p) continue;
+				if (f->address == p) return f;
+				break;
+			}
+		}
+		return nullptr;
+	}
+
+	/// Helper to get the buddy for a node, if it's adjacent
+	region_node * get_buddy(region_node *r)
+	{
+		region_node *b = r->elder() ? r->next() : r->prev();
+		if (b && b->address == r->buddy())
+			return b;
+		return nullptr;
+	}
+
+	region_node * maybe_split(region_node *reg, uintptr_t start, uintptr_t end)
+	{
+		while (reg->size > size_min) {
+			// Split if the request fits in the second half
+			if (start >= reg->half()) {
+				region_node *other = split(reg);
+				free_bucket(reg->size).sorted_insert(reg);
+				reg = other;
+			}
+
+			// Split if the request fits in the first half
+			else if (end <= reg->half()) {
+				region_node *other = split(reg);
+				free_bucket(other->size).sorted_insert(other);
+			}
+
+			// If neither, we've split as much as possible
+			else break;
+		}
+
+		return reg;
+	}
+
+	void delete_region(region_node *reg, uintptr_t start, uintptr_t end)
+	{
+		reg = maybe_split(reg, start, end);
+
+		size_t leading = start - reg->address;
+		size_t trailing = reg->end() - end;
+		if (leading > (1<<size_min) || trailing > (1<<size_min)) {
+			region_node *tail = split(reg);
+			delete_region(reg, start, reg->end());
+			delete_region(tail, tail->address, end);
+		} else {
+			m_alloc.push(reg);
+		}
+	}
+
+	region_list & used_bucket(unsigned size) { return m_used[size - size_min]; }
+	region_list & free_bucket(unsigned size) { return m_free[size - size_min]; }
+
+	static const unsigned buckets = (size_max - size_min + 1);
+
+	region_list m_free[buckets];
+	region_list m_used[buckets];
+	slab_allocator<region_type> m_alloc;
+};
+
+
+struct buddy_region
+{
+	inline int compare(const buddy_region *o) const {
+		return address > o->address ? 1 : \
+			address < o->address ? -1 : 0;
+	}
+
+	inline uintptr_t end() const { return address + (1ull << size); }
+	inline uintptr_t half() const { return address + (1ull << (size - 1)); }
+	inline bool contains(uintptr_t p) const { return p >= address && p < end(); }
+
+	inline uintptr_t buddy() const { return address ^ (1ull << size); }
+	inline bool elder() const { return address < buddy(); }
+
+	uint16_t size;
+	uintptr_t address;
+};
+
+} // namespace kutil