[project] Update Readme and remove toolchain scripts

After hitting 700 commits last week, I thought it'd be a good time to
update the project status in the Readme. This change also pulls out the
toolchain scripts that moved to jsix-os/toolchain, and updates the
Readme about that as well.

.clang-format

@@ -1,21 +0,0 @@
----
-Language:        Cpp
-BasedOnStyle:    LLVM
-
-ColumnLimit:     100
-IndentWidth:     4
-TabWidth:        4
-UseTab:          Always
-
-AccessModifierOffset: -4
-
-AlignEscapedNewlinesLeft: true
-
-AllowShortIfStatementsOnASingleLine: true
-AllowShortLoopsOnASingleLine: true
-
-AlwaysBreakAfterReturnType: AllDefinitions
-AlwaysBreakBeforeMultilineStrings: true
-AlwaysBreakTemplateDeclarations: true
-
-BreakBeforeBraces: Linux

NOTES.md

@@ -1,38 +0,0 @@
-# Design / WIP notes
-
-## TODO
-
-- Paging manager
-  - Copy-on-write pages
-  - Better page-allocation model?
-- Allow for more than one IOAPIC in ACPI module
-  - The objects get created, but GSI lookup only uses the one at index 0
-- mark kernel memory pages global
-- Serial out based on circular/bip biffer and interrupts, not spinning on
-  `write_ready()`
-- Split out more code into kutil for testing
-- AHCI / MSI interrupts on Vbox break?
-- FXSAVE to save XMM registers.
-  - optimization using #NM (0x7) to detect SSE usage
-- Clean up of process memory maps
-- Better stack tracer
-- Bootloader rewrite
-  - C++ and sharing library code for ELF, initrd, etc
-  - Parse initrd and pre-load certain ELF images, eg the process loader process?
-  - Do initial memory bootstrap?
-- Calling global ctors
-- Device Tree
-  - Actual serial driver
-  - Disk driver
-    - File system
-- Multiprocessing
-  - Fast syscalls using syscall/sysret
-
-### Build
-
-- Clean up build generator and its templates
-  - More robust objects to represent modules & targets to pass to templates
-  - Read project setup from a simple JSON/TOML/etc
-  - Better compartmentalizing when doing template inheritance
-- Move to LLD as sysroot linker
-

README.md

@@ -9,7 +9,7 @@ Roadmap* section, below.
 The design goals of the project are:
 
 * Modernity - I'm not interested in designing for legacy systems, or running on
-  all hardware out there. My target is only 64 bit architecutres, and modern
+  all hardware out there. My target is only 64 bit architectures, and modern
   commodity hardware. Currently that means x64 systems with Nehalem or newer
   CPUs and UEFI firmware. (See [this list][cpu_features] for the currently
   required CPU features.) Eventually I'd like to work on an AArch64 port,
@@ -58,12 +58,18 @@ Remaining to do:
 
 - TLB shootdowns
 - Page swapping
+- Large / huge page support
 
 _Physical page allocation: Sufficient._ The current physical page allocator
-implementation suses a group of block representing up-to-1GiB areas of usable
+implementation uses a group of blocks representing up-to-1GiB areas of usable
 memory as defined by the bootloader. Each block has a three-level bitmap
 denoting free/used pages.
 
+Future work:
+
+- Align blocks so that their first page is 1GiB-aligned, making finding free
+  large/huge pages easier.
+
 #### Multitasking
 
 _Sufficient._ The global scheduler object keeps separate ready/blocked lists
@@ -77,32 +83,37 @@ only the timekeeping task should be a separate kernel-only thread.
 #### API
 
 _In progress._ User-space tasks are able to make syscalls to the kernel via
-fast SYSCALL/SYSRET instructions.
+fast SYSCALL/SYSRET instructions. Syscalls made via `libj6` look to both the
+callee and the caller like standard SysV ABI function calls. The
+implementations are wrapped in generated wrapper functions which validate the
+request, check capabilities, and find the appropriate kernel objects or handles
+before calling the implementation functions.
 
-Major tasks still to do:
+Current work in this area:
 
-- The process initialization protocol needs to be re-built entirely.
+- The protocol for processes and their children for both initialization and
-- Processes' handles to kernel objects need the ability to check capabilities
+  sharing resources. The kernel is ignorant of this, except for providing IPC
+  capabilities.
 
 #### Hardware Support
 
   * Framebuffer driver: _In progress._ Currently on machines with a video
     device accessible by UEFI, jsix starts a user-space framebuffer driver that
     only prints out kernel logs.
-  * Serial driver: _To do._ Machines without a video device should have a
+  * Serial driver: _In progress._ The current UART driver does not expose the
-	user-space log output task like the framebuffer driver, but currently this
+    UART as an output resource yet, it merely gets the kernel logs and sends
-	is done inside the kernel.
+    them over serial.
   * USB driver: _To do_
   * AHCI (SATA) driver: _To do_
 
 ## Building
 
 jsix uses the [Ninja][] build tool, and generates the build files for it with
-the `configure` script. The build also relies on a custom sysroot, which can be
+the `configure` script. The build also relies on a custom toolchain sysroot, which can be
-downloaded via the [Peru][] tool, or built locally.
+downloaded or built using the scripts in [jsix-os/toolchain][].
 
 [Ninja]:             https://ninja-build.org
-[Peru]:     https://github.com/buildinspace/peru
+[jsix-os/toolchain]: https://github.com/jsix-os/toolchain
 
 Other build dependencies:
 
@@ -110,13 +121,11 @@ Other build dependencies:
 * [nasm][]: the assembler
 * [lld][]: the linker
 * [mtools][]: for creating the FAT image
-* [curl][]: if using `peru` below to download the sysroot
 
 [clang]:    https://clang.llvm.org
 [nasm]:     https://www.nasm.us
 [lld]:      https://lld.llvm.org
 [mtools]:   https://www.gnu.org/software/mtools/
-[curl]:     https://curl.se
 
 The `configure` script has some Python dependencies - these can be installed via
 `pip`, though doing so in a python virtual environment is recommended.
@@ -135,18 +144,14 @@ peru sync
 
 ### Setting up the sysroot
 
-Running `peru sync` as in the above section will download and unpack the
+Build or download the toolchain sysroot as mentioned above with
-toolchain into `sysroot`. 
+[jsix-os/toolchain][], and symlink the built toolchain directory as `sysroot`
+at the root of this project.
 
-#### Compiling the sysroot yourself
+```bash
-
+# Example if both the toolchain and this project are cloned under ~/src
-If you have CMake installed, runing the `scripts/build_sysroot.sh` script will
+ln -s ~/src/toolchain/toolchains/llvm-13 ~/src/jsix/sysroot
-download and build a LLVM toolchain configured for building the sysroot, and
+```
-then build the sysroot with it.
-
-Built sysroots are actually stored in `~/.local/lib/jsix/sysroots` and installed
-in the project dir via symbolic link, so having mulitple jsix working trees or
-switching sysroot versions is easy.
 
 ### Building and running jsix
 
@@ -159,3 +164,15 @@ to run jsix in QEMU `-nographic` mode.
 I personally run this either from a real debian amd64 bullseye machine or
 a windows WSL debian bullseye installation. Your mileage may vary with other
 setups and distros.
+
+### Running the test suite
+
+jsix now has the `test_runner` userspace program that runs various automated
+tests. It is not included in the default build, but if you use the `test.yml`
+manifest it will be built, and can be run with the `test.sh` script or the
+`qemu.sh` script.
+
+```bash
+./configure --manifest=assets/manifests/test.yml
+if ./test.sh; then echo "All tests passed!"; else echo "Failed."; fi
+```

peru.yaml

@@ -1,7 +0,0 @@
-imports:
-  sysroot: sysroot/
-
-curl module sysroot:
-  url: https://f000.backblazeb2.com/file/jsix-os/sysroot-llvm-11.tar.gz
-  unpack: tar
-  sha1: 7dffa388bdb2e20e1b0eb73600024e0c07d95abd

scripts/build_sysroot.sh

@@ -1,126 +0,0 @@
-#!/bin/bash
-
-set -o errexit
-set -o errtrace
-set -o pipefail
-set -o nounset
-
-function errmsg () {
-    echo "Error: $(caller)"
-    echo "Working directory not cleaned up!"
-}
-
-trap errmsg ERR
-
-LLVM_VERSION=11
-J6_TOOLCHAINS="${J6_TOOLCHAINS:-$(realpath ~/.local/lib/jsix)}"
-CHAIN_NAME="llvm-${LLVM_VERSION}"
-LLVM_BRANCH="release/${LLVM_VERSION}.x"
-
-SYSROOT="${J6_TOOLCHAINS}/sysroots/${CHAIN_NAME}"
-
-ROOT=$(realpath ${1:-$(mktemp -d "sysroot_build.XXX")})
-OUT="${ROOT}/sysroot"
-echo "Working in ${ROOT}"
-
-if [ ! -d "${ROOT}/llvm-project" ]; then
-    echo "* Downloading LLVM ${LLVM_VERSION}"
-
-    git clone -q --depth 1 \
-        --branch "${LLVM_BRANCH}" \
-        "https://github.com/llvm/llvm-project" \
-        "${ROOT}/llvm-project"
-fi
-
-if [ -d "${OUT}" ]; then
-    rm -rf "${OUT}"
-fi
-mkdir -p "${OUT}"
-
-## Stage1: build an LLVM toolchain that doesn't rely on the host libraries,
-## compiler, runtime, etc.
-
-echo "* Configuring stage1"
-
-mkdir -p "${ROOT}/stage1"
-cmake -G Ninja \
-    -S "${ROOT}/llvm-project/llvm" \
-    -B "${ROOT}/stage1" \
-	-DCLANG_DEFAULT_RTLIB=compiler-rt \
-	-DCMAKE_BUILD_TYPE=Release \
-	-DCMAKE_C_COMPILER="clang" \
-	-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-	-DCMAKE_CXX_COMPILER="clang++" \
-	-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-	-DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=lld" \
-	-DLIBCXXABI_USE_COMPILER_RT=ON \
-	-DLIBCXXABI_USE_LLVM_UNWINDER=ON \
-	-DLIBCXX_CXX_ABI=libcxxabi  \
-	-DLIBUNWIND_USE_COMPILER_RT=ON \
-	-DLLVM_BUILD_TOOLS=OFF \
-	-DLLVM_ENABLE_BINDINGS=OFF \
-	-DLLVM_ENABLE_PROJECTS="clang;lld;libcxx;libcxxabi;libunwind;compiler-rt" \
-	-DLLVM_INCLUDE_BENCHMARKS=OFF \
-	-DLLVM_INCLUDE_EXAMPLES=OFF \
-	-DLLVM_INCLUDE_TESTS=OFF \
-	-DLLVM_INCLUDE_UTILS=OFF \
-	-DLLVM_TARGETS_TO_BUILD="X86" \
-	-DLLVM_USE_LINKER="lld" \
-	2>&1 > "${ROOT}/configure_stage1.log"
-
-echo "* Building stage1"
-
-ninja -C "${ROOT}/stage1" -v > "${ROOT}/build_stage1.log"
-STAGE1="${ROOT}/stage1/bin"
-
-## Stage2: build the sysroot libraries with the stage1 toolchain
-
-echo "* Configuring stage2"
-
-mkdir -p "${ROOT}/stage2"
-cmake -G Ninja \
-    -S "${ROOT}/llvm-project/llvm" \
-    -B "${ROOT}/stage2" \
-	-DCMAKE_BUILD_TYPE=Release \
-	-DCMAKE_C_COMPILER="${STAGE1}/clang" \
-	-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-	-DCMAKE_CXX_COMPILER="${STAGE1}/clang++" \
-	-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-	-DCMAKE_INSTALL_PREFIX="${OUT}" \
-	-DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=${STAGE1}/ld.lld" \
-	-DLIBCXXABI_ENABLE_STATIC_UNWINDER=ON \
-	-DLIBCXXABI_USE_COMPILER_RT=ON \
-	-DLIBCXXABI_USE_LLVM_UNWINDER=ON \
-	-DLIBCXX_CXX_ABI=libcxxabi  \
-	-DLIBCXX_INCLUDE_BENCHMARKS=OFF \
-	-DLIBUNWIND_USE_COMPILER_RT=ON \
-	-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-elf" \
-	-DLLVM_ENABLE_BINDINGS=OFF \
-	-DLLVM_ENABLE_LIBCXX=ON \
-	-DLLVM_ENABLE_PROJECTS="libcxx;libcxxabi;libunwind" \
-	-DLLVM_INCLUDE_BENCHMARKS=OFF \
-	-DLLVM_INCLUDE_EXAMPLES=OFF \
-	-DLLVM_INCLUDE_TESTS=OFF \
-	-DLLVM_INCLUDE_TOOLS=OFF \
-	-DLLVM_INCLUDE_UTILS=OFF \
-	-DLLVM_TARGETS_TO_BUILD="X86" \
-	-DLLVM_USE_LINKER="${STAGE1}/ld.lld" \
-	2>&1 > "${ROOT}/configure_stage2.log"
-
-echo "* Building stage2"
-
-ninja -C "${ROOT}/stage2" -v > "${ROOT}/build_stage2.log"
-
-echo "* Installing stage2"
-
-ninja -C "${ROOT}/stage2" -v install/strip > "${ROOT}/install_stage2.log"
-
-echo "* Installing new sysroot"
-
-rm -rf "${SYSROOT}"
-mv "${OUT}" "${SYSROOT}"
-
-echo "* Cleaning up"
-trap - ERR
-rm -rf "${ROOT}"
-echo "Done"

scripts/readme_for_prebuilt_sysroots.md

@@ -1,14 +0,0 @@
-# jsix OS sysroot
-
-This is a pre-built sysroot for building the jsix operating system kernel,
-bootloader, and utilities. This package is provided as a convenience, and
-contains software from the following repositories.
-
-## The LLVM toolchain
-
-The LLVM sources as downloaded via git from [llvm.org][llvm] under the terms of
-the [Apache License v2.0][apache2], modified [as described here][llvmlic].
-
-[llvm]: https://llvm.org
-[apache2]: https://www.apache.org/licenses/LICENSE-2.0
-[llvmlic]: https://llvm.org/docs/DeveloperPolicy.html#new-llvm-project-license-framework