From f93d80b8d23b3a6f02ba5424f840dd231c1444d2 Mon Sep 17 00:00:00 2001 From: "Justin C. Miller" Date: Mon, 14 Feb 2022 19:52:45 -0800 Subject: [PATCH] [project] Update Readme and remove toolchain scripts After hitting 700 commits last week, I thought it'd be a good time to update the project status in the Readme. This change also pulls out the toolchain scripts that moved to jsix-os/toolchain, and updates the Readme about that as well. --- .clang-format | 21 ---- NOTES.md | 38 ------- README.md | 73 ++++++++------ peru.yaml | 7 -- scripts/build_sysroot.sh | 126 ------------------------ scripts/readme_for_prebuilt_sysroots.md | 14 --- 6 files changed, 45 insertions(+), 234 deletions(-) delete mode 100644 .clang-format delete mode 100644 NOTES.md delete mode 100644 peru.yaml delete mode 100755 scripts/build_sysroot.sh delete mode 100644 scripts/readme_for_prebuilt_sysroots.md diff --git a/.clang-format b/.clang-format deleted file mode 100644 index a598150..0000000 --- a/.clang-format +++ /dev/null @@ -1,21 +0,0 @@ ---- -Language: Cpp -BasedOnStyle: LLVM - -ColumnLimit: 100 -IndentWidth: 4 -TabWidth: 4 -UseTab: Always - -AccessModifierOffset: -4 - -AlignEscapedNewlinesLeft: true - -AllowShortIfStatementsOnASingleLine: true -AllowShortLoopsOnASingleLine: true - -AlwaysBreakAfterReturnType: AllDefinitions -AlwaysBreakBeforeMultilineStrings: true -AlwaysBreakTemplateDeclarations: true - -BreakBeforeBraces: Linux diff --git a/NOTES.md b/NOTES.md deleted file mode 100644 index 3f40974..0000000 --- a/NOTES.md +++ /dev/null @@ -1,38 +0,0 @@ -# Design / WIP notes - -## TODO - -- Paging manager - - Copy-on-write pages - - Better page-allocation model? -- Allow for more than one IOAPIC in ACPI module - - The objects get created, but GSI lookup only uses the one at index 0 -- mark kernel memory pages global -- Serial out based on circular/bip biffer and interrupts, not spinning on - `write_ready()` -- Split out more code into kutil for testing -- AHCI / MSI interrupts on Vbox break? -- FXSAVE to save XMM registers. - - optimization using #NM (0x7) to detect SSE usage -- Clean up of process memory maps -- Better stack tracer -- Bootloader rewrite - - C++ and sharing library code for ELF, initrd, etc - - Parse initrd and pre-load certain ELF images, eg the process loader process? - - Do initial memory bootstrap? -- Calling global ctors -- Device Tree - - Actual serial driver - - Disk driver - - File system -- Multiprocessing - - Fast syscalls using syscall/sysret - -### Build - -- Clean up build generator and its templates - - More robust objects to represent modules & targets to pass to templates - - Read project setup from a simple JSON/TOML/etc - - Better compartmentalizing when doing template inheritance -- Move to LLD as sysroot linker - diff --git a/README.md b/README.md index 5b7272e..7a8f2cb 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Roadmap* section, below. The design goals of the project are: * Modernity - I'm not interested in designing for legacy systems, or running on - all hardware out there. My target is only 64 bit architecutres, and modern + all hardware out there. My target is only 64 bit architectures, and modern commodity hardware. Currently that means x64 systems with Nehalem or newer CPUs and UEFI firmware. (See [this list][cpu_features] for the currently required CPU features.) Eventually I'd like to work on an AArch64 port, @@ -58,12 +58,18 @@ Remaining to do: - TLB shootdowns - Page swapping +- Large / huge page support _Physical page allocation: Sufficient._ The current physical page allocator -implementation suses a group of block representing up-to-1GiB areas of usable +implementation uses a group of blocks representing up-to-1GiB areas of usable memory as defined by the bootloader. Each block has a three-level bitmap denoting free/used pages. +Future work: + +- Align blocks so that their first page is 1GiB-aligned, making finding free + large/huge pages easier. + #### Multitasking _Sufficient._ The global scheduler object keeps separate ready/blocked lists @@ -77,32 +83,37 @@ only the timekeeping task should be a separate kernel-only thread. #### API _In progress._ User-space tasks are able to make syscalls to the kernel via -fast SYSCALL/SYSRET instructions. +fast SYSCALL/SYSRET instructions. Syscalls made via `libj6` look to both the +callee and the caller like standard SysV ABI function calls. The +implementations are wrapped in generated wrapper functions which validate the +request, check capabilities, and find the appropriate kernel objects or handles +before calling the implementation functions. -Major tasks still to do: +Current work in this area: -- The process initialization protocol needs to be re-built entirely. -- Processes' handles to kernel objects need the ability to check capabilities +- The protocol for processes and their children for both initialization and + sharing resources. The kernel is ignorant of this, except for providing IPC + capabilities. #### Hardware Support * Framebuffer driver: _In progress._ Currently on machines with a video - device accessible by UEFI, jsix starts a user-space framebuffer driver that - only prints out kernel logs. - * Serial driver: _To do._ Machines without a video device should have a - user-space log output task like the framebuffer driver, but currently this - is done inside the kernel. + device accessible by UEFI, jsix starts a user-space framebuffer driver that + only prints out kernel logs. + * Serial driver: _In progress._ The current UART driver does not expose the + UART as an output resource yet, it merely gets the kernel logs and sends + them over serial. * USB driver: _To do_ * AHCI (SATA) driver: _To do_ ## Building jsix uses the [Ninja][] build tool, and generates the build files for it with -the `configure` script. The build also relies on a custom sysroot, which can be -downloaded via the [Peru][] tool, or built locally. +the `configure` script. The build also relies on a custom toolchain sysroot, which can be +downloaded or built using the scripts in [jsix-os/toolchain][]. -[Ninja]: https://ninja-build.org -[Peru]: https://github.com/buildinspace/peru +[Ninja]: https://ninja-build.org +[jsix-os/toolchain]: https://github.com/jsix-os/toolchain Other build dependencies: @@ -110,13 +121,11 @@ Other build dependencies: * [nasm][]: the assembler * [lld][]: the linker * [mtools][]: for creating the FAT image -* [curl][]: if using `peru` below to download the sysroot [clang]: https://clang.llvm.org [nasm]: https://www.nasm.us [lld]: https://lld.llvm.org [mtools]: https://www.gnu.org/software/mtools/ -[curl]: https://curl.se The `configure` script has some Python dependencies - these can be installed via `pip`, though doing so in a python virtual environment is recommended. @@ -135,18 +144,14 @@ peru sync ### Setting up the sysroot -Running `peru sync` as in the above section will download and unpack the -toolchain into `sysroot`. +Build or download the toolchain sysroot as mentioned above with +[jsix-os/toolchain][], and symlink the built toolchain directory as `sysroot` +at the root of this project. -#### Compiling the sysroot yourself - -If you have CMake installed, runing the `scripts/build_sysroot.sh` script will -download and build a LLVM toolchain configured for building the sysroot, and -then build the sysroot with it. - -Built sysroots are actually stored in `~/.local/lib/jsix/sysroots` and installed -in the project dir via symbolic link, so having mulitple jsix working trees or -switching sysroot versions is easy. +```bash +# Example if both the toolchain and this project are cloned under ~/src +ln -s ~/src/toolchain/toolchains/llvm-13 ~/src/jsix/sysroot +``` ### Building and running jsix @@ -159,3 +164,15 @@ to run jsix in QEMU `-nographic` mode. I personally run this either from a real debian amd64 bullseye machine or a windows WSL debian bullseye installation. Your mileage may vary with other setups and distros. + +### Running the test suite + +jsix now has the `test_runner` userspace program that runs various automated +tests. It is not included in the default build, but if you use the `test.yml` +manifest it will be built, and can be run with the `test.sh` script or the +`qemu.sh` script. + +```bash +./configure --manifest=assets/manifests/test.yml +if ./test.sh; then echo "All tests passed!"; else echo "Failed."; fi +``` diff --git a/peru.yaml b/peru.yaml deleted file mode 100644 index 6c3f183..0000000 --- a/peru.yaml +++ /dev/null @@ -1,7 +0,0 @@ -imports: - sysroot: sysroot/ - -curl module sysroot: - url: https://f000.backblazeb2.com/file/jsix-os/sysroot-llvm-11.tar.gz - unpack: tar - sha1: 7dffa388bdb2e20e1b0eb73600024e0c07d95abd diff --git a/scripts/build_sysroot.sh b/scripts/build_sysroot.sh deleted file mode 100755 index ef57899..0000000 --- a/scripts/build_sysroot.sh +++ /dev/null @@ -1,126 +0,0 @@ -#!/bin/bash - -set -o errexit -set -o errtrace -set -o pipefail -set -o nounset - -function errmsg () { - echo "Error: $(caller)" - echo "Working directory not cleaned up!" -} - -trap errmsg ERR - -LLVM_VERSION=11 -J6_TOOLCHAINS="${J6_TOOLCHAINS:-$(realpath ~/.local/lib/jsix)}" -CHAIN_NAME="llvm-${LLVM_VERSION}" -LLVM_BRANCH="release/${LLVM_VERSION}.x" - -SYSROOT="${J6_TOOLCHAINS}/sysroots/${CHAIN_NAME}" - -ROOT=$(realpath ${1:-$(mktemp -d "sysroot_build.XXX")}) -OUT="${ROOT}/sysroot" -echo "Working in ${ROOT}" - -if [ ! -d "${ROOT}/llvm-project" ]; then - echo "* Downloading LLVM ${LLVM_VERSION}" - - git clone -q --depth 1 \ - --branch "${LLVM_BRANCH}" \ - "https://github.com/llvm/llvm-project" \ - "${ROOT}/llvm-project" -fi - -if [ -d "${OUT}" ]; then - rm -rf "${OUT}" -fi -mkdir -p "${OUT}" - -## Stage1: build an LLVM toolchain that doesn't rely on the host libraries, -## compiler, runtime, etc. - -echo "* Configuring stage1" - -mkdir -p "${ROOT}/stage1" -cmake -G Ninja \ - -S "${ROOT}/llvm-project/llvm" \ - -B "${ROOT}/stage1" \ - -DCLANG_DEFAULT_RTLIB=compiler-rt \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER="clang" \ - -DCMAKE_C_COMPILER_LAUNCHER=ccache \ - -DCMAKE_CXX_COMPILER="clang++" \ - -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ - -DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=lld" \ - -DLIBCXXABI_USE_COMPILER_RT=ON \ - -DLIBCXXABI_USE_LLVM_UNWINDER=ON \ - -DLIBCXX_CXX_ABI=libcxxabi \ - -DLIBUNWIND_USE_COMPILER_RT=ON \ - -DLLVM_BUILD_TOOLS=OFF \ - -DLLVM_ENABLE_BINDINGS=OFF \ - -DLLVM_ENABLE_PROJECTS="clang;lld;libcxx;libcxxabi;libunwind;compiler-rt" \ - -DLLVM_INCLUDE_BENCHMARKS=OFF \ - -DLLVM_INCLUDE_EXAMPLES=OFF \ - -DLLVM_INCLUDE_TESTS=OFF \ - -DLLVM_INCLUDE_UTILS=OFF \ - -DLLVM_TARGETS_TO_BUILD="X86" \ - -DLLVM_USE_LINKER="lld" \ - 2>&1 > "${ROOT}/configure_stage1.log" - -echo "* Building stage1" - -ninja -C "${ROOT}/stage1" -v > "${ROOT}/build_stage1.log" -STAGE1="${ROOT}/stage1/bin" - -## Stage2: build the sysroot libraries with the stage1 toolchain - -echo "* Configuring stage2" - -mkdir -p "${ROOT}/stage2" -cmake -G Ninja \ - -S "${ROOT}/llvm-project/llvm" \ - -B "${ROOT}/stage2" \ - -DCMAKE_BUILD_TYPE=Release \ - -DCMAKE_C_COMPILER="${STAGE1}/clang" \ - -DCMAKE_C_COMPILER_LAUNCHER=ccache \ - -DCMAKE_CXX_COMPILER="${STAGE1}/clang++" \ - -DCMAKE_CXX_COMPILER_LAUNCHER=ccache \ - -DCMAKE_INSTALL_PREFIX="${OUT}" \ - -DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=${STAGE1}/ld.lld" \ - -DLIBCXXABI_ENABLE_STATIC_UNWINDER=ON \ - -DLIBCXXABI_USE_COMPILER_RT=ON \ - -DLIBCXXABI_USE_LLVM_UNWINDER=ON \ - -DLIBCXX_CXX_ABI=libcxxabi \ - -DLIBCXX_INCLUDE_BENCHMARKS=OFF \ - -DLIBUNWIND_USE_COMPILER_RT=ON \ - -DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-elf" \ - -DLLVM_ENABLE_BINDINGS=OFF \ - -DLLVM_ENABLE_LIBCXX=ON \ - -DLLVM_ENABLE_PROJECTS="libcxx;libcxxabi;libunwind" \ - -DLLVM_INCLUDE_BENCHMARKS=OFF \ - -DLLVM_INCLUDE_EXAMPLES=OFF \ - -DLLVM_INCLUDE_TESTS=OFF \ - -DLLVM_INCLUDE_TOOLS=OFF \ - -DLLVM_INCLUDE_UTILS=OFF \ - -DLLVM_TARGETS_TO_BUILD="X86" \ - -DLLVM_USE_LINKER="${STAGE1}/ld.lld" \ - 2>&1 > "${ROOT}/configure_stage2.log" - -echo "* Building stage2" - -ninja -C "${ROOT}/stage2" -v > "${ROOT}/build_stage2.log" - -echo "* Installing stage2" - -ninja -C "${ROOT}/stage2" -v install/strip > "${ROOT}/install_stage2.log" - -echo "* Installing new sysroot" - -rm -rf "${SYSROOT}" -mv "${OUT}" "${SYSROOT}" - -echo "* Cleaning up" -trap - ERR -rm -rf "${ROOT}" -echo "Done" diff --git a/scripts/readme_for_prebuilt_sysroots.md b/scripts/readme_for_prebuilt_sysroots.md deleted file mode 100644 index b27643f..0000000 --- a/scripts/readme_for_prebuilt_sysroots.md +++ /dev/null @@ -1,14 +0,0 @@ -# jsix OS sysroot - -This is a pre-built sysroot for building the jsix operating system kernel, -bootloader, and utilities. This package is provided as a convenience, and -contains software from the following repositories. - -## The LLVM toolchain - -The LLVM sources as downloaded via git from [llvm.org][llvm] under the terms of -the [Apache License v2.0][apache2], modified [as described here][llvmlic]. - -[llvm]: https://llvm.org -[apache2]: https://www.apache.org/licenses/LICENSE-2.0 -[llvmlic]: https://llvm.org/docs/DeveloperPolicy.html#new-llvm-project-license-framework