justin/jsix_import
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

[project] Update Readme and remove toolchain scripts

Browse Source 
After hitting 700 commits last week, I thought it'd be a good time to
update the project status in the Readme. This change also pulls out the
toolchain scripts that moved to jsix-os/toolchain, and updates the
Readme about that as well.
This commit is contained in:
Justin C. Miller
2022-02-14 19:52:45 -08:00
parent a6632625f4
commit f93d80b8d2
6 changed files with 45 additions and 234 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
.clang-format
View File

@@ -1,21 +0,0 @@
---
Language: Cpp
BasedOnStyle: LLVM
ColumnLimit: 100
IndentWidth: 4
TabWidth: 4
UseTab: Always
AccessModifierOffset: -4
AlignEscapedNewlinesLeft: true
AllowShortIfStatementsOnASingleLine: true
AllowShortLoopsOnASingleLine: true
AlwaysBreakAfterReturnType: AllDefinitions
AlwaysBreakBeforeMultilineStrings: true
AlwaysBreakTemplateDeclarations: true
BreakBeforeBraces: Linux

38
NOTES.md
View File

@@ -1,38 +0,0 @@
# Design / WIP notes
## TODO
- Paging manager
- Copy-on-write pages
- Better page-allocation model?
- Allow for more than one IOAPIC in ACPI module
- The objects get created, but GSI lookup only uses the one at index 0
- mark kernel memory pages global
- Serial out based on circular/bip biffer and interrupts, not spinning on
`write_ready()`
- Split out more code into kutil for testing
- AHCI / MSI interrupts on Vbox break?
- FXSAVE to save XMM registers.
- optimization using #NM (0x7) to detect SSE usage
- Clean up of process memory maps
- Better stack tracer
- Bootloader rewrite
- C++ and sharing library code for ELF, initrd, etc
- Parse initrd and pre-load certain ELF images, eg the process loader process?
- Do initial memory bootstrap?
- Calling global ctors
- Device Tree
- Actual serial driver
- Disk driver
- File system
- Multiprocessing
- Fast syscalls using syscall/sysret
### Build
- Clean up build generator and its templates
- More robust objects to represent modules & targets to pass to templates
- Read project setup from a simple JSON/TOML/etc
- Better compartmentalizing when doing template inheritance
- Move to LLD as sysroot linker

73
README.md
View File

@@ -9,7 +9,7 @@ Roadmap* section, below.
The design goals of the project are:
* Modernity - I'm not interested in designing for legacy systems, or running on
all hardware out there. My target is only 64 bit architecutres, and modern
all hardware out there. My target is only 64 bit architectures, and modern
commodity hardware. Currently that means x64 systems with Nehalem or newer
CPUs and UEFI firmware. (See [this list][cpu_features] for the currently
required CPU features.) Eventually I'd like to work on an AArch64 port,
@@ -58,12 +58,18 @@ Remaining to do:
- TLB shootdowns
- Page swapping
- Large / huge page support
_Physical page allocation: Sufficient._ The current physical page allocator
implementation suses a group of block representing up-to-1GiB areas of usable
implementation uses a group of blocks representing up-to-1GiB areas of usable
memory as defined by the bootloader. Each block has a three-level bitmap
denoting free/used pages.
Future work:
- Align blocks so that their first page is 1GiB-aligned, making finding free
large/huge pages easier.
#### Multitasking
_Sufficient._ The global scheduler object keeps separate ready/blocked lists
@@ -77,32 +83,37 @@ only the timekeeping task should be a separate kernel-only thread.
#### API
_In progress._ User-space tasks are able to make syscalls to the kernel via
fast SYSCALL/SYSRET instructions.
fast SYSCALL/SYSRET instructions. Syscalls made via `libj6` look to both the
callee and the caller like standard SysV ABI function calls. The
implementations are wrapped in generated wrapper functions which validate the
request, check capabilities, and find the appropriate kernel objects or handles
before calling the implementation functions.
Major tasks still to do:
Current work in this area:
- The process initialization protocol needs to be re-built entirely.
- Processes' handles to kernel objects need the ability to check capabilities
- The protocol for processes and their children for both initialization and
sharing resources. The kernel is ignorant of this, except for providing IPC
capabilities.
#### Hardware Support
* Framebuffer driver: _In progress._ Currently on machines with a video
device accessible by UEFI, jsix starts a user-space framebuffer driver that
only prints out kernel logs.
* Serial driver: _To do._ Machines without a video device should have a
user-space log output task like the framebuffer driver, but currently this
is done inside the kernel.
device accessible by UEFI, jsix starts a user-space framebuffer driver that
only prints out kernel logs.
* Serial driver: _In progress._ The current UART driver does not expose the
UART as an output resource yet, it merely gets the kernel logs and sends
them over serial.
* USB driver: _To do_
* AHCI (SATA) driver: _To do_
## Building
jsix uses the [Ninja][] build tool, and generates the build files for it with
the `configure` script. The build also relies on a custom sysroot, which can be
downloaded via the [Peru][] tool, or built locally.
the `configure` script. The build also relies on a custom toolchain sysroot, which can be
downloaded or built using the scripts in [jsix-os/toolchain][].
[Ninja]: https://ninja-build.org
[Peru]: https://github.com/buildinspace/peru
[Ninja]: https://ninja-build.org
[jsix-os/toolchain]: https://github.com/jsix-os/toolchain
Other build dependencies:
@@ -110,13 +121,11 @@ Other build dependencies:
* [nasm][]: the assembler
* [lld][]: the linker
* [mtools][]: for creating the FAT image
* [curl][]: if using `peru` below to download the sysroot
[clang]: https://clang.llvm.org
[nasm]: https://www.nasm.us
[lld]: https://lld.llvm.org
[mtools]: https://www.gnu.org/software/mtools/
[curl]: https://curl.se
The `configure` script has some Python dependencies - these can be installed via
`pip`, though doing so in a python virtual environment is recommended.
@@ -135,18 +144,14 @@ peru sync
### Setting up the sysroot
Running `peru sync` as in the above section will download and unpack the
toolchain into `sysroot`.
Build or download the toolchain sysroot as mentioned above with
[jsix-os/toolchain][], and symlink the built toolchain directory as `sysroot`
at the root of this project.
#### Compiling the sysroot yourself
If you have CMake installed, runing the `scripts/build_sysroot.sh` script will
download and build a LLVM toolchain configured for building the sysroot, and
then build the sysroot with it.
Built sysroots are actually stored in `~/.local/lib/jsix/sysroots` and installed
in the project dir via symbolic link, so having mulitple jsix working trees or
switching sysroot versions is easy.
```bash
# Example if both the toolchain and this project are cloned under ~/src
ln -s ~/src/toolchain/toolchains/llvm-13 ~/src/jsix/sysroot
```
### Building and running jsix
@@ -159,3 +164,15 @@ to run jsix in QEMU `-nographic` mode.
I personally run this either from a real debian amd64 bullseye machine or
a windows WSL debian bullseye installation. Your mileage may vary with other
setups and distros.
### Running the test suite
jsix now has the `test_runner` userspace program that runs various automated
tests. It is not included in the default build, but if you use the `test.yml`
manifest it will be built, and can be run with the `test.sh` script or the
`qemu.sh` script.
```bash
./configure --manifest=assets/manifests/test.yml
if ./test.sh; then echo "All tests passed!"; else echo "Failed."; fi
```

7
peru.yaml
View File

@@ -1,7 +0,0 @@
imports:
sysroot: sysroot/
curl module sysroot:
url: https://f000.backblazeb2.com/file/jsix-os/sysroot-llvm-11.tar.gz
unpack: tar
sha1: 7dffa388bdb2e20e1b0eb73600024e0c07d95abd

126
scripts/build_sysroot.sh
View File

@@ -1,126 +0,0 @@
#!/bin/bash
set -o errexit
set -o errtrace
set -o pipefail
set -o nounset
function errmsg () {
echo "Error: $(caller)"
echo "Working directory not cleaned up!"
}
trap errmsg ERR
LLVM_VERSION=11
J6_TOOLCHAINS="${J6_TOOLCHAINS:-$(realpath ~/.local/lib/jsix)}"
CHAIN_NAME="llvm-${LLVM_VERSION}"
LLVM_BRANCH="release/${LLVM_VERSION}.x"
SYSROOT="${J6_TOOLCHAINS}/sysroots/${CHAIN_NAME}"
ROOT=$(realpath ${1:-$(mktemp -d "sysroot_build.XXX")})
OUT="${ROOT}/sysroot"
echo "Working in ${ROOT}"
if [ ! -d "${ROOT}/llvm-project" ]; then
echo "* Downloading LLVM ${LLVM_VERSION}"
git clone -q --depth 1 \
--branch "${LLVM_BRANCH}" \
"https://github.com/llvm/llvm-project" \
"${ROOT}/llvm-project"
fi
if [ -d "${OUT}" ]; then
rm -rf "${OUT}"
fi
mkdir -p "${OUT}"
## Stage1: build an LLVM toolchain that doesn't rely on the host libraries,
## compiler, runtime, etc.
echo "* Configuring stage1"
mkdir -p "${ROOT}/stage1"
cmake -G Ninja \
-S "${ROOT}/llvm-project/llvm" \
-B "${ROOT}/stage1" \
-DCLANG_DEFAULT_RTLIB=compiler-rt \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER="clang" \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER="clang++" \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=lld" \
-DLIBCXXABI_USE_COMPILER_RT=ON \
-DLIBCXXABI_USE_LLVM_UNWINDER=ON \
-DLIBCXX_CXX_ABI=libcxxabi \
-DLIBUNWIND_USE_COMPILER_RT=ON \
-DLLVM_BUILD_TOOLS=OFF \
-DLLVM_ENABLE_BINDINGS=OFF \
-DLLVM_ENABLE_PROJECTS="clang;lld;libcxx;libcxxabi;libunwind;compiler-rt" \
-DLLVM_INCLUDE_BENCHMARKS=OFF \
-DLLVM_INCLUDE_EXAMPLES=OFF \
-DLLVM_INCLUDE_TESTS=OFF \
-DLLVM_INCLUDE_UTILS=OFF \
-DLLVM_TARGETS_TO_BUILD="X86" \
-DLLVM_USE_LINKER="lld" \
2>&1 > "${ROOT}/configure_stage1.log"
echo "* Building stage1"
ninja -C "${ROOT}/stage1" -v > "${ROOT}/build_stage1.log"
STAGE1="${ROOT}/stage1/bin"
## Stage2: build the sysroot libraries with the stage1 toolchain
echo "* Configuring stage2"
mkdir -p "${ROOT}/stage2"
cmake -G Ninja \
-S "${ROOT}/llvm-project/llvm" \
-B "${ROOT}/stage2" \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER="${STAGE1}/clang" \
-DCMAKE_C_COMPILER_LAUNCHER=ccache \
-DCMAKE_CXX_COMPILER="${STAGE1}/clang++" \
-DCMAKE_CXX_COMPILER_LAUNCHER=ccache \
-DCMAKE_INSTALL_PREFIX="${OUT}" \
-DCMAKE_SHARED_LINKER_FLAGS="-fuse-ld=${STAGE1}/ld.lld" \
-DLIBCXXABI_ENABLE_STATIC_UNWINDER=ON \
-DLIBCXXABI_USE_COMPILER_RT=ON \
-DLIBCXXABI_USE_LLVM_UNWINDER=ON \
-DLIBCXX_CXX_ABI=libcxxabi \
-DLIBCXX_INCLUDE_BENCHMARKS=OFF \
-DLIBUNWIND_USE_COMPILER_RT=ON \
-DLLVM_DEFAULT_TARGET_TRIPLE="x86_64-unknown-elf" \
-DLLVM_ENABLE_BINDINGS=OFF \
-DLLVM_ENABLE_LIBCXX=ON \
-DLLVM_ENABLE_PROJECTS="libcxx;libcxxabi;libunwind" \
-DLLVM_INCLUDE_BENCHMARKS=OFF \
-DLLVM_INCLUDE_EXAMPLES=OFF \
-DLLVM_INCLUDE_TESTS=OFF \
-DLLVM_INCLUDE_TOOLS=OFF \
-DLLVM_INCLUDE_UTILS=OFF \
-DLLVM_TARGETS_TO_BUILD="X86" \
-DLLVM_USE_LINKER="${STAGE1}/ld.lld" \
2>&1 > "${ROOT}/configure_stage2.log"
echo "* Building stage2"
ninja -C "${ROOT}/stage2" -v > "${ROOT}/build_stage2.log"
echo "* Installing stage2"
ninja -C "${ROOT}/stage2" -v install/strip > "${ROOT}/install_stage2.log"
echo "* Installing new sysroot"
rm -rf "${SYSROOT}"
mv "${OUT}" "${SYSROOT}"
echo "* Cleaning up"
trap - ERR
rm -rf "${ROOT}"
echo "Done"

14
scripts/readme_for_prebuilt_sysroots.md
View File

@@ -1,14 +0,0 @@
# jsix OS sysroot
This is a pre-built sysroot for building the jsix operating system kernel,
bootloader, and utilities. This package is provided as a convenience, and
contains software from the following repositories.
## The LLVM toolchain
The LLVM sources as downloaded via git from [llvm.org][llvm] under the terms of
the [Apache License v2.0][apache2], modified [as described here][llvmlic].
[llvm]: https://llvm.org
[apache2]: https://www.apache.org/licenses/LICENSE-2.0
[llvmlic]: https://llvm.org/docs/DeveloperPolicy.html#new-llvm-project-license-framework