Overall, I believe TOML to be a superior configuration format than YAML
in many situations, but it gets ugly quickly when nesting data
structures. The build configs were fine in TOML, but the manifest (and
my future plans for it) got unwieldy. I also did not want different
formats for each kind of configuration on top of also having a custom
DSL for interface definitions, so I've switched all the TOML to YAML.
Also of note is that this change actually adds structure to the manifest
file, which was little more than a CSV previously.
This change adds a new interface DSL for specifying objects (with
methods) and interfaces (that expose objects, and optionally have their
own methods).
Significant changes:
- Add the new scripts/definitions Python module to parse the DSL
- Add the new definitions directory containing DSL definition files
- Use cog to generate syscall-related code in kernel and libj6
- Unify ordering of pointer + length pairs in interfaces
This change moves Bonnibel from a separate project into the jsix tree,
and alters the project configuration to be jsix-specific. (I stopped
using bonnibel for any other projects, so it's far easier to make it a
custom generator for jsix.) The build system now also uses actual python
code in `*.module` files to configure modules instead of TOML files.
Target configs (boot, kernel-mode, user-mode) now moved to separate TOML
files under `configs/` and can inherit from one another.
I'm a tabs guy. I like tabs, it's an elegant way to represent
indentation instead of brute-forcing it. But I have to admit that the
world seems to be going towards spaces, and tooling tends not to play
nice with tabs. So here we go, changing the whole repo to spaces since
I'm getting tired of all the inconsistent formatting.
I was seeing more ignored interrupts when debugging, trying to shorten
their path more. Adding a separate ISR for ignored interrupts was the
shortest path, but that caused some strange instability that I'm not in
the mood to track down.
When setting a radix other than 10, the j6stack command will start
adding the wrong values - make sure to specify a radix for the offset
explicitly.
Also show the frame pointer for each frame with the j6bt command, and
don't throw exceptions if the name of the block is unknown.
Created the framework for using different loadable panic handlers,
loaded by the bootloader. Initial panic handler is panic.serial, which
contains its own serial driver and stacktrace code.
Other related changes:
- Asserts are now based on the NMI handler - panic handlers get
installed as the NMI interrupt handler
- Changed symbol table generation: now use nm's own demangling and
sorting, and include symbol size in the table
- Move the linker script argument out of the kernel target, and into the
kernel's specific module, so that other programs (ie, panic handlers)
can use the kernel target as well
- Some asm changes to boot.s to help GDB see stack frames - but this
might not actually be all that useful
- Renamed user_rsp to just rsp in cpu_state - everything in there is
describing the 'user' state
Resurrect the existing but unused ELF library in libraries/elf, and use
it instead of boot/elf.h for parsing ELF files in the bootloader.
Also adds a const version of offset_iterator called
const_offset_iterator.
Pull this widely-useful header out of kutil, so more things can use it.
Also replace its dependency on <type_traits> by defining our own custom
basic_types.h which contains a subset of the standard's types.
Create a new usermode program, srv.init, and have it read the initial
module_page args sent to it by the bootloader. Doesn't yet do anything
useful but sets up the way for loading the rest of the programs from
srv.init.
Other (mostly) related changes:
- bootloader: The allocator now has a function for allocating init
modules out of a modules_page slab. Also changed how the allocator is
initialized and passes the allocation register and modules_page list
to efi_main().
- bootloader: Expose the simple wstrlen() to the rest of the program
- bootloader: Move check_cpu_supported() to hardware.cpp
- bootloader: Moved program_desc to loader.h and made the loader
functions take it as an argument instead of paths.
- kernel: Rename the system_map_mmio syscall to system_map_phys, and
stop having it default those VMAs to having the vm_flags::mmio flag.
Added a new flag mask, vm_flags::driver_mask, so that drivers can be
allowed to ask for the MMIO flag.
- kernel: Rename load_simple_process() to load_init_server() and got rid
of all the stack setup routines in memory_bootstrap.cpp and task.s
- Fixed formatting in config/debug.toml, undefined __linux and other
linux-specific defines, and got rid of _LIBCPP_HAS_THREAD_API_EXTERNAL
because that's just not true.
Separate the video mode setting out from the console code into video.*,
and remove the framebuffer from the kernel args, moving it to the new
init args format.
A long overdue cleanup of the src/ tree.
- Moved src/drivers to src/user because it contains more than drivers
- Removed src/drivers/ahci because it's unused - will restore it when I
make a real AHCI driver
- Removed unused src/tools
- Moved kernel.ld (the only used file under src/arch) to src/kernel for
now, if/when there's a multi-platform effort that should be figured
out as part of it
- Removed the rest of the unused src/arch
- Renamed 'fb' to 'drv.uefi_fb' and 'nulldrv' to 'testapp'
The bootloader's load_program was reproducing all loadable program
header sections into new pages. Now only do that for sections containing
BSS sections (eg, where file size and mem size do not match).
The bootloader relied on the kernel to know which parts of memory to not
allocate over. For the future shift of having the init process load
other processes instead of the kernel, the bootloader needs a mechanism
to just hand the kernel a list of allocations. This is now done through
the new bootloader allocator, which all allocation goes through. Pool
memory will not be tracked, and so can be overwritten - this means the
args structure and its other structures like programs need to be handled
right away, or copied by the kernel.
- Add bootloader allocator
- Implement a new linked-list based set of pages that act as allocation
registers
- Allow for operator new in the bootloader, which goes through the
global allocator for pool memory
- Split memory map and frame accouting code in the bootloader into
separate memory_map.* files
- Remove many includes that could be replaced by forward declaration in
the bootloader
- Add a new global template type, `counted`, which replaces the
bootloader's `buffer` type, and updated kernel args structure to use it.
- Move bootloader's pointer_manipulation.h to the global include dir
- Make offset_iterator try to return references instead of pointers to
make it more consistent with static array iteration
- Implement a stub atexit() in the bootloader to satisfy clang
The scheduler queue locks could deadlock if the timer fired before the
scoped lock destructor ran. Also, reduce lock contention by letting only
one CPU steal work at a time.
Spinlock release uses __atomic_compare_exchange_n, which overwrites the
`desired` parameter with the actual value when the compare fails. This
was causing releases to always spin when there was lock contention.
Add the object_wait_many syscall to allow programs to wait for signals
on multiple objects at once. Also removed the object argument to
thread::wait_on_signals, which does nothing with it. That information is
saved in the thread being in the object's blocked threads list.
The kernel's file header has not been verified for a long time. This
change returns file verification to the bootloader to make sure the ELF
loaded in position 0 is actually the kernel.
The kernel::args namespace is really the protocol for initializing the
kernel from the bootloader. Also, the header struct in that namespace
isn't actually a header, but a collection of parameters. This change
renames the namespace to kernel::init and the struct to args.
Update the build_sysroot to build an LLVM 11 sysroot, dealing with
LLVM's change in structure. Also re-add peru.yaml for getting a
pre-built sysroot via peru.
See: https://github.com/buildinspace/peru
Update the cpu data to point to the fake kernel process in
cpu_early_init so there can never be a race condition where the current
process may not be set.
The idle threads for the APs have intentionally tiny stacks. Logging is
currently an absolute hog of stack space, so avoid logging on the idle
stacks as much as possible.
Eventually we should instead just reclaim the physical pages used by
most of the stack instead of making them tiny.
Changing the SFINAE/enable_if strategy from a type to a constexpr
function means that it can be defined in other scopes than the functions
themselves, because of function overloading. This lets us put everything
into the kutil::bitfields namespace, and make bitfields out of enums in
other namespaces. Also took the chance to clean up the implementation a
bit.
Now that the other CPUs have been brought up, add support for scheduling
tasks on them. The scheduler now maintains separate ready/blocked lists
per CPU, and CPUs will attempt to balance load via periodic work
stealing.
Other changes as a result of this:
- The device manager no longer creates a local APIC object, but instead
just gathers relevant info from the APCI tables. Each CPU creates its
own local APIC object. This also spurred the APIC timer calibration to
become a static value, as all APICs are assumed to be symmetrical.
- Fixed a bug where the scheduler was popping the current task off of
its ready list, however the current task is never on the ready list
(except the idle task was first set up as both current and ready).
This was causing the lists to get into bad states. Now a task can only
ever be current or in a ready or blocked list.
- Got rid of the unused static process::s_processes list of all
processes, instead of trying to synchronize it via locks.
- Added spinlocks for synchronization to the scheduler and logger
objects.
KVM didn't like setting all the CR4 bits we wanted at once. I suspect
that means real hardware won't either. Delay the setting of the rest of
CR4 until after the CPU is in long mode - only set PAE and PGE from real
mode.
