I just can't get enum_bitfields to work in boot. The same code works in
other targets. None of the compiler options should change that. I gave
up, but I'm leaving these changes in because they do actually handle
const better.
This is a rather large commit that is widely focused on cleaning things
out of the 'junk drawer' that is src/include. Most notably, several
things that were put in there because they needed somewhere where both
the kernel, boot, and init could read them have been moved to a new lib,
'bootproto'.
- Moved kernel_args.h and init_args.h to bootproto as kernel.h and
init.h, respectively.
- Moved counted.h and pointer_manipulation.h into util, renaming the
latter to util/pointers.h.
- Created a new src/include/arch for very arch-dependent definitions,
and moved some kernel_memory.h constants like frame size, page table
entry count, etc to arch/amd64/memory.h. Also created arch/memory.h
which detects platform and includes the former.
- Got rid of kernel_memory.h entirely in favor of a new, cog-based
approach. The new definitions/memory_layout.csv lists memory regions
in descending order from the top of memory, their sizes, and whether
they are shared outside the kernel (ie, boot needs to know them). The
new header bootproto/memory.h exposes the addresses of the shared
regions, while the kernel's memory.h gains the start and size of all
the regions. Also renamed the badly-named page-offset area the linear
area.
- The python build scripts got a few new features: the ability to parse
the csv mentioned above in a new memory.py module; the ability to add
dependencies to existing source files (The list of files that I had to
pull out of the main list just to add them with the dependency on
memory.h was getting too large. So I put them back into the sources
list, and added the dependency post-hoc.); and the ability to
reference 'source_root', 'build_root', and 'module_root' variables in
.module files.
- Some utility functions that were in the kernel's memory.h got moved to
util/pointers.h and util/misc.h, and misc.h's byteswap was renamed
byteswap32 to be more specific.
Now that kutil has no kernel-specific code in it anymore, it can
actually be linked to by anything, so I'm renaming it 'util'.
Also, I've tried to unify the way that the system libraries from
src/libraries are #included using <> instead of "".
Other small change: util::bip_buffer got a spinlock to guard against
state corruption.
Continuing moving things out of kutil. The assert as implemented could
only ever work in the kernel, so remaining kutil uses of kassert have
been moved to including standard C assert instead.
Along the way, kassert was broken out into panic::panic and kassert,
and the panic.serial namespace was renamed panicking.
The moving of kernel-only code out of kutil continues. (See 042f061)
This commit moves the following:
- The heap allocator code
- memory.cpp/h which means:
- letting string.h be the right header for memset and memcpy, still
including an implementation of it for the kernel though, since
we're not linking libc to the kernel
- Changing calls to kalloc/kfree to new/delete in kutil containers
that aren't going to be merged into the kernel
- Fixing a problem with stdalign.h from libc, which was causing issues
for type_traits.
Part two of rearranging kutil code. (See 042f061) Removing unused kutil
headers:
I can imagine that avl_tree or slab_allocated may want to be returned to
at some point, but for now they're just clutter.
Part one of a series of code moves. The kutil library is not very
useful, as most of its code is kernel-specific. This was originally for
testing purposes, but that can be achieved in other ways with the
current build system. I find this mostly creates a strange division in
the kernel code.
Instead, I'm going to move everything kernel-specific to actually be in
the kernel, and replace kutil with just 'util' for generic utility code
I want to share.
This commit:
- Moves the logger into the kernel.
- Updates the 'printf' library used from mpaland/printf to
eyalroz/printf and moved it into the kernel, as it's only used by the
logger in kutil.
- Removes some other unused kutil headers from some files, to help
future code rearrangement.
Note that the (now redundant-seeming) log.cpp/h in kernel is currently
still there - these files are more about log output than the logging
system, and will get replaced once I add user-space log output.
Fixing two page allocation issues I came across while debugging:
- Added a spinlock to the page_table static page cache, to avoid
multiple CPUs grabbing the same page. This cache should probably
just be made into per-CPU caches.
- Fixed a bitwise math issue ("1" instead of "1ull" when working with
64-bit numbers) that made it so that pages were never marked as
allocated when allocating 32 or more.
The j6tw (j6 table walk) command to debug page tables was throwing an
exception for an integer that was too big when the default radix was 16,
because it would interpret ints as hex even without the 0x prefix. Now
j6tw explicitly converts to hex and uses the prefix to be explicit.
Updated kassert to be an actual function, and used the __builtin_*
functions for location data. Updated the panic handler protocol to
include sending location data as three more parameters. Updated the
serial panic handler to display that data along with the (optional)
message.
Add a simple ELF loader to srv.init to load and start any module_program
parameters passed from the bootloader. Also creates stacks for newly
created threads.
Also update thread creation in testapp to create stacks.
Stop creating stacks in user space for user threads, that should be done
by the thread's creator. This change adds process and stack_top
arguments to the thread_create syscall, so that threads can be created
in other processes, and given a stack address.
Also included is a fix in add_thunk_user due to the r11/flags change.
THIS COMMIT BREAKS USERSPACE. See subsequent commits for the user side
changes related to this change.
LLVM was updated by system update, and is now unhappy building because
of libc++ threads settings. I'm explicitly turning them off in base.yml
for now until I can focus on a better fix.
Bonnibel used to copy any asset into the root - now a path can be
specified for assets, and debug output goes into .debug so GDB will pick
it up automatically.
The vm_area objects had a number of issues I have been running into when
working on srv.init:
- It was impossible to map a VMA, fill it, unmap it, and hand it to
another process. Unmapping the VMA in this process would cause all the
pages to be freed, since it was removed from its last mapping.
- If a VMA was marked with vm_flag::zero, it would be zeroed out _every
time_ it was mapped into a vm_space.
- The vm_area_open class was leaking its page_tree nodes.
In order to fix these issues, the different VMA types all work slightly
differently now:
- Physical pages allocated for a VMA are now freed when the VMA is
deleted, not when it is unmapped.
- A knock-on effect from the first point is that vm_area_guarded is now
based on vm_area_open, instead of vm_area_untracked. An untracked area
cannot free its pages, since it does not track them.
- The vm_area_open type now deletes its root page_tree node. And
page_tree nodes will delete child nodes or free physical pages in
their dtors.
- vm_flag::zero has been removed; pages will need to be zeroed out
further at a higher level.
- vm_area also no longer deletes itself only on losing its last handle -
it will only self-delete when all handles _and_ mappings are gone.
The page_tree struct was doing a lot of bit manipulation to keep its
base, level, and flags in a single uint64_t. But since this is such a
large structure anyway, another word doesn't change it much and greatly
simplifies both the code and reasoning about it.
Overall, I believe TOML to be a superior configuration format than YAML
in many situations, but it gets ugly quickly when nesting data
structures. The build configs were fine in TOML, but the manifest (and
my future plans for it) got unwieldy. I also did not want different
formats for each kind of configuration on top of also having a custom
DSL for interface definitions, so I've switched all the TOML to YAML.
Also of note is that this change actually adds structure to the manifest
file, which was little more than a CSV previously.
This change adds a new interface DSL for specifying objects (with
methods) and interfaces (that expose objects, and optionally have their
own methods).
Significant changes:
- Add the new scripts/definitions Python module to parse the DSL
- Add the new definitions directory containing DSL definition files
- Use cog to generate syscall-related code in kernel and libj6
- Unify ordering of pointer + length pairs in interfaces
This change moves Bonnibel from a separate project into the jsix tree,
and alters the project configuration to be jsix-specific. (I stopped
using bonnibel for any other projects, so it's far easier to make it a
custom generator for jsix.) The build system now also uses actual python
code in `*.module` files to configure modules instead of TOML files.
Target configs (boot, kernel-mode, user-mode) now moved to separate TOML
files under `configs/` and can inherit from one another.
I'm a tabs guy. I like tabs, it's an elegant way to represent
indentation instead of brute-forcing it. But I have to admit that the
world seems to be going towards spaces, and tooling tends not to play
nice with tabs. So here we go, changing the whole repo to spaces since
I'm getting tired of all the inconsistent formatting.
I was seeing more ignored interrupts when debugging, trying to shorten
their path more. Adding a separate ISR for ignored interrupts was the
shortest path, but that caused some strange instability that I'm not in
the mood to track down.
When setting a radix other than 10, the j6stack command will start
adding the wrong values - make sure to specify a radix for the offset
explicitly.
Also show the frame pointer for each frame with the j6bt command, and
don't throw exceptions if the name of the block is unknown.
Created the framework for using different loadable panic handlers,
loaded by the bootloader. Initial panic handler is panic.serial, which
contains its own serial driver and stacktrace code.
Other related changes:
- Asserts are now based on the NMI handler - panic handlers get
installed as the NMI interrupt handler
- Changed symbol table generation: now use nm's own demangling and
sorting, and include symbol size in the table
- Move the linker script argument out of the kernel target, and into the
kernel's specific module, so that other programs (ie, panic handlers)
can use the kernel target as well
- Some asm changes to boot.s to help GDB see stack frames - but this
might not actually be all that useful
- Renamed user_rsp to just rsp in cpu_state - everything in there is
describing the 'user' state
Resurrect the existing but unused ELF library in libraries/elf, and use
it instead of boot/elf.h for parsing ELF files in the bootloader.
Also adds a const version of offset_iterator called
const_offset_iterator.
Pull this widely-useful header out of kutil, so more things can use it.
Also replace its dependency on <type_traits> by defining our own custom
basic_types.h which contains a subset of the standard's types.
Create a new usermode program, srv.init, and have it read the initial
module_page args sent to it by the bootloader. Doesn't yet do anything
useful but sets up the way for loading the rest of the programs from
srv.init.
Other (mostly) related changes:
- bootloader: The allocator now has a function for allocating init
modules out of a modules_page slab. Also changed how the allocator is
initialized and passes the allocation register and modules_page list
to efi_main().
- bootloader: Expose the simple wstrlen() to the rest of the program
- bootloader: Move check_cpu_supported() to hardware.cpp
- bootloader: Moved program_desc to loader.h and made the loader
functions take it as an argument instead of paths.
- kernel: Rename the system_map_mmio syscall to system_map_phys, and
stop having it default those VMAs to having the vm_flags::mmio flag.
Added a new flag mask, vm_flags::driver_mask, so that drivers can be
allowed to ask for the MMIO flag.
- kernel: Rename load_simple_process() to load_init_server() and got rid
of all the stack setup routines in memory_bootstrap.cpp and task.s
- Fixed formatting in config/debug.toml, undefined __linux and other
linux-specific defines, and got rid of _LIBCPP_HAS_THREAD_API_EXTERNAL
because that's just not true.
Separate the video mode setting out from the console code into video.*,
and remove the framebuffer from the kernel args, moving it to the new
init args format.
A long overdue cleanup of the src/ tree.
- Moved src/drivers to src/user because it contains more than drivers
- Removed src/drivers/ahci because it's unused - will restore it when I
make a real AHCI driver
- Removed unused src/tools
- Moved kernel.ld (the only used file under src/arch) to src/kernel for
now, if/when there's a multi-platform effort that should be figured
out as part of it
- Removed the rest of the unused src/arch
- Renamed 'fb' to 'drv.uefi_fb' and 'nulldrv' to 'testapp'
The bootloader's load_program was reproducing all loadable program
header sections into new pages. Now only do that for sections containing
BSS sections (eg, where file size and mem size do not match).
