This commit fixes the mailbox tests in test_runner, which broke when
mailbox was simplified to just use call and respond. It also fixes a
bug the tests uncovered: if the mailbox is closed while a caller is in
the reply map (ie, when its call data has been passed on to a thread
calling respond, but has yet to be responded to itself), that caller is
never awoken.
This commit changes the add_user_thunk to point to a new routine,
initialize_user_cpu, which sets all the registers that were previously
unset when starting a new user thread. The values for rdi and rsi are
popped off the initial stack values that add_user_thunk sets up, so that
user thread procs can take up to two arguments.
To suppor this, j6_thread_create gained two new arguments, which are
passed on to the thread.
This also let me finally get rid of the hack of passing an argument in
rsp when starting init.
This commit re-adds testapp to the default manifest and does some
housecleaning on the module:
- Remove the old serial.* and io.*
- Update it to use current syscall APIs
- Update it to use libj6's higher-level thread API
The debugcon logger is now separate from logger::output, and is instead
a kernel-internal thread that watches for logs and prints them to the
deubcon device.
A bip-buffer is good for producer/consumer systems, but ideally logs
will stay in the buffer until they're ousted because they need to be
overwritten. Now they're a regular ring buffer and every entry has an
incremental id. Consumers pass in the last id they've seen, and will get
the next log in the sequence.
This ended up being unused, but still probably useful: Refactor out the
"find" logic of page_tree::find_or_add (note that this is different than
the "find" logic of page_tree::find, as it potentially modifies the tree
to add a location to accommodate the page being searched for) into a new
page_tree::get_entry method. That was then used to add an add_existing
method for inserting pages into the page_tree.
In prep for the coming change to keep log entries as a true ring buffer,
move the log buffer from bss into its own memory section.
Related changes in this commit:
- New vm_area_ring, which maps a set of pages twice to allow easy linear
reading of data from a ring buffer when it wraps around the end.
- logger_init() went away, and the logger ctor is called from
mem::initialize()
- Instead of an event object, the logger just has a bare wait_queue
- util::counted::from template type changed slightly to allow easy
conversion from an intptr_t as well as a pointer
- Previously added debugcon_logger code removed - this will be added in
a separate file in a followup commit
Previously we were hard-coding loading specific files (the UART driver
and logging server) from the initrd. Now j6romfs has a for_each() method
to allow iterating all files in a directory, and init loads all programs
from /jsix/drivers and /jsix/services. Eventually this will need more
dynamic loading decisions for drivers but for now it's fine.
There have been a number of incidents lately where I've needed to see
logs but have been working on init, and broken the log output of
srv.logger. This commit adds a debug console output to io port 0x6600
if enabled at the top of logger.cpp.
I often want to use util::counted as a bool, like with a regular
pointer. There was some basic support for that, but it didn't cover
every case - now it should.
The syscall helpers.h get_handle functions should be returing
j6_err_invalid_arg if the handle they're given is j6_handle_invalid,
unless explicitly set to optional.
Make build_symbol_table.py output statistics on the symbol table it
builds, and emit warnings for zero-length symbols. Also added lengths to
several functions defined in asm that this uncovered.
Restructuring paging into an object that carries its page cache with it
and makes for simpler code. Program loading is also changed to not copy
the pages loaded from the file into new pages - we can impose a new
constraint that anything loaded by boot have a simple, page-aligned
layout so that we can just map the existing pages into the right
addresses. Also included are some linker script changes to help
accommodate this.
The symbol table needs to be passed to the panic handler very early in
the kernel, loading it in init is far less useful. Return it to the boot
directory and remove it from the initrd.
I added util::format as a replacement for other printf implementations
last year, but it sat there only being used by the kernel all this time.
Now I've templated it so that it can be used by the bootloader, and
removed printf from panic.serial as well.
Previously, to add a custom linker script, a module would need to modify
its variables after the fact to add to ldflags. Now module constructors
take a new keyword `ld_script` and handle the ldflags and dependencies
properly.
Previously, status_line would show 'ok' until something went wrong. Now,
show an empty box until either a warning or error happens, or the
cleanup happens without an issue.
Load drv.uart.elf and srv.logger.elf from the initrd and start them.
It's extremely manual and hard-coded at the moment, but it works and
they run, getting us back to where we were pre-initrd branch.
Using `-fvisibility=hidden` when building the kernel, and then
`--discard-all` when stripping it, we shave almost 100KiB off of the
resulting ELF file.
Also dropped some unused symbols from the linker script, and rearranged
the sections so that the file is able to be mapped directly into memory
instead of having each section copied.
It felt clunky to have zstd.module in src/libraries/zstd by itself, and
doesn't make much sense in src/libraries as it's an external library
anyway.
Now the ./configure script will pick up .module files in the top-level
external directory as well.
CDB seemed to be too simple for the needs of init, and squashfs is too
laden with design choices to work around Linux's APIs. This commit adds
creation of an initrd image of a new format I've called `j6romfs`.
Note that this commit currently does not work! The initrd-reading code
still needs to be added.
I've been using `const util::buffer` a lot of places where that's not
really what I mean, because I was avoiding having a separate type for
buffers with const pointers. But really this needed to happen.
The initrd image is now created by the build system, loaded by the
bootloader, and passed to srv.init, which loads it (but doesn't do
anything with it yet, so this is actually a functional regression).
This simplifies a lot of the modules code between boot and init as well:
Gone are the many subclasses of module and all the data being inline
with the module structs, except for any loaded files. Now the only
modules loaded and passed will be the initrd, and any devices only the
bootloader has knowledge of, like the UEFI framebuffer.
A new compressed initrd format for srv.init to load drivers, services,
and data from, instead of every file getting loaded by the bootloader.
This will allow for less memory allocated by the bootloader and passed
to init if not every driver or data file is loaded.
Loading, passing, and using the new initrd will be done in a coming
commit.
I discovered that QEMU has a "remote" option to the `-vnc` switch, which
causes it to reverse-connect to a VNC client on start. I've added this
into qemu.sh as the -r option.
Having main.cpp in the kernel and in the application being debugged is
annoying when setting breakpoints, so just like with main() vs
kernel_main(), kernel/main.cpp is now kernel/kernel_main.cpp.
The clang __builtin_* functions cannot be relied upon, as they may just
emit a call to the stdlib version. So this commit adds an implementation
for ceil and frexpr, as well as their float versions.
In preparation for futexes, I wanted to make kobjects a bit lighter.
Storing 32 bits of object id, and 8 bits of type (and not ending the
class in a ushort for handle count, which meant all kobjects were likely
to have a bunch of pad bytes), the kobject class data is now just one 8
byte word.
Also from this, change logs that mention threads or processes from
printing the full koid to just 2 bytes of object id from both process
and thread, which makes following the logs much easier.
This new class makes it easier for user programs to spawn threads. This
change also includes support for .hh files in modules, to differentiate
headers that are C++-only in system libraries.
The status code from thread exit had too many issues, (eg, how does it
relate to process exit code? what happens when different threads exit
with different exit codes?) and not enough value, so I'm getting rid of
it.
In the new mailbox structure, passing a j6_handle_invalid with a message
would result in a permission denied result, as the process did not have
a handle "0".
When the bootloader loads srv.init's program sections into memory, it
needed to page-align them if they weren't. srv.init's loader itself
handles this case, but the bootloader's did not.
heap_allocator::reallocate relies on the allocate and free methods so
mostly doesn't need locking, but it does touch the tracking map, so
needs to protect that with a lock.
A number of simplifications of mailboxes now that the interface is much
simpler, and synchronous.
* call and respond can now only transfer one handle at a time
* mailbox objects got rid of the message queue, and just have
wait_queues of blocked threads, and a reply_to map.
* threads now have a message_data struct on them for use by mailboxes
Instead of handles / capabilities having numeric ids that are only valid
for the owning process, they are now global in a system capabilities
table. This will allow for specifying capabilities in IPC that doesn't
need to be kernel-controlled.
Processes will still need to be granted access to given capabilities,
but that can become a simpler system call than the current method of
sending them through mailbox messages (and worse, having to translate
every one into a new capability like was the case before). In order to
track which handles a process has access to, a new node_set based on
node_map allows for an efficient storage and lookup of handles.
