These are some changes I made to debug tooling while tracking down the
bugfix in the previous commit.
Each `scripts/debug_*_alloc.gdb` script has gdb output a `*_allocs.txt`
file, which in turn can be parsed by the `scripts/parse_*_allocs.py`
script to find errors.
Initial support for XSAVE, but not XSAVEOPT or XSAVEC:
- Enable XSAVE and set up xcr0 for all CPUs
- Allocate XSAVE area for all non-kernel threads
- Call XSAVE and XRSTOR on task switch
The upcoming futex syscalls will be easier to use (and to auto verify)
if passed a pointer instead of an address, this allows for changing a
`Primitive` to a `PrimitiveRef` by adding a `*` to the end.
Now that the log ring buffer is at a fixed known address, and entries
persist even after being read, it's easy to add a command to see what's
in the buffer from GDB. Useful if there's some log messages that hadn't
yet been printed at the time of a panic.
Two minor debugging helpers:
- the GDB script was still referencing m_koid on objects, switched to
the replacement m_obj_id instead.
- finally gave in and made panic print 1-based CPU ids like GDB uses
instead of 0-based like the hardware and LITERALLY EVERYTHING ELSE
This commit fixes the mailbox tests in test_runner, which broke when
mailbox was simplified to just use call and respond. It also fixes a
bug the tests uncovered: if the mailbox is closed while a caller is in
the reply map (ie, when its call data has been passed on to a thread
calling respond, but has yet to be responded to itself), that caller is
never awoken.
This commit re-adds testapp to the default manifest and does some
housecleaning on the module:
- Remove the old serial.* and io.*
- Update it to use current syscall APIs
- Update it to use libj6's higher-level thread API
The debugcon logger is now separate from logger::output, and is instead
a kernel-internal thread that watches for logs and prints them to the
deubcon device.
The initrd image is now created by the build system, loaded by the
bootloader, and passed to srv.init, which loads it (but doesn't do
anything with it yet, so this is actually a functional regression).
This simplifies a lot of the modules code between boot and init as well:
Gone are the many subclasses of module and all the data being inline
with the module structs, except for any loaded files. Now the only
modules loaded and passed will be the initrd, and any devices only the
bootloader has knowledge of, like the UEFI framebuffer.
Adding pretty printers to aid in debugging:
* For the cap_table type so that `p g_cap_table` displays a neat table
* For node_sets of handles to easily see what handles a process owns
* For util::vector to include its contents in the output
Split the functionality of outputting kernel logs out of the UART
driver, and into a new service. The UART driver now registers a console
out channel with the service locator, which the logger service
retrieves, and then enters a loop getting logs from the kernel and
printing them out to the console.
Added profiler.h which defines classes and macros for defining profiler
objects. Also added gdb command j6prof for printing profile data. Added
the syscall_profiles profiler class and auto wrapping of syscalls with
profile objects.
Other changes in this commit:
- Made the gdb command `j6threads` argument for specifying a CPU
optional. Without an argument, it loops through all CPUs.
- Switched to -mcmodel=kernel for kernel code, which makes `call`
instructions easier to follow when debugging / looking at disassembly.
The new mailbox kernel object API offers asynchronous message-based IPC
for sending data and handles between threads, as opposed to endpoint's
synchronous model.
This change introduces test_runner, which runs unit or integration tests
and then tells the kernel to exit QEMU with a status code indicating the
number of failed tests.
The test_runner program is not loaded by default. Use the test manifest
to enable it:
./configure --manifest=assets/manifests/test.yml
A number of tests from the old src/tests have moved over. More to come,
as well as moving code from testapp before getting rid of it.
The test.sh script has been repurposed to be a "headless" version of
qemu.sh for running tests, and it exits with the appropriate exit code.
(Though ./qemu.sh gained the ability to exit with the correct exit code
as well.) Exit codes from kernel panics have been updated so that the
bash scripts should exit with code 127.
The manifest can now supply a list of boot flags, including "test".
Those get turned into the bootproto::args::flags field by the
bootloader. The kernel takes those and uses the test flag to control
enabling syscalls with the new "test" attribute, like the new
test_finish syscall, which lets automated tests call back to the kernel
to shut down the system.
The main point of this change is to allow "global" capabilities defined
on the base object type. The example here is the clone capability on all
objects, which governs the ability to clone a handle.
Related changes in this commit:
- Renamed `kobject` to `object` as far as the syscall interface is
concerned. `kobject` is the cname, but j6_cap_kobject_clone feels
clunky.
- The above change made me realize that the "object <type>" syntax for
specifying object references was also clunky, so now it's "ref <type>"
- Having to add `.object` on everywhere to access objects in
interface.exposes or object.super was cumbersome, so those properties
now return object types directly, instead of ObjectRef.
- syscall_verify.cpp.cog now generates code to check capabilities on
handles if they're specified in the definition, even when not passing
an object to the implementation function.
This change finally adds capabilities to handles. Included changes:
- j6_handle_t is now again 64 bits, with the highest 8 bits being a type
code, and the next highest 24 bits being the capability mask, so that
programs can check type/caps without calling the kernel.
- The definitions grammar now includes a `capabilities [ ]` section on
objects, to list what capabilities are relevant.
- j6/caps.h is auto-generated from object capability lists
- init_libj6 again sets __handle_self and __handle_sys, this is a bit
of a hack.
- A new syscall, j6_handle_list, will return the list of existing
handles owned by the calling process.
- syscall_verify.cpp.cog now actually checks that the needed
capabilities exist on handles before allowing the call.
Added a list of currently-set flags on the thread's state. Also stopped
tracebacks and returned instead of erroring out when they threw
gdb.MemoryError.
This commit contains a couple large, interdependent changes:
- In preparation for capability checking, the _syscall_verify_*
functions now load most handles passed in, and verify that they exist
and are of the correct type. Lists and out-handles are not converted
to objects.
- Also in preparation for capability checking, the internal
representation of handles has changed. j6_handle_t is now 32 bits, and
a new j6_cap_t (also 32 bits) is added. Handles of a process are now a
util::map<j6_handle_t, handle> where handle is a new struct containing
the id, capabilities, and object pointer.
- The kernel object definition DSL gained a few changes to support auto
generating the handle -> object conversion in the _syscall_verify_*
functions, mostly knowing the object type, and an optional "cname"
attribute on objects where their names differ from C++ code.
(Specifically vma/vm_area)
- Kernel object code and other code under kernel/objects is now in a new
obj:: namespace, because fuck you <cstdlib> for putting "system" in
the global namespace. Why even have that header then?
- Kernel object types constructed with the construct_handle helper now
have a creation_caps static member to declare what capabilities a
newly created object's handle should have.
Since we have a DSL for specifying syscalls, we can create a verificaton
method for each syscall that can cover most argument (and eventually
capability) verification instead of doing it piecemeal in each syscall
implementation, which can be more error-prone.
Now a new _syscall_verify_* function exists for every syscall, which
calls the real implementation. The syscall table for the syscall handler
now maps to these verify functions.
Other changes:
- Updated the definition grammar to allow options to have a "key:value"
style, to eventually support capabilities.
- Added an "optional" option for parameters that says a syscall will
accept a null value.
- Some bonnibel fixes, as definition file changes weren't always
properly causing updates in the build dep graph.
- The syscall implementation function signatures are no longer exposed
in syscall.h. Also, the unused syscall enum has been removed.
First attempt at a UART driver. I'm not sure it's the most stable. Now
that userspace is handling displaying logs, also removed serial and log
output support from the kernel.
The j6threads command shows the current thread, ready threads, and
blocked threads for a given CPU.
To support this, TCB structs gained a pointer to their thread (instead
of trying to do offset magic) and threads gained a pointer to their
creator. Also removed thread::from_tcb() now that the TCB has a pointer.
These commands had a number of issues. They weren't evaluating their
arguments (eg, you couldn't use a symbol name instead of a number), and
they weren't explicitly using hex when evaluating numbers, so they were
getting incorrect values when the default radix was not 10.
While bonnibel already had the concept of a manifest, which controls
what goes into the built disk image, the bootloader still had filenames
hard-coded. Now bonnibel creates a 'jsix_boot.dat' file that tells the
bootloader what it should load.
Changes include:
- Modules have two new fields: location and description. location is
their intended directory on the EFI boot volume. description is
self-explanatory, and is used in log messages.
- New class, boot::bootconfig, implements reading of jsix_boot.dat
- New header, bootproto/bootconfig.h, specifies flags used in the
manifest and jsix_boot.dat
- New python module, bonnibel/manifest.py, encapsulates reading of the
manifest and writing jsix_boot.dat
- Syntax of the manifest changed slightly, including adding flags
- Boot and Kernel target ccflags unified a bit (this was partly due to
trying to get enum_bitfields to work in boot)
- util::counted gained operator+= and new free function util::read<T>
The j6tw (j6 table walk) command to debug page tables was throwing an
exception for an integer that was too big when the default radix was 16,
because it would interpret ints as hex even without the 0x prefix. Now
j6tw explicitly converts to hex and uses the prefix to be explicit.
Overall, I believe TOML to be a superior configuration format than YAML
in many situations, but it gets ugly quickly when nesting data
structures. The build configs were fine in TOML, but the manifest (and
my future plans for it) got unwieldy. I also did not want different
formats for each kind of configuration on top of also having a custom
DSL for interface definitions, so I've switched all the TOML to YAML.
Also of note is that this change actually adds structure to the manifest
file, which was little more than a CSV previously.
This change adds a new interface DSL for specifying objects (with
methods) and interfaces (that expose objects, and optionally have their
own methods).
Significant changes:
- Add the new scripts/definitions Python module to parse the DSL
- Add the new definitions directory containing DSL definition files
- Use cog to generate syscall-related code in kernel and libj6
- Unify ordering of pointer + length pairs in interfaces
This change moves Bonnibel from a separate project into the jsix tree,
and alters the project configuration to be jsix-specific. (I stopped
using bonnibel for any other projects, so it's far easier to make it a
custom generator for jsix.) The build system now also uses actual python
code in `*.module` files to configure modules instead of TOML files.
Target configs (boot, kernel-mode, user-mode) now moved to separate TOML
files under `configs/` and can inherit from one another.
When setting a radix other than 10, the j6stack command will start
adding the wrong values - make sure to specify a radix for the offset
explicitly.
Also show the frame pointer for each frame with the j6bt command, and
don't throw exceptions if the name of the block is unknown.
Added the command "j6tw <pml4> <addr>" which takes any arguments that
evaluate to addresses or integers. It displays the full breakdown of the
page table walk for the given address, with flags.
Improve the j6stack command in two ways: first, swap the order of the
arguments, as depth is much more likely to be changed. Second, on any
exception accessing memory in the stack, print the exception and
continue instead of failing the whole command.
Create a script that builds a simple-to-read symbol table from the
output of `nm`. Include running that script over the kernel in the
build, and including that output in the initrd.
Tags: callstack debugging
GDB works far better now with QEMU's `-S` flag. No longer does it
complain about changing the target from 32 to 64 bits. Get rid of the
old `waiting` loop and `sleep` call in the GDB config for the kernel.
Tags: debugging
There are a lot of under the hood changes here:
- Move syscalls to be a dispatch table, defined by syscalls.inc
- Don't need a full process state (push_all) in syscalls now
- In push_all, define REGS instead of using offsets
- Save TWO stack pointers as well as current saved stack pointer in TCB:
- rsp0 is the base of the kernel stack for interrupts
- rsp3 is the saved user stack from cpu_data
- Update syscall numbers in nulldrv
- Some asm-debugging enhancements to the gdb script
- fork() still not working
