The great header shift: It didn't make sense to regenerate headers for
the same module for every target (boot/kernel/user) it appeared in. And
now that core headers are out of src/include, this was going to cause
problems for the new libc changes I've been working on. So I went back
to re-design how module headers work.
Pre-requisites:
- A module's public headers should all be available in one location, not
tied to target.
- No accidental includes. Another module should not be able to include
anything (creating an implicit dependency) from a module without
declaring an explicit dependency.
- Exception to the previous: libc's headers should be available to all,
at least for the freestanding headers.
New system:
- A new "public_headers" property of module declares all public headers
that should be available to dependant modules
- All public headers (after possible processing) are installed relative
to build/include/<module> with the same path as their source
- This also means no "include" dir in modules is necessary. If a header
should be included as <j6/types.h> then its source should be
src/libraries/j6/j6/types.h - this caused the most churn as all public
header sources moved one directory up.
- The "includes" property of a module is local only to that module now,
it does not create any implicit public interface
Other changes:
- The bonnibel concept of sources changed: instead of sources having
actions, they themselves are an instance of a (sub)class of Source,
which provides all the necessary information itself.
- Along with the above, rule names were standardized into <type>.<ext>,
eg "compile.cpp" or "parse.cog"
- cog and cogflags variables moved from per-target scope to global scope
in the build files.
- libc gained a more dynamic .module file
The manifest can now supply a list of boot flags, including "test".
Those get turned into the bootproto::args::flags field by the
bootloader. The kernel takes those and uses the test flag to control
enabling syscalls with the new "test" attribute, like the new
test_finish syscall, which lets automated tests call back to the kernel
to shut down the system.
If the thread waiting is the current thread, it should have the result
when it wakes. Might as well return it, so that syscalls that know
they're putting the current thread to sleep can get the result easily.
The "handle" tag on syscall parameters causes syscall_verify.cpp to pass
the resulting object as a obj::handle* instead of directly as an object
pointer. Now the handle tag is supported directly on the syscall itself
as well, causing the "self" object to be passed as a handle pointer.
The event object was missing any syscalls. Furthermore, kobject had an
old object_signal implementation (the syscall itself no longer exists),
which was removed. User code should only be able to set signals on
events.
Two minor issues: scheduler::prune wasn't formatted correctly, and
j6/caps.h was not using the ull prefix when shifting 64 bit numbers.
(It's doubtful an object would get more than 32 caps any time soon, but
better to be correct.)
The cpu.cpp/smp.cpp cleanup out of kernel_main missed an important call:
kernel_main never called smp::ready() to unblock the APs waiting for the
scheduler to be ready.
The return of slab_allocated! Now after the kutil/util/kernel giant
cleanup, this belongs squarely in the kernel, and works much better
there. Slabs are allocated via a bump pointer into a new kernel VMA,
instead of using kalloc() or allocating pages directly.
Adding the util::deque container, implemented with the util::linked_list
of arrays of items.
Also, use the deque for a kobject's blocked thread list to maintain
order instead of a vector using remove_swap().
The new zero_ok flag is similar to optional for reference parameters,
but only in cases where there is a length parameter. If that parameter
is a reference parameter itself and is null, or it is non-null and
contains a non-zero length, or there is no length parameter, then the
main parameter may not be null.
The syscall interface is designed to closely follow the System V amd64
calling convention, so that as much as possible, the call into the
assembly trampoline for the syscall sets up the call correctly. Before
this change, the only exception was using r10 (a caller-saved register
already) to stash the contents of rcx, which gets clobbered by the
syscall instruction. However, this only preserves registers for the
function call, as the stack is switched upon kernel entry, and
additional call frames have been added by the time the syscall gets back
into C++ land.
This change adds a new parameter to the syscall in rbx. Since rbx is
callee-saved, the syscall trampoline pushes it to the stack, and then
puts the address of the stack-passed arguments into rbx. Now that the
syscall implementations are wrapped in the _syscall_verify_* functions,
we can piggy-back on those to also set up the extra arguments from the
user stack.
Now, for any syscall with 7 or more arguments, the verify wrapper takes
the first six arguments normally, then gets a stack pointer (the rbx
value) as its 7th and final argument. It's then the job of the verify
wrapper to get the remaining arguments from that stack pointer and pass
them to the implementation function as normal arguments.
The main point of this change is to allow "global" capabilities defined
on the base object type. The example here is the clone capability on all
objects, which governs the ability to clone a handle.
Related changes in this commit:
- Renamed `kobject` to `object` as far as the syscall interface is
concerned. `kobject` is the cname, but j6_cap_kobject_clone feels
clunky.
- The above change made me realize that the "object <type>" syntax for
specifying object references was also clunky, so now it's "ref <type>"
- Having to add `.object` on everywhere to access objects in
interface.exposes or object.super was cumbersome, so those properties
now return object types directly, instead of ObjectRef.
- syscall_verify.cpp.cog now generates code to check capabilities on
handles if they're specified in the definition, even when not passing
an object to the implementation function.
Added the handle_clone syscall which allows for cloning a handle with
a subset of the original handle's capabilities.
Related changes:
- srv.init now calls handle_clone on its system handle, and load_program
was changed to allow this second system handle to be passed to loaded
programs instead. However, as drv.uart is still a driver AND a log
reader, this new handle is not actually passed yet.
- The definition parser was using a set for the cap list, which meant
the order (and thus values) of caps was not static.
- Some code in objects/handle.h was made more explicit about what bits
meant what.
This change finally adds capabilities to handles. Included changes:
- j6_handle_t is now again 64 bits, with the highest 8 bits being a type
code, and the next highest 24 bits being the capability mask, so that
programs can check type/caps without calling the kernel.
- The definitions grammar now includes a `capabilities [ ]` section on
objects, to list what capabilities are relevant.
- j6/caps.h is auto-generated from object capability lists
- init_libj6 again sets __handle_self and __handle_sys, this is a bit
of a hack.
- A new syscall, j6_handle_list, will return the list of existing
handles owned by the calling process.
- syscall_verify.cpp.cog now actually checks that the needed
capabilities exist on handles before allowing the call.
Channels were unused, and while they were listed in syscalls.def, they
had no syscalls listed in their interface. This change adds them back,
and updates them to the curren syscall style.
The kernel/main.cpp and kernel/memory_bootstrap.cpp files had become
something of a junk drawer. This change cleans them up in the following
ways:
- Most CPU initialization has moved to cpu.cpp, allowing several
functions to be made static and removed from cpu.h
- Multi-core startup code has moved to the new smp.h and smp.cpp, and
ap_startup.s has been renamed smp.s to match.
- run_constructors() has moved to memory_bootstrap.cpp, and all the
functionality of that file has been hidden behind a new public
interface mem::initialize().
- load_init_server() has moved from memory_bootstrap.cpp to main.cpp
Endpoints could previously crash if two senders were concurrently
writing to them, so this change adds a spinlock and protects functions
that touch the signals and blocked list.
Added methods for releasing the lock held in a scoped_lock early, as
well as reacquiring it after. Useful when, eg a thread is about to block
and should not be holding the lock while blocked.
Changes to qemu.sh:
- Now takes the -l/--log option to create the qemu log in jsix.log,
otherwise does not ask qemu to log. (Way faster operations.)
- Remove the debug-isa-exit device when starting with the --debug flag,
so that we can inspect panics
- Changed from 512M to 4G of ram
The compile_commands.json database is used by clangd to understand the
compilation settings of the project. Keep this up to date by making
ninja update it as part of the build, and have it depend on all build
and module files.
If a bip_buffer's A buffer is in the middle of being appended to, but
that append has not yet been committed, and all committed A data has
been read, the buffer would get into a bad state where m_start_r pointed
to the end of the previous A buffer, but that data is no longer
connected to either A or B. So now we make sure to check m_size_r before
considering A "done".
Added a list of currently-set flags on the thread's state. Also stopped
tracebacks and returned instead of erroring out when they threw
gdb.MemoryError.
A spinlock was recenly added to thread wait states, so that they
couldn't get stuck if their wake event happened while setting the wake
state, and cause the thread to deadlock. But the scoped_lock objects
locking that spinlock were being instantiated as temporaries and
immediately being thrown away because I forgot to name them.
The scheduler was accidentally checking the state of the _currently
running_ thread when seeing if it should promote a thread in the ready
queue. So, ie, constant-priority threads would get promoted as long as
some non-constant-priority thread was the currently-running thread.
This commit contains a couple large, interdependent changes:
- In preparation for capability checking, the _syscall_verify_*
functions now load most handles passed in, and verify that they exist
and are of the correct type. Lists and out-handles are not converted
to objects.
- Also in preparation for capability checking, the internal
representation of handles has changed. j6_handle_t is now 32 bits, and
a new j6_cap_t (also 32 bits) is added. Handles of a process are now a
util::map<j6_handle_t, handle> where handle is a new struct containing
the id, capabilities, and object pointer.
- The kernel object definition DSL gained a few changes to support auto
generating the handle -> object conversion in the _syscall_verify_*
functions, mostly knowing the object type, and an optional "cname"
attribute on objects where their names differ from C++ code.
(Specifically vma/vm_area)
- Kernel object code and other code under kernel/objects is now in a new
obj:: namespace, because fuck you <cstdlib> for putting "system" in
the global namespace. Why even have that header then?
- Kernel object types constructed with the construct_handle helper now
have a creation_caps static member to declare what capabilities a
newly created object's handle should have.
Since we have a DSL for specifying syscalls, we can create a verificaton
method for each syscall that can cover most argument (and eventually
capability) verification instead of doing it piecemeal in each syscall
implementation, which can be more error-prone.
Now a new _syscall_verify_* function exists for every syscall, which
calls the real implementation. The syscall table for the syscall handler
now maps to these verify functions.
Other changes:
- Updated the definition grammar to allow options to have a "key:value"
style, to eventually support capabilities.
- Added an "optional" option for parameters that says a syscall will
accept a null value.
- Some bonnibel fixes, as definition file changes weren't always
properly causing updates in the build dep graph.
- The syscall implementation function signatures are no longer exposed
in syscall.h. Also, the unused syscall enum has been removed.
There has been a global clock object for a while now, but scheduler was
never using it, instead still using its simple increment clock. Now it
uses the hpet clock.
First attempt at a UART driver. I'm not sure it's the most stable. Now
that userspace is handling displaying logs, also removed serial and log
output support from the kernel.
The j6threads command shows the current thread, ready threads, and
blocked threads for a given CPU.
To support this, TCB structs gained a pointer to their thread (instead
of trying to do offset magic) and threads gained a pointer to their
creator. Also removed thread::from_tcb() now that the TCB has a pointer.
The logger had a lot of code that was due to it being in kutil instead
of the kernel. Simplifying it a bit here in order to make the uart
logger easier and eventual paring down of the logger easier.
- Log areas are no longer hashes of their names, just an enum
- Log level settings are no longer saved in 4 bits, just a byte
- System signal updating is done in the logger now
This syscall allows a process to give another process access to an
object it has a handle to. The value of the handle as seen in the
receiver process is returned to the caller, so that the caller may
notify the recipient which handle was given.
In places where the "user" state is available, like interrupt handlers,
panic() and kassert() can now take an optional pointer to that user
cpu_state structure, and the panic handler will print that out as well.
The last line of the boot output was always getting cut off by anything
else getting printed to the serial port. Adding two newlines to clear
the cursor of the previous output.
These commands had a number of issues. They weren't evaluating their
arguments (eg, you couldn't use a symbol name instead of a number), and
they weren't explicitly using hex when evaluating numbers, so they were
getting incorrect values when the default radix was not 10.
A structure, system_config, which is dynamically defined by the
definitions/sysconf.yaml config, is now mapped into every user address
space. The kernel fills this with information about itself and the
running machine.
User programs access this through the new j6_sysconf fake syscall in
libj6.
See: Github bug #242
See: [frobozz blog post](https://jsix.dev/posts/frobozz/)
Tags:
This means the kernel now depends on libj6. I've added the macro
definition __j6kernel when building for the kernel target, so I can
remove parts with #ifdefs.
