The new mailbox kernel object API offers asynchronous message-based IPC
for sending data and handles between threads, as opposed to endpoint's
synchronous model.
In preparation for the new mailbox IPC model, blocking threads needed an
overhaul. The `wait_on_*` and `wake_on_*` methods are gone, and the
`block()` and `wake()` calls on threads now pass a value between the
waker and the blocked thread.
As part of this change, the concept of signals on the base kobject class
was removed, along with the queue of blocked threads waiting on any
given object. Signals are now exclusively the domain of the event object
type, and the new wait_queue utility class helps manage waiting threads
when an object does actually need this functionality. In some cases (eg,
logger) an event object is used instead of the lower-level wait_queue.
Since this change has a lot of ramifications, this large commit includes
the following additional changes:
- The j6_object_wait, j6_object_wait_many, and j6_thread_pause syscalls
have been removed.
- The j6_event_clear syscall has been removed - events are "cleared" by
reading them now. A new j6_event_wait syscall has been added to read
events.
- The generic close() method on kobject has been removed.
- The on_no_handles() method on kobject now deletes the object by
default, and needs to be overridden by classes that should not be.
- The j6_system_bind_irq syscall now takes an event handle, as well as a
signal that the IRQ should set on the event. IRQs will cause a waiting
thread to be woken with the appropriate bit set.
- Threads waking due to timeout is simplified to just having a
wake_timeout() accessor that returns a timestamp.
- The new wait_queue uses util::deque, which caused the disovery of two
bugs in the deque implementation: empty deques could still have a
single array allocated and thus return true for empty(), and new
arrays getting allocated were not being zeroed first.
- Exposed a new erase() method on util::map that takes a node pointer
instead of a key, skipping lookup.
The great header shift: It didn't make sense to regenerate headers for
the same module for every target (boot/kernel/user) it appeared in. And
now that core headers are out of src/include, this was going to cause
problems for the new libc changes I've been working on. So I went back
to re-design how module headers work.
Pre-requisites:
- A module's public headers should all be available in one location, not
tied to target.
- No accidental includes. Another module should not be able to include
anything (creating an implicit dependency) from a module without
declaring an explicit dependency.
- Exception to the previous: libc's headers should be available to all,
at least for the freestanding headers.
New system:
- A new "public_headers" property of module declares all public headers
that should be available to dependant modules
- All public headers (after possible processing) are installed relative
to build/include/<module> with the same path as their source
- This also means no "include" dir in modules is necessary. If a header
should be included as <j6/types.h> then its source should be
src/libraries/j6/j6/types.h - this caused the most churn as all public
header sources moved one directory up.
- The "includes" property of a module is local only to that module now,
it does not create any implicit public interface
Other changes:
- The bonnibel concept of sources changed: instead of sources having
actions, they themselves are an instance of a (sub)class of Source,
which provides all the necessary information itself.
- Along with the above, rule names were standardized into <type>.<ext>,
eg "compile.cpp" or "parse.cog"
- cog and cogflags variables moved from per-target scope to global scope
in the build files.
- libc gained a more dynamic .module file
The event object was missing any syscalls. Furthermore, kobject had an
old object_signal implementation (the syscall itself no longer exists),
which was removed. User code should only be able to set signals on
events.
This change finally adds capabilities to handles. Included changes:
- j6_handle_t is now again 64 bits, with the highest 8 bits being a type
code, and the next highest 24 bits being the capability mask, so that
programs can check type/caps without calling the kernel.
- The definitions grammar now includes a `capabilities [ ]` section on
objects, to list what capabilities are relevant.
- j6/caps.h is auto-generated from object capability lists
- init_libj6 again sets __handle_self and __handle_sys, this is a bit
of a hack.
- A new syscall, j6_handle_list, will return the list of existing
handles owned by the calling process.
- syscall_verify.cpp.cog now actually checks that the needed
capabilities exist on handles before allowing the call.
The kernel/main.cpp and kernel/memory_bootstrap.cpp files had become
something of a junk drawer. This change cleans them up in the following
ways:
- Most CPU initialization has moved to cpu.cpp, allowing several
functions to be made static and removed from cpu.h
- Multi-core startup code has moved to the new smp.h and smp.cpp, and
ap_startup.s has been renamed smp.s to match.
- run_constructors() has moved to memory_bootstrap.cpp, and all the
functionality of that file has been hidden behind a new public
interface mem::initialize().
- load_init_server() has moved from memory_bootstrap.cpp to main.cpp
Since we have a DSL for specifying syscalls, we can create a verificaton
method for each syscall that can cover most argument (and eventually
capability) verification instead of doing it piecemeal in each syscall
implementation, which can be more error-prone.
Now a new _syscall_verify_* function exists for every syscall, which
calls the real implementation. The syscall table for the syscall handler
now maps to these verify functions.
Other changes:
- Updated the definition grammar to allow options to have a "key:value"
style, to eventually support capabilities.
- Added an "optional" option for parameters that says a syscall will
accept a null value.
- Some bonnibel fixes, as definition file changes weren't always
properly causing updates in the build dep graph.
- The syscall implementation function signatures are no longer exposed
in syscall.h. Also, the unused syscall enum has been removed.
First attempt at a UART driver. I'm not sure it's the most stable. Now
that userspace is handling displaying logs, also removed serial and log
output support from the kernel.
A structure, system_config, which is dynamically defined by the
definitions/sysconf.yaml config, is now mapped into every user address
space. The kernel fills this with information about itself and the
running machine.
User programs access this through the new j6_sysconf fake syscall in
libj6.
See: Github bug #242
See: [frobozz blog post](https://jsix.dev/posts/frobozz/)
Tags:
This means the kernel now depends on libj6. I've added the macro
definition __j6kernel when building for the kernel target, so I can
remove parts with #ifdefs.
The last commit was a bandaid, but this needed a real fix. Now we create
a .parse_deps.phony file in every module build dir that implicitly
depends on that module's dependencies' .parse_deps.phony files, as well
as order-only depends on any cog-parsed output for that module. This
causes the cog files to get generated first if they never have been, but
still leaves real header dependency tracking to clang's depfile
generation.
While bonnibel already had the concept of a manifest, which controls
what goes into the built disk image, the bootloader still had filenames
hard-coded. Now bonnibel creates a 'jsix_boot.dat' file that tells the
bootloader what it should load.
Changes include:
- Modules have two new fields: location and description. location is
their intended directory on the EFI boot volume. description is
self-explanatory, and is used in log messages.
- New class, boot::bootconfig, implements reading of jsix_boot.dat
- New header, bootproto/bootconfig.h, specifies flags used in the
manifest and jsix_boot.dat
- New python module, bonnibel/manifest.py, encapsulates reading of the
manifest and writing jsix_boot.dat
- Syntax of the manifest changed slightly, including adding flags
- Boot and Kernel target ccflags unified a bit (this was partly due to
trying to get enum_bitfields to work in boot)
- util::counted gained operator+= and new free function util::read<T>
This is a rather large commit that is widely focused on cleaning things
out of the 'junk drawer' that is src/include. Most notably, several
things that were put in there because they needed somewhere where both
the kernel, boot, and init could read them have been moved to a new lib,
'bootproto'.
- Moved kernel_args.h and init_args.h to bootproto as kernel.h and
init.h, respectively.
- Moved counted.h and pointer_manipulation.h into util, renaming the
latter to util/pointers.h.
- Created a new src/include/arch for very arch-dependent definitions,
and moved some kernel_memory.h constants like frame size, page table
entry count, etc to arch/amd64/memory.h. Also created arch/memory.h
which detects platform and includes the former.
- Got rid of kernel_memory.h entirely in favor of a new, cog-based
approach. The new definitions/memory_layout.csv lists memory regions
in descending order from the top of memory, their sizes, and whether
they are shared outside the kernel (ie, boot needs to know them). The
new header bootproto/memory.h exposes the addresses of the shared
regions, while the kernel's memory.h gains the start and size of all
the regions. Also renamed the badly-named page-offset area the linear
area.
- The python build scripts got a few new features: the ability to parse
the csv mentioned above in a new memory.py module; the ability to add
dependencies to existing source files (The list of files that I had to
pull out of the main list just to add them with the dependency on
memory.h was getting too large. So I put them back into the sources
list, and added the dependency post-hoc.); and the ability to
reference 'source_root', 'build_root', and 'module_root' variables in
.module files.
- Some utility functions that were in the kernel's memory.h got moved to
util/pointers.h and util/misc.h, and misc.h's byteswap was renamed
byteswap32 to be more specific.
Now that kutil has no kernel-specific code in it anymore, it can
actually be linked to by anything, so I'm renaming it 'util'.
Also, I've tried to unify the way that the system libraries from
src/libraries are #included using <> instead of "".
Other small change: util::bip_buffer got a spinlock to guard against
state corruption.
Continuing moving things out of kutil. The assert as implemented could
only ever work in the kernel, so remaining kutil uses of kassert have
been moved to including standard C assert instead.
Along the way, kassert was broken out into panic::panic and kassert,
and the panic.serial namespace was renamed panicking.
The moving of kernel-only code out of kutil continues. (See 042f061)
This commit moves the following:
- The heap allocator code
- memory.cpp/h which means:
- letting string.h be the right header for memset and memcpy, still
including an implementation of it for the kernel though, since
we're not linking libc to the kernel
- Changing calls to kalloc/kfree to new/delete in kutil containers
that aren't going to be merged into the kernel
- Fixing a problem with stdalign.h from libc, which was causing issues
for type_traits.
Part one of a series of code moves. The kutil library is not very
useful, as most of its code is kernel-specific. This was originally for
testing purposes, but that can be achieved in other ways with the
current build system. I find this mostly creates a strange division in
the kernel code.
Instead, I'm going to move everything kernel-specific to actually be in
the kernel, and replace kutil with just 'util' for generic utility code
I want to share.
This commit:
- Moves the logger into the kernel.
- Updates the 'printf' library used from mpaland/printf to
eyalroz/printf and moved it into the kernel, as it's only used by the
logger in kutil.
- Removes some other unused kutil headers from some files, to help
future code rearrangement.
Note that the (now redundant-seeming) log.cpp/h in kernel is currently
still there - these files are more about log output than the logging
system, and will get replaced once I add user-space log output.
This change adds a new interface DSL for specifying objects (with
methods) and interfaces (that expose objects, and optionally have their
own methods).
Significant changes:
- Add the new scripts/definitions Python module to parse the DSL
- Add the new definitions directory containing DSL definition files
- Use cog to generate syscall-related code in kernel and libj6
- Unify ordering of pointer + length pairs in interfaces
This change moves Bonnibel from a separate project into the jsix tree,
and alters the project configuration to be jsix-specific. (I stopped
using bonnibel for any other projects, so it's far easier to make it a
custom generator for jsix.) The build system now also uses actual python
code in `*.module` files to configure modules instead of TOML files.
Target configs (boot, kernel-mode, user-mode) now moved to separate TOML
files under `configs/` and can inherit from one another.
