The UART driver would constantly hang in unpredictable spots. Turns out
it could get into a situation where it was stuck in a loop unable to
read more from the receive channel, and/or write to the serial port
buffer. Now we use a ring buffer to read as much as possible from the
receive channel, write as much as possible to the serial port buffer,
and move on without looping.
Added a release config, and fixed a few spots where optimizations broke things:
- Clang was generating incorrect code for run_ctor_list in libc's init.cpp (it
ignored a check for the end of the list)
- my rep movsb memcpy implementation used incorrect inline asm constraints, so
it was returning a pointer to the end of the copied range instead of the start.
Since this function was just inline asm anyway, I rewrote it in asm by hand in
a new memutils.s file.
The `driver_main` sinature was an alternate signature for `main`
implemented with weak symbols, but it causes linking issues when not
statically linked, and drivers are going to work differently soon
anyway. Just get rid of it for now.
This commit includes a number of changes to enable loading of PIE
executables:
- The loader in srv.init checks for a `PT_INTERP` segment in the program
its loading, and if it exists, loads the specified interpreter and
passes control to it instead of the program itself.
- Added ld.so the dynamic linker executable and set it as the
interpreter for all user-target programs.
- Program initial stack changed again to now contain a number of
possible tagged structures, including a new one for ld.so's arguments,
and for passing handles tagged with protocol ids.
- Added a stub for a new VFS protocol. Unused so far, but srv.init will
need to serve VFS requests from ld.so once I transition libraries to
shared libs for user-target programs. (Right now all executables are
PIE but statically linked, so they only need internal relocations.)
- Added 16 and 8 bit variants of `util::bitset`. This ended up not being
used, but could be useful.
Going back to letting mailboxes use variable-length data. Note that this
requires extra copies, so shared memory channels should be used for
anything in the hot path. But this allows better RPC over mailboxes and
other flexibility.
Other changes:
- added a j6::proto::sl::client class to act as a service locator
client, instead of duplicating that code in every program.
- moved protocol ids into j6/tables/protocols.inc so that C++ clients
can easily have their own API
This is the first of two rather big changes to clean up includes
throughout the project. In this commit, the implicit semi-dependency on
libc that bonnibel adds to every module is removed. Previously, I was
sloppy with includes of libc headers and include directory order. Now,
the freestanding headers from libc are split out into libc_free, and an
implicit real dependency is added onto this module, unless `no_libc` is
set to `True`. The full libc needs to be explicitly specified as a
dependency to be used.
Several things needed to change in order to do this:
- Many places use `memset` or `memcpy` that cannot depend on libc. The
kernel has basic implementations of them itself for this reason. Now
those functions are moved into the lower-level `j6/memutils.h`, and
libc merely references them. Other modules are now free to reference
those functions from libj6 instead.
- The kernel's `assert.h` was renamed kassert.h (matching its `kassert`
function) so that the new `util/assert.h` can use `__has_include` to
detect it and make sure the `assert` macro is usable in libutil code.
- Several implementation header files under `__libj6/` also moved under
the new libc_free.
- A new `include_phase` property has been added to modules for Bonnibel,
which can be "normal" (default) or "late" which uses `-idirafter`
instead of `-I` for includes.
- Since `<utility>` and `<new>` are not freestanding, implementations of
`remove_reference`, `forward`, `move`, and `swap` were added to the
`util` namespace to replace those from `std`, and `util/new.h` was
added to declare `operator new` and `operator delete`.
This commit does a number of things to start the transition of channels
from kernel to user space:
- Remove channel objects / syscalls from the kernel
- Add mutex type in libj6
- Add condition type in libj6
- Add a `ring` type flag for VMA syscalls to create ring buffers
- Implement a rudimentary shared memory channel using all of the above
Add the syscalls j6_futex_wait and j6_futex_wake. Currently marking this
as WIP as they need more testing.
Added to support futexes:
- vm_area and vm_space support for looking up physical address for a
virtual address
- libj6 mutex implementation using futex system calls
Clang will complain if main() is not declared with 0, 2, or 3 arguments.
In order to allow an extra 4th parameter, a new weak main() symbol which
just jumps to driver_main is defined, and _start passes the extra init
pointer to main.
Additionally, libc's crt0.s _start is made weak, and a matching
_libc_crt0_start symbol is defined for implementations that wish to
override _start but still call libc's _start. (Will be used by init.)
This new class makes it easier for user programs to spawn threads. This
change also includes support for .hh files in modules, to differentiate
headers that are C++-only in system libraries.
The init process now serves as a service locator for its children,
passing all children a mailbox handle on which it is serving the service
locator protocol.
In preparation for the new mailbox IPC model, blocking threads needed an
overhaul. The `wait_on_*` and `wake_on_*` methods are gone, and the
`block()` and `wake()` calls on threads now pass a value between the
waker and the blocked thread.
As part of this change, the concept of signals on the base kobject class
was removed, along with the queue of blocked threads waiting on any
given object. Signals are now exclusively the domain of the event object
type, and the new wait_queue utility class helps manage waiting threads
when an object does actually need this functionality. In some cases (eg,
logger) an event object is used instead of the lower-level wait_queue.
Since this change has a lot of ramifications, this large commit includes
the following additional changes:
- The j6_object_wait, j6_object_wait_many, and j6_thread_pause syscalls
have been removed.
- The j6_event_clear syscall has been removed - events are "cleared" by
reading them now. A new j6_event_wait syscall has been added to read
events.
- The generic close() method on kobject has been removed.
- The on_no_handles() method on kobject now deletes the object by
default, and needs to be overridden by classes that should not be.
- The j6_system_bind_irq syscall now takes an event handle, as well as a
signal that the IRQ should set on the event. IRQs will cause a waiting
thread to be woken with the appropriate bit set.
- Threads waking due to timeout is simplified to just having a
wake_timeout() accessor that returns a timestamp.
- The new wait_queue uses util::deque, which caused the disovery of two
bugs in the deque implementation: empty deques could still have a
single array allocated and thus return true for empty(), and new
arrays getting allocated were not being zeroed first.
- Exposed a new erase() method on util::map that takes a node pointer
instead of a key, skipping lookup.
The great header shift: It didn't make sense to regenerate headers for
the same module for every target (boot/kernel/user) it appeared in. And
now that core headers are out of src/include, this was going to cause
problems for the new libc changes I've been working on. So I went back
to re-design how module headers work.
Pre-requisites:
- A module's public headers should all be available in one location, not
tied to target.
- No accidental includes. Another module should not be able to include
anything (creating an implicit dependency) from a module without
declaring an explicit dependency.
- Exception to the previous: libc's headers should be available to all,
at least for the freestanding headers.
New system:
- A new "public_headers" property of module declares all public headers
that should be available to dependant modules
- All public headers (after possible processing) are installed relative
to build/include/<module> with the same path as their source
- This also means no "include" dir in modules is necessary. If a header
should be included as <j6/types.h> then its source should be
src/libraries/j6/j6/types.h - this caused the most churn as all public
header sources moved one directory up.
- The "includes" property of a module is local only to that module now,
it does not create any implicit public interface
Other changes:
- The bonnibel concept of sources changed: instead of sources having
actions, they themselves are an instance of a (sub)class of Source,
which provides all the necessary information itself.
- Along with the above, rule names were standardized into <type>.<ext>,
eg "compile.cpp" or "parse.cog"
- cog and cogflags variables moved from per-target scope to global scope
in the build files.
- libc gained a more dynamic .module file
This change finally adds capabilities to handles. Included changes:
- j6_handle_t is now again 64 bits, with the highest 8 bits being a type
code, and the next highest 24 bits being the capability mask, so that
programs can check type/caps without calling the kernel.
- The definitions grammar now includes a `capabilities [ ]` section on
objects, to list what capabilities are relevant.
- j6/caps.h is auto-generated from object capability lists
- init_libj6 again sets __handle_self and __handle_sys, this is a bit
of a hack.
- A new syscall, j6_handle_list, will return the list of existing
handles owned by the calling process.
- syscall_verify.cpp.cog now actually checks that the needed
capabilities exist on handles before allowing the call.
A structure, system_config, which is dynamically defined by the
definitions/sysconf.yaml config, is now mapped into every user address
space. The kernel fills this with information about itself and the
running machine.
User programs access this through the new j6_sysconf fake syscall in
libj6.
See: Github bug #242
See: [frobozz blog post](https://jsix.dev/posts/frobozz/)
Tags:
This means the kernel now depends on libj6. I've added the macro
definition __j6kernel when building for the kernel target, so I can
remove parts with #ifdefs.
This change adds a new interface DSL for specifying objects (with
methods) and interfaces (that expose objects, and optionally have their
own methods).
Significant changes:
- Add the new scripts/definitions Python module to parse the DSL
- Add the new definitions directory containing DSL definition files
- Use cog to generate syscall-related code in kernel and libj6
- Unify ordering of pointer + length pairs in interfaces
This change moves Bonnibel from a separate project into the jsix tree,
and alters the project configuration to be jsix-specific. (I stopped
using bonnibel for any other projects, so it's far easier to make it a
custom generator for jsix.) The build system now also uses actual python
code in `*.module` files to configure modules instead of TOML files.
Target configs (boot, kernel-mode, user-mode) now moved to separate TOML
files under `configs/` and can inherit from one another.
