Added a j6_proto_vfs_tag/_get_tag pair of messages to the VFS protocol
to allow filesystems to label themselves, and gave 6s the concept of
current fs and current working directory.
This commit adds the 6s shell, and a bunch of supporting work for it.
Major changes include:
- New shell.yaml manifest to give 6s control of the TTY instead of
srv.logger
- Changes to mailbox syscalls to add max handles array size separate
from input size. Also reversed the meaning of the similar data size
argument in those syscalls. (Using the second arg as the max array
size and the first as the current valid size allows for the auto
verify code to verify handles properly, and simplifies user-side
code.)
- New util::unique_ptr smart pointer class similar to std::unique_ptr
- New ipc::message format that uses util::unique_ptr to manage ownership
and lifetimes and avoid extra copying.
- The service locator protocol now supports multiple handles per entry
- Channels got a major overhaul. They are now split into two VMAs, each
containing a mutex, a condition, and a util::bip_buffer. The order of
the VMAs determines which end of the pipe you're on. (ie, the creator
swaps them before handing them to the other thread.) Their API also
changed to be similar to that of util::bip_buffer, to avoid extra
copies.
- util::bip_buffer now keeps its state and its buffer together, so that
there are no pointers. This allows multiple processes to share them in
shared memory, like in channels.
- The UART driver changed from keeping buffers for the serial ports to
just keeping a channel, and the serial port objects read/write
directly from/to the channel.
Known issues:
- The shell doesn't actually do anything yet. It echos its input back to
the serial line and injects a prompt on new lines.
- The shell is one character behind in printing back to the serial line.
The UART driver would constantly hang in unpredictable spots. Turns out
it could get into a situation where it was stuck in a loop unable to
read more from the receive channel, and/or write to the serial port
buffer. Now we use a ring buffer to read as much as possible from the
receive channel, write as much as possible to the serial port buffer,
and move on without looping.
The enum_bitfields system never worked quite right, and always had edge cases where name
resolution for the SFINAE would fail. Move everything over to use util::bitset, which can
be constexpr and boils down to inline integer bitops in release mode.
Improved util::bitset itself, moving the array-backed base implementation into a new
util::sized_bitset, and making the single-inttype backed implementation the base case.
Also added a distinction between | or |= (which work with real bit values) and + or +=
(which work with bit indexes).
Added a release config, and fixed a few spots where optimizations broke things:
- Clang was generating incorrect code for run_ctor_list in libc's init.cpp (it
ignored a check for the end of the list)
- my rep movsb memcpy implementation used incorrect inline asm constraints, so
it was returning a pointer to the end of the copied range instead of the start.
Since this function was just inline asm anyway, I rewrote it in asm by hand in
a new memutils.s file.
User code can now set the log area and severity of log messages. This also updates the j6_log
syscall to take these parameters, but removes all calls to it except through j6::syslog().
Added a new IPC log category, and logging of mailbox calls in it. Also added
some logs in init's service locator to help track down the bug fixed in the
previous commit.
Now the init args are a linked list - this also means ld.so can use its
own plus those of the program (eg, SLP and VFS handles). __init_libj6
now adds the head of the list to its global init_args structure, and the
j6_find_init_handle function can be used to find a handle in those args
for a given proto.
This fixes situations like the logger using the wrong mailbox for the
service locator and never finding the uart driver.
Now when loading a process, init will push all the loader args, align
the stack so that the next pointer will have correct alignment, then
push a pointer to the loader args.
Also:
* _ldso_start will pop all loader args off of the stack before jumping
to the loaded program's entrypoint, as if it had never run.
* The sentinel arg structures have a size instead of being all zeros, so
that they can be popped off properly when done.
Update init to use the new xoroshiro RNG to create a random load address for dynamic executables.
Also pass an address past the end of the loader in the loader_arg to use when loading dynamic
dependencies.
ld.so will now go through all DT_NEEDED entries in the dynamic table and load and relocate
those shared libraries as well. Lazy linking of functions via the PLT is not yet supported,
all PLT entries are looked up ahead of time by ld.so.
Stacks were not 16-byte aligned when coming into j6::thread::init_proc,
make sure it aligns stacks for SSE by adding the `force_align_arg_pointer`
attribute.
This change allows the `vma_map` and `vma_create_map` syscalls to map to
addresses other than the one specified, and therefore makes the address
parameter to those syscalls `inout` in order to return the mapped
address.
Also add the `exact` flag for specifying that mapping needs to be done
at the exact address given. If the mapping collides with another, the
new `j6_err_collision` error is returned.
Added an `API` macro in `j6/api.h` that expands to mark the given
declaration as a default-visible symbol. Also change `format` and
`vformat` to non-template functions, and make calls to `main`, `exit`,
and the library init functions in `_start` GOT-relative.
The `driver_main` sinature was an alternate signature for `main`
implemented with weak symbols, but it causes linking issues when not
statically linked, and drivers are going to work differently soon
anyway. Just get rid of it for now.
This commit includes a number of changes to enable loading of PIE
executables:
- The loader in srv.init checks for a `PT_INTERP` segment in the program
its loading, and if it exists, loads the specified interpreter and
passes control to it instead of the program itself.
- Added ld.so the dynamic linker executable and set it as the
interpreter for all user-target programs.
- Program initial stack changed again to now contain a number of
possible tagged structures, including a new one for ld.so's arguments,
and for passing handles tagged with protocol ids.
- Added a stub for a new VFS protocol. Unused so far, but srv.init will
need to serve VFS requests from ld.so once I transition libraries to
shared libs for user-target programs. (Right now all executables are
PIE but statically linked, so they only need internal relocations.)
- Added 16 and 8 bit variants of `util::bitset`. This ended up not being
used, but could be useful.
Going back to letting mailboxes use variable-length data. Note that this
requires extra copies, so shared memory channels should be used for
anything in the hot path. But this allows better RPC over mailboxes and
other flexibility.
Other changes:
- added a j6::proto::sl::client class to act as a service locator
client, instead of duplicating that code in every program.
- moved protocol ids into j6/tables/protocols.inc so that C++ clients
can easily have their own API
This change allows for parameters in definition files to have the "list"
option while also needing the handle verification. The verify function
will now iterate through the list checking capabilities and types on
every valid handle in the list.
Add extra info to the NASM `global` directive to specify the sizes of
these symbols, mostly so they look right in `nm` or `readelf` and don't
trick me into thinking something is wrong.
My `REP.MOVSB` `memcpy` implementation had marked its C++ variable
constraints as output instead of input, causing the compiler to emit
code to copy the values of `$rsi` and `$rdi` back into the `src` and
`dst` pointers, so after the copy `dst` pointed to the memory just
beyond what had been copied.
Very few places actually used the return value from `memcpy`, so this
went unnoticed for a bit..
This is the second of two big changes to clean up includes throughout
the project. Since I've started using clangd with Neovim and using
VSCode's intellisense, my former strategy of copying all header files
into place in `build/include` means that the real files don't show up in
`compile_commands.json` and so display many include errors when viewing
those header files in those tools.
That setup was mostly predicated on a desire to keep directory depths
small, but really I don't think paths like `src/libraries/j6/j6` are
much better than `src/libraries/j6/include/j6`, and the latter doesn't
have the aforementioned issues, and is clearer to the casual observer as
well.
Some additional changes:
- Added a new module flag `copy_headers` for behavior similar to the old
style, but placing headers in `$module_dir/include` instead of the
global `build/include`. This was needed for external projects that
don't follow the same source/headers folder structure - in this case,
`zstd`.
- There is no longer an associated `headers.*.ninja` for each
`module.*.ninja` file, as only parsed headers need to be listed; this
functionality has been moved back into the module's ninja file.
This is the first of two rather big changes to clean up includes
throughout the project. In this commit, the implicit semi-dependency on
libc that bonnibel adds to every module is removed. Previously, I was
sloppy with includes of libc headers and include directory order. Now,
the freestanding headers from libc are split out into libc_free, and an
implicit real dependency is added onto this module, unless `no_libc` is
set to `True`. The full libc needs to be explicitly specified as a
dependency to be used.
Several things needed to change in order to do this:
- Many places use `memset` or `memcpy` that cannot depend on libc. The
kernel has basic implementations of them itself for this reason. Now
those functions are moved into the lower-level `j6/memutils.h`, and
libc merely references them. Other modules are now free to reference
those functions from libj6 instead.
- The kernel's `assert.h` was renamed kassert.h (matching its `kassert`
function) so that the new `util/assert.h` can use `__has_include` to
detect it and make sure the `assert` macro is usable in libutil code.
- Several implementation header files under `__libj6/` also moved under
the new libc_free.
- A new `include_phase` property has been added to modules for Bonnibel,
which can be "normal" (default) or "late" which uses `-idirafter`
instead of `-I` for includes.
- Since `<utility>` and `<new>` are not freestanding, implementations of
`remove_reference`, `forward`, `move`, and `swap` were added to the
`util` namespace to replace those from `std`, and `util/new.h` was
added to declare `operator new` and `operator delete`.
When keeping track of addresses to give to the kernel for channel shared
memory, double the channel's size since the kernel will double-map the
area as a ring buffer.
This commit does a number of things to start the transition of channels
from kernel to user space:
- Remove channel objects / syscalls from the kernel
- Add mutex type in libj6
- Add condition type in libj6
- Add a `ring` type flag for VMA syscalls to create ring buffers
- Implement a rudimentary shared memory channel using all of the above
Add the syscalls j6_futex_wait and j6_futex_wake. Currently marking this
as WIP as they need more testing.
Added to support futexes:
- vm_area and vm_space support for looking up physical address for a
virtual address
- libj6 mutex implementation using futex system calls
Clang will complain if main() is not declared with 0, 2, or 3 arguments.
In order to allow an extra 4th parameter, a new weak main() symbol which
just jumps to driver_main is defined, and _start passes the extra init
pointer to main.
Additionally, libc's crt0.s _start is made weak, and a matching
_libc_crt0_start symbol is defined for implementations that wish to
override _start but still call libc's _start. (Will be used by init.)
For the coming switch to cap/handle ref-counting being the main lifetime
determiner of objects, get rid of self handles for threads and processes
to avoid circular references. Instead, passing 0 to syscalls expecting a
thread or process handle signifies "this process/thread".
This commit changes the add_user_thunk to point to a new routine,
initialize_user_cpu, which sets all the registers that were previously
unset when starting a new user thread. The values for rdi and rsi are
popped off the initial stack values that add_user_thunk sets up, so that
user thread procs can take up to two arguments.
To suppor this, j6_thread_create gained two new arguments, which are
passed on to the thread.
This also let me finally get rid of the hack of passing an argument in
rsp when starting init.
This commit re-adds testapp to the default manifest and does some
housecleaning on the module:
- Remove the old serial.* and io.*
- Update it to use current syscall APIs
- Update it to use libj6's higher-level thread API
This new class makes it easier for user programs to spawn threads. This
change also includes support for .hh files in modules, to differentiate
headers that are C++-only in system libraries.
Instead of handles / capabilities having numeric ids that are only valid
for the owning process, they are now global in a system capabilities
table. This will allow for specifying capabilities in IPC that doesn't
need to be kernel-controlled.
Processes will still need to be granted access to given capabilities,
but that can become a simpler system call than the current method of
sending them through mailbox messages (and worse, having to translate
every one into a new capability like was the case before). In order to
track which handles a process has access to, a new node_set based on
node_map allows for an efficient storage and lookup of handles.
The kernel log levels are now numerically reversed so that more-verbose
levels can be added to the end. Replaced 'debug' with 'verbose', and
added new 'spam' level.
The init process now serves as a service locator for its children,
passing all children a mailbox handle on which it is serving the service
locator protocol.
This commit adds a new flag, j6_channel_block, and a new flags param to
the channel_receive syscall. When the block flag is specified, the
caller will block waiting for data on the channel if the channel is
empty.
The new mailbox kernel object API offers asynchronous message-based IPC
for sending data and handles between threads, as opposed to endpoint's
synchronous model.
In preparation for the new mailbox IPC model, blocking threads needed an
overhaul. The `wait_on_*` and `wake_on_*` methods are gone, and the
`block()` and `wake()` calls on threads now pass a value between the
waker and the blocked thread.
As part of this change, the concept of signals on the base kobject class
was removed, along with the queue of blocked threads waiting on any
given object. Signals are now exclusively the domain of the event object
type, and the new wait_queue utility class helps manage waiting threads
when an object does actually need this functionality. In some cases (eg,
logger) an event object is used instead of the lower-level wait_queue.
Since this change has a lot of ramifications, this large commit includes
the following additional changes:
- The j6_object_wait, j6_object_wait_many, and j6_thread_pause syscalls
have been removed.
- The j6_event_clear syscall has been removed - events are "cleared" by
reading them now. A new j6_event_wait syscall has been added to read
events.
- The generic close() method on kobject has been removed.
- The on_no_handles() method on kobject now deletes the object by
default, and needs to be overridden by classes that should not be.
- The j6_system_bind_irq syscall now takes an event handle, as well as a
signal that the IRQ should set on the event. IRQs will cause a waiting
thread to be woken with the appropriate bit set.
- Threads waking due to timeout is simplified to just having a
wake_timeout() accessor that returns a timestamp.
- The new wait_queue uses util::deque, which caused the disovery of two
bugs in the deque implementation: empty deques could still have a
single array allocated and thus return true for empty(), and new
arrays getting allocated were not being zeroed first.
- Exposed a new erase() method on util::map that takes a node pointer
instead of a key, skipping lookup.
The new "noreturn" option tag on syscall methods causes those methods to
be generated with [[noreturn]] / _Noreturn to avoid clang complaining
that other functions marked noreturn, like exit(), because it can't tell
that the syscall never returns.
This new libc is mostly from scratch, with *printf() functions provided
by Marco Paland and Eyal Rozenberg's tiny printf library, and malloc and
friends provided by dlmalloc.
