The compile_commands.json database is used by clangd to understand the
compilation settings of the project. Keep this up to date by making
ninja update it as part of the build, and have it depend on all build
and module files.
If a bip_buffer's A buffer is in the middle of being appended to, but
that append has not yet been committed, and all committed A data has
been read, the buffer would get into a bad state where m_start_r pointed
to the end of the previous A buffer, but that data is no longer
connected to either A or B. So now we make sure to check m_size_r before
considering A "done".
Added a list of currently-set flags on the thread's state. Also stopped
tracebacks and returned instead of erroring out when they threw
gdb.MemoryError.
A spinlock was recenly added to thread wait states, so that they
couldn't get stuck if their wake event happened while setting the wake
state, and cause the thread to deadlock. But the scoped_lock objects
locking that spinlock were being instantiated as temporaries and
immediately being thrown away because I forgot to name them.
The scheduler was accidentally checking the state of the _currently
running_ thread when seeing if it should promote a thread in the ready
queue. So, ie, constant-priority threads would get promoted as long as
some non-constant-priority thread was the currently-running thread.
This commit contains a couple large, interdependent changes:
- In preparation for capability checking, the _syscall_verify_*
functions now load most handles passed in, and verify that they exist
and are of the correct type. Lists and out-handles are not converted
to objects.
- Also in preparation for capability checking, the internal
representation of handles has changed. j6_handle_t is now 32 bits, and
a new j6_cap_t (also 32 bits) is added. Handles of a process are now a
util::map<j6_handle_t, handle> where handle is a new struct containing
the id, capabilities, and object pointer.
- The kernel object definition DSL gained a few changes to support auto
generating the handle -> object conversion in the _syscall_verify_*
functions, mostly knowing the object type, and an optional "cname"
attribute on objects where their names differ from C++ code.
(Specifically vma/vm_area)
- Kernel object code and other code under kernel/objects is now in a new
obj:: namespace, because fuck you <cstdlib> for putting "system" in
the global namespace. Why even have that header then?
- Kernel object types constructed with the construct_handle helper now
have a creation_caps static member to declare what capabilities a
newly created object's handle should have.
Since we have a DSL for specifying syscalls, we can create a verificaton
method for each syscall that can cover most argument (and eventually
capability) verification instead of doing it piecemeal in each syscall
implementation, which can be more error-prone.
Now a new _syscall_verify_* function exists for every syscall, which
calls the real implementation. The syscall table for the syscall handler
now maps to these verify functions.
Other changes:
- Updated the definition grammar to allow options to have a "key:value"
style, to eventually support capabilities.
- Added an "optional" option for parameters that says a syscall will
accept a null value.
- Some bonnibel fixes, as definition file changes weren't always
properly causing updates in the build dep graph.
- The syscall implementation function signatures are no longer exposed
in syscall.h. Also, the unused syscall enum has been removed.
There has been a global clock object for a while now, but scheduler was
never using it, instead still using its simple increment clock. Now it
uses the hpet clock.
First attempt at a UART driver. I'm not sure it's the most stable. Now
that userspace is handling displaying logs, also removed serial and log
output support from the kernel.
The j6threads command shows the current thread, ready threads, and
blocked threads for a given CPU.
To support this, TCB structs gained a pointer to their thread (instead
of trying to do offset magic) and threads gained a pointer to their
creator. Also removed thread::from_tcb() now that the TCB has a pointer.
The logger had a lot of code that was due to it being in kutil instead
of the kernel. Simplifying it a bit here in order to make the uart
logger easier and eventual paring down of the logger easier.
- Log areas are no longer hashes of their names, just an enum
- Log level settings are no longer saved in 4 bits, just a byte
- System signal updating is done in the logger now
This syscall allows a process to give another process access to an
object it has a handle to. The value of the handle as seen in the
receiver process is returned to the caller, so that the caller may
notify the recipient which handle was given.
In places where the "user" state is available, like interrupt handlers,
panic() and kassert() can now take an optional pointer to that user
cpu_state structure, and the panic handler will print that out as well.
The last line of the boot output was always getting cut off by anything
else getting printed to the serial port. Adding two newlines to clear
the cursor of the previous output.
These commands had a number of issues. They weren't evaluating their
arguments (eg, you couldn't use a symbol name instead of a number), and
they weren't explicitly using hex when evaluating numbers, so they were
getting incorrect values when the default radix was not 10.
A structure, system_config, which is dynamically defined by the
definitions/sysconf.yaml config, is now mapped into every user address
space. The kernel fills this with information about itself and the
running machine.
User programs access this through the new j6_sysconf fake syscall in
libj6.
See: Github bug #242
See: [frobozz blog post](https://jsix.dev/posts/frobozz/)
Tags:
This means the kernel now depends on libj6. I've added the macro
definition __j6kernel when building for the kernel target, so I can
remove parts with #ifdefs.
The last commit was a bandaid, but this needed a real fix. Now we create
a .parse_deps.phony file in every module build dir that implicitly
depends on that module's dependencies' .parse_deps.phony files, as well
as order-only depends on any cog-parsed output for that module. This
causes the cog files to get generated first if they never have been, but
still leaves real header dependency tracking to clang's depfile
generation.
Kernel panics previously only stopped the calling core. This commit
re-implements the panic system to allow us to stop all cores on a panic.
Changes include:
- panic now sends an NMI to all cores. This means we can't control the
contents of their registers, so panic information has been moved to a
global struct, and the panicking cpu sets the pointer to that data in
its cpu_data.
- the panic_handler is now set up with mutexes to print appropriately
and only initialize objects once.
- copying _current_gsbase into the panic handler, and #including the
cpprt.cpp file (so that we can define NDEBUG and not have it try to
link the assert code back in)
- making the symbol data pointer in kargs an actual pointer again, not
an address - and carrying that through to the panic handler
- the number of cpus is now saved globally in the kernel as g_num_cpus
While bonnibel already had the concept of a manifest, which controls
what goes into the built disk image, the bootloader still had filenames
hard-coded. Now bonnibel creates a 'jsix_boot.dat' file that tells the
bootloader what it should load.
Changes include:
- Modules have two new fields: location and description. location is
their intended directory on the EFI boot volume. description is
self-explanatory, and is used in log messages.
- New class, boot::bootconfig, implements reading of jsix_boot.dat
- New header, bootproto/bootconfig.h, specifies flags used in the
manifest and jsix_boot.dat
- New python module, bonnibel/manifest.py, encapsulates reading of the
manifest and writing jsix_boot.dat
- Syntax of the manifest changed slightly, including adding flags
- Boot and Kernel target ccflags unified a bit (this was partly due to
trying to get enum_bitfields to work in boot)
- util::counted gained operator+= and new free function util::read<T>
I just can't get enum_bitfields to work in boot. The same code works in
other targets. None of the compiler options should change that. I gave
up, but I'm leaving these changes in because they do actually handle
const better.
This is a rather large commit that is widely focused on cleaning things
out of the 'junk drawer' that is src/include. Most notably, several
things that were put in there because they needed somewhere where both
the kernel, boot, and init could read them have been moved to a new lib,
'bootproto'.
- Moved kernel_args.h and init_args.h to bootproto as kernel.h and
init.h, respectively.
- Moved counted.h and pointer_manipulation.h into util, renaming the
latter to util/pointers.h.
- Created a new src/include/arch for very arch-dependent definitions,
and moved some kernel_memory.h constants like frame size, page table
entry count, etc to arch/amd64/memory.h. Also created arch/memory.h
which detects platform and includes the former.
- Got rid of kernel_memory.h entirely in favor of a new, cog-based
approach. The new definitions/memory_layout.csv lists memory regions
in descending order from the top of memory, their sizes, and whether
they are shared outside the kernel (ie, boot needs to know them). The
new header bootproto/memory.h exposes the addresses of the shared
regions, while the kernel's memory.h gains the start and size of all
the regions. Also renamed the badly-named page-offset area the linear
area.
- The python build scripts got a few new features: the ability to parse
the csv mentioned above in a new memory.py module; the ability to add
dependencies to existing source files (The list of files that I had to
pull out of the main list just to add them with the dependency on
memory.h was getting too large. So I put them back into the sources
list, and added the dependency post-hoc.); and the ability to
reference 'source_root', 'build_root', and 'module_root' variables in
.module files.
- Some utility functions that were in the kernel's memory.h got moved to
util/pointers.h and util/misc.h, and misc.h's byteswap was renamed
byteswap32 to be more specific.
Now that kutil has no kernel-specific code in it anymore, it can
actually be linked to by anything, so I'm renaming it 'util'.
Also, I've tried to unify the way that the system libraries from
src/libraries are #included using <> instead of "".
Other small change: util::bip_buffer got a spinlock to guard against
state corruption.
Continuing moving things out of kutil. The assert as implemented could
only ever work in the kernel, so remaining kutil uses of kassert have
been moved to including standard C assert instead.
Along the way, kassert was broken out into panic::panic and kassert,
and the panic.serial namespace was renamed panicking.
The moving of kernel-only code out of kutil continues. (See 042f061)
This commit moves the following:
- The heap allocator code
- memory.cpp/h which means:
- letting string.h be the right header for memset and memcpy, still
including an implementation of it for the kernel though, since
we're not linking libc to the kernel
- Changing calls to kalloc/kfree to new/delete in kutil containers
that aren't going to be merged into the kernel
- Fixing a problem with stdalign.h from libc, which was causing issues
for type_traits.
Part two of rearranging kutil code. (See 042f061) Removing unused kutil
headers:
I can imagine that avl_tree or slab_allocated may want to be returned to
at some point, but for now they're just clutter.
Part one of a series of code moves. The kutil library is not very
useful, as most of its code is kernel-specific. This was originally for
testing purposes, but that can be achieved in other ways with the
current build system. I find this mostly creates a strange division in
the kernel code.
Instead, I'm going to move everything kernel-specific to actually be in
the kernel, and replace kutil with just 'util' for generic utility code
I want to share.
This commit:
- Moves the logger into the kernel.
- Updates the 'printf' library used from mpaland/printf to
eyalroz/printf and moved it into the kernel, as it's only used by the
logger in kutil.
- Removes some other unused kutil headers from some files, to help
future code rearrangement.
Note that the (now redundant-seeming) log.cpp/h in kernel is currently
still there - these files are more about log output than the logging
system, and will get replaced once I add user-space log output.
Fixing two page allocation issues I came across while debugging:
- Added a spinlock to the page_table static page cache, to avoid
multiple CPUs grabbing the same page. This cache should probably
just be made into per-CPU caches.
- Fixed a bitwise math issue ("1" instead of "1ull" when working with
64-bit numbers) that made it so that pages were never marked as
allocated when allocating 32 or more.
The j6tw (j6 table walk) command to debug page tables was throwing an
exception for an integer that was too big when the default radix was 16,
because it would interpret ints as hex even without the 0x prefix. Now
j6tw explicitly converts to hex and uses the prefix to be explicit.
Updated kassert to be an actual function, and used the __builtin_*
functions for location data. Updated the panic handler protocol to
include sending location data as three more parameters. Updated the
serial panic handler to display that data along with the (optional)
message.
Add a simple ELF loader to srv.init to load and start any module_program
parameters passed from the bootloader. Also creates stacks for newly
created threads.
Also update thread creation in testapp to create stacks.
Stop creating stacks in user space for user threads, that should be done
by the thread's creator. This change adds process and stack_top
arguments to the thread_create syscall, so that threads can be created
in other processes, and given a stack address.
Also included is a fix in add_thunk_user due to the r11/flags change.
THIS COMMIT BREAKS USERSPACE. See subsequent commits for the user side
changes related to this change.
LLVM was updated by system update, and is now unhappy building because
of libc++ threads settings. I'm explicitly turning them off in base.yml
for now until I can focus on a better fix.
Bonnibel used to copy any asset into the root - now a path can be
specified for assets, and debug output goes into .debug so GDB will pick
it up automatically.
