In order to pass along arguments like the framebuffer, it's far simpler
to have that data stored along with the modules than mapping new pages
for every structure. Also now optionally pass a module's data to a
driver as init starts it.
Restructuring paging into an object that carries its page cache with it
and makes for simpler code. Program loading is also changed to not copy
the pages loaded from the file into new pages - we can impose a new
constraint that anything loaded by boot have a simple, page-aligned
layout so that we can just map the existing pages into the right
addresses. Also included are some linker script changes to help
accommodate this.
Using `-fvisibility=hidden` when building the kernel, and then
`--discard-all` when stripping it, we shave almost 100KiB off of the
resulting ELF file.
Also dropped some unused symbols from the linker script, and rearranged
the sections so that the file is able to be mapped directly into memory
instead of having each section copied.
The initrd image is now created by the build system, loaded by the
bootloader, and passed to srv.init, which loads it (but doesn't do
anything with it yet, so this is actually a functional regression).
This simplifies a lot of the modules code between boot and init as well:
Gone are the many subclasses of module and all the data being inline
with the module structs, except for any loaded files. Now the only
modules loaded and passed will be the initrd, and any devices only the
bootloader has knowledge of, like the UEFI framebuffer.
While bonnibel already had the concept of a manifest, which controls
what goes into the built disk image, the bootloader still had filenames
hard-coded. Now bonnibel creates a 'jsix_boot.dat' file that tells the
bootloader what it should load.
Changes include:
- Modules have two new fields: location and description. location is
their intended directory on the EFI boot volume. description is
self-explanatory, and is used in log messages.
- New class, boot::bootconfig, implements reading of jsix_boot.dat
- New header, bootproto/bootconfig.h, specifies flags used in the
manifest and jsix_boot.dat
- New python module, bonnibel/manifest.py, encapsulates reading of the
manifest and writing jsix_boot.dat
- Syntax of the manifest changed slightly, including adding flags
- Boot and Kernel target ccflags unified a bit (this was partly due to
trying to get enum_bitfields to work in boot)
- util::counted gained operator+= and new free function util::read<T>
This is a rather large commit that is widely focused on cleaning things
out of the 'junk drawer' that is src/include. Most notably, several
things that were put in there because they needed somewhere where both
the kernel, boot, and init could read them have been moved to a new lib,
'bootproto'.
- Moved kernel_args.h and init_args.h to bootproto as kernel.h and
init.h, respectively.
- Moved counted.h and pointer_manipulation.h into util, renaming the
latter to util/pointers.h.
- Created a new src/include/arch for very arch-dependent definitions,
and moved some kernel_memory.h constants like frame size, page table
entry count, etc to arch/amd64/memory.h. Also created arch/memory.h
which detects platform and includes the former.
- Got rid of kernel_memory.h entirely in favor of a new, cog-based
approach. The new definitions/memory_layout.csv lists memory regions
in descending order from the top of memory, their sizes, and whether
they are shared outside the kernel (ie, boot needs to know them). The
new header bootproto/memory.h exposes the addresses of the shared
regions, while the kernel's memory.h gains the start and size of all
the regions. Also renamed the badly-named page-offset area the linear
area.
- The python build scripts got a few new features: the ability to parse
the csv mentioned above in a new memory.py module; the ability to add
dependencies to existing source files (The list of files that I had to
pull out of the main list just to add them with the dependency on
memory.h was getting too large. So I put them back into the sources
list, and added the dependency post-hoc.); and the ability to
reference 'source_root', 'build_root', and 'module_root' variables in
.module files.
- Some utility functions that were in the kernel's memory.h got moved to
util/pointers.h and util/misc.h, and misc.h's byteswap was renamed
byteswap32 to be more specific.
I'm a tabs guy. I like tabs, it's an elegant way to represent
indentation instead of brute-forcing it. But I have to admit that the
world seems to be going towards spaces, and tooling tends not to play
nice with tabs. So here we go, changing the whole repo to spaces since
I'm getting tired of all the inconsistent formatting.
Create a new usermode program, srv.init, and have it read the initial
module_page args sent to it by the bootloader. Doesn't yet do anything
useful but sets up the way for loading the rest of the programs from
srv.init.
Other (mostly) related changes:
- bootloader: The allocator now has a function for allocating init
modules out of a modules_page slab. Also changed how the allocator is
initialized and passes the allocation register and modules_page list
to efi_main().
- bootloader: Expose the simple wstrlen() to the rest of the program
- bootloader: Move check_cpu_supported() to hardware.cpp
- bootloader: Moved program_desc to loader.h and made the loader
functions take it as an argument instead of paths.
- kernel: Rename the system_map_mmio syscall to system_map_phys, and
stop having it default those VMAs to having the vm_flags::mmio flag.
Added a new flag mask, vm_flags::driver_mask, so that drivers can be
allowed to ask for the MMIO flag.
- kernel: Rename load_simple_process() to load_init_server() and got rid
of all the stack setup routines in memory_bootstrap.cpp and task.s
- Fixed formatting in config/debug.toml, undefined __linux and other
linux-specific defines, and got rid of _LIBCPP_HAS_THREAD_API_EXTERNAL
because that's just not true.
The bootloader relied on the kernel to know which parts of memory to not
allocate over. For the future shift of having the init process load
other processes instead of the kernel, the bootloader needs a mechanism
to just hand the kernel a list of allocations. This is now done through
the new bootloader allocator, which all allocation goes through. Pool
memory will not be tracked, and so can be overwritten - this means the
args structure and its other structures like programs need to be handled
right away, or copied by the kernel.
- Add bootloader allocator
- Implement a new linked-list based set of pages that act as allocation
registers
- Allow for operator new in the bootloader, which goes through the
global allocator for pool memory
- Split memory map and frame accouting code in the bootloader into
separate memory_map.* files
- Remove many includes that could be replaced by forward declaration in
the bootloader
- Add a new global template type, `counted`, which replaces the
bootloader's `buffer` type, and updated kernel args structure to use it.
- Move bootloader's pointer_manipulation.h to the global include dir
- Make offset_iterator try to return references instead of pointers to
make it more consistent with static array iteration
- Implement a stub atexit() in the bootloader to satisfy clang
The kernel's file header has not been verified for a long time. This
change returns file verification to the bootloader to make sure the ELF
loaded in position 0 is actually the kernel.
The kernel::args namespace is really the protocol for initializing the
kernel from the bootloader. Also, the header struct in that namespace
isn't actually a header, but a collection of parameters. This change
renames the namespace to kernel::init and the struct to args.
Remove ELF and initrd loading from the kernel. The bootloader now loads
the initial programs, as it does with the kernel. Other files that were
in the initrd are now on the ESP, and non-program files are just passed
as modules.
Set up initial page tables for both the offset-mapped area and the
loaded kernel code and data.
* Got rid of the `loaded_elf` struct - the loader now runs after the
initial PML4 is created and maps the ELF sections itself.
* Copied in the `page_table` and `page_table_indices` from the kernel,
still need to clean this up and extract it into shared code.
* Added `page_table_cache` to the kernel args to pass along free pages
that can be used for initial page tables.
Tags: paging
Removed the frame allocation logic from page_manager and replaced it
with using an instance of frame_allocator instead. This had several
major ripple effects:
- memory_initalize() had to change to support this new world
- Where to map used blocks is now passed as a flag, since blocks don't
track their virtual address anymore
- Instead of the complicated "find N contiguous pages that can be
mapped in with one page table", we now just have the bootloader give
us some (currently 64) pages to use both for tables and scratch
space.
- frame_allocator initialization was split into two steps to allow
mapping used blocks before std::move()ing them over
- Create initrd library to support definitions and loading
- Allow tools compiled for the host machine to be built by wscript
- Create makerd tool to build initrd from manifest
- Move screenfont to initrd, so don't load framebuffer initially
More work on process page tables, including only mapping the last 2 pml4
entries (the highest 1TiB of the address space, ie, kernel space) into a
new table.
Includes the work of actually moving the kernel there, which I had
apparently done in name only previously. Oops.
Return to having the bootloader re-map the kernel into the higher
half before jumping into the kernel entrypoint, so we don't have
to juggle pointers inside the kernel.
Map is still identity-mapped. I think we need to sort and/or clean up
the map before using a higher half address. In-kernel vga output not
working yet, but do_the_set_registers() is getting called.
