WIP framebuffer issue

We have one big array of characters in scrollback, with lines of
constant width -- there's no reason to have an array of pointers when
offsets will do.

README.md

@@ -1,7 +1,9 @@
-# jsix: A toy OS kernel
+![jsix](assets/jsix.svg)
 
-**jsix** is the kernel for the hobby OS that I am currently building. It's
-far from finished, or even being usable. Instead, it's a sandbox for me to play
+# jsix: A hobby operating system
+
+**jsix** is the hobby operating system that I am currently building. It's far
+from finished, or even being usable. Instead, it's a sandbox for me to play
 with kernel-level code and explore architectures.
 
 The design goals of the project are:
@@ -9,7 +11,8 @@ The design goals of the project are:
 * Modernity - I'm not interested in designing for legacy systems, or running on
   all hardware out there. My target is only 64 bit architecutres, and modern
   commodity hardware. Currently that means x64 systems with Nehalem or newer
-  CPUs and UEFI firmware. Eventually I'd like to work on an AArch64 port,
+  CPUs and UEFI firmware. (See [this list][cpu_features] for the currently
+  required CPU features.) Eventually I'd like to work on an AArch64 port,
   partly to force myself to factor out the architecture-dependent pieces of the
   code base.
 
@@ -17,8 +20,7 @@ The design goals of the project are:
   processes as possible, in the microkernel fashion. A sub-goal of this is to
   explore where the bottlenecks of such a microkernel are now, and whether
   eschewing legacy hardware will let me design a system that's less bogged down
-  by the traditional microkernel problems. Given that there are no processes
-  yet, the kernel is monolithic by default.
+  by the traditional microkernel problems.
 
 * Exploration - I'm really mostly doing this to have fun learning and exploring
   modern OS development. Modular design may be tossed out (hopefully
@@ -31,6 +33,8 @@ name, started around the same time as this project. So I've renamed this kernel
 jsix (Always styled _jsix_ or `j6`, never capitalized) as an homage to L4, xv6,
 and my wonderful wife.
 
+[cpu_features]: https://github.com/justinian/jsix/blob/master/src/libraries/cpu/include/cpu/features.inc
+
 ## Building
 
 jsix uses the [Ninja][] build tool, and generates the build files for it with a
@@ -38,7 +42,7 @@ custom tool called [Bonnibel][]. Bonnibel can be installed with [Cargo][], or
 downloaded as a prebuilt binary from its Github repository.
 
 [Ninja]:    https://ninja-build.org
-[Bonnibel]: https://github.com/justinian/bonnibel
+[Bonnibel]: https://github.com/justinian/bonnibel_rs
 [Cargo]:    https://crates.io/crates/bonnibel
 
 Requrirements:
@@ -71,8 +75,8 @@ I personally run this either from a real debian amd64 testing/buster machine or
 a windows WSL debian testing/buster installation. The following should be
 enough to set up such a system to build the kernel:
 
-    sudo apt install qemu-system-x86 nasm clang-6.0 mtools curl
-    sudo update-alternatives /usr/bin/clang clang /usr/bin/clang-6.0 1000
-    sudo update-alternatives /usr/bin/clang++ clang++ /usr/bin/clang++-6.0 1000
-	curl -L -o pb https://github.com/justinian/bonnibel/releases/download/2.0.0/pb_linux_amd64 && chmod a+x pb
+    sudo apt install qemu-system-x86 nasm clang-10 mtools curl ninja-build
+    sudo update-alternatives /usr/bin/clang clang /usr/bin/clang-10 1000
+    sudo update-alternatives /usr/bin/clang++ clang++ /usr/bin/clang++-10 1000
+	curl -L -o pb https://github.com/justinian/bonnibel_rs/releases/download/v2.3.0/pb-linux-amd64 && chmod a+x pb
 

assets/jsix.svg

@@ -0,0 +1 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?><!-- Generator: Gravit.io --><svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" style="isolation:isolate" viewBox="176.562 356.069 211.11 113" width="211.11pt" height="113pt"><g><g><rect x="176.562" y="356.069" width="211.11" height="113" transform="matrix(1,0,0,1,0,0)" fill="rgb(255,255,255)"/><g><path d=" M 212.981 372.36 L 219.564 376.16 L 226.147 379.961 L 226.147 387.563 L 226.147 395.164 L 219.564 398.965 L 212.981 402.766 L 206.398 398.965 L 199.815 395.164 L 199.815 387.563 L 199.815 379.961 L 206.398 376.16 L 212.981 372.36 L 212.981 372.36 L 212.981 372.36 Z  M 256.292 397.366 L 262.875 401.166 L 269.458 404.967 L 269.458 412.569 L 269.458 420.17 L 262.875 423.971 L 256.292 427.772 L 249.709 423.971 L 243.126 420.17 L 243.126 412.569 L 243.126 404.967 L 249.709 401.166 L 256.292 397.366 L 256.292 397.366 Z  M 183.622 387.283 L 205.52 374.64 L 227.418 361.997 L 249.316 374.64 L 271.214 387.283 L 271.214 412.569 L 271.214 437.854 L 249.316 450.497 L 227.418 463.14 L 205.52 450.497 L 183.622 437.854 L 183.622 412.569 L 183.622 387.283 L 183.622 387.283 L 183.622 387.283 Z  M 241.855 372.36 L 248.438 376.16 L 255.021 379.961 L 255.021 387.563 L 255.021 395.164 L 248.438 398.965 L 241.855 402.766 L 235.272 398.965 L 228.689 395.164 L 228.689 387.563 L 228.689 379.961 L 235.272 376.16 L 241.855 372.36 Z " fill-rule="evenodd" fill="rgb(49,79,128)"/><path d=" M 298.642 379.579 L 291.621 379.579 L 291.621 372.558 L 298.642 372.558 L 298.642 379.579 Z  M 285.214 446.718 L 285.214 441.452 L 287.32 441.452 L 287.32 441.452 Q 289.339 441.452 290.524 440.092 L 290.524 440.092 L 290.524 440.092 Q 291.708 438.731 291.708 436.625 L 291.708 436.625 L 291.708 387.039 L 298.729 387.039 L 298.729 436.011 L 298.729 436.011 Q 298.729 440.925 295.921 443.822 L 295.921 443.822 L 295.921 443.822 Q 293.113 446.718 288.286 446.718 L 288.286 446.718 L 285.214 446.718 Z  M 306.628 432.676 L 306.628 427.41 L 314.088 427.41 L 314.088 427.41 Q 317.862 427.41 319.573 425.347 L 319.573 425.347 L 319.573 425.347 Q 321.285 423.285 321.285 419.95 L 321.285 419.95 L 321.285 419.95 Q 321.285 417.317 319.705 415.474 L 319.705 415.474 L 319.705 415.474 Q 318.125 413.631 314.966 411.174 L 314.966 411.174 L 314.966 411.174 Q 312.245 408.98 310.621 407.356 L 310.621 407.356 L 310.621 407.356 Q 308.998 405.732 307.813 403.319 L 307.813 403.319 L 307.813 403.319 Q 306.628 400.905 306.628 397.746 L 306.628 397.746 L 306.628 397.746 Q 306.628 393.095 309.744 390.067 L 309.744 390.067 L 309.744 390.067 Q 312.859 387.039 318.125 387.039 L 318.125 387.039 L 325.76 387.039 L 325.76 392.305 L 319.441 392.305 L 319.441 392.305 Q 313.21 392.305 313.21 398.185 L 313.21 398.185 L 313.21 398.185 Q 313.21 400.467 314.615 402.134 L 314.615 402.134 L 314.615 402.134 Q 316.019 403.802 319.003 406.083 L 319.003 406.083 L 319.003 406.083 Q 321.723 408.19 323.479 409.901 L 323.479 409.901 L 323.479 409.901 Q 325.234 411.613 326.463 414.202 L 326.463 414.202 L 326.463 414.202 Q 327.691 416.791 327.691 420.301 L 327.691 420.301 L 327.691 420.301 Q 327.691 426.532 324.4 429.604 L 324.4 429.604 L 324.4 429.604 Q 321.109 432.676 315.141 432.676 L 315.141 432.676 L 306.628 432.676 Z  M 342.611 379.579 L 335.59 379.579 L 335.59 372.558 L 342.611 372.558 L 342.611 379.579 Z  M 342.611 432.676 L 335.59 432.676 L 335.59 387.039 L 342.611 387.039 L 342.611 432.676 Z  M 356.126 432.676 L 348.754 432.676 L 361.392 409.77 L 349.632 387.039 L 356.39 387.039 L 364.639 403.187 L 372.977 387.039 L 379.735 387.039 L 367.974 409.77 L 380.612 432.676 L 373.24 432.676 L 364.639 416.001 L 356.126 432.676 Z " fill="rgb(49,79,128)"/></g></g></g></svg>

external/uefi/boot_services.h

@@ -14,8 +14,10 @@
 namespace uefi {
 namespace bs_impl {
 	using allocate_pages = status (*)(allocate_type, memory_type, size_t, void**);
+	using free_pages = status (*)(void*, size_t);
 	using get_memory_map = status (*)(size_t*, memory_descriptor*, size_t*, size_t*, uint32_t*);
 	using allocate_pool = status (*)(memory_type, uint64_t, void**);
+	using free_pool = status (*)(void*);
 	using handle_protocol = status (*)(handle, const guid*, void**);
 	using create_event = status (*)(evt, tpl, event_notify, void*, event*);
 	using exit_boot_services = status (*)(handle, size_t);
@@ -35,10 +37,10 @@ struct boot_services {
 
 	// Memory Services
 	bs_impl::allocate_pages allocate_pages;
-	void *free_pages;
+	bs_impl::free_pages free_pages;
 	bs_impl::get_memory_map get_memory_map;
 	bs_impl::allocate_pool allocate_pool;
-	void *free_pool;
+	bs_impl::free_pool free_pool;
 
 	// Event & Timer Services
 	bs_impl::create_event create_event;

external/uefi/networking.h

@@ -0,0 +1,390 @@
+#pragma once
+#ifndef _uefi_networking_h_
+#define _uefi_networking_h_
+
+// This Source Code Form is part of the j6-uefi-headers project and is subject
+// to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was
+// not distributed with this file, You can obtain one at
+// http://mozilla.org/MPL/2.0/.
+
+#include <stdint.h>
+#include <uefi/types.h>
+
+namespace uefi {
+
+//
+// IPv4 definitions
+//
+struct ipv4_address
+{
+	uint8_t addr[4];
+};
+
+//
+// IPv6 definitions
+//
+struct ipv6_address
+{
+	uint8_t addr[16];
+};
+
+struct ip6_address_info
+{
+    ipv6_address address;
+    uint8_t prefix_length;
+};
+
+struct ip6_route_table
+{
+    ipv6_address gateway;
+    ipv6_address destination;
+    uint8_t prefix_length;
+};
+
+enum class ip6_neighbor_state : int {
+    incomplete,
+    reachable,
+    stale,
+    delay,
+    probe,
+}
+
+struct ip6_neighbor_cache
+{
+    ipv6_address neighbor;
+    mac_address link_address;
+    ip6_neighbor_state state;
+};
+
+enum class icmpv6_type : uint8_t
+{
+    dest_unreachable = 0x1,
+    packet_too_big = 0x2,
+    time_exceeded = 0x3,
+    parameter_problem = 0x4,
+    echo_request = 0x80,
+    echo_reply = 0x81,
+    listener_query = 0x82,
+    listener_report = 0x83,
+    listener_done = 0x84,
+    router_solicit = 0x85,
+    router_advertise = 0x86,
+    neighbor_solicit = 0x87,
+    neighbor_advertise = 0x88,
+    redirect = 0x89,
+    listener_report_2 = 0x8f,
+};
+
+enum class icmpv6_code : uint8_t
+{
+    // codes for icmpv6_type::dest_unreachable
+    no_route_to_dest = 0x0,
+    comm_prohibited = 0x1,
+    beyond_scope = 0x2,
+    addr_unreachable = 0x3,
+    port_unreachable = 0x4,
+    source_addr_failed = 0x5,
+    route_rejected = 0x6,
+
+    // codes for icmpv6_type::time_exceeded
+    timeout_hop_limit = 0x0,
+    timeout_reassemble = 0x1,
+
+    // codes for icmpv6_type::parameter_problem
+    erroneous_header = 0x0,
+    unrecognize_next_hdr = 0x1,
+    unrecognize_option = 0x2,
+};
+
+struct ip6_icmp_type
+{
+    icmpv6_type type;
+    icmpv6_code code;
+};
+
+struct ip6_config_data
+{
+    uint8_t default_protocol;
+    bool accept_any_protocol;
+    bool accept_icmp_errors;
+    bool accept_promiscuous;
+    ipv6_address destination_address;
+    ipv6_address station_address;
+    uint8_t traffic_class;
+    uint8_t hop_limit;
+    uint32_t flow_label;
+    uint32_t receive_timeout;
+    uint32_t transmit_timeout;
+};
+
+struct ip6_mode_data
+{
+    bool is_started;
+    uint32_t max_packet_size;
+    ip6_config_data config_data;
+    bool is_configured;
+    uint32_t address_count;
+    ip6_address_info * address_list;
+    uint32_t group_count;
+    ipv6_address * group_table;
+    uint32_t route_count;
+    ip6_route_table * route_table;
+    uint32_t neighbor_count;
+    ip6_neighbor_cache * neighbor_cache;
+    uint32_t prefix_count;
+    ip6_address_info * prefix_table;
+    uint32_t icmp_type_count;
+     * icmp_type_list;
+};
+
+struct ip6_header
+{
+    uint8_t traffic_class_h : 4;
+    uint8_t version : 4;
+    uint8_t flow_label_h : 4;
+    uint8_t traffic_class_l : 4;
+    uint16_t flow_label_l;
+    uint16_t payload_length;
+    uint8_t next_header;
+    uint8_t hop_limit;
+    ipv6_address source_address;
+    ipv6_address destination_address;
+} __attribute__ ((packed));
+
+struct ip6_fragment_data
+{
+    uint32_t fragment_length;
+    void *fragment_buffer;
+};
+
+struct ip6_override_data
+{
+    uint8_t protocol;
+    uint8_t hop_limit;
+    uint32_t flow_label;
+};
+
+struct ip6_receive_data
+{
+    time time_stamp;
+    event recycle_signal;
+    uint32_t header_length;
+    ip6_header *header;
+    uint32_t data_length;
+    uint32_t fragment_count;
+    ip6_fragment_data fragment_table[1];
+};
+
+struct ip6_transmit_data
+{
+    ipv6_address destination_address;
+    ip6_override_data *override_data;
+
+    uint32_t ext_hdrs_length;
+    void *ext_hdrs;
+    uint8_t next_header;
+    uint32_t data_length;
+    uint32_t fragment_count;
+    ip6_fragment_data fragment_table[1];
+};
+
+struct ip6_completion_token
+{
+    event event;
+    status status;
+    union {
+        ip6_receive_data *rx_data;
+        ip6_transmit_data *tx_data;
+    } packet;
+};
+
+enum class ip6_config_data_type : int
+{
+    interface_info,
+    alt_interface_id,
+    policy,
+    dup_addr_detect_transmits,
+    manual_address,
+    gateway,
+    dns_server,
+    maximum
+};
+
+struct ip6_config_interface_info
+{
+    wchar_t name[32];
+    uint8_t if_type;
+    uint32_t hw_address_size;
+    mac_address hw_address;
+    uint32_t address_info_count;
+    ip6_address_info *address_info;
+    uint32_t route_count;
+    ip6_route_table *route_table;
+};
+
+struct ip6_config_interface_id
+{
+    uint8_t id[8];
+};
+
+enum class ip6_config_policy : int
+{
+    manual,
+    automatic
+};
+
+struct ip6_config_dup_addr_detect_transmits
+{
+    uint32_t dup_addr_detect_transmits;
+};
+
+struct ip6_config_manual_address
+{
+    ipv6_address address;
+    bool is_anycast;
+    uint8_t prefix_length;
+};
+
+//
+// IP definitions
+//
+union ip_address
+{
+	uint8_t addr[4];
+	ipv4_address v4;
+	ipv6_address v6;
+};
+
+//
+// HTTP definitions
+//
+struct httpv4_access_point
+{
+	bool use_default_address;
+	ipv4_address local_address;
+	ipv4_address local_subnet;
+	uint16_t local_port;
+};
+
+struct httpv6_access_point
+{
+	ipv6_address local_address;
+	uint16_t local_port;
+};
+
+enum class http_version : int {
+	v10,
+	v11,
+	unsupported,
+};
+
+struct http_config_data
+{
+	http_version http_version;
+	uint32_t time_out_millisec;
+	bool local_address_is_ipv6;
+	union {
+		httpv4_access_point *ipv4_node;
+		httpv6_access_point *ipv6_node;
+	} access_point;
+};
+
+enum class http_method : int {
+	get,
+	post,
+	patch,
+	options,
+	connect,
+	head,
+	put,
+	delete_,
+	trace,
+};
+
+struct http_request_data
+{
+	http_method method;
+	wchar_t *url;
+};
+
+enum class http_status_code : int {
+	unsupported,
+	continue_,
+	switching_protocols,
+	ok,
+	created,
+	accepted,
+	non_authoritative_information,
+	no_content,
+	reset_content,
+	partial_content,
+	multiple_choices,
+	moved_permanently,
+	found,
+	see_other,
+	not_modified,
+	use_proxy,
+	temporary_redirect,
+	bad_request,
+	unauthorized,
+	payment_required,
+	forbidden,
+	not_found,
+	method_not_allowed,
+	not_acceptable,
+	proxy_authentication_required,
+	request_time_out,
+	conflict,
+	gone,
+	length_required,
+	precondition_failed,
+	request_entity_too_large,
+	request_uri_too_large,
+	unsupported_media_type,
+	requested_range_not_satisfied,
+	expectation_failed,
+	internal_server_error,
+	not_implemented,
+	bad_gateway,
+	service_unavailable,
+	gateway_timeout,
+	http_version_not_supported,
+	permanent_redirect,  // I hate your decisions, uefi.
+};
+
+struct http_response_data
+{
+	http_status_code status_code;
+};
+
+struct http_header
+{
+	char *field_name;
+	char *field_value;
+};
+
+struct http_message
+{
+	union {
+		http_request_data *request;
+		http_response_data *response;
+	} data;
+
+	size_t header_count;
+	http_header *headers;
+
+	size_t body_length;
+	void *body;
+};
+
+struct http_token
+{
+	event event;
+	status status;
+	http_message *message;
+};
+
+
+} // namespace uefi
+
+#endif

external/uefi/protos/device_path.h

@@ -16,6 +16,7 @@ struct device_path
 {
     static constexpr uefi::guid guid{ 0x09576e91,0x6d3f,0x11d2,{0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     uint8_t type;
     uint8_t sub_type;
     uint16_t length;

external/uefi/protos/file_info.h

@@ -16,6 +16,7 @@ struct file_info
 {
     static constexpr uefi::guid guid{ 0x09576e92,0x6d3f,0x11d2,{0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     uint64_t size;
     uint64_t file_size;
     uint64_t physical_size;

external/uefi/protos/graphics_output.h

@@ -17,6 +17,7 @@ struct graphics_output
 {
     static constexpr uefi::guid guid{ 0x9042a9de,0x23dc,0x4a38,{0x96,0xfb,0x7a,0xde,0xd0,0x80,0x51,0x6a} };
 
+
     inline uefi::status query_mode(uint32_t mode_number, uint64_t * size_of_info, uefi::graphics_output_mode_info ** info) {
         return _query_mode(this, mode_number, size_of_info, info);
     }

external/uefi/protos/http.h

@@ -0,0 +1,72 @@
+#pragma once
+#ifndef _uefi_protos_http_h_
+#define _uefi_protos_http_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+#include <uefi/networking.h>
+
+namespace uefi {
+namespace protos {
+struct http;
+
+struct http
+{
+    static constexpr uefi::guid guid{ 0x7a59b29b,0x910b,0x4171,{0x82,0x42,0xa8,0x5a,0x0d,0xf2,0x5b,0x5b} };
+    static constexpr uefi::guid service_binding{ 0xbdc8e6af,0xd9bc,0x4379,{0xa7,0x2a,0xe0,0xc4,0xe7,0x5d,0xae,0x1c} };
+
+
+    inline uefi::status get_mode_data(uefi::http_config_data * http_config_data) {
+        return _get_mode_data(this, http_config_data);
+    }
+
+    inline uefi::status configure(uefi::http_config_data * http_config_data) {
+        return _configure(this, http_config_data);
+    }
+
+    inline uefi::status request(uefi::http_token * token) {
+        return _request(this, token);
+    }
+
+    inline uefi::status cancel(uefi::http_token * token) {
+        return _cancel(this, token);
+    }
+
+    inline uefi::status response(uefi::http_token * token) {
+        return _response(this, token);
+    }
+
+    inline uefi::status poll() {
+        return _poll(this);
+    }
+
+
+protected:
+    using _get_mode_data_def = uefi::status (*)(uefi::protos::http *, uefi::http_config_data *);
+    _get_mode_data_def _get_mode_data;
+
+    using _configure_def = uefi::status (*)(uefi::protos::http *, uefi::http_config_data *);
+    _configure_def _configure;
+
+    using _request_def = uefi::status (*)(uefi::protos::http *, uefi::http_token *);
+    _request_def _request;
+
+    using _cancel_def = uefi::status (*)(uefi::protos::http *, uefi::http_token *);
+    _cancel_def _cancel;
+
+    using _response_def = uefi::status (*)(uefi::protos::http *, uefi::http_token *);
+    _response_def _response;
+
+    using _poll_def = uefi::status (*)(uefi::protos::http *);
+    _poll_def _poll;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_http_h_

external/uefi/protos/ip6.h

@@ -0,0 +1,93 @@
+#pragma once
+#ifndef _uefi_protos_ip6_h_
+#define _uefi_protos_ip6_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+#include <uefi/networking.h>
+
+namespace uefi {
+namespace protos {
+struct ip6;
+
+struct ip6
+{
+    static constexpr uefi::guid guid{ 0x2c8759d5,0x5c2d,0x66ef,{0x92,0x5f,0xb6,0x6c,0x10,0x19,0x57,0xe2} };
+    static constexpr uefi::guid service_binding{ 0xec835dd3,0xfe0f,0x617b,{0xa6,0x21,0xb3,0x50,0xc3,0xe1,0x33,0x88} };
+
+
+    inline uefi::status get_mode_data(uefi::ip6_mode_data * ip6_mode_data, uefi::managed_network_config_data * mnp_config_data, uefi::simple_network_mode * snp_config_data) {
+        return _get_mode_data(this, ip6_mode_data, mnp_config_data, snp_config_data);
+    }
+
+    inline uefi::status configure(uefi::ip6_config_data * ip6_config_data) {
+        return _configure(this, ip6_config_data);
+    }
+
+    inline uefi::status groups(bool join_flag, uefi::ipv6_address * group_address) {
+        return _groups(this, join_flag, group_address);
+    }
+
+    inline uefi::status routes(bool delete_route, uefi::ipv6_address * destination, uint8_t prefix_length, uefi::ipv6_address * gateway_address) {
+        return _routes(this, delete_route, destination, prefix_length, gateway_address);
+    }
+
+    inline uefi::status neighbors(bool delete_flag, uefi::ipv6_address * target_ip6_address, uefi::mac_address * target_link_address, uint32_t timeout, bool override) {
+        return _neighbors(this, delete_flag, target_ip6_address, target_link_address, timeout, override);
+    }
+
+    inline uefi::status transmit(uefi::ip6_completion_token * token) {
+        return _transmit(this, token);
+    }
+
+    inline uefi::status receive(uefi::ip6_completion_token * token) {
+        return _receive(this, token);
+    }
+
+    inline uefi::status cancel(uefi::ip6_completion_token * token) {
+        return _cancel(this, token);
+    }
+
+    inline uefi::status poll() {
+        return _poll(this);
+    }
+
+
+protected:
+    using _get_mode_data_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_mode_data *, uefi::managed_network_config_data *, uefi::simple_network_mode *);
+    _get_mode_data_def _get_mode_data;
+
+    using _configure_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_config_data *);
+    _configure_def _configure;
+
+    using _groups_def = uefi::status (*)(uefi::protos::ip6 *, bool, uefi::ipv6_address *);
+    _groups_def _groups;
+
+    using _routes_def = uefi::status (*)(uefi::protos::ip6 *, bool, uefi::ipv6_address *, uint8_t, uefi::ipv6_address *);
+    _routes_def _routes;
+
+    using _neighbors_def = uefi::status (*)(uefi::protos::ip6 *, bool, uefi::ipv6_address *, uefi::mac_address *, uint32_t, bool);
+    _neighbors_def _neighbors;
+
+    using _transmit_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_completion_token *);
+    _transmit_def _transmit;
+
+    using _receive_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_completion_token *);
+    _receive_def _receive;
+
+    using _cancel_def = uefi::status (*)(uefi::protos::ip6 *, uefi::ip6_completion_token *);
+    _cancel_def _cancel;
+
+    using _poll_def = uefi::status (*)(uefi::protos::ip6 *);
+    _poll_def _poll;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_ip6_h_

external/uefi/protos/ip6_config.h

@@ -0,0 +1,57 @@
+#pragma once
+#ifndef _uefi_protos_ip6_config_h_
+#define _uefi_protos_ip6_config_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+#include <uefi/networking.h>
+
+namespace uefi {
+namespace protos {
+struct ip6_config;
+
+struct ip6_config
+{
+    static constexpr uefi::guid guid{ 0x937fe521,0x95ae,0x4d1a,{0x89,0x29,0x48,0xbc,0xd9,0x0a,0xd3,0x1a} };
+
+
+    inline uefi::status set_data(uefi::ip6_config_data_type data_type, size_t data_size, void * data) {
+        return _set_data(this, data_type, data_size, data);
+    }
+
+    inline uefi::status get_data(uefi::ip6_config_data_type data_type, size_t data_size, void * data) {
+        return _get_data(this, data_type, data_size, data);
+    }
+
+    inline uefi::status register_data_notify(uefi::ip6_config_data_type data_type, uefi::event event) {
+        return _register_data_notify(this, data_type, event);
+    }
+
+    inline uefi::status unregister_data_notify(uefi::ip6_config_data_type data_type, uefi::event event) {
+        return _unregister_data_notify(this, data_type, event);
+    }
+
+
+protected:
+    using _set_data_def = uefi::status (*)(uefi::protos::ip6_config *, uefi::ip6_config_data_type, size_t, void *);
+    _set_data_def _set_data;
+
+    using _get_data_def = uefi::status (*)(uefi::protos::ip6_config *, uefi::ip6_config_data_type, size_t, void *);
+    _get_data_def _get_data;
+
+    using _register_data_notify_def = uefi::status (*)(uefi::protos::ip6_config *, uefi::ip6_config_data_type, uefi::event);
+    _register_data_notify_def _register_data_notify;
+
+    using _unregister_data_notify_def = uefi::status (*)(uefi::protos::ip6_config *, uefi::ip6_config_data_type, uefi::event);
+    _unregister_data_notify_def _unregister_data_notify;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_ip6_config_h_

external/uefi/protos/load_file.h

@@ -0,0 +1,36 @@
+#pragma once
+#ifndef _uefi_protos_load_file_h_
+#define _uefi_protos_load_file_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+#include <uefi/protos/device_path.h>
+
+namespace uefi {
+namespace protos {
+struct load_file;
+
+struct load_file
+{
+    static constexpr uefi::guid guid{ {0x56ec3091,0x954c,0x11d2,{0x8e,0x3f,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
+
+
+    inline uefi::status load_file(uefi::protos::device_path * file_path, bool boot_policy, size_t * buffer_size, void * buffer) {
+        return _load_file(this, file_path, boot_policy, buffer_size, buffer);
+    }
+
+
+protected:
+    using _load_file_def = uefi::status (*)(uefi::protos::load_file *, uefi::protos::device_path *, bool, size_t *, void *);
+    _load_file_def _load_file;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_load_file_h_

external/uefi/protos/loaded_image.h

@@ -18,6 +18,7 @@ struct loaded_image
 {
     static constexpr uefi::guid guid{ 0x5b1b31a1,0x9562,0x11d2,{0x8e,0x3f,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     inline uefi::status unload(uefi::handle image_handle) {
         return _unload(image_handle);
     }

external/uefi/protos/service_binding.h

@@ -0,0 +1,41 @@
+#pragma once
+#ifndef _uefi_protos_service_binding_h_
+#define _uefi_protos_service_binding_h_
+
+// This file was auto generated by the j6-uefi-headers project. Please see
+// https://github.com/justinian/j6-uefi-headers for more information.
+
+#include <uefi/guid.h>
+#include <uefi/types.h>
+
+namespace uefi {
+namespace protos {
+struct service_binding;
+
+struct service_binding
+{
+
+
+    inline uefi::status create_child(uefi::handle * child_handle) {
+        return _create_child(this, child_handle);
+    }
+
+    inline uefi::status destroy_child(uefi::handle child_handle) {
+        return _destroy_child(this, child_handle);
+    }
+
+
+protected:
+    using _create_child_def = uefi::status (*)(uefi::protos::service_binding *, uefi::handle *);
+    _create_child_def _create_child;
+
+    using _destroy_child_def = uefi::status (*)(uefi::protos::service_binding *, uefi::handle);
+    _destroy_child_def _destroy_child;
+
+public:
+};
+
+} // namespace protos
+} // namespace uefi
+
+#endif // _uefi_protos_service_binding_h_

external/uefi/protos/simple_file_system.h

@@ -17,6 +17,7 @@ struct simple_file_system
 {
     static constexpr uefi::guid guid{ 0x0964e5b22,0x6459,0x11d2,{0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     inline uefi::status open_volume(uefi::protos::file ** root) {
         return _open_volume(this, root);
     }

external/uefi/protos/simple_text_output.h

@@ -17,6 +17,7 @@ struct simple_text_output
 {
     static constexpr uefi::guid guid{ 0x387477c2,0x69c7,0x11d2,{0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b} };
 
+
     inline uefi::status reset(bool extended_verification) {
         return _reset(this, extended_verification);
     }

modules.yaml

@@ -29,7 +29,6 @@ modules:
             - src/kernel/interrupts.cpp
             - src/kernel/interrupts.s
             - src/kernel/io.cpp
-            - src/kernel/loader.s
             - src/kernel/log.cpp
             - src/kernel/main.cpp
             - src/kernel/memory_bootstrap.cpp
@@ -42,6 +41,7 @@ modules:
             - src/kernel/objects/system.cpp
             - src/kernel/objects/vm_area.cpp
             - src/kernel/page_table.cpp
+            - src/kernel/page_tree.cpp
             - src/kernel/pci.cpp
             - src/kernel/scheduler.cpp
             - src/kernel/serial.cpp
@@ -56,7 +56,6 @@ modules:
             - src/kernel/syscalls/thread.cpp
             - src/kernel/syscalls/vm_area.cpp
             - src/kernel/task.s
-            - src/kernel/vm_mapper.cpp
             - src/kernel/vm_space.cpp
 
     boot:
@@ -121,12 +120,22 @@ modules:
         source:
             - src/libraries/cpu/cpu.cpp
 
+    j6:
+      kind: lib
+      output: libj6.a
+      includes:
+        - src/libraries/j6/include
+      target: user
+      source:
+        - src/libraries/j6/syscalls.s
 
     libc:
         kind: lib
         output: libc.a
         includes:
           - src/libraries/libc/include
+        deps:
+            - j6
         target: user
         defines:
             - DISABLE_SSE
@@ -145,7 +154,6 @@ modules:
             - src/libraries/libc/arch/x86_64/_Exit.s
             - src/libraries/libc/arch/x86_64/crt0.s
             - src/libraries/libc/arch/x86_64/init_libc.c
-            - src/libraries/libc/arch/x86_64/syscalls.s
             - src/libraries/libc/ctype/isalnum.c
             - src/libraries/libc/ctype/isalpha.c
             - src/libraries/libc/ctype/isblank.c
@@ -314,6 +322,7 @@ modules:
             - src/tests/main.cpp
             - src/tests/map.cpp
             - src/tests/vector.cpp
+
 overlays:
     - url: https://f000.backblazeb2.com/file/jsix-os/sysroot-llvm8-20190706.tar.bz2
       path: sysroot

src/boot/console.cpp

@@ -55,7 +55,7 @@ console::console(uefi::boot_services *bs, uefi::protos::simple_text_output *out)
 	m_out->output_string(GIT_VERSION_WIDE);
 
 	m_out->set_attribute(uefi::attribute::light_gray);
-	m_out->output_string(L" booting...\r\n\n");
+	m_out->output_string(L" booting...\r\n");
 
 	if (m_fb.type != kernel::args::fb_type::none) {
 		wchar_t const * type = nullptr;

src/boot/error.cpp

@@ -35,27 +35,28 @@ message(uefi::status status)
 }
 
 [[ noreturn ]] static void
-cpu_assert(uefi::status s, const wchar_t *message)
+cpu_assert(uefi::status s, const wchar_t *message, size_t line)
 {
 	asm volatile (
 		"movq $0xeeeeeeebadbadbad, %%r8;"
 		"movq %0, %%r9;"
 		"movq %1, %%r10;"
+		"movq %2, %%r11;"
 		"movq $0, %%rdx;"
 		"divq %%rdx;"
 		:
-		: "r"((uint64_t)s), "r"(message)
+		: "r"((uint64_t)s), "r"(message), "r"(line)
 		: "rax", "rdx", "r8", "r9", "r10");
 	while (1) asm("hlt");
 }
 
 [[ noreturn ]] void
-raise(uefi::status status, const wchar_t *message)
+raise(uefi::status status, const wchar_t *message, size_t line)
 {
 	if(status_line::fail(message, status))
 		while (1) asm("hlt");
 	else
-		cpu_assert(status, message);
+		cpu_assert(status, message, line);
 }
 
 

src/boot/error.h

@@ -12,7 +12,7 @@ class console;
 namespace error {
 
 /// Halt or exit the program with the given error status/message
-[[ noreturn ]] void raise(uefi::status status, const wchar_t *message);
+[[ noreturn ]] void raise(uefi::status status, const wchar_t *message, size_t line = 0);
 
 const wchar_t * message(uefi::status status);
 
@@ -28,6 +28,6 @@ void debug_break();
 #define try_or_raise(s, m) \
 	do { \
 		uefi::status _s = (s); \
-		if (uefi::is_error(_s)) ::boot::error::raise(_s, (m)); \
+		if (uefi::is_error(_s)) ::boot::error::raise(_s, (m), __LINE__); \
 	} while(0)
 

src/boot/fs.cpp

@@ -78,7 +78,7 @@ file::load(uefi::memory_type mem_type)
 		m_file->read(&size, data),
 		L"Could not read from file");
 
-	return { .data = data, .size = size };
+	return { .size = size,  .data = data };
 }
 
 file

src/boot/guids.cpp

@@ -1,13 +0,0 @@
-#include "guids.h"
-
-#define GUID(dw, w1, w2, b1, b2, b3, b4, b5, b6, b7, b8, name) \
-	EFI_GUID name __attribute__((section(".rodata"))) = {dw, w1, w2, {b1, b2, b3, b4, b5, b6, b7, b8}}
-#include "guids.inc"
-#undef GUID
-
-int is_guid(EFI_GUID *a, EFI_GUID *b)
-{
-	uint64_t *ai = (uint64_t *)a;
-	uint64_t *bi = (uint64_t *)b;
-	return ai[0] == bi[0] && ai[1] == bi[1];
-}

src/boot/guids.h

@@ -1,8 +0,0 @@
-#pragma once
-#include <efi/efi.h>
-
-int is_guid(EFI_GUID *a, EFI_GUID *b);
-
-#define GUID(dw, w1, w2, b1, b2, b3, b4, b5, b6, b7, b8, name) extern EFI_GUID name
-#include "guids.inc"
-#undef GUID

src/boot/guids.inc

@@ -1,11 +0,0 @@
-GUID(0xeb9d2d30,0x2d88,0x11d3,0x9a,0x16,0x00,0x90,0x27,0x3f,0xc1,0x4d, guid_acpi1);
-GUID(0x8868e871,0xe4f1,0x11d3,0xbc,0x22,0x00,0x80,0xc7,0x3c,0x88,0x81, guid_acpi2);
-GUID(0x09576e92,0x6d3f,0x11d2,0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b, guid_file_info);
-GUID(0x9042a9de,0x23dc,0x4a38,0x96,0xfb,0x7a,0xde,0xd0,0x80,0x51,0x6a, guid_gfx_out);
-GUID(0x964e5b22,0x6459,0x11d2,0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b, guid_simple_filesystem);
-GUID(0x09576e91,0x6d3f,0x11d2,0x8e,0x39,0x00,0xa0,0xc9,0x69,0x72,0x3b, guid_device_path);
-GUID(0x8b843e20,0x8132,0x4852,0x90,0xcc,0x55,0x1a,0x4e,0x4a,0x7f,0x1c, guid_device_path_to_text);
-
-GUID(0x10d0669c,0x9ec6,0x4268,0xbc,0x48,0xff,0x74,0x75,0x21,0xfe,0x07, guid_jsix_vendor);
-
-// vim: ft=c

src/boot/hardware.cpp

@@ -37,8 +37,25 @@ find_acpi_table(uefi::system_table *st)
 	return reinterpret_cast<void*>(acpi1_table);
 }
 
+static uint64_t
+rdmsr(uint32_t addr)
+{
+	uint32_t low, high;
+	__asm__ __volatile__ ("rdmsr" : "=a"(low), "=d"(high) : "c"(addr));
+	return (static_cast<uint64_t>(high) << 32) | low;
+}
+
+static void
+wrmsr(uint32_t addr, uint64_t value)
+{
+	uint32_t low = value & 0xffffffff;
+	uint32_t high = value >> 32;
+	__asm__ __volatile__ ("wrmsr" :: "c"(addr), "a"(low), "d"(high));
+}
+
+
 void
-setup_cr4()
+setup_control_regs()
 {
 	uint64_t cr4 = 0;
 	asm volatile ( "mov %%cr4, %0" : "=r" (cr4) );
@@ -49,6 +66,15 @@ setup_cr4()
 		0x020000 | // Enable PCIDs
 		0;
 	asm volatile ( "mov %0, %%cr4" :: "r" (cr4) );
+
+	// Set up IA32_EFER
+	constexpr uint32_t IA32_EFER = 0xC0000080;
+	uint64_t efer = rdmsr(IA32_EFER);
+	efer |=
+		0x0001 | // Enable SYSCALL
+		0x0800 | // Enable NX bit
+		0;
+	wrmsr(IA32_EFER, efer);
 }
 
 } // namespace hw

src/boot/hardware.h

@@ -13,8 +13,8 @@ namespace hw {
 /// significant bit set to 1.
 void * find_acpi_table(uefi::system_table *st);
 
-/// Enable CPU options in CR4 for the kernel starting state.
-void setup_cr4();
+/// Enable CPU options in CR4 etc for the kernel starting state.
+void setup_control_regs();
 
 } // namespace hw
 } // namespace boot

src/boot/loader.cpp

@@ -27,7 +27,7 @@ load_file(
 	fs::file file = disk.open(path);
 	buffer b = file.load(type);
 
-	console::print(L"    Loaded at: 0x%lx, %d bytes\r\n", b.data, b.size);
+	//console::print(L"    Loaded at: 0x%lx, %d bytes\r\n", b.data, b.size);
 	return b;
 }
 
@@ -87,6 +87,9 @@ load_program(
 
 	bs->set_mem(pages, total_size, 0);
 
+	program.base = prog_base;
+	program.total_size = total_size;
+	program.num_sections = 0;
 	for (int i = 0; i < header->ph_num; ++i) {
 		ptrdiff_t offset = header->ph_offset + i * header->ph_entsize;
 		const elf::program_header *pheader =
@@ -95,14 +98,18 @@ load_program(
 		if (pheader->type != elf::PT_LOAD)
 			continue;
 
+		args::program_section &section = program.sections[program.num_sections++];
+
 		void *src_start = offset_ptr<void>(data.data, pheader->offset);
 		void *dest_start = offset_ptr<void>(pages, pheader->vaddr - prog_base);
+
 		bs->copy_mem(dest_start, src_start, pheader->file_size);
+		section.phys_addr = reinterpret_cast<uintptr_t>(dest_start);
+		section.virt_addr = pheader->vaddr;
+		section.size = pheader->mem_size;
+		section.type = static_cast<args::section_flags>(pheader->flags);
 	}
 
-	program.phys_addr = reinterpret_cast<uintptr_t>(pages);
-	program.size = total_size;
-	program.virt_addr = prog_base;
 	program.entrypoint = header->entrypoint;
 }
 

src/boot/main.cpp

@@ -152,18 +152,13 @@ uefi_preboot(uefi::handle image, uefi::system_table *st)
 		loader::load_program(program, desc.name, buf, bs);
 	}
 
-	for (unsigned i = 0; i < args->num_modules; ++i) {
-		args::module &mod = args->modules[i];
-		change_pointer(mod.location);
-	}
-
 	return args;
 }
 
 memory::efi_mem_map
 uefi_exit(args::header *args, uefi::handle image, uefi::boot_services *bs)
 {
-	status_line status {L"Exiting UEFI"};
+	status_line status {L"Exiting UEFI", nullptr, false};
 
 	memory::efi_mem_map map =
 		memory::build_kernel_mem_map(args, bs);
@@ -191,17 +186,24 @@ efi_main(uefi::handle image, uefi::system_table *st)
 	args->video = con.fb();
 	status_bar status {con.fb()}; // Switch to fb status display
 
+	// Map the kernel to the appropriate address
 	args::program &kernel = args->programs[0];
-	paging::map_pages(args, kernel.phys_addr, kernel.virt_addr, kernel.size);
+	for (auto &section : kernel.sections)
+		if (section.size)
+			paging::map_section(args, section);
+
+	memory::fix_frame_blocks(args);
+
 	kernel::entrypoint kentry =
 		reinterpret_cast<kernel::entrypoint>(kernel.entrypoint);
 	status.next();
 
+
+	hw::setup_control_regs();
 	memory::virtualize(args->pml4, map, st->runtime_services);
 	status.next();
 
 	change_pointer(args->pml4);
-	hw::setup_cr4();
 	status.next();
 
 	kentry(args);

src/boot/memory.cpp

@@ -14,6 +14,8 @@ namespace memory {
 
 using mem_entry = kernel::args::mem_entry;
 using mem_type = kernel::args::mem_type;
+using frame_block = kernel::args::frame_block;
+using kernel::args::frames_per_block;
 
 size_t fixup_pointer_index = 0;
 void **fixup_pointers[64];
@@ -107,29 +109,143 @@ can_merge(mem_entry &prev, mem_type type, uefi::memory_descriptor *next)
 }
 
 void
-get_uefi_mappings(efi_mem_map *map, bool allocate, uefi::boot_services *bs)
+get_uefi_mappings(efi_mem_map &map, uefi::boot_services *bs)
 {
-	size_t length = 0;
+	size_t length = map.total;
 	uefi::status status = bs->get_memory_map(
-		&length, nullptr, &map->key, &map->size, &map->version);
+		&length, map.entries, &map.key, &map.size, &map.version);
+	map.length = length;
+
+	if (status == uefi::status::success)
+		return;
 
 	if (status != uefi::status::buffer_too_small)
 		error::raise(status, L"Error getting memory map size");
 
-	map->length = length;
-
-	if (allocate) {
-		map->length += 10*map->size;
-
+	if (map.entries) {
 		try_or_raise(
-			bs->allocate_pool(
-				uefi::memory_type::loader_data, map->length,
-				reinterpret_cast<void**>(&map->entries)),
-			L"Allocating space for memory map");
+			bs->free_pool(reinterpret_cast<void*>(map.entries)),
+			L"Freeing previous memory map space");
+	}
 
-		try_or_raise(
-			bs->get_memory_map(&map->length, map->entries, &map->key, &map->size, &map->version),
-			L"Getting UEFI memory map");
+	map.total = length + 10*map.size;
+
+	try_or_raise(
+		bs->allocate_pool(
+			uefi::memory_type::loader_data, map.total,
+			reinterpret_cast<void**>(&map.entries)),
+		L"Allocating space for memory map");
+
+	map.length = map.total;
+	try_or_raise(
+		bs->get_memory_map(&map.length, map.entries, &map.key, &map.size, &map.version),
+		L"Getting UEFI memory map");
+}
+
+inline size_t bitmap_size(size_t frames) { return (frames + 63) / 64; }
+inline size_t num_blocks(size_t frames) { return (frames + (frames_per_block-1)) / frames_per_block; }
+
+void
+build_kernel_frame_blocks(const mem_entry *map, size_t nent, kernel::args::header *args, uefi::boot_services *bs)
+{
+	status_line status {L"Creating kernel frame accounting map"};
+
+	size_t block_count = 0;
+	size_t total_bitmap_size = 0;
+	for (size_t i = 0; i < nent; ++i) {
+		const mem_entry &ent = map[i];
+		if (ent.type != mem_type::free)
+			continue;
+
+		block_count += num_blocks(ent.pages);
+		total_bitmap_size += bitmap_size(ent.pages) * sizeof(uint64_t);
+	}
+
+	size_t total_size = block_count * sizeof(frame_block) + total_bitmap_size;
+
+	frame_block *blocks = nullptr;
+	try_or_raise(
+		bs->allocate_pages(
+			uefi::allocate_type::any_pages,
+			uefi::memory_type::loader_data,
+			bytes_to_pages(total_size),
+			reinterpret_cast<void**>(&blocks)),
+		L"Error allocating kernel frame block space");
+
+	frame_block *next_block = blocks;
+	for (size_t i = 0; i < nent; ++i) {
+		const mem_entry &ent = map[i];
+		if (ent.type != mem_type::free)
+			continue;
+
+		size_t page_count = ent.pages;
+		uintptr_t base_addr = ent.start;
+		while (page_count) {
+			frame_block *blk = next_block++;
+			bs->set_mem(blk, sizeof(frame_block), 0);
+
+			blk->attrs = ent.attr;
+			blk->base = base_addr;
+			base_addr += frames_per_block * page_size;
+
+			if (page_count >= frames_per_block) {
+				page_count -= frames_per_block;
+				blk->count = frames_per_block;
+				blk->map1 = ~0ull;
+				bs->set_mem(blk->map2, sizeof(blk->map2), 0xff);
+			} else {
+				blk->count = page_count;
+				unsigned i = 0;
+
+				uint64_t b1 = (page_count + 4095) / 4096;
+				blk->map1 = (1 << b1) - 1;
+
+				uint64_t b2 = (page_count + 63) / 64;
+				uint64_t b2q = b2 / 64;
+				uint64_t b2r = b2 % 64;
+				bs->set_mem(blk->map2, b2q, 0xff);
+				blk->map2[b2q] = (1 << b2r) - 1;
+				break;
+			}
+		}
+	}
+
+	uint64_t *bitmap = reinterpret_cast<uint64_t*>(next_block);
+	bs->set_mem(bitmap, total_bitmap_size, 0);
+	for (unsigned i = 0; i < block_count; ++i) {
+		frame_block &blk = blocks[i];
+		blk.bitmap = bitmap;
+
+		size_t b = blk.count / 64;
+		size_t r = blk.count % 64;
+		bs->set_mem(blk.bitmap, b*8, 0xff);
+		blk.bitmap[b] = (1 << r) - 1;
+
+		bitmap += bitmap_size(blk.count);
+	}
+
+	args->frame_block_count = block_count;
+	args->frame_block_pages = bytes_to_pages(total_size);
+	args->frame_blocks = blocks;
+}
+
+void
+fix_frame_blocks(kernel::args::header *args)
+{
+	// Map the frame blocks to the appropriate address
+	paging::map_pages(args,
+		reinterpret_cast<uintptr_t>(args->frame_blocks),
+		::memory::bitmap_start,
+		args->frame_block_pages,
+		true, false);
+
+	uintptr_t offset = ::memory::bitmap_start -
+		reinterpret_cast<uintptr_t>(args->frame_blocks);
+
+	for (unsigned i = 0; i < args->frame_block_count; ++i) {
+		frame_block &blk = args->frame_blocks[i];
+		blk.bitmap = reinterpret_cast<uint64_t*>(
+			reinterpret_cast<uintptr_t>(blk.bitmap) + offset);
 	}
 }
 
@@ -139,7 +255,7 @@ build_kernel_mem_map(kernel::args::header *args, uefi::boot_services *bs)
 	status_line status {L"Creating kernel memory map"};
 
 	efi_mem_map map;
-	get_uefi_mappings(&map, false, bs);
+	get_uefi_mappings(map, bs);
 
 	size_t map_size = map.num_entries() * sizeof(mem_entry);
 
@@ -153,14 +269,14 @@ build_kernel_mem_map(kernel::args::header *args, uefi::boot_services *bs)
 		L"Error allocating kernel memory map module space");
 
 	bs->set_mem(kernel_map, map_size, 0);
-	get_uefi_mappings(&map, true, bs);
+	get_uefi_mappings(map, bs);
 
-	size_t i = 0;
+	size_t nent = 0;
 	bool first = true;
 	for (auto desc : map) {
 		/*
 		// EFI map dump
-		console::print(L"   Range %lx (%lx) %x(%s) [%lu]\r\n",
+		console::print(L"   eRange %lx (%lx) %x(%s) [%lu]\r\n",
 			desc->physical_start, desc->attribute, desc->type, memory_type_name(desc->type), desc->number_of_pages);
 		*/
 
@@ -176,11 +292,8 @@ build_kernel_mem_map(kernel::args::header *args, uefi::boot_services *bs)
 			case uefi::memory_type::boot_services_code:
 			case uefi::memory_type::boot_services_data:
 			case uefi::memory_type::conventional_memory:
-				type = mem_type::free;
-				break;
-
 			case uefi::memory_type::loader_data:
-				type = mem_type::pending;
+				type = mem_type::free;
 				break;
 
 			case uefi::memory_type::runtime_services_code:
@@ -210,18 +323,19 @@ build_kernel_mem_map(kernel::args::header *args, uefi::boot_services *bs)
 		// TODO: validate uefi's map is sorted
 		if (first) {
 			first = false;
-			kernel_map[i].start = desc->physical_start;
-			kernel_map[i].pages = desc->number_of_pages;
-			kernel_map[i].type = type;
-			kernel_map[i].attr = (desc->attribute & 0xffffffff);
+			mem_entry &ent = kernel_map[nent++];
+			ent.start = desc->physical_start;
+			ent.pages = desc->number_of_pages;
+			ent.type = type;
+			ent.attr = (desc->attribute & 0xffffffff);
 			continue;
 		}
 
-		mem_entry &prev = kernel_map[i];
+		mem_entry &prev = kernel_map[nent - 1];
 		if (can_merge(prev, type, desc)) {
 			prev.pages += desc->number_of_pages;
 		} else {
-			mem_entry &next = kernel_map[++i];
+			mem_entry &next = kernel_map[nent++];
 			next.start = desc->physical_start;
 			next.pages = desc->number_of_pages;
 			next.type = type;
@@ -231,17 +345,19 @@ build_kernel_mem_map(kernel::args::header *args, uefi::boot_services *bs)
 
 	// Give just the actually-set entries in the header
 	args->mem_map = kernel_map;
-	args->map_count = i;
+	args->map_count = nent;
 
 	/*
 	// kernel map dump
-	for (unsigned i = 0; i < args->map_count; ++i) {
+	for (unsigned i = 0; i < nent; ++i) {
 		const kernel::args::mem_entry &e = kernel_map[i];
-		console::print(L"   Range %lx (%lx) %x(%s) [%lu]\r\n",
+		console::print(L"   kRange %lx (%lx) %x(%s) [%lu]\r\n",
 			e.start, e.attr, e.type, kernel_memory_type_name(e.type), e.pages);
 	}
 	*/
 
+	build_kernel_frame_blocks(kernel_map, nent, args, bs);
+	get_uefi_mappings(map, bs);
 	return map;
 }
 

src/boot/memory.h

@@ -41,12 +41,13 @@ struct efi_mem_map
 	using iterator = offset_iterator<desc>;
 
 	size_t length;     ///< Total length of the map data
+	size_t total;      ///< Total allocated space for map data
 	size_t size;       ///< Size of an entry in the array
 	size_t key;        ///< Key for detecting changes
 	uint32_t version;  ///< Version of the `memory_descriptor` struct
 	desc *entries;     ///< The array of UEFI descriptors
 
-	efi_mem_map() : length(0), size(0), key(0), version(0), entries(nullptr) {}
+	efi_mem_map() : length(0), total(0), size(0), key(0), version(0), entries(nullptr) {}
 
 	/// Get the count of entries in the array
 	inline size_t num_entries() const { return length / size; }
@@ -62,6 +63,14 @@ struct efi_mem_map
 /// \returns  The uefi memory map used to build the kernel map
 efi_mem_map build_kernel_mem_map(kernel::args::header *args, uefi::boot_services *bs);
 
+/// Create the kernel frame allocation maps
+void build_kernel_frame_blocks(
+		const kernel::args::mem_entry *map, size_t nent,
+		kernel::args::header *args, uefi::boot_services *bs);
+
+/// Map the frame allocation maps to the right spot and fix up pointers
+void fix_frame_blocks(kernel::args::header *args);
+
 /// Activate the given memory mappings. Sets the given page tables live as well
 /// as informs UEFI runtime services of the new mappings.
 /// \arg pml4  The root page table for the new mappings

src/boot/paging.cpp

@@ -15,7 +15,7 @@ using memory::page_size;
 using ::memory::pml4e_kernel;
 using ::memory::table_entries;
 
-// Flags: 0 0 0 1  0 0 0 0  0 0 1 1 = 0x0103
+// Flags: 0 0 0 1  0 0 0 0  0 0 0 1 = 0x0101
 //         IGN  |  | | | |  | | | +- Present
 //              |  | | | |  | | +--- Writeable
 //              |  | | | |  | +----- Usermode access (supervisor only)
@@ -26,7 +26,7 @@ using ::memory::table_entries;
 //              |  +---------------- PAT (determining memory type for page)
 //              +------------------- Global
 /// Page table entry flags for entries pointing at a page
-constexpr uint16_t page_flags = 0x103;
+constexpr uint64_t page_flags = 0x101;
 
 // Flags: 0  0 0 0 1  1 0 0 0  1 0 1 1 = 0x018b
 //        |   IGN  |  | | | |  | | | +- Present
@@ -40,7 +40,7 @@ constexpr uint16_t page_flags = 0x103;
 //        |        +------------------- Global
 //        +---------------------------- PAT (determining memory type for page)
 /// Page table entry flags for entries pointing at a huge page
-constexpr uint16_t huge_page_flags = 0x183;
+constexpr uint64_t huge_page_flags = 0x18b;
 
 // Flags: 0 0 0 0  0 0 0 0  0 0 1 1 = 0x0003
 //        IGNORED  | | | |  | | | +- Present
@@ -52,7 +52,7 @@ constexpr uint16_t huge_page_flags = 0x183;
 //                 | +-------------- Ignored
 //                 +---------------- Reserved 0 (Table pointer, not page)
 /// Page table entry flags for entries pointing at another table
-constexpr uint16_t table_flags = 0x003;
+constexpr uint64_t table_flags = 0x003;
 
 /// Iterator over page table entries.
 template <unsigned D = 4>
@@ -118,7 +118,7 @@ private:
 
 		if (!(parent_ent & 1)) {
 			if (!m_cache_count--)
-				error::raise(uefi::status::out_of_resources, L"Page table cache empty");
+				error::raise(uefi::status::out_of_resources, L"Page table cache empty", 0x7ab1e5);
 
 			page_table *table = reinterpret_cast<page_table*>(m_page_cache);
 			m_page_cache = offset_ptr<void>(m_page_cache, page_size);
@@ -191,7 +191,7 @@ allocate_tables(kernel::args::header *args, uefi::boot_services *bs)
 	status_line status(L"Allocating initial page tables");
 
 	static constexpr size_t pd_tables = 256;   // number of pages for kernelspace PDs
-	static constexpr size_t extra_tables = 49; // number of extra pages
+	static constexpr size_t extra_tables = 64; // number of extra pages
 
 	// number of pages for kernelspace PDs + PML4
 	static constexpr size_t kernel_tables = pd_tables + 1;
@@ -212,6 +212,7 @@ allocate_tables(kernel::args::header *args, uefi::boot_services *bs)
 	page_table *pml4 = reinterpret_cast<page_table*>(addr);
 
 	args->pml4 = pml4;
+	args->table_pages = tables_needed;
 	args->table_count = tables_needed - 1;
 	args->page_tables = offset_ptr<void>(addr, page_size);
 
@@ -220,27 +221,42 @@ allocate_tables(kernel::args::header *args, uefi::boot_services *bs)
 	add_kernel_pds(pml4, args->page_tables, args->table_count);
 	add_offset_mappings(pml4, args->page_tables, args->table_count);
 
-	console::print(L"    Set up initial mappings, %d spare tables.\r\n", args->table_count);
+	//console::print(L"    Set up initial mappings, %d spare tables.\r\n", args->table_count);
+}
+
+template <typename E>
+constexpr bool has_flag(E set, E flag) {
+	return
+		(static_cast<uint64_t>(set) & static_cast<uint64_t>(flag)) ==
+		 static_cast<uint64_t>(flag);
 }
 
 void
 map_pages(
 	kernel::args::header *args,
 	uintptr_t phys, uintptr_t virt,
-	size_t size)
+	size_t count, bool write_flag, bool exe_flag)
 {
+	if (!count)
+		return;
+
 	paging::page_table *pml4 =
 		reinterpret_cast<paging::page_table*>(args->pml4);
 
-	size_t pages = memory::bytes_to_pages(size);
 	page_entry_iterator<4> iterator{
 		virt, pml4,
 		args->page_tables,
 		args->table_count};
 
+	uint64_t flags = page_flags;
+	if (!exe_flag)
+		flags |= (1ull << 63); // set NX bit
+	if (write_flag)
+		flags |= 2;
+
 	while (true) {
-		*iterator = phys | page_flags;
-		if (--pages == 0)
+		*iterator = phys | flags;
+		if (--count == 0)
 			break;
 
 		iterator.increment();
@@ -248,6 +264,24 @@ map_pages(
 	}
 }
 
+void
+map_section(
+	kernel::args::header *args,
+	const kernel::args::program_section &section)
+{
+	using kernel::args::section_flags;
+
+	size_t pages = memory::bytes_to_pages(section.size);
+
+	map_pages(
+		args,
+		section.phys_addr,
+		section.virt_addr,
+		pages,
+		has_flag(section.type, section_flags::write),
+		has_flag(section.type, section_flags::execute));
+}
+
 
 } // namespace paging
 } // namespace boot

src/boot/paging.h

@@ -38,15 +38,22 @@ void allocate_tables(
 /// tables in the current PML4.
 void add_current_mappings(page_table *new_pml4);
 
-/// Map a physical address to a virtual address in the given page tables.
-/// \arg args  The kernel args header, used for the page table cache and pml4
-/// \arg phys  The phyiscal address to map in
-/// \arg virt  The virtual address to map in
-/// \arg size  The size in bytes of the mapping 
+/// Map physical memory pages to virtual addresses in the given page tables.
+/// \arg args    The kernel args header, used for the page table cache and pml4
+/// \arg section The program section to load
 void map_pages(
 	kernel::args::header *args,
 	uintptr_t phys, uintptr_t virt,
-	size_t bytes);
+	size_t count, bool write_flag, bool exe_flag);
+
+/// Map a program section in physical memory to its virtual address in the
+/// given page tables.
+/// \arg args    The kernel args header, used for the page table cache and pml4
+/// \arg section The program section to load
+void map_section(
+	kernel::args::header *args,
+	const kernel::args::program_section &section);
+
 
 } // namespace paging
 } // namespace boot

src/boot/status.cpp

@@ -30,7 +30,8 @@ status *status::s_current = nullptr;
 unsigned status::s_current_type = 0;
 unsigned status_bar::s_count = 0;
 
-status_line::status_line(const wchar_t *message, const wchar_t *context) :
+status_line::status_line(const wchar_t *message, const wchar_t *context, bool fails_clean) :
+	status(fails_clean),
 	m_level(level_ok),
 	m_depth(0),
 	m_outer(nullptr)
@@ -148,6 +149,7 @@ status_line::do_fail(const wchar_t *message, uefi::status status)
 
 
 status_bar::status_bar(kernel::args::framebuffer const &fb) :
+	status(fb.size),
 	m_outer(nullptr)
 {
 	m_size = (fb.vertical / num_boxes) - 1;

src/boot/status.h

@@ -17,6 +17,8 @@ namespace boot {
 class status
 {
 public:
+	status(bool fails_clean = true) : m_fails_clean(fails_clean) {}
+
 	virtual void do_warn(const wchar_t *message, uefi::status status) = 0;
 	virtual void do_fail(const wchar_t *message, uefi::status status) = 0;
 
@@ -28,7 +30,7 @@ public:
 	inline static bool warn(const wchar_t *message, uefi::status status = uefi::status::success) {
 		if (!s_current) return false;
 		s_current->do_warn(message, status);
-		return true;
+		return s_current->m_fails_clean;
 	}
 
 	/// Set the state to error, and print a message. If the state is already at
@@ -39,12 +41,15 @@ public:
 	inline static bool fail(const wchar_t *message, uefi::status status) {
 		if (!s_current) return false;
 		s_current->do_fail(message, status);
-		return true;
+		return s_current->m_fails_clean;
 	}
 
 protected:
 	static status *s_current;
 	static unsigned s_current_type;
+
+private:
+	bool m_fails_clean;
 };
 
 /// Scoped status line reporter. Prints a message and an "OK" if no errors
@@ -59,7 +64,11 @@ public:
 	/// Constructor.
 	/// \arg message  Description of the operation in progress
 	/// \arg context  If non-null, printed after `message` and a colon
-	status_line(const wchar_t *message, const wchar_t *context = nullptr);
+	/// \arg fails_clean If true, this object can handle printing failure
+	status_line(
+		const wchar_t *message,
+		const wchar_t *context = nullptr,
+		bool fails_clean = true);
 	~status_line();
 
 	virtual void do_warn(const wchar_t *message, uefi::status status) override;

src/drivers/fb/font.cpp

@@ -62,6 +62,9 @@ font::draw_glyph(
 		unsigned x,
 		unsigned y) const
 {
+	if (glyph >= m_count)
+		return;
+
 	unsigned bwidth = (m_sizex+7)/8;
 	uint8_t const *data = m_data + (glyph * glyph_bytes());
 

src/drivers/fb/main.cpp

@@ -4,11 +4,11 @@
 
 #include "j6/init.h"
 #include "j6/errors.h"
+#include "j6/flags.h"
 #include "j6/signals.h"
+#include "j6/syscalls.h"
 #include "j6/types.h"
 
-#include <j6libc/syscalls.h>
-
 #include "font.h"
 #include "screen.h"
 #include "scrollback.h"
@@ -19,6 +19,7 @@ extern "C" {
 }
 
 extern j6_handle_t __handle_sys;
+extern j6_handle_t __handle_self;
 
 struct entry
 {
@@ -29,10 +30,36 @@ struct entry
 	char message[0];
 };
 
+void
+draw_stuff(screen &scr, font &fnt)
+{
+	screen::pixel_t fg = scr.color(0xb0, 0xb0, 0xb0);
+	screen::pixel_t bg = scr.color(49, 79, 128);
+
+	unsigned h = fnt.height();
+	unsigned w = fnt.width();
+
+	unsigned lines = scr.height()/h;
+	unsigned columns = scr.width()/w;
+
+	for (unsigned y = 0; y < lines; ++y) {
+		for (unsigned x = 0; x < columns; ++x) {
+			char d = (x + y * columns) % 10 + '0';
+			fnt.draw_glyph(scr, d, fg, bg, x*w, y*h);
+		}
+	}
+
+	for (unsigned y = 0; y < scr.height(); ++y) {
+		for (unsigned x = 0; x < scr.width(); x += 61) {
+			scr.draw_pixel(x, y, 0xffffff);
+		}
+	}
+}
+
 int
 main(int argc, const char **argv)
 {
-	_syscall_system_log("fb driver starting");
+	j6_system_log("fb driver starting");
 
 	size_t initc = 0;
 	j6_init_value *initv = nullptr;
@@ -41,20 +68,40 @@ main(int argc, const char **argv)
 	j6_init_framebuffer *fb = nullptr;
 	for (unsigned i = 0; i < initc; ++i) {
 		if (initv[i].type == j6_init_desc_framebuffer) {
-			fb = reinterpret_cast<j6_init_framebuffer*>(initv[i].value);
+			fb = reinterpret_cast<j6_init_framebuffer*>(initv[i].data);
 			break;
 		}
 	}
 
-	if (!fb || fb->addr == nullptr) {
-		_syscall_system_log("fb driver didn't find a framebuffer, exiting");
+	if (!fb || fb->addr == 0) {
+		j6_system_log("fb driver didn't find a framebuffer, exiting");
 		return 1;
 	}
 
+	j6_handle_t fb_handle = j6_handle_invalid;
+	uint32_t flags =
+		j6_vm_flag_write |
+		j6_vm_flag_write_combine;
+	j6_status_t s = j6_system_map_mmio(__handle_sys, &fb_handle, fb->addr, fb->size, flags);
+	if (s != j6_status_ok) {
+		return s;
+	}
+
+	s = j6_vma_map(fb_handle, __handle_self, fb->addr);
+	if (s != j6_status_ok) {
+		return s;
+	}
+
 	const screen::pixel_order order = (fb->flags & 1) ?
 		screen::pixel_order::bgr8 : screen::pixel_order::rgb8;
 
-	screen scr(fb->addr, fb->horizontal, fb->vertical, order);
+	screen scr(
+		reinterpret_cast<void*>(fb->addr),
+		fb->horizontal,
+		fb->vertical,
+		fb->scanline,
+		order);
+
 	font fnt;
 
 	screen::pixel_t fg = scr.color(0xb0, 0xb0, 0xb0);
@@ -62,6 +109,10 @@ main(int argc, const char **argv)
 	scr.fill(bg);
 	scr.update();
 
+	draw_stuff(scr, fnt);
+	scr.update();
+
+	/*
 	constexpr int margin = 2;
 	const unsigned xstride = (margin + fnt.width());
 	const unsigned ystride = (margin + fnt.height());
@@ -73,12 +124,26 @@ main(int argc, const char **argv)
 	int pending = 0;
 	constexpr int pending_threshold = 10;
 
-	char message_buffer[256];
+	j6_handle_t sys = __handle_sys;
+	size_t buffer_size = 0;
+	void *message_buffer = nullptr;
+
 	while (true) {
-		size_t size = sizeof(message_buffer);
-		_syscall_system_get_log(__handle_sys, message_buffer, &size);
-		if (size != 0) {
-			entry *e = reinterpret_cast<entry*>(&message_buffer);
+		size_t size = buffer_size;
+		j6_status_t s = j6_system_get_log(sys, message_buffer, &size);
+
+		if (s == j6_err_insufficient) {
+			free(message_buffer);
+			message_buffer = malloc(size);
+			buffer_size = size;
+			continue;
+		} else if (s != j6_status_ok) {
+			j6_system_log("fb driver got error from get_log, quitting");
+			return s;
+		}
+
+		if (size > 0) {
+			entry *e = reinterpret_cast<entry*>(message_buffer);
 
 			size_t eom = e->bytes - sizeof(entry);
 			e->message[eom] = 0;
@@ -97,9 +162,9 @@ main(int argc, const char **argv)
 			}
 		}
 	}
+	*/
 
-
-	_syscall_system_log("fb driver done, exiting");
+	j6_system_log("fb driver done, exiting");
 	return 0;
 }
 

src/drivers/fb/screen.cpp

@@ -2,13 +2,15 @@
 #include <string.h>
 #include "screen.h"
 
-screen::screen(void *addr, unsigned hres, unsigned vres, pixel_order order) :
-	m_fb(static_cast<pixel_t *>(addr)),
+screen::screen(volatile void *addr, unsigned hres, unsigned vres, unsigned scanline, pixel_order order) :
+	m_fb(static_cast<volatile pixel_t *>(addr)),
 	m_order(order),
+	m_scanline(scanline),
 	m_resx(hres),
 	m_resy(vres)
 {
-	m_back = reinterpret_cast<pixel_t*>(malloc(hres*vres*sizeof(pixel_t)));
+	m_back = reinterpret_cast<pixel_t*>(malloc(scanline*vres*sizeof(pixel_t)));
+	//m_back = const_cast<pixel_t*>(m_fb);
 }
 
 screen::pixel_t
@@ -32,19 +34,25 @@ screen::color(uint8_t r, uint8_t g, uint8_t b) const
 void
 screen::fill(pixel_t color)
 {
-	const size_t len = m_resx * m_resy;
-	for (size_t i = 0; i < len; ++i)
-		m_back[i] = color;
-}
-
-void
-screen::draw_pixel(unsigned x, unsigned y, pixel_t color)
-{
-	m_back[x + y * m_resx] = color;
+	const size_t len = m_scanline * m_resy;
+	asm volatile ( "rep stosl" : :
+		"a"(color), "c"(len), "D"(m_back) );
 }
 
 void
 screen::update()
 {
-	memcpy(m_fb, m_back, m_resx*m_resy*sizeof(pixel_t));
+	/*
+	const size_t len = m_scanline * m_resy * sizeof(pixel_t);
+	asm volatile ( "rep movsb" : :
+		"c"(len), "S"(m_back), "D"(m_fb) );
+	*/
+	const size_t len = m_scanline * m_resy;
+	for (size_t i = 0; i < len; ++i) {
+		m_fb[i] = m_back[i];
+		/*
+		for (int j = 0; j < 10000; ++j)
+			volatile char c = j;
+		*/
+	}
 }

src/drivers/fb/screen.h

@@ -9,7 +9,7 @@ public:
 
 	enum class pixel_order : uint8_t { bgr8, rgb8, };
 
-	screen(void *addr, unsigned hres, unsigned vres, pixel_order order);
+	screen(volatile void *addr, unsigned hres, unsigned vres, unsigned scanline, pixel_order order);
 
 	unsigned width() const { return m_resx; }
 	unsigned height() const { return m_resy; }
@@ -17,13 +17,21 @@ public:
 	pixel_t color(uint8_t r, uint8_t g, uint8_t b) const;
 
 	void fill(pixel_t color);
-	void draw_pixel(unsigned x, unsigned y, pixel_t color);
+
+	inline void draw_pixel(unsigned x, unsigned y, pixel_t color) {
+		const size_t index = x + y * m_scanline;
+		if (x > m_resx || y > m_resy)
+			return;
+		m_back[index] = color;
+	}
 
 	void update();
 
 private:
-	pixel_t *m_fb, *m_back;
+	volatile pixel_t *m_fb;
+	pixel_t *m_back;
 	pixel_order m_order;
+	unsigned m_scanline;
 	unsigned m_resx, m_resy;
 
 	screen() = delete;

src/drivers/fb/scrollback.cpp

@@ -12,10 +12,6 @@ scrollback::scrollback(unsigned lines, unsigned cols, unsigned margin) :
 	m_margin {margin}
 {
 	m_data = reinterpret_cast<char*>(malloc(lines*cols));
-	m_lines = reinterpret_cast<char**>(malloc(lines*sizeof(char*)));
-	for (unsigned i = 0; i < lines; ++i)
-		m_lines[i] = &m_data[i*cols];
-
 	memset(m_data, ' ', lines*cols);
 }
 
@@ -27,15 +23,17 @@ scrollback::add_line(const char *line, size_t len)
 	if (len > m_cols)
 		len = m_cols;
 
-	memcpy(m_lines[i], line, len);
+	char *start = m_data + (i * m_cols);
+	memcpy(start, line, len);
 	if (len < m_cols)
-		memset(m_lines[i]+len, ' ', m_cols - len);
+		memset(start + len, ' ', m_cols - len);
 }
 
 char *
 scrollback::get_line(unsigned i)
 {
-	return m_lines[(i+m_count)%m_rows];
+	unsigned line = (i + m_count) % m_rows;
+	return &m_data[line*m_cols];
 }
 
 void
@@ -48,7 +46,7 @@ scrollback::render(screen &scr, font &fnt)
 	const unsigned ystride = (m_margin + fnt.height());
 
 	for (unsigned y = 0; y < m_rows; ++y) {
-		char *line = get_line(y);
+		char *line = &m_data[y*m_cols];
 		for (unsigned x = 0; x < m_cols; ++x) {
 			fnt.draw_glyph(scr, line[x], fg, bg, m_margin+x*xstride, m_margin+y*ystride);
 		}

src/drivers/fb/scrollback.h

@@ -17,7 +17,6 @@ public:
 
 private:
 	char *m_data;
-	char **m_lines;
 	unsigned m_rows, m_cols;
 	unsigned m_start;
 	unsigned m_count;

src/drivers/nulldrv/main.cpp

@@ -4,8 +4,7 @@
 #include "j6/types.h"
 #include "j6/errors.h"
 #include "j6/signals.h"
-
-#include <j6libc/syscalls.h>
+#include "j6/syscalls.h"
 
 #include "io.h"
 #include "serial.h"
@@ -15,40 +14,39 @@ extern j6_handle_t __handle_sys;
 j6_handle_t endp = j6_handle_invalid;
 
 extern "C" {
-	void _init_libc(j6_process_init *);
 	int main(int, const char **);
 }
 
 void
 thread_proc()
 {
-	_syscall_system_log("sub thread starting");
+	j6_system_log("sub thread starting");
 
 	char buffer[512];
 	size_t len = sizeof(buffer);
 	j6_tag_t tag = 0;
-	j6_status_t result = _syscall_endpoint_receive(endp, &tag, &len, (void*)buffer);
+	j6_status_t result = j6_endpoint_receive(endp, &tag, &len, (void*)buffer);
 	if (result != j6_status_ok)
-		_syscall_thread_exit(result);
+		j6_thread_exit(result);
 
-	_syscall_system_log("sub thread received message");
+	j6_system_log("sub thread received message");
 
 	for (int i = 0; i < len; ++i)
 		if (buffer[i] >= 'A' && buffer[i] <= 'Z')
 			buffer[i] += 0x20;
 
 	tag++;
-	result = _syscall_endpoint_send(endp, tag, len, (void*)buffer);
+	result = j6_endpoint_send(endp, tag, len, (void*)buffer);
 	if (result != j6_status_ok)
-		_syscall_thread_exit(result);
+		j6_thread_exit(result);
 
-	_syscall_system_log("sub thread sent message");
+	j6_system_log("sub thread sent message");
 
 	for (int i = 1; i < 5; ++i)
-		_syscall_thread_sleep(i*10);
+		j6_thread_sleep(i*10);
 
-	_syscall_system_log("sub thread exiting");
-	_syscall_thread_exit(0);
+	j6_system_log("sub thread exiting");
+	j6_thread_exit(0);
 }
 
 int
@@ -57,10 +55,10 @@ main(int argc, const char **argv)
 	j6_handle_t child = j6_handle_invalid;
 	j6_signal_t out = 0;
 
-	_syscall_system_log("main thread starting");
+	j6_system_log("main thread starting");
 
 	for (int i = 0; i < argc; ++i)
-		_syscall_system_log(argv[i]);
+		j6_system_log(argv[i]);
 
 	void *base = malloc(0x1000);
 	if (!base)
@@ -70,43 +68,48 @@ main(int argc, const char **argv)
 	for (int i = 0; i < 3; ++i)
 		vma_ptr[i*100] = uint64_t(i);
 
-	_syscall_system_log("main thread wrote to memory area");
+	j6_system_log("main thread wrote to memory area");
 
-	j6_status_t result = _syscall_endpoint_create(&endp);
+	j6_status_t result = j6_endpoint_create(&endp);
 	if (result != j6_status_ok)
 		return result;
 
-	_syscall_system_log("main thread created endpoint");
+	j6_system_log("main thread created endpoint");
 
-	result = _syscall_thread_create(reinterpret_cast<void*>(&thread_proc), &child);
+	result = j6_thread_create(reinterpret_cast<void*>(&thread_proc), &child);
 	if (result != j6_status_ok)
 		return result;
 
-	_syscall_system_log("main thread created sub thread");
+	j6_system_log("main thread created sub thread");
 
 	char message[] = "MAIN THREAD SUCCESSFULLY CALLED SENDRECV IF THIS IS LOWERCASE";
 	size_t size = sizeof(message);
 	j6_tag_t tag = 16;
-	result = _syscall_endpoint_sendrecv(endp, &tag, &size, (void*)message);
+	result = j6_endpoint_sendrecv(endp, &tag, &size, (void*)message);
 	if (result != j6_status_ok)
 		return result;
 
 	if (tag != 17)
-		_syscall_system_log("GOT WRONG TAG FROM SENDRECV");
+		j6_system_log("GOT WRONG TAG FROM SENDRECV");
 
-	result = _syscall_system_bind_irq(__handle_sys, endp, 3);
+	result = j6_system_bind_irq(__handle_sys, endp, 3);
 	if (result != j6_status_ok)
 		return result;
 
-	_syscall_system_log(message);
+	j6_system_log(message);
 
-	_syscall_system_log("main thread waiting on child");
-
-	result = _syscall_object_wait(child, -1ull, &out);
+	j6_system_log("main thread waiting on child");
+	result = j6_object_wait(child, -1ull, &out);
 	if (result != j6_status_ok)
 		return result;
 
-	_syscall_system_log("main testing irqs");
+	j6_system_log("main thread creating a new process");
+	j6_handle_t child_proc = j6_handle_invalid;
+	result = j6_process_create(&child_proc);
+	if (result != j6_status_ok)
+		return result;
+
+	j6_system_log("main testing irqs");
 
 
 	serial_port com2(COM2);
@@ -120,21 +123,21 @@ main(int argc, const char **argv)
 
 	size_t len = 0;
 	while (true) {
-		result = _syscall_endpoint_receive(endp, &tag, &len, nullptr);
+		result = j6_endpoint_receive(endp, &tag, &len, nullptr);
 		if (result != j6_status_ok)
 			return result;
 
 		if (j6_tag_is_irq(tag))
-			_syscall_system_log("main thread got irq!");
+			j6_system_log("main thread got irq!");
 	}
 
-	_syscall_system_log("main thread closing endpoint");
+	j6_system_log("main thread closing endpoint");
 
-	result = _syscall_object_close(endp);
+	result = j6_object_close(endp);
 	if (result != j6_status_ok)
 		return result;
 
-	_syscall_system_log("main thread done, exiting");
+	j6_system_log("main thread done, exiting");
 	return 0;
 }
 

src/include/j6/flags.h

@@ -0,0 +1,10 @@
+#pragma once
+/// \file flags.h
+/// Enums used as flags for syscalls
+
+enum j6_vm_flags {
+#define VM_FLAG(name, v) j6_vm_flag_ ## name = v,
+#include "j6/tables/vm_flags.inc"
+#undef VM_FLAG
+	j6_vm_flags_MAX
+};

src/include/j6/init.h

@@ -3,25 +3,32 @@
 /// Types used in process and thread initialization
 
 #include <stdint.h>
+#include "j6/types.h"
 
 enum j6_init_type {					// `value` is a:
-	j6_init_handle_system,			// Handle to the system
-	j6_init_handle_process,			// Handle to this process
-	j6_init_handle_thread,			// Handle to this thread
-	j6_init_handle_space,			// Handle to this process' address space
+	j6_init_handle_self,			// Handle to the system
+	j6_init_handle_other,			// Handle to this process
 	j6_init_desc_framebuffer		// Pointer to a j6_init_framebuffer descriptor
 };
 
+struct j6_typed_handle {
+	enum j6_object_type type;
+	j6_handle_t handle;
+};
+
 struct j6_init_value {
 	enum j6_init_type type;
-	uint64_t value;
+	union {
+		struct j6_typed_handle handle;
+		void *data;
+	};
 };
 
 /// Structure defining a framebuffer.
 /// `flags` has the following bits:
 ///   0-3:   Pixel layout. 0000: rgb8, 0001: bgr8
 struct j6_init_framebuffer {
-	void* addr;
+	uintptr_t addr;
 	size_t size;
 	uint32_t vertical;
 	uint32_t horizontal;

src/include/j6/signals.h

@@ -8,9 +8,42 @@
 
 // Signals 16-47 are defined per-object-type
 
-// Process signals
+// Event signals
+#define j7_signal_event00           (1ull << 16)
+#define j7_signal_event01           (1ull << 17)
+#define j7_signal_event02           (1ull << 18)
+#define j7_signal_event03           (1ull << 19)
+#define j7_signal_event04           (1ull << 20)
+#define j7_signal_event05           (1ull << 21)
+#define j7_signal_event06           (1ull << 22)
+#define j7_signal_event07           (1ull << 23)
+#define j7_signal_event08           (1ull << 24)
+#define j7_signal_event09           (1ull << 25)
+#define j7_signal_event10           (1ull << 26)
+#define j7_signal_event11           (1ull << 27)
+#define j7_signal_event12           (1ull << 28)
+#define j7_signal_event13           (1ull << 29)
+#define j7_signal_event14           (1ull << 30)
+#define j7_signal_event15           (1ull << 31)
+#define j7_signal_event16           (1ull << 32)
+#define j7_signal_event17           (1ull << 33)
+#define j7_signal_event18           (1ull << 34)
+#define j7_signal_event19           (1ull << 35)
+#define j7_signal_event20           (1ull << 36)
+#define j7_signal_event21           (1ull << 37)
+#define j7_signal_event22           (1ull << 38)
+#define j7_signal_event23           (1ull << 39)
+#define j7_signal_event24           (1ull << 40)
+#define j7_signal_event25           (1ull << 41)
+#define j7_signal_event26           (1ull << 42)
+#define j7_signal_event27           (1ull << 43)
+#define j7_signal_event28           (1ull << 44)
+#define j7_signal_event29           (1ull << 45)
+#define j7_signal_event30           (1ull << 46)
+#define j7_signal_event31           (1ull << 47)
 
-// Thread signals
+// System signals
+#define j6_signal_system_has_log	(1ull << 16)
 
 // Channel signals
 #define j6_signal_channel_can_send	(1ull << 16)

src/include/j6/tables/log_areas.inc

@@ -1,13 +1,14 @@
 LOG(apic,   info);
 LOG(device, debug);
-LOG(paging, warn);
+LOG(paging, info);
 LOG(driver, info);
-LOG(memory, info);
+LOG(memory, debug);
 LOG(fs,     info);
-LOG(task,   debug);
-LOG(loader, info);
+LOG(task,   info);
+LOG(sched,  info);
+LOG(loader, debug);
 LOG(boot,   debug);
-LOG(syscall,debug);
+LOG(syscall,info);
 LOG(vmem,   debug);
 LOG(objs,   debug);
 LOG(timer,  debug);

src/include/j6/tables/object_types.inc

@@ -0,0 +1,12 @@
+OBJECT_TYPE( none,     0x00 )
+
+OBJECT_TYPE( system,   0x01 )
+
+OBJECT_TYPE( event,    0x02 )
+OBJECT_TYPE( channel,  0x03 )
+OBJECT_TYPE( endpoint, 0x04 )
+
+OBJECT_TYPE( vma,      0x05 )
+
+OBJECT_TYPE( process,  0x06 )
+OBJECT_TYPE( thread,   0x07 )

src/include/j6/tables/syscalls.inc

@@ -1,17 +1,21 @@
 SYSCALL(0x00, system_log,        const char *)
 SYSCALL(0x01, system_noop,       void)
-SYSCALL(0x02, system_get_log,    j6_handle_t, char *, size_t *)
+SYSCALL(0x02, system_get_log,    j6_handle_t, void *, size_t *)
 SYSCALL(0x03, system_bind_irq,   j6_handle_t, j6_handle_t, unsigned)
+SYSCALL(0x04, system_map_mmio,   j6_handle_t, j6_handle_t *, uintptr_t, size_t, uint32_t)
 
 SYSCALL(0x08, object_koid,       j6_handle_t, j6_koid_t *)
 SYSCALL(0x09, object_wait,       j6_handle_t, j6_signal_t, j6_signal_t *)
 SYSCALL(0x0a, object_signal,     j6_handle_t, j6_signal_t)
 SYSCALL(0x0b, object_close,      j6_handle_t)
 
-SYSCALL(0x10, process_exit,      int64_t)
+SYSCALL(0x10, process_create,    j6_handle_t *)
+SYSCALL(0x11, process_start,     j6_handle_t, uintptr_t, j6_handle_t *, size_t)
+SYSCALL(0x11, process_kill,      j6_handle_t)
+SYSCALL(0x17, process_exit,      int32_t)
 
 SYSCALL(0x18, thread_create,     void *, j6_handle_t *)
-SYSCALL(0x19, thread_exit,       int64_t)
+SYSCALL(0x19, thread_exit,       int32_t)
 SYSCALL(0x1a, thread_pause,      void)
 SYSCALL(0x1b, thread_sleep,      uint64_t)
 
@@ -26,6 +30,6 @@ SYSCALL(0x2b, endpoint_sendrecv, j6_handle_t, j6_tag_t *, size_t *, void *)
 
 SYSCALL(0x30, vma_create,        j6_handle_t *, size_t, uint32_t)
 SYSCALL(0x31, vma_create_map,    j6_handle_t *, size_t, uintptr_t, uint32_t)
-SYSCALL(0x32, vma_map,           j6_handle_t, uintptr_t)
-SYSCALL(0x33, vma_unmap,         j6_handle_t)
+SYSCALL(0x32, vma_map,           j6_handle_t, j6_handle_t, uintptr_t)
+SYSCALL(0x33, vma_unmap,         j6_handle_t, j6_handle_t)
 SYSCALL(0x34, vma_resize,        j6_handle_t, size_t *)

src/include/j6/tables/vm_flags.inc

@@ -0,0 +1,9 @@
+VM_FLAG( none,            0x00000000)
+VM_FLAG( write,           0x00000001)
+VM_FLAG( exec,            0x00000002)
+VM_FLAG( zero,            0x00000010)
+VM_FLAG( contiguous,      0x00000020)
+VM_FLAG( large_pages,     0x00000100)
+VM_FLAG( huge_pages,      0x00000200)
+VM_FLAG( write_combine,   0x00001000)
+VM_FLAG( mmio,            0x00010000)

src/include/j6/types.h

@@ -36,15 +36,10 @@ typedef uint64_t j6_handle_t;
 #define j6_handle_id_mask 0xffffffffull
 #define j6_handle_invalid 0xffffffffull
 
-/// A process' initial data structure for communicating with the system
-struct j6_process_init
-{
-	j6_handle_t process;
-	j6_handle_t handles[3];
-};
+enum j6_object_type {
+#define OBJECT_TYPE( name, val ) j6_object_type_ ## name = val,
+#include "j6/tables/object_types.inc"
+#undef OBJECT_TYPE
 
-/// A thread's initial data structure
-struct j6_thread_init
-{
-	j6_handle_t thread;
+	j6_object_type_max
 };

src/include/kernel_args.h

@@ -35,11 +35,26 @@ struct framebuffer {
 	fb_type type;
 };
 
-struct program {
+enum class section_flags : uint32_t {
+	none    = 0,
+	execute = 1,
+	write   = 2,
+	read    = 4,
+};
+
+struct program_section {
 	uintptr_t phys_addr;
 	uintptr_t virt_addr;
+	uint32_t size;
+	section_flags type;
+};
+
+struct program {
 	uintptr_t entrypoint;
-	size_t size;
+	uintptr_t base;
+	size_t total_size;
+	size_t num_sections;
+	program_section sections[3];
 };
 
 enum class mem_type : uint32_t {
@@ -60,6 +75,18 @@ struct mem_entry
 	uint32_t attr;
 };
 
+constexpr size_t frames_per_block = 64 * 64 * 64;
+
+struct frame_block
+{
+	uintptr_t base;
+	uint32_t count;
+	uint32_t attrs;
+	uint64_t map1;
+	uint64_t map2[64];
+	uint64_t *bitmap;
+};
+
 enum class boot_flags : uint16_t {
 	none  = 0x0000,
 	debug = 0x0001
@@ -73,6 +100,7 @@ struct header {
 	void *pml4;
 	void *page_tables;
 	size_t table_count;
+	size_t table_pages;
 
 	program *programs;
 	size_t num_programs;
@@ -83,6 +111,10 @@ struct header {
 	mem_entry *mem_map;
 	size_t map_count;
 
+	frame_block *frame_blocks;
+	size_t frame_block_count;
+	size_t frame_block_pages;
+
 	void *runtime_services;
 	void *acpi_table;
 

src/include/kernel_memory.h

@@ -17,7 +17,7 @@ namespace memory {
 	constexpr uintptr_t page_offset = 0xffffc00000000000ull;
 
 	/// Max number of pages for a kernel stack
-	constexpr unsigned kernel_stack_pages = 4;
+	constexpr unsigned kernel_stack_pages = 2;
 
 	/// Max number of pages for a kernel buffer
 	constexpr unsigned kernel_buffer_pages = 16;
@@ -35,11 +35,17 @@ namespace memory {
 	constexpr uintptr_t stacks_start = heap_start - kernel_max_stacks;
 
 	/// Max size of kernel buffers area
-	constexpr size_t kernel_max_buffers = 0x10000000000ull; // 1TiB
+	constexpr size_t kernel_max_buffers = 0x8000000000ull; // 512GiB
 
 	/// Start of kernel buffers
 	constexpr uintptr_t buffers_start = stacks_start - kernel_max_buffers;
 
+	/// Max size of kernel bitmap area
+	constexpr size_t kernel_max_bitmap = 0x8000000000ull; // 512GiB
+
+	/// Start of kernel bitmap
+	constexpr uintptr_t bitmap_start = buffers_start - kernel_max_bitmap;
+
 	/// First kernel space PML4 entry
 	constexpr unsigned pml4e_kernel = 256;
 
@@ -58,6 +64,11 @@ namespace memory {
 		return reinterpret_cast<T*>(a|page_offset);
 	}
 
+	/// Convert a physical address to a virtual one (in the offset-mapped area)
+	template <typename T> T * to_virtual(T *p) {
+		return to_virtual<T>(reinterpret_cast<uintptr_t>(p));
+	}
+
 	/// Get the number of pages needed for a given number of bytes.
 	/// \arg bytes  The number of bytes desired
 	/// \returns    The number of pages needed to contain the desired bytes

src/kernel/acpi_tables.h

@@ -23,7 +23,7 @@ struct acpi_table_header
 } __attribute__ ((packed));
 
 #define TABLE_HEADER(signature) \
-	static const uint32_t type_id = kutil::byteswap(signature); \
+	static constexpr uint32_t type_id = kutil::byteswap(signature); \
 	acpi_table_header header;
 
 
@@ -198,3 +198,14 @@ struct acpi_hpet
 	uint8_t attributes;
 } __attribute__ ((packed));
 
+struct acpi_bgrt
+{
+	TABLE_HEADER('BGRT');
+	uint16_t version;
+	uint8_t status;
+	uint8_t type;
+	uintptr_t address;
+	uint32_t offset_x;
+	uint32_t offset_y;
+} __attribute__ ((packed));
+

src/kernel/apic.cpp

@@ -68,7 +68,7 @@ lapic::calibrate_timer()
 	set_divisor(1);
 	apic_write(m_base, lapic_timer_init, initial);
 
-	uint64_t us = 200000;
+	uint64_t us = 20000;
 	clock::get().spinwait(us);
 
 	uint32_t remaining = apic_read(m_base, lapic_timer_cur);

src/kernel/clock.cpp

@@ -16,7 +16,7 @@ clock::clock(uint64_t rate, clock::source source_func, void *data) :
 void
 clock::spinwait(uint64_t us) const
 {
-	uint64_t when = m_source(m_data) + us;
+	uint64_t when = value() + us;
 	while (value() < when);
 }
 

src/kernel/device_manager.cpp

@@ -38,7 +38,7 @@ struct acpi2_rsdp
 	uint32_t rsdt_address;
 
 	uint32_t length;
-	uint64_t xsdt_address;
+	acpi_table_header *xsdt_address;
 	uint8_t checksum20;
 	uint8_t reserved[3];
 } __attribute__ ((packed));
@@ -63,7 +63,7 @@ void irq4_callback(void *)
 
 
 device_manager::device_manager() :
-	m_lapic(0)
+	m_lapic(nullptr)
 {
 	m_irqs.ensure_capacity(32);
 	m_irqs.set_size(16);
@@ -73,13 +73,20 @@ device_manager::device_manager() :
 	m_irqs[2] = ignore_endpoint;
 }
 
+template <typename T> static const T *
+check_get_table(const acpi_table_header *header)
+{
+	kassert(header && header->validate(T::type_id), "Invalid ACPI table.");
+	return reinterpret_cast<const T *>(header);
+}
+
 void
 device_manager::parse_acpi(const void *root_table)
 {
 	kassert(root_table != 0, "ACPI root table pointer is null.");
 
-	const acpi1_rsdp *acpi1 =
-		reinterpret_cast<const acpi1_rsdp *>(root_table);
+	const acpi1_rsdp *acpi1 = memory::to_virtual(
+		reinterpret_cast<const acpi1_rsdp *>(root_table));
 
 	for (int i = 0; i < sizeof(acpi1->signature); ++i)
 		kassert(acpi1->signature[i] == expected_signature[i],
@@ -96,7 +103,7 @@ device_manager::parse_acpi(const void *root_table)
 	sum = kutil::checksum(acpi2, sizeof(acpi2_rsdp), sizeof(acpi1_rsdp));
 	kassert(sum == 0, "ACPI 2.0 RSDP checksum mismatch.");
 
-	load_xsdt(reinterpret_cast<const acpi_xsdt *>(acpi2->xsdt_address));
+	load_xsdt(memory::to_virtual(acpi2->xsdt_address));
 }
 
 ioapic *
@@ -112,9 +119,9 @@ put_sig(char *into, uint32_t type)
 }
 
 void
-device_manager::load_xsdt(const acpi_xsdt *xsdt)
+device_manager::load_xsdt(const acpi_table_header *header)
 {
-	kassert(xsdt && acpi_validate(xsdt), "Invalid ACPI XSDT.");
+	const auto *xsdt = check_get_table<acpi_xsdt>(header);
 
 	char sig[5] = {0,0,0,0,0};
 	log::info(logs::device, "ACPI 2.0+ tables loading");
@@ -124,7 +131,8 @@ device_manager::load_xsdt(const acpi_xsdt *xsdt)
 
 	size_t num_tables = acpi_table_entries(xsdt, sizeof(void*));
 	for (size_t i = 0; i < num_tables; ++i) {
-		const acpi_table_header *header = xsdt->headers[i];
+		const acpi_table_header *header =
+			memory::to_virtual(xsdt->headers[i]);
 
 		put_sig(sig, header->type);
 		log::debug(logs::device, "  Found table %s", sig);
@@ -133,15 +141,15 @@ device_manager::load_xsdt(const acpi_xsdt *xsdt)
 
 		switch (header->type) {
 		case acpi_apic::type_id:
-			load_apic(reinterpret_cast<const acpi_apic *>(header));
+			load_apic(header);
 			break;
 
 		case acpi_mcfg::type_id:
-			load_mcfg(reinterpret_cast<const acpi_mcfg *>(header));
+			load_mcfg(header);
 			break;
 
 		case acpi_hpet::type_id:
-			load_hpet(reinterpret_cast<const acpi_hpet *>(header));
+			load_hpet(header);
 			break;
 
 		default:
@@ -151,8 +159,10 @@ device_manager::load_xsdt(const acpi_xsdt *xsdt)
 }
 
 void
-device_manager::load_apic(const acpi_apic *apic)
+device_manager::load_apic(const acpi_table_header *header)
 {
+	const auto *apic = check_get_table<acpi_apic>(header);
+
 	uintptr_t local = apic->local_address;
 	m_lapic = new lapic(local, isr::isrSpurious);
 
@@ -236,19 +246,23 @@ device_manager::load_apic(const acpi_apic *apic)
 		p += length;
 	}
 
+	/*
 	for (uint8_t i = 0; i < m_ioapics[0].get_num_gsi(); ++i) {
 		switch (i) {
 			case 2: break;
 			default: m_ioapics[0].mask(i, false);
 		}
 	}
+	*/
 
 	m_lapic->enable();
 }
 
 void
-device_manager::load_mcfg(const acpi_mcfg *mcfg)
+device_manager::load_mcfg(const acpi_table_header *header)
 {
+	const auto *mcfg = check_get_table<acpi_mcfg>(header);
+
 	size_t count = acpi_table_entries(mcfg, sizeof(acpi_mcfg_entry));
 	m_pci.set_size(count);
 	m_devices.set_capacity(16);
@@ -269,8 +283,10 @@ device_manager::load_mcfg(const acpi_mcfg *mcfg)
 }
 
 void
-device_manager::load_hpet(const acpi_hpet *hpet)
+device_manager::load_hpet(const acpi_table_header *header)
 {
+	const auto *hpet = check_get_table<acpi_hpet>(header);
+
 	log::debug(logs::device, "  Found HPET device #%3d: base %016lx  pmin %d  attr %02x",
 			hpet->index, hpet->base_address.address, hpet->periodic_min, hpet->attributes);
 
@@ -311,6 +327,13 @@ device_manager::probe_pci()
 	}
 }
 
+static uint64_t
+fake_clock_source(void*)
+{
+	static uint64_t value = 0;
+	return value++;
+}
+
 void
 device_manager::init_drivers()
 {
@@ -329,18 +352,20 @@ device_manager::init_drivers()
 		ahcid.register_device(&device);
 	}
 	*/
+	clock *master_clock = nullptr;
 	if (m_hpets.count() > 0) {
 		hpet &h = m_hpets[0];
 		h.enable();
 
 		// becomes the singleton
-		clock *master_clock = new clock(h.rate(), hpet_clock_source, &h);
-		kassert(master_clock, "Failed to allocate master clock");
+		master_clock = new clock(h.rate(), hpet_clock_source, &h);
 		log::info(logs::clock, "Created master clock using HPET 0: Rate %d", h.rate());
 	} else {
-		//TODO: APIC clock?
-		kassert(0, "No HPET master clock");
+		//TODO: Other clocks, APIC clock?
+		master_clock = new clock(5000, fake_clock_source, nullptr);
 	}
+
+	kassert(master_clock, "Failed to allocate master clock");
 }
 
 bool

src/kernel/device_manager.h

@@ -6,10 +6,7 @@
 #include "hpet.h"
 #include "pci.h"
 
-struct acpi_xsdt;
-struct acpi_apic;
-struct acpi_mcfg;
-struct acpi_hpet;
+struct acpi_table_header;
 class block_device;
 class endpoint;
 
@@ -100,19 +97,19 @@ public:
 private:
 	/// Parse the ACPI XSDT and load relevant sub-tables.
 	/// \arg xsdt  Pointer to the XSDT from the firmware
-	void load_xsdt(const acpi_xsdt *xsdt);
+	void load_xsdt(const acpi_table_header *xsdt);
 
 	/// Parse the ACPI MADT and initialize APICs from it.
 	/// \arg apic  Pointer to the MADT from the XSDT
-	void load_apic(const acpi_apic *apic);
+	void load_apic(const acpi_table_header *apic);
 
 	/// Parse the ACPI MCFG and initialize PCIe from it.
 	/// \arg mcfg  Pointer to the MCFG from the XSDT
-	void load_mcfg(const acpi_mcfg *mcfg);
+	void load_mcfg(const acpi_table_header *mcfg);
 
 	/// Parse the ACPI HPET and initialize an HPET from it.
 	/// \arg hpet  Pointer to the HPET from the XSDT
-	void load_hpet(const acpi_hpet *hpet);
+	void load_hpet(const acpi_table_header *hpet);
 
 	/// Probe the PCIe busses and add found devices to our
 	/// device list. The device list is destroyed and rebuilt.

src/kernel/frame_allocator.cpp

@@ -2,20 +2,11 @@
 #include "kutil/assert.h"
 #include "kutil/memory.h"
 #include "frame_allocator.h"
+#include "kernel_args.h"
+#include "kernel_memory.h"
+#include "log.h"
 
 using memory::frame_size;
-using memory::page_offset;
-using frame_block_node = kutil::list_node<frame_block>;
-
-int
-frame_block::compare(const frame_block &rhs) const
-{
-	if (address < rhs.address)
-		return -1;
-	else if (address > rhs.address)
-		return 1;
-	return 0;
-}
 
 
 frame_allocator &
@@ -25,54 +16,142 @@ frame_allocator::get()
 	return g_frame_allocator;
 }
 
-frame_allocator::frame_allocator() {}
+frame_allocator::frame_allocator(kernel::args::frame_block *frames, size_t count) :
+	m_blocks(frames),
+	m_count(count)
+{
+}
+
+inline unsigned
+bsf(uint64_t v)
+{
+	asm ("tzcntq %q0, %q1" : "=r"(v) : "r"(v) : "cc");
+	return v;
+}
 
 size_t
 frame_allocator::allocate(size_t count, uintptr_t *address)
 {
-	kassert(!m_free.empty(), "frame_allocator::pop_frames ran out of free frames!");
-	if (m_free.empty())
-		return 0;
+	for (long i = m_count - 1; i >= 0; ++i) {
+		frame_block &block = m_blocks[i];
 
-	auto *first = m_free.front();
+		if (!block.map1)
+			continue;
 
-	if (count >= first->count) {
-		*address = first->address;
-		m_free.remove(first);
-		return first->count;
-	} else {
-		first->count -= count;
-		*address = first->address + (first->count * frame_size);
-		return count;
+		// Tree walk to find the first available page
+		unsigned o1 = bsf(block.map1);
+
+		uint64_t m2 = block.map2[o1];
+		unsigned o2 = bsf(m2);
+
+		uint64_t m3 = block.bitmap[(o1 << 6) + o2];
+		unsigned o3 = bsf(m3);
+
+		unsigned frame = (o1 << 12) + (o2 << 6) + o3;
+
+		// See how many contiguous pages are here
+		unsigned n = bsf(~m3 >> o3);
+		if (n > count)
+			n = count;
+
+		*address = block.base + frame * frame_size;
+
+		// Clear the bits to mark these pages allocated
+		m3 &= ~(((1 << n) - 1) << o3);
+		block.bitmap[(o1 << 6) + o2] = m3;
+		if (!m3) {
+			// if that was it for this group, clear the next level bit
+			m2 &= ~(1 << o2);
+			block.map2[o1] = m2;
+
+			if (!m2) {
+				// if that was cleared too, update the top level
+				block.map1 &= ~(1 << o1);
+			}
+		}
+
+		return n;
 	}
-}
 
-inline uintptr_t end(frame_block *node) { return node->address + node->count * frame_size; }
+	kassert(false, "frame_allocator ran out of free frames!");
+	return 0;
+}
 
 void
 frame_allocator::free(uintptr_t address, size_t count)
 {
 	kassert(address % frame_size == 0, "Trying to free a non page-aligned frame!");
 
-	frame_block_node *node =
-		reinterpret_cast<frame_block_node*>(address + page_offset);
+	if (!count)
+		return;
 
-	kutil::memset(node, 0, sizeof(frame_block_node));
-	node->address = address;
-	node->count = count;
+	for (long i = 0; i < m_count; ++i) {
+		frame_block &block = m_blocks[i];
+		uintptr_t end = block.base + block.count * frame_size;
 
-	m_free.sorted_insert(node);
+		if (address < block.base || address >= end)
+			continue;
 
-	frame_block_node *next = node->next();
-	if (next && end(node) == next->address) {
-		node->count += next->count;
-		m_free.remove(next);
-	}
+		uint64_t frame = (address - block.base) >> 12;
+		unsigned o1 = (frame >> 12) & 0x3f;
+		unsigned o2 = (frame >> 6) & 0x3f;
+		unsigned o3 = frame & 0x3f;
 
-	frame_block_node *prev = node->prev();
-	if (prev && end(prev) == address) {
-		prev->count += node->count;
-		m_free.remove(node);
+		while (count--) {
+			block.map1 |= (1 << o1);
+			block.map2[o1] |= (1 << o2);
+			block.bitmap[o2] |= (1 << o3);
+			if (++o3 == 64) {
+				o3 = 0;
+				if (++o2 == 64) {
+					o2 = 0;
+					++o1;
+					kassert(o1 < 64, "Tried to free pages past the end of a block");
+				}
+			}
+		}
+	}
+}
+
+void
+frame_allocator::used(uintptr_t address, size_t count)
+{
+	kassert(address % frame_size == 0, "Trying to mark a non page-aligned frame!");
+
+	if (!count)
+		return;
+
+	for (long i = 0; i < m_count; ++i) {
+		frame_block &block = m_blocks[i];
+		uintptr_t end = block.base + block.count * frame_size;
+
+		if (address < block.base || address >= end)
+			continue;
+
+		uint64_t frame = (address - block.base) >> 12;
+		unsigned o1 = (frame >> 12) & 0x3f;
+		unsigned o2 = (frame >> 6) & 0x3f;
+		unsigned o3 = frame & 0x3f;
+
+		while (count--) {
+			block.bitmap[o2] &= ~(1 << o3);
+			if (!block.bitmap[o2]) {
+				block.map2[o1] &= ~(1 << o2);
+
+				if (!block.map2[o1]) {
+					block.map1 &= ~(1 << o1);
+				}
+			}
+
+			if (++o3 == 64) {
+				o3 = 0;
+				if (++o2 == 64) {
+					o2 = 0;
+					++o1;
+					kassert(o1 < 64, "Tried to mark pages past the end of a block");
+				}
+			}
+		}
 	}
 }
 

src/kernel/frame_allocator.h

@@ -4,17 +4,21 @@
 
 #include <stdint.h>
 
-#include "kutil/linked_list.h"
-
-struct frame_block;
-using frame_block_list = kutil::linked_list<frame_block>;
+namespace kernel {
+namespace args {
+	struct frame_block;
+}}
 
 /// Allocator for physical memory frames
 class frame_allocator
 {
 public:
-	/// Default constructor
-	frame_allocator();
+	using frame_block = kernel::args::frame_block;
+
+	/// Constructor
+	/// \arg blocks The bootloader-supplied frame bitmap block list
+	/// \arg count  Number of entries in the block list
+	frame_allocator(frame_block *frames, size_t count);
 
 	/// Get free frames from the free list. Only frames from the first free block
 	/// are returned, so the number may be less than requested, but they will
@@ -29,26 +33,18 @@ public:
 	/// \arg count    The number of frames to be freed
 	void free(uintptr_t address, size_t count);
 
+	/// Mark frames as used
+	/// \arg address  The physical address of the first frame to free
+	/// \arg count    The number of frames to be freed
+	void used(uintptr_t address, size_t count);
+
 	/// Get the global frame allocator
 	static frame_allocator & get();
 
 private:
-	frame_block_list m_free; ///< Free frames list
+	frame_block *m_blocks;
+	long m_count;
 
+	frame_allocator() = delete;
 	frame_allocator(const frame_allocator &) = delete;
 };
-
-
-/// A block of contiguous frames. Each `frame_block` represents contiguous
-/// physical frames with the same attributes.
-struct frame_block
-{
-	uintptr_t address;
-	uint32_t count;
-
-	/// Compare two blocks by address.
-	/// \arg rhs   The right-hand comparator
-	/// \returns   <0 if this is sorts earlier, >0 if this sorts later, 0 for equal
-	int compare(const frame_block &rhs) const;
-};
-

src/kernel/gdt.cpp

@@ -4,6 +4,7 @@
 #include "kutil/enum_bitfields.h"
 #include "kutil/memory.h"
 #include "console.h"
+#include "kernel_memory.h"
 #include "log.h"
 
 
@@ -136,19 +137,55 @@ idt_set_entry(uint8_t i, uint64_t addr, uint16_t selector, uint8_t flags)
 }
 
 void
-tss_set_stack(int ring, uintptr_t rsp)
+tss_set_stack(unsigned ring, uintptr_t rsp)
 {
 	kassert(ring < 3, "Bad ring passed to tss_set_stack.");
 	g_tss.rsp[ring] = rsp;
 }
 
 uintptr_t
-tss_get_stack(int ring)
+tss_get_stack(unsigned ring)
 {
 	kassert(ring < 3, "Bad ring passed to tss_get_stack.");
 	return g_tss.rsp[ring];
 }
 
+void
+idt_set_ist(unsigned i, unsigned ist)
+{
+	g_idt_table[i].ist = ist;
+}
+
+void
+tss_set_ist(unsigned ist, uintptr_t rsp)
+{
+	kassert(ist > 0 && ist < 7, "Bad ist passed to tss_set_ist.");
+	g_tss.ist[ist] = rsp;
+}
+
+void
+ist_increment(unsigned i)
+{
+	uint8_t ist = g_idt_table[i].ist;
+	if (ist)
+		g_tss.ist[ist] += memory::frame_size;
+}
+
+void
+ist_decrement(unsigned i)
+{
+	uint8_t ist = g_idt_table[i].ist;
+	if (ist)
+		g_tss.ist[ist] -= memory::frame_size;
+}
+
+uintptr_t
+tss_get_ist(unsigned ist)
+{
+	kassert(ist > 0 && ist < 7, "Bad ist passed to tss_get_ist.");
+	return g_tss.ist[ist];
+}
+
 void
 gdt_init()
 {
@@ -184,14 +221,14 @@ gdt_init()
 }
 
 void
-gdt_dump(int index)
+gdt_dump(unsigned index)
 {
 	const table_ptr &table = g_gdtr;
 
 	console *cons = console::get();
 
-	int start = 0;
-	int count = (table.limit + 1) / sizeof(gdt_descriptor);
+	unsigned start = 0;
+	unsigned count = (table.limit + 1) / sizeof(gdt_descriptor);
 	if (index != -1) {
 		start = index;
 		count = 1;
@@ -240,13 +277,13 @@ gdt_dump(int index)
 }
 
 void
-idt_dump(int index)
+idt_dump(unsigned index)
 {
 	const table_ptr &table = g_idtr;
 
 
-	int start = 0;
-	int count = (table.limit + 1) / sizeof(idt_descriptor);
+	unsigned start = 0;
+	unsigned count = (table.limit + 1) / sizeof(idt_descriptor);
 	if (index != -1) {
 		start = index;
 		count = 1;

src/kernel/gdt.h

@@ -17,17 +17,42 @@ void idt_set_entry(uint8_t i, uint64_t addr, uint16_t selector, uint8_t flags);
 /// Set the stack pointer for a given ring in the TSS
 /// \arg ring  Ring to set for (0-2)
 /// \arg rsp   Stack pointer to set
-void tss_set_stack(int ring, uintptr_t rsp);
+void tss_set_stack(unsigned ring, uintptr_t rsp);
 
 /// Get the stack pointer for a given ring in the TSS
 /// \arg ring  Ring to get (0-2)
 /// \returns   Stack pointers for that ring
-uintptr_t tss_get_stack(int ring);
+uintptr_t tss_get_stack(unsigned ring);
+
+/// Set the given IDT entry to use the given IST entry
+/// \arg i     Which IDT entry to set
+/// \arg ist   Which IST entry to set (1-7)
+void idt_set_ist(unsigned i, unsigned ist);
+
+/// Set the stack pointer for a given IST in the TSS
+/// \arg ist   Which IST entry to set (1-7)
+/// \arg rsp   Stack pointer to set
+void tss_set_ist(unsigned ist, uintptr_t rsp);
+
+/// Increment the stack pointer for the given vector,
+/// if it's using an IST entry
+/// \arg i     Which IDT entry to use
+void ist_increment(unsigned i);
+
+/// Decrement the stack pointer for the given vector,
+/// if it's using an IST entry
+/// \arg i     Which IDT entry to use
+void ist_decrement(unsigned i);
+
+/// Get the stack pointer for a given IST in the TSS
+/// \arg ring  Which IST entry to get (1-7)
+/// \returns   Stack pointers for that IST entry
+uintptr_t tss_get_ist(unsigned ist);
 
 /// Dump information about the current GDT to the screen
 /// \arg index  Which entry to print, or -1 for all entries
-void gdt_dump(int index = -1);
+void gdt_dump(unsigned index = -1);
 
 /// Dump information about the current IDT to the screen
 /// \arg index  Which entry to print, or -1 for all entries
-void idt_dump(int index = -1);
+void idt_dump(unsigned index = -1);

src/kernel/interrupt_isrs.inc

@@ -1,36 +1,36 @@
-ISR (0x00, isrDivideByZero)
-ISR (0x01, isrDebug)
-ISR (0x02, isrNMI)
-ISR (0x03, isrBreakpoint)
-ISR (0x04, isrOverflow)
-ISR (0x05, isrBRE)
-ISR (0x06, isrInvalidOp)
-ISR (0x07, isrDNA)
-EISR(0x08, isrDoubleFault)
-ISR (0x09, isrCoprocessor)
-EISR(0x0a, isrInvalidTSS)
-EISR(0x0b, isrSegmentNP)
-EISR(0x0c, isrSSFault)
-EISR(0x0d, isrGPFault)
-EISR(0x0e, isrPageFault)
-ISR (0x0f, isr15)
+ISR (0x00, 0, isrDivideByZero)
+ISR (0x01, 0, isrDebug)
+ISR (0x02, 1, isrNMI)
+ISR (0x03, 0, isrBreakpoint)
+ISR (0x04, 0, isrOverflow)
+ISR (0x05, 0, isrBRE)
+ISR (0x06, 0, isrInvalidOp)
+ISR (0x07, 0, isrDNA)
+EISR(0x08, 2, isrDoubleFault)
+ISR (0x09, 0, isrCoprocessor)
+EISR(0x0a, 0, isrInvalidTSS)
+EISR(0x0b, 0, isrSegmentNP)
+EISR(0x0c, 0, isrSSFault)
+EISR(0x0d, 0, isrGPFault)
+EISR(0x0e, 3, isrPageFault)
+ISR (0x0f, 0, isr15)
 
-ISR (0x10, isrX87FPE)
-ISR (0x11, isrAlignmentChk)
-ISR (0x12, isrMachineChk)
-ISR (0x13, isrSIMDFPE)
-ISR (0x14, isrVirt)
-ISR (0x15, isr21)
-ISR (0x16, isr22)
-ISR (0x17, isr23)
-ISR (0x18, isr24)
-ISR (0x19, isr25)
-ISR (0x1a, isr26)
-ISR (0x1b, isr27)
-ISR (0x1c, isr28)
-ISR (0x1d, isr29)
-ISR (0x1e, isrSecurity)
-ISR (0x1f, isr31)
+ISR (0x10, 0, isrX87FPE)
+ISR (0x11, 0, isrAlignmentChk)
+ISR (0x12, 0, isrMachineChk)
+ISR (0x13, 0, isrSIMDFPE)
+ISR (0x14, 0, isrVirt)
+ISR (0x15, 0, isr21)
+ISR (0x16, 0, isr22)
+ISR (0x17, 0, isr23)
+ISR (0x18, 0, isr24)
+ISR (0x19, 0, isr25)
+ISR (0x1a, 0, isr26)
+ISR (0x1b, 0, isr27)
+ISR (0x1c, 0, isr28)
+ISR (0x1d, 0, isr29)
+ISR (0x1e, 0, isrSecurity)
+ISR (0x1f, 0, isr31)
 
 IRQ (0x20, 0x00, irq00)
 IRQ (0x21, 0x01, irq01)
@@ -237,28 +237,27 @@ IRQ (0xde, 0xbe, irqBE)
 IRQ (0xdf, 0xbf, irqBF)
 
 
-ISR (0xe0, isrTimer)
-ISR (0xe1, isrLINT0)
-ISR (0xe2, isrLINT1)
-ISR (0xe4, isrAssert)
+ISR (0xe0, 0, isrTimer)
+ISR (0xe1, 0, isrLINT0)
+ISR (0xe2, 0, isrLINT1)
+ISR (0xe4, 0, isrAssert)
 
-UISR(0xee, isrSyscall)
-ISR (0xef, isrSpurious)
+ISR (0xef, 0, isrSpurious)
 
-ISR (0xf0, isrIgnore0)
-ISR (0xf1, isrIgnore1)
-ISR (0xf2, isrIgnore2)
-ISR (0xf3, isrIgnore3)
-ISR (0xf4, isrIgnore4)
-ISR (0xf5, isrIgnore5)
-ISR (0xf6, isrIgnore6)
-ISR (0xf7, isrIgnore7)
+ISR (0xf0, 0, isrIgnore0)
+ISR (0xf1, 0, isrIgnore1)
+ISR (0xf2, 0, isrIgnore2)
+ISR (0xf3, 0, isrIgnore3)
+ISR (0xf4, 0, isrIgnore4)
+ISR (0xf5, 0, isrIgnore5)
+ISR (0xf6, 0, isrIgnore6)
+ISR (0xf7, 0, isrIgnore7)
 
-ISR (0xf8, isrIgnore8)
-ISR (0xf9, isrIgnore9)
-ISR (0xfa, isrIgnoreA)
-ISR (0xfb, isrIgnoreB)
-ISR (0xfc, isrIgnoreC)
-ISR (0xfd, isrIgnoreD)
-ISR (0xfe, isrIgnoreE)
-ISR (0xff, isrIgnoreF)
+ISR (0xf8, 0, isrIgnore8)
+ISR (0xf9, 0, isrIgnore9)
+ISR (0xfa, 0, isrIgnoreA)
+ISR (0xfb, 0, isrIgnoreB)
+ISR (0xfc, 0, isrIgnoreC)
+ISR (0xfd, 0, isrIgnoreD)
+ISR (0xfe, 0, isrIgnoreE)
+ISR (0xff, 0, isrIgnoreF)

src/kernel/interrupts.cpp

@@ -28,13 +28,11 @@ extern "C" {
 	void isr_handler(cpu_state*);
 	void irq_handler(cpu_state*);
 
-#define ISR(i, name)     extern void name ();
-#define EISR(i, name)    extern void name ();
-#define UISR(i, name)    extern void name ();
+#define ISR(i, s, name)     extern void name ();
+#define EISR(i, s, name)    extern void name ();
 #define IRQ(i, q, name)  extern void name ();
 #include "interrupt_isrs.inc"
 #undef IRQ
-#undef UISR
 #undef EISR
 #undef ISR
 }
@@ -50,13 +48,11 @@ uint8_t
 get_irq(unsigned vector)
 {
 	switch (vector) {
-#define ISR(i, name)
-#define EISR(i, name)
-#define UISR(i, name)
+#define ISR(i, s, name)
+#define EISR(i, s, name)
 #define IRQ(i, q, name) case i : return q;
 #include "interrupt_isrs.inc"
 #undef IRQ
-#undef UISR
 #undef EISR
 #undef ISR
 
@@ -87,13 +83,11 @@ disable_legacy_pic()
 void
 interrupts_init()
 {
-#define ISR(i, name)     idt_set_entry(i, reinterpret_cast<uint64_t>(& name), 0x08, 0x8e);
-#define EISR(i, name)    idt_set_entry(i, reinterpret_cast<uint64_t>(& name), 0x08, 0x8e);
-#define UISR(i, name)    idt_set_entry(i, reinterpret_cast<uint64_t>(& name), 0x08, 0xee);
+#define ISR(i, s, name)     idt_set_entry(i, reinterpret_cast<uint64_t>(& name), 0x08, 0x8e);
+#define EISR(i, s, name)    idt_set_entry(i, reinterpret_cast<uint64_t>(& name), 0x08, 0x8e);
 #define IRQ(i, q, name)  idt_set_entry(i, reinterpret_cast<uint64_t>(& name), 0x08, 0x8e);
 #include "interrupt_isrs.inc"
 #undef IRQ
-#undef UISR
 #undef EISR
 #undef ISR
 
@@ -106,8 +100,10 @@ void
 isr_handler(cpu_state *regs)
 {
 	console *cons = console::get();
+	uint8_t vector = regs->interrupt & 0xff;
+	ist_decrement(vector);
 
-	switch (static_cast<isr>(regs->interrupt & 0xff)) {
+	switch (static_cast<isr>(vector)) {
 
 	case isr::isrDebug: {
 			cons->set_color(11);
@@ -279,6 +275,7 @@ isr_handler(cpu_state *regs)
 		print_stacktrace(2);
 		_halt();
 	}
+	ist_increment(vector);
 	*reinterpret_cast<uint32_t *>(apic_eoi_addr) = 0;
 }
 
@@ -292,8 +289,8 @@ irq_handler(cpu_state *regs)
 		cons->printf("\nReceived unknown IRQ: %d (vec %d)\n",
 				irq, regs->interrupt);
 		cons->set_color();
+
 		print_regs(*regs);
-		_halt();
 	}
 
 	*reinterpret_cast<uint32_t *>(apic_eoi_addr) = 0;

src/kernel/interrupts.h

@@ -7,13 +7,11 @@
 /// Enum of all defined ISR/IRQ vectors
 enum class isr : uint8_t
 {
-#define ISR(i, name)     name = i,
-#define EISR(i, name)    name = i,
-#define UISR(i, name)    name = i,
+#define ISR(i, s, name)     name = i,
+#define EISR(i, s, name)    name = i,
 #define IRQ(i, q, name)  name = i,
 #include "interrupt_isrs.inc"
 #undef IRQ
-#undef UISR
 #undef EISR
 #undef ISR
 

src/kernel/interrupts.s

@@ -53,9 +53,8 @@ isr_handler_return:
 		jmp irq_handler_prelude
 %endmacro
 
-%define EISR(i, name)     EMIT_EISR name, i   ; ISR with error code
-%define UISR(i, name)     EMIT_ISR name, i    ; ISR callable from user space
-%define  ISR(i, name)     EMIT_ISR name, i
+%define EISR(i, s, name)     EMIT_EISR name, i   ; ISR with error code
+%define  ISR(i, s, name)     EMIT_ISR name, i
 %define  IRQ(i, q, name)  EMIT_IRQ name, i
 
 section .isrs

src/kernel/loader.s

@@ -1,24 +0,0 @@
-%include "push_all.inc"
-
-extern load_process_image
-
-global preloaded_process_init
-preloaded_process_init:
-
-	; create_process already pushed the arguments for load_process_image and
-	; the following iretq onto the stack for us
-
-	pop rdi ; the physical address of the program image
-	pop rsi ; the virtual address of the program image
-	pop rdx ; the size in bytes of the program image
-	pop rcx ; the address of this thread's TCB
-
-	call load_process_image
-
-	; user rsp is now in rax, put it in the right place for iret
-	mov [rsp + 0x18], rax
-
-	; the entrypoint should already be on the stack
-	swapgs
-	iretq
-

src/kernel/log.cpp

@@ -1,8 +1,10 @@
+#include "j6/signals.h"
 #include "kutil/memory.h"
 #include "kutil/no_construct.h"
 #include "console.h"
 #include "log.h"
-#include "scheduler.h"
+#include "objects/system.h"
+#include "objects/thread.h"
 
 static uint8_t log_buffer[0x10000];
 
@@ -26,29 +28,54 @@ output_log(log::area_t area, log::level severity, const char *message)
 	cons->set_color();
 }
 
+static void
+log_flush()
+{
+	system &sys = system::get();
+	sys.assert_signal(j6_signal_system_has_log);
+}
+
 void
 logger_task()
 {
-	uint8_t buffer[257];
-	auto *ent = reinterpret_cast<log::logger::entry *>(buffer);
 	auto *cons = console::get();
 
 	log::info(logs::task, "Starting kernel logger task");
 	g_logger.set_immediate(nullptr);
+	g_logger.set_flush(log_flush);
 
-	scheduler &s = scheduler::get();
+	thread &self = thread::current();
+	system &sys = system::get();
+
+	size_t buffer_size = 1;
+	uint8_t *buffer = nullptr;
 
 	while (true) {
-		if(g_logger.get_entry(buffer, sizeof(buffer))) {
+		size_t size = g_logger.get_entry(buffer, buffer_size);
+		if (size > buffer_size) {
+			while (size > buffer_size) buffer_size *= 2;
+			kutil::kfree(buffer);
+			buffer = reinterpret_cast<uint8_t*>(kutil::kalloc(buffer_size));
+			kassert(buffer, "Could not allocate logger task buffer");
+			continue;
+		}
+
+		if(size) {
+			auto *ent = reinterpret_cast<log::logger::entry *>(buffer);
 			buffer[ent->bytes] = 0;
+
 			cons->set_color(level_colors[static_cast<int>(ent->severity)]);
 			cons->printf("%7s %5s: %s\n",
 				g_logger.area_name(ent->area),
 				g_logger.level_name(ent->severity),
 				ent->message);
 			cons->set_color();
-		} else {
-			s.schedule();
+		}
+
+		if (!g_logger.has_log()) {
+			sys.deassert_signal(j6_signal_system_has_log);
+			sys.add_blocked_thread(&self);
+			self.wait_on_signals(&sys, j6_signal_system_has_log);
 		}
 	}
 }

src/kernel/main.cpp

@@ -35,12 +35,13 @@ extern "C" {
 
 extern void __kernel_assert(const char *, unsigned, const char *);
 
+using namespace kernel;
+
 /// Bootstrap the memory managers.
 void setup_pat();
-void memory_initialize_pre_ctors(kernel::args::header *kargs);
-void memory_initialize_post_ctors(kernel::args::header *kargs);
-
-using namespace kernel;
+void memory_initialize_pre_ctors(args::header &kargs);
+void memory_initialize_post_ctors(args::header &kargs);
+process * load_simple_process(args::program &program);
 
 /// TODO: not this. this is awful.
 args::framebuffer *fb = nullptr;
@@ -67,46 +68,6 @@ run_constructors()
 	}
 }
 
-channel *std_out = nullptr;
-
-void
-stdout_task()
-{
-	uint8_t buffer[257];
-	auto *ent = reinterpret_cast<log::logger::entry *>(buffer);
-	auto *cons = console::get();
-
-	log::info(logs::task, "Starting kernel stdout task");
-
-	scheduler &s = scheduler::get();
-	thread *th = thread::from_tcb(s.current());
-
-	while (true) {
-		j6_signal_t current = std_out->signals();
-		if (!(current & j6_signal_channel_can_recv)) {
-			th->wait_on_signals(std_out, j6_signal_channel_can_recv);
-			s.schedule();
-		}
-
-		size_t n = 256;
-		j6_status_t status = std_out->dequeue(&n, buffer);
-		if (status != j6_status_ok) {
-			log::warn(logs::task, "Kernel stdout error: %x", status);
-			return;
-		}
-
-		buffer[n] = 0;
-		const char *s = reinterpret_cast<const char *>(buffer);
-
-		while (n) {
-			size_t r = cons->puts(s);
-			n -= r + 1;
-			s += r + 1;
-		}
-	}
-}
-
-
 void
 kernel_main(args::header *header)
 {
@@ -115,6 +76,7 @@ kernel_main(args::header *header)
 	init_console();
 	logger_init();
 
+	cpu_validate();
 	setup_pat();
 
 	bool has_video = false;
@@ -123,25 +85,30 @@ kernel_main(args::header *header)
 		fb = memory::to_virtual<args::framebuffer>(reinterpret_cast<uintptr_t>(&header->video));
 
 		const args::framebuffer &video = header->video;
-		log::debug(logs::boot, "Framebuffer: %dx%d[%d] type %s @ %016llx",
-			video.horizontal, video.vertical, video.scanline, video.type, video.phys_addr);
+		log::debug(logs::boot, "Framebuffer: %dx%d[%d] type %d @ %llx size %llx",
+			video.horizontal,
+			video.vertical,
+			video.scanline,
+			video.type,
+			video.phys_addr,
+			video.size);
 		logger_clear_immediate();
 	}
 
 	gdt_init();
 	interrupts_init();
 
-	memory_initialize_pre_ctors(header);
+	memory_initialize_pre_ctors(*header);
 	run_constructors();
-	memory_initialize_post_ctors(header);
-
-	cpu_validate();
+	memory_initialize_post_ctors(*header);
 
 	for (size_t i = 0; i < header->num_modules; ++i) {
 		args::module &mod = header->modules[i];
+		void *virt = memory::to_virtual<void>(mod.location);
+
 		switch (mod.type) {
 		case args::mod_type::symbol_table:
-			new symbol_table {mod.location, mod.size};
+			new symbol_table {virt, mod.size};
 			break;
 
 		default:
@@ -157,9 +124,9 @@ kernel_main(args::header *header)
 	device_manager &devices = device_manager::get();
 	devices.parse_acpi(header->acpi_table);
 
-	interrupts_enable();
 	devices.init_drivers();
 	devices.get_lapic()->calibrate_timer();
+	interrupts_enable();
 
 	/*
 	block_device *disk = devices->get_block_device(0);
@@ -188,24 +155,12 @@ kernel_main(args::header *header)
 	syscall_enable();
 	scheduler *sched = new scheduler(devices.get_lapic());
 
-	std_out = new channel;
-
 	// Skip program 0, which is the kernel itself
-	for (size_t i = 1; i < header->num_programs; ++i) {
-		args::program &prog = header->programs[i];
-		thread *th = sched->load_process(prog.phys_addr, prog.virt_addr, prog.size, prog.entrypoint); 
-		if (i == 2) {
-			//th->set_state(thread::state::constant);
-		}
-	}
+	for (unsigned i = 1; i < header->num_programs; ++i)
+		load_simple_process(header->programs[i]);
 
 	if (!has_video)
-		sched->create_kernel_task(logger_task, scheduler::max_priority-1, true);
-	sched->create_kernel_task(stdout_task, scheduler::max_priority-1, true);
-
-	const char stdout_message[] = "Hello on the fake stdout channel\n";
-	size_t message_size = sizeof(stdout_message);
-	std_out->enqueue(&message_size, reinterpret_cast<const void*>(stdout_message));
+		sched->create_kernel_task(logger_task, scheduler::max_priority/2, true);
 
 	sched->start();
 }

src/kernel/memory_bootstrap.cpp

@@ -1,31 +1,31 @@
 #include <utility>
 
 #include "kernel_args.h"
+#include "j6/init.h"
 
 #include "kutil/assert.h"
 #include "kutil/heap_allocator.h"
 #include "kutil/no_construct.h"
 
+#include "device_manager.h"
 #include "frame_allocator.h"
+#include "gdt.h"
 #include "io.h"
 #include "log.h"
 #include "msr.h"
 #include "objects/process.h"
+#include "objects/thread.h"
+#include "objects/system.h"
 #include "objects/vm_area.h"
 #include "vm_space.h"
 
-using memory::frame_size;
 using memory::heap_start;
 using memory::kernel_max_heap;
-using memory::kernel_offset;
-using memory::heap_start;
-using memory::page_offset;
-using memory::pml4e_kernel;
-using memory::pml4e_offset;
-using memory::table_entries;
 
 using namespace kernel;
 
+extern "C" void initialize_main_thread();
+extern "C" uintptr_t initialize_main_user_stack();
 
 // These objects are initialized _before_ global constructors are called,
 // so we don't want them to have global constructors at all, lest they
@@ -36,20 +36,20 @@ kutil::heap_allocator &g_kernel_heap = __g_kernel_heap_storage.value;
 static kutil::no_construct<frame_allocator> __g_frame_allocator_storage;
 frame_allocator &g_frame_allocator = __g_frame_allocator_storage.value;
 
-static kutil::no_construct<vm_area_open> __g_kernel_heap_area_storage;
-vm_area_open &g_kernel_heap_area = __g_kernel_heap_area_storage.value;
+static kutil::no_construct<vm_area_untracked> __g_kernel_heap_area_storage;
+vm_area_untracked &g_kernel_heap_area = __g_kernel_heap_area_storage.value;
 
-vm_area_buffers g_kernel_stacks {
+vm_area_guarded g_kernel_stacks {
+	memory::stacks_start,
+	memory::kernel_stack_pages,
 	memory::kernel_max_stacks,
-	vm_space::kernel_space(),
-	vm_flags::write,
-	memory::kernel_stack_pages};
+	vm_flags::write};
 
-vm_area_buffers g_kernel_buffers {
+vm_area_guarded g_kernel_buffers {
+	memory::buffers_start,
+	memory::kernel_buffer_pages,
 	memory::kernel_max_buffers,
-	vm_space::kernel_space(),
-	vm_flags::write,
-	memory::kernel_buffer_pages};
+	vm_flags::write};
 
 void * operator new(size_t size)           { return g_kernel_heap.allocate(size); }
 void * operator new [] (size_t size)       { return g_kernel_heap.allocate(size); }
@@ -57,49 +57,91 @@ void operator delete (void *p) noexcept    { return g_kernel_heap.free(p); }
 void operator delete [] (void *p) noexcept { return g_kernel_heap.free(p); }
 
 namespace kutil {
-void * kalloc(size_t size) { return g_kernel_heap.allocate(size); }
-void kfree(void *p) { return g_kernel_heap.free(p); }
+	void * kalloc(size_t size) { return g_kernel_heap.allocate(size); }
+	void kfree(void *p) { return g_kernel_heap.free(p); }
 }
 
-/*
-void walk_page_table(
-	page_table *table,
-	page_table::level level,
-	uintptr_t &current_start,
-	size_t &current_bytes,
-	vm_area &karea)
+void
+memory_initialize_pre_ctors(args::header &kargs)
 {
-	constexpr size_t huge_page_size = (1ull<<30);
-	constexpr size_t large_page_size = (1ull<<21);
+	using kernel::args::frame_block;
 
-	for (unsigned i = 0; i < table_entries; ++i) {
-		page_table *next = table->get(i);
-		if (!next) {
-			if (current_bytes)
-				karea.commit(current_start, current_bytes);
-			current_start = 0;
-			current_bytes = 0;
-			continue;
-		} else if (table->is_page(level, i)) {
-			if (!current_bytes)
-				current_start = reinterpret_cast<uintptr_t>(next);
-			current_bytes +=
-				(level == page_table::level::pt
-					? frame_size
-					: level == page_table::level::pd
-						? large_page_size
-						: huge_page_size);
-		} else {
-			page_table::level deeper =
-				static_cast<page_table::level>(
-					static_cast<unsigned>(level) + 1);
+	new (&g_kernel_heap) kutil::heap_allocator {heap_start, kernel_max_heap};
 
-			walk_page_table(
-				next, deeper, current_start, current_bytes, kspace);
+	frame_block *blocks = reinterpret_cast<frame_block*>(memory::bitmap_start);
+	new (&g_frame_allocator) frame_allocator {blocks, kargs.frame_block_count};
+
+	// Mark all the things the bootloader allocated for us as used
+	g_frame_allocator.used(
+		reinterpret_cast<uintptr_t>(kargs.frame_blocks),
+		kargs.frame_block_pages);
+
+	g_frame_allocator.used(
+		reinterpret_cast<uintptr_t>(kargs.pml4),
+		kargs.table_pages);
+
+	for (unsigned i = 0; i < kargs.num_modules; ++i) {
+		const kernel::args::module &mod = kargs.modules[i];
+		g_frame_allocator.used(
+			reinterpret_cast<uintptr_t>(mod.location),
+			memory::page_count(mod.size));
+	}
+
+	for (unsigned i = 0; i < kargs.num_programs; ++i) {
+		const kernel::args::program &prog = kargs.programs[i];
+		for (auto &sect : prog.sections) {
+			if (!sect.size) continue;
+			g_frame_allocator.used(
+				sect.phys_addr,
+				memory::page_count(sect.size));
 		}
 	}
+
+	page_table *kpml4 = reinterpret_cast<page_table*>(kargs.pml4);
+	process *kp = process::create_kernel_process(kpml4);
+	vm_space &vm = kp->space();
+
+	vm_area *heap = new (&g_kernel_heap_area)
+		vm_area_untracked(kernel_max_heap, vm_flags::write);
+
+	vm.add(heap_start, heap);
+}
+
+void
+memory_initialize_post_ctors(args::header &kargs)
+{
+	vm_space &vm = vm_space::kernel_space();
+	vm.add(memory::stacks_start, &g_kernel_stacks);
+	vm.add(memory::buffers_start, &g_kernel_buffers);
+
+	g_frame_allocator.free(
+		reinterpret_cast<uintptr_t>(kargs.page_tables),
+		kargs.table_count);
+
+	using memory::frame_size;
+	using memory::kernel_stack_pages;
+	constexpr size_t stack_size = kernel_stack_pages * frame_size;
+
+	for (int ist = 1; ist <= 3; ++ist) {
+		uintptr_t bottom = g_kernel_stacks.get_section();
+		log::debug(logs::boot, "Installing IST%d stack at %llx", ist, bottom);
+
+		// Pre-realize and xerothese stacks, they're no good
+		// if they page fault
+		kutil::memset(reinterpret_cast<void*>(bottom), 0, stack_size);
+
+		// Skip two entries to be the null frame
+		tss_set_ist(ist, bottom + stack_size - 2 * sizeof(uintptr_t));
+	}
+
+#define ISR(i, s, name)   if (s) { idt_set_ist(i, s); }
+#define EISR(i, s, name)  if (s) { idt_set_ist(i, s); }
+#define IRQ(i, q, name)
+#include "interrupt_isrs.inc"
+#undef IRQ
+#undef EISR
+#undef ISR
 }
-*/
 
 static void
 log_mtrrs()
@@ -166,60 +208,103 @@ setup_pat()
 }
 
 
-void
-memory_initialize_pre_ctors(args::header *kargs)
+process *
+load_simple_process(args::program &program)
 {
-	new (&g_kernel_heap) kutil::heap_allocator {heap_start, kernel_max_heap};
-	new (&g_frame_allocator) frame_allocator;
+	using kernel::args::section_flags;
 
-	args::mem_entry *entries = kargs->mem_map;
-	const size_t count = kargs->map_count;
-	for (unsigned i = 0; i < count; ++i) {
-		// TODO: use entry attributes
-		// TODO: copy anything we need from "pending" memory and free it
-		args::mem_entry &e = entries[i];
-		if (e.type == args::mem_type::free)
-			g_frame_allocator.free(e.start, e.pages);
+	process *p = new process;
+	vm_space &space = p->space();
+
+	for (const auto &sect : program.sections) {
+		vm_flags flags =
+			(bitfield_has(sect.type, section_flags::execute) ? vm_flags::exec : vm_flags::none) |
+			(bitfield_has(sect.type, section_flags::write) ? vm_flags::write : vm_flags::none);
+
+		vm_area *vma = new vm_area_fixed(sect.phys_addr, sect.size, flags);
+		space.add(sect.virt_addr, vma);
 	}
 
-	page_table *kpml4 = reinterpret_cast<page_table*>(kargs->pml4);
-	process *kp = process::create_kernel_process(kpml4);
-	vm_space &vm = kp->space();
+	uint64_t iopl = (3ull << 12);
+	uintptr_t trampoline = reinterpret_cast<uintptr_t>(initialize_main_thread);
 
-	vm_area *heap = new (&g_kernel_heap_area)
-		vm_area_open(memory::kernel_max_heap, vm, vm_flags::write);
+	thread *main = p->create_thread();
+	main->add_thunk_user(program.entrypoint, trampoline, iopl);
+	main->set_state(thread::state::ready);
 
-	vm.add(memory::heap_start, heap);
+	return p;
 }
 
-void
-memory_initialize_post_ctors(args::header *kargs)
+template <typename T>
+inline T * push(uintptr_t &rsp, size_t size = sizeof(T)) {
+	rsp -= size;
+	T *p = reinterpret_cast<T*>(rsp);
+	rsp &= ~(sizeof(uint64_t)-1); // Align the stack
+	return p;
+}
+
+uintptr_t
+initialize_main_user_stack()
 {
-	/*
-	uintptr_t current_start = 0;
-	size_t current_bytes = 0;
+	process &proc = process::current();
+	thread &th = thread::current();
+	TCB *tcb = th.tcb();
 
-	// TODO: Should we exclude the top of this area? (eg, buffers, stacks, etc)
-	page_table *kpml4 = reinterpret_cast<page_table*>(kargs->pml4);
-	for (unsigned i = pml4e_kernel; i < pml4e_offset; ++i) {
-		page_table *pdp = kpml4->get(i);
-		kassert(pdp, "Bootloader did not create all kernelspace PDs");
+	const char message[] = "Hello from the kernel!";
+	char *message_arg = push<char>(tcb->rsp3, sizeof(message));
+	kutil::memcpy(message_arg, message, sizeof(message));
 
-		walk_page_table(
-			pdp, page_table::level::pdp,
-			current_start, current_bytes,
-			g_kernel_space);
+	j6_init_value *initv = nullptr;
+	unsigned n = 0;
+
+	extern args::framebuffer *fb;
+	if (fb) {
+		j6_init_framebuffer *fb_desc = push<j6_init_framebuffer>(tcb->rsp3);
+		kutil::memset(fb_desc, 0, sizeof(j6_init_framebuffer));
+
+		fb_desc->addr = fb->phys_addr;
+		fb_desc->size = fb->size;
+		fb_desc->vertical = fb->vertical;
+		fb_desc->horizontal = fb->horizontal;
+		fb_desc->scanline = fb->scanline;
+
+		if (fb->type == kernel::args::fb_type::bgr8)
+			fb_desc->flags |= 1;
+
+		initv = push<j6_init_value>(tcb->rsp3);
+		initv->type = j6_init_desc_framebuffer;
+		initv->data = fb_desc;
+		++n;
 	}
 
-	if (current_bytes)
-		g_kernel_space.commit(current_start, current_bytes);
-	*/
-	vm_space &vm = vm_space::kernel_space();
-	vm.add(memory::stacks_start, &g_kernel_stacks);
-	vm.add(memory::buffers_start, &g_kernel_buffers);
 
-	g_frame_allocator.free(
-		reinterpret_cast<uintptr_t>(kargs->page_tables),
-		kargs->table_count);
+	initv = push<j6_init_value>(tcb->rsp3);
+	initv->type = j6_init_handle_other;
+	initv->handle.type = j6_object_type_system;
+	initv->handle.handle = proc.add_handle(&system::get());
+	++n;
 
+	initv = push<j6_init_value>(tcb->rsp3);
+	initv->type = j6_init_handle_self;
+	initv->handle.type = j6_object_type_process;
+	initv->handle.handle = proc.self_handle();
+	++n;
+
+	initv = push<j6_init_value>(tcb->rsp3);
+	initv->type = j6_init_handle_self;
+	initv->handle.type = j6_object_type_thread;
+	initv->handle.handle = th.self_handle();
+	++n;
+
+	uint64_t *initc = push<uint64_t>(tcb->rsp3);
+	*initc = n;
+
+	char **argv0 = push<char*>(tcb->rsp3);
+	*argv0 = message_arg;
+
+	uint64_t *argc = push<uint64_t>(tcb->rsp3);
+	*argc = 1;
+
+	th.clear_state(thread::state::loading);
+	return tcb->rsp3;
 }

src/kernel/objects/channel.cpp

@@ -4,13 +4,13 @@
 #include "objects/channel.h"
 #include "objects/vm_area.h"
 
-extern vm_area_buffers g_kernel_buffers;
+extern vm_area_guarded g_kernel_buffers;
 
 constexpr size_t buffer_bytes = memory::kernel_buffer_pages * memory::frame_size;
 
 channel::channel() :
 	m_len(0),
-	m_data(g_kernel_buffers.get_buffer()),
+	m_data(g_kernel_buffers.get_section()),
 	m_buffer(reinterpret_cast<uint8_t*>(m_data), buffer_bytes),
 	kobject(kobject::type::channel, j6_signal_channel_can_send)
 {
@@ -79,7 +79,7 @@ void
 channel::close()
 {
 	kobject::close();
-	g_kernel_buffers.return_buffer(m_data);
+	g_kernel_buffers.return_section(m_data);
 }
 
 void

src/kernel/objects/kobject.h

@@ -16,17 +16,9 @@ public:
 	/// Types of kernel objects.
 	enum class type : uint16_t
 	{
-		none,
-		system,
-
-		event,
-		channel,
-		endpoint,
-
-		vma,
-
-		process,
-		thread,
+#define OBJECT_TYPE( name, val ) name = val,
+#include "j6/tables/object_types.inc"
+#undef OBJECT_TYPE
 
 		max
 	};

src/kernel/objects/process.cpp

@@ -17,13 +17,13 @@ kutil::vector<process*> process::s_processes;
 
 process::process() :
 	kobject {kobject::type::process},
-	m_next_handle {0},
+	m_next_handle {1},
 	m_state {state::running}
 {
 	s_processes.append(this);
 
 	j6_handle_t self = add_handle(this);
-	kassert(self == self_handle(), "Process self-handle is not 0");
+	kassert(self == self_handle(), "Process self-handle is not 1");
 }
 
 // The "kernel process"-only constructor
@@ -52,7 +52,7 @@ process::create_kernel_process(page_table *pml4)
 }
 
 void
-process::exit(unsigned code)
+process::exit(int32_t code)
 {
 	// TODO: make this thread-safe
 	m_state = state::exited;
@@ -93,7 +93,7 @@ process::update()
 thread *
 process::create_thread(uint8_t priority, bool user)
 {
-	if (priority == default_pri)
+	if (priority == default_priority)
 		priority = scheduler::default_priority;
 
 	thread *th = new thread(*this, priority);
@@ -102,11 +102,13 @@ process::create_thread(uint8_t priority, bool user)
 	if (user) {
 		uintptr_t stack_top = stacks_top - (m_threads.count() * stack_size);
 
-		vm_area *vma = new vm_area_open(stack_size, m_space,
-				vm_flags::zero|vm_flags::write);
+		vm_flags flags = vm_flags::zero|vm_flags::write;
+		vm_area *vma = new vm_area_open(stack_size, flags);
 		m_space.add(stack_top - stack_size, vma);
 
-		th->tcb()->rsp3 = stack_top;
+		// Space for null frame - because the page gets zeroed on
+		// allocation, just pointing rsp here does the trick
+		th->tcb()->rsp3 = stack_top - 2 * sizeof(uint64_t);
 	}
 
 	m_threads.append(th);
@@ -123,6 +125,8 @@ process::thread_exited(thread *th)
 	remove_handle(th->self_handle());
 	delete th;
 
+	// TODO: delete the thread's stack VMA
+
 	if (m_threads.count() == 0) {
 		exit(status);
 		return true;

src/kernel/objects/process.h

@@ -16,10 +16,10 @@ public:
 	constexpr static uintptr_t stacks_top = 0x0000800000000000;
 
 	/// Size of userspace thread stacks
-	constexpr static size_t stack_size = 0x4000;
+	constexpr static size_t stack_size = 0x4000000; // 64MiB
 
 	/// Value that represents default priority
-	constexpr static uint8_t default_pri = 0xff;
+	constexpr static uint8_t default_priority = 0xff;
 
 	/// Constructor.
 	process();
@@ -34,7 +34,7 @@ public:
 
 	/// Terminate this process.
 	/// \arg code   The return code to exit with.
-	void exit(unsigned code);
+	void exit(int32_t code);
 
 	/// Update internal bookkeeping about threads.
 	void update();
@@ -46,7 +46,7 @@ public:
 	/// \args priority  The new thread's scheduling priority
 	/// \args user      If true, create a userspace stack for this thread
 	/// \returns        The newly created thread object
-	thread * create_thread(uint8_t priorty = default_pri, bool user = true);
+	thread * create_thread(uint8_t priorty = default_priority, bool user = true);
 
 	/// Start tracking an object with a handle.
 	/// \args obj  The object this handle refers to
@@ -69,7 +69,7 @@ public:
 	bool thread_exited(thread *th);
 
 	/// Get the handle for this process to refer to itself
-	inline j6_handle_t self_handle() const { return 0; }
+	inline j6_handle_t self_handle() const { return 1; }
 
 	/// Get the process object that owns kernel threads and the
 	/// kernel address space
@@ -84,7 +84,7 @@ private:
 	// This constructor is called by create_kernel_process
 	process(page_table *kpml4);
 
-	uint32_t m_return_code;
+	int32_t m_return_code;
 
 	vm_space m_space;
 

src/kernel/objects/system.h

@@ -10,7 +10,7 @@ class system :
 public:
 	static constexpr kobject::type type = kobject::type::event;
 
-	inline static system * get() { return &s_instance; }
+	inline static system & get() { return s_instance; }
 
 private:
 	static system s_instance;

src/kernel/objects/thread.cpp

@@ -9,7 +9,7 @@
 extern "C" void kernel_to_user_trampoline();
 static constexpr j6_signal_t thread_default_signals = 0;
 
-extern vm_area_buffers g_kernel_stacks;
+extern vm_area_guarded g_kernel_stacks;
 
 thread::thread(process &parent, uint8_t pri, uintptr_t rsp0) :
 	kobject(kobject::type::thread, thread_default_signals),
@@ -32,7 +32,7 @@ thread::thread(process &parent, uint8_t pri, uintptr_t rsp0) :
 
 thread::~thread()
 {
-	g_kernel_stacks.return_buffer(m_tcb.kernel_stack);
+	g_kernel_stacks.return_section(m_tcb.kernel_stack);
 }
 
 thread *
@@ -136,7 +136,7 @@ thread::wake_on_result(kobject *obj, j6_status_t result)
 }
 
 void
-thread::exit(uint32_t code)
+thread::exit(int32_t code)
 {
 	m_return_code = code;
 	set_state(state::exited);
@@ -149,6 +149,10 @@ thread::exit(uint32_t code)
 void
 thread::add_thunk_kernel(uintptr_t rip)
 {
+	// This adds just enough values to the top of the
+	// kernel stack to come out of task_switch correctly
+	// and start executing at rip (still in kernel mode)
+
 	m_tcb.rsp -= sizeof(uintptr_t) * 7;
 	uintptr_t *stack = reinterpret_cast<uintptr_t*>(m_tcb.rsp);
 
@@ -162,15 +166,24 @@ thread::add_thunk_kernel(uintptr_t rip)
 }
 
 void
-thread::add_thunk_user(uintptr_t rip)
+thread::add_thunk_user(uintptr_t rip3, uintptr_t rip0, uint64_t flags)
 {
+	// This sets up the stack to:
+	// a) come out of task_switch and return to rip0 (default is the
+	//    kernel/user trampoline) (via add_thunk_kernel) - if this is
+	//    changed, it needs to end up at the trampoline with the stack
+	//    as it was
+	// b) come out of the kernel/user trampoline and start executing
+	//    in user mode at rip
+
 	m_tcb.rsp -= sizeof(uintptr_t) * 8;
 	uintptr_t *stack = reinterpret_cast<uintptr_t*>(m_tcb.rsp);
+	flags |= 0x200;
 
-	stack[7] = rip;        // return rip in rcx
+	stack[7] = rip3;       // return rip in rcx
 	stack[6] = m_tcb.rsp3; // rbp
 	stack[5] = 0xbbbbbbbb; // rbx
-	stack[4] = 0x00000200; // r11 sets RFLAGS
+	stack[4] = flags;      // r11 sets RFLAGS
 	stack[3] = 0x12121212; // r12
 	stack[2] = 0x13131313; // r13
 	stack[1] = 0x14141414; // r14
@@ -178,7 +191,7 @@ thread::add_thunk_user(uintptr_t rip)
 
 	static const uintptr_t trampoline =
 		reinterpret_cast<uintptr_t>(kernel_to_user_trampoline);
-	add_thunk_kernel(trampoline);
+	add_thunk_kernel(rip0 ? rip0 : trampoline);
 }
 
 void
@@ -191,7 +204,7 @@ thread::setup_kernel_stack()
 	constexpr unsigned null_frame_entries = 2;
 	constexpr size_t null_frame_size = null_frame_entries * sizeof(uint64_t);
 
-	uintptr_t stack_addr = g_kernel_stacks.get_buffer();
+	uintptr_t stack_addr = g_kernel_stacks.get_section();
 	uintptr_t stack_end = stack_addr + stack_bytes;
 
 	uint64_t *null_frame = reinterpret_cast<uint64_t*>(stack_end - null_frame_size);

src/kernel/objects/thread.h

@@ -131,15 +131,18 @@ public:
 
 	/// Terminate this thread.
 	/// \arg code   The return code to exit with.
-	void exit(unsigned code);
+	void exit(int32_t code);
 
 	/// Add a stack header that returns to the given address in kernel space.
 	/// \arg rip  The address to return to, must be kernel space
 	void add_thunk_kernel(uintptr_t rip);
 
-	/// Add a stack header that returns to the given address in user space.
-	/// \arg rip  The address to return to, must be user space
-	void add_thunk_user(uintptr_t rip);
+	/// Add a stack header that returns to the given address in user space
+	/// via a function in kernel space.
+	/// \arg rip3  The user space address to return to
+	/// \arg rip0  The kernel function to pass through, optional
+	/// \arg flags Extra RFLAGS values to set, optional
+	void add_thunk_user(uintptr_t rip3, uintptr_t rip0 = 0, uint64_t flags = 0);
 
 	/// Get the handle representing this thread to its process
 	j6_handle_t self_handle() const { return m_self_handle; }
@@ -173,7 +176,7 @@ private:
 	wait_type m_wait_type;
 	// There should be 1 byte of padding here
 
-	uint32_t m_return_code;
+	int32_t m_return_code;
 
 	uint64_t m_wait_data;
 	j6_status_t m_wait_result;

src/kernel/objects/vm_area.cpp

@@ -1,5 +1,7 @@
+#include "frame_allocator.h"
 #include "kernel_memory.h"
 #include "objects/vm_area.h"
+#include "page_tree.h"
 #include "vm_space.h"
 
 using memory::frame_size;
@@ -7,30 +9,31 @@ using memory::frame_size;
 vm_area::vm_area(size_t size, vm_flags flags) :
 	m_size {size},
 	m_flags {flags},
+	m_spaces {m_vector_static, 0, static_size},
 	kobject {kobject::type::vma}
 {
 }
 
 vm_area::~vm_area() {}
 
-size_t
-vm_area::resize(size_t size)
+bool
+vm_area::add_to(vm_space *space)
 {
-	if (mapper().can_resize(size))
-		m_size = size;
-	return m_size;
+	for (auto *s : m_spaces) {
+		if (s == space)
+			return true;
+	}
+	m_spaces.append(space);
+	return true;
 }
 
-void
-vm_area::commit(uintptr_t phys, uintptr_t offset, size_t count)
+bool
+vm_area::remove_from(vm_space *space)
 {
-	mapper().map(offset, count, phys);
-}
-
-void
-vm_area::uncommit(uintptr_t offset, size_t count)
-{
-	mapper().unmap(offset, count);
+	m_spaces.remove_swap(space);
+	return
+		!m_spaces.count() &&
+		!(m_flags && vm_flags::mmio);
 }
 
 void
@@ -40,48 +43,99 @@ vm_area::on_no_handles()
 	delete this;
 }
 
+size_t
+vm_area::resize(size_t size)
+{
+	if (can_resize(size))
+		m_size = size;
+	return m_size;
+}
 
+bool
+vm_area::can_resize(size_t size)
+{
+	for (auto *space : m_spaces)
+		if (!space->can_resize(*this, size))
+			return false;
+	return true;
+}
 
-vm_area_shared::vm_area_shared(size_t size, vm_flags flags) :
-	m_mapper {*this},
+vm_area_fixed::vm_area_fixed(uintptr_t start, size_t size, vm_flags flags) :
+	m_start {start},
 	vm_area {size, flags}
 {
 }
 
-vm_area_shared::~vm_area_shared()
+vm_area_fixed::~vm_area_fixed()
 {
 }
 
-
-vm_area_open::vm_area_open(size_t size, vm_space &space, vm_flags flags) :
-	m_mapper(*this, space),
-	vm_area(size, flags)
+size_t vm_area_fixed::resize(size_t size)
 {
+	// Not resizable
+	return m_size;
 }
 
-void
-vm_area_open::commit(uintptr_t phys, uintptr_t offset, size_t count)
+bool vm_area_fixed::get_page(uintptr_t offset, uintptr_t &phys)
 {
-	m_mapper.map(offset, count, phys);
-}
+	if (offset > m_size)
+		return false;
 
-void
-vm_area_open::uncommit(uintptr_t offset, size_t count)
-{
-	m_mapper.unmap(offset, count);
+	phys = m_start + offset;
+	return true;
 }
 
 
-vm_area_buffers::vm_area_buffers(size_t size, vm_space &space, vm_flags flags, size_t buf_pages) :
-	m_mapper {*this, space},
+vm_area_untracked::vm_area_untracked(size_t size, vm_flags flags) :
+	vm_area {size, flags}
+{
+}
+
+vm_area_untracked::~vm_area_untracked()
+{
+}
+
+bool
+vm_area_untracked::get_page(uintptr_t offset, uintptr_t &phys)
+{
+	if (offset > m_size)
+		return false;
+
+	return frame_allocator::get().allocate(1, &phys);
+}
+
+bool
+vm_area_untracked::add_to(vm_space *space)
+{
+	if (!m_spaces.count())
+		return vm_area::add_to(space);
+	return m_spaces[0] == space;
+}
+
+
+vm_area_open::vm_area_open(size_t size, vm_flags flags) :
+	m_mapped {nullptr},
+	vm_area {size, flags}
+{
+}
+
+bool
+vm_area_open::get_page(uintptr_t offset, uintptr_t &phys)
+{
+	return page_tree::find_or_add(m_mapped, offset, phys);
+}
+
+
+vm_area_guarded::vm_area_guarded(uintptr_t start, size_t buf_pages, size_t size, vm_flags flags) :
+	m_start {start},
 	m_pages {buf_pages},
 	m_next {memory::frame_size},
-	vm_area {size, flags}
+	vm_area_untracked {size, flags}
 {
 }
 
 uintptr_t
-vm_area_buffers::get_buffer()
+vm_area_guarded::get_section()
 {
 	if (m_cache.count() > 0) {
 		return m_cache.pop();
@@ -89,33 +143,27 @@ vm_area_buffers::get_buffer()
 
 	uintptr_t addr = m_next;
 	m_next += (m_pages + 1) * memory::frame_size;
-	return m_mapper.space().lookup(*this, addr);
+	return m_start + addr;
 }
 
 void
-vm_area_buffers::return_buffer(uintptr_t addr)
+vm_area_guarded::return_section(uintptr_t addr)
 {
 	m_cache.append(addr);
 }
 
 bool
-vm_area_buffers::allowed(uintptr_t offset) const
+vm_area_guarded::get_page(uintptr_t offset, uintptr_t &phys)
 {
-	if (offset >= m_next) return false;
+	if (offset > m_next)
+		return false;
 
-	// Buffers are m_pages big plus 1 leading guard page
-	return memory::page_align_down(offset) % (m_pages+1);
-}
-
-void
-vm_area_buffers::commit(uintptr_t phys, uintptr_t offset, size_t count)
-{
-	m_mapper.map(offset, count, phys);
-}
-
-void
-vm_area_buffers::uncommit(uintptr_t offset, size_t count)
-{
-	m_mapper.unmap(offset, count);
+	// make sure this isn't in a guard page. (sections are
+	// m_pages big plus 1 leading guard page, so page 0 is
+	// invalid)
+	if ((offset >> 12) % (m_pages+1) == 0)
+		return false;
+
+	return vm_area_untracked::get_page(offset, phys);
 }
 

src/kernel/objects/vm_area.h

@@ -11,26 +11,15 @@
 
 #include "kernel_memory.h"
 #include "objects/kobject.h"
-#include "vm_mapper.h"
 
+class page_tree;
 class vm_space;
 
 enum class vm_flags : uint32_t
 {
-	none            = 0x00000000,
-
-	write           = 0x00000001,
-	exec            = 0x00000002,
-
-	zero            = 0x00000010, 
-	contiguous      = 0x00000020,
-
-	large_pages     = 0x00000100,
-	huge_pages      = 0x00000200,
-
-	mmio            = 0x00010000,
-	write_combine   = 0x00020000,
-
+#define VM_FLAG(name, v) name = v,
+#include "j6/tables/vm_flags.inc"
+#undef VM_FLAG
 	user_mask       = 0x0000ffff  ///< flags allowed via syscall
 };
 
@@ -55,136 +44,131 @@ public:
 	/// Get the flags set for this area
 	inline vm_flags flags() const { return m_flags; }
 
+	/// Track that this area was added to a vm_space
+	/// \arg space  The space to add this area to
+	/// \returns    False if this area cannot be added
+	virtual bool add_to(vm_space *space);
+
+	/// Track that this area was removed frm a vm_space
+	/// \arg space  The space that is removing this area
+	/// \returns    True if the removing space should free the pages
+	///             mapped for this area
+	virtual bool remove_from(vm_space *space);
+
 	/// Change the virtual size of the memory area. This may cause
 	/// deallocation if the new size is smaller than the current size.
 	/// Note that if resizing is unsuccessful, the previous size will
 	/// be returned.
 	/// \arg size  The desired new virtual size
 	/// \returns   The new virtual size
-	size_t resize(size_t size);
+	virtual size_t resize(size_t size);
 
-	/// Get the mapper object that maps this area to address spaces
-	virtual vm_mapper & mapper() = 0;
-	virtual const vm_mapper & mapper() const = 0;
-
-	/// Check whether allocation at the given offset is allowed
-	virtual bool allowed(uintptr_t offset) const { return true; }
-
-	/// Commit contiguous physical pages to this area
-	/// \arg phys   The physical address of the first page
-	/// \arg offset The offset from the start of this area these pages represent
-	/// \arg count  The number of pages
-	virtual void commit(uintptr_t phys, uintptr_t offset, size_t count);
-
-	/// Uncommit physical pages from this area
-	/// \arg offset The offset from the start of this area these pages represent
-	/// \arg count  The number of pages
-	virtual void uncommit(uintptr_t offset, size_t count);
+	/// Get the physical page for the given offset
+	/// \arg offset The offset into the VMA
+	/// \arg phys   [out] Receives the physical page address, if any
+	/// \returns    True if there should be a page at the given offset
+	virtual bool get_page(uintptr_t offset, uintptr_t &phys) = 0;
 
 protected:
 	virtual void on_no_handles() override;
+	bool can_resize(size_t size);
 
 	size_t m_size;
 	vm_flags m_flags;
+	kutil::vector<vm_space*> m_spaces;
+
+	// Initial static space for m_spaces - most areas will never grow
+	// beyond this size, so avoid allocations
+	static constexpr size_t static_size = 2;
+	vm_space *m_vector_static[static_size];
 };
 
 
-/// The standard, sharable, user-controllable VMA type
-class vm_area_shared :
+/// A shareable but non-allocatable memory area of contiguous physical
+/// addresses (like mmio)
+class vm_area_fixed :
 	public vm_area
 {
 public:
 	/// Constructor.
-	/// \arg size  Initial virtual size of the memory area
+	/// \arg start Starting physical address of this area
+	/// \arg size  Size of the physical memory area
 	/// \arg flags Flags for this memory area
-	vm_area_shared(size_t size, vm_flags flags = vm_flags::none);
-	virtual ~vm_area_shared();
+	vm_area_fixed(uintptr_t start, size_t size, vm_flags flags = vm_flags::none);
+	virtual ~vm_area_fixed();
 
-	virtual vm_mapper & mapper() override { return m_mapper; }
-	virtual const vm_mapper & mapper() const override { return m_mapper; }
+	virtual size_t resize(size_t size) override;
+	virtual bool get_page(uintptr_t offset, uintptr_t &phys) override;
 
 private:
-	vm_mapper_multi m_mapper;
+	uintptr_t m_start;
 };
 
 
-/// Area that allows open allocation (eg, kernel heap)
+/// Area that allows open allocation
 class vm_area_open :
 	public vm_area
 {
 public:
 	/// Constructor.
 	/// \arg size  Initial virtual size of the memory area
-	/// \arg space The address space this area belongs to
 	/// \arg flags Flags for this memory area
-	vm_area_open(size_t size, vm_space &space, vm_flags flags);
+	vm_area_open(size_t size, vm_flags flags);
 
-	virtual vm_mapper & mapper() override { return m_mapper; }
-	virtual const vm_mapper & mapper() const override { return m_mapper; }
-
-	virtual void commit(uintptr_t phys, uintptr_t offset, size_t count) override;
-	virtual void uncommit(uintptr_t offset, size_t count) override;
+	virtual bool get_page(uintptr_t offset, uintptr_t &phys) override;
 
 private:
-	vm_mapper_single m_mapper;
+	page_tree *m_mapped;
 };
 
 
-/// Area split into standard-sized segments
-class vm_area_buffers :
-	public vm_area
-{
-public:
-	/// Constructor.
-	/// \arg size      Initial virtual size of the memory area
-	/// \arg space     The address space this area belongs to
-	/// \arg flags     Flags for this memory area
-	/// \arg buf_pages Pages in an individual buffer
-	vm_area_buffers(
-		size_t size,
-		vm_space &space,
-		vm_flags flags,
-		size_t buf_pages);
-
-	/// Get an available stack address
-	uintptr_t get_buffer();
-
-	/// Return a buffer address to the available pool
-	void return_buffer(uintptr_t addr);
-
-	virtual vm_mapper & mapper() override { return m_mapper; }
-	virtual const vm_mapper & mapper() const override { return m_mapper; }
-
-	virtual bool allowed(uintptr_t offset) const override;
-	virtual void commit(uintptr_t phys, uintptr_t offset, size_t count) override;
-	virtual void uncommit(uintptr_t offset, size_t count) override;
-
-private:
-	vm_mapper_single m_mapper;
-	kutil::vector<uintptr_t> m_cache;
-	size_t m_pages;
-	uintptr_t m_next;
-};
-
-
-/// Area backed by an external source (like a loaded program)
-class vm_area_backed :
+/// Area that does not track its allocations and thus cannot be shared
+class vm_area_untracked :
 	public vm_area
 {
 public:
 	/// Constructor.
 	/// \arg size  Initial virtual size of the memory area
 	/// \arg flags Flags for this memory area
-	vm_area_backed(size_t size, vm_flags flags);
+	vm_area_untracked(size_t size, vm_flags flags);
+	virtual ~vm_area_untracked();
 
-	virtual vm_mapper & mapper() override { return m_mapper; }
-	virtual const vm_mapper & mapper() const override { return m_mapper; }
-
-	virtual void commit(uintptr_t phys, uintptr_t offset, size_t count) override;
-	virtual void uncommit(uintptr_t offset, size_t count) override;
-
-private:
-	vm_mapper_multi m_mapper;
+	virtual bool add_to(vm_space *space) override;
+	virtual bool get_page(uintptr_t offset, uintptr_t &phys) override;
 };
 
+
+/// Area split into standard-sized segments, separated by guard pages.
+/// Based on vm_area_untracked, can not be shared.
+class vm_area_guarded :
+	public vm_area_untracked
+{
+public:
+	/// Constructor.
+	/// \arg start     Initial address where this area is mapped
+	/// \arg sec_pages Pages in an individual section
+	/// \arg size      Initial virtual size of the memory area
+	/// \arg flags     Flags for this memory area
+	vm_area_guarded(
+		uintptr_t start,
+		size_t sec_pages,
+		size_t size,
+		vm_flags flags);
+
+	/// Get an available section in this area
+	uintptr_t get_section();
+
+	/// Return a section address to the available pool
+	void return_section(uintptr_t addr);
+
+	virtual bool get_page(uintptr_t offset, uintptr_t &phys) override;
+
+private:
+	kutil::vector<uintptr_t> m_cache;
+	uintptr_t m_start;
+	size_t m_pages;
+	uintptr_t m_next;
+};
+
+
 IS_BITFIELD(vm_flags);

src/kernel/page_tree.cpp

@@ -0,0 +1,152 @@
+#include "kutil/assert.h"
+#include "kutil/memory.h"
+#include "frame_allocator.h"
+#include "page_tree.h"
+
+// Page tree levels map the following parts of a pagewise offset:
+// (Note that a level 0's entries are physical page addrs, the rest
+// map other page_tree nodes)
+//
+// Level 0: 0000000003f    64 pages / 256 KiB 
+// Level 1: 00000000fc0    4K pages /  16 MiB
+// Level 2: 0000003f000  256K pages /   1 GiB
+// Level 3: 00000fc0000   16M pages /  64 GiB
+// Level 4: 0003f000000    1G pages /   4 TiB
+// Level 5: 00fc0000000   64G pages / 256 TiB
+// Level 6: 3f000000000    4T pages /  16 PiB -- Not supported until 5-level paging
+
+static constexpr unsigned max_level = 5;
+static constexpr unsigned bits_per_level = 6;
+
+inline uint64_t to_word(uint64_t base, uint64_t level, uint64_t flags = 0) {
+	// Clear out the non-appropriate bits for this level
+	base &= (~0x3full << (level*bits_per_level));
+
+	return
+		(base & 0x3ffffffffff) |
+		((level & 0x7) << 42) |
+		((flags & 0x7ffff) << 45);
+}
+
+inline uint64_t to_base(uint64_t word) {
+	return word & 0x3ffffffffff;
+}
+
+inline uint64_t to_level(uint64_t word) {
+	return (word >> 42) & 0x3f;
+}
+
+inline uint64_t to_flags(uint64_t word) {
+	return (word >> 45);
+}
+
+inline bool contains(uint64_t page_off, uint64_t word, uint8_t &index) {
+	uint64_t base = to_base(word);
+	uint64_t bits = to_level(word) * bits_per_level;
+	index = (page_off >> bits) & 0x3f;
+	return (page_off & (~0x3full << bits)) != base;
+}
+
+inline uint64_t index_for(uint64_t page_off, uint8_t level) {
+	return (page_off >> (level*bits_per_level))  & 0x3f;
+}
+
+page_tree::page_tree(uint64_t base, uint8_t level) :
+	m_base {to_word(base, level)}
+{
+	kutil::memset(m_entries, 0, sizeof(m_entries));
+}
+
+bool
+page_tree::find(const page_tree *root, uint64_t offset, uintptr_t &page)
+{
+	uint64_t page_off = offset >> 12; // change to pagewise offset
+	page_tree const *node = root;
+	while (node) {
+		uint8_t level = to_level(node->m_base);
+		uint8_t index = 0;
+		if (!contains(page_off, node->m_base, index))
+			return false;
+
+		if (!level) {
+			uintptr_t entry = node->m_entries[index].entry;
+			page = entry & ~1ull; // bit 0 marks 'present'
+			return (entry & 1);
+		}
+
+		node = node->m_entries[index].child;
+	}
+
+	return false;
+}
+
+bool
+page_tree::find_or_add(page_tree * &root, uint64_t offset, uintptr_t &page)
+{
+	uint64_t page_off = offset >> 12; // change to pagewise offset
+	page_tree *level0 = nullptr;
+
+	if (!root) {
+		// There's no root yet, just make a level0 and make it
+		// the root.
+		level0 = new page_tree(page_off, 0);
+		root = level0;
+	} else {
+		// Find or insert an existing level0
+		page_tree **parent = &root;
+		page_tree *node = root;
+		uint8_t parent_level = max_level + 1;
+
+		while (node) {
+			uint8_t level = to_level(node->m_base);
+			uint8_t index = 0;
+			if (!contains(page_off, node->m_base, index)) {
+				// We found a valid parent but the slot where this node should
+				// go contains another node. Insert an intermediate parent of
+				// this node and a new level0 into the parent.
+				uint64_t other = to_base(node->m_base);
+				uint8_t lcl = parent_level;
+				while (index_for(page_off, lcl) == index_for(other, lcl))
+					--lcl;
+
+				page_tree *inter = new page_tree(page_off, lcl);
+				inter->m_entries[index_for(other, lcl)].child = node;
+				*parent = inter;
+
+				level0 = new page_tree(page_off, 0);
+				inter->m_entries[index_for(page_off, lcl)].child = level0;
+				break;
+			}
+
+			if (!level) {
+				level0 = node;
+				break;
+			}
+
+			parent = &node->m_entries[index].child;
+			node = *parent;
+		}
+
+		kassert( node || parent, "Both node and parent were null in find_or_add");
+
+		if (!node) {
+			// We found a parent with an empty spot where this node should
+			// be. Insert a new level0 there.
+			level0 = new page_tree(page_off, 0);
+			*parent = level0;
+		}
+	}
+
+	kassert(level0, "Got through find_or_add without a level0");
+	uint8_t index = index_for(page_off, 0);
+	uint64_t &ent = level0->m_entries[index].entry;
+	if (!(ent & 1)) {
+		// No entry for this page exists, so make one
+		if (!frame_allocator::get().allocate(1, &ent))
+			return false;
+		ent |= 1;
+	}
+
+	page = ent & ~0xfffull;
+	return true;
+}

src/kernel/page_tree.h

@@ -0,0 +1,39 @@
+#pragma once
+/// \file page_tree.h
+/// Definition of mapped page tracking structure and related definitions
+
+#include <stdint.h>
+
+/// A radix tree node that tracks mapped pages
+class page_tree
+{
+public:
+	/// Get the physical address of the page at the given offset.
+	/// \arg root    The root node of the tree
+	/// \arg offset  Offset into the VMA, in bytes
+	/// \arg page    [out] Receives the page physical address, if found
+	/// \returns     True if a page was found
+	static bool find(const page_tree *root, uint64_t offset, uintptr_t &page);
+
+	/// Get the physical address of the page at the given offset. If one does
+	/// not exist yet, allocate a page, insert it, and return that.
+	/// \arg root    [inout] The root node of the tree. This pointer may be updated.
+	/// \arg offset  Offset into the VMA, in bytes
+	/// \arg page    [out] Receives the page physical address, if found
+	/// \returns     True if a page was found
+	static bool find_or_add(page_tree * &root, uint64_t offset, uintptr_t &page);
+
+private:
+	page_tree(uint64_t base, uint8_t level);
+
+	/// Stores the page offset of the start of this node's pages in bits 0:41
+	/// and the depth of tree this node represents in bits 42:44 (0-7)
+	uint64_t m_base;
+
+	/// For a level 0 node, the entries area all physical page addresses.
+	/// Other nodes contain pointers to child tree nodes.
+	union {
+		uintptr_t entry;
+		page_tree *child;
+	} m_entries[64];
+};

src/kernel/scheduler.cpp

@@ -7,6 +7,7 @@
 #include "console.h"
 #include "cpu.h"
 #include "debug.h"
+#include "device_manager.h"
 #include "gdt.h"
 #include "interrupts.h"
 #include "io.h"
@@ -27,16 +28,9 @@
 
 scheduler *scheduler::s_instance = nullptr;
 
-extern kernel::args::framebuffer *fb;
-
 const uint64_t rflags_noint = 0x002;
 const uint64_t rflags_int = 0x202;
 
-extern "C" {
-	void preloaded_process_init();
-	uintptr_t load_process_image(uintptr_t phys, uintptr_t virt, size_t bytes, TCB *tcb);
-};
-
 extern uint64_t idle_stack_end;
 
 scheduler::scheduler(lapic *apic) :
@@ -75,90 +69,13 @@ inline T * push(uintptr_t &rsp, size_t size = sizeof(T)) {
 	return p;
 }
 
-uintptr_t
-load_process_image(uintptr_t phys, uintptr_t virt, size_t bytes, TCB *tcb)
-{
-	using memory::page_align_down;
-	using memory::page_align_up;
-
-	// We're now in the process space for this process, allocate memory for the
-	// process code and load it
-	process &proc = process::current();
-	thread &th = thread::current();
-	vm_space &space = proc.space();
-
-	vm_area *vma = new vm_area_open(bytes, space, vm_flags::zero|vm_flags::write);
-	space.add(virt, vma);
-	vma->commit(phys, 0, memory::page_count(bytes));
-
-	// double zero stack sentinel
-	*push<uint64_t>(tcb->rsp3) = 0;
-	*push<uint64_t>(tcb->rsp3) = 0;
-
-	const char message[] = "Hello from the kernel!";
-	char *message_arg = push<char>(tcb->rsp3, sizeof(message));
-	kutil::memcpy(message_arg, message, sizeof(message));
-
-	j6_init_framebuffer *fb_desc = push<j6_init_framebuffer>(tcb->rsp3);
-	fb_desc->addr = fb ? reinterpret_cast<void*>(0x100000000) : nullptr;
-	fb_desc->size = fb ? fb->size : 0;
-	fb_desc->vertical = fb ? fb->vertical : 0;
-	fb_desc->horizontal = fb ? fb->horizontal : 0;
-	fb_desc->scanline = fb ? fb->scanline : 0;
-	fb_desc->flags = 0;
-
-	if (fb && fb->type == kernel::args::fb_type::bgr8)
-		fb_desc->flags |= 1;
-
-	j6_init_value *initv = push<j6_init_value>(tcb->rsp3);
-	initv->type = j6_init_handle_system;
-	initv->value = static_cast<uint64_t>(proc.add_handle(system::get()));
-
-	initv = push<j6_init_value>(tcb->rsp3);
-	initv->type = j6_init_handle_process;
-	initv->value = static_cast<uint64_t>(proc.self_handle());
-
-	initv = push<j6_init_value>(tcb->rsp3);
-	initv->type = j6_init_handle_thread;
-	initv->value = static_cast<uint64_t>(th.self_handle());
-
-	initv = push<j6_init_value>(tcb->rsp3);
-	initv->type = j6_init_handle_space;
-	//initv->value = static_cast<uint64_t>(proc.add_handle(&space));
-
-	initv = push<j6_init_value>(tcb->rsp3);
-	initv->type = j6_init_desc_framebuffer;
-	initv->value = reinterpret_cast<uint64_t>(fb_desc);
-
-	uint64_t *initc = push<uint64_t>(tcb->rsp3);
-	*initc = 5;
-
-	char **argv0 = push<char*>(tcb->rsp3);
-	*argv0 = message_arg;
-
-	uint64_t *argc = push<uint64_t>(tcb->rsp3);
-	*argc = 1;
-
-	// Crazypants framebuffer part
-	if (fb) {
-		vma = new vm_area_open(fb->size, space, vm_flags::write|vm_flags::mmio|vm_flags::write_combine);
-		space.add(0x100000000, vma);
-		vma->commit(fb->phys_addr, 0, memory::page_count(fb->size));
-	}
-
-	th.clear_state(thread::state::loading);
-	return tcb->rsp3;
-}
-
 thread *
 scheduler::create_process(bool user)
 {
 	process *p = new process;
 	thread *th = p->create_thread(default_priority, user);
 
-	auto *tcb = th->tcb();
-	tcb->time_left = quantum(default_priority);
-
+	TCB *tcb = th->tcb();
 	log::debug(logs::task, "Creating thread %llx, priority %d, time slice %d",
 			th->koid(), tcb->priority, tcb->time_left);
 
@@ -166,48 +83,6 @@ scheduler::create_process(bool user)
 	return th;
 }
 
-thread *
-scheduler::load_process(uintptr_t phys, uintptr_t virt, size_t size, uintptr_t entry)
-{
-
-	uint16_t kcs = (1 << 3) | 0; // Kernel CS is GDT entry 1, ring 0
-	uint16_t cs = (5 << 3) | 3;  // User CS is GDT entry 5, ring 3
-
-	uint16_t kss = (2 << 3) | 0; // Kernel SS is GDT entry 2, ring 0
-	uint16_t ss = (4 << 3) | 3;  // User SS is GDT entry 4, ring 3
-
-	thread* th = create_process(true);
-	auto *tcb = th->tcb();
-
-	// Create an initial kernel stack space
-	uintptr_t *stack = reinterpret_cast<uintptr_t *>(tcb->rsp0) - 9;
-
-	// Pass args to preloaded_process_init on the stack
-	stack[0] = reinterpret_cast<uintptr_t>(phys);
-	stack[1] = reinterpret_cast<uintptr_t>(virt);
-	stack[2] = reinterpret_cast<uintptr_t>(size);
-	stack[3] = reinterpret_cast<uintptr_t>(tcb);
-
-	tcb->rsp = reinterpret_cast<uintptr_t>(stack);
-	th->add_thunk_kernel(reinterpret_cast<uintptr_t>(preloaded_process_init));
-
-	// Arguments for iret - rip will be pushed on before these
-	stack[4] = reinterpret_cast<uintptr_t>(entry);
-	stack[5] = cs;
-	stack[6] = rflags_int | (3 << 12);
-	stack[7] = process::stacks_top;
-	stack[8] = ss;
-
-	tcb->rsp3 = process::stacks_top;
-
-	log::debug(logs::task, "Loading thread %llx  pri %d", th->koid(), tcb->priority);
-	log::debug(logs::task, "     RSP  %016lx", tcb->rsp);
-	log::debug(logs::task, "     RSP0 %016lx", tcb->rsp0);
-	log::debug(logs::task, "     PML4 %016lx", tcb->pml4);
-
-	return th;
-}
-
 void
 scheduler::create_kernel_task(void (*task)(), uint8_t priority, bool constant)
 {
@@ -237,12 +112,20 @@ scheduler::quantum(int priority)
 void
 scheduler::start()
 {
-	log::info(logs::task, "Starting scheduler.");
+	log::info(logs::sched, "Starting scheduler.");
 	wrmsr(msr::ia32_gs_base, reinterpret_cast<uintptr_t>(&bsp_cpu_data));
 	m_apic->enable_timer(isr::isrTimer, false);
 	m_apic->reset_timer(10);
 }
 
+void
+scheduler::add_thread(TCB *t)
+{
+	m_blocked.push_back(static_cast<tcb_node*>(t));
+	t->time_left = quantum(t->priority);
+
+}
+
 void scheduler::prune(uint64_t now)
 {
 	// Find processes that are ready or have exited and
@@ -279,7 +162,7 @@ void scheduler::prune(uint64_t now)
 				delete &p;
 		} else {
 			m_blocked.remove(remove);
-			log::debug(logs::task, "Prune: readying unblocked thread %llx", th->koid());
+			log::debug(logs::sched, "Prune: readying unblocked thread %llx", th->koid());
 			m_runlists[remove->priority].push_back(remove);
 		}
 	}
@@ -309,7 +192,7 @@ scheduler::check_promotions(uint64_t now)
 				tcb->priority -= 1;
 				tcb->time_left = quantum(tcb->priority);
 				m_runlists[tcb->priority].push_back(tcb);
-				log::debug(logs::task, "Scheduler promoting thread %llx, priority %d",
+				log::info(logs::sched, "Scheduler promoting thread %llx, priority %d",
 						th->koid(), tcb->priority);
 			}
 		}
@@ -331,7 +214,7 @@ scheduler::schedule()
 		if (priority < max_priority && !constant) {
 			// Process used its whole timeslice, demote it
 			++m_current->priority;
-			log::debug(logs::task, "Scheduler  demoting thread %llx, priority %d",
+			log::info(logs::sched, "Scheduler  demoting thread %llx, priority %d",
 					th->koid(), m_current->priority);
 		}
 		m_current->time_left = quantum(m_current->priority);
@@ -373,11 +256,11 @@ scheduler::schedule()
 		bsp_cpu_data.p = &next_thread->parent();
 		m_current = next;
 
-		log::debug(logs::task, "Scheduler switching threads %llx->%llx",
+		log::debug(logs::sched, "Scheduler switching threads %llx->%llx",
 				th->koid(), next_thread->koid());
-		log::debug(logs::task, "    priority %d time left %d @ %lld.",
+		log::debug(logs::sched, "    priority %d time left %d @ %lld.",
 				m_current->priority, m_current->time_left, m_clock);
-		log::debug(logs::task, "    PML4 %llx", m_current->pml4);
+		log::debug(logs::sched, "    PML4 %llx", m_current->pml4);
 
 		task_switch(m_current);
 	}

src/kernel/scheduler.h

@@ -5,6 +5,11 @@
 #include <stdint.h>
 #include "objects/thread.h"
 
+namespace kernel {
+namespace args {
+	struct program;
+}}
+
 class lapic;
 class process;
 struct page_table;
@@ -41,12 +46,9 @@ public:
 	scheduler(lapic *apic);
 
 	/// Create a new process from a program image in memory.
-	/// \arg phys  Physical address of the loaded program image
-	/// \arg virt  Virtual address of the loaded program image
-	/// \arg size  Size of the program image, in bytes
-	/// \arg entry Virtual address of the program entrypoint
-	/// \returns   The main thread of the loaded process
-	thread * load_process(uintptr_t phys, uintptr_t virt, size_t size, uintptr_t entry);
+	/// \arg program  The descriptor of the pogram in memory
+	/// \returns      The main thread of the loaded process
+	thread * load_process(kernel::args::program &program);
 
 	/// Create a new kernel task
 	/// \arg proc     Function to run as a kernel task
@@ -58,7 +60,7 @@ public:
 			bool constant = false);
 
 	/// Get the quantum for a given priority.
-	uint32_t quantum(int priority);
+	static uint32_t quantum(int priority);
 
 	/// Start the scheduler working. This may involve starting
 	/// timer interrupts or other preemption methods.
@@ -71,7 +73,9 @@ public:
 	/// \returns  A pointer to the current thread's TCB
 	inline TCB * current() { return m_current; }
 
-	inline void add_thread(TCB *t) { m_blocked.push_back(static_cast<tcb_node*>(t)); }
+	/// Start scheduling a new thread.
+	/// \arg t  The new thread's TCB
+	void add_thread(TCB *t);
 
 	/// Get a reference to the system scheduler
 	/// \returns  A reference to the global system scheduler

src/kernel/syscall.cpp

@@ -13,10 +13,6 @@ extern "C" {
 	void syscall_handler_prelude();
 }
 
-namespace syscalls {
-
-} // namespace syscalls
-
 uintptr_t syscall_registry[static_cast<unsigned>(syscall::MAX)];
 const char * syscall_names[static_cast<unsigned>(syscall::MAX)];
 
@@ -44,86 +40,9 @@ syscall_invalid(uint64_t call)
 	_halt();
 }
 
-/*
-void
-syscall_dispatch(cpu_state *regs)
-{
-	console *cons = console::get();
-	syscall call = static_cast<syscall>(regs->rax);
-
-	auto &s = scheduler::get();
-	auto *p = s.current();
-
-	switch (call) {
-	case syscall::noop:
-		break;
-
-	case syscall::debug:
-		cons->set_color(11);
-		cons->printf("\nProcess %d: Received DEBUG syscall\n", p->pid);
-		cons->set_color();
-		print_regs(*regs);
-		cons->printf("\n         Syscall enters: %8d\n", __counter_syscall_enter);
-		cons->printf("         Syscall sysret: %8d\n", __counter_syscall_sysret);
-		break;
-
-	case syscall::send:
-		{
-			pid_t target = regs->rdi;
-			uintptr_t data = regs->rsi;
-
-			cons->set_color(11);
-			cons->printf("\nProcess %d: Received SEND syscall, target %d, data %016lx\n", p->pid, target, data);
-			cons->set_color();
-
-			if (p->wait_on_send(target))
-				s.schedule();
-		}
-		break;
-
-	case syscall::receive:
-		{
-			pid_t source = regs->rdi;
-			uintptr_t data = regs->rsi;
-
-			cons->set_color(11);
-			cons->printf("\nProcess %d: Received RECEIVE syscall, source %d, dat %016lx\n", p->pid, source, data);
-			cons->set_color();
-
-			if (p->wait_on_receive(source))
-				s.schedule();
-		}
-		break;
-
-	case syscall::fork:
-		{
-			cons->set_color(11);
-			cons->printf("\nProcess %d: Received FORK syscall\n", p->pid);
-			cons->set_color();
-
-			pid_t pid = p->fork(regs);
-			cons->printf("\n   fork returning %d\n", pid);
-			regs->rax = pid;
-		}
-		break;
-
-	default:
-		cons->set_color(9);
-		cons->printf("\nReceived unknown syscall: %02x\n", call);
-		cons->set_color();
-		_halt();
-		break;
-	}
-}
-*/
-
 void
 syscall_enable()
 {
-	// IA32_EFER - set bit 0, syscall enable
-	uint64_t efer = rdmsr(msr::ia32_efer);
-	wrmsr(msr::ia32_efer, efer|1);
-
 	// IA32_STAR - high 32 bits contain k+u CS
 	// Kernel CS: GDT[1] ring 0 bits[47:32]
 	//   User CS: GDT[3] ring 3 bits[63:48]
@@ -150,7 +69,7 @@ syscall_enable()
 	syscall_names[id] = #name; \
 	static_assert( id <= num_calls, "Syscall " #name " has id > syscall::MAX" ); \
 	log::debug(logs::syscall, "Enabling syscall 0x%02x as " #name , id);
-#include "syscalls.inc"
+#include "j6/tables/syscalls.inc"
 #undef SYSCALL
 }
 

src/kernel/syscall.h

@@ -7,8 +7,8 @@ struct cpu_state;
 
 enum class syscall : uint64_t
 {
-#define SYSCALL(id, name, result, ...) name = id,
-#include "syscalls.inc"
+#define SYSCALL(id, name, ...) name = id,
+#include "j6/tables/syscalls.inc"
 #undef SYSCALL
 
 	// Maximum syscall id. If you change this, also change
@@ -21,6 +21,6 @@ void syscall_enable();
 namespace syscalls
 {
 #define SYSCALL(id, name, ...) j6_status_t name (__VA_ARGS__);
-#include "syscalls.inc"
+#include "j6/tables/syscalls.inc"
 #undef SYSCALL
 }

src/kernel/syscall.s

@@ -1,4 +1,3 @@
-%include "push_all.inc"
 %include "tasking.inc"
 
 ; Make sure to keep MAX_SYSCALLS in sync with
@@ -12,7 +11,6 @@ extern syscall_registry
 extern syscall_invalid
 
 global syscall_handler_prelude
-global syscall_handler_prelude.return
 syscall_handler_prelude:
 	swapgs
 	mov [gs:CPU_DATA.rsp3], rsp
@@ -55,7 +53,8 @@ syscall_handler_prelude:
 
 	inc qword [rel __counter_syscall_sysret]
 
-.return:
+global kernel_to_user_trampoline
+kernel_to_user_trampoline:
 	pop r15
 	pop r14
 	pop r13

src/kernel/syscalls/endpoint.cpp

@@ -36,7 +36,7 @@ endpoint_receive(j6_handle_t handle, j6_tag_t *tag, size_t *len, void *data)
 	if (!e) return j6_err_invalid_arg;
 
 	j6_tag_t out_tag = j6_tag_invalid;
-	size_t out_len = 0;
+	size_t out_len = *len;
 	j6_status_t s = e->receive(&out_tag, &out_len, data);
 	*tag = out_tag;
 	*len = out_len;
@@ -57,7 +57,7 @@ endpoint_sendrecv(j6_handle_t handle, j6_tag_t *tag, size_t *len, void *data)
 		return status;
 
 	j6_tag_t out_tag = j6_tag_invalid;
-	size_t out_len = 0;
+	size_t out_len = *len;
 	j6_status_t s = e->receive(&out_tag, &out_len, data);
 	*tag = out_tag;
 	*len = out_len;

src/kernel/syscalls/process.cpp

@@ -3,18 +3,56 @@
 
 #include "log.h"
 #include "objects/process.h"
+#include "syscalls/helpers.h"
 
 namespace syscalls {
 
 j6_status_t
-process_exit(int64_t status)
+process_create(j6_handle_t *handle)
+{
+	process *child = construct_handle<process>(handle);
+	log::debug(logs::task, "Process %llx created", child->koid());
+	return j6_status_ok;
+}
+
+j6_status_t
+process_start(j6_handle_t handle, uintptr_t entrypoint, j6_handle_t *handles, size_t handle_count)
 {
 	process &p = process::current();
-	log::debug(logs::syscall, "Process %llx exiting with code %d", p.koid(), status);
+	process *c = get_handle<process>(handle);
+	if (handle_count && !handles)
+		return j6_err_invalid_arg;
+
+	for (size_t i = 0; i < handle_count; ++i) {
+		kobject *o = p.lookup_handle(handles[i]);
+		if (o) c->add_handle(o);
+	}
+
+	return j6_err_nyi;
+}
+
+j6_status_t
+process_kill(j6_handle_t handle)
+{
+	process &p = process::current();
+	process *c = get_handle<process>(handle);
+	if (!c) return j6_err_invalid_arg;
+
+	log::debug(logs::task, "Process %llx killed by process %llx", c->koid(), p.koid());
+	c->exit(-1u);
+
+	return j6_status_ok;
+}
+
+j6_status_t
+process_exit(int32_t status)
+{
+	process &p = process::current();
+	log::debug(logs::task, "Process %llx exiting with code %d", p.koid(), status);
 
 	p.exit(status);
 
-	log::error(logs::syscall, "returned to exit syscall");
+	log::error(logs::task, "returned to exit syscall");
 	return j6_err_unexpected;
 }
 

src/kernel/syscalls/system.cpp

@@ -5,6 +5,8 @@
 #include "log.h"
 #include "objects/endpoint.h"
 #include "objects/thread.h"
+#include "objects/system.h"
+#include "objects/vm_area.h"
 #include "syscalls/helpers.h"
 
 extern log::logger &g_logger;
@@ -31,10 +33,17 @@ system_noop()
 }
 
 j6_status_t
-system_get_log(j6_handle_t sys, char *buffer, size_t *size)
+system_get_log(j6_handle_t sys, void *buffer, size_t *size)
 {
+	if (!size || (*size && !buffer))
+		return j6_err_invalid_arg;
+
+	size_t orig_size = *size;
 	*size = g_logger.get_entry(buffer, *size);
-	return j6_status_ok;
+	if (!g_logger.has_log())
+		system::get().deassert_signal(j6_signal_system_has_log);
+
+	return (*size > orig_size) ? j6_err_insufficient : j6_status_ok;
 }
 
 j6_status_t
@@ -50,4 +59,16 @@ system_bind_irq(j6_handle_t sys, j6_handle_t endp, unsigned irq)
 	return j6_err_invalid_arg;
 }
 
+j6_status_t
+system_map_mmio(j6_handle_t sys, j6_handle_t *vma_handle, uintptr_t phys_addr, size_t size, uint32_t flags)
+{
+	// TODO: check capabilities on sys handle
+	if (!vma_handle) return j6_err_invalid_arg;
+
+	vm_flags vmf = vm_flags::mmio | (static_cast<vm_flags>(flags) & vm_flags::user_mask);
+	construct_handle<vm_area_fixed>(vma_handle, phys_addr, size, vmf);
+
+	return j6_status_ok;
+}
+
 } // namespace syscalls

src/kernel/syscalls/thread.cpp

@@ -15,24 +15,24 @@ thread_create(void *rip, j6_handle_t *handle)
 
 	thread *child = p.create_thread();
 	child->add_thunk_user(reinterpret_cast<uintptr_t>(rip));
-	*handle = p.self_handle();
+	*handle = child->self_handle();
 	child->clear_state(thread::state::loading);
 	child->set_state(thread::state::ready);
 
-	log::debug(logs::syscall, "Thread %llx spawned new thread %llx, handle %d",
+	log::debug(logs::task, "Thread %llx spawned new thread %llx, handle %d",
 		parent.koid(), child->koid(), *handle);
 
 	return j6_status_ok;
 }
 
 j6_status_t
-thread_exit(int64_t status)
+thread_exit(int32_t status)
 {
 	thread &th = thread::current();
-	log::debug(logs::syscall, "Thread %llx exiting with code %d", th.koid(), status);
+	log::debug(logs::task, "Thread %llx exiting with code %d", th.koid(), status);
 	th.exit(status);
 
-	log::error(logs::syscall, "returned to exit syscall");
+	log::error(logs::task, "returned to exit syscall");
 	return j6_err_unexpected;
 }
 
@@ -48,7 +48,7 @@ j6_status_t
 thread_sleep(uint64_t til)
 {
 	thread &th = thread::current();
-	log::debug(logs::syscall, "Thread %llx sleeping until %llu", th.koid(), til);
+	log::debug(logs::task, "Thread %llx sleeping until %llu", th.koid(), til);
 
 	th.wait_on_time(til);
 	return j6_status_ok;

src/kernel/syscalls/vm_area.cpp

@@ -14,7 +14,7 @@ j6_status_t
 vma_create(j6_handle_t *handle, size_t size, uint32_t flags)
 {
 	vm_flags f = vm_flags::user_mask & flags;
-	construct_handle<vm_area_shared>(handle, size, f);
+	construct_handle<vm_area_open>(handle, size, f);
 	return j6_status_ok;
 }
 
@@ -22,28 +22,34 @@ j6_status_t
 vma_create_map(j6_handle_t *handle, size_t size, uintptr_t base, uint32_t flags)
 {
 	vm_flags f = vm_flags::user_mask & flags;
-	vm_area *a = construct_handle<vm_area_shared>(handle, size, f);
+	vm_area *a = construct_handle<vm_area_open>(handle, size, f);
 	process::current().space().add(base, a);
 	return j6_status_ok;
 }
 
 j6_status_t
-vma_map(j6_handle_t handle, uintptr_t base)
+vma_map(j6_handle_t handle, j6_handle_t proc, uintptr_t base)
 {
 	vm_area *a = get_handle<vm_area>(handle);
 	if (!a) return j6_err_invalid_arg;
 
-	process::current().space().add(base, a);
+	process *p = get_handle<process>(proc);
+	if (!p) return j6_err_invalid_arg;
+
+	p->space().add(base, a);
 	return j6_status_ok;
 }
 
 j6_status_t
-vma_unmap(j6_handle_t handle)
+vma_unmap(j6_handle_t handle, j6_handle_t proc)
 {
 	vm_area *a = get_handle<vm_area>(handle);
 	if (!a) return j6_err_invalid_arg;
 
-	process::current().space().remove(a);
+	process *p = get_handle<process>(proc);
+	if (!p) return j6_err_invalid_arg;
+
+	p->space().remove(a);
 	return j6_status_ok;
 }
 

src/kernel/task.s

@@ -55,8 +55,15 @@ task_switch:
 	ret
 
 
-extern syscall_handler_prelude.return
-global kernel_to_user_trampoline
-kernel_to_user_trampoline:
-	jmp syscall_handler_prelude.return
+extern initialize_main_user_stack
+extern kernel_to_user_trampoline
+global initialize_main_thread
+initialize_main_thread:
+	call initialize_main_user_stack
 
+	; user rsp is now in rax, put it in the right place for sysret
+	mov [rsp + 0x30], rax
+	mov [gs:CPU_DATA.rsp3], rax
+
+	; the entrypoint should already be on the stack
+	jmp kernel_to_user_trampoline

src/kernel/vm_mapper.cpp

@@ -1,108 +0,0 @@
-#include "objects/vm_area.h"
-#include "vm_mapper.h"
-#include "vm_space.h"
-
-
-vm_mapper_single::vm_mapper_single(vm_area &area, vm_space &space) :
-	m_area(area), m_space(space)
-{}
-
-vm_mapper_single::~vm_mapper_single()
-{
-	m_space.clear(m_area, 0, memory::page_count(m_area.size()), true);
-}
-
-bool
-vm_mapper_single::can_resize(size_t size) const
-{
-	return m_space.can_resize(m_area, size);
-}
-
-void
-vm_mapper_single::map(uintptr_t offset, size_t count, uintptr_t phys)
-{
-	m_space.page_in(m_area, offset, phys, count);
-}
-
-void
-vm_mapper_single::unmap(uintptr_t offset, size_t count)
-{
-	m_space.clear(m_area, offset, count, true);
-}
-
-void
-vm_mapper_single::remove(vm_space *space)
-{
-	size_t count = memory::page_count(m_area.size());
-	bool keep = m_area.flags() && vm_flags::mmio;
-	m_space.clear(m_area, 0, count, !keep);
-}
-
-
-
-vm_mapper_multi::vm_mapper_multi(vm_area &area) :
-	m_area(area)
-{
-}
-
-vm_mapper_multi::~vm_mapper_multi()
-{
-	if (!m_spaces.count())
-		return;
-
-	size_t count = memory::page_count(m_area.size());
-
-	for (int i = 1; i < m_spaces.count(); ++i)
-		m_spaces[i]->clear(m_area, 0, count);
-
-	m_spaces[0]->clear(m_area, 0, count, true);
-}
-
-bool
-vm_mapper_multi::can_resize(size_t size) const
-{
-	for (auto &it : m_spaces)
-		if (!it->can_resize(m_area, size))
-			return false;
-	return true;
-}
-
-void
-vm_mapper_multi::map(uintptr_t offset, size_t count, uintptr_t phys)
-{
-	for (auto &it : m_spaces)
-		it->page_in(m_area, offset, phys, count);
-}
-
-void
-vm_mapper_multi::unmap(uintptr_t offset, size_t count)
-{
-	for (auto &it : m_spaces)
-		it->clear(m_area, offset, count);
-}
-
-void
-vm_mapper_multi::add(vm_space *space)
-{
-	if (m_spaces.count()) {
-		vm_space *source = m_spaces[0];
-		space->copy_from(*source, m_area);
-	}
-
-	m_spaces.append(space);
-}
-
-void
-vm_mapper_multi::remove(vm_space *space)
-{
-	size_t count = memory::page_count(m_area.size());
-	bool keep = m_area.flags() && vm_flags::mmio;
-
-	for (int i = 0; i < m_spaces.count(); ++i) {
-		if (m_spaces[i] == space) {
-			m_spaces.remove_swap_at(i);
-			keep &= m_spaces.count() > 0;
-			space->clear(m_area, 0, count, !keep);
-		}
-	}
-}

src/kernel/vm_mapper.h

@@ -1,83 +0,0 @@
-#pragma once
-/// \file vm_mapper.h
-/// VMA to address space mapping interface and implementing objects
-
-#include <stdint.h>
-#include "kutil/vector.h"
-
-class vm_area;
-class vm_space;
-
-/// An interface to map vm_areas to one or more vm_spaces
-class vm_mapper
-{
-public:
-	virtual ~vm_mapper() {}
-
-	/// Check whether the owning VMA can be resized to the given size.
-	/// \arg size The desired size
-	/// \returns  True if resize is possible
-	virtual bool can_resize(size_t size) const = 0;
-
-	/// Map the given physical pages into the owning VMA at the given offset
-	/// \arg offset Offset into the VMA of the requested virtual address
-	/// \arg count  Number of contiguous physical pages to map
-	/// \arg phys   The starting physical address of the pages
-	virtual void map(uintptr_t offset, size_t count, uintptr_t phys) = 0;
-
-	/// Unmap the pages corresponding to the given offset from the owning VMA
-	/// \arg offset Offset into the VMA of the requested virtual address
-	/// \arg count  Number of pages to unmap
-	virtual void unmap(uintptr_t offset, size_t count) = 0;
-
-	/// Add the given address space to the list of spaces the owning VMA is
-	/// mapped to, if applicable.
-	virtual void add(vm_space *space) {}
-
-	/// Remove the given address space from the list of spaces the owning VMA
-	/// is mapped to, if applicable.
-	virtual void remove(vm_space *space) {}
-};
-
-
-/// A vm_mapper that maps a VMA to a single vm_space
-class vm_mapper_single :
-	public vm_mapper
-{
-public:
-	vm_mapper_single(vm_area &area, vm_space &space);
-	virtual ~vm_mapper_single();
-
-	virtual bool can_resize(size_t size) const override;
-	virtual void map(uintptr_t offset, size_t count, uintptr_t phys) override;
-	virtual void unmap(uintptr_t offset, size_t count) override;
-
-	vm_space & space() { return m_space; }
-
-	virtual void remove(vm_space *space) override;
-
-private:
-	vm_area &m_area;
-	vm_space &m_space;
-};
-
-
-/// A vm_mapper that maps a VMA to multiple vm_spaces
-class vm_mapper_multi :
-	public vm_mapper
-{
-public:
-	vm_mapper_multi(vm_area &area);
-	virtual ~vm_mapper_multi();
-
-	virtual bool can_resize(size_t size) const override;
-	virtual void map(uintptr_t offset, size_t count, uintptr_t phys) override;
-	virtual void unmap(uintptr_t offset, size_t count) override;
-	virtual void add(vm_space *space) override;
-	virtual void remove(vm_space *space) override;
-
-private:
-	vm_area &m_area;
-	kutil::vector<vm_space*> m_spaces;
-};
-

src/kernel/vm_space.cpp

@@ -7,7 +7,8 @@
 #include "vm_space.h"
 
 // The initial memory for the array of areas for the kernel space
-static uint64_t kernel_areas[16];
+constexpr size_t num_kernel_areas = 8;
+static uint64_t kernel_areas[num_kernel_areas * 2];
 
 int
 vm_space::area::compare(const vm_space::area &o) const
@@ -28,7 +29,7 @@ vm_space::area::operator==(const vm_space::area &o) const
 vm_space::vm_space(page_table *p) :
 	m_kernel {true},
 	m_pml4 {p},
-	m_areas {reinterpret_cast<vm_space::area*>(kernel_areas), 0, 8}
+	m_areas {reinterpret_cast<vm_space::area*>(kernel_areas), 0, num_kernel_areas}
 {}
 
 vm_space::vm_space() :
@@ -44,15 +45,15 @@ vm_space::vm_space() :
 
 vm_space::~vm_space()
 {
-	for (auto &a : m_areas)
-		a.area->mapper().remove(this);
+	for (auto &a : m_areas) {
+		bool free = a.area->remove_from(this);
+		clear(*a.area, 0, memory::page_count(a.area->size()), free);
+		a.area->handle_release();
+	}
 
 	kassert(!is_kernel(), "Kernel vm_space destructor!");
-
-	vm_space &kernel = kernel_space();
-
 	if (active())
-		kernel.activate();
+		kernel_space().activate();
 
 	// All VMAs have been removed by now, so just
 	// free all remaining pages and tables
@@ -70,7 +71,7 @@ vm_space::add(uintptr_t base, vm_area *area)
 {
 	//TODO: check for collisions
 	m_areas.sorted_insert({base, area});
-	area->mapper().add(this);
+	area->add_to(this);
 	area->handle_retain();
 	return true;
 }
@@ -80,8 +81,9 @@ vm_space::remove(vm_area *area)
 {
 	for (auto &a : m_areas) {
 		if (a.area == area) {
+			bool free = area->remove_from(this);
+			clear(*area, 0, memory::page_count(area->size()), free);
 			m_areas.remove(a);
-			area->mapper().remove(this);
 			area->handle_release();
 			return true;
 		}
@@ -259,27 +261,25 @@ vm_space::initialize_tcb(TCB &tcb)
 bool
 vm_space::handle_fault(uintptr_t addr, fault_type fault)
 {
-	uintptr_t page = addr & ~0xfffull;
-
 	// TODO: Handle more fult types
 	if (fault && fault_type::present)
 		return false;
 
 	uintptr_t base = 0;
 	vm_area *area = get(addr, &base);
-
-	if (!area || !area->allowed(page-base))
+	if (!area)
 		return false;
 
-	uintptr_t phys = 0;
-	size_t n = frame_allocator::get().allocate(1, &phys);
-	kassert(n, "Failed to allocate a new page during page fault");
+	uintptr_t offset = (addr & ~0xfffull) - base;
+	uintptr_t phys_page = 0;
+	if (!area->get_page(offset, phys_page))
+		return false;
 
+	void *mem = memory::to_virtual<void>(phys_page);
 	if (area->flags() && vm_flags::zero)
-		kutil::memset(memory::to_virtual<void>(phys), 0, memory::frame_size);
+		kutil::memset(mem, 0, memory::frame_size);
 
-	uintptr_t offset = page - base;
-	area->commit(phys, offset, 1);
+	page_in(*area, offset, phys_page, 1);
 	return true;
 }
 

src/kernel/vm_space.h

@@ -79,6 +79,13 @@ public:
 		fetch    = 0x10
 	};
 
+	/// Allocate pages into virtual memory. May allocate less than requested.
+	/// \arg virt  The virtual address at which to allocate
+	/// \arg count The number of pages to allocate
+	/// \arg phys  [out] The physical address of the pages allocated
+	/// \returns   The number of pages actually allocated
+	size_t allocate(uintptr_t virt, size_t count, uintptr_t *phys);
+
 	/// Handle a page fault.
 	/// \arg addr  Address which caused the fault
 	/// \arg ft    Flags from the interrupt about the kind of fault
@@ -98,7 +105,7 @@ public:
 	static size_t copy(vm_space &source, vm_space &dest, void *from, void *to, size_t length);
 
 private:
-	friend class vm_mapper_single;
+	friend class vm_area;
 	friend class vm_mapper_multi;
 
 	/// Find a given VMA in this address space

src/libraries/cpu/include/cpu/features.inc

@@ -13,6 +13,7 @@ CPU_FEATURE_REQ(pat,        0x00000001, 0, edx, 16)
 CPU_FEATURE_REQ(fxsr,       0x00000001, 0, edx, 24)
 
 CPU_FEATURE_OPT(fsgsbase,   0x00000007, 0, ebx,  0)
+CPU_FEATURE_OPT(bmi1,       0x00000007, 0, ebx,  3)
 CPU_FEATURE_OPT(invpcid,    0x00000007, 0, ebx, 10)
 
 CPU_FEATURE_OPT(pku,        0x00000007, 0, ecx,  3)

src/libraries/j6/include/j6/syscalls.h

@@ -6,8 +6,8 @@
 extern "C" {
 #endif
 
-#define SYSCALL(n, name, ...) j6_status_t _syscall_ ## name (__VA_ARGS__);
-#include "syscalls.inc"
+#define SYSCALL(n, name, ...) j6_status_t j6_ ## name (__VA_ARGS__);
+#include "j6/tables/syscalls.inc"
 #undef SYSCALL
 
 #ifdef __cplusplus

src/libraries/j6/syscalls.s

@@ -1,6 +1,6 @@
 %macro SYSCALL 2
-	global _syscall_%1
-	_syscall_%1:
+	global j6_%1
+	j6_%1:
 		push rbp
 		mov rbp, rsp
 
@@ -25,4 +25,4 @@
 %define SYSCALL(n, name, a, b, c, d) SYSCALL name, n
 %define SYSCALL(n, name, a, b, c, d, e) SYSCALL name, n
 
-%include "syscalls.inc"
+%include "j6/tables/syscalls.inc"

src/libraries/kutil/heap_allocator.cpp

@@ -62,10 +62,13 @@ private:
 };
 
 
-heap_allocator::heap_allocator() : m_next(0), m_size(0) {}
+heap_allocator::heap_allocator() : m_start {0}, m_end {0} {}
 
 heap_allocator::heap_allocator(uintptr_t start, size_t size) :
-	m_next(start), m_size(size), m_allocated_size(0)
+	m_start {start},
+	m_end {start+size},
+	m_blocks {0},
+	m_allocated_size {0}
 {
 	kutil::memset(m_free, 0, sizeof(m_free));
 }
@@ -97,6 +100,10 @@ heap_allocator::free(void *p)
 {
 	if (!p) return;
 
+	uintptr_t addr = reinterpret_cast<uintptr_t>(p);
+	kassert(addr >= m_start && addr < m_end,
+		"Attempt to free non-heap pointer");
+
 	mem_header *header = reinterpret_cast<mem_header *>(p);
 	header -= 1; // p points after the header
 	header->set_used(false);
@@ -133,12 +140,12 @@ heap_allocator::ensure_block(unsigned order)
 
 	if (order == max_order) {
 		size_t bytes = (1 << max_order);
-		if (bytes <= m_size) {
-			mem_header *next = reinterpret_cast<mem_header *>(m_next);
-			new (next) mem_header(nullptr, nullptr, order);
-			get_free(order) = next;
-			m_next += bytes;
-			m_size -= bytes;
+		uintptr_t next = m_start + m_blocks * bytes;
+		if (next + bytes <= m_end) {
+			mem_header *nextp = reinterpret_cast<mem_header *>(next);
+			new (nextp) mem_header(nullptr, nullptr, order);
+			get_free(order) = nextp;
+			++m_blocks;
 		}
 	} else {
 		mem_header *orig = pop_free(order + 1);

src/libraries/kutil/include/kutil/enum_bitfields.h

@@ -131,10 +131,13 @@ operator ! (E rhs)
 }
 
 template <typename E>
-constexpr typename std::enable_if<is_enum_bitfield<E>::value,bool>::type
+constexpr bool
 bitfield_has(E set, E flag)
 {
-	return (set & flag) == flag;
+	return
+		(static_cast<typename std::underlying_type<E>::type>(set) &
+		 static_cast<typename std::underlying_type<E>::type>(flag)) ==
+		 static_cast<typename std::underlying_type<E>::type>(flag);
 }
 
 // Overload the logical-and operator to be 'bitwise-and, bool-cast'

src/libraries/kutil/include/kutil/heap_allocator.h

@@ -23,11 +23,11 @@ public:
 	/// \arg length  The amount of memory to allocate, in bytes
 	/// \returns     A pointer to the allocated memory, or nullptr if
 	///              allocation failed.
-	virtual void * allocate(size_t length);
+	void * allocate(size_t length);
 
 	/// Free a previous allocation.
 	/// \arg p  A pointer previously retuned by allocate()
-	virtual void free(void *p);
+	void free(void *p);
 
 	/// Minimum block size is (2^min_order). Must be at least 6.
 	static const unsigned min_order = 6;
@@ -52,8 +52,8 @@ protected:
 	/// \returns    A detached block of the given order
 	mem_header * pop_free(unsigned order);
 
-	uintptr_t m_next;
-	size_t m_size;
+	uintptr_t m_start, m_end;
+	size_t m_blocks;
 	mem_header *m_free[max_order - min_order + 1];
 	size_t m_allocated_size;
 

src/libraries/kutil/include/kutil/logger.h

@@ -19,17 +19,20 @@ class logger
 {
 public:
 	/// Callback type for immediate-mode logging
-	typedef void (*immediate)(area_t, level, const char *);
+	typedef void (*immediate_cb)(area_t, level, const char *);
+
+	/// Callback type for log flushing
+	typedef void (*flush_cb)();
 
 	/// Default constructor. Creates a logger without a backing store.
 	/// \arg output  Immediate-mode logging output function
-	logger(immediate output = nullptr);
+	logger(immediate_cb output = nullptr);
 
 	/// Constructor. Logs are written to the given buffer.
 	/// \arg buffer  Buffer to which logs are written
 	/// \arg size    Size of `buffer`, in bytes
 	/// \arg output  Immediate-mode logging output function
-	logger(uint8_t *buffer, size_t size, immediate output = nullptr);
+	logger(uint8_t *buffer, size_t size, immediate_cb output = nullptr);
 
 	/// Register a log area for future use.
 	/// \arg area      The key for the new area
@@ -38,7 +41,10 @@ public:
 	void register_area(area_t area, const char *name, level verbosity);
 
 	/// Register an immediate-mode log callback
-	inline void set_immediate(immediate cb) { m_immediate = cb; }
+	inline void set_immediate(immediate_cb cb) { m_immediate = cb; }
+
+	/// Register a flush callback
+	inline void set_flush(flush_cb cb) { m_flush = cb; }
 
 	/// Get the default logger.
 	inline logger & get() { return *s_log; }
@@ -77,9 +83,13 @@ public:
 	/// Get the next log entry from the buffer
 	/// \arg buffer  The buffer to copy the log message into
 	/// \arg size    Size of the passed-in buffer, in bytes
-	/// \returns     Number of bytes copied into the buffer
+	/// \returns     The size of the log entry (if larger than the
+	///              buffer, then no data was copied)
 	size_t get_entry(void *buffer, size_t size);
 
+	/// Get whether there is currently data in the log buffer
+	inline bool has_log() const { return m_buffer.size(); }
+
 private:
 	friend void debug(area_t area, const char *fmt, ...);
 	friend void info (area_t area, const char *fmt, ...);
@@ -95,7 +105,8 @@ private:
 	static const unsigned num_areas = 1 << (sizeof(area_t) * 8);
 	uint8_t m_levels[num_areas / 2];
 	const char *m_names[num_areas];
-	immediate m_immediate;
+	immediate_cb m_immediate;
+	flush_cb m_flush;
 
 	uint8_t m_sequence;
 
@@ -115,7 +126,7 @@ void fatal(area_t area, const char *fmt, ...);
 
 namespace logs {
 #define LOG(name, lvl) extern const log::area_t name;
-#include "log_areas.inc"
+#include "j6/tables/log_areas.inc"
 #undef LOG
 } // namespace logs