Change log (most important only):
- unify CPU context structures - now we have only one version -
`PAL_CONTEXT` - which is shared between LibOS and PALs and it should
depend only on the host architecture (not OS),
- syscalls emulation changed:
- dedicated LibOS stack is now used for syscalls emulation,
- removed one indirection level in syscalls table - now it stores
`shim_do_*` functions directly,
- signal handling - completely rewritten:
- all signal queues use proper locking schemes now,
- signals are handled *only* when returning to the user app from LibOS
or PAL,
- nested signals are now possible,
- the app is allowed to jump out of signal handler with the same
sematics as on normal Linux,
- signal altstack is now fully supported,
- syscall restarting is now supported,
- doing a backtrace from the signal handler works properly,
- disallow injecting host-level signals, with one exception, see
`sys.enable_sigterm_injection` manifest option for more details.
This commit also fixes `pseudo_name_ops::list_name()` function pointer
type: `size_t len` argument instead of `int len`. It also adds a
regression test to exercise the newly supported /sys pseudo filesystem.
Instead of 'loader.debug_type', introduce 'loader.log_level'
and 'loader.log_file', along with a set of definitions for
logging at a chosen level.
For now, the call sites keep using the legacy macros (SGX_DBG and
debug()), because converting them all will conflict with other
big changes in the code base. The existing LibOS calls are
assumed to be at 'info' level.
- Use the same mechanism (debug_map) in Pal/Linux and Pal/Linux-SGX.
Previously, Pal/Linux emulated the _r_debug structure, normally
maintained by ld.so, but that cannot be done in SGX outer PAL,
because it's loaded by ld.so already.
- Maintain the debug maps outside of SGX enclave. This allows
initializing them before enclave start, and potentially makes
them easier to use.
- Initialize PAL debug map before enclave start. Previously, this
was done from inside the enclave, so you couldn't set a
breakpoint too early (e.g. in pal_linux_main).
- Store only load address, without list of sections. This is to
avoid parsing the list of sections just to report them to the
debugger. Unfortunately, the GDB version that we support still
needs these sections, but we can retrieve them in GDB plugin.
- Move Python GDB code related to debug maps to a common file.
This is the next part of the great loader rework, with a lot of breaking changes:
- Complete removal of the "trusted children" thing - now children
processes can be spawned arbitrarily and from arbitrary mountpoint
types, without any additional configuration needed.
- There's a new, required option in the manifest: `libos.entrypoint` - it
specifies the URI to the entry binary in the first process. There's no
need anymore to name the manifest and the first binary identically.
- On SGX, the main binary is not measured in MRENCLAVE anymore - only
PAL, LibOS and the manifest are measured. This is enough to bind
MRENCLAVE to a specific entrypoint user executable if wanted - it
just has to be mounted as a trusted file.
- All Graphene SGX enclaves have now exactly the same MRENCLAVE. This is
a hash of a "Graphene stub", which can "fork" into one of two states
in runtime: initial process or child. The initial process creates a
new "Graphene namespace" with a clean state, it can also be attested
remotely (contrary to child processes). The initial process can spawn
children processes by spawning a Graphene stub and directing it to
start in the child mode. It then attests it locally, and if
successful, establishes an encrypted pipe, "connects" to its own
namespace and treats as trusted (including sending protected files
key).
- Now, there's only one, central manifest describing the initial state
of a Graphene instance which can be spawned from it (previously, each
process required a separate manifest which could have different
configuration - which wasn't actually supported and didn't make sense
design-wise). One downside of central manifests is that all processes
require the same enclave configuration (e.g. size), but that was
already the case so far because of broken checkpointing code. Also,
this is only a temporary problem, which will cease to exist after the
introduction of EDMM.
- `sgx.static_address` was renamed to `sgx.nonpie_binary` and now has to
be inserted manually by users (`sgx_sign` tools doesn't know about the
binaries run inside, which can be even provided or generated in
runtime by the user's workload).
- Caveat: the memory gap for non-PIE executables was removed because it
requires adding a new option to the manifest to be cleanly
implemented. This is left for some future loader rework PR.
Previously, if futex wait returned -EINTR it was returned from
`_DkEventWaitTimeout` too. Now if the waiting condition no longer holds
we ignore EINTR and treat that as a successful wait.
This commit also removes redundant, copy-pasted code (_DkEventWait) and
unused Pal API function (DkEventWait).
This function was empty and it is responsibility of the caller (Pal
level) of specific exception handling function (LibOS level) to return
from the exception.
The profiling system instruments the asynchronous enclave exit
(AEX). Depending on configuration, we either snapshot the IP
value, or dump registers and (portion of) stack. The 'perf report'
tool can be used to generate a report from the samples.
These functions (despite documentation comment saying exactly the
opposite) accepted only positive error values, but most of the callsites
passed negative error codes instead. This resulted in Graphene printing
"Unknown error" instead of a proper error details.
This is a major refactor of the way manifests are loaded and handled,
which will be followed by a complete rework of the loader code (which
will include e.g. centralized config).
Changes/fixes:
- Huge part of manifest handling was refactored and untangled.
- Starting without a manifest is now disallowed. This was actually
accidentally broken for some time and no one complained. It also makes
little sense in practice and in Graphene's overall design, e.g. it
conflicts with protected argv.
- Now we only allow starting by giving the executable, not manifest (the
magic resolution logic was removed).
- Now manifests are sent over pipes between parent and children, instead
of children finding and loading them on their own. This is a
preparation for the upcoming centralized manifests change.
- Previously manifests were parsed 2 times on Linux and 3 times on
Linux-SGX (by untrusted PAL, trusted PAL and LibOS). This is now
fixed.
- The common `pal_main()` now requires that the backend-specific PAL
loader loads the manifest before calling it. SGX code already has to
do it (for proper initialization), so let's unify this interface for
all PALs.
- Fix for a PAL crash when manifest size was divisible by page size
(sic!). NULL termination was missing, but most of the time the padding
to page size saved Graphene from crashing.
Logging to file was broken, because the PAL file write operation
required the user to provide an absolute offset, and LibOS always
provided an offset of 0. This worked when logging to stdout, but
in case of a regular file, it kept overwriting the beginning of
file.
To fix that, we introduce a a special DkDebugLog call. This is a
better solution than tracking the file offset manually, because
the offset would need to be synchronized across different threads
and processes, and debug logs should be as simple as possible. At
the same time, we don't want PAL to provide a generic "append to
a file" mechanism, because it makes I/O less deterministic.
Previously, process communication (channel between parent and newly
created child) was protected via TLS only during send/receive of the
checkpoint; after that the channel was downgraded from TLS to
plaintext. The reason for this downgrade is historical (IPC was
complicated, and we wanted to have at least some TLS at the time).
This commit fixes this issue: Graphene now always uses TLS on IPC.
I don't know any reason why would stating the file name we're in be
helpful for anything. Moreover, this information was incorrect in a few
cases (copy-paste bugs, probably).
Additionally, a few minor type/formatting fixes included.
Previously, we introduced `sgx.zero_heap_on_demand` in Linux-SGX as a
knob to trade off runtime degradation on memory allocations for faster
enclave start-up times. This was an incorrect fix because Linux-SGX's
`_DkVirtualMemoryAlloc()` always zeroess the requested memory region,
so there was a double-zero of the heap at runtime. Note that LibOS
layer silently assumes that `_DkVirtualMemoryAlloc()` zeroes out the
memory, and many applications rely on this (Apache, Blender in my
experiments). Thus, this commit keeps the zero-out in
`_DkVirtualMemoryAlloc()` and removes zero-outs on enclave init and in
`get_enclave_pages()`. This renders `sgx.zero_heap_on_demand`
useless, so this manifest option is also removed. Also note that this
commit doesn't introduce any performance degradation (in fact, now
Graphene behaves as if `sgx.zero_heap_on_demand = 1` always).
It turns out that our guard pages were incorrectly handled (i.e. not re-
added to LibOS VMA list) on SGX when execve was optimized to re-use the
same enclave, which caused exec_same test to crash from time to time
(when ASLR put heap on a guard page).
Static guard pages aren't too useful and introduce unnecessary
complexity to our code, so we decided to just delete them in order to
fix this bug.
This commit adds the ability to provision the wrap (master) key for
protected files at runtime (in contrast to previous approach of
hard-coding `protected_files_key` in the manifest as a temporary
solution). This is achieved as follows:
- New PAL API `DkSetProtectedFilesKey()` is added.
- New writable pseudo-file `/dev/attestation/protected_files_key` is
added. It calls `DkSetProtectedFilesKey()` after it was written to.
- New `SECRET_PROVISION_SET_PF_KEY` option is added to the Secret
Provisioning library. If it is set, the library assumes that the
first provisioned secret is the wrap key for PF and writes it into
the new pseudo-file.
The Secret Provisioning example `ra-tls-secret-prov` is updated to
include the new protected-files client. This client receives the wrap
key for PF via secret provisioning and reads & outputs the protected
file `files/input.txt`.
*NOTE*: The current implementation of provisioning the wrap key does
not work for `loader.argv_src_file` and `loader.env_src_file` if they
point to protected files (because provisioning happens after setting
up arguments and environment variables).
Previously, Graphene always performed ASLR at the LibOS layer. ASLR
may lead to a situation when one mmap allocates an object in the
middle of address space, and there is no space for a later mmap of
a large object. This is problematic in restricted environments such
as SGX enclaves. In particular, `large_mmap` LibOS test failed
occasionally because it only has 8GB of enclave size and it may
mmap first objects somewhere in the middle (around 4GB address) and
then fail to find any space for a large 4GB mmap.
This commit adds "loader.insecure__disable_aslr" manifest option.
If it set to one, ASLR is disabled and mappings become deterministic
which guarantees programs like `large_mmap` never fail due to ENOMEM.
Previously, Graphene simply forwarded SIGPIPE generated by the host to
LibOS/app. Unfortunately, SIGPIPE generation is a process-wide feature
and there is no portable way to restrict it only to a subset of pipes,
UNIX domain sockets, etc. This led to sporadic Graphene failures
because Graphene's internal use of pipes and sockets may result in an
unexpected (to application) SIGPIPE.
This commit removes the forwarding of SIGPIPE. Instead, PALs explicitly
ignore SIGPIPE. This forces the host to return EPIPE error code, which
is checked only on a subset of LibOS handles (the ones created by the
app), and if required, LibOS generates a SIGPIPE for the application.
While adding this logic, the whole PAL exception code was refactored,
both in Linux and Linux-SGX. Tests for SIGPIPE are now enabled for
both Linux and Linux-SGX PALs.
This patch adds a new PAL_EVENT_PIPE to handle EPIPE signals
and forward them to the signal handler installed in LibOS.
If the application does not have a signal handler installed,
it will terminate the application with SIGPIPE exit code.
Support for SIGPIPE in Linux-SGX PAL will be added in a follow-up
commit.
Introduce PAL_ERROR_CONNFAILED_PIPE and treat EPIPE separately
from ECONNRESET.
The effects of this patch on LTP are:
from:
writev01.c:139: FAIL: write to closed pipe, expected: -1 (EPIPE), got: -1 (ECONNRESET)
to:
writev01.c:139: PASS: write to closed pipe, expected: -1 (EPIPE), got: -1 (EPIPE)
AND:
from:
write05.c:82: FAIL: write() failed unexpectedly, expected EPIPE: ECONNRESET
to:
write05.c:87: FAIL: sigpipe_cnt = 0
writev01 now works correctly, so this commit enables it.
This commit completely reworks VMA subsystem along with its usages.
New version should be: cleaner (easier to maintain), faster and allow
for bookkeeping requests from Pal.
It also fixes some bugs and inconsistencies found in the process and
changes brk and mmap/munmap implementations (at least partially).
Currently various flags in file and memory syscalls work mostly by an
accident, because values of some of them align with corresponding Linux
syscall flags. Some APIs weren't that lucky though - e.g.
DkStreamOpen(..., /*options=*/PAL_OPTION_CLOEXEC) deletes file contents
(sic!) intead of opening it with O_CLOEXEC. This is because
PAL_OPTION_CLOEXEC == O_TRUNC.
This commit fixes all this mess and also adds asserts to check validity
of flags passed to Dk* handlers.
New DkAttestationReport() API retrieves the attestation report (local)
from the local attestation mechanism. Currently, it is implemented
only for Linux-SGX PAL and stubbed for all other PALs. The Linux-SGX
implementation retrieves the SGX report via EREPORT instruction.
The caller of this new API may learn sizes of report_data, target_info,
and report structs. The caller may also obtain current target info.
Finally, the caller may obtain the report for use in local attestation.
A simple test case is added to PAL tests.
Previously, Graphene used the notation "pipe:<uint32_t>" to emulate
pipes, socketpairs, and UNIX domain sockets. In particular, pipes
and socketpairs received random integer IDs, and sockets received
deterministic integer IDs. However, 32-bit randomly generated IDs
may collide quite often. Since pipe IDs/names should *not* repeat
(otherwise e.g. derived crypto keys will be reused), this commit
changes pipe IDs (pipeid) from uint32_t to char[96], and pipe IDs
(names) become 256-bit random sequences.
New DkAttestationQuote() API retrieves the attestation quote from the
underlying host-OS attestation mechanism. Currently, it is implemented
only for Linux-SGX PAL and stubbed for all other PALs. The Linux-SGX
implementation retrieves the SGX quote via sgx_get_quote() which
communicates with the Quoting Enclave via AESM service. The caller of
this new API may forward the obtained quote to the remote user for
remote attestation.
Previously, Graphene with SGX performed self-remote attestation
by retrieving the SGX quote from the Quoting Enclave, sending it
to the Intel Attestation Service via Curl (in an HTTPS request),
and parsing the received from IAS remote-attestation certificate.
This self-attestation functionality is meaningless and is removed.
Moreover, since EPID-based remote attestation requires client key
(Ocp-Apim-Subscription-Key), and this key must be kept secret,
specifying it in the Graphene manifest (as was done previously) is
insecure. Therefore, the whole remote attestation functionality is
moved out of Graphene and to another trusted party (or to the app
on top of Graphene). Only the SGX quote retrieval from the Quoting
Enclave is kept in Graphene.
Previously, Graphene in execve() emulation silently terminated the
intermediate process after sending all necessary metadata. For example,
`bash -c ls` would spawn (host clone+execve) a new Graphene process with
`ls` and exit the "intermediate" process (`bash`) immediately, without
waiting for `ls` to finish. This deviation from standard execve()
behavior resulted in the host shell becoming detached from the Graphene
process.
This commit simply forces this intermediate process to wait() until
the child terminates. This way the host shell stays attached, and
all timings and pipes behave correctly.
This patch auto-generates asm offsets for Pal/Linux-SGX and uses them.
Also to share the generation logic among Pal and LibOS, header file for
it is created under Pal.
Signed-off-by: Isaku Yamahata <isaku.yamahata@gmail.com>
Borys Popławski