- Extract parts that are common for all hosts
- Remove some outdated/unnecessary options, we should now be
closer to default configuration
- Disable libthread_db loading (does not work and crashes GDB 9.2)
- Disable pagination when loading debug maps
- Use the same mechanism (debug_map) in Pal/Linux and Pal/Linux-SGX.
Previously, Pal/Linux emulated the _r_debug structure, normally
maintained by ld.so, but that cannot be done in SGX outer PAL,
because it's loaded by ld.so already.
- Maintain the debug maps outside of SGX enclave. This allows
initializing them before enclave start, and potentially makes
them easier to use.
- Initialize PAL debug map before enclave start. Previously, this
was done from inside the enclave, so you couldn't set a
breakpoint too early (e.g. in pal_linux_main).
- Store only load address, without list of sections. This is to
avoid parsing the list of sections just to report them to the
debugger. Unfortunately, the GDB version that we support still
needs these sections, but we can retrieve them in GDB plugin.
- Move Python GDB code related to debug maps to a common file.
This is a major refactor of the way manifests are loaded and handled,
which will be followed by a complete rework of the loader code (which
will include e.g. centralized config).
Changes/fixes:
- Huge part of manifest handling was refactored and untangled.
- Starting without a manifest is now disallowed. This was actually
accidentally broken for some time and no one complained. It also makes
little sense in practice and in Graphene's overall design, e.g. it
conflicts with protected argv.
- Now we only allow starting by giving the executable, not manifest (the
magic resolution logic was removed).
- Now manifests are sent over pipes between parent and children, instead
of children finding and loading them on their own. This is a
preparation for the upcoming centralized manifests change.
- Previously manifests were parsed 2 times on Linux and 3 times on
Linux-SGX (by untrusted PAL, trusted PAL and LibOS). This is now
fixed.
- The common `pal_main()` now requires that the backend-specific PAL
loader loads the manifest before calling it. SGX code already has to
do it (for proper initialization), so let's unify this interface for
all PALs.
- Fix for a PAL crash when manifest size was divisible by page size
(sic!). NULL termination was missing, but most of the time the padding
to page size saved Graphene from crashing.
Pylint output was filtered so that many files with existing pylint
violations were allowed to stay broken.
I made sure all files pass pylint, but whitelisted some rules that
we commonly disable:
* missing docstrings: most of the code is tests/internal anyway
* invalid-name: too many violations, and we commonly use one- or
two-character names (like "a, b" or "t1, t2") which is
disallowed by this rule; we could tweak it and then fix
remaining violations such as camel-case or lowercase constants
* fixme: we leave TODOs as a matter of practice, same as in C
* high-level style rules like too-few-* and too-many-*,
no-self-use
Hopefully that will make using pylint less annoying, while also
catching serious issues (such as unused variables or imports).
Graphene had some paths to internal files generated at compile time and
hardcoded into the output binary, which disallowed e.g. moving the
Graphene directory after compilation.
Protected files (PF) are a new type of file that can be specified in
the manifest (SGX only). They are encrypted on disk and transparently
decrypted when accessed by the Graphene payload.
Other features:
- data is integrity protected (tamper resistance)
- file swap protection (a PF can only be accessed when in a specific path)
- transparency (Graphene payload sees PFs as regular files, no need to modify
the payload)
See Linux-SGX/protected-files directory for implementation. PF format is
based on protected files from the SGX SDK:
https://github.com/intel/linux-sgx/tree/master/sdk/protected_fs
The following new manifest elements are added:
sgx.protected_files_key = <16-byte hex value>
sgx.protected_files.<name> = file:<host path>
sgx.protected_files_key specifies the encryption key and is only a temporary
implementation. This key should be provisioned with local/remote attestation
in the future.
Paths specifying PF entries can be files or directories. If a directory is
specified, all files/directories within are registered as protected
recursively (and are expected to be encrypted in the PF format).
Linux-SGX/tools directory contains the pf_crypt utility that converts files
to/from the protected format.
For detection of SGX/non-SGX (for example in regression tests) always
use the SGX environment variable. To generate launch/EINIT tokens use
the new 'sgx-tokens' Make target.
- Deduplicate much of setup code.
- Allow running non-sandbox and sandbox code in single run.
- Use pytest.
- Generate JUnit-XML file for Jenkins.
- Document running a subset of regression tests.
This fixes two problems with subprocess handling:
1. Use subprocess.communicate to avoid that the pipe can fill up.
2. Use a process group to ensure that we also kill the subprocess'
children in case of a timeout.
Note: This does not cover the case that the child returns successfully
but leaves other child processes running. AFAICS this is not easy since
subprocess.communicate() wait(2)s for the process so the pid is no
longer guaranteed to be valid to kill. So we leave this to the outside
environment (Jenkins or whatever).
The regression python scripts are not marked executable. Also they
expect that the Makefile has setup the environment correctly so just
marking them executable would not be useful. So drop the unused shebang
to avoid encoding the interpreter in multiple places.
Script/regression.py is a library so it also doesn't need a shebang.
- Fix a severe bug caused by merging #87. The patch does not check the return values from get_config_entries_size(), while the return values can easily be negative (-PAL_ERROR_INVAL). Any usage of this function should carefully check the return values and also pass the size into get_config_entries() to prevent buffer overflow.
- Fix a minor bug in handling IPC disconnection. The callback ipc_child_exit() never receives a "term_signal" parameter. The callback is called when thepipe to a child process unexpectedly terminates, which can only mean the child process has crashed. I believe the proper signal to send is SIGKILL.
- Remove a compilation warning caused by type mismatch of __malloc().
- Allow binding TCP socket to ANY address
- enable LD_PRELOAD in glibc
- disable output buffering in LTP tests to reveal omitted passes
- Remove a double-unlock in shim_async.c
- Replace all int with size_t or ssize_t in Graphene configuration API
- Add new passed LTP tests
* Fix a PAL unit test
* Make the PAL unit tests fail properly if one that should work doesn't.
* Get consistent indexes with and without graphene ipc module
* A little debugging output for ltp flakiness. Seems to make things a little more stable.
* A bugfix for issue #80. The migration is not complete when a file-backed VMA is larger than
the file size, but the file size is not page-aligned. In Linux, if the file size is smaller
than the mapped memory, accessing the remaining memory that is not backed by the file will
trigger a SIGBUS. However, it is fine to access the remaining memory within the last page that
still overlaps with file, and won't trigger a SIGBUS. This area is commonly used in ELF binary
to store uninitialized, static variables. To improve fork latency, Graphene chooses to
truncate the migration size as the file size, but missed the memory which belongs to the last
file-backed page. The migration size should be aligned up to the page size.
* Fix assertion lines in PAL and LibOS
* Make sure the return code is correct for the LTP script, bump up some timeouts
* Only run gipc tests when the module is loaded, for the purposes of getting CI to work.
* Build fix
Paweł Marczewski