When child thread exits, it wakes up its parent if CLONE_CHILD_CLEARTID was set
during clone() call. Previously, this was done by the child thread itself as
part of its own clean-up in release_clear_child_id(). But this child thread is
still alive at this point and uses some resources, most notably the stack (that
might have been provided by the parent) and the SGX TCS slot. Upon waking up,
the parent might decide to free that stack (as Pthreads do) or re-use the TCS
slot, causing data races.
This commit introduces a correct emulation of CLONE_CHILD_CLEARTID:
- A new argument `PAL_PTR clear_child_tid` is added to DkThreadExit();
it points to internal Graphene memory that is erased on child exit to notify
Async Helper thread.
- At PAL layer, when thread finally exits, it sets PAL-level *clear_child_tid = 0
(corresponds to &clear_child_tid_val_pal at LibOS level); this signals to LibOS
layer that the thread stopped using resources.
- At LibOS layer, Async Helper thread is set up to wait for the signal from
PAL; it is now the responsibility of Async Helper thread to call
release_clear_child_id() to wake up the parent thread.
- Async Helper thread waits for clear_child_tid_val_pal == 0 and then sets
the actual clear_child_tid to 0 and wakes up the waiting parent.
Note that for Linux-SGX PAL, clear_child_tid is set to 0 not immediately
but as part of handle_thread_reset, otherwise the TCS slot could be still
occupied when LibOS wakes up the parent.
As a side effect, the LibOS code for threads/process exit is cleaned up.
This commit also fixes all regression tests to use the new signature of
DkThreadExit() and increases the number of SGX threads slightly (to
accommodate the newly used Async Helper thread).
This adds a test for thread cleanup. It tests two things:
1. Thred exit in general works, both through return as well as
DkThreadExit.
2. If there's a thread limit, like on SGX, it tests that after a thread
has finished it no longer counts against the limit (currently
broken, see issue #517).
Michał Kowalczyk