commit e40e64e23e3eaca95b9474e15000349b2c5203fb
author Boian Petkantchin <boian.petkantchin@amd.com> Fri Aug 02 20:29:43 2024 +0300
committer GitHub <noreply@github.com> Fri Aug 02 13:29:43 2024 -0400
tree e959339850268c75373554aec97268aa8089abf7
parent 509d5f721498f4a957e54187d7de5c77892bfb44

[runtime][cuda] Propagate errors through semaphores (#18095)

Mirror the change 3fd336f2850e3965a26fab27a70e160d87588552 from the HAL
HIP driver.

Adds proper handling of errors that occur when executing operations on
the device or when a semaphore fails in the wait list. These errors will
propagate to downstream semaphores that are in the signal list of the
operation.

This change includes some refactoring of the pending queue actions:
* Make the context hold a sticky status instead of just an status code.
* Remove the worker threads' "error" state. This can be handled by the
context's status.
* Remove "exit committed" thread state in favor of standard thread
joining.
* Make the "exit requested" thread state a separate boolean variable and
guard against submitting more work after an exit is requested. Also wait
on all work to complete before exiting worker threads, not just on the
currently ran actions.
* Make pending work items increment immediately when an action is
enqueued instead of when scheduled on the CUDA stream. This is required
to properly count outstanding work.
* Remove and merge some of the redundant state for the worker and
completion threads.
* Remove reference counting from the pending queue actions context. It
has a clear owner, which is the device.
* Rework when the threads exit, which is pretty much only when exit is
requested and there is no more queued or executing actions. Errors don't
cause the threads to exit.

Here is not included moving the destruction and cleanup of actions from
the worker thread to the completion thread. This is an optimization and
code simplification that is now possible since we are not using HIP
stream callbacks, so we could do that right after an action completes.
Technically, actions get destroyed on the completion thread as well when
not on the happy path and actions fail.