tree 6d8de651eedb6199321de38cd928f8f8d1718b65
parent c5faffb268237a3f08515d0845d6652c27765dfa
author Srikrishna Iyer <sriyer@google.com> 1614672951 -0800
committer Srikrishna Iyer <46467186+sriyerg@users.noreply.github.com> 1615868759 -0700

[dvsim] Implement LsfLauncher

This is a first cut implementation of the LsfLauncher. There are several
items left as TODOs - they will be addressed later.

This implementation dispatches all targets (builds, runs, cov etc) as
job arrays by default. Builds are run discretely (array of 1 job) since
we consider each build to have specific job requirements that cannot be
shared with other builds (cpu/mem/disk/stack usage settings - these will
be added in future). Runs pertaining to a build is dispatched as an
array. The associated changes made to other sources support the array
generation.

The job polling is not done by invoking bjobs or bhist, but by looking
for the LSF job output file (unique for each array index), which gets
written to only AFTER the job is complete. This offers a really fast way
to test for completion rather than invoking bjobs or bhist, which bring
the system to a crawl when invoked for 20k tests in flight. This largely
works for now, but we need to explore other options such as using IBM's
Platform LSF Python APIs (future work!).

What launcher system to pick is decided by `DVSIM_LAUNCHER` variable.
In addition, this PR also adds support for Python virtualenv to isolate
project-specific python requirements that need to be met when running
tasks on remote machines used by several other projects as well.

Signed-off-by: Srikrishna Iyer <sriyer@google.com>