commit 27179e2a79600b97aba5ba9ab6bcea3a9103ad73
author Okwan Kwon <okwan@google.com> Tue Apr 18 12:59:24 2023 -0700
committer GitHub <noreply@github.com> Tue Apr 18 12:59:24 2023 -0700
tree ca72b3535b1497a2c09061ae2a85571fdf5d4775
parent c20535560e71c85ea325f3582692b020edb8b236

Use MPI for NCCL unique ID exchange by default (#12902)

Now the default implementation of the channel provider uses MPI to
exchange NCCL unique ID.

A few MPI functions are dynamically called just like how NCCL functions
are called. With the default implementation, OpenMPI must be available
in the system to run a module with collective operations. (Note that We
can't use MPICH-variants because of the difference in defining
MPI_Datatype and MPI_Comm.)

The device index is overridden by the MPI rank automatically.

Here is the command line before and after.

before
```
NCCL_COMM_ID=127.0.0.1:8000 iree-run-module --cuda_nccl_default_rank=0 --cuda_nccl_default_count=2 --device=cuda://0 --module=all_gather.vmfb --function=all_gather_dim_0 & \
NCCL_COMM_ID=127.0.0.1:8000 iree-run-module --cuda_nccl_default_rank=1 --cuda_nccl_default_count=2 --device=cuda://1 --module=all_gather.vmfb --function=all_gather_dim_0
```

after
```
mpirun -np 2 iree-run-module --device=cuda --module=all_gather.vmfb --function=all_gather_dim_0
```