[Codegen] Limit async scope in pipelining (#24350)

Currently, the pipelining assumes that all `amdgpu.gather_to_lds`
operations are part of the loop that is being pipelined and marks them
as async, with waits inserted in the pipelined loop.

This assumption was fine for the workloads so far, but with work
underway to enable pipelining for attention, this no longer holds, as
the load of e.g. the `Q` matrix can be outside of the loop. Operations
outside the loop can also be marked async, but that requires inserting
marks and waits before the loop.

This PR fixes this by recording a marker in the block before the loop
and later on inserting marks and waits for async operations before the
marker.

This is part of https://github.com/iree-org/iree/issues/23782.

Assisted-by: Claude Code and Codex

---------

Signed-off-by: Lukas Sommer <lukas.sommer@amd.com>