Fix instruction overflow/underflow problem

There are undefined behaviors of integer type overflow/underflow in
arithmetic/logic operations.

Cast the inputs to the proper data type before the operation to avoid
undefined behaviors.

Modify the test input vectors to ensure the min/max of the input type is covered.

PiperOrigin-RevId: 551429757