[Tokenizer] Fix rstrip flag: remove match rejection, add whitespace consumption (#23805)

## Summary

`flags_allow_match()` treated `rstrip=true` as a match condition,
rejecting tokens not followed by whitespace or end of input. Per
[HuggingFace's
implementation(https://github.com/huggingface/tokenizers/blob/main/tokenizers/src/tokenizer/added_vocabulary.rs),
`rstrip` is a post-processing directive that consumes trailing
whitespace after the match, not a match gate. This caused special tokens
like `<|user|>` in Phi-4-mini to fall through to BPE subword
tokenization when followed by text.

**Example** (Phi-4-mini-instruct, token `<|user|>` has `rstrip=true`):

Input: "<|user|>Hello"
Before fix: [27, 91, 1428, 91, 29, 13225] ← <|user|> split into subwords
After fix: [200021, 13225] ← matches HuggingFace output



## Changes

- Remove the rstrip rejection branch from `flags_allow_match()` in
`special_tokens.c`
- Add rstrip whitespace consumption in `tokenizer.c` after a successful
match
- Add `matched_flags` field to encode state to propagate flags to the
caller

## Test results

### HuggingFace smoketest (`huggingface_smoketest.py`)

**1667/1667** tokenization comparisons pass across ~80 HuggingFace
models (0 mismatches).

76 additional tests fail to **load** (not tokenization mismatches) —
these are tiktoken models listed in the HF smoketest that aren't valid
HuggingFace model identifiers. They're tested by the dedicated tiktoken
smoketest instead.

### Tiktoken smoketest (`tiktoken_smoketest.py`)

**72/76** tokenization comparisons pass across 4 tiktoken encodings
(cl100k_base, o200k_base, r50k_base, p50k_base).

4 failures — **identical between upstream and this PR** (pre-existing):

| Encoding | Failing test | Cause |
|----------|-------------|-------|
| cl100k_base | `special_token_endoftext` | IREE matches
`<\|endoftext\|>` as special token; tiktoken's `encode_ordinary` treats
it as literal text |
| o200k_base | `special_token_endoftext` | Same |
| r50k_base | `special_token_endoftext` | Same |
| p50k_base | `special_token_endoftext` | Same |

Root cause: the IREE tokenizer has no "encode ordinary" mode (equivalent
to tiktoken's `disallowed_special=()`).

Signed-off-by: Jorn <jorn.tuyls@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>