Custom modules in Python sample

This sample illustrates how to define a custom module in the Python API, with a pure Python implementation, and compiling an overall program that can use it.

This builds on the capabilities ot the custom_module sample, which demonstrates C-based extension modules -- applying the same basics to Python. Some features are not yet implemented on the Python side, and the API is lower level than we should ultimately have. However, as is demonstrated, it can do some not trivial things.

Sample description

To show off some of the capabilities, this sample:

• Demonstrates how to define a custom Python function which accepts both a buffer and a variant list. Within the implementation, the buffer is wrapped by a numpy array for use.
• Module state is kept for the detokenizer state, keeping track of whether we are at the start of text or sentence. Real detokenizers are much more complex and would likely involve an opaque module custom type (not yet implemented in Python).
• A global in the main program is used to accumulate fragments by the @detokenizer.accumtokens function.
• The @detokenizer.jointokens will format and emit the text corresponding to accumulated tokens, respecting sentence boundaries and previous state.
• A reset function is exported which resets the accumulated tokens and the detokenizer state.

A real text model would be organized differently, but this example should suffice to show that many of these advanced integration concepts are just simple code.

A future version of this sample will embed the detokenizer vocabulary as rodata in the main module and use that to initialize the internal lookup table.