| # Custom modules in Python sample | |
| This sample illustrates how to define a custom module in the Python API, | |
| with a pure Python implementation, and compiling an overall program that | |
| can use it. | |
| This builds on the capabilities ot the `custom_module` sample, which | |
| demonstrates C-based extension modules -- applying the same basics to | |
| Python. Some features are not yet implemented on the Python side, and | |
| the API is lower level than we should ultimately have. However, as | |
| is demonstrated, it can do some not trivial things. | |
| ## Sample description | |
| To show off some of the capabilities, this sample: | |
| * Demonstrates how to define a custom Python function which accepts both | |
| a buffer and a variant list. Within the implementation, the buffer is | |
| wrapped by a numpy array for use. | |
| * Module state is kept for the detokenizer state, keeping track of whether | |
| we are at the start of text or sentence. Real detokenizers are much | |
| more complex and would likely involve an opaque module custom type | |
| (not yet implemented in Python). | |
| * A global in the main program is used to accumulate fragments by | |
| the `@detokenizer.accumtokens` function. | |
| * The `@detokenizer.jointokens` will format and emit the text corresponding | |
| to accumulated tokens, respecting sentence boundaries and previous | |
| state. | |
| * A `reset` function is exported which resets the accumulated tokens and | |
| the detokenizer state. | |
| A real text model would be organized differently, but this example should | |
| suffice to show that many of these advanced integration concepts are just | |
| simple code. | |
| A future version of this sample will embed the detokenizer vocabulary as | |
| rodata in the main module and use that to initialize the internal lookup | |
| table. |