Integrate RISP4ML into Shodan ML Toolchain using Fssd model
This CL is Part 2 of the comprehensive 3-part changes of integrating RISP4ML into Shodan ML toolchain. This change focuses on integrating RISP4ML using the FSSD model as an example.
I have verifed that the output@RISP4ML running on Qemu/Renode completely matches with the output@RISPML running on google3.
Details of the 3-part changes:
(Part 1) Completely re-wrote RISP4ML in plain C.
- https://spacebeaker-review.googlesource.com/c/shodan/sw/vec-iree/+/17180
- The original RISP4ML on google3 was written in C++. The memory usage with C++ RISP4ML was significantly larger. Thus, I have completely re-written RISP4ML in plain C. The memory increase with the plain-C RISP4ML is much smaller.
- Made the input/output shape to be configurable arguments instead of constants.
- Made low-level optimizations and simplifications.
(Part 2) Integrated plain-C RISP4ML into Shodan ML toolchain (this CL)
- Use fssd_25_8bit_v2 as an example to load the binary (raw bayer) file and go through RISP4ML toolchain, and feed the RISP4ML output into IREE flow.
- The fssd_25_8bit_v2 example is up and running, and the correctness of output has been verified (compare output@plain-C risp4ml against output@google3).
(Part 3) Added unit tests for plain-C RISP4ML
- Added unit tests based on pw_unit_test
- Build unit tests in iree_cc_binary
- Run and check unit tests via lit_test
Change-Id: I90eb31cb4b5cecfee79ad0952025ad97e12b9b52
diff --git a/samples/quant_model/CMakeLists.txt b/samples/quant_model/CMakeLists.txt
index c0ed7c3..e3e70ff 100644
--- a/samples/quant_model/CMakeLists.txt
+++ b/samples/quant_model/CMakeLists.txt
@@ -159,6 +159,20 @@
QUANT
)
+if(${BUILD_INTERNAL_MODELS})
+
+iree_model_input(
+ NAME
+ fssd_quant_input
+ SHAPE
+ "1, 480, 640, 1"
+ SRC
+ "$ENV{ROOTDIR}/sw/vec_iree/samples/risp4ml/test_data/faces_480x640_uint8_numpy_bayer.bin"
+ QUANT
+)
+
+endif(${BUILD_INTERNAL_MODELS})
+
#-------------------------------------------------------------------------------
# Binaries to execute the MLIR bytecode modules
#-------------------------------------------------------------------------------
@@ -334,9 +348,11 @@
SRCS
"fssd_25_8bit_v2.c"
DEPS
+ ::fssd_quant_input_c
::fssd_25_8bit_v2_bytecode_module_static
::fssd_25_8bit_v2_bytecode_module_static_c
iree::vm::bytecode_module
+ samples::risp4ml::pipeline::pipeline
samples::util::util
LINKOPTS
"LINKER:--defsym=__stack_size__=100k"
@@ -348,8 +364,10 @@
SRCS
"fssd_25_8bit_v2.c"
DEPS
+ ::fssd_quant_input_c
::fssd_25_8bit_v2_c_module_static_c
::fssd_25_8bit_v2_c_module_static_emitc
+ samples::risp4ml::pipeline::pipeline
samples::util::util
LINKOPTS
"LINKER:--defsym=__stack_size__=100k"
diff --git a/samples/quant_model/fssd_25_8bit_v2.c b/samples/quant_model/fssd_25_8bit_v2.c
index 7a272da..0ba243e 100644
--- a/samples/quant_model/fssd_25_8bit_v2.c
+++ b/samples/quant_model/fssd_25_8bit_v2.c
@@ -6,9 +6,11 @@
#include "iree/base/api.h"
#include "iree/hal/api.h"
+#include "samples/risp4ml/pipeline/pipeline.h"
#include "samples/util/util.h"
// Compiled module embedded here to avoid file IO:
+#include "samples/quant_model/fssd_quant_input_c.h"
#if !defined(BUILD_EMITC)
#include "samples/quant_model/fssd_25_8bit_v2_bytecode_module_static.h"
#include "samples/quant_model/fssd_25_8bit_v2_bytecode_module_static_c.h"
@@ -60,13 +62,13 @@
iree_status_t load_input_data(const MlModel *model, void **buffer,
iree_byte_span_t **byte_span) {
iree_status_t result = alloc_input_buffer(model, buffer);
- // Populate initial value
- srand(11111111);
- if (iree_status_is_ok(result)) {
- for (int i = 0; i < model->input_length[0]; ++i) {
- ((uint8_t *)*buffer)[i] = (uint8_t)rand();
- }
- }
+
+ ImageU8 input = {
+ .width = 640, .height = 480, .num_channels = 1, .data = fssd_quant_input};
+ ImageU8 output = {
+ .width = 320, .height = 320, .num_channels = 3, .data = buffer[0]};
+ isp_pipeline(&input, &output);
+
byte_span[0] = malloc(sizeof(iree_byte_span_t));
*byte_span[0] = iree_make_byte_span(
buffer[0], model->input_size_bytes[0] * model->input_length[0]);
