Update TF docs to reflect bazel deprecation (#4171)
diff --git a/docs/developing_iree/e2e_benchmarking.md b/docs/developing_iree/e2e_benchmarking.md index 7b5e404..607ef35 100644 --- a/docs/developing_iree/e2e_benchmarking.md +++ b/docs/developing_iree/e2e_benchmarking.md
@@ -1,26 +1,49 @@ # Benchmark IREE and TFLite +<!-- TODO(meadowlark): Update this doc once the API is stable and change default to cmake. --> + +> Note:<br> +> The TensorFlow integrations are currently being + refactored. The `bazel` build is deprecated. Refer to + https://google.github.io/iree/get-started/getting-started-python for a general + overview of how to build and execute the e2e tests. + We use our end-to-end TensorFlow integration tests to test compilation and numerical accuracy, and to generate compilation and benchmarking artifacts. This allows us to validate that our benchmarks are behaving as we expect them to, and to run them using valid inputs for each model. This guide assumes that you can run the tensorflow integration tests. See -[this doc](https://google.github.io/iree/developing-iree/tensorflow-integrations) +[this doc](https://google.github.io/iree/get-started/getting-started-python) for more information. That doc also covers writing new tests, which you'll need to do if you'd like to benchmark a new TensorFlow model. ## 1. Run IREE's E2E TensorFlow tests to generate the benchmarking artifacts -The `get_e2e_artifacts.py` script compiles and tests all of our integrated -TensorFlow models, and gathers their compilation and benchmarking artifacts in -`/tmp/iree/modules/`. +```shell +# Continuing from the "Running Python Tests" section of the doc linked above: +$ cd ../iree-build/ +``` ```shell -# By default `get_e2e_artifacts.py` will run all of our test suites, including -# those that take a long time to complete, so we specify -# `--test_suites=e2e_tests` to only run the smaller tests. -$ python3 ./scripts/get_e2e_artifacts.py --test_suites=e2e_tests +# --target_backends: All tests allow you to specify one or more backends to generate benchmarking artifacts for. +# --artifacts_dir: The default location for these artifacts is under /tmp/iree/modules +$ python ../iree/integrations/tensorflow/e2e/matrix_ops_static.py \ + --target_backends=iree_vmla +# View the generated artifacts: +$ tree /tmp/iree/modules/MatrixOpsStaticModule/ +``` + +```shell +# Some tests use additional flags to specify features of the Module to test/benchmark: +# --model: the tf.keras.applications model to test +# --data: the dataset (and corresponding image shapes) to create the model for +$ python ../iree/integrations/tensorflow/e2e/keras/applications/applications_test.py \ + --target_backends=iree_vmla \ + --model=MobileNetV3 \ + --data=imagenet +# View the generated artifacts: +$ tree /tmp/iree/modules/MobileNetV3/ ``` Each test/module has a folder with the following artifacts (filtered to only @@ -92,32 +115,6 @@ └── graph_path ``` -### Optional: Compile the Keras Applications Vision tests - -The vision tests take a while to run, so we exclude them from the CI and -wildcard expansion. They can be run by invoking the following test suite: - -```shell -$ python3 ./scripts/get_e2e_artifacts.py --test_suites=imagenet_non_hermetic_tests -``` - -The previous command compiles `MobileNet`, `MobileNetV2` and `ResNet50` to run -on `cifar10` and `imagenet` weights on all backends. The artifacts generated by -this test suite are slightly different than those above in that they are -organized by `/tmp/iree/modules/ModelName/Dataset/backends` instead of just by -`/tmp/iree/modules/ModelName/backends`. - -### Optional: Manually get the benchmarking artifacts for a specific test - -You can manually get the benchmarking artifacts for a specific test by using -`bazel run` on the `_manual` binary target we create for each test. This will -automatically store the benchmarking artifacts in `/tmp/iree/modules/`. - -```shell -$ bazel run //integrations/tensorflow/e2e:matrix_ops_static_test_manual -- \ - --target_backends=iree_vmla,tflite -``` - ## 2. Benchmarking IREE on desktop ### 2.1 Optional: Build the `iree-benchmark-module` @@ -161,13 +158,12 @@ --flagfile="/tmp/iree/modules/MatrixOpsStaticModule/iree_vmla/traces/matmul_lhs_batch/flagfile" ``` -If you ran the Keras Applications vision test suite, then you'll be able to -benchmark `ResNet50`, `MobileNet` or `MobileNetV2` with `cifar10` or `imagenet` -weights. For example: +If you ran `applications_test.py` then you'll be able to benchmark `MobileNetV3` +on `imagenet` input shapes. For example: ```shell $ ./bazel-bin/iree/tools/iree-benchmark-module \ - --flagfile="/tmp/iree/modules/ResNet50/cifar10/iree_vmla/traces/predict/flagfile" + --flagfile="/tmp/iree/modules/MobileNetV3/imagenet/iree_vmla/traces/predict/flagfile" ``` ## 3. Benchmarking TFLite on desktop
diff --git a/integrations/tensorflow/e2e/README.md b/integrations/tensorflow/e2e/README.md index 4a7e179..1210edb 100644 --- a/integrations/tensorflow/e2e/README.md +++ b/integrations/tensorflow/e2e/README.md
@@ -1,5 +1,13 @@ # TensorFlow e2e tests +<!-- TODO(meadowlark): Update this doc once the API is stable. --> + +> Note:<br> +> The TensorFlow integrations are currently being + refactored. The `bazel` build is deprecated. Refer to + https://google.github.io/iree/get-started/getting-started-python for a general + overview of how to build and execute the e2e tests. + This is a collection of e2e tests that compile a TensorFlow model with IREE (and potentially TFLite), run it on multiple backends, and crosscheck the results.
diff --git a/scripts/get_e2e_artifacts.py b/scripts/get_e2e_artifacts.py index 3b5fd62..61a5938 100755 --- a/scripts/get_e2e_artifacts.py +++ b/scripts/get_e2e_artifacts.py
@@ -155,6 +155,13 @@ def main(argv): del argv # Unused. + print("The bazel integrations build and tests are deprecated. This script " + "may be reworked in the future. For the time being refer to " + "https://google.github.io/iree/get-started/getting-started-python " + "and https://google.github.io/iree/developing-iree/e2e-benchmarking " + "for information on how to run TensorFlow benchmarks.") + exit(1) + # Convert test suite shorthands to full test suite targets. test_suites = [SUITE_NAME_TO_TARGET[suite] for suite in FLAGS.test_suites]