update android package and readme

Author: linyiLYi

NOTICE

@@ -0,0 +1,5 @@
+PoseMon 让爷康康
+Copyright (c) Lin Yi
+
+This project is forked from TensorFlow Examples -- Copyright 2021 The TensorFlow Authors. All Rights Reserved. -- available 
+under the Apache 2.0 license (https://github.com/tensorflow/examples/blob/master/LICENSE).

README.md

@@ -1,74 +1,46 @@
-“让爷康康”是一款手机 AI 应用程序，可以监测不良坐姿并进行语音提示
-# TensorFlow Lite Pose Estimation Android Demo
+# PoseMon 让爷康康
 
-### Overview
-This is an app that continuously detects the body parts in the frames seen by
-your device's camera. These instructions walk you through building and running
-the demo on an Android device. Camera captures are discarded immediately after
-use, nothing is stored or saved.
+## 介绍
 
-The app demonstrates how to use 4 models:
+<image align="right" src="doc_images/screenshot_icon.jpg" alt="Application Icon" width=17%>
 
-* Single pose models: The model can estimate the pose of only one person in the
-input image. If the input image contains multiple persons, the detection result
-can be largely incorrect.
-   * PoseNet
-   * MoveNet Lightning
-   * MoveNet Thunder
-* Multi pose models: The model can estimate pose of multiple persons in the
-input image.
-   * MoveNet MultiPose: Support up to 6 persons.
+“让爷康康”是一款应用于安卓平台的手机应用，可以实时监测不良坐姿并给出语音提示。本项目主要基于 [Tensorflow Lite 官方示例 - 姿态估计](https://github.com/tensorflow/examples/tree/master/lite/examples/pose_estimation/android)实现，其中 AI 部分包含用于姿态估计的 [MoveNet](https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html)，以及用于对姿态进行分类的[全连接网络](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/tutorials/pose_classification.ipynb)。本应用不需要联网使用，所有 AI 特性均在手机本地运行，不需要将视频画面传输至外部服务器，仅需要摄像头权限用于获取姿态画面。视频介绍可以点击 [bilibili]() 或 [YouTube]()。
 
-See this [blog post](https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html)
-for a comparison between these models.
+## 在 Android Studio 中编译程序并运行
 
-![Demo Image](posenetimage.png)
+### 准备工作
 
-## Build the demo using Android Studio
+* 安卓项目的编译需要 Android Studio，可以进入[官方网站](
+ https://developer.android.com/studio/install?hl=zh-cn)按照说明进行下载安装。
 
-### Prerequisites
+* 需要准备一部安卓手机。
 
-* If you don't have it already, install **[Android Studio](
- https://developer.android.com/studio/index.html)** 4.2 or
- above, following the instructions on the website.
+### 编译程序
 
-* Android device and Android development environment with minimum API 21.
+* 通过 `git clone` 克隆本项目，或者以压缩包形式下载项目文件并解压。
 
-### Building
-* Open Android Studio, and from the `Welcome` screen, select
-`Open an existing Android Studio project`.
+* 打开 Android Studio，在初始的 `Welcome` 界面选择
+`Open an existing Android Studio project`，打开项目中的安卓工程文件夹。
 
-* From the `Open File or Project` window that appears, navigate to and select
- the `lite/examples/pose_estimation/android` directory from wherever you
- cloned the `tensorflow/examples` GitHub repo. Click `OK`.
+* 安卓工程文件位于本项目的 `android/` 文件夹下。在 Android Studio 的提示窗口中选择该文件夹。项目打开后软件可能会提示需要进行 Gradle 同步，同意并等待同步完成即可。
 
-* If it asks you to do a `Gradle Sync`, click `OK`.
+* 将处于开发者模式的手机通过 USB 线连接到电脑，具体连接方法可以参考[官方教程](https://developer.android.com/studio/run/device?hl=zh-cn)。如果程序顶部工具栏右侧正确显示了你的手机型号，说明设备连接成功。
 
-* You may also need to install various platforms and tools, if you get errors
- like `Failed to find target with hash string 'android-21'` and similar. Click
- the `Run` button (the green arrow) or select `Run` > `Run 'android'` from the
- top menu. You may need to rebuild the project using `Build` > `Rebuild Project`.
+* 如果是首次安装 Android Studio，可能还需要安装一系列开发工具。点击软件界面右上角的绿色三角按钮`Run 'app'`直接运行程序。如果有需要安装的工具，系统会进行提示，按照提示依次安装即可。
 
-* If it asks you to use `Instant Run`, click `Proceed Without Instant Run`.
+### 模型介绍
 
-* Also, you need to have an Android device plugged in with developer options
- enabled at this point. See **[here](
- https://developer.android.com/studio/run/device)** for more details
- on setting up developer devices.
+本项目需要用到两个神经网络模型文件，均已包含在本项目中，不需要额外下载。第一个是 `int8` 格式的 MoveNet Thunder 神经网络模型，可以点击[官方模型文件链接](https://tfhub.dev/google/lite-model/movenet/singlepose/thunder/tflite/int8/4)进一步了解。[MoveNet](https://blog.tensorflow.org/2021/05/next-generation-pose-detection-with-movenet-and-tensorflowjs.html) 是谷歌推出的轻量级人体姿态估计模型，有 Thunder 和 Lightning 两个版本。其中 Thunder 版本运行速度较慢，但准确率更高，本项目使用的是 Thunder 版本。该版本又分为 `float16`、`int8` 两种数据格式。其中 `float16` 模型只能在通用 GPU 上运行，而 `int8` 模型既可以运行于通用 GPU 之上，也可以在高通骁龙处理器的 [Hexagon DSP 数字信号处理器](https://developer.qualcomm.com/software/hexagon-dsp-sdk/dsp-processor)上运行。运行在 Hexagon 处理器上时，AI 程序运行速度更快、也更省电，建议对 AI 模型进行移动部署时优先选择 Hexagon 处理器。目前谷歌也推出了自研的 Google Tensor 处理器，最新型号为 Tensor G2，如何调用 Tensor 处理器的 AI 加速单元尚不清楚，未来拿到设备实测确认后会更新文档。
 
+#### 训练自己的分类网络
 
-### Model used
-Downloading, extraction and placement in assets folder has been managed
- automatically by `download.gradle`.
+<image align="right" src="doc_images/labeled_movenet_result.png" alt="17 Keypoints detected by MoveNet" width=17%>
 
-If you explicitly want to download the model, you can download it from here:
+除了 MoveNet Thunder，本项目还使用了一个简单的全连接网络对 MoveNet 输出的姿态信息（人体 17 个关键点的坐标）进行分类，用来判断画面中的人处于“标准坐姿”、“翘二郎腿”、“脖子前倾驼背”中的哪一种状态。关于该分类网络的介绍以及训练过程实际演示，可以参考 Tensorflow Lite 的 [Jupyter Notebook 教程](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/tutorials/pose_classification.ipynb)，或是本项目中修改并注释过的[版本]()。本项目为了对“标准坐姿”、“翘二郎腿”、“脖子前倾驼背”三种姿态进行分类，为每种姿态采集了约 300 张照片作为训练集（共 876 张照片），为每种姿态采集了约 30 张作为测试集（共 74 张照片）。其中训练集与测试集为不同人物主体，以此来在训练过程中及时发现模型的过拟合问题。训练数据应存放于 `main/pose_data/train/` 路径下的 `standard`、`crossleg`、`forwardhead` 
+三个文件夹中，测试数据则位于 `main/pose_data/test/` 路径下。本项目中用于训练分类网络的 [Jupyter Notebook]() 会将原始数据自动转化为训练数据包，在此过程中生成每张照片的 MoveNet 检测结果，并将每张照片标记为三种姿态中的一种，最后将所有信息存储在 `main/pose_data/train_data.csv`、`main/pose_data/test_data.csv`，并生成记录标签信息的文本文件 `main/pose_data/pose_labels.txt`。在 Notebook 中训练完毕后，在 `main/pose_data/` 路径下会自动生成 `.tflite` 权重文件，导入至 Android Studio 项目中，替换掉本项目中的 `android\app\src\main\assets\classifier.tflite` 即可使用。
 
-* [Posenet](https://storage.googleapis.com/download.tensorflow.org/models/tflite/posenet_mobilenet_v1_100_257x257_multi_kpt_stripped.tflite)
-* [Movenet Lightning](https://tfhub.dev/google/movenet/singlepose/lightning/)
-* [Movenet Thunder](https://tfhub.dev/google/movenet/singlepose/thunder/)
-* [Movenet MultiPose](https://tfhub.dev/google/movenet/multipose/lightning/)
+### 运行效果
 
-### Additional Note
-_Please do not delete the assets folder content_. If you explicitly deleted the
- files, then please choose `Build` > `Rebuild` from menu to re-download the
- deleted model files into assets folder.
+
+## 鸣谢
+本项目主要基于 [Tensorflow Lite Pose Estimation 示例项目](https://github.com/tensorflow/examples/tree/master/lite/examples/pose_estimation/android)，离不开 [Tensorflow](https://www.tensorflow.org/?hl=zh-cn)、[Jupyter Notebook](https://jupyter.org/) 等开源框架、开源开发工具。感谢各位程序工作者对开源社区的贡献！

android/app/build.gradle

@@ -8,7 +8,7 @@ android {
     buildToolsVersion "30.0.3"
 
     defaultConfig {
-        applicationId "org.tensorflow.lite.examples.poseestimation"
+        applicationId "lyi.linyi.posemon"
         minSdkVersion 23
         targetSdkVersion 30
         versionCode 1
@@ -32,9 +32,6 @@ android {
     }
 }
 
-// Download tflite model
-apply from:"download.gradle"
-
 dependencies {
 
     implementation "org.jetbrains.kotlin:kotlin-stdlib:$kotlin_version"

android/app/src/androidTest/java/org/tensorflow/lite/examples/poseestimation/ml/EvaluationUtils.kt android/app/src/androidTest/java/lyi/linyi/lite/examples/posemon/ml/EvaluationUtils.kt

@@ -14,7 +14,7 @@ limitations under the License.
 ==============================================================================
 */
 
-package org.tensorflow.lite.examples.poseestimation.ml
+package lyi.linyi.lite.examples.posemon.ml
 
 import android.graphics.Bitmap
 import android.graphics.BitmapFactory
@@ -23,8 +23,8 @@ import android.graphics.PointF
 import androidx.test.platform.app.InstrumentationRegistry
 import com.google.common.truth.Truth.assertThat
 import com.google.common.truth.Truth.assertWithMessage
-import org.tensorflow.lite.examples.poseestimation.data.BodyPart
-import org.tensorflow.lite.examples.poseestimation.data.Person
+import org.tensorflow.posemon.data.BodyPart
+import org.tensorflow.posemon.data.Person
 import java.io.BufferedReader
 import java.io.InputStreamReader
 import kotlin.math.pow
@@ -49,7 +49,11 @@ object EvaluationUtils {
             assertWithMessage("$bodyPart must exist").that(keypoint).isNotNull()
 
             val detectedPointF = keypoint!!.coordinate
-            val distanceFromExpectedPointF = distance(detectedPointF, expectedPointF)
+            val distanceFromExpectedPointF =
+                lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.distance(
+                    detectedPointF,
+                    expectedPointF
+                )
             assertWithMessage("Detected $bodyPart must be close to expected result")
                 .that(distanceFromExpectedPointF).isAtMost(acceptableError)
         }

android/app/src/androidTest/java/org/tensorflow/lite/examples/poseestimation/ml/MovenetLightningTest.kt android/app/src/androidTest/java/lyi/linyi/lite/examples/posemon/ml/MovenetLightningTest.kt

@@ -14,7 +14,7 @@ limitations under the License.
 ==============================================================================
 */
 
-package org.tensorflow.lite.examples.poseestimation.ml
+package lyi.linyi.lite.examples.posemon.ml
 
 import android.content.Context
 import android.graphics.PointF
@@ -23,8 +23,11 @@ import androidx.test.platform.app.InstrumentationRegistry
 import org.junit.Before
 import org.junit.Test
 import org.junit.runner.RunWith
-import org.tensorflow.lite.examples.poseestimation.data.BodyPart
-import org.tensorflow.lite.examples.poseestimation.data.Device
+import org.tensorflow.posemon.data.BodyPart
+import org.tensorflow.posemon.data.Device
+import org.tensorflow.posemon.ml.ModelType
+import org.tensorflow.posemon.ml.MoveNet
+import org.tensorflow.posemon.ml.PoseDetector
 
 @RunWith(AndroidJUnit4::class)
 class MovenetLightningTest {
@@ -44,20 +47,20 @@ class MovenetLightningTest {
         appContext = InstrumentationRegistry.getInstrumentation().targetContext
         poseDetector = MoveNet.create(appContext, Device.CPU, ModelType.Lightning)
         expectedDetectionResult =
-            EvaluationUtils.loadCSVAsset("pose_landmark_truth.csv")
+            lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.loadCSVAsset("pose_landmark_truth.csv")
     }
 
     @Test
     fun testPoseEstimationResultWithImage1() {
-        val input = EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE1)
+        val input = lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE1)
 
         // As Movenet use previous frame to optimize detection result, we run it multiple times
         // using the same image to improve result.
         poseDetector.estimatePoses(input)
         poseDetector.estimatePoses(input)
         poseDetector.estimatePoses(input)
         val person = poseDetector.estimatePoses(input)[0]
-        EvaluationUtils.assertPoseDetectionResult(
+        lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.assertPoseDetectionResult(
             person,
             expectedDetectionResult[0],
             ACCEPTABLE_ERROR
@@ -66,15 +69,15 @@ class MovenetLightningTest {
 
     @Test
     fun testPoseEstimationResultWithImage2() {
-        val input = EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE2)
+        val input = lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE2)
 
         // As Movenet use previous frame to optimize detection result, we run it multiple times
         // using the same image to improve result.
         poseDetector.estimatePoses(input)
         poseDetector.estimatePoses(input)
         poseDetector.estimatePoses(input)
         val person = poseDetector.estimatePoses(input)[0]
-        EvaluationUtils.assertPoseDetectionResult(
+        lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.assertPoseDetectionResult(
             person,
             expectedDetectionResult[1],
             ACCEPTABLE_ERROR

android/app/src/androidTest/java/org/tensorflow/lite/examples/poseestimation/ml/MovenetMultiPoseTest.kt android/app/src/androidTest/java/lyi/linyi/lite/examples/posemon/ml/MovenetMultiPoseTest.kt

@@ -14,7 +14,7 @@ limitations under the License.
 ==============================================================================
 */
 
-package org.tensorflow.lite.examples.poseestimation.ml
+package lyi.linyi.lite.examples.posemon.ml
 
 import android.content.Context
 import android.graphics.Bitmap
@@ -24,10 +24,10 @@ import androidx.test.platform.app.InstrumentationRegistry
 import org.junit.Before
 import org.junit.Test
 import org.junit.runner.RunWith
-import org.tensorflow.lite.examples.poseestimation.data.BodyPart
-import org.tensorflow.lite.examples.poseestimation.data.Device
-import org.tensorflow.lite.examples.poseestimation.ml.MoveNetMultiPose
-import org.tensorflow.lite.examples.poseestimation.ml.Type
+import org.tensorflow.posemon.data.BodyPart
+import org.tensorflow.posemon.data.Device
+import org.tensorflow.posemon.ml.MoveNetMultiPose
+import org.tensorflow.posemon.ml.Type
 
 @RunWith(AndroidJUnit4::class)
 class MovenetMultiPoseTest {
@@ -46,11 +46,11 @@ class MovenetMultiPoseTest {
     fun setup() {
         appContext = InstrumentationRegistry.getInstrumentation().targetContext
         poseDetector = MoveNetMultiPose.create(appContext, Device.CPU, Type.Dynamic)
-        val input1 = EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE1)
-        val input2 = EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE2)
-        inputFinal = EvaluationUtils.hConcat(input1, input2)
+        val input1 = lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE1)
+        val input2 = lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE2)
+        inputFinal = lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.hConcat(input1, input2)
         expectedDetectionResult =
-                EvaluationUtils.loadCSVAsset("pose_landmark_truth.csv")
+                lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.loadCSVAsset("pose_landmark_truth.csv")
 
         // update coordination of the pose_landmark_truth.csv corresponding to the new input image
         for ((_, value) in expectedDetectionResult[1]) {
@@ -66,13 +66,13 @@ class MovenetMultiPoseTest {
         // Sort the results so that the person on the right side come first.
         val sortedPersons = persons.sortedBy { it.boundingBox?.left }
 
-        EvaluationUtils.assertPoseDetectionResult(
+        lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.assertPoseDetectionResult(
                 sortedPersons[0],
                 expectedDetectionResult[0],
                 ACCEPTABLE_ERROR
         )
 
-        EvaluationUtils.assertPoseDetectionResult(
+        lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.assertPoseDetectionResult(
                 sortedPersons[1],
                 expectedDetectionResult[1],
                 ACCEPTABLE_ERROR

android/app/src/androidTest/java/org/tensorflow/lite/examples/poseestimation/ml/MovenetThunderTest.kt android/app/src/androidTest/java/lyi/linyi/lite/examples/posemon/ml/MovenetThunderTest.kt

@@ -14,7 +14,7 @@ limitations under the License.
 ==============================================================================
 */
 
-package org.tensorflow.lite.examples.poseestimation.ml
+package lyi.linyi.lite.examples.posemon.ml
 
 import android.content.Context
 import android.graphics.PointF
@@ -23,8 +23,11 @@ import androidx.test.platform.app.InstrumentationRegistry
 import org.junit.Before
 import org.junit.Test
 import org.junit.runner.RunWith
-import org.tensorflow.lite.examples.poseestimation.data.BodyPart
-import org.tensorflow.lite.examples.poseestimation.data.Device
+import org.tensorflow.posemon.data.BodyPart
+import org.tensorflow.posemon.data.Device
+import org.tensorflow.posemon.ml.ModelType
+import org.tensorflow.posemon.ml.MoveNet
+import org.tensorflow.posemon.ml.PoseDetector
 
 @RunWith(AndroidJUnit4::class)
 class MovenetThunderTest {
@@ -44,20 +47,20 @@ class MovenetThunderTest {
         appContext = InstrumentationRegistry.getInstrumentation().targetContext
         poseDetector = MoveNet.create(appContext, Device.CPU, ModelType.Thunder)
         expectedDetectionResult =
-            EvaluationUtils.loadCSVAsset("pose_landmark_truth.csv")
+            lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.loadCSVAsset("pose_landmark_truth.csv")
     }
 
     @Test
     fun testPoseEstimationResultWithImage1() {
-        val input = EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE1)
+        val input = lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE1)
 
         // As Movenet use previous frame to optimize detection result, we run it multiple times
         // using the same image to improve result.
         poseDetector.estimatePoses(input)
         poseDetector.estimatePoses(input)
         poseDetector.estimatePoses(input)
         val person = poseDetector.estimatePoses(input)[0]
-        EvaluationUtils.assertPoseDetectionResult(
+        lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.assertPoseDetectionResult(
             person,
             expectedDetectionResult[0],
             ACCEPTABLE_ERROR
@@ -66,15 +69,15 @@ class MovenetThunderTest {
 
     @Test
     fun testPoseEstimationResultWithImage2() {
-        val input = EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE2)
+        val input = lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.loadBitmapAssetByName(TEST_INPUT_IMAGE2)
 
         // As Movenet use previous frame to optimize detection result, we run it multiple times
         // using the same image to improve result.
         poseDetector.estimatePoses(input)
         poseDetector.estimatePoses(input)
         poseDetector.estimatePoses(input)
         val person = poseDetector.estimatePoses(input)[0]
-        EvaluationUtils.assertPoseDetectionResult(
+        lyi.linyi.lite.examples.posemon.ml.EvaluationUtils.assertPoseDetectionResult(
             person,
             expectedDetectionResult[1],
             ACCEPTABLE_ERROR