OpenVINO を使用したライブ 3D 人物の姿勢推定#
この Jupyter ノートブックはオンラインで起動でき、ブラウザーのウィンドウで対話型環境を開きます。ローカルにインストールすることもできます。次のオプションのいずれかを選択します:
このノートブックは、ウェブカメラ経由で OpenVINO を使用したライブ 3D 人物の姿勢推定を示します。Open Model Zoo の human-pose-estimation-3d-0001 モデルを使用します。このノートブックの最後では、ウェブカメラからのライブ推論結果が表示されます (利用可能な場合)。あるいは、ビデオファイルをアップロードしてアルゴリズムをテストすることもできます。Jupyter 拡張機能が適切にインストールされていること、および ``README.md`` で提案されているように JupyterLab を使用してデモを実行していることを確認してください。
注: ウェブカメラを使用するには、ウェブカメラを備えたコンピューター上でこの Jupyter ノートブックを実行する必要があります。リモートサーバー上で実行すると、ウェブカメラは機能しなくなります。ただし、最終ステップではビデオに対して推論を行うことができます。このデモでは、WebGL と統合された ``Three.js`` の Python インターフェイスを利用して、モデル推論からのデータを処理します。これらの結果は処理されノートブックに表示されます。
結果が正しく表示されることを確認するには、次のオペレーティング・システムのいずれかの推奨ブラウザーでコードを実行します: Ubuntu*、Windows*: Chrome macOS*: Safari
目次:
必要条件#
最新の Jupyter ノートブックのリリースを使用すると、``pythreejs`` 拡張子が正しく表示されない場合があります。したがって、代わりに Jupyter Lab を使用することを推奨します。
%pip install pythreejs "openvino-dev>=2024.0.0" "opencv-python" "torch" "onnx" --extra-index-url https://download.pytorch.org/whl/cpuLooking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
Collecting pythreejs
Using cached pythreejs-2.4.2-py3-none-any.whl.metadata (5.4 kB)
Collecting openvino-dev>=2024.0.0
Using cached openvino_dev-2024.2.0-15519-py3-none-any.whl.metadata (16 kB)
Collecting opencv-python
Using cached opencv_python-4.10.0.84-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (20 kB)
Collecting torch
Using cached https://download.pytorch.org/whl/cpu/torch-2.3.1%2Bcpu-cp38-cp38-linux_x86_64.whl (190.4 MB)
Collecting onnx
Using cached onnx-1.16.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (16 kB)
Requirement already satisfied: ipywidgets>=7.2.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pythreejs) (8.1.3)
Collecting ipydatawidgets>=1.1.1 (from pythreejs)
Using cached ipydatawidgets-4.3.5-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting numpy (from pythreejs)
Using cached numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (5.6 kB)
Requirement already satisfied: traitlets in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pythreejs) (5.14.3)
Requirement already satisfied: defusedxml>=0.7.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino-dev>=2024.0.0) (0.7.1)
Collecting networkx<=3.1.0 (from openvino-dev>=2024.0.0)
Using cached networkx-3.1-py3-none-any.whl.metadata (5.3 kB)
Collecting openvino-telemetry>=2023.2.1 (from openvino-dev>=2024.0.0)
Using cached openvino_telemetry-2024.1.0-py3-none-any.whl.metadata (2.3 kB)
Requirement already satisfied: packaging in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino-dev>=2024.0.0) (24.1)
Requirement already satisfied: pyyaml>=5.4.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino-dev>=2024.0.0) (6.0.1)
Requirement already satisfied: requests>=2.25.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from openvino-dev>=2024.0.0) (2.32.0)
Collecting openvino==2024.2.0 (from openvino-dev>=2024.0.0)
Using cached openvino-2024.2.0-15519-cp38-cp38-manylinux2014_x86_64.whl.metadata (8.9 kB)
Collecting filelock (from torch)
Using cached filelock-3.15.4-py3-none-any.whl.metadata (2.9 kB)
Requirement already satisfied: typing-extensions>=4.8.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch) (4.12.2)
Collecting sympy (from torch)
Using cached sympy-1.13.0-py3-none-any.whl.metadata (12 kB)
Requirement already satisfied: jinja2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from torch) (3.1.4)
Collecting fsspec (from torch)
Using cached fsspec-2024.6.1-py3-none-any.whl.metadata (11 kB)
Collecting protobuf>=3.20.2 (from onnx)
Using cached protobuf-5.27.2-cp38-abi3-manylinux2014_x86_64.whl.metadata (592 bytes)
Collecting traittypes>=0.2.0 (from ipydatawidgets>=1.1.1->pythreejs)
Using cached traittypes-0.2.1-py2.py3-none-any.whl.metadata (1.0 kB)
Requirement already satisfied: comm>=0.1.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets>=7.2.1->pythreejs) (0.2.2)
Requirement already satisfied: ipython>=6.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets>=7.2.1->pythreejs) (8.12.3)
Requirement already satisfied: widgetsnbextension~=4.0.11 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets>=7.2.1->pythreejs) (4.0.11)
Requirement already satisfied: jupyterlab-widgets~=3.0.11 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipywidgets>=7.2.1->pythreejs) (3.0.11)
Requirement already satisfied: charset-normalizer<4,>=2 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.25.1->openvino-dev>=2024.0.0) (3.3.2)
Requirement already satisfied: idna<4,>=2.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.25.1->openvino-dev>=2024.0.0) (3.7)
Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.25.1->openvino-dev>=2024.0.0) (2.2.2)
Requirement already satisfied: certifi>=2017.4.17 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from requests>=2.25.1->openvino-dev>=2024.0.0) (2024.7.4)
Requirement already satisfied: MarkupSafe>=2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jinja2->torch) (2.1.5)
Collecting mpmath<1.4,>=1.1.0 (from sympy->torch)
Using cached https://download.pytorch.org/whl/mpmath-1.3.0-py3-none-any.whl (536 kB)
Requirement already satisfied: backcall in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (0.2.0)
Requirement already satisfied: decorator in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (5.1.1)
Requirement already satisfied: jedi>=0.16 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (0.19.1)
Requirement already satisfied: matplotlib-inline in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (0.1.7)
Requirement already satisfied: pickleshare in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (0.7.5)
Requirement already satisfied: prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (3.0.47)
Requirement already satisfied: pygments>=2.4.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (2.18.0)
Requirement already satisfied: stack-data in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (0.6.3)
Requirement already satisfied: pexpect>4.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (4.9.0)
Requirement already satisfied: parso<0.9.0,>=0.8.3 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (0.8.4)
Requirement already satisfied: ptyprocess>=0.5 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (0.7.0)
Requirement already satisfied: wcwidth in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from prompt-toolkit!=3.0.37,<3.1.0,>=3.0.30->ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (0.2.13)
Requirement already satisfied: executing>=1.2.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (2.0.1)
Requirement already satisfied: asttokens>=2.1.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (2.4.1)
Requirement already satisfied: pure-eval in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from stack-data->ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (0.2.2)
Requirement already satisfied: six>=1.12.0 in /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages (from asttokens>=2.1.0->stack-data->ipython>=6.1.0->ipywidgets>=7.2.1->pythreejs) (1.16.0)
Using cached pythreejs-2.4.2-py3-none-any.whl (3.4 MB)
Using cached openvino_dev-2024.2.0-15519-py3-none-any.whl (4.7 MB)
Using cached openvino-2024.2.0-15519-cp38-cp38-manylinux2014_x86_64.whl (38.7 MB)
Using cached opencv_python-4.10.0.84-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (62.5 MB)
Using cached onnx-1.16.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
Using cached ipydatawidgets-4.3.5-py2.py3-none-any.whl (271 kB)
Using cached networkx-3.1-py3-none-any.whl (2.1 MB)
Using cached numpy-1.24.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
Using cached openvino_telemetry-2024.1.0-py3-none-any.whl (23 kB)
Using cached protobuf-5.27.2-cp38-abi3-manylinux2014_x86_64.whl (309 kB)
Using cached filelock-3.15.4-py3-none-any.whl (16 kB)
Using cached fsspec-2024.6.1-py3-none-any.whl (177 kB)
Using cached sympy-1.13.0-py3-none-any.whl (6.2 MB)
Using cached traittypes-0.2.1-py2.py3-none-any.whl (8.6 kB)
Installing collected packages: openvino-telemetry, mpmath, traittypes, sympy, protobuf, numpy, networkx, fsspec, filelock, torch, openvino, opencv-python, onnx, openvino-dev, ipydatawidgets, pythreejs
Successfully installed filelock-3.15.4 fsspec-2024.6.1 ipydatawidgets-4.3.5 mpmath-1.3.0 networkx-3.1 numpy-1.24.4 onnx-1.16.1 opencv-python-4.10.0.84 openvino-2024.2.0 openvino-dev-2024.2.0 openvino-telemetry-2024.1.0 protobuf-5.27.2 pythreejs-2.4.2 sympy-1.13.0 torch-2.3.1+cpu traittypes-0.2.1
Note: you may need to restart the kernel to use updated packages.インポート#
import collections
import time
from pathlib import Path
import cv2
import ipywidgets as widgets
import numpy as np
from IPython.display import clear_output, display
import openvino as ov
# `notebook_utils` モジュールを取得
import requests
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py",
)
with open("notebook_utils.py", "w") as f:
f.write(r.text)
r = requests.get(
url="https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/engine3js.py",
)
with open("engine3js.py", "w") as f:
f.write(r.text)
import notebook_utils as utils
import engine3js as engineモデル#
モデルのダウンロード#
ここでは、openvino-dev パッケージのコマンドライン・ツールである omz_downloader を使用します。omz_downloader は、自動的にディレクトリー構造を作成し、選択したモデルをダウンロードします。
# モデルがダウンロードされるディレクトリー
base_model_dir = "model"
# Open Model Zoo で命名されたモデル名
model_name = "human-pose-estimation-3d-0001"
# 選択された精度 (FP32, FP16)
precision = "FP32"
BASE_MODEL_NAME = f"{base_model_dir}/public/{model_name}/{model_name}"
model_path = Path(BASE_MODEL_NAME).with_suffix(".pth")
onnx_path = Path(BASE_MODEL_NAME).with_suffix(".onnx")
ir_model_path = f"model/public/{model_name}/{precision}/{model_name}.xml"
model_weights_path = f"model/public/{model_name}/{precision}/{model_name}.bin"
if not model_path.exists():
download_command = f"omz_downloader " f"--name {model_name} " f"--output_dir {base_model_dir}"
! $download_command################|| Downloading human-pose-estimation-3d-0001 ||################
========== Downloading model/public/human-pose-estimation-3d-0001/human-pose-estimation-3d-0001.tar.gz
========== Unpacking model/public/human-pose-estimation-3d-0001/human-pose-estimation-3d-0001.tar.gzモデルを OpenVINO IR 形式に変換#
選択されたモデルはパブリック・ディレクトリーから取得されます。つまり、OpenVINO 中間表現 (OpenVINO IR) に変換する必要があります。omz_converter を使用して、ONNX 形式モデルを OpenVINO IR 形式に変換します。
if not onnx_path.exists():
convert_command = (
f"omz_converter " f"--name {model_name} " f"--precisions {precision} " f"--download_dir {base_model_dir} " f"--output_dir {base_model_dir}"
)
! $convert_command
========== Converting human-pose-estimation-3d-0001 to ONNX
Conversion to ONNX command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/lib/python3.8/site-packages/omz_tools/internal_scripts/pytorch_to_onnx.py --model-path=model/public/human-pose-estimation-3d-0001 --model-name=PoseEstimationWithMobileNet --model-param=is_convertible_by_mo=True --import-module=model --weights=model/public/human-pose-estimation-3d-0001/human-pose-estimation-3d-0001.pth --input-shape=1,3,256,448 --input-names=data --output-names=features,heatmaps,pafs --output-file=model/public/human-pose-estimation-3d-0001/human-pose-estimation-3d-0001.onnx
ONNX check passed successfully.
========== Converting human-pose-estimation-3d-0001 to IR (FP32)
Conversion command: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/bin/python -- /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/.venv/bin/mo --framework=onnx --output_dir=model/public/human-pose-estimation-3d-0001/FP32 --model_name=human-pose-estimation-3d-0001 --input=data '--mean_values=data[128.0,128.0,128.0]' '--scale_values=data[255.0,255.0,255.0]' --output=features,heatmaps,pafs --input_model=model/public/human-pose-estimation-3d-0001/human-pose-estimation-3d-0001.onnx '--layout=data(NCHW)' '--input_shape=[1, 3, 256, 448]' --compress_to_fp16=False
[ INFO ] MO command line tool is considered as the legacy conversion API as of OpenVINO 2023.2 release. In 2025.0 MO command line tool and openvino.tools.mo.convert_model() will be removed. Please use OpenVINO Model Converter (OVC) or openvino.convert_model(). OVC represents a lightweight alternative of MO and provides simplified model conversion API. Find more information about transition from MO to OVC at https://docs.openvino.ai/2023.2/openvino_docs_OV_Converter_UG_prepare_model_convert_model_MO_OVC_transition.html
[ SUCCESS ] Generated IR version 11 model.
[ SUCCESS ] XML file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/notebooks/3D-pose-estimation-webcam/model/public/human-pose-estimation-3d-0001/FP32/human-pose-estimation-3d-0001.xml
[ SUCCESS ] BIN file: /opt/home/k8sworker/ci-ai/cibuilds/ov-notebook/OVNotebookOps-727/.workspace/scm/ov-notebook/notebooks/3D-pose-estimation-webcam/model/public/human-pose-estimation-3d-0001/FP32/human-pose-estimation-3d-0001.bin推論デバイスの選択#
OpenVINO を使用して推論を実行するためにドロップダウン・リストからデバイスを選択します
core = ov.Core()
device = widgets.Dropdown(
options=core.available_devices + ["AUTO"],
value="AUTO",
description="Device:",
disabled=False,
)
deviceDropdown(description='Device:', index=1, options=('CPU', 'AUTO'), value='AUTO')モデルのロード#
変換されたモデルは、ベンダー、モデル名、精度を示す固定構造に配置されます。
まず、推論エンジンである OpenVINO ランタイムを初期化します。次に、.bin および .xml ファイルからネットワーク・アーキテクチャーとモデルの重みを読み取り、目的のデバイス用にコンパイルします。次に、コンパイルされたモデルを推論するため推論要求が作成されます。
# 推論エンジンを初期化
core = ov.Core()
# ファイルからネットワークと対応する重みを読み取り
model = core.read_model(model=ir_model_path, weights=model_weights_path)
# 指定されたデバイスにモデルをロード
compiled_model = core.compile_model(model=model, device_name=device.value)
infer_request = compiled_model.create_infer_request()
input_tensor_name = model.inputs[0].get_any_name()
# ノードの入力名と出力名を取得
input_layer = compiled_model.input(0)
output_layers = list(compiled_model.outputs)モデルの入力は入力画像からのデータであり、出力はヒートマップ、PAF (パーツ・アフィニティー・フィールド)、およびフィーチャーです。
input_layer.any_name, [o.any_name for o in output_layers]('data', ['features', 'heatmaps', 'pafs'])処理#
モデル推論#
ビデオファイルまたはライブ Web カメラからキャプチャーされたフレームは、3D モデルの入力として使用されます。これは、出力ヒートマップ、PAF (パーツ・アフィニティー・フィールド)、およびフィーチャーを取得する方法です。
def model_infer(scaled_img, stride):
"""
Run model inference on the input image
Parameters:
scaled_img: resized image according to the input size of the model
stride: int, the stride of the window
"""
# 画像から余分なスペースを削除
img = scaled_img[
0 : scaled_img.shape[0] - (scaled_img.shape[0] % stride),
0 : scaled_img.shape[1] - (scaled_img.shape[1] % stride),
]
img = np.transpose(img, (2, 0, 1))[None,]
infer_request.infer({input_tensor_name: img})
# 3 つの推論結果のセットを取得
results = {name: infer_request.get_tensor(name).data[:] for name in {"features", "heatmaps", "pafs"}}
# 結果を取得
results = (results["features"][0], results["heatmaps"][0], results["pafs"][0])
return results2D ポーズ・オーバーレイの描画#
推論結果を取得した後、結果の画像に人体の構造を描画できるように、関節の接続を事前に定義する必要があります。関節は円として描画され、手足は線で描画されます。このコードは、Open Model Zoo の 3D Human Pose Estimation Demo をベースにしています。
# 3D エッジ・インデックス配列
body_edges = np.array(
[
[0, 1],
[0, 9],
[9, 10],
[10, 11], # neck - r_shoulder - r_elbow - r_wrist
[0, 3],
[3, 4],
[4, 5], # neck - l_shoulder - l_elbow - l_wrist
[1, 15],
[15, 16], # nose - l_eye - l_ear
[1, 17],
[17, 18], # nose - r_eye - r_ear
[0, 6],
[6, 7],
[7, 8], # neck - l_hip - l_knee - l_ankle
[0, 12],
[12, 13],
[13, 14], # neck - r_hip - r_knee - r_ankle
]
)
body_edges_2d = np.array(
[
[0, 1], # neck - nose
[1, 16],
[16, 18], # nose - l_eye - l_ear
[1, 15],
[15, 17], # nose - r_eye - r_ear
[0, 3],
[3, 4],
[4, 5], # neck - l_shoulder - l_elbow - l_wrist
[0, 9],
[9, 10],
[10, 11], # neck - r_shoulder - r_elbow - r_wrist
[0, 6],
[6, 7],
[7, 8], # neck - l_hip - l_knee - l_ankle
[0, 12],
[12, 13],
[13, 14], # neck - r_hip - r_knee - r_ankle
]
)
def draw_poses(frame, poses_2d, scaled_img, use_popup):
"""
Draw 2D pose overlays on the image to visualize estimated poses.
Joints are drawn as circles and limbs are drawn as lines.
:param frame: the input image
:param poses_2d: array of human joint pairs
"""
for pose in poses_2d:
pose = np.array(pose[0:-1]).reshape((-1, 3)).transpose()
was_found = pose[2] > 0
pose[0], pose[1] = (
pose[0] * frame.shape[1] / scaled_img.shape[1],
pose[1] * frame.shape[0] / scaled_img.shape[0],
)
# Draw joints.
for edge in body_edges_2d:
if was_found[edge[0]] and was_found[edge[1]]:
cv2.line(
frame,
tuple(pose[0:2, edge[0]].astype(np.int32)),
tuple(pose[0:2, edge[1]].astype(np.int32)),
(255, 255, 0),
4,
cv2.LINE_AA,
)
# Draw limbs.
for kpt_id in range(pose.shape[1]):
if pose[2, kpt_id] != -1:
cv2.circle(
frame,
tuple(pose[0:2, kpt_id].astype(np.int32)),
3,
(0, 255, 255),
-1,
cv2.LINE_AA,
)
return frameメイン処理関数#
指定されたソースで 3D 姿勢推定を実行します。ウェブカメラのフィードまたはビデオファイルのどちらかである可能性があります。
def run_pose_estimation(source=0, flip=False, use_popup=False, skip_frames=0):
"""
2D image as input, using OpenVINO as inference backend,
get joints 3D coordinates, and draw 3D human skeleton in the scene
:param source: The webcam number to feed the video stream with primary webcam set to "0", or the video path.
:param flip: To be used by VideoPlayer function for flipping capture image.
:param use_popup: False for showing encoded frames over this notebook, True for creating a popup window.
:param skip_frames: Number of frames to skip at the beginning of the video.
"""
focal_length = -1 # デフォルト
stride = 8
player = None
skeleton_set = None
try:
# ターゲット fps で再生するビデオプレーヤーを作成
# video_path はカメラからフレームを取得
# 最初の N フレームをスキップしてビデオを早送り。'skip_first_frames’ を変更
player = utils.VideoPlayer(source, flip=flip, fps=30, skip_first_frames=skip_frames)
# キャプチャーを開始
player.start()
input_image = player.next()
# ウィンドウサイズを設定
resize_scale = 450 / input_image.shape[1]
windows_width = int(input_image.shape[1] * resize_scale)
windows_height = int(input_image.shape[0] * resize_scale)
# 視覚化ライブラリーを使用
engine3D = engine.Engine3js(grid=True, axis=True, view_width=windows_width, view_height=windows_height)
if use_popup:
# このノートブックに 3D の人間のポーズを表示し、ポップアップ・ウィンドウに原点フレームを表示
display(engine3D.renderer)
title = "Press ESC to Exit"
cv2.namedWindow(title, cv2.WINDOW_KEEPRATIO | cv2.WINDOW_AUTOSIZE)
else:
# 2D 画像ボックスを設定し、ノートブックに人間のポーズと画像の両方を表示
imgbox = widgets.Image(format="jpg", height=windows_height, width=windows_width)
display(widgets.HBox([engine3D.renderer, imgbox]))
skeleton = engine.Skeleton(body_edges=body_edges)
processing_times = collections.deque()
while True:
# フレームをグラブ
frame = player.next()
if frame is None:
print("Source ended")
break
# ニューラル・ネットワークの入力に合わせて画像のサイズを変更して暗さを変更
# (https://github.com/openvinotoolkit/open_model_zoo/tree/master/models/public/human-pose-estimation-3d-0001 を参照)
scaled_img = cv2.resize(frame, dsize=(model.inputs[0].shape[3], model.inputs[0].shape[2]))
if focal_length < 0: # 焦点距離は不明
focal_length = np.float32(0.8 * scaled_img.shape[1])
# 推論を開始
start_time = time.time()
# 結果を取得
inference_result = model_infer(scaled_img, stride)
# 推論を停止
stop_time = time.time()
processing_times.append(stop_time - start_time)
# データのポイントツーポイント座標を処理
poses_3d, poses_2d = engine.parse_poses(inference_result, 1, stride, focal_length, True)
# 最後の 200 フレームの処理時間を使用
if len(processing_times) > 200:
processing_times.popleft()
processing_time = np.mean(processing_times) * 1000
fps = 1000 / processing_time
if len(poses_3d) > 0:
# ここから、関数 "draw_poses" を使用して 3D ポイントの位置を回転、
# または、以下の正しいマッピングを直接行うことで、画面上にオブジェクト画像を適切に表示
poses_3d_copy = poses_3d.copy()
x = poses_3d_copy[:, 0::4]
y = poses_3d_copy[:, 1::4]
z = poses_3d_copy[:, 2::4]
poses_3d[:, 0::4], poses_3d[:, 1::4], poses_3d[:, 2::4] = (
-z + np.ones(poses_3d[:, 2::4].shape) * 200,
-y + np.ones(poses_3d[:, 2::4].shape) * 100,
-x,
)
poses_3d = poses_3d.reshape(poses_3d.shape[0], 19, -1)[:, :, 0:3]
people = skeleton(poses_3d=poses_3d)
try:
engine3D.scene_remove(skeleton_set)
except Exception:
pass
engine3D.scene_add(people)
skeleton_set = people
# 2D 描画
frame = draw_poses(frame, poses_2d, scaled_img, use_popup)
else:
try:
engine3D.scene_remove(skeleton_set) skeleton_set = None
except Exception:
pass
cv2.putText(
frame,
f"Inference time: {processing_time:.1f}ms ({fps:.1f} FPS)",
(10, 30),
cv2.FONT_HERSHEY_COMPLEX,
0.7,
(0, 0, 255),
1,
cv2.LINE_AA,
)
if use_popup:
cv2.imshow(title, frame)
key = cv2.waitKey(1)
# escape = 27、ESC キーを押して終了
if key == 27:
break
else:
# numpy 配列を jpg にエンコード
imgbox.value = cv2.imencode(
".jpg",
frame,
params=[cv2.IMWRITE_JPEG_QUALITY, 90],
)[1].tobytes()
engine3D.renderer.render(engine3D.scene, engine3D.cam)
except KeyboardInterrupt:
print("Interrupted")
except RuntimeError as e:
print(e)
finally:
clear_output()
if player is not None:
# キャプチャーを停止
player.stop()
if use_popup:
cv2.destroyAllWindows()
if skeleton_set:
engine3D.scene_remove(skeleton_set)実行#
Web カメラをビデオ入力として使用して実行します。デフォルトでは、プライマリー Web カメラは source=0 に設定されます。複数のウェブカメラがある場合、0 から始まる連続した番号が割り当てられます。前面カメラを使用する場合は、flip=True を設定します。一部のウェブブラウザー、特に Mozilla Firefox ではちらつきが発生する場合があります。ちらつきが発生する場合、use_popup=True を設定してください。
注:
- このノートブックをウェブカメラで使用するには、ウェブカメラを備えたコンピューター上でノートブックを実行する必要があります。ノートブックをサーバー (Binder など) 上で実行する場合、ウェブカメラは機能しません。
- このノートブックをリモート・コンピューター (Binder など) で実行する場合、ポップアップ・モードは機能しない可能性があります。
ウェブカメラがない場合でも、ビデオファイルを使用してこのデモを実行できます。OpenCV でサポートされている形式であればどれでも機能します。
次の方法を使用すると、左側の画像をクリックしてマウスを移動して操作できます。
USE_WEBCAM = False
cam_id = 0
video_path = "https://github.com/intel-iot-devkit/sample-videos/raw/master/face-demographics-walking.mp4"
source = cam_id if USE_WEBCAM else video_path
run_pose_estimation(source=source, flip=isinstance(source, int), use_popup=False)