本文共 19565 字，大约阅读时间需要 65 分钟。
by Gaurav Kaila
由高拉夫·凯拉(Gaurav Kaila)
Object detection models are some of the most sophisticated deep learning models. They’re capable of localizing and classifying objects in real time both in images and videos. But what good is a model if it cannot be used for production?
对象检测模型是一些最复杂的深度学习模型。 它们能够实时定位和分类图像和视频中的对象。 但是，如果模型不能用于生产，那有什么好处呢？
Thanks to the wonderful guys at TensorFlow, we have TensorFlow serving that is capable of serving our models in production. There are some really good articles on TensorFlow serving to get you started such as one and one.
感谢TensorFlow的出色人才，我们拥有TensorFlow服务，能够在生产中为我们的模型提供服务。 TensorFlow上有一些非常好的文章可以帮助您入门，例如和 。
This article will focus on how we can serve Object Detection Models specifically with TF Serving. It is motivated by the lack of a good resource online that explains how to create production-ready object detection models and TF-serving environments using Docker. We’ll also discuss how to serve the model and create a client side script to access it. Our architecture will look something like this:
本文将重点介绍如何特别通过TF服务来提供对象检测模型 。 它的原因是缺乏在线资源，该资源解释了如何使用Docker创建生产就绪的对象检测模型和TF服务环境。 我们还将讨论如何提供模型并创建客户端脚本来访问它。 我们的架构如下所示：
In the spirit of not reinventing the wheel, I have taken help from resources available in the for this tutorial. I assume you have cloned the object detection API from TensorFlow — but if not, do the following:
本着不浪费时间的精神，我从本教程的中的可用资源中获取了帮助。 我假设您已经从TensorFlow中克隆了对象检测API，但如果没有，请执行以下操作：
# Clone tensorlfow models repohttps://github.com/tensorflow/models.gitcd models/research/object_detection
Assuming you have trained your object detection model using TensorFlow, you will have the following four files saved in your disk:
假设您已经使用TensorFlow训练了对象检测模型，那么将在磁盘中保存以下四个文件：
These files can be used for . Or we can use f script to consisting of the model architecture and weights in one file. This is useful for testing purposes on your local machine, but is not suitable for a production environment.
这些文件可以直接用于 。 或者，我们可以使用f 脚本 ，该由一个文件中的模型架构和权重组成。 这对于在本地计算机上进行测试很有用，但不适用于生产环境。
To create models ready for serving, we will tweak the file available on the object detection API Github. The original script available on the repo does not save the Variables that are required for serving. Use the following exporter.py script instead of the original TensorFlow one.
要创建准备服务的模型，我们将调整对象检测API Github上可用的文件。 存储库上可用的原始脚本不会保存投放所需的变量 。 使用以下exporter.py脚本代替原始的TensorFlow脚本。
The following changes have been made to the above exporter.py:
对上述exporter.py进行了以下更改：
Change to the _write_saved_model method. This is required, as the original python script does not save variables which are required for serving the model. Now instead of using the frozen_graph_def, we use the trained_checkpoint_prefix that has the weights of the model as variables. (credits to this )
更改为_write_saved_model方法。 这是必需的，因为原始的python脚本不会保存服务模型所需的变量 。 现在，而不是使用frozen_graph_def，我们使用具有模型变量的权重trained_checkpoint_prefix。 (此贷方)
2. Change the calling function from frozen_graph_def to trained_checkpoint_prefix by following:
2.将调用函数从freeze_graph_def更改为trained_checkpoint_prefix 依照指示：
3. Comment out code that saves files to disk not required during serving:
3.注释掉在服务期间不需要将文件保存到磁盘的代码：
Now you are all ready to create your model that can be used for serving. The following code can help you achieve that:
现在，您已经准备好创建可用于服务的模型。 以下代码可以帮助您实现这一目标：
Here’s an explanation of that code:
这是该代码的解释：
Each object detection model has a configuration which needs to be passed to the export_model.py. This consists of information regarding the model architecture. For more information, refer to this .
每个对象检测模型都有一个配置，需要将该配置传递给export_model.py。 这包括有关模型架构的信息。 有关更多信息，请参考此 。
The get_configs_from_pipeline_file method creates a dictionary from the configuration file, and the create_pipeline_proto_from_configs method creates a proto buffer object from this dictionary.
get_configs_from_pipeline_file方法 从配置文件创建一个字典，然后create_pipeline_proto_from_configs方法从该字典创建一个原型缓冲区对象。
input_checkpoint is the path to model.ckpt of the trained model.
input_checkpoint是model.ckpt训练的模型的路径。
model_version_id is an integer for the current version of the model. This is required by TF-serving for the versioning of models.
model_version_id是模型当前版本的整数。 TF服务需要使用此功能来进行模型版本控制。
object_detection.exporter will save the model in the following format:
object_detection.exporter将以以下格式保存模型：
1/ is the model version, saved_model.pb. It contains the model architecture, and the variables directory has the weights for the model. This model is ready to be served.
1 /是模型版本， saved_model.pb。 它包含模型架构，并且变量目录具有模型的权重。 该模型已准备就绪。
Docker is a software tool that lets you package software into standardised units for development, shipment and deployment. Docker container image is a lightweight, stand-alone, executable package of a piece of software that includes everything needed to run it: code, runtime, system tools, system libraries, settings.
Docker是一种软件工具，可让您将软件打包到标准化单元中，以进行开发，运输和部署。 Docker容器映像是一个软件的轻量级独立可执行软件包，包括运行该映像所需的一切：代码，运行时，系统工具，系统库，设置。
In short, Docker lets us you isolate your application and its dependancies in a stand-alone package that can be used anywhere and anytime without having to worry about installing code and system dependancies.
简而言之，Docker使我们能够将您的应用程序及其依赖关系隔离在一个独立的程序包中，该程序包可随时随地使用，而不必担心安装代码和系统依赖关系。
Our motivation for using docker for TensorFlow serving is that we can ship our container to run on the cloud and easily scale our service without having to install any dependancies again.
我们使用Docker进行TensorFlow服务的动机是我们可以将容器运送到云上运行，并轻松扩展我们的服务而无需再次安装任何依赖关系。
of TensorFlow serving describes how to build it from source. It’s good but I (and a ) had problems compiling it in the docker container. So we will go over the steps one-by-one here.
TensorFlow服务的描述了如何从源代码构建它。 很好，但是我(和 )在docker容器中编译它时遇到了问题。 因此，我们将在这里一步一步地进行操作。
Build the container using the official docker image
使用官方docker镜像构建容器
Assuming you have cloned the official TensorFlow serving repo as described in the last part, you can build the docker image by doing the following:
假设您已如上一部分中所述克隆了正式的TensorFlow服务存储库，则可以通过执行以下操作来构建Docker映像：
# Move to the directory of the docker filescd ./serving/tensorflow_serving/tools/docker/
# Build the image (CPU)docker build --pull -t $USER/tensorflow-serving-devel-cpu -f Dockerfile.devel .
or
# Build the image (GPU)docker build --pull -t $USER/tensorflow-serving-devel-gpu -f Dockerfile.devel-gpu .
Before starting the docker container, increase the memory (to 10–12 GBs) and CPUs (to 4–6) available to the container in the preferences section of the docker app. Building TensorFlow serving is a memory-intensive process and the default parameters might not work. Once done, you can start the container like this:
在启动Docker容器之前，请在Docker应用程序的“首选项”部分中，为容器提供可用的内存(至10–12 GB)和CPU(至4–6)。 构建TensorFlow服务是一个占用大量内存的过程，默认参数可能不起作用。 完成后，您可以像这样启动容器：
[FOR CPU]docker run -it -p 9000:9000 $USER/tensorflow-serving-devel-cpu /bin/bash
or
[FOR GPU]docker run -it -p 9000:9000 $USER/tensorflow-serving-devel-gpu /bin/bash
In the container, do the following:
在容器中，执行以下操作：
[FOR CPU]# Clone the TensorFlow serving Github repo in the containergit clone --recurse-submodules https://github.com/tensorflow/servingcd serving/tensorflow
# Configure TensorFlow./configurecd ..
# Build TensorFlow servingbazel build -c opt --copt=-msse4.1 --copt=-msse4.2 tensorflow_serving/...
or
[FOR GPU]# TensorFlow serving Github repo is already present in the container # so do not need to clone again# Configure TensorFlow with CUDA by accepting (-y) --# with_CUDA_support flagcd serving/tensorflow./configure
# Build TensorFlow serving with CUDA bazel build -c opt --copt=-msse4.1 --copt=-msse4.2 --copt=-mavx --copt=-mavx2 --copt=-mfma --copt=-O3 --copt=/usr/local/cuda tensorflow_serving/...
The build process can take up to an hour depending the host system and docker configuration. Once the build is finished without any errors, you can test if the model server is running:
根据主机系统和docker配置的不同，构建过程最多可能需要一个小时。 一旦构建完成且没有任何错误，您可以测试模型服务器是否正在运行：
bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server
The output should look something like this:
输出应如下所示：
Flags:
--port=8500 int32 port to listen on
--enable_batching=false bool enable batching
--batching_parameters_file="" string If non-empty, read an ascii BatchingParameters protobuf from the supplied file name and use the contained values instead of the defaults.
--model_config_file="" string If non-empty, read an ascii ModelServerConfig protobuf from the supplied file name, and serve the models in that file. This config file can be used to specify multiple models to serve and other advanced parameters including non-default version policy. (If used, --model_name, --model_base_path are ignored.)
--model_name="default" string name of model (ignored if --model_config_file flag is set
--model_base_path="" string path to export (ignored if --model_config_file flag is set, otherwise required)
--file_system_poll_wait_seconds=1 int32 interval in seconds between each poll of the file system for new model version
--tensorflow_session_parallelism=0 int64 Number of threads to use for running a Tensorflow session. Auto-configured by default.Note that this option is ignored if --platform_config_file is non-empty.
--platform_config_file="" string If non-empty, read an ascii PlatformConfigMap protobuf from the supplied file name, and use that platform config instead of the Tensorflow platform. (If used, --enable_batching is ignored.)
Your serving environment is now ready to be used. Exit the container and commit the changes in the container to an image. You can do this like so:
现在可以使用您的服务环境了。 退出容器并将容器中的更改提交到映像。 您可以这样做：
# Find the container Iddocker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
# Commit the changes[FOR CPU]docker commit ${CONTAINER ID} $USER/tensorflow-serving-devel-cpu or
[FOR GPU]docker commit ${CONTAINER ID} $USER/tensorflow-serving-devel-gpu docker exec -it ${CONTAINER ID} /bin/bash Note: For the TensorFlow serving container to access the GPUs on your host system, you need to on your system and run the container like this:
注意：为了使TensorFlow服务容器访问主机系统上的GPU，您需要在系统上并运行容器，如下所示：
nvidia-docker docker run -it -p 9000:9000 $USER/tensorflow-serving-devel-gpu /bin/bash
You can then check your GPU usage inside the container by using the nvidia-smi cmd.
然后，您可以使用nvidia-smi检查容器内的GPU使用情况 指令
As I’ve seen on a number of Github issues (see resources), people are unable to compile TensorFlow serving on docker. So I have pre-built docker images for both CPU and GPU support.
正如我在许多Github问题上看到的那样(请参阅参考资料)，人们无法在docker上编译TensorFlow服务。 因此，我已经为CPU和GPU支持预先构建了docker映像。
You can find them at my or you can pull the images down like this:
您可以在我的找到它们，也可以像这样下拉图像：
[FOR CPU]docker pull gauravkaila/tf_serving_cpu
or
[FOR GPU]docker pull gauravkaila/tf_serving_gpu
gRPC (Google’s Remote Procedure Call) is Google’s HTTP2 wrapped RPC protocol. This allows a client running on a computer to access a remote computer, via the computer network, and call a “function” on that remote computer as if the function was local to the client.
gRPC(Google的远程过程调用)是Google的HTTP2包装的RPC协议。 这允许在计算机上运行的客户端通过计算机网络访问远程计算机，并在该远程计算机上调用“功能”，就好像该功能是客户端本地的一样。
TensorFlow serving uses this protocol to serve models for inference. According to the
TensorFlow服务使用此协议为推理服务模型。 根据
In gRPC a client application can directly call methods on a server application on a different machine as if it was a local object, making it easier for you to create distributed applications and services.
在gRPC中，客户端应用程序可以直接在其他计算机上的服务器应用程序上调用方法，就好像它是本地对象一样，这使您更容易创建分布式应用程序和服务。
Here, the gRPC server is our docker container running the TensorFlow serving service, and our client is in python that requests this service for inference.
在这里，gRPC服务器是运行TensorFlow服务服务的docker容器，而我们的客户端位于python中，请求此服务进行推断。
gRPC uses to serialise structured data as well as define the parameters and return responses for the callable methods. It is language-neutral and platform-neutral. It has a structured language which then compiles the transport serialisation code into your chosen language to be included in your project. It transmits data in binary format which is smaller and faster compared to good old JSON and XML.
gRPC使用 序列化结构化数据以及定义参数并返回可调用方法的响应。 它与语言无关，与平台无关。 它具有结构化语言，然后将传输序列化代码编译为您选择的语言，以包含在您的项目中。 它以二进制格式传输数据，与老式的JSON和XML相比，该格式更小，更快。
A TensorFlow serving request can be one of three types:
TensorFlow服务请求可以是以下三种类型之一：
As the problem at hand here is a prediction problem, we will be using the prediction RPC API. For this we need the predict protobuf’s available on the , and we need to convert them into our language-specific code (i.e. Python).
由于这里的问题是预测问题，因此我们将使用预测RPC API。 为此，我们需要可在上使用的预测protobuf，并将其转换为特定于语言的代码(即Python)。
You can do this yourself, or go the easy way and download the python files from this We will use this protobuf object to create a prediction request in our client.
您可以自己执行此操作，也可以通过简单的方法从此下载python文件 我们将使用此protobuf对象在客户端中创建预测请求。
Template of a Prediction RPC client
预测RPC客户端的模板
Stub is a piece of code that is used to convert parameters during a remote procedure call (RPC). As the client and the server sit in different address spaces, the parameter sent from the client to the server (and vice-versa) needs to be converted so that the remote server computer perceives the RPC as a local function call. The stub used here is the code generated from the predict protobuf as described above.
存根是一段代码，用于在远程过程调用(RPC)期间转换参数。 由于客户端和服务器位于不同的地址空间中，因此需要转换从客户端发送到服务器的参数(反之亦然)，以便远程服务器计算机将RPC视为本地函数调用。 此处使用的存根是如上所述的从预测protobuf生成的代码。
As described in the previous part, our TensorFlow serving service will run in a docker container with ports open to the outside world. Assuming the docker image is available, the container can be started like this:
如前一部分所述，我们的TensorFlow服务将在一个docker容器中运行，并向外界开放端口。 假设Docker映像可用，则可以像这样启动容器：
$ docker run -it -d -P --name tf_serving_cpu -p 3000:3000 gauravkaila/tf_serving_cpu
Here, the port 3000 is open to the world and the client can access the TensorFlow serving service via this port. Export the model directory created in the first part to a folder inside the container:
在这里，端口3000向世界开放，客户端可以通过该端口访问TensorFlow服务服务。 将在第一部分中创建的模型目录导出到容器内的文件夹中：
$ docker cp /path/to/model tf_serving_cpu:/path/to/destination
To run the service, move into the container and start:
要运行该服务，请移入容器并启动：
# Move to the serving/ directory$ cd serving/
# Start the service$ bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=3000 --model_name=obj_det--model_base_path=/path/to/dest &> obj_det &
Make sure the model_name flag has the same name as specified in the client. The output is logged in obj_det . If all went well, you will be able to see the following output when you type:
确保model_name标志具有与客户端中指定的名称相同的名称。 输出记录在obj_det中 。 如果一切顺利，键入时您将能够看到以下输出：
$ tail -f obj_det
tensorflow_serving/model_servers/main.cc:288] Running ModelServer at 0.0.0.0:3000 …
tensorflow_serving / model_servers / main.cc：288]在0.0.0.0:3000上运行ModelServer…
The model is being served and is ready to be used by our client.
该模型正在提供中，可供客户使用。
The aim of an object detection model is to visualise the bounding boxes of the located objects on the image. In order to visualise the final image with the bounding boxes, we will use the file from the TensorFlow object detection API.
对象检测模型的目的是可视化图像上已定位对象的边界框。 为了使带有边框的最终图像可视化，我们将使用 TensorFlow对象检测API中的文件。
We can access the individual outputs from the result like this:
我们可以从结果中访问单个输出，如下所示：
boxes = result.outputs['detection_boxes'].float_valclasses = result.outputs['detection_classes'].float_valscores = result.outputs['detection_scores'].float_val
This returns protobuf objects that can be fed into the visualization_utils.py file:
这将返回可以添加到visualization_utils.py文件中的protobuf对象：
image_vis = vis_util.visualize_boxes_and_labels_on_image_array( {input_image}, np.reshape(boxes,[100,4]), np.squeeze(classes).astype(np.int32), np.squeeze(scores), category_index, use_normalized_coordinates=True, line_thickness=8) The final client script will look like this:
最终的客户端脚本将如下所示：
Sending in a test image of a clock, our final output should look something like this. Note: the model used here is a faster RCNN pre-trained on COCO dataset for which class number 85 corresponds to a clock.
发送时钟的测试图像后，我们的最终输出应如下所示。 注意 ：这里使用的模型是在COCO数据集上预训练的更快的RCNN，其类别号85对应于时钟。
outputs {key: “detection_boxes”value {dtype: DT_FLOATtensor_shape {dim {size: 1}dim {size: 300}dim {size: 4}}float_val: 0.24750074744224548float_val: 0.17159424722194672float_val: 0.9083144068717957float_val: 0.797699511051178 outputs {key: "detection_classes"value {dtype: DT_FLOATtensor_shape {dim {size: 1}dim {size: 300}}float_val: 85.0 outputs {key: "detection_scores"value {dtype: DT_FLOATtensor_shape {dim {size: 1}dim {size: 300}}float_val: 0.9963208436965942 We started of with an object detection use-case to demonstrate the power of TensorFlow serving. We exported our trained model to a format expected by TensorFlow serving, compiled TF-serving using Docker, and created a client script that could request the model server for inference.
我们从对象检测用例开始，以演示TensorFlow服务的功能。 我们将训练后的模型导出为TensorFlow服务期望的格式，使用Docker编译了TF服务，并创建了可以请求模型服务器进行推理的客户端脚本。
Github issues:
Github问题：
About the author: Gaurav is a machine learning engineer at The Dock, Accenture’s premier research and innovation centre in Dublin, Ireland. His interests include building scalable deep learning systems for computer vision applications. Find more at
关于作者： Gaurav是埃森哲位于爱尔兰都柏林的首要研究与创新中心The Dock的机器学习工程师。 他的兴趣包括为计算机视觉应用程序构建可扩展的深度学习系统。 在找到更多
翻译自:
转载地址：http://dfzzd.baihongyu.com/