MobileNetv2-SSDLite 
Environment
- Ubuntu 16.04
- CUDA 9.0
- cuDNN 7
- caffe-ssd
- Tensorflow-GPU v1.12.0
- [Docker] nvidia/cuda:9.0-cudnn7-devel-ubuntu16.04 here
Procedure
Advance_preparation
$ sudo apt-get update;sudo apt-get upgrade -y
$ sudo apt-get install -y autoconf automake libtool curl make g++ unzip wget git nano
$ sudo apt-get install -y --no-install-recommends libboost-all-dev libopenblas-dev libsnappy-dev
$ sudo apt-get install -y libgflags-dev libgoogle-glog-dev liblmdb-dev libleveldb-dev
$ sudo apt-get install -y libhdf5-10 libhdf5-serial-dev libhdf5-dev libhdf5-cpp-11
$ sudo apt-get install -y python3-dev python3-numpy python3-skimage gfortran libturbojpeg
$ sudo apt-get install -y python-dev python-numpy python-skimage
$ sudo apt-get install -y python3-pip python-pip
$ sudo -H pip install pip --upgrade
$ sudo -H pip3 install pip --upgrade
$ sudo -H pip install opencv-python
$ sudo -H pip3 install opencv-python
$ cd ~
$ wget https://github.com/protocolbuffers/protobuf/releases/download/v2.6.1/protobuf-2.6.1.zip
$ unzip protobuf-2.6.1.zip;rm protobuf-2.6.1.zip
$ cd protobuf-2.6.1
$ ./autogen.sh
$ ./configure
$ make
$ sudo make install
$ sudo ldconfig
Custom_Caffe_Clone
$ cd ~
$ git clone https://github.com/chuanqi305/ssd.git
$ cd ssd
$ cp Makefile.config.example Makefile.config
$ nano Makefile.config
Edit_Makefile.config
## Refer to http://caffe.berkeleyvision.org/installation.html
# Contributions simplifying and improving our build system are welcome!
# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1
# CPU-only switch (uncomment to build without GPU support).
# CPU_ONLY := 1
# uncomment to disable IO dependencies and corresponding data layers
USE_OPENCV := 0
# USE_LEVELDB := 0
# USE_LMDB := 0
# uncomment to allow MDB_NOLOCK when reading LMDB files (only if necessary)
# You should not set this flag if you will be reading LMDBs with any
# possibility of simultaneous read and write
# ALLOW_LMDB_NOLOCK := 1
# Uncomment if you're using OpenCV 3
# OPENCV_VERSION := 3
# To customize your choice of compiler, uncomment and set the following.
# N.B. the default for Linux is g++ and the default for OSX is clang++
# CUSTOM_CXX := g++
# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
# On Ubuntu 14.04, if cuda tools are installed via
# "sudo apt-get install nvidia-cuda-toolkit" then use this instead:
# CUDA_DIR := /usr
# CUDA architecture setting: going with all of them.
# For CUDA < 6.0, comment the lines after *_35 for compatibility.
CUDA_ARCH := -gencode arch=compute_30,code=sm_30 \
-gencode arch=compute_35,code=sm_35 \
-gencode arch=compute_50,code=sm_50 \
-gencode arch=compute_52,code=sm_52 \
-gencode arch=compute_61,code=sm_61
# BLAS choice:
# atlas for ATLAS (default)
# mkl for MKL
# open for OpenBlas
# BLAS := atlas
BLAS := open
# Custom (MKL/ATLAS/OpenBLAS) include and lib directories.
# Leave commented to accept the defaults for your choice of BLAS
# (which should work)!
# BLAS_INCLUDE := /path/to/your/blas
# BLAS_LIB := /path/to/your/blas
# Homebrew puts openblas in a directory that is not on the standard search path
# BLAS_INCLUDE := $(shell brew --prefix openblas)/include
# BLAS_LIB := $(shell brew --prefix openblas)/lib
# This is required only if you will compile the matlab interface.
# MATLAB directory should contain the mex binary in /bin.
# MATLAB_DIR := /usr/local
# MATLAB_DIR := /Applications/MATLAB_R2012b.app
# NOTE: this is required only if you will compile the python interface.
# We need to be able to find Python.h and numpy/arrayobject.h.
# PYTHON_INCLUDE := /usr/include/python2.7 \
# /usr/lib/python2.7/dist-packages/numpy/core/include
# Anaconda Python distribution is quite popular. Include path:
# Verify anaconda location, sometimes it's in root.
# ANACONDA_HOME := $(HOME)/anaconda2
# PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
$(ANACONDA_HOME)/include/python2.7 \
$(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \
# Uncomment to use Python 3 (default is Python 2)
PYTHON_LIBRARIES := boost_python-py35 python3.5m
PYTHON_INCLUDE := /usr/include/python3.5m \
/usr/lib/python3.5/dist-packages/numpy/core/include \
/usr/local/lib/python3.5/dist-packages/numpy/core/include
# We need to be able to find libpythonX.X.so or .dylib.
PYTHON_LIB := /usr/lib
# PYTHON_LIB := $(ANACONDA_HOME)/lib
# Homebrew installs numpy in a non standard path (keg only)
# PYTHON_INCLUDE += $(dir $(shell python -c 'import numpy.core; print(numpy.core.__file__)'))/include
# PYTHON_LIB += $(shell brew --prefix numpy)/lib
# Uncomment to support layers written in Python (will link against Python libs)
WITH_PYTHON_LAYER := 1
# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial
# If Homebrew is installed at a non standard location (for example your home directory) and you use it for general dependencies
# INCLUDE_DIRS += $(shell brew --prefix)/include
# LIBRARY_DIRS += $(shell brew --prefix)/lib
# Uncomment to use `pkg-config` to specify OpenCV library paths.
# (Usually not necessary -- OpenCV libraries are normally installed in one of the above $LIBRARY_DIRS.)
# USE_PKG_CONFIG := 1
# N.B. both build and distribute dirs are cleared on `make clean`
BUILD_DIR := build
DISTRIBUTE_DIR := distribute
# Uncomment for debugging. Does not work on OSX due to https://github.com/BVLC/caffe/issues/171
# DEBUG := 1
# The ID of the GPU that 'make runtest' will use to run unit tests.
TEST_GPUID := 0
# enable pretty build (comment to see full commands)
Q ?= @
Custom_Caffe_Build
$ sudo -H pip install -r python/requirements.txt
$ sudo -H pip3 install -r python/requirements.txt
$ rm -rf .build_release
$ rm -rf build
$ make all -j8
$ make pycaffe
$ export PYTHONPATH=/<caffe path>/python/:$PYTHONPATH
Download_MobileNetv2-SSDLite_COCO_Model
$ cd ~
$ git clone https://github.com/chuanqi305/MobileNetv2-SSDLite.git
$ cd ~/MobileNetv2-SSDLite/ssdlite
$ wget http://download.tensorflow.org/models/object_detection/ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz
$ tar -zxvf ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz
$ rm ssdlite_mobilenet_v2_coco_2018_05_09.tar.gz
Edit_weight_output_program
$ cd ~/MobileNetv2-SSDLite/ssdlite
$ nano dump_tensorflow_weights.py
dump_tensorflow_weights.py
import tensorflow as tf
import cv2
import numpy as np
import os
def graph_create(graphpath):
with tf.gfile.FastGFile(graphpath, 'rb') as graphfile:
graphdef = tf.GraphDef()
graphdef.ParseFromString(graphfile.read())
return tf.import_graph_def(graphdef, name='',return_elements=[])
graph_create("ssdlite_mobilenet_v2_coco_2018_05_09/frozen_inference_graph.pb")
save_dir = ('output')
if not os.path.exists(save_dir):
os.mkdir(save_dir)
with tf.Session() as sess:
tensors = [tensor for tensor in tf.get_default_graph().as_graph_def().node]
for t in tensors:
if ('weights' in t.name \
or 'bias' in t.name \
or 'moving_variance' in t.name \
or 'moving_mean' in t.name \
or 'beta' in t.name \
or 'gamma' in t.name \
or 'BatchNorm/batchnorm/sub' in t.name \
or 'BatchNorm/batchnorm/mul' in t.name) \
and ('sub' in t.name or 'mul/' in t.name or 'read' in t.name):
ts = tf.get_default_graph().get_tensor_by_name(t.name + ":0")
data = ts.eval()
#print(t.name)
names = t.name.split('/')
layer_name = None
paratype = None
if 'BatchNorm/batchnorm/sub' in t.name or 'biases' in t.name:
paratype = 'biases'
elif 'BatchNorm/batchnorm/mul' in t.name:
paratype = 'weights_scale'
elif 'weights' in t.name:
paratype = 'weights'
elif 'moving_variance' in t.name:
paratype = 'bn_moving_variance'
elif 'moving_mean' in t.name:
paratype = 'bn_moving_mean'
elif 'beta' in t.name:
paratype = 'beta'
elif 'gamma' in t.name:
paratype = 'gamma'
if names[2] == 'Conv' or names[2] == 'Conv_1':
layer_name = names[2]
elif 'expanded_conv' in names[2]:
layer_name = names[2].replace('expanded_', '') + '_' + names[3]
elif 'layer_19' in names[2]:
substr = names[2].split('_')
layer_name = 'layer_19_' + substr[2] + '_' + substr[4]
if 'depthwise' in names[2]:
layer_name += "_depthwise"
elif 'BoxPredictor' in names[0]:
layer_name = names[0] + '_' + names[1]
output_name = layer_name + "_" + paratype
print(output_name)
#print ts.get_shape()
if len(data.shape) == 4:
caffe_weights = data.transpose(3, 2, 0, 1)
origin_shape = caffe_weights.shape
boxes = 0
if 'depthwise' not in output_name:
if output_name.find('BoxEncodingPredictor') != -1:
boxes = caffe_weights.shape[0] // 4
elif output_name.find('ClassPredictor') != -1:
boxes = caffe_weights.shape[0] // 91
if output_name.find('BoxEncodingPredictor') != -1:
tmp = caffe_weights.reshape(boxes, 4, -1).copy()
new_weights = np.zeros(tmp.shape, dtype=np.float32)
#tf order: [y, x, h, w]
#caffe order: [x, y, w, h]
if 'BoxPredictor_0/BoxEncodingPredictor/weights' in t.name:
#caffe first box layer [(0.2, 1.0), (0.2, 2.0), (0.2, 0.5)]
#tf first box layer [(0.1, 1.0), (0.2, 2.0), (0.2, 0.5)]
#adjust the box by weights and bias change
new_weights[:, 0] = tmp[:, 1] * 0.5
new_weights[:, 1] = tmp[:, 0] * 0.5
else:
new_weights[:, 0] = tmp[:, 1]
new_weights[:, 1] = tmp[:, 0]
new_weights[:, 2] = tmp[:, 3]
new_weights[:, 3] = tmp[:, 2]
caffe_weights = new_weights.reshape(origin_shape).copy()
if output_name.find('BoxEncodingPredictor') != -1 or \
output_name.find('ClassPredictor') != -1:
tmp = caffe_weights.reshape(boxes, -1).copy()
new_weights = np.zeros(tmp.shape, dtype=np.float32)
#tf aspect ratio: [1, 2, 3, 0.5, 0.333333333, 1]
#caffe aspect ratio:[1, 1, 2, 3, 0.5, 0.333333333]
if boxes == 6:
new_weights[0] = tmp[0]
new_weights[1] = tmp[5]
new_weights[2] = tmp[1]
new_weights[3] = tmp[2]
new_weights[4] = tmp[3]
new_weights[5] = tmp[4]
caffe_weights = new_weights.reshape(origin_shape).copy()
caffe_weights.tofile(os.path.join(save_dir, output_name + '.dat'))
print(caffe_weights.shape)
else:
caffe_bias = data
boxes = 0
if 'depthwise' not in output_name:
if output_name.find('BoxEncodingPredictor') != -1:
boxes = caffe_bias.shape[0] // 4
elif output_name.find('ClassPredictor') != -1:
boxes = caffe_bias.shape[0] // 91
if output_name.find('BoxEncodingPredictor') != -1:
tmp = caffe_bias.reshape(boxes, 4).copy()
new_bias = np.zeros(tmp.shape, dtype=np.float32)
new_bias[:, 0] = tmp[:, 1]
new_bias[:, 1] = tmp[:, 0]
new_bias[:, 2] = tmp[:, 3]
new_bias[:, 3] = tmp[:, 2]
caffe_bias = new_bias.flatten().copy()
if output_name.find('BoxEncodingPredictor') != -1 or \
output_name.find('ClassPredictor') != -1:
tmp = caffe_bias.reshape(boxes, -1).copy()
new_bias = np.zeros(tmp.shape, dtype=np.float32)
if boxes == 6:
new_bias[0] = tmp[0]
new_bias[1] = tmp[5]
new_bias[2] = tmp[1]
new_bias[3] = tmp[2]
new_bias[4] = tmp[3]
new_bias[5] = tmp[4]
caffe_bias = new_bias.flatten()
elif 'BoxPredictor_0/BoxEncodingPredictor/biases' in t.name:
#caffe first box layer [(0.2, 1.0), (0.2, 2.0), (0.2, 0.5)]
#tf first box layer [(0.1, 1.0), (0.2, 2.0), (0.2, 0.5)]
#adjust the box by weights and bias change
new_bias[0,:2] = tmp[0,:2] * 0.5
new_bias[0,2] = tmp[0,2] + (np.log(0.5) / 0.2)
new_bias[0,3] = tmp[0,3] + (np.log(0.5) / 0.2)
new_bias[1] = tmp[1]
new_bias[2] = tmp[2]
caffe_bias = new_bias.flatten()
print(caffe_bias.shape)
caffe_bias.tofile(os.path.join(save_dir, output_name + '.dat'))
Edit_weight_conversion_program
$ sudo -H pip3 install tensorflow-gpu==1.12.0 --upgrade
$ sudo -H pip3 install numpy --upgrade
$ python3 dump_tensorflow_weights.py
$ cp load_caffe_weights.py BK_load_caffe_weights.py
$ nano load_caffe_weights.py
Before
# Before editing Load_caffe_weights.py
caffe_root = '/home/yaochuanqi/work/ssd/caffe/'
for key in net.params.iterkeys():
print key
After
# After editing load_caffe_weights.py
caffe_root = '/path/to/your/ssd-caffe/'
[ex] caffe_root = '/ssd/'
for key in net.params.keys():
print(key)
load_caffe_weights.py
import numpy as np
import sys,os
caffe_root = '/ssd/'
sys.path.insert(0, caffe_root + 'python')
import caffe
deploy_proto = 'deploy.prototxt'
save_model = 'deploy.caffemodel'
weights_dir = 'output'
box_layers = ['conv_13/expand', 'Conv_1', 'layer_19_2_2', 'layer_19_2_3', 'layer_19_2_4', 'layer_19_2_5']
def load_weights(path, shape=None):
weights = None
if shape is None:
weights = np.fromfile(path, dtype=np.float32)
else:
weights = np.fromfile(path, dtype=np.float32).reshape(shape)
os.unlink(path)
return weights
def load_data(net):
#for key in net.params.iterkeys():
for key in net.params.keys():
if type(net.params[key]) is caffe._caffe.BlobVec:
print(key)
if 'mbox' not in key and (key.startswith("conv") or key.startswith("Conv") or key.startswith("layer")):
print('conv')
if key.endswith("/bn"):
prefix = weights_dir + '/' + key.replace('/', '_')
net.params[key][0].data[...] = load_weights(prefix + '_moving_mean.dat')
net.params[key][1].data[...] = load_weights(prefix + '_moving_variance.dat')
net.params[key][2].data[...] = np.ones(net.params[key][2].data.shape, dtype=np.float32)
elif key.endswith("/scale"):
prefix = weights_dir + '/' + key.replace('scale','bn').replace('/', '_')
net.params[key][0].data[...] = load_weights(prefix + '_gamma.dat')
net.params[key][1].data[...] = load_weights(prefix + '_beta.dat')
else:
prefix = weights_dir + '/' + key.replace('/', '_')
ws = np.ones((net.params[key][0].data.shape[0], 1, 1, 1), dtype=np.float32)
if os.path.exists(prefix + '_weights_scale.dat'):
ws = load_weights(prefix + '_weights_scale.dat', ws.shape)
net.params[key][0].data[...] = load_weights(prefix + '_weights.dat', net.params[key][0].data.shape) * ws
if len(net.params[key]) > 1:
net.params[key][1].data[...] = load_weights(prefix + '_biases.dat')
elif 'mbox_loc/depthwise' in key or 'mbox_conf/depthwise' in key:
prefix = key[0:key.find('_mbox')]
index = box_layers.index(prefix)
if 'mbox_loc' in key:
prefix = weights_dir + '/BoxPredictor_' + str(index) + '_BoxEncodingPredictor_depthwise'
else:
prefix = weights_dir + '/BoxPredictor_' + str(index) + '_ClassPredictor_depthwise'
if key.endswith("/bn"):
net.params[key][0].data[...] = load_weights(prefix + '_bn_moving_mean.dat')
net.params[key][1].data[...] = load_weights(prefix + '_bn_moving_variance.dat')
net.params[key][2].data[...] = np.ones(net.params[key][2].data.shape, dtype=np.float32)
elif key.endswith("/scale"):
net.params[key][0].data[...] = load_weights(prefix + '_gamma.dat')
net.params[key][1].data[...] = load_weights(prefix + '_beta.dat')
else:
print(key)
net.params[key][0].data[...] = load_weights(prefix + '_weights.dat', net.params[key][0].data.shape)
if len(net.params[key]) > 1:
net.params[key][1].data[...] = load_weights(prefix + '_biases.dat')
elif key.endswith("mbox_loc"):
prefix = key.replace("_mbox_loc", "")
index = box_layers.index(prefix)
prefix = weights_dir + '/BoxPredictor_' + str(index) + '_BoxEncodingPredictor'
net.params[key][0].data[...] = load_weights(prefix + '_weights.dat', net.params[key][0].data.shape)
net.params[key][1].data[...] = load_weights(prefix + '_biases.dat')
elif key.endswith("mbox_conf"):
prefix = key.replace("_mbox_conf", "")
index = box_layers.index(prefix)
prefix = weights_dir + '/BoxPredictor_' + str(index) + '_ClassPredictor'
net.params[key][0].data[...] = load_weights(prefix + '_weights.dat', net.params[key][0].data.shape)
net.params[key][1].data[...] = load_weights(prefix + '_biases.dat')
else:
print("error key " + key)
net_deploy = caffe.Net(deploy_proto, caffe.TEST)
load_data(net_deploy)
net_deploy.save(save_model)
Weight_conversion_(Tensorflow->Caffe)
$ python3 load_caffe_weights.py
Editing_MS-COCO_to_Pascal_VOC_conversion_program
$ cp coco2voc.py BK_coco2voc.py
$ nano coco2voc.py
Before
# Before editing coco2voc.py
caffe_root = '/home/yaochuanqi/work/ssd/caffe/'
for key in net.params.iterkeys():
x = wt.shape[0] / 91
After
# After editing coco2voc.py
caffe_root = '/path/to/your/ssd-caffe/'
[ex] caffe_root = '/opt/movidius/ssd-caffe/'
for key in net.params.keys():
x = int(wt.shape[0] / 91)
MS-COCO_to_Pascal_VOC_conversion
$ python3 coco2voc.py
Finally, Learning data of the voc model is generated.
Destination＿path
ssdlite/deploy_voc.caffemodel
ssdlite/voc/deploy.prototxt