\documentclass[10pt]{article}

\usepackage{fullpage}
\usepackage{setspace}
\usepackage{parskip}
\usepackage{titlesec}
\usepackage[section]{placeins}
\usepackage{xcolor}
\usepackage{breakcites}
\usepackage{lineno}
\usepackage{hyphenat}





\PassOptionsToPackage{hyphens}{url}
\usepackage[colorlinks = true,
            linkcolor = blue,
            urlcolor  = blue,
            citecolor = blue,
            anchorcolor = blue]{hyperref}
\usepackage{etoolbox}
\makeatletter
\patchcmd\@combinedblfloats{\box\@outputbox}{\unvbox\@outputbox}{}{%
  \errmessage{\noexpand\@combinedblfloats could not be patched}%
}%
\makeatother


\usepackage{natbib}




\renewenvironment{abstract}
  {{\bfseries\noindent{\abstractname}\par\nobreak}\footnotesize}
  {\bigskip}

\titlespacing{\section}{0pt}{*3}{*1}
\titlespacing{\subsection}{0pt}{*2}{*0.5}
\titlespacing{\subsubsection}{0pt}{*1.5}{0pt}


\usepackage{authblk}


\usepackage{graphicx}
\usepackage[space]{grffile}
\usepackage{latexsym}
\usepackage{textcomp}
\usepackage{longtable}
\usepackage{tabulary}
\usepackage{booktabs,array,multirow}
\usepackage{amsfonts,amsmath,amssymb}
\providecommand\citet{\cite}
\providecommand\citep{\cite}
\providecommand\citealt{\cite}
% You can conditionalize code for latexml or normal latex using this.
\newif\iflatexml\latexmlfalse
\providecommand{\tightlist}{\setlength{\itemsep}{0pt}\setlength{\parskip}{0pt}}%

\AtBeginDocument{\DeclareGraphicsExtensions{.pdf,.PDF,.eps,.EPS,.png,.PNG,.tif,.TIF,.jpg,.JPG,.jpeg,.JPEG}}

\usepackage[utf8]{inputenc}
\usepackage[ngerman,english]{babel}



\usepackage{float}








\begin{document}

\title{A Depthwise Separable Convolution Hardware Accelerator for ShuffleNetV2}



\author[1]{Linshuang Li}%
\author[1]{Dihu Chen}%
\author[1]{Tao Su}%
\affil[1]{Sun Yat-Sen University}%


\vspace{-1em}



  
  \date{May 29, 2024}


\begingroup
\let\center\flushleft
\let\endcenter\endflushleft
\maketitle
\endgroup





\selectlanguage{english}
\begin{abstract}
Convolutional neural networks (CNNs) have been widely applied in the
field of computer vision with the development of artificial
intelligence. MobileNet and ShuffleNet, among other depthwise separable
convolutional neural networks, have gained significant advantages in
deploying on resource-constrained embedded devices due to their
characteristics such as fewer parameters and higher computational
efficiency compared to previous networks. In this paper, we focus on the
hardware implementation of ShuffleNetV2. We optimized the network
structure. Feature channel numbers, pooling modes, and channel shuffle
modes are modified, resulting in a 1.09\% increase in accuracy while
reducing the parameter count by 0.18M. Additionally, we implement a
highly parallel hardware accelerator on the Xillinx xczu9eg FPGA, which
supports both standard convolution and depthwise convolution. The power
consumption of this accelerator is only 7.3W while achieving an energy
efficiency of 13.45 GOPS/W. The running frame rate achieves 675.7 fps.%
\end{abstract}%



\sloppy


A Depthwise Separable Convolution Hardware Accelerator for ShuffleNetV2

Linshuang Li, Dihu Chen, Tao Su

School of Electronics and Information Technology, Sun Yat-Sen
Univer-sity, Guangzhou, Guangdong, People's Republic of China

Email: stscdh@mail.sysu.edu.cn

Convolutional neural networks (CNNs) have been widely applied in the
field of computer vision with the development of artificial
intelligence. MobileNet and ShuffleNet, among other depthwise separable
convolutional neural networks, have gained significant advantages in
deploying on resource-constrained embedded devices due to their
characteristics such as fewer parameters and higher computational
efficiency compared to previous networks. In this paper, we focus on the
hardware implementation of ShuffleNetV2. We optimized the network
structure. Feature channel numbers, pooling modes, and channel shuffle
modes are modified, resulting in a 1.09\% increase in accuracy while
reducing the parameter count by 0.18M. Additionally, we implement a
highly parallel hardware accelerator on the Xillinx xczu9eg FPGA, which
supports both standard convolution and depthwise convolution. The power
consumption of this accelerator is only 7.3W while achieving an energy
efficiency of 13.45 GOPS/W. The running frame rate achieves 675.7 fps.

\emph{Introduction:} Nowadays, with the development of artificial
intelligence, Convolutional Neural Networks (CNNs) as one of its
representative algorithms have received increasing attention. Due to its
fast speed and small model size, depthwise separable convolutional
neural networks have gained significant advantages deploying on embedded
terminals.

However, depthwise separable convolutions decouple traditional
convolutions into depthwise convolution (DwC) and pointwise convolution
(PwC) {[}1{]}. Therefore, conventional CNN accelerators are no longer
suitable for performing computations in depthwise separable
convolutional neural networks. Based on current research, the pipelined
computing architecture introduced an additional feature bank to prefetch
data from off-chip memory {[}2{]}, the introduction of additional
storage units leads to increased hardware resources and data read/write
time consumptions. The reconfigurable architecture considered the
combination of different computation modes in the network model that
supports both PwC and DwC calculations {[}3{]}. This structure cannot
guarantee that all modules are in an operational state during
computation, leading to inefficient utilization of computing resources
and resulting in wastage. It does not support complex operations such as
grouped convolution and network shuffle operations as well. Although
traditional channel shuffle mode separately handled the shortcut branch
and concating the results after PwC {[}4{]}, it still resulted in a
significant amount of memory read and write operations, leading to high
latency.

Based on the above observations, the main contributions of this paper
are as follows. We redesigned the structure of the depthwise separable
convolution ShuffleNetV2. The network channels, pooling mode and channel
shuffle mode are optimized, resulting in a 12.9\% decrease in the
parameter count and 1.09\% accuracy increase. We also proposed a
hardware accelerator that supports both DwC and PwC, allowing them to
fully utilize and share the hardware resources of the computing array.
Achieving Energy efficiency ratio with minimal FPGA hardware resource
utilization, resulting in a frame rate of 675.7fps for image processing.

\emph{Design details:} The purpose of this paper is to implement the
computation of depthwise separable convolution on resource-constrained
embedded terminals, enabling fast and efficient image classification.
Firstly, the pooling layer after the first convolution layer is removed
to ensure that more effective feature information enters the
construction block for feature extraction. Secondly, to ensure the
utilization of processing elements (PEs) in the convolution computing
array, the number of channels after each down sampling unit is changed
from 29\selectlanguage{ngerman}× to 27×. This operation allows for the full utilization and
sharing of resources on the same computational array by both DwC and
PwC, without wasting hardware resources. The simplified ShuffleNetV2
structure is shown in Table 1. In addition, to further compress the
network, we quantized the weights by converting them from 32-bit
floating-point numbers to 8-bit fixed-point numbers, with 3 bits for the
integer part and 5 bits for the fractional part.

\emph{Table 1. Simplified ShuffleNetV2 network structure}\selectlanguage{english}
\begin{longtable}[]{@{}lllll@{}}
\toprule
Layer & Output size & Repeat & Stride & Output Channel\tabularnewline
\midrule
\endhead
Image & 224\selectlanguage{ngerman}×224 & & & 3\tabularnewline
Conv1 & 112\selectlanguage{ngerman}×112 & 1 & 2 & 27\tabularnewline
Stage2 & 112\selectlanguage{ngerman}×112 & 1 & 2 & 108\tabularnewline
& 112\selectlanguage{ngerman}×112 & 3 & 1 & 108\tabularnewline
Stage3 & 56\selectlanguage{ngerman}×56 & 1 & 2 & 216\tabularnewline
& 56\selectlanguage{ngerman}×56 & 7 & 1 & 216\tabularnewline
Stage4 & 28\selectlanguage{ngerman}×28 & 1 & 2 & 432\tabularnewline
& 28\selectlanguage{ngerman}×28 & 3 & 1 & 432\tabularnewline
Conv5 & 7\selectlanguage{ngerman}×7 & 1 & 1 & 1024\tabularnewline
Maxpool & 1\selectlanguage{ngerman}×1 & & &\tabularnewline
FC & & & & 100\tabularnewline
\bottomrule
\end{longtable}

The core of the accelerator is a highly parallel array that supports
both standard convolution and DwC. For standard convolution, we employed
channel-level parallelism in the convolution. In one clock cycle, each
row of the convolution array reads the convolution window at the same
position from l input channels, while each column reads the convolution
kernel weights from m output feature maps. The array uses a total
of\emph{l} \selectlanguage{ngerman}× \emph{m} PEs to perform the element-wise multiplication.
The input feature maps and weights are accessed in address order, a
parallel approach suitable for natural data storage patterns. Moreover,
only a single read operation is required for the same data within the
same clock cycle, resulting in reduced bandwidth requirements.
After\emph{D\textsubscript{k}} × \emph{D\textsubscript{k}} (kernel size)
cycles, a set of convolution results is obtained. Then, the sliding
window moves to the next position to traverse the entire feature map,
and the computation continues for the next input feature map channel.
This process repeats until all m output feature maps have completed the
convolution, and then the calculation starts for the next group
of\emph{l} × \emph{m} channel dimensions of the feature maps.\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image1/image1}

\end{center}
\end{figure}

Fig 1  \emph{Convolution computing array for DwC mode.}

For a DwC with both input and output channel parallelism of \emph{l} ,
only \emph{l} groups of PEs can be used simultaneously when implemented
on traditional CNN accelerators, while the rest of the PEs in the
computation array will be idle. Therefore, to ensure that the
computation engine shares the same computation array for both DwC and
standard convolution modes, and to achieve high resource utilization
efficiency during computation, an additional control module is used to
manage the depth convolution mode. As shown in Fig 1, the standard
convolution computation array is divided into \emph{q} image processing
units (PPE). Each PPE reads \emph{l} different channels of input data
and the corresponding convolution kernel weights from the buffers. Each
PPE also contains \emph{l} groups of window processing units (WPE) for
multiplication. Each WPE performs parallel computation on a single
sliding window. Since the depth convolution layer has a uniform 3\selectlanguage{ngerman}×3
kernel size, the parallelism of PEs in each WPE is \emph{p} =9.

In each clock cycle, the computation array first reads \emph{l}
×\emph{p}convolution weights from the weight buffer for each column.
When the input feature map channels remain the same, the same batch of
convolution kernel weights can be used for each convolution operation,
so the weights can be loaded only once. Then, \emph{p} pixels of a
single convolution window from \emph{l} input channels of
\emph{q}different images are read from the input buffer and loaded to
different PEs for convolution. Therefore, the number of parallelizable
multiplications is \emph{q} ×\emph{l} ×\emph{p} . The convolution window
is traversed by prioritizing the entire feature map before moving on to
the next set of channel feature map computations, until all
\emph{q}different input images have completed the convolution. Besides,
the size of the convolution array is determined to be 27×27. The
parallelism for both standard convolution and DwC is equal, ensuring
that both convolution modes can fully utilize the hardware resources of
the computation module without wasting.

After the computation is completed in the convolution array, the feature
map data enters the post-processing module for further processing,
including the addition tree module, activation module, pooling module,
and channel shuffle module. The hardware accelerator architecture is
shown in Fig 2. After all the computation is finished, the output
feature map data would be sent to the output buffer and then returned to
the BRAM for the next round of computation.\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image3/image3}

\end{center}
\end{figure}

Fig 2  \emph{Architecture of the accelerator.}

For the traditional channel shuffle operation, channels are selected
alternately from two groups of feature maps and recombined into a new
output feature map, which is then transferred to BRAM as the input for
the next convolution. In this paper, the channel shuffle method is
modified by partitioning the output feature maps internally into groups
of 4 channels. Channels are then selected alternately from the two
groups and recombined into a new output feature map, as illustrated in
Fig 3. This approach maintained the advantage of increasing
inter-channel information exchange while reducing the number of memory
read/write operations by 75\%, significantly reducing memory access
time.\selectlanguage{english}
\begin{figure}[H]
\begin{center}
\includegraphics[width=0.70\columnwidth]{figures/image5/image5}

\end{center}
\end{figure}

Fig 3  \emph{Comparison between original and optimized channel shuffle
mode.}

\emph{Experiments and results:} The network we proposed is trained and
tested using the PyTorch deep learning framework. The test dataset we
used is CIFAR100. The simplified ShuffleNetV2 network proposed in this
paper is compared with traditional networks, and the comparison results
are shown in the table I. In the case of small input feature map size,
the network achieves a recognition accuracy improvement of 1.33\% and
1.09\% compared to the original network. At the same time, the number of
parameters is reduced from 1.4M to 1.22M, effectively reducing the
storage resources and data access on the hardware.

\emph{Table 2. Comparison with original network}\selectlanguage{english}
\begin{longtable}[]{@{}llll@{}}
\toprule
\begin{minipage}[b]{0.24\columnwidth}\raggedright\strut
\strut
\end{minipage} & \begin{minipage}[b]{0.24\columnwidth}\raggedright\strut
Original network\strut
\end{minipage} & \begin{minipage}[b]{0.24\columnwidth}\raggedright\strut
Simplified network\strut
\end{minipage} & \begin{minipage}[b]{0.24\columnwidth}\raggedright\strut
Simplified network\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
Data format\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
float32\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
float32\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
int8\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
Accuracy(\%)\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
69.40\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
\textbf{70.73}\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
\textbf{70.49}\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
Parameters(M)\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
1.40\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
\textbf{1.22}\strut
\end{minipage} & \begin{minipage}[t]{0.24\columnwidth}\raggedright\strut
\textbf{1.22}\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

The hardware acceleration system was deployed on Xilinx FPGA chip
xczu9eg, with a clock frequency of 180 MHz. The GPU used was NVIDIA
GeForce GTX 1080 Ti. The execution times of ShuffleNetV2 on FPGA and GPU
are shown in the table II, representing the average time for classifying
1250 images. The speed on FPGA is 1.12 times faster than that on GPU,
with a decrease in recognition accuracy of only 0.66\%. Since it takes
146M FLOPS to perform the whole network, we can calculate the GPU's
energy efficiency as 0.44. Our work has achieved an energy efficiency
30.57 times higher than that of the GPU.

\emph{Table 3. Comparison with GPU}\selectlanguage{english}
\begin{longtable}[]{@{}lll@{}}
\toprule
\begin{minipage}[b]{0.30\columnwidth}\raggedright\strut
Device\strut
\end{minipage} & \begin{minipage}[b]{0.30\columnwidth}\raggedright\strut
FPGA xczu9eg\strut
\end{minipage} & \begin{minipage}[b]{0.30\columnwidth}\raggedright\strut
GPU\strut
\end{minipage}\tabularnewline
\midrule
\endhead
\begin{minipage}[t]{0.30\columnwidth}\raggedright\strut
Power(W)\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright\strut
\textbf{7.3}\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright\strut
220\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.32\columnwidth}\raggedright\strut
Power efficiency (GOPS/W)\strut
\end{minipage} & \begin{minipage}[t]{0.32\columnwidth}\raggedright\strut
\textbf{13.45}\strut
\end{minipage} & \begin{minipage}[t]{0.32\columnwidth}\raggedright\strut
0.44\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.30\columnwidth}\raggedright\strut
Execution time(ms)\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright\strut
\textbf{1.48}\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright\strut
1.66\strut
\end{minipage}\tabularnewline
\begin{minipage}[t]{0.30\columnwidth}\raggedright\strut
Accuracy(\%)\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright\strut
69.83\strut
\end{minipage} & \begin{minipage}[t]{0.30\columnwidth}\raggedright\strut
70.49\strut
\end{minipage}\tabularnewline
\bottomrule
\end{longtable}

We also compared our work with others work {[}2{]}{[}4{]}{[}5{]}{[}6{]}.
We achieved a computational efficiency that is 1.75 times higher than
{[}4{]}. Compared to {[}5{]}, we achieved a throughput and frame rate
that are 2.08 times and 11.5 times higher respectively, even at a lower
clock frequency. These comparisons demonstrate significant advantages of
our work in terms of computational speed and hardware resource
consumption.

\emph{Table 4. Comparison with other work}\selectlanguage{english}
\begin{longtable}[]{@{}llllll@{}}
\toprule
& {[}2{]} & {[}4{]} & {[}5{]} & {[}6{]} & ours\tabularnewline
\midrule
\endhead
Device & zu2eg & 7z045 & zu3eg & 7z045 & zu9eg\tabularnewline
BRAM & 145 & 213 & 170 & 64 & 190\tabularnewline
LUT & 31198 & 105000 & 24130 & 69666 & \textbf{18573}\tabularnewline
DSP & 212 & - & - & 385 & 729\tabularnewline
Throughput(GOPS) & - & 56.1 & 47.1 & 33.6 &
\textbf{98.2}\tabularnewline
Power (W) & - & -- & 5.5 & - & 7.3\tabularnewline
Energy efficiency (GOPS/W) & - & - & 8.56 & - &
\textbf{13.45}\tabularnewline
Frame rate(fps) & 205.3 & 291.5 & 96.5 & 240 &
\textbf{675.7}\tabularnewline
\bottomrule
\end{longtable}

\emph{Conclusion:} This paper presents a deep separable convolutional
neural network accelerator designed specifically for ShuffleNetV2. Based
on the features of ShuffleNetV2, optimizations are made to the network
structure, achieving a 1.09\% increase in accuracy while reducing the
parameters by 0.18M. The paper also proposes a reconfigurable hardware
accelerator that supports both PwC and DwC. The power consumption of
this accelerator is only 7.3W while achieving a power efficiency of
13.45 GOPS/W. The running frame rate achieves 675.7 fps.

\emph{Acknowledgments:} The authors thank to the support by the Science
and Technology Program of Guangdong Province under Grant
2022B0701180001.

References

\begin{enumerate}
\tightlist
\item
  N. Ma \emph{et al} , ``ShuffleNet V2: Practical Guidelines for
  Efficient CNN Architecture Design,'' in Proceedings of the European
  conference on computer vision (ECCV) ~pp. 116-131, doi:
  ~\emph{arXiv:1807.11164v1}.
\item
  D. Wu \emph{et al} ., ''A High-Performance CNN Processor Based on FPGA
  for MobileNets,'' 2019 29th International Conference on Field
  Programmable Logic and Applications (FPL), Barcelona, Spain, 2019, pp.
  136-143, doi:\emph{10.1109/FPL.2019.00030}.
\item
  Y. -G. Chen, H. -Y. Chiang, C. -W. Hsu, T. -H. Hsieh and J. -Y. Jou,
  ''A Reconfigurable Accelerator Design for Quantized Depthwise
  Separable Convolutions,'' 2021 18th International SoC Design
  Conference (ISOCC), Jeju Island, Korea, Republic of, 2021, pp.
  290-291, doi:\emph{10.1109/ISOCC53507.2021.9613976}.
\item
  Z. Fan, W. Hu, H. Guo, F. Liu and D. Xu, ''Hardware and Algorithm
  Co-Optimization for pointwise convolution and channel shuffle in
  ShuffleNet V2,'' 2021 IEEE International Conference on Systems, Man,
  and Cybernetics (SMC), Melbourne, Australia, 2021, pp. 3212-3217,
  doi:\emph{10.1109/SMC52423.2021.9659057}.
\item
  Y. Yang \emph{et al} , Synetgy: ``Algorithm-hardware co-design for
  convnet accelerators on embedded fpgas,'' Proceedings of the 2019
  ACM/SIGDA international symposium on field-programmable gate arrays.
  2019, pp. 23-32, doi:\emph{10.1145/3289602.3293902}.
\item
  Y. Lin \emph{et al} , ''A High-speed Low-cost CNN Inference
  Accelerator for Depthwise Separable Convolution,'' 2020 IEEE
  International Conference on Integrated Circuits, Technologies and
  Applications (ICTA), Nanjing, China, 2020, pp. 63-64,
  doi:\emph{10.1109/ICTA50426.2020.9332057}.
\end{enumerate}

\selectlanguage{english}
\FloatBarrier
\end{document}

