In a mobile environment like Android, if you choose OpenCV for production, one of your important goals is to bring down the size of the library and also make it performance ready.
OpenCV is an awesome library with tons of Computer Vision algorithms but usually, you use a small subset of these algorithms in your application. Hence, it makes perfect sense to include what is required and leave out the rest.
Static vs. Dynamic Library
A library can be compiled statically along with your application code.
Alternatively, it can be dynamically linked at runtime.
In this tutorial we will create a dynamic library (i.e. shared object (.so)).
Environment and library versions
Android is a versatile OS and can run on multiple hardwares, from mobile phones to IOT devices, Raspberry Pi, to various single board computers and therefore it is important to cross compile the code for that particular Instruction Set Architecture (ISA).
As we are building OpenCV for Android, we would need its built tools called Native Development Kit (NDK).
This demo has been tested on the following
- Ubuntu 16.4
- cmake 3.7.2
- NDK r14b
- OpenCV 3.4.1
- Target armeabi-v7a (ARM based)
- Android API 23
Step 1: Download and Setup NDK
Download NDK and unzip it to your work area. There are various versions of the NDK, hence it is important to read the release notes and pick the one you need for your project.
For this tutorial, we will use Android NDK Release 14b on Ubuntu inside a Docker container. But this can be on any Linux box.
root@dc:/opt/android-ndk-r14b# ls
ndk-build ndk-gdb ndk-which prebuilt shader-tools source.properties sysroot build ndk-depends ndk-stack platforms python-packages simpleperf sources toolchains
We will build the standalone toolchain for OpenCV compilation,
root@dc:/opt/android-ndk-r14b# ./build/tools/make_standalone_toolchain.py \
--arch arm \
--api 23 \
--install-dir /tmp/my-android-toolchain
This command will create toolchain based on –arch and –api that i need for my project and install them in /tmp/my-android-toolchain directory.
Step 2: Setup ANDROID_NDK Path
Often ANDROID_NDK path is set to the prebuilt toolchain, that comes along with the NDK. But we will use the one that we built in the previous step, we need to let cmake know this important fact and hence set the environment variable as:
$ export ANDROID_STANDALONE_TOOLCHAIN=/tmp/my-android-toolchain/
That’s pretty much our build environment needs to know, there are no other variables required. If you already have ANDROID_NDK set, please unset it.
Step 3: Install Ninja and Ant
Building for Android also needs a few important packages called Ninja and Ant. Let’s install them:
$ sudo apt-get install ninja-build ant
Step 4: Download and Install OpenCV
Download OpenCV and install it as mentioned in the article. Either you can compile the way blog suggested or just download all the packages, OpenCV repository and leave it there.
Note : Because we are compiling for Android so we can skip the compilation part mentioned in the blog.
OpenCV
Once, the above instructions have been followed, follow the instructions below:
$ cd opencv/
$ mkdir build $ cd build $ cmake \
-DCMAKE_TOOLCHAIN_FILE=../platforms/android/android.toolchain.cmake \
-DANDROID_STL=gnustl_shared \
-DANDROID_NATIVE_API_LEVEL=23 ..
OpenCV_contrib
If you need opencv_contrib module, download them it and make the following changes cmake as below:
$ cd opencv/
$ mkdir build
$ cd build
$ cmake \
-DCMAKE_TOOLCHAIN_FILE=../platforms/android/android.toolchain.cmake \
-OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \
-DANDROID_STL=gnustl_shared \
-DANDROID_NATIVE_API_LEVEL=23 ..
The output will show the configuration for this build and this is the step where you can change the configuration of your likes.
Neon and VFPV3 optimizations
For ARM based devices, optimization i usually enable using:
- NEON
- VFPV3
Check out preamble of platforms/android/android.toolchain.cmake for various configuration options.
Make
Finally, we are ready to build with the following command
$ make -j $nproc
Step 5: Check build
Hopefully the build goes fine and you have ready ELF output in the form of .so (shared object), you can also generate output as .a, for this you need to enable appropriate flag in the cmake. The output would look like following:
root@dc:/opencv/build/lib/armeabi-v7a# ls
libopencv_calib3d.a libopencv_flann.a libopencv_java3.so libopencv_shape.a libopencv_video.a libopencv_core.a libopencv_highgui.a libopencv_ml.a libopencv_stitching.a libopencv_videoio.a libopencv_dnn.a libopencv_imgcodecs.a libopencv_objdetect.a libopencv_superres.a libopencv_videostab.a libopencv_features2d.a libopencv_imgproc.a libopencv_photo.a libopencv_ts.a
libopencv_java3.so is what you need for your android project. However, if you look at the size of it, its around 9.5MB
root@dc:/opencv/build/lib/armeabi-v7a# du -h
libopencv_java3.so 9.4M libopencv_java3.so
That is because it has all the modules included. To verify this, run following command:
$ strings -a libopencv_java3.so | grep .cpp
Step 6: Optimize Library Size
Lets trim the library for our needs. Let us say we need just two modules core and imageproc.
root@dc:/opencv/build/lib/armeabi-v7a# /tmp/my-android-toolchain/bin/arm-linux-androideabi-g++ \
-shared -o libopencv_tiny.so \
--sysroot=/tmp/my-android-toolchain/sysroot \
-Wl,—-whole-archive libopencv_core.a \
libopencv_imgproc.a -Wl,—-no-whole-archive
Now check the file type and size of linopencv_tiny.so
root@dc:/opencv/build/lib/armeabi-v7a# file libopencv_tiny.so
libopencv_tiny.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /system/bin/linker, not stripped
root@dc:/opencv/build/lib/armeabi-v7a# du -h libopencv_tiny.so
8.3M libopencv_tiny.so
That’s better than before, but this is non-striped, let us strip it to reduce the size even further.
root@dc:/opencv/build/lib/armeabi-v7a# /tmp/my-android-toolchain/bin/arm-linux-androideabi-strip --strip-unneeded libopencv_tiny.so
root@dc:/opencv/build/lib/armeabi-v7a# file libopencv_tiny.so
libopencv_tiny.so: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /system/bin/linker, stripped
root@dc:/opencv/build/lib/armeabi-v7a# du -h libopencv_tiny.so
3.4M libopencv_tiny.so
Now, 3.4MB is about a 2.5x smaller than of where we started.
Step 7 (optional): Bonus round
You can further drill down and pick each .o file rather than .a file, that can be achieved using ar tool as show below.
$ /tmp/my-android-toolchain/bin/arm-linux-androideabi-ar x libopencv_core.a
ar tool extracts all the .o from .a file and later you can make .so from them the same way we created before, the command is shown below
root@dc:/opencv/build/lib/armeabi-v7a# /tmp/my-android-toolchain/bin/arm-linux-androideabi-g++ \
-shared -o libopencv_tiny.so \
--sysroot=/tmp/my-android-toolchain/sysroot \
-Wl, — whole-archive *.o -Wl, — no-whole-archive
Step 8: Create .so for your application
Linking is where the efforts ends, and let us cross that milestone too. As this is cross compiled for a specific platform, it is important to include few extra libraries in the .so. These extra libraries are completely 3rd party and platform specific. If you do not include them you will get linking error, hence let us do it:
root@dc:/opencv/build/lib/armeabi-v7a# /tmp/my-android-toolchain/bin/arm-linux-androideabi-g++ \
-L/opencv/build/3rdparty/lib/armeabi-v7a -lz -llog \
-ljnigraphics -ltegra_hal -lcpufeatures \
-shared -Wl,-soname,libopencv_tiny.so -o libopencv_tiny.so \
--sysroot=/tmp/my-android-toolchain/sysroot \
-Wl,--whole-archive libopencv_core.a libopencv_imgproc.a \
-Wl,--no-whole-archive
Dependencies should look like the following, few of these extra libraries are .a so they won’t show up in the dependencies.
root@dc:/opencv/build/lib/armeabi-v7a# readelf -d libopencv_tiny.so Dynamic section at offset 0x3615d8 contains 33 entries: Tag Type Name/Value 0x00000003 (PLTGOT) 0x362b18 0x00000002 (PLTRELSZ) 2488 (bytes) 0x00000017 (JMPREL) 0x3f674 0x00000014 (PLTREL) REL 0x00000011 (REL) 0x28cb4 0x00000012 (RELSZ) 92608 (bytes) 0x00000013 (RELENT) 8 (bytes) 0x6ffffffa (RELCOUNT) 11570 0x00000006 (SYMTAB) 0x148 0x0000000b (SYMENT) 16 (bytes) 0x00000005 (STRTAB) 0xaf78 0x0000000a (STRSZ) 97135 (bytes) 0x00000004 (HASH) 0x22ae8 0x00000001 (NEEDED) Shared library: [libz.so] 0x00000001 (NEEDED) Shared library: [liblog.so] 0x00000001 (NEEDED) Shared library: [libjnigraphics.so] 0x00000001 (NEEDED) Shared library: [libm.so] 0x00000001 (NEEDED) Shared library: [libc.so] 0x00000001 (NEEDED) Shared library: [libdl.so] 0x0000000e (SONAME) Library soname: [libopencv_tiny.so] 0x0000001a (FINI_ARRAY) 0x35e7b4 0x0000001c (FINI_ARRAYSZ) 8 (bytes) 0x00000019 (INIT_ARRAY) 0x362554 0x0000001b (INIT_ARRAYSZ) 132 (bytes) 0x00000010 (SYMBOLIC) 0x0 0x0000001e (FLAGS) SYMBOLIC BIND_NOW 0x6ffffffb (FLAGS_1) Flags: NOW 0x6ffffff0 (VERSYM) 0x27690 0x6ffffffc (VERDEF) 0x28c58 0x6ffffffd (VERDEFNUM) 1 0x6ffffffe (VERNEED) 0x28c74 0x6fffffff (VERNEEDNUM) 2 0x00000000 (NULL) 0x0
Make sure you strip the newly created .so before checking the size.
Now, your shared object ( libopencv_tiny.so ) is ready for android deployment. Often it is given the name libopencv_java3.so where 3 is the version. Do not worry it is just a file name and it will still work. However, if you want to make it android complaint you can rename it to libopencv_java3.so
Step 9: Linking with sample code
Let us say we have following code in the file demo.cpp inside directory ./demo, that you want to link with created the .so
#include "opencv2/opencv.hpp"
#include <iostream>
int main() {
std::cout << "OpenCV Version: " << CV_VERSION << std::endl;
return 0;
}
We MUST use the same toolchain to compile and link demo.cpp. Can you guess why?
Ok! now that you are thinking, let me quickly start the compilation:
root@dc: /tmp/my-android-toolchain/bin/arm-linux-androideabi-g++ -L/demo -I/demo/include/ -Wall -o demo demo.cpp -lopencv_tiny
I have moved the generated libopencv_tiny.so to folder ./demo and all the opencv includes (i.e. .hpp), this is just to demonstrated that we are in isolation and not using /usr/local/include.
Now, coming back to my question, why same toolchain?
root@5c7fe9a72d74:/demo# readelf -h libopencv_tiny.so
ELF Header: Magic: 7f 45 4c 46 01 01 01 00 00 00 00 00 00 00 00 00 Class: ELF32 Data: 2's complement, little endian Version: 1 (current) OS/ABI: UNIX - System V ABI Version: 0 Type: DYN (Shared object file) Machine: ARM Version: 0x1 Entry point address: 0x0 Start of program headers: 52 (bytes into file) Start of section headers: 3561860 (bytes into file) Flags: 0x5000200, Version5 EABI, soft-float ABI Size of this header: 52 (bytes) Size of program headers: 32 (bytes) Number of program headers: 8 Size of section headers: 40 (bytes) Number of section headers: 27 Section header string table index: 26
Did you get it? Ok, let me just say it: The .so is cross compiled for a specific arch type and that is ARM where as out host machine is x86 based linux. This is the reason, you need to run the executable file only on the same target machine type.
Step 10: Include path to .so in LD_LIBRARY_PATH
If you build it on x86, you must also tell the system where the .so resides. You can keep it in any directory of your project or workarea, as this is runtime linking. Make sure ld is able to find it. The best way to do this is by updating an environment variable:
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/path/to/your/.so
Now, say you have become very generous and want to share it with everyone on the system. For that you need to have root privileges. You will need this for two reasons:
- To put the library in a standard system location, probably /usr/lib or /usr/local/lib, which normal users don’t have write access to.
- You will need to modify the ld.so config file and cache. As root, do the following:
cp /demo/libopencv_tiny.so /usr/local/lib
chmod 0755 /usr/local/lib/libopencv_tiny.so
We need to inform ld that it’s available for use, hence let’s update the cache:
ldconfig
Now, we do not need LD_LIBRARY_PATH, so let’s clear it too.
CAUTION: You must clear your own path from LD_LIBRARY_PATH and leave the rest as is.
unset LD_LIBRARY_PATH
Try the same steps on various different ARCH, and please share your experience.
Subscribe
If you liked this article, please subscribe to our newsletter. You will also receive a free Computer Vision Resource guide. In our newsletter, we share Computer Vision, Machine Learning and AI tutorials written in Python and C++ using OpenCV, Dlib, Keras, Tensorflow, CoreML, and Caffe.
</blockquote”>
