Saturday, December 5, 2015

Using gradle to build NDK projects

When I started writing this blog two years ago, I originally planned to write more frequently about various topics. However, the blog ended up with only two posts with tips and tricks for NDK debugging of library projects. Today, NDK debugging of library projects is officially supported with NDK r10e (you can use ndk-gdb --package=app.package.name) and you do not need to perform hacks as described in my earlier posts.

In this article I'm about to write how we at our company solved problem of building our native codebase with gradle and how we use it in production. Note that what I will describe now is not an official way of using NDK from gradle with gradle android plugin. I am writing this in hope that new gradle experimental plugin that will support NDK will have all the features we need and have implemented with hacks I am about to describe. So let's hope these will not be required anymore in the future.

Let's start with what we want to achieve. We have a native codebase with JNI wrappers. We want to create a gradle script that will be able to build AAR that can be distributed to third parties and which will contain everything a client needs to get the functionality we offer with a single reference to AAR library that we delivered. At our company we build multiple products from single codebase. Each product needs to have different set of Java classes that are delivered and different native library that is built with different configuration. Therefore, we use product flavors supported from android gradle plugin. For each product flavor, our gradle script has to support 3 build types: debug, release and distribute.

In debug build type, java code is not obfuscated and native code must be built with debug logging enabled and without compiler optimization. In release build type, java code is both obfuscated and optimized with ProGuard and native code is build with optimization, but debug logging is kept enabled. Distribute build type is same as release, except debug logging is disabled. In our development process, we want to use debug build type to debug java and native code with debuggers (step-by-step mode), release build type to test that everything works OK and to debug obvious bugs and distribute build type to build final AAR that will be distributed to clients.

So, it is obvious that we need to ensure our gradle script can build both java and native code in all combinations of flavors and build types. Even more, when we moved from Eclipse to Android studio we had a mature Android.mk file that was used to build native code in various flavors. Since Eclipse and ant did not support product flavors, we created an Android.mk file that could build different versions of native library based on configuration variables that were stored in several configuration makefiles that were given to ndk-build via command line argument. So when writing our gradle script, we will make sure that we reuse the mature Android.mk file that was used for building native code.

Before starting the writing of gradle script, we will assume that your Android.mk is written in such way that it can build different versions of native library simply by setting BUILD_SETTINGS variable that contains a name of makefile that defines variables that control the build. In our Android.mk this is done with following code snippet:

BUILD_SETTINGS ?= build_settings.mk

# include variables that control the build
include $(LOCAL_PATH)/build_settings/$(BUILD_SETTINGS)

I will not give the details of how Android.mk controls the build of native code. I will only assume that it can build different versions of native code with configuration given with makefile specified by BUILD_SETTINGS variable.

Now we can start writing our gradle script. As a first thing, we will need to setup a local.properties file which will contain settings that are specific for each developer. Let's put these lines in local.properties file of our library project that will build the AAR:

sdk.dir=/Users/dodo/android-sdks
ndk.dir=/Users/dodo/android-sdks/android-ndk
ndk.ccache=/usr/local/bin/ccache

The sdk.dir variable should already be set by Android studio and will contain a path where developer has installed Android SDK. The ndk.dir variable is not set by Android studio, unless you are trying out gradle-experimental plugin. You should set this variable to location where you have installed your NDK. In this document, I assume usage of NDK r10e. The ndk.ccache variable should be set either to null or to path containing the ccache binary. If developer has ccache installed, it can use it to speed up repeated native compilation. If developer does not have it installed, it will not be used.

OK, let's finally start with build.gradle file of our library project. First, we need to load the variables from local.properties file because those variables are not exposed to us in any way by Android gradle plugin.

String userDir = System.getProperty("user.dir")
File fDir = file(userDir);
if(fDir.name != "LibRecognizer") {
    userDir = userDir + "/LibRecognizer"
}

String localProperties = userDir + "/local.properties"
FileInputStream localPropsFs
try {
    localPropsFs = new FileInputStream(localProperties)
} catch (Exception e) {
    throw new GradleException("Please create a " + localProperties + " file!")
}
Properties localProps = new Properties()
localProps.load(localPropsFs)
String androidSdkDir = localProps.getProperty('sdk.dir')

String ndkDir = localProps.getProperty('ndk.dir')
if(ndkDir == null) {
    throw new GradleException("Please define 'ndk.dir' variable in your local.properties file!")
}

String ndkCache = localProps.getProperty('ndk.ccache')

if(ndkCache == null) {
    ndkCache = ""
}
String originalDirectory = userDir + '/src/main'

So, we first obtain the system property called user.dir which will contain the path where our root project is located. Next, as we want this script to work both for our library as nested project (i.e. as subproject of root project that contains our library and a test application) and as root project (i.e. when we just want to build AAR), we must check if userDir is already same as name of our library (name of our library is LibRecognizer) and if it isn't, then we must correct the userDir to point directly to library project folder. This will ensure that userDir will always contain a path to folder containing build.gradle file of our LibRecognizer library project.

Next, we load the properties from local.properties file and store them to variables for later use. We now continue our build.gradle file with more or less known code:

apply plugin: 'com.android.library'

android {
    compileSdkVersion 23
    buildToolsVersion '23.0.2'

    defaultConfig {
        minSdkVersion 10
        targetSdkVersion 23
    }
    // to be continued

If you have ever created a library module in android studio, previous lines should not be unfamiliar to you. Now, let's define build types, but for each build type we want to store some settings that we will need later. In order to do that, we first must define default settings that will apply for each build type and then simply override the defaults:

// continued from above
buildTypes.whenObjectAdded { buildType ->
    buildType.ext {
        ndkDebug = true
        appOptim = "release"
        stfu = false
        onlyActiveArch = true
    }
}

buildTypes {
    debug {
        minifyEnabled false
        ext {
            ndkDebug = true
            appOptim = "debug"
            stfu = false
            onlyActiveArch = true
        }
    }
    release {
        minifyEnabled true
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt')
        ext {
            ndkDebug = true
            appOptim = "release"
            stfu = false
            onlyActiveArch = true
        }
    }
    distribute {
        minifyEnabled true
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt')
        ext {
            ndkDebug = false
            appOptim = "release"
            stfu = true
            onlyActiveArch = false
        }
    }
}
// to be continued

OK, let's explain this code snippet: whenever we create a new build type, we want to initialize its ext property (a 'property' in each gradle object that lets you write anything into it) with default values of variables that we will need later. Let's see what those variables are:

  • ndkDebug - boolean that defines whether or not native code will be built with NDK_DEBUG=1 that will besides libNative.so create files gdbserver and gdb.setup that enable native debugging of application
  • appOptim - a string that defines the optimization level inside NDK. The APP_OPTIM variable will be set to value of this variable
  • stfu - a boolean that will be transformed into value of STFU variable understood by our Android.mk file. If this variable is set to true, Android.mk will make sure all log outputs of native library are silenced.
  • onlyActiveArch - a boolean that defines whether we want to build only ABI of currently connected device. The idea for this actually came from iOS, where Xcode let's you build only architecture of currently connected iPhone (it does not build 64-bit binary if you connected iPhone 5). We will write our script in a such way that if this variable is set to true, it will be able to detect CPU architecture of connected device (or emulator) and build native library only for this architecture, thus saving time.
So, as you can see, debug build type will have ProGuard disabled, will allow debug logs, will have native code debuggable and will be built only for architecture of connected device. Release build type will have ProGuard enabled, will allow debug logs, will have native code debuggable, although optimized by compiler and will also be built only for architecture of connected device. Distribute build type will have ProGuard enabled, no logs, no debuggable native code and will be built for all architectures specified by product flavor.

Now, let's define product flavors and their settings:


// continued from above
productFlavors.whenObjectAdded { flavor ->
    flavor.ext {
        buildSettings = 'build_settings.mk'
        ndkAbi = 'armeabi armeabi-v7a x86 arm64-v8a'
        app_stl = 'gnustl_static'
        ndkToolChainVersion = '4.9'
    }
}

productFlavors {
    photomath {
        proguardFiles 'proguard/keep.pro'
        ext {
            buildSettings = 'build_settings_photomath.mk'
            ndkAbi = 'armeabi-v7a x86 arm64-v8a'
        }
    }
    blinkBarcode {
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard/common.pro', 'proguard/barcode.pro'
        ext {
            buildSettings = 'build_settings_barcode.mk'
            ndkAbi = 'armeabi armeabi-v7a x86 arm64-v8a x86_64 mips mips64'
        }
    }
    blinkid {
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard/common.pro', 'proguard/blinkid.pro'
        ext {
            buildSettings = 'build_settings_blinkid.mk'
            ndkAbi = 'armeabi-v7a x86 arm64-v8a'
        }
    }
    blinkOcr {
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard/common.pro', 'proguard/blinkocr.pro'
        ext {
            buildSettings = 'build_settings_blinkocr.mk'
            ndkAbi = 'armeabi-v7a x86 arm64-v8a'
        }
    }
    blinkPhotoPay {
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard/common.pro', 'proguard/photopay.pro'
        ext {
            buildSettings = 'build_settings_photopay_all.mk'
        }
    }
}
// to be continued

Same as with build type, we define default values of build properties for each new flavor and then override those values in concrete flavors. Let't explain the variables:
  • buildSettings - name of the makefile that contains definitions of all variables that control the build of native code
  • ndkAbi - list of supported architectures. When not using onlyActiveArch, architectures listed in this string will be built.
  • app_stl - STL version that will be used (we can use different STL for each flavor if we want to)
  • ndkToolChainVersion - version of NDK toolchain that will be used for building this flavor (yes, we can build one flavor with GCC 4.9, and another with Clang)
So, now we can define our product flavors and how they will be built. PhotoPay will be built for armv6, armv7, arm64 and x86, BlinkBarcode for all available architectures and BlinkOCR, BlinkID and PhotoMath for armv7, arm64 and x86. All flavors will use GNU STL with 4.9 toolchain (however, we allow selecting that per flavor). Besides that, we also defined which ProGuard rule files will be used for each flavor (we are building a library so we need to make sure that API classes are not obfuscated).

Now, let's complete the android block in our build.gradle file:

    // allow publishing of other than default build flavors
    publishNonDefault true
    sourceSets.main {
        // prevent automatic generation of Android.mk
        jni.srcDirs = []
    }
} // this brace closes android block

OK, with publishNonDefault we will ensure that all flavors of our AAR will be published and available to apps for use, and by setting jni.srcDirs to empty array we will ensure that default NDK provided by gradle plugin will not be used. If you use recent versions of gradle plugin for Android, and have some c++ files in your src/main/jni and do not set this to empty array, you will end up with error that will tell your that gradle NDK is deprecated and that you should consider using gradle-experimental plugin. We tried that and concluded that gradle-experimental is still not mature enough for our needs (we hope it soon will be).

Now comes the fun part - making it work. Simply defining all those variables in build types and product flavors will do nothing if we do not used them somehow. After we close the android block in build.gradle, we will be able to iterate over every library variant (i.e. build type - product flavor combination) and in this iteration we will spawn tasks that will compile the native code for us and package the native compilation results into AAR as specified by AAR specification.

Let's begin step by step:

android.libraryVariants.all { variant ->
    def flavorProps = variant.productFlavors.get(0).ext
    def buildTypeProps = variant.buildType.ext
    def buildTypeName = variant.buildType.name
    def flavorName = variant.productFlavors.get(0).name

    String ndkBuildDir = "${buildDir}/intermediates/ndk/${flavorName}/${buildTypeName}"

       // to be continued

We start the iteration and for each library variant we first need to obtain the ext properties of build type and flavor as defined earlier. We obtain flavor's ext property in flavorProps variable and build type's ext property in buildTypeProps variable. We also obtain flavor's name and build type's name because we will need them to create appropriate folder structure. Then, we need to define a folder where native compilation will take place. I decided that this folder will be in intermediates/ndk folder inside build directory. You can choose that path as you wish, as long as it is not in conflict with any of the paths used by other gradle tasks.

Now, let's continue iteration by defining a NDK build task:

// continued from above
def ndkBuildTask = project.tasks.create "${variant.name}NdkBuild", Exec
ndkBuildTask.doFirst {
    // create directory structure
    mkdir(ndkBuildDir)

    // create project.properties file
    def propsFile = new File(ndkBuildDir + "/project.properties")
    propsFile << "target=android-21\n"
    propsFile << "android.library=true\n"
    // delete file if it existed before
    delete userDir + "/build.${variant.name}.conf"
    // create conf file
    def conf = new File(userDir + "/build.${variant.name}.conf")
    conf << "export PATH=\$PATH:/usr/local/bin\n"
    conf << "export SRC_ROOT=" + originalDirectory + "\n"
    conf << "export NDK_DIR=" + ndkDir + "\n"
    conf << "export ANDROID_SDK_DIR=" + androidSdkDir + "\n"
    conf << "export ONLY_ACTIVE_ARCH=${buildTypeProps.onlyActiveArch\n"
    conf << "export NDK_CCACHE=" + ndkCache + "\n"
    conf << "export NDK_PROJECT_PATH=${ndkBuildDir}\n"
    conf << "export APP_BUILD_SCRIPT=${originalDirectory}/jni/Android.mk\n"
    conf << "export NDK_APPLICATION_MK=${originalDirectory}/jni/Application.mk\n"
    conf << "export BUILD_SETTINGS=${flavorProps.buildSettings}\n"
    conf << "export APP_ABI=\"${flavorProps.ndkAbi}\"\n"
    conf << "export APP_STL=\"${flavorProps.app_stl}\"\n"
    conf << "export NDK_TOOLCHAIN_VERSION=\"${flavorProps.ndkToolChainVersion}\"\n"
    if (buildTypeProps.ndkDebug) {
        conf << "export NDK_DEBUG=1\n"
    }
    conf << "export APP_OPTIM=${buildTypeProps.appOptim}\n"
    if (buildTypeProps.stfu) {
        conf << "export STFU=true\n"
    }
}
ndkBuildTask.commandLine userDir + "/gradle-ndk-build", userDir + "/build.${variant.name}.conf"
ndkBuildTask.doLast {
    // delete conf file
    delete userDir + "/build.${variant.name}.conf"
}
ndkBuildTask.description "Builds a native code for ${variant.name}"

def copyJniArtifacts = project.tasks.create "${variant.name}CopyJNIArtifacts", Copy
copyJniArtifacts.from "${ndkBuildDir}/libs"
copyJniArtifacts.into "${buildDir}/intermediates/bundles/${flavorName}/${buildTypeName}/jni"
copyJniArtifacts.dependsOn ndkBuildTask
// to be continued


Our NDK build task will be Exec task that will execute a shell script named gradle-ndk-build that accepts one parameter - a name of configuration file that contains variables required for shell script. I will later describe the script, now let's focus on gradle task. So, task is named ${variant.name}NdkBuild which means that for each build type - product flavor combination we will have one flavorNameBuildTypeNameNdkBuild task. This task will first prepare a configuration file named build.flavorNameBuildTypeName.conf and project.properties file (required for ndk-build script that comes with Android NDK), then execute shell script called gradle-ndk-build found inside library's project directory using that prepared configuration file and finally when native code is built, task will delete configuration file as it is not required anymore.

After we have created task that will build native libs, we need to ensure those native libs will end up in AAR. AAR specification specifies that native libs must be placed in folder jni inside AAR, so we create a copy task that will copy the result of build into into buildDir/intermediates/bundles/productFlavor/buildType/jni folder. Why this folder? Well, intermediates/bundles path inside gradle build folder contains all files that will be zipped into final AAR for each product flavor - build type combination.

We are almost done with gradle, we just need to ensure this task will get executed when we build our project. The simpliest way to do it is to make ndkBuild task a dependency of javaCompile task. This will ensure that NDK is build always before Java. You can do this with following line:

// continued from above
variant.javaCompile.dependsOn copyJniArtifacts

This will work correctly if you have android gradle plugin 1.3.1 or older. However, in gradle plugin 1.5.0 this will not work. Why? Because of something called Transform API. This API is very cool as it gives you ability to register custom transformations of .class files before they are transformed into dex. However, for some very strange reason, this transformation step performs some transformations of native libraries built with gradle. Since gradle does not support NDK (it's support is deprecated as mentioned earlier), it is very weird why then task called transformNative_libsWithSyncJniLibsForFlavorBuildType is executed and even weirder that this task, if executed after copyJniArtifacts will clear the jni folder in AAR bundle.

At the time of writing this post, I really don't know how to prevent this task from executing or doing whatever it is doing, I devised a simple hack that will solve my problem - I will make sure that copyJniArtifacts task gets executed after this task. Here is the hack:

// continued from above
// gradle plugin 1.5.0 and newer adds this task which messes up data from copyJniArtifacts
def naughtyTaskName = "transformNative_libsWithSyncJniLibsFor${flavorName.capitalize()}${buildTypeName.capitalize()}"
def naughtyTask = project.tasks[naughtyTaskName]
copyJniArtifacts.dependsOn naughtyTask

def bundleTaskName = "bundle${flavorName.capitalize()}${buildTypeName.capitalize()}"
def bundleTask = project.tasks[bundleTaskName]
bundleTask.dependsOn nativeLibsDeploy

So, here we first calculate the name of this weird task (I really want to know what this task does and how to use Transform API to perform this - if anyone can explain that to me, please do so in the comments), then obtain that task from map of all tasks in project and make our copyJniArtifacts task depend on it.

However, since this weird task happens after java compilation, we cannot now make copyJniArtifacts dependency of javaCompile, so we will use the same hack to obtain the bundle task which zips everything into AAR and make sure copyJniArtifacts happens before that.

Now we only need to close the brace of android.libraryVariants.all iteration loop and we are done with build.gradle. You may note that we didn't do anything to perform cleaning of native code. This is not required because we performed our NDK build inside gradle build directory, and default clean task will delete that entire directory - this is good for us.

All that is now left is to show how a gradle-ndk-build shell script works. Here is the script written in BASH (sorry, windows people, but I am not adept at Windows batch files - if you are, feel free to rewrite this shell script into .bat file):

#!/bin/bash

DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

source $1

# determine number of cores
darwin=false
case "`uname`" in
  Darwin* )
    darwin=true
    ;;
esac

JOBS=8
if $darwin; then
    JOBS=`sysctl hw.ncpu | awk '{print $2}'`
else
    JOBS=`grep -c ^processor /proc/cpuinfo`
fi

JOBS=$((JOBS+1))

pushd $SRC_ROOT

params="NDK_CCACHE=$NDK_CCACHE NDK_PROJECT_PATH=$NDK_PROJECT_PATH APP_STL=$APP_STL APP_BUILD_SCRIPT=$APP_BUILD_SCRIPT NDK_APPLICATION_MK=$NDK_APPLICATION_MK BUILD_SETTINGS=$BUILD_SETTINGS APP_OPTIM=$APP_OPTIM NDK_TOOLCHAIN_VERSION=$NDK_TOOLCHAIN_VERSION"

if [ ! -z "$NDK_DEBUG" ]; then
    params="$params NDK_DEBUG=$NDK_DEBUG"
fi

if [ "$STFU" == "true" ]; then
    params="$params STFU=true"
fi

if [ "$ONLY_ACTIVE_ARCH" == "true" ]; then
    echo "Detecting architecture of connected device(s)..."
    ARCHS=`$DIR/detectArch $1`
    echo "Detected '$ARCHS'"
    if [[ ! -z "$ARCHS" ]]; then
        APP_ABI=$ARCHS
    else
        echo "Error: Please connect your device or boot emulator in order to perform build or set 'onlyActiveArch' to false." >&2
        exit 1
    fi
fi

echo "Executing $NDK_DIR/ndk-build $params APP_ABI=\"$APP_ABI\" -j$JOBS"
$NDK_DIR/ndk-build $params APP_ABI="$APP_ABI" -j$JOBS  || exit 1
popd

First, we need to determine our current directory and execute commands from configuration file given as argument. This will ensure that all variables defined in configuration file (which is generated in gradle task) are available in shell environment. Since we want to have as fast as possible compilation, we need to detect number of processor cores of host system so we can execute build on multiple cores. The code shows a detection methods for OS X and for Linux.

After that, we prepare a list of parameters that will be given to ndk-build script that ships with Android NDK. As you can see, the list is a simple copy of variables given in configuration file.

If ONLY_ACTIVE_ARCH variable is set to true, we call the detectArch shell script which will return a list of ABI's of connected devices or empty list. If the script has detected one ore more device architectures, then we set the APP_ABI (the NDK builtin variable) to detected architectures, and if script hasn't detected anything, we fail the build. Finally, we call the ndk-build script that will perform a build of native code.

Finally, let's see how we can detect CPU architectures of connected Android devices using shell script called detectArch:

#!/bin/bash

source $1

ADB=$ANDROID_SDK_DIR/platform-tools/adb

UNKNOWN="unknown"
ARMv6="armeabi-v6"
ARMv6_CLEAN="armeabi"
ARMv7="armeabi-v7a"
ARMv8="arm64-v8a"
X86="x86"

detectArch() {

    local MAYBE_ARMv7
    local MAYBE_ARMv8
    local MAYBE_x86
    local MAYBE_ARMv6

    MAYBE_ARMv7=`$ADB -s $1 shell cat /proc/cpuinfo | grep 'ARMv7'`

    if [[ ! -z "$MAYBE_ARMv7" ]]; then
        echo $ARMv7
        return 0
    fi

    MAYBE_ARMv8=`$ADB -s $1 shell cat /proc/cpuinfo | grep 'AArch64'`

    if [[ ! -z "$MAYBE_ARMv8" ]]; then
        echo $ARMv8
        return 0
    fi

    MAYBE_x86=`$ADB -s $1 shell cat /proc/cpuinfo | grep 'GenuineIntel'`

    if [[ ! -z "$MAYBE_x86" ]]; then
        echo $X86
        return 0
    fi

    MAYBE_ARMv6=`$ADB -s $1 shell cat /proc/cpuinfo | grep 'ARMv6'`

    if [[ ! -z "$MAYBE_ARMv6" ]]; then
        echo $ARMv6
        return 0
    fi

    echo $UNKNOWN
}

listDevices() {
    local DEVICES
    DEVICES=""
    local ADB_OUTPUT
    ADB_OUTPUT=`$ADB devices | awk '{print $1}'`
    for device in $ADB_OUTPUT; do
        if [[ ! "$device" == "List" ]]; then
            DEVICES="$DEVICES $device"
        fi
    done
    echo $DEVICES
}

deviceList=$(listDevices)

if [[ -z "deviceList" ]]; then
    exit 0
fi

archList=""

for device in $deviceList; do
    arch=$(detectArch $device)
    if [[ "$arch" == "$UNKNOWN" ]]; then
        continue
    fi
    alreadyExists=`echo $archList | grep $arch`
    if [[ -z "$alreadyExists" ]]; then
        archList="$archList $arch"
    fi
done

archList="${archList//$ARMv6/$ARMv6_CLEAN}"

echo $archList

So, this script also needs a configuration file as parameter to obtain path to where Android SDK is installed. This is required because script will need to use adb tool to detect connected devices CPU architecture.

The idea of script is very simple: script first lists ID of every connected Android device and then for each device it detects the CPU architecture by reading its /proc/cpuinfo file. The script also makes sure that it will not report duplicates (for that reason function detectArch returns armeabi-v6 for arm6 devices instead of armeabi because armeabi is substring of armeabi-v7a and it may detect duplicates if we have connected both ARMv6 and ARMv7 device - the replacement of armeabi-v6 string with armeabi string is done in second to last line).

So this is it. You have now learned how we at MicroBlink use gradle to build NDK code for each flavor-buildType combination with different settings, how we achieved compilation only for architecture of connected device for development purposes and how we avoided problems introduced with android gradle plugin 1.5.0.

I hope this post will help people around the world in setting up their NDK projects to work with Android Studio and I hope that new gradle-experimental android plugin will soon have all those features that I've shown here and even more.

P.S. Sorry for lots of whitespace inside code blocks - Blogger's WYSIWYG editor is not very code-friendly (actually, when pasting code from Android Studio, What You See Is Not What You Get).

P.P.S. I would like to thank all the people on Stack Overflow and across various blogs for sharing their experiences that helped me building this tutorial, especially ph0b blog post which taught me the basics of gradle NDK almost one year ago when I started migration of our codebase from Ant and Eclipse to Gradle and Android Studio.