Saturday, December 5, 2015

Using gradle to build NDK projects

When I started writing this blog two years ago, I originally planned to write more frequently about various topics. However, the blog ended up with only two posts with tips and tricks for NDK debugging of library projects. Today, NDK debugging of library projects is officially supported with NDK r10e (you can use ndk-gdb --package=app.package.name) and you do not need to perform hacks as described in my earlier posts.

In this article I'm about to write how we at our company solved problem of building our native codebase with gradle and how we use it in production. Note that what I will describe now is not an official way of using NDK from gradle with gradle android plugin. I am writing this in hope that new gradle experimental plugin that will support NDK will have all the features we need and have implemented with hacks I am about to describe. So let's hope these will not be required anymore in the future.

Let's start with what we want to achieve. We have a native codebase with JNI wrappers. We want to create a gradle script that will be able to build AAR that can be distributed to third parties and which will contain everything a client needs to get the functionality we offer with a single reference to AAR library that we delivered. At our company we build multiple products from single codebase. Each product needs to have different set of Java classes that are delivered and different native library that is built with different configuration. Therefore, we use product flavors supported from android gradle plugin. For each product flavor, our gradle script has to support 3 build types: debug, release and distribute.

In debug build type, java code is not obfuscated and native code must be built with debug logging enabled and without compiler optimization. In release build type, java code is both obfuscated and optimized with ProGuard and native code is build with optimization, but debug logging is kept enabled. Distribute build type is same as release, except debug logging is disabled. In our development process, we want to use debug build type to debug java and native code with debuggers (step-by-step mode), release build type to test that everything works OK and to debug obvious bugs and distribute build type to build final AAR that will be distributed to clients.

So, it is obvious that we need to ensure our gradle script can build both java and native code in all combinations of flavors and build types. Even more, when we moved from Eclipse to Android studio we had a mature Android.mk file that was used to build native code in various flavors. Since Eclipse and ant did not support product flavors, we created an Android.mk file that could build different versions of native library based on configuration variables that were stored in several configuration makefiles that were given to ndk-build via command line argument. So when writing our gradle script, we will make sure that we reuse the mature Android.mk file that was used for building native code.

Before starting the writing of gradle script, we will assume that your Android.mk is written in such way that it can build different versions of native library simply by setting BUILD_SETTINGS variable that contains a name of makefile that defines variables that control the build. In our Android.mk this is done with following code snippet:

BUILD_SETTINGS ?= build_settings.mk

# include variables that control the build
include $(LOCAL_PATH)/build_settings/$(BUILD_SETTINGS)

I will not give the details of how Android.mk controls the build of native code. I will only assume that it can build different versions of native code with configuration given with makefile specified by BUILD_SETTINGS variable.

Now we can start writing our gradle script. As a first thing, we will need to setup a local.properties file which will contain settings that are specific for each developer. Let's put these lines in local.properties file of our library project that will build the AAR:

sdk.dir=/Users/dodo/android-sdks
ndk.dir=/Users/dodo/android-sdks/android-ndk
ndk.ccache=/usr/local/bin/ccache

The sdk.dir variable should already be set by Android studio and will contain a path where developer has installed Android SDK. The ndk.dir variable is not set by Android studio, unless you are trying out gradle-experimental plugin. You should set this variable to location where you have installed your NDK. In this document, I assume usage of NDK r10e. The ndk.ccache variable should be set either to null or to path containing the ccache binary. If developer has ccache installed, it can use it to speed up repeated native compilation. If developer does not have it installed, it will not be used.

OK, let's finally start with build.gradle file of our library project. First, we need to load the variables from local.properties file because those variables are not exposed to us in any way by Android gradle plugin.

String userDir = System.getProperty("user.dir")
File fDir = file(userDir);
if(fDir.name != "LibRecognizer") {
    userDir = userDir + "/LibRecognizer"
}

String localProperties = userDir + "/local.properties"
FileInputStream localPropsFs
try {
    localPropsFs = new FileInputStream(localProperties)
} catch (Exception e) {
    throw new GradleException("Please create a " + localProperties + " file!")
}
Properties localProps = new Properties()
localProps.load(localPropsFs)
String androidSdkDir = localProps.getProperty('sdk.dir')

String ndkDir = localProps.getProperty('ndk.dir')
if(ndkDir == null) {
    throw new GradleException("Please define 'ndk.dir' variable in your local.properties file!")
}

String ndkCache = localProps.getProperty('ndk.ccache')

if(ndkCache == null) {
    ndkCache = ""
}
String originalDirectory = userDir + '/src/main'

So, we first obtain the system property called user.dir which will contain the path where our root project is located. Next, as we want this script to work both for our library as nested project (i.e. as subproject of root project that contains our library and a test application) and as root project (i.e. when we just want to build AAR), we must check if userDir is already same as name of our library (name of our library is LibRecognizer) and if it isn't, then we must correct the userDir to point directly to library project folder. This will ensure that userDir will always contain a path to folder containing build.gradle file of our LibRecognizer library project.

Next, we load the properties from local.properties file and store them to variables for later use. We now continue our build.gradle file with more or less known code:

apply plugin: 'com.android.library'

android {
    compileSdkVersion 23
    buildToolsVersion '23.0.2'

    defaultConfig {
        minSdkVersion 10
        targetSdkVersion 23
    }
    // to be continued

If you have ever created a library module in android studio, previous lines should not be unfamiliar to you. Now, let's define build types, but for each build type we want to store some settings that we will need later. In order to do that, we first must define default settings that will apply for each build type and then simply override the defaults:

// continued from above
buildTypes.whenObjectAdded { buildType ->
    buildType.ext {
        ndkDebug = true
        appOptim = "release"
        stfu = false
        onlyActiveArch = true
    }
}

buildTypes {
    debug {
        minifyEnabled false
        ext {
            ndkDebug = true
            appOptim = "debug"
            stfu = false
            onlyActiveArch = true
        }
    }
    release {
        minifyEnabled true
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt')
        ext {
            ndkDebug = true
            appOptim = "release"
            stfu = false
            onlyActiveArch = true
        }
    }
    distribute {
        minifyEnabled true
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt')
        ext {
            ndkDebug = false
            appOptim = "release"
            stfu = true
            onlyActiveArch = false
        }
    }
}
// to be continued

OK, let's explain this code snippet: whenever we create a new build type, we want to initialize its ext property (a 'property' in each gradle object that lets you write anything into it) with default values of variables that we will need later. Let's see what those variables are:

  • ndkDebug - boolean that defines whether or not native code will be built with NDK_DEBUG=1 that will besides libNative.so create files gdbserver and gdb.setup that enable native debugging of application
  • appOptim - a string that defines the optimization level inside NDK. The APP_OPTIM variable will be set to value of this variable
  • stfu - a boolean that will be transformed into value of STFU variable understood by our Android.mk file. If this variable is set to true, Android.mk will make sure all log outputs of native library are silenced.
  • onlyActiveArch - a boolean that defines whether we want to build only ABI of currently connected device. The idea for this actually came from iOS, where Xcode let's you build only architecture of currently connected iPhone (it does not build 64-bit binary if you connected iPhone 5). We will write our script in a such way that if this variable is set to true, it will be able to detect CPU architecture of connected device (or emulator) and build native library only for this architecture, thus saving time.
So, as you can see, debug build type will have ProGuard disabled, will allow debug logs, will have native code debuggable and will be built only for architecture of connected device. Release build type will have ProGuard enabled, will allow debug logs, will have native code debuggable, although optimized by compiler and will also be built only for architecture of connected device. Distribute build type will have ProGuard enabled, no logs, no debuggable native code and will be built for all architectures specified by product flavor.

Now, let's define product flavors and their settings:


// continued from above
productFlavors.whenObjectAdded { flavor ->
    flavor.ext {
        buildSettings = 'build_settings.mk'
        ndkAbi = 'armeabi armeabi-v7a x86 arm64-v8a'
        app_stl = 'gnustl_static'
        ndkToolChainVersion = '4.9'
    }
}

productFlavors {
    photomath {
        proguardFiles 'proguard/keep.pro'
        ext {
            buildSettings = 'build_settings_photomath.mk'
            ndkAbi = 'armeabi-v7a x86 arm64-v8a'
        }
    }
    blinkBarcode {
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard/common.pro', 'proguard/barcode.pro'
        ext {
            buildSettings = 'build_settings_barcode.mk'
            ndkAbi = 'armeabi armeabi-v7a x86 arm64-v8a x86_64 mips mips64'
        }
    }
    blinkid {
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard/common.pro', 'proguard/blinkid.pro'
        ext {
            buildSettings = 'build_settings_blinkid.mk'
            ndkAbi = 'armeabi-v7a x86 arm64-v8a'
        }
    }
    blinkOcr {
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard/common.pro', 'proguard/blinkocr.pro'
        ext {
            buildSettings = 'build_settings_blinkocr.mk'
            ndkAbi = 'armeabi-v7a x86 arm64-v8a'
        }
    }
    blinkPhotoPay {
        proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard/common.pro', 'proguard/photopay.pro'
        ext {
            buildSettings = 'build_settings_photopay_all.mk'
        }
    }
}
// to be continued

Same as with build type, we define default values of build properties for each new flavor and then override those values in concrete flavors. Let't explain the variables:
  • buildSettings - name of the makefile that contains definitions of all variables that control the build of native code
  • ndkAbi - list of supported architectures. When not using onlyActiveArch, architectures listed in this string will be built.
  • app_stl - STL version that will be used (we can use different STL for each flavor if we want to)
  • ndkToolChainVersion - version of NDK toolchain that will be used for building this flavor (yes, we can build one flavor with GCC 4.9, and another with Clang)
So, now we can define our product flavors and how they will be built. PhotoPay will be built for armv6, armv7, arm64 and x86, BlinkBarcode for all available architectures and BlinkOCR, BlinkID and PhotoMath for armv7, arm64 and x86. All flavors will use GNU STL with 4.9 toolchain (however, we allow selecting that per flavor). Besides that, we also defined which ProGuard rule files will be used for each flavor (we are building a library so we need to make sure that API classes are not obfuscated).

Now, let's complete the android block in our build.gradle file:

    // allow publishing of other than default build flavors
    publishNonDefault true
    sourceSets.main {
        // prevent automatic generation of Android.mk
        jni.srcDirs = []
    }
} // this brace closes android block

OK, with publishNonDefault we will ensure that all flavors of our AAR will be published and available to apps for use, and by setting jni.srcDirs to empty array we will ensure that default NDK provided by gradle plugin will not be used. If you use recent versions of gradle plugin for Android, and have some c++ files in your src/main/jni and do not set this to empty array, you will end up with error that will tell your that gradle NDK is deprecated and that you should consider using gradle-experimental plugin. We tried that and concluded that gradle-experimental is still not mature enough for our needs (we hope it soon will be).

Now comes the fun part - making it work. Simply defining all those variables in build types and product flavors will do nothing if we do not used them somehow. After we close the android block in build.gradle, we will be able to iterate over every library variant (i.e. build type - product flavor combination) and in this iteration we will spawn tasks that will compile the native code for us and package the native compilation results into AAR as specified by AAR specification.

Let's begin step by step:

android.libraryVariants.all { variant ->
    def flavorProps = variant.productFlavors.get(0).ext
    def buildTypeProps = variant.buildType.ext
    def buildTypeName = variant.buildType.name
    def flavorName = variant.productFlavors.get(0).name

    String ndkBuildDir = "${buildDir}/intermediates/ndk/${flavorName}/${buildTypeName}"

       // to be continued

We start the iteration and for each library variant we first need to obtain the ext properties of build type and flavor as defined earlier. We obtain flavor's ext property in flavorProps variable and build type's ext property in buildTypeProps variable. We also obtain flavor's name and build type's name because we will need them to create appropriate folder structure. Then, we need to define a folder where native compilation will take place. I decided that this folder will be in intermediates/ndk folder inside build directory. You can choose that path as you wish, as long as it is not in conflict with any of the paths used by other gradle tasks.

Now, let's continue iteration by defining a NDK build task:

// continued from above
def ndkBuildTask = project.tasks.create "${variant.name}NdkBuild", Exec
ndkBuildTask.doFirst {
    // create directory structure
    mkdir(ndkBuildDir)

    // create project.properties file
    def propsFile = new File(ndkBuildDir + "/project.properties")
    propsFile << "target=android-21\n"
    propsFile << "android.library=true\n"
    // delete file if it existed before
    delete userDir + "/build.${variant.name}.conf"
    // create conf file
    def conf = new File(userDir + "/build.${variant.name}.conf")
    conf << "export PATH=\$PATH:/usr/local/bin\n"
    conf << "export SRC_ROOT=" + originalDirectory + "\n"
    conf << "export NDK_DIR=" + ndkDir + "\n"
    conf << "export ANDROID_SDK_DIR=" + androidSdkDir + "\n"
    conf << "export ONLY_ACTIVE_ARCH=${buildTypeProps.onlyActiveArch\n"
    conf << "export NDK_CCACHE=" + ndkCache + "\n"
    conf << "export NDK_PROJECT_PATH=${ndkBuildDir}\n"
    conf << "export APP_BUILD_SCRIPT=${originalDirectory}/jni/Android.mk\n"
    conf << "export NDK_APPLICATION_MK=${originalDirectory}/jni/Application.mk\n"
    conf << "export BUILD_SETTINGS=${flavorProps.buildSettings}\n"
    conf << "export APP_ABI=\"${flavorProps.ndkAbi}\"\n"
    conf << "export APP_STL=\"${flavorProps.app_stl}\"\n"
    conf << "export NDK_TOOLCHAIN_VERSION=\"${flavorProps.ndkToolChainVersion}\"\n"
    if (buildTypeProps.ndkDebug) {
        conf << "export NDK_DEBUG=1\n"
    }
    conf << "export APP_OPTIM=${buildTypeProps.appOptim}\n"
    if (buildTypeProps.stfu) {
        conf << "export STFU=true\n"
    }
}
ndkBuildTask.commandLine userDir + "/gradle-ndk-build", userDir + "/build.${variant.name}.conf"
ndkBuildTask.doLast {
    // delete conf file
    delete userDir + "/build.${variant.name}.conf"
}
ndkBuildTask.description "Builds a native code for ${variant.name}"

def copyJniArtifacts = project.tasks.create "${variant.name}CopyJNIArtifacts", Copy
copyJniArtifacts.from "${ndkBuildDir}/libs"
copyJniArtifacts.into "${buildDir}/intermediates/bundles/${flavorName}/${buildTypeName}/jni"
copyJniArtifacts.dependsOn ndkBuildTask
// to be continued


Our NDK build task will be Exec task that will execute a shell script named gradle-ndk-build that accepts one parameter - a name of configuration file that contains variables required for shell script. I will later describe the script, now let's focus on gradle task. So, task is named ${variant.name}NdkBuild which means that for each build type - product flavor combination we will have one flavorNameBuildTypeNameNdkBuild task. This task will first prepare a configuration file named build.flavorNameBuildTypeName.conf and project.properties file (required for ndk-build script that comes with Android NDK), then execute shell script called gradle-ndk-build found inside library's project directory using that prepared configuration file and finally when native code is built, task will delete configuration file as it is not required anymore.

After we have created task that will build native libs, we need to ensure those native libs will end up in AAR. AAR specification specifies that native libs must be placed in folder jni inside AAR, so we create a copy task that will copy the result of build into into buildDir/intermediates/bundles/productFlavor/buildType/jni folder. Why this folder? Well, intermediates/bundles path inside gradle build folder contains all files that will be zipped into final AAR for each product flavor - build type combination.

We are almost done with gradle, we just need to ensure this task will get executed when we build our project. The simpliest way to do it is to make ndkBuild task a dependency of javaCompile task. This will ensure that NDK is build always before Java. You can do this with following line:

// continued from above
variant.javaCompile.dependsOn copyJniArtifacts

This will work correctly if you have android gradle plugin 1.3.1 or older. However, in gradle plugin 1.5.0 this will not work. Why? Because of something called Transform API. This API is very cool as it gives you ability to register custom transformations of .class files before they are transformed into dex. However, for some very strange reason, this transformation step performs some transformations of native libraries built with gradle. Since gradle does not support NDK (it's support is deprecated as mentioned earlier), it is very weird why then task called transformNative_libsWithSyncJniLibsForFlavorBuildType is executed and even weirder that this task, if executed after copyJniArtifacts will clear the jni folder in AAR bundle.

At the time of writing this post, I really don't know how to prevent this task from executing or doing whatever it is doing, I devised a simple hack that will solve my problem - I will make sure that copyJniArtifacts task gets executed after this task. Here is the hack:

// continued from above
// gradle plugin 1.5.0 and newer adds this task which messes up data from copyJniArtifacts
def naughtyTaskName = "transformNative_libsWithSyncJniLibsFor${flavorName.capitalize()}${buildTypeName.capitalize()}"
def naughtyTask = project.tasks[naughtyTaskName]
copyJniArtifacts.dependsOn naughtyTask

def bundleTaskName = "bundle${flavorName.capitalize()}${buildTypeName.capitalize()}"
def bundleTask = project.tasks[bundleTaskName]
bundleTask.dependsOn nativeLibsDeploy

So, here we first calculate the name of this weird task (I really want to know what this task does and how to use Transform API to perform this - if anyone can explain that to me, please do so in the comments), then obtain that task from map of all tasks in project and make our copyJniArtifacts task depend on it.

However, since this weird task happens after java compilation, we cannot now make copyJniArtifacts dependency of javaCompile, so we will use the same hack to obtain the bundle task which zips everything into AAR and make sure copyJniArtifacts happens before that.

Now we only need to close the brace of android.libraryVariants.all iteration loop and we are done with build.gradle. You may note that we didn't do anything to perform cleaning of native code. This is not required because we performed our NDK build inside gradle build directory, and default clean task will delete that entire directory - this is good for us.

All that is now left is to show how a gradle-ndk-build shell script works. Here is the script written in BASH (sorry, windows people, but I am not adept at Windows batch files - if you are, feel free to rewrite this shell script into .bat file):

#!/bin/bash

DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

source $1

# determine number of cores
darwin=false
case "`uname`" in
  Darwin* )
    darwin=true
    ;;
esac

JOBS=8
if $darwin; then
    JOBS=`sysctl hw.ncpu | awk '{print $2}'`
else
    JOBS=`grep -c ^processor /proc/cpuinfo`
fi

JOBS=$((JOBS+1))

pushd $SRC_ROOT

params="NDK_CCACHE=$NDK_CCACHE NDK_PROJECT_PATH=$NDK_PROJECT_PATH APP_STL=$APP_STL APP_BUILD_SCRIPT=$APP_BUILD_SCRIPT NDK_APPLICATION_MK=$NDK_APPLICATION_MK BUILD_SETTINGS=$BUILD_SETTINGS APP_OPTIM=$APP_OPTIM NDK_TOOLCHAIN_VERSION=$NDK_TOOLCHAIN_VERSION"

if [ ! -z "$NDK_DEBUG" ]; then
    params="$params NDK_DEBUG=$NDK_DEBUG"
fi

if [ "$STFU" == "true" ]; then
    params="$params STFU=true"
fi

if [ "$ONLY_ACTIVE_ARCH" == "true" ]; then
    echo "Detecting architecture of connected device(s)..."
    ARCHS=`$DIR/detectArch $1`
    echo "Detected '$ARCHS'"
    if [[ ! -z "$ARCHS" ]]; then
        APP_ABI=$ARCHS
    else
        echo "Error: Please connect your device or boot emulator in order to perform build or set 'onlyActiveArch' to false." >&2
        exit 1
    fi
fi

echo "Executing $NDK_DIR/ndk-build $params APP_ABI=\"$APP_ABI\" -j$JOBS"
$NDK_DIR/ndk-build $params APP_ABI="$APP_ABI" -j$JOBS  || exit 1
popd

First, we need to determine our current directory and execute commands from configuration file given as argument. This will ensure that all variables defined in configuration file (which is generated in gradle task) are available in shell environment. Since we want to have as fast as possible compilation, we need to detect number of processor cores of host system so we can execute build on multiple cores. The code shows a detection methods for OS X and for Linux.

After that, we prepare a list of parameters that will be given to ndk-build script that ships with Android NDK. As you can see, the list is a simple copy of variables given in configuration file.

If ONLY_ACTIVE_ARCH variable is set to true, we call the detectArch shell script which will return a list of ABI's of connected devices or empty list. If the script has detected one ore more device architectures, then we set the APP_ABI (the NDK builtin variable) to detected architectures, and if script hasn't detected anything, we fail the build. Finally, we call the ndk-build script that will perform a build of native code.

Finally, let's see how we can detect CPU architectures of connected Android devices using shell script called detectArch:

#!/bin/bash

source $1

ADB=$ANDROID_SDK_DIR/platform-tools/adb

UNKNOWN="unknown"
ARMv6="armeabi-v6"
ARMv6_CLEAN="armeabi"
ARMv7="armeabi-v7a"
ARMv8="arm64-v8a"
X86="x86"

detectArch() {

    local MAYBE_ARMv7
    local MAYBE_ARMv8
    local MAYBE_x86
    local MAYBE_ARMv6

    MAYBE_ARMv7=`$ADB -s $1 shell cat /proc/cpuinfo | grep 'ARMv7'`

    if [[ ! -z "$MAYBE_ARMv7" ]]; then
        echo $ARMv7
        return 0
    fi

    MAYBE_ARMv8=`$ADB -s $1 shell cat /proc/cpuinfo | grep 'AArch64'`

    if [[ ! -z "$MAYBE_ARMv8" ]]; then
        echo $ARMv8
        return 0
    fi

    MAYBE_x86=`$ADB -s $1 shell cat /proc/cpuinfo | grep 'GenuineIntel'`

    if [[ ! -z "$MAYBE_x86" ]]; then
        echo $X86
        return 0
    fi

    MAYBE_ARMv6=`$ADB -s $1 shell cat /proc/cpuinfo | grep 'ARMv6'`

    if [[ ! -z "$MAYBE_ARMv6" ]]; then
        echo $ARMv6
        return 0
    fi

    echo $UNKNOWN
}

listDevices() {
    local DEVICES
    DEVICES=""
    local ADB_OUTPUT
    ADB_OUTPUT=`$ADB devices | awk '{print $1}'`
    for device in $ADB_OUTPUT; do
        if [[ ! "$device" == "List" ]]; then
            DEVICES="$DEVICES $device"
        fi
    done
    echo $DEVICES
}

deviceList=$(listDevices)

if [[ -z "deviceList" ]]; then
    exit 0
fi

archList=""

for device in $deviceList; do
    arch=$(detectArch $device)
    if [[ "$arch" == "$UNKNOWN" ]]; then
        continue
    fi
    alreadyExists=`echo $archList | grep $arch`
    if [[ -z "$alreadyExists" ]]; then
        archList="$archList $arch"
    fi
done

archList="${archList//$ARMv6/$ARMv6_CLEAN}"

echo $archList

So, this script also needs a configuration file as parameter to obtain path to where Android SDK is installed. This is required because script will need to use adb tool to detect connected devices CPU architecture.

The idea of script is very simple: script first lists ID of every connected Android device and then for each device it detects the CPU architecture by reading its /proc/cpuinfo file. The script also makes sure that it will not report duplicates (for that reason function detectArch returns armeabi-v6 for arm6 devices instead of armeabi because armeabi is substring of armeabi-v7a and it may detect duplicates if we have connected both ARMv6 and ARMv7 device - the replacement of armeabi-v6 string with armeabi string is done in second to last line).

So this is it. You have now learned how we at MicroBlink use gradle to build NDK code for each flavor-buildType combination with different settings, how we achieved compilation only for architecture of connected device for development purposes and how we avoided problems introduced with android gradle plugin 1.5.0.

I hope this post will help people around the world in setting up their NDK projects to work with Android Studio and I hope that new gradle-experimental android plugin will soon have all those features that I've shown here and even more.

P.S. Sorry for lots of whitespace inside code blocks - Blogger's WYSIWYG editor is not very code-friendly (actually, when pasting code from Android Studio, What You See Is Not What You Get).

P.P.S. I would like to thank all the people on Stack Overflow and across various blogs for sharing their experiences that helped me building this tutorial, especially ph0b blog post which taught me the basics of gradle NDK almost one year ago when I started migration of our codebase from Ant and Eclipse to Gradle and Android Studio.

Wednesday, September 18, 2013

Limiting bandwidth on a rooted Android phone

While developing Android applications that use network communication it is often required to test the app in poor network conditions. Android emulator has an option -netdelay with which you can tell emulator to simulate gprs, edge, 3g or similar network conditions. The main problem with using emulator for simulating poor network conditions is that Android emulator is slow. Second problem is that you cannot use it to take pictures with camera. And what if you want to create an app that takes an picture and uploads it to a web server? You can only test such app on a real device. And how can you simulate poor network conditions on a real device?

Well, if you have SIM card in your phone, you can restrict it to use only gprs/edge. But if you have multiple phones for development and neither of them has SIM card in it? Or even better, what if you want to test against local server running on your computer? You need to limit Wi-Fi bandwidth somehow.

Fortunately, this is possible if you have a rooted phone. Android is essentially a Linux, and Linux has a tool called tc. Tc is short for traffic control, and this is exactly what it does. You can find numerous tutorials all over the internet how to use tc to optimise your server. I've found two such a tutorials on this site and this site. Solution from one site did the trick for me with upload limiting, and solution from other did the trick for download limiting. Therefore, I've made a hybrid script so it worked for me.

Here is it:
 #!/system/bin/bash  
 #  
 # tc uses the following units when passed as a parameter.  
 # kbps: Kilobytes per second  
 # mbps: Megabytes per second  
 # kbit: Kilobits per second  
 # mbit: Megabits per second  
 # bps: Bytes per second  
 #    Amounts of data can be specified in:  
 #    kb or k: Kilobytes  
 #    mb or m: Megabytes  
 #    mbit: Megabits  
 #    kbit: Kilobits  
 # To get the byte figure from bits, divide the number by 8 bit  
 #  
 #  
 # Name of the traffic control command.  
 TC=tc  
 # The network interface we're planning on limiting bandwidth.  
 IF=wlan0       # Interface  
 # Download limit (in mega bits)  
 DNLD=1mbit     # DOWNLOAD Limit  
 # Upload limit (in mega bits)  
 UPLD=1mbit     # UPLOAD Limit  
 # IP address of the machine we are controlling  
 IP=192.168.5.14   # Host IP  
 start() {  
 # Limit download  
 $TC qdisc add dev $IF handle ffff: ingress                  
 $TC filter add dev $IF parent ffff: protocol ip prio 50 u32 match ip src $IP/32 police rate $DNLD burst 100k drop flowid :1  
 # Limit upload  
 $TC qdisc add dev $IF root handle 1: htb default 30  
 $TC class add dev $IF parent 1: classid 1:1 htb rate $UPLD  
 $TC filter add dev $IF protocol ip parent 1:0 prio 1 u32 match ip dst $IP/32 flowid 1:1  
 }  
 stop() {  
 # Stop the bandwidth shaping.  
 $TC qdisc del dev $IF ingress  
 $TC qdisc del dev $IF root  
 }  
 restart() {  
 # Self-explanatory.  
 stop  
 sleep 1  
 start  
 }  
 show() {  
 # Display status of traffic control status.  
 $TC -s qdisc ls dev $IF  
 }  
 case "$1" in  
 start)  
 echo -n "Starting bandwidth shaping: "  
 start  
 echo "done"  
 ;;  
 stop)  
 echo -n "Stopping bandwidth shaping: "  
 stop  
 echo "done"  
 ;;  
 restart)  
 echo -n "Restarting bandwidth shaping: "  
 restart  
 echo "done"  
 ;;  
 show)  
 echo "Bandwidth shaping status for $IF:"  
 show  
 echo ""  
 ;;  
 *)  
 pwd=$(pwd)  
 echo "Usage: tc.bash {start|stop|restart|show}"  
 ;;  
 esac   
 exit 0  

Put this script in a file called limitBandwidth.sh, and transfer the file on your rooted phone (for example by using adb). I've placed it in folder /sdcard/.

So now, type adb shell to enter device's shell and execute following command:
su -c "sh /sdcard/limitBandwidth.sh start". After that your phone should have limited bandwidth towards the IP address you set in the script. If you want to limit bandwidth towards all addresses, just set IP to 0.0.0.0.

In my opinion it is a better option to limit bandwidth only towards your development machine that runs the test server. You can then use the phone normally for browsing and streaming videos without any limits in bandwidth whilst all requests towards your local machine will have limited bandwidth.

I've tested the script on Samsung Galaxy S2 with CyanogenMod 10.3. If you find it working on other rooted phone, please tell in comments that it works.

If it doesn't work, tell in comments that it doesn't work and attach an error. I cannot promise that I could make it work on that phone (most probably I would not), but if I get my hands on such phone, I will definitely investigate why it is not working.

Monday, March 11, 2013

Android native code debugging with Eclipse

OK, this has been covered in numerous blogs and forum posts, but I will post this tutorial for multiple reasons. The first reason is that this will be a reminder for me, and the second one is the fact that all tutorials cover the steps for debugging android application projects that have native part; but what about debugging android library project that has native part? What if you have a situation like this: you have an android library project that has native part and you have pure java-based android application project which uses the library project and you have bug in your native part. How to debug this? Well, you can always add native support to your application project and then make the debugging, but why doing that, if there is an easier way to solve this?

For this tutorial I will assume that you have the latest version of Eclipse, Android SDK and Android NDK installed. In my case, eclipse is at version Juno, Android SDK is at revision 21, and I'm using NDK r8d. Of course, this tutorial should work with earlier versions of NDK and SDK, but I haven't tested that.

OK, let's start:
First, you need to adapt the ndk-gdb script which is located in Android NDK dir. Create a copy of that script; let's call it ndk-gdb-eclipse. Open the copied file in your favorite editor and find the following snippet:


    PACKAGE_NAME=`run_awk_manifest_script extract-package-name.awk`
    log "Found package name: $PACKAGE_NAME"
    if [ $? != 0 -o "$PACKAGE_NAME" = "<none>" ] ; then
        echo "ERROR: Could not extract package name from $PROJECT/$MANIFEST."
        echo "       Please check that the file is well-formed!"
        exit 1
    fi

Now, edit this so it will look like this:

if [[ -z "$PACKAGE_NAME" ]]; then
    PACKAGE_NAME=`run_awk_manifest_script extract-package-name.awk`
    log "Found package name: $PACKAGE_NAME"
    if [ $? != 0 -o "$PACKAGE_NAME" = "<none>" ] ; then
        echo "ERROR: Could not extract package name from $PROJECT/$MANIFEST."
        echo "       Please check that the file is well-formed!"
        exit 1
    fi
else
    log "Using predefined package: $PACKAGE_NAME"
fi

OK, what does that do? This code actually checks if the variable PACKAGE_NAME is set, and if it is not set, it sets it to the name of the package for Android project in current directory. This works fine if you are trying to debug an application, but if you try to debug Android library project, the found package name will be wrong (the name of the package in the library does not need to be the same as the name of the package of application that uses the library).

OK, don't close the text editor yet, there is a still one little change that needs to be done. Scroll to the end of the script. The last line should be as this:
$GDBCLIENT -x `native_path $GDBSETUP`
Comment-out this line by prefixing it with #. The changed line should look like this:

#$GDBCLIENT -x `native_path $GDBSETUP` 
 OK, now save the file - the hard hacker-style work is done. Actually not yet, we will still write a little helper bash script which is actually not needed, but vastly eases the later usage. So, open your editor and put the following code inside it:

 #!/bin/bash
 # activate remote port to pipe forwarding
 PACKAGE_NAME=$1

 if [[ -z "$PACKAGE_NAME" ]]; then
  echo "No package name defined :-("
 else
  export PACKAGE_NAME
  ndk-gdb-eclipse --force --verbose
 fi

Save the file as "android-debug" and make it executable with chmod +x android-debug.

Next, you should build your native code with ndk-build NDK_DEBUG=1, which will enable native debugging. The easiest way to ensure that your native code is always built with debugging support is to make eclipse do that for you. Right click the project that contains native code, choose properties, click the C/C++ Build tab, uncheck "Use default build command" and add NDK_DEBUG=1 to build command (see the picture).

Example of C/C++ build settings for android library project with native code
OK, now connect your phone or start android emulator and start the application project which will use the library project you are about to debug. You can start the application by either selecting "Run" or "Debug". If you select "Run", you will be able to debug only the native part, and if you select "Debug", you will be able to debug both the native and the java part of your project.

Just to be sure that everything will work, add android:debuggable="true" to application tag in your AndroidManifest.xml. In my case, this is required only for debugging of java part of library project, but not for native debug.

OK, the application is now started on your phone, but before being able to debug the native part, you still have to perform some steps. First, open your terminal and navigate to root folder of library project that contains the native code. Run the following command:
android-debug test.application.packagename
So, "android-debug" is the script that you have created earlier, and test.application.packagename is the name of the package of application that uses the library project (not the package of library project). For example, if your library project is in package com.test.library and your application is in package net.cool.application, you should run "android-debug net.cool.application".
The script will then run the gdbserver on phone that will attach to the process of application you started. Now, all you have to do is connect your eclipse to that gdbserver and debugging can start. Warning, on some phones, deploying of gdbserver may not succeed. Unfortunately, on such phones, debugging the native code is not possible.

To connect your eclipse to the gdbserver on the phone do the following: open the Debug Configurations dialog in Eclipse (little arrow next to the button for starting the debug, and choosing "Debug Configurations..." option.

Create new C/C++ Remote Application launch configuration as seen on the following screenshot:


If you don't have C/C++ Remote Application launch configuration option, then you should install C/C++ hardware debug support from CDT update site.
In C/C++ application field enter the full path to app_process file that will be found in libraries ./obj/local/armeabi folder. If you can't find that file, that means that android-debug script has failed.

Next, click the debugger tab and enter the full path to arm-linux-androideabi-gdb file that is found in android-ndk-dir/toolchains/arm-linux-androideabi-4.6/prebuilt/linux-x86/bin/ folder (I'm not sure where this is found in windows or mac, but it should be similar) to the GDB debugger field and into the GDB command file put the full path to the gdb.setup file that was created by ndk-build script (it should be found next to .so file in libs/armeabi folder). See the next screenshot for example:


Finally, open the Connection tab and set the port number to 5039.


And that is all. Click the Debug button and start putting breakpoints to your native code. When phone hits them, the execution will be suspended and you will be able to examine variables, do steps and everything else you are used to do when debugging C/C++ code with eclipse CDT.

I hope this wasn't too complicated...