Code, Design, and Growth at SeatGeek

Jobs at SeatGeek

We are growing fast, and have lots of open positions!

Explore Career Opportunities at SeatGeek

Continuous Integration for Android Applications

In our quest to make everything as automated as possible, the SeatGeek dev team has been using Travis-CI to build our Android application apks automatically. Since the process of doing Android-based CI is a bit difficult to get correct without trial and error, we’ll document it here for everyone else to use!

A few caveats

Travis-CI does not support Java 6 - Oracle JDK 7 and 8 are available, as is the OpenJDK. We encountered an issue related to jar-signing, which was easily fixed by adding the following to our app’s pom.xml file:

<!-- Fix for jar signing with java 1.7
http://stackoverflow.com/questions/8738962/what-kind-of-pitfals-exist-for-the-android-apk-signing/9567153#9567153 -->
<configuration>
   <arguments>
       <argument>-sigalg</argument><argument>MD5withRSA</argument>
       <argument>-digestalg</argument><argument>SHA1</argument>
   </arguments>
</configuration>

Travis sometimes upgrades their infrastructure. You may need to install packages from source to get everything working. The only time this has bitten us is with the Maven version, which we’ll discuss below.

Travis currently does not have the android SDKs by default, so you need to manage this yourself. We ended up building a script to cache them, though there is an outstanding PR that may fix the issue, so be on the lookout.

Initial Setup

Since we upload APKs to S3, we’ve included a few gems in our Gemfile:

source 'https://rubygems.org'

gem 'travis', '1.5.4'
gem 'travis-artifacts', '~>0.1'

In our .travis.yml, we’ll want to specify the java build-type and the jdk version:

language: java
jdk: oraclejdk7

Next, you’ll want to specify some environment variables for the ANDROID_TARGET and ANDROID_ABI. SeatGeek uses a single matrix entry, though you are welcome to build for multiple targets:

env:
  matrix:
    - ANDROID_TARGET=android-18  ANDROID_ABI=armeabi-v7a

We also upload the build apk to our S3 for exposure using our build-artifacts application, so we have some S3-related environment variables. Note that we encrypt our S3 access_key_id and secret_access_key, which is something I highly recommend doing:

env:
  global:
    - "ARTIFACTS_AWS_REGION=us-east-1"
    - "ARTIFACTS_S3_BUCKET=shared_bucket"
    # travis-artifacts ARTIFACTS_AWS_ACCESS_KEY_ID
    - secure: "SOME_SECURE_ACCESS_KEY_ID"
    # travis-artifacts ARTIFACTS_AWS_SECRET_ACCESS_KEY
    - secure: "SOME_SECURE_SECRET_ACCESS_KEY"

I recommend enabling the apt cache. We need to install some system packages in order to run the android build tools, so this will cut down some time from your build. In our testing, it was ~4 minutes.

cache:
  - apt

If you have setup the build-artifacts tool, then you’ll need a notification entry for the app:

notifications:
  webhooks:
    urls:
      # build-artifacts app
      - http://ancient-foot-stomps-alligator.herokuapp.com/travisci/

before_install: The Meat and Potatoes

Next is the before_install step. This is the meat of our setup, and was the result of quite a bit of trial and effort. I’ll be candid and state it took ~100 builds to figure out everything. The android devs were confused about whether the app actually worked, to say the least.

before_install:
  - sudo apt-get install -qq --force-yes expect libgd2-xpm ia32-libs ia32-libs-multiarch s3cmd > /dev/null

  - export ANDROID_HOME=${HOME}/android-sdk-linux/
  - export ANDROID_APP_DIR=${PWD}
  - export PATH=${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/platform-tools:${HOME}/.bundle/ruby/1.9.1/bin
  - export GEM_PATH=${HOME}/.bundle/ruby/1.9.1:${GEM_PATH}

  # Install 3.0.5 maven
  - bin/install_maven

  # Install Ruby requirements
  - bin/cache_deps -a $ARTIFACTS_AWS_ACCESS_KEY_ID -b $ARTIFACTS_S3_BUCKET -d 'Gemfile.lock' -e $HOME -f ".bundle" -i "bin/install_gems" -p "android/bundles" -s $ARTIFACTS_AWS_SECRET_ACCESS_KEY

  # Install Android requirements
  - bin/cache_deps -a $ARTIFACTS_AWS_ACCESS_KEY_ID -b $ARTIFACTS_S3_BUCKET -d 'deps.txt' -e $HOME -f "android-sdk-linux" -i "./bin/install_sdk" -p "android/deps" -s $ARTIFACTS_AWS_SECRET_ACCESS_KEY

I’ll explain this step-by-step.

  1. We need to install some system packages for the android dev tools. They won’t run without these, unfortunately, and the packages aren’t included by default in TravisCI.
  2. We set some android-related environment variables to ensure the app can be properly built. The PATH and GEM_PATH are overriden because we want to specify where the build tools live, as well as use a custom path for the bundle requirements.
  3. If you use maven in any way, then your best bet is to force the installation of maven 3.0.x. This ticket explains it in more detail than I can in a few words.
  4. At the time I originally implemented this, there was no bundler caching. There is bundler caching built into Travis-CI now, but it is untested in our setup. Probably works, so worth trying!
  5. We cache all android sdk requirements using a custom dependency file which I’ll outline below. Trying to hit the Android repositories for the SDK is error-prone, and you’ll eventually be throttled, which balloons a 4-minute build to a 45-minute build.

Helper scripts

You’ll notice we have two scripts in use, cache_deps, install_maven, install_gems, and install_sdk. They are outlined below:

install_maven

The install_maven command was actually created today - we realized recent updates in TravisCI broke all of our builds, and the change that broke things was upgrading to Maven 3.1.1. Thankfully the awesome folks at Travis lent me a built-vm to futz with and I was able to fix the issue. This can’t be easily fixed on their end without requiring everyone use an older version of Maven, which is undesirable, so this script is available for all to use!

#!/usr/bin/env bash

VERSION=3.0.5

if [ -d /usr/local/maven-3.* ]; then
  echo "- Removing existing maven 3.x installation"
  sudo rm -fr /usr/local/maven-3.*
fi
if [ -L /usr/local/maven ]; then
  echo "- Removing old maven symlink"
  sudo rm /usr/local/maven
fi

echo "- Downloading maven ${VERSION}"
curl -O http://apache.mirrors.tds.net/maven/maven-3/$VERSION/binaries/apache-maven-$VERSION-bin.tar.gz 2>/dev/null
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to download maven"
  exit $retval
fi

echo "- Extracting maven ${VERSION}"
tar -zxf apache-maven-$VERSION-bin.tar.gz > /dev/null
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to extract maven"
  exit $retval
fi

echo "- Moving maven ${VERSION} to /usr/local/maven-${VERSION}"
sudo mv apache-maven-$VERSION /usr/local/maven-$VERSION
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to extract maven"
  exit $retval
fi

echo "- Symlinking /usr/local/maven-${VERSION} /usr/local/maven"
sudo ln -s /usr/local/maven-$VERSION /usr/local/maven
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to extract maven"
  exit $retval
fi

echo "- Updating alternatives for maven"
sudo update-alternatives --install /usr/bin/mvn mvn /usr/local/maven-$VERSION/bin/mvn 1
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to update package alternatives"
  exit $retval
fi

echo "- Maven ${VERSION} successfully upgraded!"

The script is pretty self-explanatory, but feel free to ping us with questions on it.

cache_deps

We wrote this script to cache a directory of dependencies to S3 and retrieve them for later use. It uses an md5 sha of the specified dependency file to figure out if it needs to regenerate the cache. It is loosely based on the wad.

This is untested outside of the usage in this blog post, though it should work for Python requirements as well, as the idea is the same.

This command requires s3cmd, which we installed in our before_install step.

#!/usr/bin/env bash

ARCHIVE_FOLDER_NAME=false
ARTIFACT_PREFIX=""
DEPENDENCY_FILE=false
EXTRACT_PATH=false
INSTALL_COMMAND="true"
ROOT_PATH=`pwd`
S3_ACCESS_ID=false
S3_BUCKET="shared_bucket"
S3_SECRET_KEY=false

LOG () { echo -e "[LOG] $1"; }
RUNCOMMAND () { echo -e "[CMD] $1" && eval $1; }

while getopts "a:b:d:e:f:h:i:p:r:s:" opt
do
  case $opt
  in
    a)
      S3_ACCESS_ID=$OPTARG
      ;;
    b)
      S3_BUCKET=$OPTARG
      ;;
    d)
      DEPENDENCY_FILE=$OPTARG
      ;;
    e)
      EXTRACT_PATH=$OPTARG
      ;;
    f)
      ARCHIVE_FOLDER_NAME=$OPTARG
      ;;
    h)
      echo "cache_deps"
      echo ""
      echo "usage:"
      echo "  -a S3_ACCESS_ID        - S3 Access ID"
      echo "  -b S3_BUCKET           - S3 Bucket name"
      echo "  -B ARCHIVE_BASE        - Base path to where the folder being archived"
      echo "  -d DEPENDENCY_FILE     - File that manages dependencies"
      echo "  -e EXTRACT_PATH        - Path to extract dependencies into"
      echo "  -f ARCHIVE_FOLDER_NAME - Name of folder to archive"
      echo "  -h                     - This help screen"
      echo "  -i INSTALL_COMMAND     - Command to run to install dependencies"
      echo "  -p ARTIFACT_PREFIX     - Prefix to use for artifact uploads"
      echo "  -s S3_SECRET_KEY       - S3 secret key"
      exit 0
      ;;
    i)
      INSTALL_COMMAND="$OPTARG"
      ;;
    p)
      ARTIFACT_PREFIX=$OPTARG
      ;;
    r)
      ROOT_PATH=$OPTARG
      ;;
    s)
      S3_SECRET_KEY=$OPTARG
      ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      exit 1
      ;;
  esac
done

ROOT_PATH=${ROOT_PATH%/}
EXTRACT_PATH=${EXTRACT_PATH%/}

unamestr=`uname`
if [[ "$unamestr" == 'Linux' ]]; then
   ARCHIVE_NAME=$(md5sum $DEPENDENCY_FILE | awk '{ print $1 }')
else
   ARCHIVE_NAME=$(md5 $DEPENDENCY_FILE | awk '{ print $4 }')
fi
GZIP_FILENAME="${ARCHIVE_NAME}.tar.gz"
GZIP_FILEDIR="${ROOT_PATH}/tmp"
GZIP_FILEPATH="${GZIP_FILEDIR}/${GZIP_FILENAME}"
S3_PATH="${ARTIFACT_PREFIX}/${GZIP_FILENAME}"

setup () {
  if which s3cmd >/dev/null; then echo ""; else
    LOG "Missing s3cmd in PATH"
    exit 1
  fi

  RUNCOMMAND "ensure_config"
  RUNCOMMAND "get_archive"

  if [ $? -eq 0 ]; then
    LOG "Archive installed"
  else
    LOG "Archive not available on S3"
    RUNCOMMAND "install_dependencies"
    installed=$?

    if [ $installed -eq 0 ]; then
      RUNCOMMAND "put_archive"
    else
      echo "Failed properly fetch or install archive. Please review the logs."
      exit 1
    fi
  fi

  return $?
}

get_archive () {
  RUNCOMMAND "s3_read"
  if [ $? -eq 0 ]; then
    LOG "S3 Read succeeded, extracting archive to ${EXTRACT_PATH}"
    RUNCOMMAND "tar -xzf ${GZIP_FILEPATH} -C ${EXTRACT_PATH}"
    return $?
  fi

  return 1
}

install_dependencies () {
  LOG "Installing dependencies"
  RUNCOMMAND $INSTALL_COMMAND
  return $?
}

put_archive () {
  RUNCOMMAND "zip_archive"
  if [ $? -eq 0 ]; then
    RUNCOMMAND "s3_write"
  fi

  return $?
}

s3_read () {
  if [ -f $GZIP_FILEPATH ]; then
    LOG "Removing archive from filesystem"
    rm -rf $GZIP_FILEPATH
  fi

  LOG "Trying to fetch Wad from S3"
  RUNCOMMAND "mkdir -p $GZIP_FILEDIR"
  RUNCOMMAND "s3cmd get s3://$S3_BUCKET/$S3_PATH tmp/$GZIP_FILENAME >/dev/null"
  return $?
}

zip_archive () {
  LOG "Creating Wad with tar ($GZIP_FILEPATH)"
  RUNCOMMAND "tar -czvf $GZIP_FILEPATH -C $EXTRACT_PATH $ARCHIVE_FOLDER_NAME"
  return $?
}

s3_write () {
  LOG "Trying to write Wad to S3"
  RUNCOMMAND "s3cmd put --acl-public $GZIP_FILEPATH s3://$S3_BUCKET/$S3_PATH >/dev/null"
  if [ $? -eq 0 ]; then
    LOG "Wrote Wad to S3"
    return 0
  else
    LOG "Failed to write to S3, debug with 'wad -h'"
    return 1
  fi
}

ensure_config () {
  tee ~/.s3cfg > /dev/null <<EOF
[default]
access_key = $S3_ACCESS_ID
secret_key = $S3_SECRET_KEY
EOF
}

setup
rm -rf ~/.s3cfg

install_gems

Pretty simple script, it simple installs ruby gems:

#!/usr/bin/env bash

bundle install --path ~/.bundle --without='development production' --deployment

install_sdk

This script is a bit odd. It deals with the absolute nonsense that is the download api for the Android SDKs. I cannot guarantee this will always work - there is a hack for a case where it stopped working because installing sysimg-18 required two agreements be accepted - but it does currently work.

We use a deps.txt file format that is pretty simple to grok:

platform-tools
tools
build-tools-18.0.1
android-18
android-17
addon-google_apis-google-18
extra-android-m2repository
extra-android-support
extra-google-admob_ads_sdk
extra-google-analytics_sdk_v2
extra-google-gcm
extra-google-google_play_services
extra-google-m2repository
extra-google-play_apk_expansion
extra-google-play_billing
extra-google-play_licensing
extra-google-webdriver

No tricks there, just a text file with the SDK elements we want installed. The following is the script itself:

#!/usr/bin/env bash

LOG () { echo -e "[LOG] $1"; }
RUNCOMMAND () { echo -e "[CMD] $1" && eval $1; }

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ -z "$ANDROID_HOME" ]; then ANDROID_HOME="${PWD}/android-sdk-linux/"; fi
echo $ANDROID_HOME
exit

# Install base Android SDK
RUNCOMMAND "wget http://example.com/android-sdk_r22.0.5-linux.tgz"
RUNCOMMAND "tar xzf android-sdk_r22.0.5-linux.tgz"

# install android build tools
# - sudo apt-get install --force-yes unzip
RUNCOMMAND "wget http://EXAMPLE.COM/build-tools_r18.0.1-linux.tgz"
RUNCOMMAND "tar -xf build-tools_r18.0.1-linux.tgz -C $ANDROID_HOME"
RUNCOMMAND "mkdir -p $ANDROID_HOME/build-tools/"
RUNCOMMAND "mv $ANDROID_HOME/android-4.3 $ANDROID_HOME/build-tools/18.0.1"

# Install required Android components.
# For a full list, run `android list sdk -a --extended`
# Note that sysimg-18 downloads the ARM, x86 and MIPS images (we should optimize this).
# Other relevant API's

for dep in `cat $DIR/../deps.txt`; do
    echo "Installing $dep"
    expect <<DONE
        set timeout -1

        # install dependencies
        spawn android update sdk --filter $dep --no-ui --force --all
        match_max 1000000

        # Look for prompt
        expect "*?\[y\/n\]*"

        # Accept the prompt

        send -- "yes\r"

        # send blank line (\r) to make sure we get back to gui
        send -- "\r"

        expect eof
DONE
done

echo "Installing sysimg-18"
expect <<DONE
    set timeout -1
    # install dependencies
    spawn android update sdk --filter sysimg-18 --no-ui --force --all
    match_max 1000000

    # Look for prompt
    expect "*android-sdk-license-bcbbd656*"

    # Accept the prompt
    send -- "yes\r"

    # Look for prompt
    expect "*intel-android-sysimage-license-1ea702d1*"

    # Accept the prompt
    send -- "yes\r"

    # send blank line (\r) to make sure we get back to gui
    send -- "\r"

    expect eof
DONE

A few notes:

  • You’ll need to specify a url to the base Android SDK. This isn’t provided by us. We upload ours to S3.
  • You’ll also need to specify a url for the android build_tools. Again, we suggest using S3.
  • If you use different version of the sdk than we do, then feel free to modify this script.

Finishing up

We’ll want to skip the installation of requirements, since we more or less took care of it in the before_install. Feel free to move that to this section. We simply skipped it for simplicity:

install: true

Next, we’ll want to actually run tests. Since we use maven, performing a mvn clean package will not only create a package, but will also ensure that all tests pass before doing so:

script: mvn clean package

For those who wish to use our build-artifacts tool, you’ll want to use the travis-artifacts gem to upload your apk. Here is what we do:

after_success:
  - "cd android/target"
  - "bundle exec travis-artifacts upload --path android-1.0-SNAPSHOT-aligned.apk --target-path android-app/$TRAVIS_BUILD_NUMBER"
  - "bundle exec travis-artifacts upload --path android-1.0-SNAPSHOT-aligned.apk --target-path android-app/latest"

Note that these are production releases. Debug code is turned off, so if you want to enable that, you’ll need to adjust your maven settings.

Some closing thoughts

One advantage to our setup is that it allows developers to write code without necessarily having an Android dev installation setup. I use it for this at my home computer when quickly testing bug fixes. While it isn’t a complete solution, it does bring us to that last mile.

A few possible improvements:

  • Use bundler caching instead of a custom caching script
  • Switching to native Android SDK support should simplify the setup of an Android application, hopefully removing much trial and error.
  • Another possible success would be to use HTTP caching. Travis does offer this, but you’ll need to ask directly, and it’s likely not going to be approved for large files which you should place in your S3 ;)
  • Fixing Maven 3.1.x support will remove a bit of the complexity, though that seems unlikely to occur in the near future.
  • Build times aren’t extremely quick. On a recent Macbook Pro, we see build times of a minute or less. Travis averages ~5 minutes for us. This is therefore better used for branch integration, and potentially for creating releases for the Play Store.

We’re pretty happy with the final outcome, and while we have had to work on it every so often, the tweaks are pretty easy and usually apply to our Vagrant development environment.

Shout out to the Travis-CI folks for being so supportive as we’ve abused their systems :)

Comments