ChairNerd

Code, Design & Growth at

Open Sourcing Our Admin Panel

The first version of the SeatGeek Dev Challenge. Crack open the beers.

In a land before time, SeatGeek created an extremely hackable admin panel. Its primary purpose was to stir the curiosity of developers who might be looking for a new job. You can read more about it in this previous post.

While the response over the years to the dev challenge was great, we retired the dev challenge over a year ago. Please stop trying to hack into our backend! (If you do happen to be hacking our website, we’d appreciate a heads-up on any vulnerabilities you find at hi@seatgeek.com Responsible disclosure and whatnot.

In order to cater to the curious, I took the opportunity to open source the dev challenge. It’s a small Sinatra app that you can run locally and hack to your heart’s content.

A few notes about the panel:

  • Configuration is done using environment variables with a few sane defaults.
  • You’ll need a local SMTP server to send email notifications of new applicants. We used postfix at the time, but you can use whatever you’d like.
  • Applicant resumes are stored on disk. Yes, we know, there are probably better ways than what we did, but since the box was in a DMZ, it was probably okay. Not like we weren’t trying to have you hack us anyhow.
  • Ruby 1.9.3 is what we used to deploy—actually 1.9.1 at the time, but it works with 1.9.3—but no guarantees that it will work with a newer Ruby. Pull requests welcome!

We’d like to thank all the thousands of developers who have hacked our backend over the years. Don’t worry, we’ll have a new challenge soon.

In the meantime, we’re still hiring engineers.

Spatial Data Structures for Better Map Interactions

Last week we launched a feature that even the most die-hard SeatGeek fans probably didn’t notice. However, we think that this feature makes a huge difference in usability and overall user experience, even if only at a subconscious level. You can now interact with the section and row shapes themselves, rather than just section/row markers.

For anyone not familar with our maps, here is an example of what one looks like:

Map example

Each of those markers represents a ticket or a group of tickets in that section. Until recently, all of the map interactions revolved around those markers. In order to find out more about the tickets in a section or row, the user would have to hover or click on the marker itself.

Fitts’s Law

One major concept in human-computer interaction is Fitts’s Law. Fitts’s law models the time it takes for a user to move a pointing device (e.g. cursor) over an object. In order to decrease the time to select an object, one can do one of two things: decrease the distance between the cursor and the object, or increase the size of the object. On SeatGeek’s maps we are constrained by the layout of venues, so our only option is to increase the marker size.

The natural way to increase the target area of a marker is to expand it to the shape of its section. However, it turns out this isn’t straightforward.

How Our Map Pages Work

First, a little background on how our map pages work. We use an excellent library, Leaflet, as the foundation for our interactive maps. The maps themselves start out as vector files. These are then rasterized into a bunch of tiles, such as this:

Example of a tile

Leaflet handles the logic for mouse interactions and displaying the correct tiles depending on the current view. The markers are a custom Leaflet layer (for performance reasons, but that is a whole other blog post). Then, we overlay a vector path as an SVG layer when a marker is either hovered or clicked.

SVG Highlighting

First Attempt at Section Interaction

A while back, when we first implemented our current style of maps, we considered adding polygon interaction instead of just using the markers. Given that we had the SVG paths of all of the shapes for the highlighting purposes, we decided to add all of these SVG elements to the map so that we could use the built-in event handling that browsers provide.

Unfortunately, that resulted in terrible performance on the map page. At the row level, we can have as many as a few thousand SVG elements being drawn at the same time. Combine that with all the markers we have to draw, and the map grinds to a halt. We decided to shelve the section interaction and move on to other features.

A Renewed Attempt

With the launch of our new map design, called the Omnibox, click and hover interactions became much more central to the interface.

The breakthrough was realizing that we could implement our own logic for hit-testing, or checking if a given mouse position is inside of a polygon. This means we didn’t have to add any additional elements to the DOM (like we did before with the SVG elements).

The naive approach would be to iterate through every polygon and check if the mouse is inside it using the ray casting algorithm.

However, we can do even better. By using a spatial data structure, such as an R-Tree, we can reduce the lookup complexity from linear to logarithmic. Below is an example of a 2D R-Tree. Imagine that the lowest level of the tree contains references to the actual polygons. Each node in the tree represents a bounding box that is the union of all the children below it in the tree.

R-Tree Example

Luckily enough, we were able to find a leaflet-compatible implementation of an R-Tree by Calvin Metcalf. Basically our use of it looks like this:

  1. On page load, convert all of the SVG shapes to geoJSON and insert into the R-Tree.
  2. On mouse events, transform the mouse position to the map coordinate system and make a throttled call to our intersection testing function.
  3. Call a search on the R-Tree with the transformed position of the mouse.
  4. The R-Tree will return a list of all the leaves (shapes) whose bounding boxes had been intersected.
  5. Iterate through the shapes and perform the ray casting algorithm to check for intersection.
  6. Return the intersected shape.

Results

The user can now hover over any point in the polygon, instead of just the marker! It works on sections:

And it works on rows:

Most importantly, all of this works without any impact on performance.

Event Pages Through the Ages

If you think about SeatGeek like we do, then in your head you probably picture an event page. You know, the page that has the venue map, the colorful dots, and the big list of tickets from all over the web, ranked by Deal Score. All the best reasons to know & love SeatGeek are encapsulated in this one single page. And it is not only the functional core of SeatGeek, it’s also our most highly-trafficked page type.

With so much riding on the event page, we’re constantly working on incremental and under-the-hood improvements. We normally avoid committing obvious, disruptive changes, but a few times in SeatGeek’s history we’ve launched major redesigns of our event page—the most recent of which happened earlier today.

Here I’ll give an overview of the latest changes and, for posterity, a quick tour through earlier SeatGeek event page history.

Today’s release

Inspiration

In the year and a half since we launched the last major version of the event page we started making mobile apps. Designing for mobile devices forced us to reconsider the SeatGeek experience from scratch, and once we launched our apps—in particular our iPad app—they became new sources of inspiration for the website. For example, we began to think much harder about conservation of screen real estate.

Internally, today’s milestone inherited the name “Omnibox” from an eponymous Google feature. Not Chrome’s address bar, but rather a more obscure reference to a CSS class found in the new Google Maps’ control panel. Although many people have griped about Google Maps’ recent update, we admired the idea of having a single page element whose content could change based on interactions with the underlying map.

What changed

In the main sidebar, we swapped our large, color-filled section headers and deal score labels for more elegant circles and lines that more closely resemble our latest iOS designs. We also moved the filter controls and box office link from the top of the sidebar to the bottom. The result is that ticket listings get higher precendence on the page.

Old sidebar vs. new sidebar, side by side

The new version of the section info view (below, on the right) looks very similar to the old, with the notable exception that it doesn’t appear in a popover overlaid on the map, but rather in the sidebar. Popovers had a lot of annoying characteristics, not least of which was that they were starting to feel decidedly web-1.0-y. As an added bonus, under the new sidebar scheme, it’s now possible to apply search filters to tickets within individual sections.

Section Info view, side by side

If you can believe it, the old version of the ticket info view (below, on the left) was actually a second popover that appeared beside the first popover containing section info. Now that all this information is in the sidebar, the map won’t get cluttered (which was especially problematic on smaller viewports), and the ticket details are much more legible.

Ticket Info view, side by side

Last but not least, we moved the old event info bar (seen in the top half of the image below) into the site header. This frees up more space for the venue map. In order to make room for event info in the new site header, we consolidated the category page links (i.e. “MLB”, “NFL”, etc.) into a dropdown off the main SeatGeek logo.

Event info

To really get a feel for the wonders of the new event page, you should really check it out. Go try a search, or check out one of these random event pages.

Earlier event pages

Here we take a walk down memory lane, through the annals of SeatGeek event page history. We’ll begin with the very first 2009-era event page before venue maps even existed and end on today’s latest event page release, ten notable iterations later.

Full disclosure: there’s a gratuitous, self-indulgent number of screenshots ahead. Only the most obsessive SeatGeek fans need read any further.

#1 The original SeatGeek event page was launched—along with SeatGeek itself—in September 2009. It contained no venue maps. SeatGeek was all about price forecasting, and making recommendations about whether to buy a ticket now, or wait to buy. If you wanted to buy, there was a table of tickets from various sellers, sorted by price.

1

#2 In early 2010, SeatGeek licensed the rights to use venue maps, provided by a third party named SeatQuest (now defunct). According to engineer #1, working with SeatQuest maps was reportedly a nightmare.

2

#3 Before long, ticket listings and venue maps started stealing screen real estate away from the price forecast part of the page.

3

#4 The event page’s first major redesign happened in Summer 2010.

4

#5 Soon after the Summer 2010 redesign, we scrapped SeatQuest in favor of our own venue maps, which should look a lot more familiar to current SeatGeek users. Also worth pointing out that by now the price forecast feature is relegated to a small-ish button area above the map, and restricted to signed-in users.

5

#6 Sometime in early 2011, we made the long-standing switch from a lefty sidebar to a righty—a change that would persist all the way until yesterday.

6

#7 In mid/late 2011, we redesigned the site again. Note the dark blue primary colors, and the new styling for the sidebar.

7

#8 In the first half of 2012, the dark blue from the previous version softened into the lighter SeatGeek blue of today.

8

#9 This update featured some new sidebar styling and abolished the permanently overlaid search bar in favor of a more compact event info bar. This version reigned supreme from Fall 2012 all the way until March 12, 2014.

9

#10 Omnibox: The cleanest SG event page yet. (Note the lefty sidebar, a clear throwback to the year 2010.)

10

Celebrating Valentine’s Day Late With Some iPad Love

Back in December, we released an epic update to our iOS app that featured tons of big additions and improvements for iPhones. But this time, it’s all about the iPad.*

This is much more than just a quick catch-up release. We took stock of all the additions from v2.0 for iPhone, and thought holistically about what the ideal experience for each one would be like on a tablet. So, iPad users, sorry about the wait. But we think you’ll find that it was well worth it.

* Well, technically it’s not all about the iPad. We also made a few UI tweaks to the iPhone app, and added a “Recent Search” list to both apps so you don’t have to keep typing the same things over and over again. Anyway, the rest of this blog post is focused on big, new iPad features.

Demo time

Like gifs? Great! Here’s one:

iPad demo gif

If that was too fast for you, we’ve also got a few higher-res screenshots that you can inspect more closely down below. But to really get the best feel for the new app, we’d strongly suggest installing it now →

Logging In

Now that you can log in (you couldn’t before), we can remember things for you – like artists, teams, and events that you’re tracking. Later, when you’re logged in to SeatGeek somewhere else like on your laptop or your iPhone, you’ll have access to all the same preferences.

You can log in with a SeatGeek account or a Facebook account. (One nice thing about Facebook accounts is that we can help you automatically track performers for you if you’ve already liked them on FB.)

Log-in modal screenshot

Tracking

As noted above, you can now track artists, teams, and events right in the app. Tracked items are easily accessible in your new My Events and My Performers screens. We’ll even send you reminders when new shows are announced or when a tracked event is approaching. Plus, now that we know a bit more about what you like, our event recommendations for you will get better and better.

Here’s what the My Performers screen looks like:

My Performers screen

To track an event, just tap the heart in the corner of any event screen:

Event view with heart

What! JT’s coming to town?!

Push notification screenshot

Redesigned Home Screen

We thought to ourselves why bother making an update if you can’t tell the difference right away? So we also added some new hotness to the home screen:

New home screen

Tapping the Explore area near the bottom lets you browse all upcoming events near you:

New explore view

Go get it

Now that you know what’s in store for you, all that’s left is to hit the App Store to make sure you’re running the latest version.

As always, we hope you like it and will let us know if you find bugs or have any cool ideas.

It’s Business Time

A simple python utility for calculating business time aware timedeltas between two datetimes

Earlier this week we released businesstime, a simple python library for doing math with datetimes that’s aware of weekends, holidays and business hours. We’re using it to more accurately calculate shipping costs for tickets listed on SeatGeek.

Here’s a quick demo of how it works:

>>> datetime(2013, 12, 26, 5) - datetime(2013, 12, 23, 12)
datetime.timedelta(2, 61200)
>>> bt = businesstime.BusinessTime(holidays=businesstime.USFederalHolidays())
>>> bt.businesstimedelta(datetime(2013, 12, 23, 12), datetime(2013, 12, 26, 5))
datetime.timedelta(1, 18000)

You can grab it on pypi or checkout the code on github. We hope you find it useful!

SeatGeek for Android 1.2, Ready for Update

Earlier this week we shipped version 1.2 of our now two-month-old Android app. There were a few main focuses for this release:

Tablet support. Whereas the previous version was built for smartphones only, the new release extends support to all the popular 7- and 10-inch Android tablets. (This one’s for you, Nexus 7 and Galaxy Tab owners.)

Landscape support. When the app was smartphone-only we could get away with only building for portrait, but tablet users sure do love to hold their devices sideways. So you’ll find that 1.2 supports landscape mode – for both smartphones and tablets.

Landcape mode on a 10-inch tablet

Performance & stability. Since version 1.0 launched, we’ve kept a close eye on app reviews and error logs, tweaked the internals, squashed some bugs, and cured as many crashes as we could diagnose. The result is that the latest version should run fast and smooth.

We’ve also added support for deep-linking. If you’re a developer or an API partner looking to refer app users to SeatGeek, you’ll find the relevant docs at platform.seatgeek.com.

And, last but not least, we’ve stepped up our game with a new, more Android-esque app icon:

New SeatGeek for Android app icon

If you haven’t already, hit the Play Store now to get yourself a fresh copy of the app. And be sure to let us know what you think.

Cross-Event Search

Cross-Event Search header

Today we’re excited to announce the public launch of Cross-Event Search, a new feature that helps users find the best tickets across multiple events at once.

Suppose you live in NYC and want to see what all the Book of Mormon hype is about. You want to go to whatever upcoming show is cheapest. With Cross-Event Search you can see the listings for all future performances (over 400 events!) in a single UI. Here’s the live page for that search.

The ability to search for & compare tickets for more than one event at once is not only a SeatGeek first, but also – to our knowledge – a ticketing first.

When to use it

Cross-Event Search comes in handy when you have more than one chance to see an event and – rather than pick solely by date – you prefer to choose based on which event has the lowest price, the closest seats, or the best deal.

The basic formula is: one performer + one venue + multiple events.

In addition to the Book of Mormon example above, here are a few more common use cases:

  • Home games: The Giants’ season is underway and you want to take advantage of some nice San Francisco weather and a 9-game home stand with a group of friends. You don’t care who they’re playing — you just want to get into the park and have plenty of money left over for beer & hotdogs. This page compares all Giants’ home games in the first two weeks in June.
  • Back-to-back concerts: Jay-Z has graced your local arena with multiple nights of concerts. Tickets originally sold out in minutes, but you’re sure there must be at least one show on SeatGeek with a great deal.

How to use it

To start Cross-Event Searching, visit any performer page on SeatGeek and click the Compare Events button. (This will work for any major sports team, music artist, theater show, etc.)

Step 1

Check the boxes next to the events that you want to include in your ticket search, and click the green Compare N Events button. (You’ll also be able to change your selections on the next page.)

Step 2

Next you’ll be taken to a regular-looking SeatGeek event page, where the venue map & sidebar will be populated with ticket listings from all your selected events.

Deal Scores for each event are indexed to the same scale, so you can browse for tickets just as you would normally. In other words, an 89 on one night is still a better deal than a 79 on another night. (See also: Absolute Deal Score.)

Step 3

To adjust which events are included in your search, click the Comparing tickets for… button at the top of your event page and check or uncheck boxes next to a few more events.

Step 4

Finally live

As many patient SG feature-requesters can attest, Cross-Event Search has been a long time coming. Historically, our underlying software was not structured to support queries like this. But that changed in June 2013 with a first place-winning SeatGeek hackathon project named “Sandcrab,” which paved the way for Cross-Event Search.

We’re pumped to release this feature into the wild. As with everything we build, please, please hit us up with feedback.

FanSnap Is Now Part of SeatGeek

We’re thrilled to announce that we have acquired FanSnap from its parent company, Wize Commerce, and have integrated it into SeatGeek.com.

A Bit of History

When we started SeatGeek in 2009, FanSnap’s incumbency cast quite a shadow on us. They were two years old at that point, with a formidable team, an impressive set of advisors, and more than $15 million in venture financing. We had an idea and a couple of laptops.

Then, as now, we had great respect for the FanSnap team and what they built. FanSnap helped make ticket search a thing. They introduced users to the benefit of seeing multiple ticket sellers on the same screen, showing just how often the secondary market can have the best deal for any game or show.

In the years since, we assembled an incredible team and built what we hope you’ll agree is the best event and ticket site on the web. When we were given the opportunity to purchase FanSnap—a company with a nearly identical mission to ours—we jumped at the opportunity.

For FanSnap Users: The Transition

We’ll be sunsetting the FanSnap website and redirecting all FanSnap pages to analogous pages on SeatGeek. For example, www.fansnap.com/phish-tickets now directs to www.seatgeek.com/phish-tickets. So any bookmarks you have will work automatically, and any FanSnap links you find floating around the web will seamlessly take you to the correct page on SeatGeek. Merging FanSnap into SeatGeek lets us remain laser-focused on giving everyone the best experience on a single website rather than splitting our efforts across two sites.

We think loyal FanSnap users formerly unacquainted with SeatGeek will be pleasantly surprised by their new home. All the tickets available on FanSnap (and a few that aren’t) are also available here. Plus, we’ve got some extra stuff for you:

  • See exactly where you’ll sit with our beautiful interactive maps for over 3,000 venues
  • Get the best bang-for-your-buck with Deal Score™, our rating system that tells you which tickets give you the best experience for the least money
  • Find out when your favorite bands announce shows by tracking performers on SeatGeek
  • Get event recommendations tailored to your personal tastes
  • Search for tickets on your phone with our native iOS and Android apps

For SeatGeek Users

If you are a SeatGeek user, you will have the same experience today that you had yesterday. As time goes on, we’ll be able to use FanSnap’s data and technology to improve the experience for everyone. In the meantime, you won’t notice any changes.

What’s Next

Lest you worry, nobody over here is resting on laurels. We’ve got some huge stuff in store for you in 2014, all aimed at making it simpler, cheaper, and more fun to discover events and buy tickets. Stay tuned.

SeatGeek for iOS v2.0

Just in time for the winter holidays, SeatGeek for iPhone & iPad has undergone a complete redesign.

Our main goal for this release was to bring more of the awesome features found in the SeatGeek web app to the iPhone. This is the app we’ve wanted to build since day one, and now that it’s here we couldn’t be happier. Here are some of the best new features:

  • Facebook and SeatGeek login*
  • Event and Artist tracking*
  • Push notifications
  • Updated explore and homescreen

* iPhone only. iPad, you’re next.

Let’s see it in action

Accounts

The first thing you’ll notice when you launch the new app is a prompt to log in. You can choose to log in via Facebook, or with your regular SeatGeek account. If you don’t log in you can’t use any of our awesome new features, so you should probably do this one.

SeatGeek Android ticket view screenshot

Track Teams & Bands

Do you find yourself constantly refreshing Justin Bieber’s tour page waiting for him to add more dates? Give your keyboard a rest, we’ve got you covered. Let us know who your favorite teams and artists are and we’ll let you know when they’re coming to town. Just press the heart icon on any artist’s screen and they will be added to your list of tracked performers.

SeatGeek Android ticket view screenshot

Track Events

Are those tickets to next Sunday’s Monster Truck Rally just a little too expensive? Tap the heart icon on any event and we’ll add it to your list of tracked events. It’s an easy way to keep tabs on prices to everything you want to see. We’ll even remind you when the event is coming up so you don’t forget to snag tickets.

SeatGeek Android ticket view screenshot

Push Notifications

Whenever one of your tracked performers announces new shows in your area we’ll let you know. You can forget the pain of hearing you missed one of DJ Corn Meal’s sets because you had no idea he was playing (trust us it hurts). We’ll give you a heads up a few days before any events you’re tracking too, just so you don’t forget.

SeatGeek Android ticket view screenshot

Redesigned Homescreen

The new Explore view lives right in the homescreen. If you’re bored, or looking for some mid-week date inspiration, just swipe up on the homescreen and check out all the awesome live events coming up in your area.

SeatGeek Android ticket view screenshot

Zones

Admittedly not the sexiest of features, but still pretty significant. Sometimes, tickets to an event like the Super Bowl aren’t sold as exact seats but are sold in general areas of the venue called zones. Previously we had no way to display that information, but now our maps can handle it. If you’re looking for tickets to the big game, you can use our maps to get a good idea of where your seats will be.

SeatGeek Android ticket view screenshot

A very mobile year!

It’s been a wild 12 months for SeatGeek in the mobile world

  • December 2012: Launched our first native mobile app on the iPhone
  • July 2013: Released SeatGeek on the iPad too
  • September 2013: Released our redesigned app for iOS 7
  • November 2013: Launched SeatGeek for Android
  • December 2013: Released SeatGeek for iPhone 2.0
  • Up next: User accounts and tracking on Android & iPad

Continuous Integration for Android Applications

In our quest to make everything as automated as possible, the SeatGeek dev team has been using Travis-CI to build our Android application apks automatically. Since the process of doing Android-based CI is a bit difficult to get correct without trial and error, we’ll document it here for everyone else to use!

A few caveats

Travis-CI does not support Java 6 - Oracle JDK 7 and 8 are available, as is the OpenJDK. We encountered an issue related to jar-signing, which was easily fixed by adding the following to our app’s pom.xml file:

<!-- Fix for jar signing with java 1.7
http://stackoverflow.com/questions/8738962/what-kind-of-pitfals-exist-for-the-android-apk-signing/9567153#9567153 -->
<configuration>
   <arguments>
       <argument>-sigalg</argument><argument>MD5withRSA</argument>
       <argument>-digestalg</argument><argument>SHA1</argument>
   </arguments>
</configuration>

Travis sometimes upgrades their infrastructure. You may need to install packages from source to get everything working. The only time this has bitten us is with the Maven version, which we’ll discuss below.

Travis currently does not have the android SDKs by default, so you need to manage this yourself. We ended up building a script to cache them, though there is an outstanding PR that may fix the issue, so be on the lookout.

Initial Setup

Since we upload APKs to S3, we’ve included a few gems in our Gemfile:

source 'https://rubygems.org'

gem 'travis', '1.5.4'
gem 'travis-artifacts', '~>0.1'

In our .travis.yml, we’ll want to specify the java build-type and the jdk version:

language: java
jdk: oraclejdk7

Next, you’ll want to specify some environment variables for the ANDROID_TARGET and ANDROID_ABI. SeatGeek uses a single matrix entry, though you are welcome to build for multiple targets:

env:
  matrix:
    - ANDROID_TARGET=android-18  ANDROID_ABI=armeabi-v7a

We also upload the build apk to our S3 for exposure using our build-artifacts application, so we have some S3-related environment variables. Note that we encrypt our S3 access_key_id and secret_access_key, which is something I highly recommend doing:

env:
  global:
    - "ARTIFACTS_AWS_REGION=us-east-1"
    - "ARTIFACTS_S3_BUCKET=shared_bucket"
    # travis-artifacts ARTIFACTS_AWS_ACCESS_KEY_ID
    - secure: "SOME_SECURE_ACCESS_KEY_ID"
    # travis-artifacts ARTIFACTS_AWS_SECRET_ACCESS_KEY
    - secure: "SOME_SECURE_SECRET_ACCESS_KEY"

I recommend enabling the apt cache. We need to install some system packages in order to run the android build tools, so this will cut down some time from your build. In our testing, it was ~4 minutes.

cache:
  - apt

If you have setup the build-artifacts tool, then you’ll need a notification entry for the app:

notifications:
  webhooks:
    urls:
      # build-artifacts app
      - http://ancient-foot-stomps-alligator.herokuapp.com/travisci/

before_install: The Meat and Potatoes

Next is the before_install step. This is the meat of our setup, and was the result of quite a bit of trial and effort. I’ll be candid and state it took ~100 builds to figure out everything. The android devs were confused about whether the app actually worked, to say the least.

before_install:
  - sudo apt-get install -qq --force-yes expect libgd2-xpm ia32-libs ia32-libs-multiarch s3cmd > /dev/null

  - export ANDROID_HOME=${HOME}/android-sdk-linux/
  - export ANDROID_APP_DIR=${PWD}
  - export PATH=${PATH}:${ANDROID_HOME}/tools:${ANDROID_HOME}/platform-tools:${HOME}/.bundle/ruby/1.9.1/bin
  - export GEM_PATH=${HOME}/.bundle/ruby/1.9.1:${GEM_PATH}

  # Install 3.0.5 maven
  - bin/install_maven

  # Install Ruby requirements
  - bin/cache_deps -a $ARTIFACTS_AWS_ACCESS_KEY_ID -b $ARTIFACTS_S3_BUCKET -d 'Gemfile.lock' -e $HOME -f ".bundle" -i "bin/install_gems" -p "android/bundles" -s $ARTIFACTS_AWS_SECRET_ACCESS_KEY

  # Install Android requirements
  - bin/cache_deps -a $ARTIFACTS_AWS_ACCESS_KEY_ID -b $ARTIFACTS_S3_BUCKET -d 'deps.txt' -e $HOME -f "android-sdk-linux" -i "./bin/install_sdk" -p "android/deps" -s $ARTIFACTS_AWS_SECRET_ACCESS_KEY

I’ll explain this step-by-step.

  1. We need to install some system packages for the android dev tools. They won’t run without these, unfortunately, and the packages aren’t included by default in TravisCI.
  2. We set some android-related environment variables to ensure the app can be properly built. The PATH and GEM_PATH are overriden because we want to specify where the build tools live, as well as use a custom path for the bundle requirements.
  3. If you use maven in any way, then your best bet is to force the installation of maven 3.0.x. This ticket explains it in more detail than I can in a few words.
  4. At the time I originally implemented this, there was no bundler caching. There is bundler caching built into Travis-CI now, but it is untested in our setup. Probably works, so worth trying!
  5. We cache all android sdk requirements using a custom dependency file which I’ll outline below. Trying to hit the Android repositories for the SDK is error-prone, and you’ll eventually be throttled, which balloons a 4-minute build to a 45-minute build.

Helper scripts

You’ll notice we have two scripts in use, cache_deps, install_maven, install_gems, and install_sdk. They are outlined below:

install_maven

The install_maven command was actually created today - we realized recent updates in TravisCI broke all of our builds, and the change that broke things was upgrading to Maven 3.1.1. Thankfully the awesome folks at Travis lent me a built-vm to futz with and I was able to fix the issue. This can’t be easily fixed on their end without requiring everyone use an older version of Maven, which is undesirable, so this script is available for all to use!

#!/usr/bin/env bash

VERSION=3.0.5

if [ -d /usr/local/maven-3.* ]; then
  echo "- Removing existing maven 3.x installation"
  sudo rm -fr /usr/local/maven-3.*
fi
if [ -L /usr/local/maven ]; then
  echo "- Removing old maven symlink"
  sudo rm /usr/local/maven
fi

echo "- Downloading maven ${VERSION}"
curl -O http://apache.mirrors.tds.net/maven/maven-3/$VERSION/binaries/apache-maven-$VERSION-bin.tar.gz 2>/dev/null
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to download maven"
  exit $retval
fi

echo "- Extracting maven ${VERSION}"
tar -zxf apache-maven-$VERSION-bin.tar.gz > /dev/null
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to extract maven"
  exit $retval
fi

echo "- Moving maven ${VERSION} to /usr/local/maven-${VERSION}"
sudo mv apache-maven-$VERSION /usr/local/maven-$VERSION
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to extract maven"
  exit $retval
fi

echo "- Symlinking /usr/local/maven-${VERSION} /usr/local/maven"
sudo ln -s /usr/local/maven-$VERSION /usr/local/maven
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to extract maven"
  exit $retval
fi

echo "- Updating alternatives for maven"
sudo update-alternatives --install /usr/bin/mvn mvn /usr/local/maven-$VERSION/bin/mvn 1
retval=$?
if [ $retval -ne 0 ]; then
  echo "- Failed to update package alternatives"
  exit $retval
fi

echo "- Maven ${VERSION} successfully upgraded!"

The script is pretty self-explanatory, but feel free to ping us with questions on it.

cache_deps

We wrote this script to cache a directory of dependencies to S3 and retrieve them for later use. It uses an md5 sha of the specified dependency file to figure out if it needs to regenerate the cache. It is loosely based on the wad.

This is untested outside of the usage in this blog post, though it should work for Python requirements as well, as the idea is the same.

This command requires s3cmd, which we installed in our before_install step.

#!/usr/bin/env bash

ARCHIVE_FOLDER_NAME=false
ARTIFACT_PREFIX=""
DEPENDENCY_FILE=false
EXTRACT_PATH=false
INSTALL_COMMAND="true"
ROOT_PATH=`pwd`
S3_ACCESS_ID=false
S3_BUCKET="shared_bucket"
S3_SECRET_KEY=false

LOG () { echo -e "[LOG] $1"; }
RUNCOMMAND () { echo -e "[CMD] $1" && eval $1; }

while getopts "a:b:d:e:f:h:i:p:r:s:" opt
do
  case $opt
  in
    a)
      S3_ACCESS_ID=$OPTARG
      ;;
    b)
      S3_BUCKET=$OPTARG
      ;;
    d)
      DEPENDENCY_FILE=$OPTARG
      ;;
    e)
      EXTRACT_PATH=$OPTARG
      ;;
    f)
      ARCHIVE_FOLDER_NAME=$OPTARG
      ;;
    h)
      echo "cache_deps"
      echo ""
      echo "usage:"
      echo "  -a S3_ACCESS_ID        - S3 Access ID"
      echo "  -b S3_BUCKET           - S3 Bucket name"
      echo "  -B ARCHIVE_BASE        - Base path to where the folder being archived"
      echo "  -d DEPENDENCY_FILE     - File that manages dependencies"
      echo "  -e EXTRACT_PATH        - Path to extract dependencies into"
      echo "  -f ARCHIVE_FOLDER_NAME - Name of folder to archive"
      echo "  -h                     - This help screen"
      echo "  -i INSTALL_COMMAND     - Command to run to install dependencies"
      echo "  -p ARTIFACT_PREFIX     - Prefix to use for artifact uploads"
      echo "  -s S3_SECRET_KEY       - S3 secret key"
      exit 0
      ;;
    i)
      INSTALL_COMMAND="$OPTARG"
      ;;
    p)
      ARTIFACT_PREFIX=$OPTARG
      ;;
    r)
      ROOT_PATH=$OPTARG
      ;;
    s)
      S3_SECRET_KEY=$OPTARG
      ;;
    \?)
      echo "Invalid option: -$OPTARG" >&2
      exit 1
      ;;
  esac
done

ROOT_PATH=${ROOT_PATH%/}
EXTRACT_PATH=${EXTRACT_PATH%/}

unamestr=`uname`
if [[ "$unamestr" == 'Linux' ]]; then
   ARCHIVE_NAME=$(md5sum $DEPENDENCY_FILE | awk '{ print $1 }')
else
   ARCHIVE_NAME=$(md5 $DEPENDENCY_FILE | awk '{ print $4 }')
fi
GZIP_FILENAME="${ARCHIVE_NAME}.tar.gz"
GZIP_FILEDIR="${ROOT_PATH}/tmp"
GZIP_FILEPATH="${GZIP_FILEDIR}/${GZIP_FILENAME}"
S3_PATH="${ARTIFACT_PREFIX}/${GZIP_FILENAME}"

setup () {
  if which s3cmd >/dev/null; then echo ""; else
    LOG "Missing s3cmd in PATH"
    exit 1
  fi

  RUNCOMMAND "ensure_config"
  RUNCOMMAND "get_archive"

  if [ $? -eq 0 ]; then
    LOG "Archive installed"
  else
    LOG "Archive not available on S3"
    RUNCOMMAND "install_dependencies"
    installed=$?

    if [ $installed -eq 0 ]; then
      RUNCOMMAND "put_archive"
    else
      echo "Failed properly fetch or install archive. Please review the logs."
      exit 1
    fi
  fi

  return $?
}

get_archive () {
  RUNCOMMAND "s3_read"
  if [ $? -eq 0 ]; then
    LOG "S3 Read succeeded, extracting archive to ${EXTRACT_PATH}"
    RUNCOMMAND "tar -xzf ${GZIP_FILEPATH} -C ${EXTRACT_PATH}"
    return $?
  fi

  return 1
}

install_dependencies () {
  LOG "Installing dependencies"
  RUNCOMMAND $INSTALL_COMMAND
  return $?
}

put_archive () {
  RUNCOMMAND "zip_archive"
  if [ $? -eq 0 ]; then
    RUNCOMMAND "s3_write"
  fi

  return $?
}

s3_read () {
  if [ -f $GZIP_FILEPATH ]; then
    LOG "Removing archive from filesystem"
    rm -rf $GZIP_FILEPATH
  fi

  LOG "Trying to fetch Wad from S3"
  RUNCOMMAND "mkdir -p $GZIP_FILEDIR"
  RUNCOMMAND "s3cmd get s3://$S3_BUCKET/$S3_PATH tmp/$GZIP_FILENAME >/dev/null"
  return $?
}

zip_archive () {
  LOG "Creating Wad with tar ($GZIP_FILEPATH)"
  RUNCOMMAND "tar -czvf $GZIP_FILEPATH -C $EXTRACT_PATH $ARCHIVE_FOLDER_NAME"
  return $?
}

s3_write () {
  LOG "Trying to write Wad to S3"
  RUNCOMMAND "s3cmd put --acl-public $GZIP_FILEPATH s3://$S3_BUCKET/$S3_PATH >/dev/null"
  if [ $? -eq 0 ]; then
    LOG "Wrote Wad to S3"
    return 0
  else
    LOG "Failed to write to S3, debug with 'wad -h'"
    return 1
  fi
}

ensure_config () {
  tee ~/.s3cfg > /dev/null <<EOF
[default]
access_key = $S3_ACCESS_ID
secret_key = $S3_SECRET_KEY
EOF
}

setup
rm -rf ~/.s3cfg

install_gems

Pretty simple script, it simple installs ruby gems:

#!/usr/bin/env bash

bundle install --path ~/.bundle --without='development production' --deployment

install_sdk

This script is a bit odd. It deals with the absolute nonsense that is the download api for the Android SDKs. I cannot guarantee this will always work - there is a hack for a case where it stopped working because installing sysimg-18 required two agreements be accepted - but it does currently work.

We use a deps.txt file format that is pretty simple to grok:

platform-tools
tools
build-tools-18.0.1
android-18
android-17
addon-google_apis-google-18
extra-android-m2repository
extra-android-support
extra-google-admob_ads_sdk
extra-google-analytics_sdk_v2
extra-google-gcm
extra-google-google_play_services
extra-google-m2repository
extra-google-play_apk_expansion
extra-google-play_billing
extra-google-play_licensing
extra-google-webdriver

No tricks there, just a text file with the SDK elements we want installed. The following is the script itself:

#!/usr/bin/env bash

LOG () { echo -e "[LOG] $1"; }
RUNCOMMAND () { echo -e "[CMD] $1" && eval $1; }

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"

if [ -z "$ANDROID_HOME" ]; then ANDROID_HOME="${PWD}/android-sdk-linux/"; fi
echo $ANDROID_HOME
exit

# Install base Android SDK
RUNCOMMAND "wget http://example.com/android-sdk_r22.0.5-linux.tgz"
RUNCOMMAND "tar xzf android-sdk_r22.0.5-linux.tgz"

# install android build tools
# - sudo apt-get install --force-yes unzip
RUNCOMMAND "wget http://EXAMPLE.COM/build-tools_r18.0.1-linux.tgz"
RUNCOMMAND "tar -xf build-tools_r18.0.1-linux.tgz -C $ANDROID_HOME"
RUNCOMMAND "mkdir -p $ANDROID_HOME/build-tools/"
RUNCOMMAND "mv $ANDROID_HOME/android-4.3 $ANDROID_HOME/build-tools/18.0.1"

# Install required Android components.
# For a full list, run `android list sdk -a --extended`
# Note that sysimg-18 downloads the ARM, x86 and MIPS images (we should optimize this).
# Other relevant API's

for dep in `cat $DIR/../deps.txt`; do
    echo "Installing $dep"
    expect <<DONE
        set timeout -1

        # install dependencies
        spawn android update sdk --filter $dep --no-ui --force --all
        match_max 1000000

        # Look for prompt
        expect "*?\[y\/n\]*"

        # Accept the prompt

        send -- "yes\r"

        # send blank line (\r) to make sure we get back to gui
        send -- "\r"

        expect eof
DONE
done

echo "Installing sysimg-18"
expect <<DONE
    set timeout -1
    # install dependencies
    spawn android update sdk --filter sysimg-18 --no-ui --force --all
    match_max 1000000

    # Look for prompt
    expect "*android-sdk-license-bcbbd656*"

    # Accept the prompt
    send -- "yes\r"

    # Look for prompt
    expect "*intel-android-sysimage-license-1ea702d1*"

    # Accept the prompt
    send -- "yes\r"

    # send blank line (\r) to make sure we get back to gui
    send -- "\r"

    expect eof
DONE

A few notes:

  • You’ll need to specify a url to the base Android SDK. This isn’t provided by us. We upload ours to S3.
  • You’ll also need to specify a url for the android build_tools. Again, we suggest using S3.
  • If you use different version of the sdk than we do, then feel free to modify this script.

Finishing up

We’ll want to skip the installation of requirements, since we more or less took care of it in the before_install. Feel free to move that to this section. We simply skipped it for simplicity:

install: true

Next, we’ll want to actually run tests. Since we use maven, performing a mvn clean package will not only create a package, but will also ensure that all tests pass before doing so:

script: mvn clean package

For those who wish to use our build-artifacts tool, you’ll want to use the travis-artifacts gem to upload your apk. Here is what we do:

after_success:
  - "cd android/target"
  - "bundle exec travis-artifacts upload --path android-1.0-SNAPSHOT-aligned.apk --target-path android-app/$TRAVIS_BUILD_NUMBER"
  - "bundle exec travis-artifacts upload --path android-1.0-SNAPSHOT-aligned.apk --target-path android-app/latest"

Note that these are production releases. Debug code is turned off, so if you want to enable that, you’ll need to adjust your maven settings.

Some closing thoughts

One advantage to our setup is that it allows developers to write code without necessarily having an Android dev installation setup. I use it for this at my home computer when quickly testing bug fixes. While it isn’t a complete solution, it does bring us to that last mile.

A few possible improvements:

  • Use bundler caching instead of a custom caching script
  • Switching to native Android SDK support should simplify the setup of an Android application, hopefully removing much trial and error.
  • Another possible success would be to use HTTP caching. Travis does offer this, but you’ll need to ask directly, and it’s likely not going to be approved for large files which you should place in your S3 ;)
  • Fixing Maven 3.1.x support will remove a bit of the complexity, though that seems unlikely to occur in the near future.
  • Build times aren’t extremely quick. On a recent Macbook Pro, we see build times of a minute or less. Travis averages ~5 minutes for us. This is therefore better used for branch integration, and potentially for creating releases for the Play Store.

We’re pretty happy with the final outcome, and while we have had to work on it every so often, the tweaks are pretty easy and usually apply to our Vagrant development environment.

Shout out to the Travis-CI folks for being so supportive as we’ve abused their systems :)