This post is the first of a two-part series on using Vault in production. Both posts are slightly redacted forms of internal documentation. This post will cover why we chose our specific workflow, and the second post will cover day-to-day usage of Vault.

Problems

Sensitive credentials and keys are stored in certain code repositories (Github).

• Anyone with access to Github has access to these credentials.
• Anyone who has checked out code has these sensitive credentials on their hard drive.
• Key rollovers are a very difficult, manual process.

Sensitive credentials and keys are stored in plain text.

• Anyone who can see these credentials can use them.

Shared credentials and keys are used in numerous places.

• Generating a meaningful audit log is difficult.

Goals

1. Encrypt sensitive credentials and keys at rest.
2. Store sensitive credentials and keys in a central, remote, network accessible location.
3. Gate and audit access to sensitive credentials and keys.
4. Provide a unique identifier to each user/agent (per auditing purposes).

Solution

Vault

By leveraging Vault, we can meet all of our goals.

1. Encrypt sensitive credentials and keys at rest.

Vault encrypts data all stored data at rest.

2. Store sensitive credentials and keys in a central, remote, network accessible location.

Vault is a highly available secret management solution that is network accessible via its HTTP API or via running a local client.

3. Gate and audit access to sensitive credentials and keys. 4. Provide a unique identifier to each user/agent (per auditing purposes).

Vault allows for per user, per machine, or per app credentials controlling access as granularly as needed or desired. In addition, all requests and key usages are recorded in Vault’s logs or syslog which can be shipped to a centralized logging solution.

Implementation Strategy

While Vault provides the primitives and tools, we still need to form a process that understands and works with SeatGeek both now and in the future. With encryption and auditing handled, our job is to store and provide access to secrets as well as manage tokens.

NOTE: The following assumes knowledge about specific Vault features, general AWS knowledge, and SeatGeek’s Base AMI.

Storing Secrets

At SeatGeek (and most other software shops), the two most common types of secrets are the following:

1. Per Enviroment

   This includes secrets that the same for every machine or application, but differ based on the current environment. They are also commonly or can be used by all machines or applications, which is important to note.

   Examples: New Relic, PagerDuty

2. Per Application

   This includes secrets that differ between applications, where an application is the combination of itself and the environment in which in runs. This also includes secrets that are not common to every application, regardless if one value is always used.

   Examples: Braintree Token, Spreedly Key, Sentry DSN

To address these two use cases, we will be using Vault’s generic secret backend.

The reasons for using this backend are simplicity and flexibilty. It allows for arbitrary key-value pairs to be stored, encrypted, and retrieved from Vault without the need or use of third party services.

The generic secret backend allows for key-value pairs to be written under the namespace secret, and can be associated with various ACL’s. The currently used schema is of the following form:

1

secret/ENVIRONMENT/APP/KEY value=VALUE

Here, the top level under the secret namespace is ENVIRONMENT, with each APP getting its own bucket per ENVIRONMENT in which KEYs are written. Vault KEYs can contain a dictionary of key-value pairs themselves, and so the secret VALUE is written to the key value.

NOTE: bucket == namespace

The following environments exist:

1
2
3
4

production
staging
management
test

Each app will have a bucket created when it is configured to launch in a given environment. Additionally, for our per environment secrets there is a common bucket under each ENVIRONMENT namespace.

Examples of secrets in the wild:

1
2

Staging New Relic key
secret/staging/common/NEW_RELIC value=THISITHEKEY

1
2

Production API Spreedly Token
secret/production/api/SPREEDLY_TOKEN value=THISISTHETOKEN

Accessing Secrets

The basic premise here is a client authenticates and is granted a token. That token, among other things, is associated with a role and corresponding set of authorizations in the form of policies or permissions.

Authentication

At SeatGeek (for the time being), there are two Vault clients we need to worry about:

1. Developers

   These are people who write code at SeatGeek. Developers should be granted enough access to be able to do their jobs while keeping our sensitive information secure and our applications running.

2. Machines

   These includes any servers running with SeatGeek infrastructure. Machines should be able to self-authenticate in order to retrieve necessary secrets for provisioning and running applications.

NOTE: This workflow differs for Admins who are granted root tokens, no permission restrictions here.

To provide these levels of access, two different Vault authentication strategies will be used specifically github authentication and app-id authentication.

The github authentication strategy was chosen here as we are already using it as a means of authenticating people for internal applications, and so some user grouping has already been done.

The app-id authentication strategy is used for roughly the same reasons as the generic secret backend. It is the simplest and most flexible to implement without relying on other systems.

Successful authentication via either of these methods results in a Vault token, which can be used to retrieve secrets.

Our github authentication includes simply allowing anyone in the SeatGeek Github organization on the team-developers team to be able to request and retrieve a Vault token. This is done by making a Vault login request with a Github personal access token. While this does not include everyone who writes code, it handles the majority of users for now.

Our app-id strategy reserved for machine authentication is highly dependent on AWS and our newer infrastructure strategies. When an AWS machine boots up, it can be configured to run with an IAM Role. This role is unique per application per environment, and also includes an id which can be retrieved from an instance’s metadata on the machine itself. Using this information, all SeatGeek IAM roles are whitelisted within Vault against their matching app and associated with a IP Range that corresponds environment’s VPC IP Range. This is our user-id in Vault terms. Machines can then make a Vault login request with the app they are responsible for running (applied during configuration management) and their IAM Role Instance Profile ID (attachment id). Assuming all pieces line up (IP address, app id, IAM Role Instance Profile ID), a Vault token is granted.

Additionally and only for machine authentication, there is a ENVIRONMENT-base-ami role that all machines can authenticate as. This allows for all machines on boot to be able to retrieve environment secrets via Vault’s app-id strategy without knowing which app is to be deployed. This is/would primarily be used to be able to test the Base AMI in isolation in our environments.

In both of these app-id authentication scenarios, the user-id is the machine IAM Role Id. However, when applications authenticate, the user-id is app-IAM_ROLE_ID. user-id’s must be unique, and this allows for us to have two user-id’s for a give IAM Role along with the appropriate configuration.

In the latest release of Vault, the app-id strategy has been deprecated in favor a new app-role strategy. Ultimately we will migrate from app-id to app-role with roughly the same implementation but are currently held back by the version of Vault (0.6.0) and the vault-ruby (0.6.0) gem we are using.

Authorization

Vault implements authorization via its own ACL’s or policies. These provide a set of permissions which can be scoped to various operations within Vault, typically indicated by namespaces. In the case of obtaining secrets, that namespace is secret. Additionally, these ACL’s can be associated with the various authentication strategies. A more generic way to think of it is a client authenticates and is granted a token. That token, among other things, is associated with a role and corresponding set of policies (same as other authentication/authorization strategies).

The current policies are used to control access to Vault secrets:

1
2
3
4

staging-read-only
testing-read-write
ENVIRONMENT-APP-read-only
ENVIRONMENT-common-read-only

As far as developer authorization, all Github users are granted staging-read-only and testing-read-write, which if not obvious, means that any secret under the staging namespace can be read, and free reign with the testing namspace. production read-only access will be granted on a per application bases to service owners, and be implemented via Github teams.

As far as machine authorization, machines are granted the ENVIRONMENT-APP-read-only and ENVIRONMENT-common-read-only. As such, machines can access the common bucket and their app bucket within their ENVIRONMENT, nothing else. Cross ENVIRONMENT and cross app secret access is currently disabled and discouraged, although this might be revisited in the future.

Important to note here is the inability for non-Admins to write or update anything in Vault. These permissions are currently restricted to members of the Operations team, but this will surely be revisited in the future.

Token Managment

As of now, Vault tokens last forever once granted. This is a temporary measure that allows for simplicity of use, but additionally tooling will allow for this be changed.

Causes for Concern

1. Admins are granted root tokens
2. Developer authentication and authorization is reliant on Github
3. Machine credentials can be used on other machines within an IP Range
4. Assumptions are made around machines running a single application
5. Tokens last forever and be reused if retrieved
6. Vault is not using TLS
7. Metrics are not currently sent anywhere
8. No ui solution for managing secrets
9. Not possible to easily assume an application’s environment

Strategic Improvements

Admin Tokens

Currently, Admins are granted root tokens without permission restricitons. The latest version of Vault (0.6.2) has changed the ways in which root tokens are created/used, and as such, these could be substitued for Admin tokens or tokens with equivalent or slightly less permissions granted.

Developer Authentication/Authorization

With a centralized login system, developers would be able to authenticate with means other than Github potentially being more flexible and less dependent on a 3rd party. Permission granularity could also be provided on a per user basis allowing for trusted production access (ex: service owner access).

Machine Authentication

While we are already leveraging AWS for machine authentication, there are improvements in Vault to make this simpler and more secure. This integration would tie us tighter to AWS infrastructure, but it is doubtful we would run servers elsewhere, and if so we have an existing strategy.

These improvements involve allowing machines to one time authenticate with AWS dynamic metadata, addressing the issue of credential (re)use on different machines. Machines can be currently whitelisted by IAM Role or AMI.

App Authentication

We currently have a decent strategy for machine authentication, but our application authentication lacks flexibility. Specifically we assume that a single machine is running a single application and as such has a single IAM Role with the appropriate permissions for that application. This does not work if multiple applications coexist on a single machine, or if an application is broken up into tiers.

A way to combat this is to have application authentication use a different mechanism than machine authentication. This will require a revisit but will most likely leverage Vault’s Cubbyhole to multi-application scoped tokens via one time tokens.

Token Management

Tokens last forever currently, and should have leases and TTL’s. This would involve additional work to renew token leases as necessary.

TLS

TLS is disabled on our Vault cluster as it is addressed only within our internal network. With the requirement of TLS for all HTTP 2.0 connections, this will be revisited in the future and most likely with Vault serving as an internal CA.

Metrics and Monitoring

We are still in the early stages of adoption and use, but Vault has support for shipping application stats via a few means including StatsD.

Web UI

Either writing or adopting an existing open source solution would be extremely beneficial, as it would remove the burden of managing secrets from the Operations team while also allowing developers more control over how their applications are configured.

Locally Assuming App Roles

There is currently no way to run a command locally using the credentials in staging/production for a given application. Something like a .env file writer or a foreman-style command runner for our application manifests could go a long way in allowing developers to run services locally while simulating an environment.

Vault Configuration

https://www.vaultproject.io/docs/config/index.html

Below lists our current Vault configuration, which takes into account the following conditions:

1. Vault is running within our internal network and is not publicly accessible.
2. Consul is already being used

1
2
3
4
5
6
7
8
9

backend "consul" {
  address = "127.0.0.1:8500"
  path = "vault"
}

listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

Vault differentiates itself from other secret management services with its high availabilty option, and we leverage the Consul backend to deliver that. The Consul client is already configured to run on all of our machines (with default port mappings), with our Vault servers being no different. This also means that all data is stored encrypted in Consul, and so the Consul install should also be highly available.

As Vault is run within our internal network (and for other reasons), TLS is disabled. While this is desireable, we need to do additional work to make internal TLS usage a reality. Vault is also running on the standard default port of 8200 and listening on all network interfaces.

If you think these kinds of things are interesting, consider working with us as an Infrastructure Engineer at SeatGeek. Or, if infrastructure isn’t your thing, we have other openings in engineering and beyond!

Reference: https://sreeninet.wordpress.com/2016/10/01/vault-use-cases/

At SeatGeek, our mission is to help you see more live events. While we are very proud of how easy it is to find and buy tickets for events you already have in mind, we believe there is quite a bit we can improve when it comes to discovering new events. This blog post discusses what we’ve tried in the past, a couple features we’ve recently launched, and our larger vision for improving event discovery moving forward.

The Past

Our first major attempt at improving event discovery goes back to 2011 when we launched a calendar-based “Pandora for Live Events” called Columbus. This feature recommended events based on your favorite teams and performers. Despite our best efforts, Columbus wasn’t the big hit we expected. It had low usage numbers and didn’t translate well into a mobile-first world, so we decided to sunset¹ this feature several years ago.

The Present Recent Past

Since then, our event discovery experience has consisted of primarily:

1. Search
2. Location based recommendations
3. The ability to keep track of your favorite teams and performers

Here is how we surfaced these features in our mobile apps not too long ago:

This version was useful when you had a specific event in mind or if you were a power user with dozens of tracked performers, but left a majority of people yearning for more. It wasn’t useful for answering more general types of questions such as:

• Are there any good comedy shows in New York tonight?
• What are the most popular shows in San Francisco?
• What are some affordable Broadway shows when I visit New York in 3 weeks?
• What rock concerts are playing in Los Angeles this weekend?

Said differently, this version was good at search, but had room for improvement for browsing and general discovery:

Where to Start?

Once we decided to focus on improving event discovery, we next needed to figure out where to start. The ideas were endless: social integrations, date filters, price filters, genre filters, semantic search, ability to sample songs, ability to upvote and downvote recommendations, a notifications tab, revamped onboarding, and countless others. In addition to all these ideas, there was an equally large number of ways to implement each idea. Do we want to have one feed similar to Facebook? Multiple lists on the screen like Netflix? A card approach that focuses on gathering preferences for every event? We knew we wanted to take an iterative approach where each feature stands on its own but also builds towards an ambitious long term goal², but it wasn’t immediately clear where we should begin, how these items would fit together, and what exactly the long term goal would be.

Larger Vision

After carefully reviewing our options, we decided to guide our approach by looking at the relationship between types of user input (specific objective v general objective v little to no objective at all) with the different levels of personalization complexity required to return relevant results for a given feature.³

Search

For example, the bottom left of this chart describes search functionality, where there is a specific input (“Yankees tickets,” “Along Came Polly,” “Sweetgreen”) and a small amount of personalization needed to return a relevant result:

Browse

Moving up and to the right leads you to browsing features, where inputs are more general in nature compared to searching for a specific team or event. The more general nature of these inputs (“Comedy shows this weekend,” “Coffee in Nolita,” “A romantic comedy”) requires a bit more work to return relevant results.

List Based Discovery

Moving up and to the right again puts us in the discovery realm (“Find me something fun to do,” “Find me a good movie to watch right now,” “Find me a good place to eat”). For discovery based features, like Netflix’s homepage, there is no explicit input initially, which makes it a bit harder to return relevant results (in the case of Netflix both relevant lists and relevant shows or movies within each list). List based discovery offers many different types of recommendations, making it a bit easier to provide at least one relevant result.

Feed Based Discovery

While it’s no easy task to return relevant list based recommendations, there’s another level of personalization complexity required to surface 1-2 highly relevant pieces of content just by opening the app, as is exemplified by Facebook’s News Feed and Spotify’s Discover Weekly. This requires a sophisticated personalization strategy to do well.

Looking at the relationship between user input and personalization complexity helped us answer the questions mentioned above related to where we should begin, how these items would fit together, and what exactly the long term goal would be. We’d start by focusing on the browse experience because it would require minimal personalization complexity. We’d then move towards a list based discovery experience, and ultimately aim for the ambitious goal of having a feed based discovery experience that would show people extremely relevant content simply by opening the app:

The Present

Based on this high level plan, we recently launched several new features that improve the search and browsing experience. These include

1. The ability to find events for a specific date or date range
2. The ability to browse and search by specific categories and genres (Music, Sports, Comedy, Theater, etc.)

The Future

Over the next several months we are going to continue to improve the browse and discovery experience. While we have a general idea about what we think will work best, we plan to run experiments at every step along the way to understand what users actually find most useful. There are plenty of apps that provide great discovery experiences (Netflix, Facebook, Spotify), but that doesn’t necessarily mean their approach will translate well for the types of items people are searching for on SeatGeek. Here’s a preview of a couple future experiments:

We hope these new features help you find more events to attend with your friends! As always, we’d love to hear any feedback or suggestions you have to improve the app!

Help Us Make More Experiences

If you think these kinds of things are interesting, consider working with us on the iOS team at SeatGeek. Or, if you’re not an iOS Engineer, we have other openings as well!

As you know, iOS gives us a few different ways to create our user interfaces. Here at SeatGeek, we build our views in code. We’ve found that using auto layout together with the Masonry library makes it pretty easy to write our layout code, and pretty easy to maintain it later. Readability aside, though, auto layout does come with some complexities and gotchas that can be frustrating at first. This blog post is a list of some stuff we wish we’d known when we started.

Helper libraries:

• Masonry provides a nice DSL for creating auto layout constraints in code
• SnapKit is the Swift equivalent, pretty much exactly the same thing

UIView subclasses:

• For custom UIView subclasses that use auto layout, usually we need to override +requiresConstraintBasedLayout to return true. If you do everything perfectly and see an empty screen, it could be because this is missing.

Intrinsic content size:

• The native UIView subclasses provided in iOS, like UILabel or UIImageView, usually know how big they should be, so you often need only specify position when laying them out.
• Our own custom UIView subclasses can also have intrinsic content size, if we set up constraints to drive it. For example, it’s pretty common to drive intrinsic height by relating the top subview’s top edge to the view’s top edge, the bottoms and tops of the succeeding views to each other as we go down, and then the bottom subview’s bottom edge to the view’s bottom edge.
• Or, our views can have their content size driven by the containing view’s layout. For example, it’s common for the container to control the width, since we don’t usually have horizontal scrolling. We often still have the widths of the subviews fully related as in the height example above, but in this case, rather than the subviews driving the instrinsic width of their container, the container drives the widths of the subviews.
• It’s very common to have both approaches: the container drives the width from outside, but the subviews drive the height (or the contentSize.height, for a scroll view, see below) from the inside.
• Using intrinsic content size results in a lot less layout code — often you’ll see only relative / absolute positioning to locate the subviews. Really, subview size is often an internal detail, and it’s nice not to be hard coding it everywhere a reusable view is used.
• Of course, we do often constrain subview sizes, especially widths. For example with a UILabel, I’d often set the width, but use its intrinsic height based on the font.
• As discussed above, custom UIView subclasses we make can also have intrinsic size. We get this for free if we use auto layout. Otherwise we can override -intrinsicContentSize to make it easy for auto layout clients.

Scroll views:

• For scroll views, we usually want the container to specify the frame, so we set up constraints that do that. Then we want the contentSize.width to be determined by the container (usually the same as the frame width), but we want the scroll view’s contents to drive the height.
• contentSize.height is easy — just make sure that you have constraints as discussed above to determine the height. Typically the intrinsic heights of the subviews together with vertical spacing constraints to the scroll view do it.
• contentSize.width usually requires a trick. If you just relate the scroll view width to its container, that sets the scroll view’s frame, not the content size width. The solution I’ve usually used is to constrain contentSize.width by relating a subview of the scroll view to the containing view. If there is no suitable subview handy, I’ll add a dummy view only for that purpose.

1
2
3
4
5
6
7
8
9
10
11
12
13

[self.view addSubview:self.scrollView];
[self.scrollView mas_makeConstraints:^(MASConstraintMaker *make) {
    make.edges.equalTo(self.view);
}];

UIView *dummyView = [UIView new];
[self.scrollView addSubview:dummyView];
[dummyView mas_makeConstraints:^(MASConstraintMaker *make) {
    make.left.right.equalTo(self.scrollView);
    make.left.right.equalTo(self.view);
}];

// We go on to add the subviews to the scroll view, with horizontal and vertical constraints

Content compression resistance and hugging:

• Sometimes we need to give auto layout a little more information to know which fields are OK to stretch beyond their instrinsic size, and which ones are not. For example: you have an image with a label to the right of it. The left of the image is related to the left of the container, the right to the left of the label, and the right of the label to the right of the container. You want the image to be its natural size, and the label to stretch to the full available width, even if the text is currently shorter. If iOS stretches the image horizontally instead of the label, you can add this code:

1

[self.imageView setContentHuggingPriority:UILayoutPriorityRequired forAxis:UILayoutConstraintAxisHorizontal];

Multiline labels:

• Unfortunately, multiline labels sometimes don’t behave the way you’d expect unless you set preferredMaxLayoutWidth. The problem with setting that width at constraint creation time is that you give up dynamic layout update if the container width changes. One solution is to use a UILabel subclass containing a small fix. There’s been such a subclass at the last 3 places I’ve worked:

1
2
3
4
5
6
7
8

@implementation SGLabel

- (void)layoutSubviews {
    self.preferredMaxLayoutWidth = self.bounds.size.width;
    [super layoutSubviews];
}

@end

Centering groups of views:

• To center groups of views, normally we wrap them in a container view and center that.

Changing layout later on:

• Since auto layout is a bit more flexible than autoresizing masks, often an initial set of constraints is all you ever need. If the view’s bounds are changed, iOS will automatically update the layout using the existing constraints. In these cases, you can create the constraints just one time in -viewDidLoad or a UIView initializer.
• Every once in a while, you may need different constraints when the view is in different states. A common example is when a view that affects the layout of other views is hidden or shown. In that case, you can create the constraints in the -viewDidUpdateConstraints or -updateConstraints methods. When a state change that affects the constraints occurs, just call -setNeedsUpdateConstraints on the view. However, this means your constraint creation code will be run multiple times, so it needs to be idempotent —- see the next section.

Writing idempotent constraint code:

• Whenever our constraint creation code may be run more than once, we need to be careful not to add on a new set of duplicate constraints each time, some of which may be in conflict, spew warnings to the console, and cause your layout to be chaotic as iOS breaks constraints randomly to get a set it can satisfy.
• First, never use -mas_makeConstraints in code that may run more than once. Use either -mas_updateConstraints (if you always make the same constraints but their constant values change), or -mas_remakeConstraints (if you make different constraints for different states). Very rarely as a last resort, you might need to save the objects returned by these Masonry methods and invoke -uninstall on each.
• You might wonder why we need to use those different Masonry methods. Why can’t we can’t just clear all the constraints out of our view before adding them again with -mas_makeConstraints? It turns out this is tricky, because some of the constraints that you manage in -viewDidUpdateConstraints or -updateConstraints are actually owned by subviews. It’s not easy to remove just the constraints you’re about to re-add. Some constraints that your containing view manages (which you should NOT remove) may actually be owned by your view. Similarly, some constraints that your view manages (which you should remove) may actually be owned by subviews. You only want to clear the constraints that your view manages, that you’re going to add again. Masonry provided the above methods to make that easy.
• Couple gotchas with the UIViewController -viewDidUpdateConstraints method: (1) it’s often called more than once even when you didn’t call -setNeedsUpdateConstraints, and (2) sometimes you need to set up your constraints earlier in -viewDidLoad for everything to work nice. Because of this, I got in the habit of always making my constraints in a helper method -addOrUpdateConstraints, and always making them idempotent as above. I call the helper from -viewDidLoad, and add a call from -viewDidUpdateConstraints if necessary. That also keeps -viewDidLoad from getting too cluttered up.
• Gotcha with the UIView -updateConstraints method: the super call must be made, and is supposed to go at the end.

Animations:

• When you invoke -setNeedsUpdateConstraints on a view, or update a constant value inside a constraint directly, iOS will mark the view as needing layout, and set new frames using the updated constraints when it gets around to the next layout pass. The frames are not updated synchronously. If you need frames to update at a certain time, for example within an animation block, invoke -layoutIfNeeded on the view. Constraints are not animated, frames are, just like always.

Gotchas:

• Sometimes table header views behave strangely if auto layout is used, on iOS 7 & 8 at least. For those cases and other rare times where auto layout just doesn’t seem to work right, I usually time box it and fall back to another method like programatically setting frames in -layoutSubviews / -viewDidLayoutSubviews.

Let us Know

We hope some of these tips prove helpful. We’re curious what useful techniques you’ve discovered, so feel free to let us know in the comments!

Help Us Take it to the Next Level

If you think this kind of stuff is interesting, consider working with us on the iOS team at SeatGeek. Or, if you’re not an iOS Engineer, we have other openings as well!

Background

Earlier this week Apple released a major upgrade to Messages as part of iOS 10. In addition to being able to send different styles of texts and respond to messages with reactions, iOS 10 also allows developers to create app extensions that let users interact with apps directly within Messages. Going to live events is an incredibly social activity, making iMessage Apps and SeatGeek a perfect match.

Looking for a fun event to go to tonight? Want to send your friend an extra ticket at the last minute? No problem, the SeatGeek iMessage App allows you to easily share events and send tickets to friends, all without leaving your conversation.

Sharing Events

Our iMessage app makes it easy to browse and share events with your friends. By default, we suggest popular events in your area. If you’re a regular SeatGeek user, we suggest events you’re tracking and upcoming events by your favorite performers:

You can also search for any event you have in mind:

Sending Tickets

In addition to sharing events, our iMessage app allows you to quickly send tickets to your friends. All you need to do is tap “Send Tickets” and you will be shown tickets for all upcoming events you’ve purchased on SeatGeek. Tapping on an individual ticket will then send it to your friend. No more waiting at the gate for your friend that is perpetually late.

Find More Fun

We hope our iMessage app makes it easier to see more live events with your friends. Try it out yourself by downloading it on the App Store!

Help Us Make More Experiences

If you think these kinds of things are interesting, consider working with us on the iOS team at SeatGeek. Or, if you’re not an iOS Engineer, we have other openings as well!

Listening Suggestions is a new feature of the SeatGeek app for Android, available now on the Play Store. Listening Suggestions shows you a rich notification when you’re listening to an artist or band that has a concert coming up nearby in the next few months.

We’re all super pumped about this feature. Jack—our CEO—often talks about how experiences, more than anything else, are a meaningful part of a person’s happiness. That’s why we created Listening Suggestions, and built it the way we did. We wanted to create something that didn’t just let people know about concerts nearby, but did it in a way that could truly wow a person. Strong colors and artist imagery come together with useful information to create a truly interesting notification to let you know about an artist’s show. We’re incredibly excited to see more people take advantage of it!

Turn it on!

Once you enable Listening Suggestions in the Android app—either via the dialog that pops up when you open the app, via Settings, or from the notification that comes up when you play music—you’ll be on your way to getting great event recommendations for artists you listen to.

The notification is silent, so it won’t disrupt your listening experience, but is easy to notice when it shows up. Tap the notification to view and buy tickets in the SeatGeek app, or track the event right from the notification.

Google Play Music & SoundCloud

Listening Suggestions works out of the box with Google Play Music and SoundCloud. All you have to do is listen to music, and we’ll let you know when an artist has an event coming up!

Spotify

If you’re using Spotify, you’ll have to enable a special setting within the Spotify app to have this work for you. Don’t worry, it’s not too difficult. Within Spotify’s settings, turn on “Broadcast Device Status”.

With that finished, just go back to doing what you always do. Listen to awesome music, and when we find a show, you’ll know immediately.

The updated Android app is out today in the Play Store, so check it out and let us know what you think! (Sorry iPhone users, this is an Android-only feature)

We recently announced partnerships with Major League Soccer and Sporting Kansas City. Today we’re announcing SeatGeek Open, the platform behind it all.

Fifty years ago fans had to queue in line at the box office – cash in hand – to pay money for a ticket. Throughput was a function of box office attendants. The Internet made it much easier to sell a ticket; you needed one server to handle an on-sale and not a bunch of attendants. But, in the last 20 years not much has changed in the ticketing world. SeatGeek Open brings ticketing to the 21st century. We provide the technical infrastructure where any site can function as a box office.

The core premise of SeatGeek Open is in the name: openness. This is in stark contrast to the closed nature of the ticketing industry today. Teams and artists will be able to distribute their tickets wherever they choose. We’re letting anyone tap into our APIs and use them for free. Imagine renting an Airbnb in New York City and seeing the option to buy tickets to the upcoming Drake show at Barclays Center. Or perhaps you are shopping for your favorite team’s merchandise on Amazon and want to add tickets to your cart. SeatGeek Open makes it possible.

Stripe and Braintree empower app developers by reducing the messy world of payments to clean, well-documented APIs. SeatGeek Open does the same for ticketing. Developers can integrate tickets into their app with a few hours of work. Our open APIs encourage other developers to build new services which will enable teams to run more effectively. Rightsholders will be able to choose from dozens of services like pricing, analytics, marketing automation, and CRM.

Most importantly, SeatGeek Open means a better experience for fans. Every ticket they purchase (primary or secondary) on any site will be verified. The closed ecosystem perpetuated by Ticketmaster forces fans to stumble through a clunky user experience to buy a ticket. A distributed box office forces every ticket seller to step their game up. Fans have myriad choices of where to buy tickets, and it’s incumbent on us to build the best experience for the fan.

Our partnership with TopTix enabled SeatGeek Open. We wanted an API-driven inventory management partner who sold tickets at scale. In six months our software will be live at Sporting Kansas City, proving the benefits of open distribution.

In technology, the only thing constant is change. Ticketing has been an exception and SeatGeek is changing that.

In July, we outlined a fundamental problem with the ticket industry. In a word, the ticket industry is closed. It should be open. Industry leaders loved our post, but they questioned “so what are you going to do about it?” We’re proud to announce a partnership with Major League Soccer and a radically open ticketing system for soccer fans.

Our partner, MLS, is a young league. In fact, MLS celebrated its 20th birthday this year. With an unrelenting focus on the fan, MLS is now the ascendant sports league. We see this in our data. MLS sales are quadrupling year over year with the highest rates of mobile utilization and last-minute purchasing of any league. To accelerate their rapid growth, MLS needed a novel approach to ticketing, one that focused squarely on the fan.

MLS has a young, mobile-focused fan, the type of audience that other leagues covet. Soccer fans spend time in dozens of mobile apps: from e-commerce sites like SeatGeek, Uber, and AirBnb to social media platforms like Facebook, Snapchat, YouTube, and Twitter. We are building a new type of ticket distribution, one that will empower dozens of sites and apps to sell tickets on behalf of a team. The premise is simple – put the tickets where the fans are.

Fans won’t have to wait long. Through a partnership with TopTix, we will be powering Sporting Kansas City in 2017 with our new, radically open platform. SKC’s tickets will be listed on dozens of existing and emerging ticketing and non-ticketing platforms. SeatGeek users will have access to secondary and primary tickets for Sporting Kansas City. Users will be able to purchase tickets in a few taps, and if they cannot make an event, re-selling tickets is dead simple using SeatGeek Marketplace.

Our ticketing service will revolutionize how Sporting Kansas City (and other teams) run their business. Rather than relying exclusively on their marketing to drive sales, SKC can put inventory in the hands of other companies, elevating their brand via new channels. Even more, SKC will better understand which fans are attending their games by gathering purchase data from dozens of sources their tickets sell on. Sporting Kansas City is an undisputed leader in technology and innovation. They immediately understood the power of open distribution, and we couldn’t be more excited to launch with them.

1996 – the year of the first MLS game – also marks another noteworthy event in our industry’s history. Ticketmaster sold its first ticket over the Internet. Yet, the Internet’s great promise of openness never affected the ticketing industry the way it transformed other industries. Innovation and change stagnated. Twenty years later, SeatGeek and Major League Soccer will have a chance to rewrite a new future, one that gives unprecedented control to rightsholders and access to fans.

At SeatGeek, we think about our venue maps as “maps” rather than “seating charts.” Therefore, we aim to make our venue maps as good as the best interactive geographic maps in the industry, such as those by Google and Mapbox.

As part of this aim, we recently made a major change to our venue maps on our web platforms. We now render a vast majority of our maps client-side using vector tiles and Mapbox GL. This is the first major user-facing change stemming from our GIS pipeline and opens the door for some really cool things to come.

Background

For those not familiar with modern web maps, it may be helpful to take a step back and define some concepts. There are two primary ways in which to define images in computer graphics: raster and vector.

Raster Graphics

In raster graphics, images are defined using a rectangular grid of pixels. Pixels are samples of a source image, typically represented by red, green, and blue values. Some common raster file formats include JPEG, TIFF, PNG, etc., with the primary distinguishing feature being the particular flavor of lossy or lossless compression involved.

Due to the fact that these images are based on pixels, they are resolution-dependent. Therefore, an image created at a particular resolution only looks correct at that resolution. Displayed at any other size, the image must be resampled. Through this process, things like diagonal lines in the source image can appear as “staircases” or “jaggies” in an enlarged image. Real life is not like CSI; you can only “enhance” a raster image so much.

Vector Graphics

Unlike raster graphics, vector graphics don’t store information for every pixel of the image. Rather, vector images are defined using higher-level primitives, such as polygons, bezier curves, and ellipses. Due to this, vector graphics are not resolution-dependent; they can be accurately rendered at any scale, regardless of the rendering device resolution. Common vector file types include EPS, AI, and SVG.

For many graphics, vector representations are substantially smaller files, since they do not store the color value of each pixel. As you can see with the SeatGeek logo, a shape can be defined by just a few nodes and control points. Vector graphics are particularly useful for images which are composed of well defined geometric shapes and curves, such as fonts, logos, and maps.

Raster Maps

Until just a few years ago, almost all web-based maps employed the same strategy: displaying a specific area and zoom level of a map using a collection of raster image tiles (e.g. 256px PNGs). Tiles are particularly useful at producing the effect of a very large, interactive image on the client while being highly cacheable.

However, raster tiles have some downsides. Probably the most noticeable to the end user is that the tiles only exist at integer zoom levels. This means that any zoom action results in a “jump” between different zoom levels. Many libraries cover this using CSS transformations, scaling up the lower resolution tiles and replacing them with the higher resolution tiles as they load.

Additionally, because the tiles are rendered on the server, there is no way to manipulate or style the map on the client without using overlays, like we did with canvas tiles.

Vector Maps

Unlike raster maps, vector maps are rendered entirely by the client. Although they use the same tiling strategy as raster maps, these tiles only store geometries and metadata. The client-side renderer (Mapbox GL) takes the data from the tiles and style definitions and draws maps at a very high frame rate.

Even though vector tiles are also only defined at integer zoom levels, the renderer can render any zoom level. This means we can initialize our venue maps to be the perfect size for every screen and give the user an incredibly smooth experience.

In web apps, vector maps have only become practical in the last few years. For any moderately complex map to render at 60FPS, the renderer needs to be WebGL-based. WebGL permits the use of shader code in order to take advantage of dedicated GPUs, which are able to draw scenes much more efficently than any CPU-executed JavaScript. Unfortunately, it has taken a while for WebGL to reach a vast majority of users’ browsers. As recently as 2012, the only major offering of vector maps was Apple Maps, and that was only offered in native applications.

Thankfully, we have seen tremendous growth in WebGL support in the last couple of years. According to our tracking, over 95% of our visitors have a WebGL enabled browsers. Only for a sliver of traffic and a small number of older maps do we fall back to our Leaflet.js raster maps.

Next Steps

One of the major reasons we chose Mapbox GL is because of its cross platform support and we are actively working on bringing vector maps to our native applications.

Additionally, some of the design and UI decisions we made were reflections of the limitations of our legacy mapping stack. Now that we have the ability to fully access and manipulate the map data on the client, we are free to completely rethink how our maps work.

Final Thoughts

First of all, we want to give a shout-out to Mapbox. Our maps have benefited greatly from their open source work. They set a great example of how companies can contribute to the tech community and have been an inspiration for our efforts to do more of the same.

Finally, if these are the types of projects that excite you, come work with us!

Last month, former Ticketmaster CEO Nathan Hubbard published a piece on “why you can’t get a ticket to the NBA Finals.” It’s a good article and Nathan does a great job diagnosing problems with the status quo – including how arbitrageurs use bots to buy tickets before real fans get the chance.

While I agree with Nathan that we need to solve the problem of how to get more fans to more events at a fair price, I disagree on the solution.

Hubbard proposes restricting how fans can transfer tickets, locking them down within a closed platform. This technique is called “paperless ticketing” – if you’ve ever been to a concert that demands you swipe your credit card to get in, then you’re familiar with it. The idea is that by making it harder to transfer ownership, non-fans have less incentive to buy and resell tickets.

Paperless ticketing can be effective in reducing resale activity in some cases, but with it comes substantial downsides ^[1]. I think there’s a better, more open way to improve the experience for everyone involved: leagues, teams, venues, and most of all, fans. And it’s not too different from what has worked in other technology categories.

But first, a quick aside…

The power of the internet lies in how it enables developers everywhere to create technology to improve lives. Open approaches – ones that start with easy-to-use software, priced transparently and fairly, and built for others to share and improve – are transforming major sectors globally, from payments (Braintree & Stripe), to hosting (AWS), to communications (Twilio), to mobile (Android).

But somehow ticketing has been left behind. It remains closed, uneaten by software. Why?

The short answer is that it’s always been a pay-to-play business.

Ticketmaster (and its parent company, Live Nation) – and a handful of competitors – have used the power of their pocketbooks for years to lock up teams, leagues, and arena owners with rigid long-term contracts that force these stakeholders to use their platforms exclusively, leaving consumers with little choice other than to accept their considerable fees.

As a result, live entertainment remains largely closed, stuck in a world of pre-internet thinking. The core API endpoints of the live event commerce experience – purchasing, delivery, transfer, fan identity, access – are locked down within legacy primary ticket vendors, and are inaccessible to developers, artists and teams ^[2].

The end result? Everyone is unhappy. Fans are forced to deal with an awful checkout experience, are nickeled-and-dimed with fees at every turn, and are unable to openly transfer or sell tickets. Teams and artists aren’t able to manage their businesses effectively, e.g. they cannot move prices in real-time ^[3]. It’s no wonder that when you add all this up, legacy ticketing companies are some of the most loathed businesses in America, year in and year out.

It’s within this closed status quo – of legacy ticket companies controlling the ticketing rights and then selling through a locked-down experience – that Hubbard suggests the answer is an even more closed platform.

I’d rather make it open. And the good news for fans is that, behind closed doors, an increasing number of industry stakeholders, including teams and leagues – anchored by a new, tech-savvy generation of owners – are thinking similar thoughts.

What do we mean by an open ticketing ecosystem? Imagine a world where…

• Artists and teams can sell their tickets on any website or app, whether it’s their own, Facebook Messenger, Amazon, every ticket site, or anywhere else. (In the status quo, artists and teams can only sell tickets on their ticket vendor’s site. Unlike most other ecommerce categories, they can’t leverage omnichannel distribution to sell across the web ^[4]. Think about how strange that is…not even Apple, the most iconic brand in America, sells exclusively on Apple.com!)
• Fans can, in turn, buy tickets anywhere. They can also sell or transfer tickets anytime. Can’t make it to a game last-minute and want to send your tickets to your favorite co-worker on Slack? Cool. Want to list your tickets on StubHub and reissue a new barcode to the buyer? No problem. You own the tickets. You decide what to do with them.
• Using open APIs, enterprising software developers can write apps that improve the fan experience. Maybe a team wants to sell tickets from within Uber, bundled with an UberPOOL ride to the game? Easy to do. And Uber can further use the ticket platform API to make sure your carpool buddies are fans of the same team (it’ll make for a more fun ride!).

So how would an open ecosystem help artists get tickets in the hands of fans, not bots? There’s an obvious answer, and there’s a more important answer…

The obvious answer is that openness allows ticket sellers to better understand the identity of buyers. In the legacy ticketing status quo, a bot doesn’t need to prove it’s a real person, it just needs a credit card. A smarter system could ask a buyer to authenticate via Facebook, Twitter, or Spotify. It’s more difficult (and less scalable) to create an account on those platforms with many real friends/followers/listens than it is to get a new credit card ^[5].

But the more important answer is that I don’t know what the best solution looks like. No single person does. A core feature of the API-driven open web is that disparate software systems can work together to solve problems more effectively than a single, monolithic one. One company can’t possibly build the best version of everything, including Ticketmaster. The free market of ideas usually wins. That brings me back to a solution Hubbard mentioned in his piece…

“Artists and teams can use technology to design a screening system that gets below-market-priced tickets directly to passionate fans who will use them. This is an aspirational but unproven concept, one that seems risky to implement at scale in a way that doesn’t violate privacy or completely piss customers off.”

I’m confident that the best solution to that problem can best be found by a bunch of smart developers trying a bunch of different approaches that we haven’t thought of yet.

There’s reason for optimism within live entertainment; we’re finding an increasing number of industry stakeholders see the world in this way. Last year Eventbrite launched Spectrum, a marketplace of services that easily integrate on top of Eventbrite. NYC-based Ticket Evolution created an elegant API that allows any third-party to access a large swath of resale ticket inventory that can be sold and delivered anywhere on the web. At SeatGeek, we’ve opened nearly all of the data on our platform via our API and encourage any seller to list inventory via our Checkout APIs or our consumer marketplace. We also allow artists and teams to reach millions of incremental fans by working with their primary ticket companies to list inventory directly on SeatGeek.

At the end of the day, artists and teams care most about creating great experiences for fans, and fans care most about enjoying them. Closed platforms add friction to the ecosystem. Open platforms at their core are enabling, harnessing the power (and collective wisdom) of the internet to solve existing problems and create heretofore unconceived experiences. And the best part? That means more fans, at more events, having more fun.

This post was also published on TechCrunch.

Notes

Welcome to SeatGeek Employee Spotlights - an opportunity to meet the fantastic folks on our world-class team.

By day, we’re a group of talented developers, designers, marketers, and businessfolk working together to build something new and different. We represent live event junkies of every kind: diehard sports fans, passionate concert-goers, sophisticated theater enthusiasts, and more. From our lives outside the office and before SeatGeek, we all have interesting stories to tell.

Up next: Adam Waxman, Product Manager.

Where were you born?
I was born outside of Cleveland, Ohio, and lived there until I went off to college.

Have you lived in NYC since college?
After school I lived in Atlanta for a year and a half. While there are many great aspects of Atlanta (weather, food, live music scene, etc.), I was interested in moving to a more walkable city with a greater percentage of people from all over the country (and world), so I decided to move to NYC.

Where did you go to school?
I went to Emory in Atlanta, GA. Initially interested in the business school, I ended up studying Economics, Math, and Philosophy after being convinced by a great freshman professor that liberal arts was the way to go. He told me it was important to be able to communicate clearly and learn how to learn rather than just memorize formulas. I couldn’t agree more with that mindset given how quickly the world changes these days.

So how’d you get your technical skills?
About a year after school I decided I wanted to build an app to help friends make plans. This forced me to learn basic tech lingo and ultimately convinced me to quit my job in finance to learn how to code. I spent my first 6 months going through online tutorials and building out various website ideas, and then decided to go to a 12 week developer bootcamp to hone my skills. Upon graduation I was lucky enough to stumble upon a unique role at SeatGeek where I’d be able to continue to hone my developer skills initially but ultimately move towards a product management role as the company continued to grow.

How would your friends describe you in 3 words?
Curious, genuine, goofy.

Best project you’ve worked on at SeatGeek
I really enjoyed working on our rebranding, which we launched earlier this year. It took a bit longer than expected, but I couldn’t be happier with the end result and I was able to learn an incredible amount along the way.

Three “fun facts” about yourself that people would be surprised to know
1. I grew up with 7 dogs
2. I don’t eat red meat (and haven’t in 15 years)
3. I tend to fall asleep a majority of the time I watch a movie, even the most action-packed movies

Favorite place(s) to hang out in NYC
I’m a big fan of the Westside Highway for both running and hanging out on a nice summer day. I also love reading a good book or working on a side project at a nice coffee shop. My current favorites are Grounded, Matchabar, Cafe Grumpy, and the FIKA on 16th and 6th.

Favorite SeatGeek snack
I’m a huge fan of the Cliff for Kids fruit ropes. I was turned on to these when I worked at a summer camp and one of the campers always had one packed in his lunchbox but never ate it. I couldn’t let it go to waste!

Why do you love SeatGeek?
I love the people I get to work with and learn from on a daily basis. Everyone is super talented but also very humble and collaborative. I also love the live event space - it’s fun working on a product that my friends and I use on a regular basis.

Favorite part of the new office?
I am a huge fan of the little personal booths/couches on wheels. When I first saw them I thought they were a bit silly but they’re actually very functional and comfortable.