Last Summer we decided to update the look and feel of our apps. This post shares our process from start to finish. It’s broken up into four sections:

1. Why we prioritized a redesign
2. Our goals
3. Our plan of attack
4. The (latest) results

1. Why Prioritize a Redesign?

There are many reasons to prioritize a redesign: fix usability issues, re-prioritize the problems you’re solving, align with the latest design trends, improve intra-app and inter-app consistency, as part of a rebrand, etc. While this project was initially triggered by wanting to make sure our designs were consistent with the new iOS 11 styles, we ultimately decided to expand the scope and prioritize a more comprehensive redesign across all our platforms. At a high level we aimed to:

1. Create a more trustworthy experience
2. Remove unnecessary distractions
3. Design and develop new features more efficiently moving forward

Creating a More Trustworthy Experience

People are (rightly in many ways) skeptical of the secondary ticketing industry. Accordingly, it’s important we do whatever we can to make a good first impression. Using consistent messaging and styling across a user’s entire experience (subway ad, app store, onboarding, emails, intra-app, etc) can go a long way in helping users feel more confident about buying tickets on SeatGeek. While we are proud of many aspects of SeatGeek, the consistency within and across our apps was not one of them before we started this project.

Removing Unnecessary Distractions

Simplicity is one of our core product values. While we generally try to avoid unnecessary elements and features throughout the design and development process, our iterative approach can lead to some nonessential features slipping through the cracks. We saw this project as an opportunity to cut back the visual noise and put more emphasis back on the core features of our apps.

More Easily Designing & Developing New Features Moving Forward

Over the past couple years there have been many advances in frontend technologies and design tools to make it easier to create, maintain, and use a modular design system. On the design side, you can create reusable symbols in Sketch and share these symbols across files using Sketch Libraries. This isn’t even taking into account all of the new tools like Figma and Invision Studio that have been built from the ground up with components in mind. On the development side, you can now build a react component once, document it using Storybook, and reuse it throughout your application. If you’re really fancy you can even leverage Airbnb’s React Sketch app to automatically generate these React components in your Sketch files. With a bit of upfront investment, all these new technologies can radically improve the speed and ease at which you build new features, and also ensure you’re being as consistent as possible.

2. Goals of the Redesign

We took the above ideas and distilled them down to the following success criteria:

• Being more consistent
  • Intra-platform (including latest OS trends)
  • Cross-platform
  • Between marketing and product design touch points
• Having the designs and interface work in a modular manner
• Doing the above without hurting conversion rates¹ or upsetting core SeatGeek users²

3. Plan of Attack

When starting this initiative, we knew we wanted to take an iterative approach, where each phase could stand its own but also build towards a more ambitious long term goal.³ You may be thinking, “How the hell do you iteratively roll out a redesign?” If you are, you’re not alone. However, when we started thinking more about our goals, we realized that we could accomplish a lot of what we were aiming for (consistency, modularity, and simplicity) without radically changing the look and feel or structure of the app.

Based on the above, we decided to break out this redesign into two phases. The first (main) phase consisted of updating the look and feel of our apps and making sure all elements are part of a cohesive system. This phase did not involve any radical structural changes, so we’ve been referring to it as a “Visual Refresh.” This foundation laid the groundwork for the second (much longer, practically never-ending) phase of using this new modular foundation when working on future projects.

To help us learn even more quickly (and partly due to design capacity) we decided to work on this one platform at a time. We started working on the modular design system on iOS, then took those learnings to other mobile platforms, and are just now beginning to apply this system to our desktop site. Additionally, to go along with the iterative spirit we focused our efforts solely on the core functionality of the apps to help keep the scope manageable.⁴

4. How’d It Go?

What Went Well

While we’re still far from finished, overall we’re extremely pleased with how this project has gone so far. As of now we’ve launched Phase 1 across iOS, Android, and mobile web, and are currently begin the design stage of Phase 1 on desktop. While in some ways the launch of the visual refresh was a bit anticlimactic, when working on a project like this that’s not necessarily a bad thing; the apps are way more consistent, the new designs work in a very modular fashion, and we did the above without hurting conversion rates or upsetting core SeatGeek users. Additionally, we now have a great new foundation to more quickly build incredibly consistent and delightful features. Given that this project was highly visible, some examples will be more effective at sharing the results.

What Could Have Gone Better

While we’re pleased with how the initial phase of the project went overall, there were certainly things that could have gone better.

For starters, it took us much longer than expected. In particular, we underestimated the amount of time it took to apply the system to the long tail screens and features⁵. There were also some more custom elements we decided to incorporate into the designs (custom nav bar, custom animation when selecting a ticket) that ended up taking longer than expected to get right. While these ended up being some of the more delightful pieces of the new version, they did drag the project on a bit longer than expected.

Related, while we were able to design most of the core screens in a modular fashion, we ended up deciding not to implement all of it modularly in order to make time for other initiatives. While modular designs alone are a step in the right direction, translating this modularity into code is vital to fully take advantage of the benefits of such a system.

Lastly, while there is certainly much more consistency both intra- and inter-platform, there is still a lot of room for improvement. In particular, we’ve found it more challenging than expected to balance platform paradigms with cross-platform brand consistency.

On the bright side, all these challenges have been great learning opportunities. Given that this is a never-ending type of project, we’re able to put these lessons to work right away!

What’s Next?

We’re currently focusing on applying the same strategy to our desktop site. There are a lot more features and screens on desktop, so it will likely take us a bit longer than any of the platforms we’ve already worked on. In addition to the desktop visual refresh, we’re also beginning to use the new system for all new features. Referrals (which we’re in the midst of beta testing!) is a great example of the new system in action:

We’re also improving the design system based off of what we’ve learned actually using it. These improvements include making it more accessible, putting more emphasis on component language consistency, and more clearly communicating the rationale behind certain patterns. We’re really excited about the progress we’ve made, but also realize that we’re really only getting started.

Whether you’re trying to get a redesign prioritized or in the process of planning for a similar project, we hope you have found it helpful to understand how we approached this type of project. We’re always looking for ways to improve, so if you have any design system suggestions (or questions), don’t hesitate to reach out to design@seatgeek.com.

Notes

¹We decided not to specifically aim to improve conversions because (as you’ll read later), we decided to approach this in a very iterative manner, so much so that we didn’t really expect to drastically affect conversions one way or the other. However, given that when you make large changes there is always a chance that you introduce bugs or affect the experience more that you expected, we wanted to make sure we didn’t negatively affect any metrics.

²In general people don’t like change, so it’s important to be mindful that any large changes may leave a sour taste in people that are very familiar with your platform even if there are good reasons for those changes.

³Allowing us to test hypotheses early on and learn from our mistakes earlier as famously depicted in this great slide.

⁴We still made sure the experience looked decent on non-core features, but we didn’t spend time building each from scratch and making sure it worked seamlessly with the new modular system.

⁵Despite the focus being on core features, we still need to make sure that the new look and feel worked across other areas of the apps.

The past year has been an extraordinarily exciting one for SeatGeek. In 2017, we’ve built the industry’s best seating previews, signed a deal with the world’s biggest social media network, launched our primary ticketing brand, and partnered with our first-ever NFL and NBA teams.

But in the last month of the year, we’re lucky to add one more accolade to the list. It’s an important one to us because it’s about our team. We’re thrilled that for the second straight year, SeatGeek is on Glassdoor’s list of Best Places to Work in 2018.

Something noteworthy about this award is that there was no self-nomination process. The scores are based on SeatGeek employee feedback given voluntarily (and anonymously) on Glassdoor over the past year. It’s humbling to make the list two years in a row – there were only nine other companies in the US that did so.

The fact that SeatGeek is a great place to work is one of the things we’re most proud of as a company. Working here has its perks: SeatGeek’s social calendar is full of parties and retreats, our kitchen pantry is on par with that of a five-star restaurant, and our monthly ticket perk helps employees attend their favorite live events.

But what really makes this a place to do great work – and have fun – is less tangible. SeatGeek is is an amazing place to contribute, grow, and build something special. The team believes in the power of live entertainment to make people happier, and there is a genuine passion for building a product that enables great experiences. The environment here encourages that, and the reviews reflect it.

As a company, we have some lofty goals for 2018 and beyond. We operate in an industry that has long been overdue for a fundamental shift, and SeatGeek is a catalyst for change in ticketing. We’ll stay focused on launching great products, partnering with amazing clients, and building out a distribution network for ticketing across the internet.

But one of our most important goals is to continue to be a fixture on awards lists like Glassdoor’s. As our team continues to grow, we’re fortunate to have found people that continue to embody SeatGeek’s core values. The people at SeatGeek that bring these values to life are our most valuable asset, and are what we hear about most often when employees talk about their favorite parts of working here.

If this sounds interesting to you, come work with us at SeatGeek! We have a number of roles open across engineering, marketing, business development, and more.

Cultural diversity is an important ingredient in creativity. At the tip of our process and teamwork, we leverage the different ways we use the internet and user interfaces to help us craft a well-rounded product and user experience. While cultural diversity can often be taken for granted being in New York City (where we call home, and can arguably be considered the “melting pot of the world”) we at times will hire extraordinary talent, no matter which city they call home.

I’m a Senior Director of Engineering at SeatGeek, and my team consists of engineers from 5 countries across 3 continents: North America, Europe and Africa. My situation is not at all unique in the engineering industry. Distributed team management is often a challenge many organizations face as they scale and sometimes hire extraordinary talent outside of their borders.

In my career, I’ve been involved in rapid scale, with remote teams, across many timezones and different working cultures. This type of scale and distributed team organization would not have been successful without travel, regular facetime, lots of communication, good processes and supporting software. So, naturally, one of the first things after I started at SeatGeek and joined my team was book flights to Copenhagen to meet some of them.

Now, I realize there are very good blog posts, books and talks about distributed team management and I certainly don’t have all the answers (I read those blogs and books, too!), but I felt it would be interesting to you to hear about my trip to Copenhagen, and share with you where we learned to be better with our distributed team.

Here’s what went down:

Saturday & Sunday - Day 1 & 2 - Copenhagen:

It was great being back in Scandinavia (I used to live in Stockholm, Sweden). The weather was cool, the food was delicious (fish, pickled everything, breads, cheeses…) and the beer (Mikkeller ftw) was plentiful.

If you ever get out to Copenhagen, here’s my hit list:

• Warpigs
  • An American style BBQ joint which is founded by Mikkeller and Three Floyds brewing. Expect nothing but amazing beer flights and hearty meats.
• Mikkeller everything
  • Mikkeller is a tour de force of beer in Copenhagen. Their locations around Copenhagen are all worth checking off your list.
• Nørrebro neighborhoods.
  • The northern neighborhoods of Copenhagen are tree lined, quaint and picturesque. Easy to get lost and found again.

Monday - Day 3 - Meets and Greets:

One of the immediate thoughts I had walking into workspaces is that it felt just as I remembered what any good Scandinavian workspace should be. Lots of plants, the low northern sunlight creeping through some shades, the smell of coffee, subtle music playing in the background… the atmosphere was very familiar. The floor the workspace is on is packed with other Danish tech startups. Some companies were game makers, others were building PaaS software, and there was everything in between. This workspace definitely had the better corner views, though. Great vibes all round, I loved being surrounded by other technologists and watching the Danish startup culture in full flight.

Day Agenda:

• SeatGeek news, updates & open conversation
• Distributed Team Effectiveness (Open Conversation)
• 1:1’s!
• Dinner and beer.

SeatGeek news, updates & open conversation:

We sat around the workspaces in an open conversation and discussed things that have been going on in the SeatGeek offices of New York. As a distributed team you can sometimes miss out on the regular news updates that you can get at the HQ. I learned that we can be better at sharing milestones and regular news, which at first may not seem mission critical, but really are vital to building the SeatGeek culture across the pond.

Distributed Team Effectiveness:

This was a great discussion around how we can be better working in a distributed team model. Hot topics like being social, communication and process and tools were discussed at length.

Being Social:

While we don’t follow “Capital A” agile strictly speaking, we do conduct the typical scrum ceremonies. I learned that we can be better in our standups. Given that standups are the one time of the day that the whole team is together, we should give our best effort to make it as effective and useful as possible. One idea was that since we are a little loose in our agile ceremonies, given the distributed nature of the team we thought that being a little more social in those meetings would help grow relationships and allow us time to bond. Having a few offtopic points about our weekends, or hobbies are a great way to learn about each other. This has been a great tweak to our process and I know there is some great back chatter amongst the team. We do our best to stay the course during the stand up, and enjoy some light chatter afterwards.

We also decided to have an “always open” Google hangout for people to join and chat like a party line. This has proven to be a great idea and a lot of fun!

Communication:

We felt that the communication channels we use should be consistent. Sometimes an issue will be discussed in a private chat and not in the team chat, or sometimes an issue is discussed in a ticket and not updated in any chat channels. We agreed to be better at communication and work to moving the conversation into a single channel, and to use our tools to help communication be more visible to the wider team.

Process and tools:

We recognized that there needs to be a little more planning in place to avoid being blocked by the 6 hour time gap between Copenhagen, Africa and NY being online. Let’s use our tools to understand the bottlenecks and try to resolve them and any blockers before EOD.

We have been using Geekbot as our stand-up / stand-down bot in Slack. It’s working really well for us.

Dinner and beer:

One of our engineers, having just returned from a vacation to Croatia, decided to take all of us to a Croatian restaurant in the center of Copenhagen - Restaurant Dubrovnik (don’t worry, I got my fair share of Danish cuisine in). We ate some delicious Croatian fare and started to get into the nitty-gritty of our work.

It was getting a little warm in that restaurant, so we decided to move to a craft beer bar called Taphouse. Taphouse is a pretty cool bar, owned by a Danish engineer with a love for beer and tech (perfect!). It had digital displays for what beers are currently on tap and the lights of the bar would flash when a new keg got hooked up – us geeks loved that. Great selection of beer!

Our discussions went long into the night, and without realizing it an hour had passed with empty glasses due to rich conversation. We decided to call it a night and pick things up in the morning.

Tuesday - Day 4 - Wrap up:

We were off to an early start and continued our conversations around being distributed, our engineering duties and plans for the future. A quick taxi ride to the airport and I was back on my way to NYC… a whirlwind trip but the facetime is invaluable and sets grounds for a great long lasting relationship.

In summary:

Overall, traveling, whether you’re going there or coming here is going to be the catalyst for a great engineering culture with distributed teams. Some of my best friendships were forged through working in a distributed team. I have been on the other side of the fence (living in Stockholm and working for a NYC-based company) and I understand that it takes extra effort to be effective and optimized. If you encourage a great communication culture, and use good processes, and travel, you’re already heading in the right direction.

Some reading / viewing:

• Agile Techniques to Manage Distributed Teams
• 8 tips on how to best manage distributed teams
• Building and Scaling a Distributed and Inclusive Team – Mathias Meyer

Karl Stanton is a Senior Director of Engineering at SeatGeek with a core focus on technical leadership, career development and distributed team management.

Welcome to SeatGeek Employee Spotlights – an opportunity to meet the fantastic folks on our world-class team.

By day, we’re a group of talented developers, designers, marketers, and businessfolk working together to build something new and different. But we are also fans and live event junkies of every kind: diehard sports fans, passionate concert-goers, sophisticated theater enthusiasts, and more. From our lives outside the office and within, we all have interesting stories to tell.

Up next: Lily Dai from our finance team!

Where were you born?
Los Angeles

Have you always lived in NYC?
No, I just moved from LA six months ago.

Where did you go to school?
I went to the University of Southern California and studied Business and Film with a double major in Accounting

Any funny roommate or apartment stories in NYC? Feels like everyone has at least one…
We were friends on Facebook, but we never talked until I reached out to see if she needed a roommate. Can’t think of any funny stories, but we once had 12 pints of Halo Top in the freezer. I called it the “Juice Cleanse”.

How would your friends describe you in 3 words?
Goofy, perceptive, adventurous

Best project you’ve worked on at SeatGeek
We’re currently redoing our forecast for the next 18 months. We’re talking to the product, marketing, and recruiting teams and learning what their needs are. I’m diving a lot deeper into our business. (I know all the secrets.)

What is your dream project to work on at SeatGeek?
My dream project would be to do a deep dive into our snack inventory and find out what’s the most popular. Personally, some of the protein bars taste like chalk, so figuring out which can go and what new ones we can get.

3 “fun facts” about yourself that people would be surprised to know
* You can pour 5 cans of any beverage into a frisbee disk, and I can drink it very quickly. * I fixed cars with my dad for about 10 years. If I’m ever stranded, I can fix the car. * I’ve taught my kitten to fetch.

Favorite place(s) to hang out in NYC
Prospect Park. I’ve been there about 30 times, and I still get lost almost every time. I also love that NYC has a bunch of queer spaces to hang out.

Best vacation you’ve ever taken
My post-college trip with my friends to East Asia. We went to six countries. We had island lifestyle and ancient temples. It was a great way to close out college plus everything was a dollar. We had an oceanfront hotel room for 50 bucks.

Favorite SeatGeek snack
Peanut butter pretzels

Why do you love SeatGeek?
Besides our business model, I love our culture. We had a pride themed happy hour. I almost teared up at how supportive everyone was of our LGBTQ co-workers. It was the perfect example of how supportive and open-minded everyone is here.

Favorite part of the office?
I love the pods in the lounge. They’re super comfy, and you could take a nap if you wanted to while watching the people at Blink work out.

We launched our primary ticketing platform a year ago with one overarching goal: to get more fans to live events. Through our open API approach, an open ticketing ecosystem will create opportunities to increase distribution, empower teams and artists to sell on the platforms of their choice, and eliminate fraud through a process of barcode verification. At the end of the day, our goal is to increase discovery of live events by using the power of the open web, putting tickets where fans are already spending time online.

Today we’re taking a major step toward that goal, and are thrilled to add Facebook as a distribution partner for SeatGeek. As the world’s largest social media network, Facebook powers an incredible amount of delightful discovery of information for their users, whether it is through a news post, long-lost childhood friend, or adorable cat video. We’re stoked to be able to add a piece of the live entertainment world to that.

SeatGeek primary ticketing client Sporting Kansas City will be the first to utilize this new partnership, selling tickets directly to fans through Facebook. We’re excited to expand this partnership to more SeatGeek partners in 2018 as the industry moves toward openness.

While we have already seen great success with SeatGeek distribution partners such as Gametime and TicketNetwork, adding the world’s largest social network to the mix is a game changer for both clients and fans. Exposing inventory directly to fans on Facebook will make for a seamless shopping experience, resulting in more tickets sold, more events attended, and most importantly, more fun for fans.

Our Facebook partnership comes at an exciting time for SeatGeek, coinciding with the recent announcement of SeatGeek Enterprise, our powerful stack of primary ticketing services solidified into one brand. SeatGeek Enterprise promotes our vision of an open ticketing world, in which venues and rightsholders enjoy unprecedented flexibility, transparency and monetization potential. Our open distribution approach is a crucial piece of the SeatGeek Enterprise offering and one of the most important ways we are empowering rightsholders and putting more control in the hands of teams and venues.

We’re thrilled for fans to begin to see this partnership in action in their newsfeeds, as we continue to put the fan first in an industry that for too long has been more than happy to sit on its hands when it comes to innovation.

Welcome to SeatGeek Employee Spotlights – an opportunity to meet the fantastic folks on our world-class team.

By day, we’re a group of talented developers, designers, marketers, and businessfolk working together to build something new and different. But we are also fans and live event junkies of every kind: diehard sports fans, passionate concert-goers, sophisticated theater enthusiasts, and more. From our lives outside the office and within, we all have interesting stories to tell.

Up next: Erin Elsham, on our People Ops team!

Where were you born?
Toledo, Ohio. Well, I was born there – we moved to Iowa after a year, then to Kansas. My dad worked in the agriculture business, so we moved around the midwest with him a bit.

Have you always lived in NYC?
Nope! I packed 2 suitcases and bought a one-way ticket from Kansas City 6 years ago.

It’s kind of funny – I went to college in Kansas as well. I was the odd one out of my really close friends and went to Kansas State, even though I was a Kansas University fan growing up, so that was a funny rivalry thing. New York City, for some reason, draws a lot of Kansas people – there are a couple of well-known KU bars here, and I have some friends from growing up who had moved out here right after school. I went to visit one of my best friends and her husband who were living in Williamsburg, and at the time I had just gotten out of school and was bored with what i was doing, so I moved out there. They were nice to let me stay with them for a bit. I literally packed two suitcases and moved, and six years later here I am.

Where did you go to school?
Kansas State University, Go Jayhawks! Oh wait, that’s KU (I grew up a KU fan). That’s the thing - I went to school at K-State, but was not into K-State sports.

Any funny roommate or apartment stories in NYC? Feels like everyone has at least one…
I’ve always lived in WIlliamsburg, and once we found an apartment with a backyard, we decided we’d stay there for a while. Because it’s a shared backyard, we’re friends with all of the tenants now, so if there’s ever a backyard party or barbecue, everyone is invited. One time, a friend of a friend who does photo shoots and works with modeling agencies asked if she could do a shoot back there. They brought in a really fancy model and props that totally confused the neighborhood, it was pretty funny.

Here’s a picture.

How would your friends describe you in 3 words?
Full of surprises!

I’ve been told that I’m like a stealth bomber – I’m pretty quiet for the most part, but I listen to everything, so I always kind of know what’s going on. Even with friends and family, I’ll know what’s going on and plan something that will surprise them in one way or another. For example, my boyfriend’s 30th birthday: he really likes Wet Hot American Summer, so I planned a themed party in our backyard and made everyone wear 70s camp gear. He had no idea. Also, the SeatGeek portraits – nobody knew that was coming. I like surprising people.

Best project you’ve worked on at SeatGeek?
There are a ton! I love to show off how fun we are as a company, so employee portraits and Life@SeatGeek Instagram account are the top!

What is your dream project to work on at SeatGeek?
Workation was a blast planning last year and I can’t wait to plan an even better event for this year. It’s a huge challenge planning an event for 150 people, but I’ve got some good ideas and am super excited.

What are three “fun facts” about yourself that people would be surprised to know?
I’m an aerialist. I perform trapeze in seasonal circus cabarets in Brooklyn, wear funky costumes and fly in the air – think that’s the most fun fact I’ve got. It’s all volunteer, and I choose my own songs and make the costumes I wear during shows. I’ve been doing it for about 5 and a half years at the same circus school in Williamsburg, am really close with my teacher and have made some amazing friends there.

Favorite place(s) to hang out in NYC?
Backyards and rooftops – there’s nothing better than living in the borough with the view! There are a lot of cool places, but they’re always too crowded – once you find someone with rooftop access, you’ve made it. The beach is fun to go to as well – I usually try Rockaway.

Best vacation you’ve ever taken?
So far, my first “real” vacation as an adult with no plans and no weddings was Puerto Rico with my fiance three years ago. I’ve been a bridesmaid like seven times, and have had a wedding almost every single year, so I haven’t taken a lot of vacations because it was all weddings for a while. Puerto Rico is great – it’s cheap, fun, and the water is clear.

Favorite SeatGeek snack?
Eggs? Do those count? Is that weird?

Why do you love SeatGeek?
It’s the people – everyone’s cool. They’re all so amazing, friendly and welcoming. So many smiles. There is not a single person I don’t feel like I could talk to.

Favorite part of the new office?
SG Portrait wall – sorry, I’m biased! That was the most fun I’ve had on a project. I had to be secretive about it (which I love), and we hired an outside designer for it and worked with them to get the details right and continue to work together for SG 1-year anniversaries!

seatgeek open sourced seatgeek/docker-build-cacher

Builds a service with docker and caches the intermediate stages

At SeatGeek we use Multi-stage Dockerfiles to build the container images that we deploy to production. We have found them to be a great and simple way of building projects with dependencies in different languages or tools. If you are not familiar with multi-stage Dockerfiles, we recommend you take a look at this blog post.

In our first days of using them in our build pipeline, we found a few shortcomings that were making our deploys take longer than they should have. We traced these shortcomings to a missing key feature: It is not possible to carry statically generated cache files from one build to another once certain source files in the project change.

For example when building our frontend pipeline we have to invoke yarn first to get all the npm packages. But this command can only be executed after adding the yarn.lock and package.json files to the Docker container. Because of the nature of how Docker caching works, this meant that each time those files are modified, the node_modules folder cached in previous built was also trashed. As you may already know, building that folder from scratch is not a cheap operation.

Here’s an example that illustrates the issue.

Imagine you create a generic Dockerfile for building node projects

1
2
3
4
5
6
7
8
9
10
11
12
13
14

FROM nodejs

RUN apt-get install nodejs yarn

WORKDIR /app

# Whenever this image is used execute these triggers
ONBUILD ADD package.json yarn.lock .

# Dowanload npm packages
ONBUILD RUN yarn

# Build the assets pipeline
ONBUILD RUN yarn run dist

We can now build and tag a Docker image with for building yarn based projects

1

docker build -t nodejs-build .

The tagged image can be used in a generic way like this:

1
2
3
4
5
6
7
8
9
10
11

# Automatically build yarn dependencies
FROM nodejs-build as nodedeps

# Build the final container image
FROM scratch

# Copy the generated app.js from yarn run dist
COPY --from=nodedeps /app/app.js .

# Rest of the Dockerfile
...

So far so good, we have build a pretty lean docker image that discards the node_modules folder and only keeps the final artifact. For example a set of js bundles from a React application.

It’s also very fast to build! This is because each individual step is cleverly cached by Docker during the build processes. That is, as long as none of the steps or files used in the step have changed.

And that’s exactly where the problem is: Whenever the package.json or yarn.lock files change, Docker will trash all the files in node_modules directory as well as the cached yarn packages and will start downloading from scratch, linking and building every single dependency.

That’s far from ideal, as it takes significant time to rebuild all dependencies. What if we could make a change to the process so that changes to those files do not bust the yarn cache? It turns out we can!

Enter docker-build-cacher`

We have built a slim utility that helps overcome the problem by providing a way to build the Dockerfile and cache all of the intermediate stages. On subsequent builds, it will make sure that the static cache files that were generated during previous builds will also be present.

The effect it has should be obvious: your builds will be consistently fast, at the cost of a bit of extra disk space.

Building and caching is done in separate steps. The first step is a replacement for the docker build command and the second step is the cache persisting phase.

1
2
3
4
5
6

export APP_NAME=fancyapp
export GIT_BRANCH=master # Used to internally tag cache artifacts
export DOCKER_TAG=fancyapp:latest

docker-build-cacher build # This will build the docker file
docker-build-cacher cache # This will cache each of the stage results separately

How It Works

The docker-build-cacher tool works by parsing the Dockerfile and extracting COPY or ADD instructions nested inside ONBUILD for each of the stages found in the file.

It will compare the source files present in such COPY or ADD instructions to check for changes. If it detects changes, it rewrites the Dockerfile on the fly, such that FROM directives in each of the stages use the locally cached images instead of the original base image.

The effect this FROM swap has is that disk state for the image is preserved between builds.

docker-build-cacher is available now on GitHub under the BSD 3-Clause License.

Make sure to grab the binary files from the releases page

If you think these kinds of things are interesting, consider working with us as a Software Engineer at SeatGeek. Or, if backend development isn’t your thing, we have other openings in engineering and beyond!

seatgeek open sourced seatgeek/hashi-helper

Disaster Recovery and Configuration Management for Consul and Vault

This post is the first of a bonus series on tooling for the Hashi-stack - Consul, Nomad, Vault. We also recommend our previous series on using Vault in production.

Configuration Management for your Configuration

In our initial Vault rollout, one pain-point we quickly came across was managing Consul and Vault configuration. We use both Hashicorp tools for managing secrets and access control across our entire infrastructure, and knowing what configuration was setup where in each cluster is quite critical. On top of this, disaster recovery quickly became an issue we knew we needed to tackle before a more broad roll-out.

One of our Systems Engineers started looking at what our needs were in regards to properly managing Consul and Vault configuration, and came up with a wonderful workflow through the use of a tool we like to call hashi-helper. We use hashi-helper internally to manage multiple clusters in different environments via a git repository that contains our canonical configuration. It is now pretty trivial for us to:

• Standup a vault cluster using our normal provisioning toolchain.
• Unseal vault via gpg key via the normal vault tooling, or keybase via hashi-helper vault-unseal-keybase.
• Provision mounts, policies, and secrets using hashi-helper vault-push-all.
• Provision custom registered consul services via hashi-helper consul-push-services.
• Manage all of our configuration via either blackbox gpg-encrypted files or AWS KMS encryption through tooling such as sm.

Example Workflow

For those curious about what a typical workflow might look like, the following directory structure may be suitable for a typical organization with a Consul/Vault cluster per-environment:

• /${env}/apps/${app}.hcl (encrypted) Vault secrets for an application in a specific environment.
• /${env}/auth/${name}.hcl (encrypted) Vault auth backends for an specific environment ${env}.
• /${env}/consul_services/${type}.hcl (cleartext) List of static Consul services that should be made available in an specific environment ${env}.
• /${env}/databases/${name}/_mount.hcl (encrypted) Vault secret backend configuration for an specific mount ${name} in ${env}.
• /${env}/databases/${name}/*.hcl (cleartext) Vault secret backend configuration for a specific Vault role belonging to mount ${name} in ${env}.

Here is an example for managing Vault Secrets:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# environment name must match the directory name
environment "production" {

  # application name must match the file name
  application "XXXX" {

    # Vault policy granting any user with policy XXXX-read-only read+list access to all secrets
    policy "XXXX-read-only" {
      path "secret/XXXX/*" {
        capabilities = ["read", "list"]
      }
    }

    # an sample secret, will be written to secrets/XXXX/API_URL in Vault
    secret "API_URL" {
      value = "http://localhost:8181"
    }
  }
}

hashi-helper is available now on GitHub under the BSD 3-Clause License.

If you think these kinds of things are interesting, consider working with us as a Software Engineer at SeatGeek. Or, if backend development isn’t your thing, we have other openings in engineering and beyond!

We believe that modern technology can be a force for good in live entertainment. That’s why we created SeatGeek Open – a platform which enables teams, artists, and fans to buy and sell tickets across the open web. Openness means harnessing the power of the internet to create better experiences.

In order to launch SeatGeek Open last August, we needed to find a partner that shared our vision, and whose powerful box office technology would enable a true API-driven entertainment platform. We did an exhaustive worldwide search, but in the end the selection of Israel-based TopTix was remarkably clear and unequivocal. They brought unprecedented technology and incredible talent to SeatGeek Open.

Today, we’re over-the-moon excited to announce that TopTix is joining SeatGeek.

TopTix was started in 2000 in Karmiel, Israel by two remarkable entrepreneurs, Eli Dagan and Yehuda Yuval. Since then, they’ve grown to 115 employees across four continents. As we’ve gotten to know the people at TopTix, we’ve been struck by how similar they are to the people at SeatGeek – bright, humble, and driven by a core belief in the transformative power of technology. We couldn’t be more stoked that this extraordinary group is now part of our team.

TopTix’s primary ticketing platform, called SRO, is by far the strongest and most modern backend ticketing platform on earth. It serves more than 500 venues, and processes over 80 million tickets a year in 16 countries across the globe. Current TopTix clients range from museums and theaters to festivals and sports teams, including well-known organizations such as the Ravinia Festival, the Royal Dutch Football Association, West End theaters, and several English Premier League clubs.

We’re excited to continue to support all current TopTix clients from TopTix’s seven global offices. The TopTix engineering team will remain 100% focused on SRO, working to further extend the platform with all of SeatGeek’s resources now behind them.

We believe the future of ticketing is open. Openness means enabling teams and venues to fill their venues. Most importantly, it means making fans happier.

Joining forces with TopTix will allow us to make SeatGeek Open an even more powerful platform and to create more great experiences for fans. SeatGeek and TopTix have already helped Sporting Kansas City, our first client, sell more tickets and reach new fans, while giving supporters easy access to the games they love. Now that we’re officially operating as one, we’re feverishly excited about the potential this unlocks and our future together as a single united company.

This post is the second of a two-part series on using Vault in production. Both posts are slightly redacted forms of internal documentation. This post covers day-to-day usage of Vault, while the previous post covered our specific workflow.

Please note that some of the referenced tooling is not currently publicly available.

Not all of our practices will apply to your situation, and there are cases certainly where our setup may be suboptimal for your environment.

Using Vault

This is the tl;dr you were looking for…

For Developers

Developers should expect credentials to live in environment variables that can be loaded into an app when needed. These credential names are specified within an app.json manifest file, and Ops should be contacted to place their values in Vault. The deploy process currently uses the vault api to retrieve appropriate credentials for a service, transparent to the development staff.

The basic flow for reading or writing credentials is the following:

1. Login to Vault and receive a token
2. Make a request with the token to read or write

Schema

Secrets accordingly to the following convention:

1

secret/ENVIRONMENT/APP/KEY value=VALUE

Login

Note: Developer Vault logins require a Github Access Token (https://github.com/blog/1509-personal-api-tokens)

CLI:

1

vault auth -method=github token=GITHUB_ACCESS_TOKEN

Upon success, a Vault token will be stored at $HOME/.vault-token.

HTTP_API:

1
2
3

curl \
  -L http://vault.service.consul:8200/v1/auth/github/login \
  -d '{ "token": "GITHUB_ACCESS_TOKEN" }'

List

CLI:

1

vault list secret/path/to/bucket

This will use the token at $HOME/.vault-token if it exists. Otherwise, you will need to login via vault auth first.

HTTP API:

1
2
3
4

curl \
    -H "X-Vault-Token: VAULT_TOKEN" \
    -X GET \
    -L http://vault.service.consul:8200/v1/secret/path/to/bucket?list=true

Read

CLI:

1

vault read secret/path/to/key

This will use the token at $HOME/.vault-token if it exists. Otherwise, you will need to login via vault auth first.

HTTP API:

1
2
3
4

curl \
    -H "X-Vault-Token: VAULT_TOKEN" \
    -X GET \
    -L http://vault.service.consul:8200/v1/secret/path/to/key

Write

CLI:

1
2

vault write secret/path/to/key \
    value=THISISASECRET

This will use the token at $HOME/.vault-token if it exists. Otherwise, you will need to login via vault auth first.

HTTP API:

1
2
3
4
5
6

curl \
    -H "X-Vault-Token: VAULT_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{"value":"THISISASECRET"}' \
    -L http://vault.service.consul:8200/v1/secret/path/to/key

For Admins

Initialization

Required reading: https://www.vaultproject.io/intro/getting-started/deploy.html

vault init is used to bootstrap a new Vault cluster. This generates a number of keys and requires a majority threshold of these keys in order to unseal Vault (more on unsealing below).

For SeatGeek’s Vault cluster, vault init has been run as follows:

1
2
3
4

vault init \
    -key-shares=5
    -key-threshold=3
    -pgp-keys = "keybase:OPS1,keybase:OPS2,keybase:OPS3,keybase:OPS4,keybase:OPS5"

Note: Local public key files can also submitted for the pgp-keys option

Initializing Vault this way leverages its support for authorizing users to be able to unseal Vault via their private GPG keys. This method was chosen as we already using blackbox to encrypt secrets within certain repositories.

When vault is initialized, an unseal tokens are printed out for each pgp key specified. The order of these keys matches the order in which the pgp keys were specified, and each can only be decrypted by the corresponding pgp key. Those unseal tokens should be securely distributed to the corresponding operations engineer and stored in a secure fashion. Loss of keys exceeding the threshold will result in a loss of ability to unseal the cluster.

Vault should only need to be reinitialized if all of the data in lost, which for SeatGeek would be in a loss of the Consul Cluster.

Unsealing

Vault boots up in a sealed state, and in this state no requests are answered. Each machine within a Vault cluster can be in a sealed or active state, and all must be unsealed before answering any requests. There is also a standby state, in which the machine is unsealed but not primary, and is ready for failover if the currently active primary dies.

A Vault machine can be unsealed via the following command:

1
2

export VAULT_ADDR="http://ip.for.vault.instance:8200"
echo "$VAULT_UNSEAL_KEY" | base64 -D | keybase pgp decrypt | xargs vault unseal

The VAULT_UNSEAL_KEY is specific to each user who was specified in the vault init command. All unseal keys were distributed at the time of initialization.

Note: This requires having the vault binary installed locally.

Root Token

An initial root token is created when the Vault Cluster is initialized. Other root tokens are created from this token, and as such, if a root token is needed it must be created by an existing holder of a root token.

A new root token can be created from an existing root token via the following command:

1

vault token-create -metadata "name=ADMIN_NAME" -display-name="ADMIN_USER_NAME" -orphan -no-default-policy

Note: You must be logged in with a root token in order to run this command

In the emergency case that a new root token needs to be created, the following command can be run:

1

vault generate-root

This operation requires a majority of unseal key holders to execute.

Note: At this time of writing, Vault 0.6.2 has deprecated this workflow surrounding root tokens and our usage is subject to change in the future.

Provisiong a New Service

When provisioning a new service, secrets can simply be written to the appropriate bucket (secret/ENVIRONMENT/APP_NAME/KEY value=VALUE). Everything under secret/ is a “key” and all necessary paths will be created.

Whitelisting a New Service

On first boot of infrastructure, the Jenkins Whitelist deploy job, a dependency of the Jenkins Configure job, will be run. If not already, the new application will be created and its corresponding IAM Role will be whitelisted. This job also ensures all other IAM Role whitelists are up to date in Vault.

To ensure that Jenkins has the correct permissions, a special role allowing it access to write auth and policy documents should be written to Vault. The following can be used to create the policy, which is stored with all other custom vault policies:

1

vault policy-write ENV-jenkins data/vault/policies/env-jenkins-policy.hcl

Policies must be audited on a regular basis, consistent with all other internal auditing processes.

When used in contexts that do not easily support passing roles, you can create a vault token for this or any policy. The following creates a renewable token that is valid for 60 seconds:

1
2

TOKEN_ID="$(uuidgen)"
vault token-create -policy="ENV-jenkins" -display-name="ENV-jenkins" -id="$TOKEN_ID" -ttl=60s

Rekey and Key Rotation

Vault’s usage of unseal keys is based on Shamir’s secret sharing algorithm.

https://en.wikipedia.org/wiki/Shamir%27s_Secret_Sharing

The vault rekey command allows for the recreation of unseal keys as well as changing the number of key shares and key threshold. This is useful for adding or removing Vault admins.

The vault rotate command is used to change the encryption key used by Vault. This does not require anything other than a root token. Vault will continue to stay online and responsive during a rotate operation.

Disaster Response

In the case of an emergency, Vault should be sealed immediately via:

1

vault seal

This will prevent any actions or requests to be performed against the Vault server, and gives time to investigate the cause of the issue and an appropriate solution.

1. A secret stored in Vault is leaked A new secret should be generated and replaced in Vault, with a key rotation following.
2. Vault user credentials are leaked The user credentials should be revoked and a key rotation should be performed.
3. Vault unseal keys are leaked A rekey should be performed.

If you think these kinds of things are interesting, consider working with us as a Software Engineer at SeatGeek. Or, if backend development isn’t your thing, we have other openings in engineering and beyond!