**TL;DR:** We built OAuth2 authentication and authorization layer via nginx middleware using lua. If you intend on performing this, read the docs, automate what you can, and carry rations.

As SeatGeek has grown over the years, we’ve amassed quite a few different administrative interfaces for various tasks. We regularly build new modules to export data for news outlets, our own blog posts, infographics, etc. We also regularly build internal dev tools to handle things such as deployment, operations visualization, event curation etc. In the course of doing that, we’ve also used and created a few different interfaces for authentication:

• Github/Google Oauth
• Our internal SeatGeek User System
• Basic Auth
• Hardcoded logins

Obviously, this is subpar. The myriad of authentication systems makes it difficult to abstract features such as access levels and general permissioning for various datastores.

One System to auth them all

We did a bit of research about what sort of setup would solve our problems. This turned up Odin, which works well for authenticating users against Google Apps. Unfortunately, it would require us to use Apache, and we are pretty married to Nginx as a frontend for our backend applications.

As luck would have it, I came across a post by mixlr referencing their usage of Lua at the Nginx level for:

• Modifying response headers
• Rewriting requests internally
• Selectively denying access to hosts based on IP

The last one in that set seemed interesting. Thus began the journey in package management hell.

Building Nginx with Lua Support

Lua support for Nginx is not distributed with the core Nginx source, and as such any testing would require us to build pacakges for both OS X–for testing purposes–and Linux - for deployment.

Custom Nginx for OS X

For Mac OS X, we promote the usage of the Homebrew for package management. Nginx does not come with many modules enabled in the base formula for one very good reason:

The problem is that NGINX has so many options that adding them all to a formula would be batshit insane and adding some of them to a formula opens the door to adding all of them and associated insanity. - Charlie Sharpsteen, @sharpie

So we needed to build our own. Preferably in a manner that would allow further customization in case we need more features in the future. Fortunately, modifying homebrew packages is quite straightforward.

We want to have a workspace for working on the recipe:

cd ~
mkdir -p src
cd src

Next we need the formula itself. You can do one of the following to retrieve it:

• Go spelunking in your HOMEBREW_PREFIX directory - usually /usr/local - for the nginx.rb
• Have the github url memorized as an api and wget https://raw.github.com/mxcl/homebrew/master/Library/Formula/nginx.rb
• Simply output your formula using brew cat nginx > nginx.rb

If we brew install ./nginx.rb, that will install the recipe contained within that file. Since this is a completely custom nginx installation, we’ll want to rename the formula so that future brew upgrade calls do not nix our customizations.

mv nginx.rb nginx-custom.rb
cat nginx-custom.rb | sed 's/class Nginx/class NginxCustom/' >> tmp
rm nginx-custom.rb
mv tmp nginx-custom.rb

We’re now ready to add new modules to our compilation step. Thankfully this is easy, we just need to collect all the custom modules from passed arguments to the brew install command. The following bit of ruby takes care of this:

# Collects arguments from ARGV
def collect_modules regex=nil
    ARGV.select { |arg| arg.match(regex) != nil }.collect { |arg| arg.gsub(regex, '') }
end

# Get nginx modules that are not compiled in by default specified in ARGV
def nginx_modules; collect_modules(/^--include-module-/); end

# Get nginx modules that are available on github specified in ARGV
def add_from_github; collect_modules(/^--add-github-module=/); end

# Get nginx modules from mdounin's hg repository specified in ARGV
def add_from_mdounin; collect_modules(/^--add-mdounin-module=/); end

# Retrieve a repository from github
def fetch_from_github name
    name, repository = name.split('/')
    raise "You must specify a repository name for github modules" if repository.nil?

    puts "- adding #{repository} from github..."
    `git clone -q git://github.com/#{name}/#{repository} modules/#{name}/#{repository}`
    path = Dir.pwd + '/modules/' + name + '/' + repository
end

# Retrieve a tar of a package from mdounin
def fetch_from_mdounin name
    name, hash = name.split('#')
    raise "You must specify a commit sha for mdounin modules" if hash.nil?

    puts "- adding #{name} from mdounin..."
    `mkdir -p modules/mdounin && cd $_ ; curl -s -O http://mdounin.ru/hg/#{name}/archive/#{hash}.tar.gz; tar -zxf #{hash}.tar.gz`
    path = Dir.pwd + '/modules/mdounin/' + name + '-' + hash
end

The above helper methods allow us to specify new modules to include on the command line and retrieve the modules from their respective locations. At this point, we’ll need to modify the nginx-custom.rb recipe to include the flags and retrieve the packages, around line 58:

nginx_modules.each { |name| args << "--with-#{name}"; puts "- adding #{name} module" }
add_from_github.each { |name| args <<  "--add-module=#{fetch_from_github(name)}" }
add_from_mdounin.each { |name| args <<  "--add-module=#{fetch_from_mdounin(name)}" }

At this point, we can compile a custom version of nginx with our own modules.

brew install ./nginx-custom.rb \
    --add-github-module=agentzh/chunkin-nginx-module \
    --include-module-http_gzip_static_module \
    --add-mdounin-module=ngx_http_auth_request_module#a29d74804ff1

We’ve provided this formula as a tap for you convenience at seatgeek/homebrew-formulae.

Custom Nginx for Debian

We typically deploy to some flavor of Debian–usually Ubuntu–for our production servers. As such, it would be nice to simply run dpkg -i nginx-custom to have our customized package installed. The steps to doing so are relatively simple once you’ve gone through them.

Some notes for those researching custom debian/ubuntu packaging:

• It is possible to get the debian package source using apt-get source PACKAGE_NAME
• Debian package building is generally governed by a rules file, which you’ll need some sed-fu to manipulate
• You can update deb dependencies by modifying the control file. Note that there are some meta-dependencies specified herein that you’ll not want to remove, but these are easy to identify.
• New releases must always have a section in the changelog, otherwise the package may not be upgraded to because it may have already been installed. You should use tags in the form +tag_name to idenfity changes from the baseline package with your own additions. I also personally append a number - starting from 0 - signifying the release number of the package.
• Most of these changes can be automated in some fashion, but it appears as though there are no simple command line tools for creating custom releases of packages. That’s definitely something we’re interested in, so feel free to link to tooling to do so if you know of anything.

While running this process is great, I have built a small bash script that should automate the majority of the process. It is available as a gist on github.

It only took 90 nginx package builds before I realized the process was scriptable.

OAuth ALL the things

Now that it is possible to test and deploy a Lua script embedded within Nginx, we can move on to actually writing some Lua.

The nginx-lua module provides quite a few helper functions and variables for accessing most of Nginx’s abilities, so it is quite possible to force OAuth authentication via the access_by_lua directive provided by the module.

When using the *_by_lua_file directives, nginx must be reloaded for code changes to take effect.

I built a simple OAuth2 provider for SeatGeek in NodeJS. This part is simple, and you can likely find something off the box in your language of choice.

Next, our OAuth API uses JSON for handling token, access level, and re-authentication responses, so we needed to install the lua-cjson module.

# install lua-cjson
if [ ! -d lua-cjson-2.1.0 ]; then
    tar zxf lua-cjson-2.1.0.tar.gz
fi
cd lua-cjson-2.1.0
sed 's/i686/x86_64/' /usr/share/lua/5.1/luarocks/config.lua > /usr/share/lua/5.1/luarocks/config.lua-tmp
rm /usr/share/lua/5.1/luarocks/config.lua
mv /usr/share/lua/5.1/luarocks/config.lua-tmp /usr/share/lua/5.1/luarocks/config.lua
luarocks make

My OAuth provider uses the query-string for sending error messages on authentication, so I needed to support that in my Lua script:

local args = ngx.req.get_uri_args()
if args.error and args.error == "access_denied" then
    ngx.status = ngx.HTTP_UNAUTHORIZED
    ngx.say("{\"status\": 401, \"message\": \""..args.error_description.."\"}")
    return ngx.exit(ngx.HTTP_OK)
end

Now that we’ve handled our base error case, we’ll set a cookie for the access token. In my case, the cookie expires before the access token actually expires so that I can use the cookie to renew my access token.

local access_token = ngx.var.cookie_SGAccessToken
if access_token then
    ngx.header["Set-Cookie"] = "SGAccessToken="..access_token.."; path=/;Max-Age=3000"
end

At this point, we’ve handled error responses from the api, and stored the access_token away for later retrieval. We now need to ensure the oauth process actually kicks off. In this block, we’ll want to:

• Start the oauth process if there is no access_token stored and we are not in the middle of it
• Retrieve the user access_token from the oauth api if the oauth access code is present in the query string arguments
• Deny users with invalid access codes

Reading the docs on available nginx-lua functions and variables can clear up some issues, and perhaps show you various ways in which you can access certain request/response information

At this point we need to retrieve data from our api to retrieve an access token. Nginx-lua provides the ngx.location.capture method, which can be used to retrieve the response from any internal endpoint within redis. This means we cannot call something like [https://seatgeek.com/ncaa-football-tickets?oq=ncaa+football+tickets(https://seatgeek.com/ncaa-football-tickets) directly, but would need to use proxy_pass in order to wrap the external url in an internal endpoint.

My convention for these endpoints is to prefix them with an _ (underscore), and normally blocked against direct access.

-- first lets check for a code where we retrieve
-- credentials from the api
if not access_token or args.code then
    if args.code then
        -- internal-oauth:1337/access_token
        local res = ngx.location.capture("/_access_token?client_id="..app_id.."&client_secret="..app_secret.."&code="..args.code)

        -- kill all invalid responses immediately
        if res.status ~= 200 then
            ngx.status = res.status
            ngx.say(res.body)
            ngx.exit(ngx.HTTP_OK)
        end

        -- decode the token
        local text = res.body
        local json = cjson.decode(text)
        access_token = json.access_token
    end

    -- both the cookie and proxy_pass token retrieval failed
    if not access_token then
        -- Track the endpoint they wanted access to so we can transparently redirect them back
        ngx.header["Set-Cookie"] = "SGRedirectBack="..nginx_uri.."; path=/;Max-Age=120"

        -- Redirect to the /oauth endpoint, request access to ALL scopes
        return ngx.redirect("internal-oauth:1337/oauth?client_id="..app_id.."&scope=all")
    end
end

At this point in the Lua script, you should have a - hopefully! - valid access_token. We can use this against your whatever endpoint you have setup to provide user information. In my endpoint, I respond with a 401 status code if the user has zero access, 403 if their token is expired, and access_level information via a simple integer in the json response.

-- ensure we have a user with the proper access app-level
-- internal-oauth:1337/accessible
local res = ngx.location.capture("/_user", {args = { access_token = access_token } } )
if res.status ~= 200 then
    -- delete their bad token
    ngx.header["Set-Cookie"] = "SGAccessToken=deleted; path=/; Expires=Thu, 01-Jan-1970 00:00:01 GMT"

    -- Redirect 403 forbidden back to the oauth endpoint, as their stored token was somehow bad
    if res.status == 403 then
        return ngx.redirect("https://seatgeek.com/oauth?client_id="..app_id.."&scope=all")
    end

    -- Disallow access
    ngx.status = res.status
    ngx.say("{"status": 503, "message": "Error accessing api/me for credentials"}")
    return ngx.exit(ngx.HTTP_OK)
end

Now that we’ve verified that the user is indeed authenticated and has some level of access, we can check their access level against whatever we define is the access level for the current endpoint. I personally delete the SGAccessToken at this step so that the user has the ability to log into a different user, but that is up to you.

local json = cjson.decode(res.body)
-- Ensure we have the minimum for access_level to this resource
if json.access_level < 255 then
    -- Expire their stored token
    ngx.header["Set-Cookie"] = "SGAccessToken=deleted; path=/; Expires=Thu, 01-Jan-1970 00:00:01 GMT"

    -- Disallow access
    ngx.status = ngx.HTTP_UNAUTHORIZED
    ngx.say("{\"status\": 403, \"message\": \"USER_ID"..json.user_id.." has no access to this resource\"}")
    return ngx.exit(ngx.HTTP_OK)
end

-- Store the access_token within a cookie
ngx.header["Set-Cookie"] = "SGAccessToken="..access_token.."; path=/;Max-Age=3000"

-- Support redirection back to your request if necessary
local redirect_back = ngx.var.cookie_SGRedirectBack
if redirect_back then
    ngx.header["Set-Cookie"] = "SGRedirectBack=deleted; path=/; Expires=Thu, 01-Jan-1970 00:00:01 GMT"
    return ngx.redirect(redirect_back)
end

Now we just need to tell our current app who is logged in via some headers. You can reuse REMOTE_USER if you have some requirement that this replace basic auth, but otherwise anything is fair game.

-- Set some headers for use within the protected endpoint
ngx.req.set_header("X-USER-ACCESS-LEVEL", json.access_level)
ngx.req.set_header("X-USER-EMAIL", json.email)

I can now access these http headers like any others within my applications, replacing hundreds of lines of code and hours of work reimplementing authentication yet again.

Nginx and Lua, sitting in a tree

At this point, we should have a working lua script that we can use to block/deny access. We can place this into a file on disk and then use access_by_lua_file to use it within our nginx site. At SeatGeek, we use Chef to template out config files, though you can use Puppet, Fabric, or whatever else you’d like to do so.

Below is the simplest nginx site you can use to get this entire thing running. You’ll also want to check out the access.lua - available here - which is the compilation of the above lua script.

# The app we are proxying to
upstream production-app {
  server localhost:8080;
}

# The internal oauth provider
upstream internal-oauth {
  server localhost:1337;
}

server {
  listen       80;
  server_name  private.example.com;
  root         /apps;
  charset      utf-8;

  # This will run for everything but subrequests
  access_by_lua_file "/etc/nginx/access.lua";

  # Used in a subrequest
  location /_access_token { proxy_pass http://internal-oauth/oauth/access_token; }
  location /_user { proxy_pass http://internal-oauth/user; }

  location / {
    proxy_set_header  X-Real-IP  $remote_addr;
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  Host $http_host;
    proxy_redirect    off;
    proxy_max_temp_file_size 0;

    if (!-f $request_filename) {
      proxy_pass http://production-app;
      break;
    }
  }

}

Further Considerations

While this setup has worked really well for us, I’d like to point out some shortcomings:

• The above code is a simplification of our access_by_lua script. We also handle POST request saving, inject JS into pages to renew the session automatically, handle token renewal etc. You may not need these features, and in fact, I didn’t think I’d need them until we started testing this system on our internal systems.
• We had some endpoints which were available via basic auth for certain background tasks. These had to be reworked so that the data was retrieved from an external store, such as S3. Be aware that this may not always be possible, so oauth may not be the answer in your case.
• Oauth2 was simply the standard I chose. In theory, you could use Facebook Auth to achieve similar results. You may also combine this approach with rate-limiting, or storing various access levels in a datastore such as redis for easy manipulation and retrieval within your Lua script. If you were really bored, you could reimplement Basic Auth within Lua, it’s just up to you.
• There are no test harnesses for systems such as these. Test-junkies will cringe when they realize it’s going to be integration testing for a while. You can likely rerun the above by injecting variable mocks into the global scope and then executing scripts, but it’s not the ideal setup.
• You still need to modify apps to recognize your new access headers. Internal tools will be easiest, but you may need to make certain concessions for vendor software.

tl;dr with links

The above blog post combined nginx-lua and our internal oauth provider to enable using OAuth for access control to our infrastructure.

Links

• SeatGeek Homebrew Formulae with customizable nginx
• nginx_release.sh for building nginx debs
• access.lua and nginx-site

Further Reading

• HttpLuaModule
• proxy_pass, like mod_proxy, but for nginx
• Lua usage at Mixlr
• OAuth for Apache
• Homebrew OS X Package Manager

Also

SeatGeek is hiring UI Developers and Web Engineers. Your first tasks will be to make my OAuth application pretty and write some tests for a bit of Lua code… (kidding)

If you have any questions about the project just let us know in the comments!