David Beath (Posts about hosting)

Nginx and ELB Proxy Protocol Forwarding

David Beath — Mon, 17 Apr 2017 10:35:06 GMT

Recently I noticed that an application I'm hosting on AWS Elastic Beanstalk wasn't logging the Client IP Address of my users. After a lot of digging I found the issue was to do with the configuration of the Nginx servers running on the EC2 instances and in my application's Docker container, and in the TCP passthrough configuration of the Elastic Load Balancer.

So here's a quick post about my particular setup and the configuration I used to fix this issue of Client IP Addresses not being forwarded.

Elastic Beanstalk Setup

In a little more detail, my setup consists of an application hosted within a Docker container running on EC2 instances configured by Elastic Beanstalk, which sit behind an Elastic Load Balancer (ELB). Because of issues with Certificate support in ELB, I'm not using AWS's Certificate Manager or using the built-in SSL termination in ELB, but instead using TCP passthrough on the ELB, and using a custom Nginx configuration on each EC2 instance to terminate the SSL connection. The Docker container on each instance also has it's own Nginx configuration, which proxies the application server, in my case UWSGI.

The steps that a request takes to get to the application are:

User makes an HTTPS request.
ELB uses TCP passthrough to pass the encrypted request to an EC2 instance.
Nginx in EC2 decrypts the HTTPS request and passes the HTTP to it's Docker container.
The Nginx server on Docker proxies the request to UWSGI.
The application hosted by UWSGI handles the request.

In all, the parts that you need to configure to forward the Client IP Address are the TCP passthrough on ELB and each of the two Nginx servers.

Proxy Protocol

The first problem is that if you're using a TCP load balancer to pass through the request, the load balancer will not add an X-Forwarded-For header, and so the downstream Nginx server will only see the IP Address of the load balancer. The problem and part of the solution is described in better detail here. The linked solution is a little different from what I'm going to show below, because I'm proxying the request another hop down the line, but please read the link to better understand what's going on.

The first step is to enable Proxy Protocol support on the Elastic Load Balancer. AWS has a good tutorial on how to do that.

Once the ELB has Proxy Protocol support enabled, you can configure the Nginx server on the EC2 instances. The relevant parts of the configuration look something like this:

log_format elb_log '$proxy_protocol_addr - $remote_user'
                    '[$time_local] "$request" '
                    '$status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent"';

server {
    listen 443 ssl http2 proxy_protocol;

    server_name localhost;

    set_real_ip_from  172.31.0.0/20;
    real_ip_header    proxy_protocol;

    access_log /var/log/nginx/elb-access.log elb_log;

    # SSL Certificate settings go here.

    location / {
        proxy_pass          https://docker;
        proxy_http_version  1.1;

        proxy_set_header Connection         "";
        proxy_set_header Host               $host;
        proxy_set_header X-Real-IP          $proxy_protocol_addr;
        proxy_set_header X-Forwarded-For    $proxy_protocol_addr;
        proxy_set_header X-Forwarded-Proto  $scheme;
    }
}

Here's what's happening in the above config:

The listen directive now has proxy_protocol.
set_real_ip_from tells Nginx the real CIDR range of addresses that the ELB is using.
real_ip_header tells us that we're using the proxy_protocol value to access the client headers.
X-Real-IP and X-Forwarded-For use the $proxy_protocal_addr, which is the client IP information taken from the proxy protocol.
The elb_log format is now using the $proxy_protocal_addr for the client IP.

Also note that the listen directive can support both http2 and proxy_protocol, so if your version of Nginx supports HTTP/2 then you can enable that just fine. However, the proxy_http_version directive in this case is still 1.1, as it doesn't make sense to use HTTP/2 for proxy backends.

If you deployed the above config, you should now see that Nginx server on the EC2 instance is now logging the correct client IP Address.

Multiple Nginx Proxies

Now that we have the Nginx server on the EC2 instance receiving the correct Client IP from the Load Balancer, we need to foward that IP on to the Nginx server in the Docker container.

The X-Forwarded-For and X-Real-IP directives in the above config will properly forward the Client IP from the EC2 instance's Nginx server. However, Nginx appends each proxy's IP address to the X-Forwarded-For header, as described in more detail here.

The solution to this is in the last Nginx proxy configuration is to include the IP address ranges of all previous known proxies in the set_real_ip_from directive. You can then set the real_ip_header directive to X-Forwarded-For. The final piece is to set real_ip_recursive to on, which will return the leftmost (original) IP instead of the rightmost (most recent) IP from the X-Forwarded-For header.

My Nginx configuration for the Docker container now looks like this:

upstream app {
    server unix:///tmp/uwsgi.sock;
}

server {
    listen 80 default_server;

    set_real_ip_from    172.31.0.0/20;    # IP of the ELB
    set_real_ip_from    172.17.0.0/20;    # IP of the EC2 instance
    real_ip_header      X-Forwarded-For;
    real_ip_recursive   on;

    location / {
        uwsgi_pass app;
        include uwsgi_params;
    }
}

You can see that I've included the IP ranges of both the ELB and the EC2 instance. With a config similar to this you should now see the Nginx instance in the Docker container log the correct Client IP, and pass that IP through to your application.

Installing Tiny Tiny RSS from scratch

David Beath — Wed, 02 Oct 2013 03:43:39 GMT

Considering that a couple of times now I've been a proponent of using an RSS reader, I figured it was time I wrote a tutorial on how to install Tiny Tiny RSS, using Nginx as the server and PostgreSQL for the database. While as with any piece of software there are a number of tutorials and guides already out there, I've found that none of them provided a complete instruction set that didn't have me running into and searching for solutions to multiple errors. To that end, this tutorial aims to both provide as complete a set of fool-proof instructions as I can make, both for newbies to self-hosting, and simply as a reference for my future self should I need to do this again.

This set of instructions assumes that you're running a Debian based Linux distro. I've tested this setup on a clean Ubuntu Server 12.04 virtual machine, and the settings are roughly the same as I'm using on my Debian server. It's assumed that you either know how to set up a server already, or are capable of reading a tutorial to do so (If not, then why are you reading this? Go use Feedly). There are some very good tutorials at Digital Ocean and Linode, and I'd recommend either of them for hosting purposes.

Installation

On to the install. While not strictly necessary, the first thing I recommend is to make sure that your server is using the correct time, so install ntp to keep the system clock updated.

> sudo apt-get install ntp

For most of the commands in this tutorial you'll need super user privileges, so you can either run them as root (not recommended) or use sudo like I've used in this tutorial. You'll also need to edit a bunch of files in a text editor. I've used nano for this tutorial because it's easier to use and you can just copy the commands straight, but I usually use vim.

PHP

Unfortunately, despite the fact that PHP sucks (seriously, read the article, even if you disagree it's an interesting read), Tiny Tiny RSS is built with PHP. However, just because a language sucks doesn't mean that programs written in it necessarily have to, and tt-rss is the best of the self-hosted RSS readers. Still, that means you're going to have to install PHP, and get it working nicely with Nginx.

Install PHP 5 and the components necessary to get it playing nicely with everything else.

> sudo apt-get install php5 php5-fpm php5-curl php5-pgsql php5-gd php5-mcrypt php5-cli

There's one line in php.ini you'll need to change, which will prevent a possible security issue.

> sudo nano /etc/php5/fpm/php.ini

Change this line:

cgi.fix_pathinfo=1

To this:

cgi.fix_pathinfo=0

The other change is to make PHP-FPM listen on a UNIX socket rather than a TCP socket.

> sudo nano /etc/php5/fpm/pool.d/www.conf

Change the line:

listen = 127.0.0.1:9000

To:

listen = /var/run/php5-fpm.sock

Finally, restart PHP.

> sudo /etc/init.d/php5-fpm restart

Nginx

Now that PHP is configured, you'll want to configure Nginx. There's a very good primer by Martin Fjordvald on the basics of Nginx configuration which you can read to get an understanding of what's going on. If you want to dig deeper for various other settings, the documentation for Nginx is quite thorough.

This tutorial is going to have you build Nginx from source, since the .deb packages are not always up to date. It's really not that much harder than setting it up from a package install anyway.

First, install the packages required to compile the source.

> sudo apt-get install gcc make libpcre3-dev openssl libssl-dev

I also prefer to have Nginx use a dedicated system account with no login or password access for a bit of extra security.

> sudo adduser --system --no-create-home --disabled-login --disabled-password --group nginx

Then download and unpack the latest Nginx version.

> wget https://nginx.org/download/nginx-1.5.6.tar.gz

> tar -xzvf nginx-1.5.6.tar.gz

> cd nginx-1.5.6

The configure command sets the parameters for the build. This is where you can change the default install directory and the different modules that Nginx uses. Nginx doesn't allow you to change modules without rebuilding, so you'll need to select or deselect any modules here. The only module I'll add is the http_ssl module, which will allow you to add SSL to your site later if you wish. I also prefer to just use the default install directory, so I won't change that, but I'll set the user and group Nginx uses to the one I created earlier.

> ./configure --user=nginx --group=nginx --with-http_ssl_module

Now we can build and install Nginx.

> make && sudo make install

Nginx Configuration

Change to the directory you installed Nginx to:

> cd /usr/local/nginx

There are various methods people use for controlling the sites on their server. The .deb install uses a sites-enabled and sites-disabled style configuration, but since I don't have many sites on my server, I prefer to keep each site configuration file in the conf/ folder, and include each one manually in nginx.conf.

So, for this tutorial, we'll create a ttrss.conf file.

> sudo nano conf/ttrss.conf

My ttrss.conf looks something like this (except I have SSL enabled):

# Tiny Tiny RSS Configuration

server {
  listen 80;
  server_name domainname www.domainname;

  root /var/www/ttrss;
  index index.php;

  error_log /var/log/nginx/ttrss.error.log;
  access_log /var/log/nginx/ttrss.access.log;

  location / {
    try_files $uri $uri/ /index.php;
  }

  location ~ \.php$ {
    include fastcgi.conf; # don't use fastcgi_params
    fastcgi_pass unix:/var/run/php5-fpm.sock;
    fastcgi_index index.php;
  }
}

A brief primer:

listen: the port that the server listens for incoming connections on.
server_name: the FQDN (fully qualified domain name) that the server uses for this connection. I set this to listen on both example.com and www.example.com.
root: the root directory that the site's files are located at.
index: the index file that the server directs traffic to. In this case you only need index.php, but can also include index.html or .htm, or any other file you want to use as a homepage.
error_log: the location of the error log for this site.
access_log: the location of the access log for this site. The first location block accepts all incoming requests:
try_files: this just makes the URL search engine friendly. Not really necessary in this case, but it doesn't hurt. The second location block catches all incoming php requests:
include fastcgi.conf: this file comes with Nginx and includes all the parameters for using fastcgi. It's almost the same as fastcgi_params, but you'll have issues using fastcgi_params in Ubuntu 12.04, unless you include the line SCRIPT_FILENAME $document_root$fastcgi_script_name.
fastcgi_pass: the socket that fastcgi passes requests through.
fastcgi_index: the index file that fastcgi uses.

Next, you'll need to edit nginx.conf.

> sudo nano conf/nginx.conf

The only change that you really need to make to this file is to add the line include ttrss.conf in the http block, but this is my recommended configuration below.

# Nginx Configuration

user nginx nginx;
worker_processes auto; # optimal value is number of cpu cores

error_log /var/log/nginx/error.log

events {
  worker_connections 2048;
}

http {
  # Virtual Host Configuration
  include ttrss.conf;

  # Basic Settings
  include mime.types;
  default_type application/octet-stream;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay off;

  client_header_timeout 20s;
  client_body_timeout 20s;
  send_timeout 20s;

  # Disable Nginx version number in error pages and Server header
  server_tokens off;

  # Silently block all undefined vhost access
  server {
    server_name _;
    return 444;
  }

  # Gzip Settings
  gzip on;
  gzip_disable "msie6";

  gzip_comp_level 5;
  gzip_min_length 256;

  gzip_vary on;
  gzip_proxied any;
  gzip_types *;
}

Finally, start Nginx.

> sudo /usr/local/nginx/sbin/nginx

If you need to make any changes to Nginx after it's been started, just open and change the config files. Then run this command to reload the Nginx with the new configuration.

> sudo /usr/local/nginx/sbin/nginx -s reload

Postgres

Tiny Tiny RSS recommends using PostgreSQL over MySQL for the database, as PostgreSQL is slightly faster. For a single user, it probably wouldn't make much difference, depending on how many feeds you have. The instructions for installing PostgreSQL are mostly taken from the Ubuntu community wiki here.

Install PostgreSQL.

> sudo apt-get install postgresql

> sudo -u postgres psql postgres

Change the password for the postgres user. Type in the command below, hit enter, then enter as password. Make sure to save all your passwords in a good password manager somewhere. I recommend KeePass.

> \password postgres

While you're still in the postgres command line, create a database and database user for tt-rss. The password must be entered surrounded by single quotes.

postgres=# CREATE USER ttrss WITH PASSWORD 'password';

postgres=# CREATE DATABASE ttrssdb;

postgres=# GRANT ALL PRIVILEGES ON DATABASE ttrssdb to ttrss;

Exit the postgres command line.

postgres=# \q

From this point, you might be able to get tt-rss to access the database, but I wasn't until I edited a line in the postgresql config to allow access.

> sudo nano /etc/postgresql/9.1/main/pg_hba.conf

Add this line to allow tt-rss to use the database:

local all ttrss md5

Now restart PostgreSQL.

> sudo service postgresql restart

Tiny Tiny RSS

Finally, it's now time to install Tiny Tiny RSS itself. Yeah, I know it takes a while to get here. I assure you, it's worth it in the end, and you shouldn't have to worry about it much at all once you've got it properly set up. The installation instructions for tt-rss can be be found here, but once again they seem somewhat incomplete.

First, make a directory for tt-rss to be served from. This should be the same as the directory you specified as root in your ttrss.conf file for Nginx.

> sudo mkdir -p /var/www/ttrss

You'll need to set the owner of the directory to the user that you're installing and running tt-rss as.

> sudo chown -R user:group /var/www/

Now change to the directory above.

> cd /var/www

Download and extract the latest version of Tiny Tiny RSS.

> wget https://github.com/gothfox/Tiny-Tiny-RSS/archive/1.10.tar.gz

> tar -xzvf 1.10.tar.gz

Rename the extracted directory so that it matches the root directory you specified earlier, then open the directory.

> mv Tiny-Tiny-RSS-1.10/ ttrss

> cd /var/www/ttrss

The following folders need to have their permissions changed to be writable by anyone with an account on the system, or tt-rss won't be able to save files there.

> sudo chmod -R 777 cache/images/ cache/js/ cache/export/ cache/upload/ feed-icons/ lock/

Now you should be able to navigate in your browser to the site that you are hosting tt-rss from. This should be the same as the server_name you specified in ttrss.conf. If you installed tt-rss into a sub-folder of the domain, then the address will be https://domainname/ttrss/. If you got the configuration correct, you should see the Tiny Tiny RSS install page.

The installation page will have a few fields to enter. These will be the credentials for tt-rss to use the database, as well as the full URL that you'll access tt-rss from. Once you've entered the database details, click the 'Test Configuration' button. If there are errors, you'll get a message saying so, and it shouldn't be too hard to figure out what to do from there. If it's good to go, you'll see a text box with a lot of configuration lines. Copy that config data to your clipboard, and then into a config file on your server with:

> sudo nano /var/www/ttrss/config.php

Save that file, and reload tt-rss in your browser. You should now be able to login with the default user admin and password password. Once you're logged in, change the admin settings in the preferences at top right.

The final required step is to set up automatic updating of your feeds. You can skip this, but you'll have to manually click on each feed to check for updates if you don't. The screen command will permanently run feed updates in the background, defaulting to once every 30 minutes.

> screen -d -m php ./update_daemon2.php

Conclusion

You should now have a working installation of Tiny Tiny RSS on your server, accessible from anywhere in the world. If you've used RSS Readers before, you can import your feeds from an OPML file, otherwise, find some good blogs and start adding feeds.

A New Blog

David Beath — Fri, 23 Aug 2013 09:23:54 GMT

So, I guess it's time I started to write a blog. I've always read that it's recommended in IT to write a blog, or maybe in any industry these days, in order to have some sort of recognisable professional presence. Or, at the very least, to keep your rants off Facebook.

At any rate, there are plenty of things I could talk about, and I will. I tend mostly to have strong opinions on Politics, Information Technology, Economics, the Environment, and any intersection between the bunch, though not necessarily in that order. I think I'll start off properly with a politics post next.

Anyway, I figure I might as well write about the tools I used to set this blog up. Starting from the bottom of the stack, my server is a VPS hosted by Linode. I like that I have full control over my machine, without having to worry about the hardware. Even the lowest tier plan gets me more than enough machine for my needs, and the documentation and tools they have are really good. Definitely recommended.

The server OS is Debian, because for me it was either that or Ubuntu Server for the easy and familiar option. Fanboys can feel free to flame away at whatever choice is made, but I like something stable.

For the HTTP server itself, I'm using nginx. Apache is just to mainstream, and from everything I've read it doesn't scale as well. Not that scaling should be a problem for me for a long time, if ever, but you never know. I've found nginx to be just as easy, if not easier, to understand and configure compared to Apache anyway.

Finally, I use Nikola to generate this blog. Nikola is a static blog generator, so all the files on this site are just plain HTML, no CMS needed. The site design is basically just an unmodified Foundation theme, which for now looks good enough. Besides, if you're not using an RSS reader to read this blog (once you've first visited and decided to keep reading of course), then you damned well should be. Why on earth would you manually visit blogs to read?

On the non-essential side of things, analytics on the blog are handled by Piwik, so that I don't have to send all my data to Google Analytics. The thing I use this server for the most though, is Tiny Tiny RSS. I find that using an RSS reader is essential for keeping up with the world. Like most everyone else, I used to use Google Reader, but of course anyone who hasn't been living under a rock this year knows the story of how they shut it down, and the gnashing and wailing of teeth that ensued. I contributed my part of that, on my tiny corner of Google Plus and Facebook.

Just like many others, I decided that if I couldn't trust even Google with my stuff, then it might be better to go self-hosted. Tiny Tiny RSS was the self hosted solution with the most recommendations, and I've been pretty happy with it so far. It's nice to know that you've got a service that no-one is going to arbitrarily remove, just so long as you have a good backup policy in place.

Someday soon, I might get round to routing my mail through my server as well, though I might like to have it on a different one. A single point of failure for everything I rely on makes me nervous.

In conclusion, Welcome to my Blog, and don't be afraid to look into self hosting if you're at all technically inclined. It's the 21st century, and all the information you need is out there.