X-Forwarded-For Headers (Loadbalancers)

If you have a server behind a loadbalancer then your web servers access logs may not log the correct IP address, they may log localhost or the ipaddress of the load balancer. Some hosting companies such as Rackspace have their LoadBalancer add a specific header called X-Forwarded-For headers. You can make a few changes to your Apache or nginx config files and vhosts to resolve this issue. Your log files should then read correctly.

Note: If you have SSL termination on your server then the following guides will not work. A load balancer is not able to inject a FORWARDED_TO header into the request because it is not able to decrypt and re-encrypt the packets.

Quick Links:
Apache (CentOS)
Nginx (CentOS)
Nginx + Varnish (CentOS)

 

Log files

Before: 127.0.0.1 - - [09/Jan/2015:11:18:11 +0000] "GET / HTTP/1.1" 200 35891
After: 94.236.7.x - - [09/Jan/2015:11:21:14 +0000] "GET / HTTP/1.1" 200 35891



Apache – CentOS

You will first need to edit /etc/httpd/conf/httpd.conf

Once you are in this file you need to locate the LogFormat section. The line you will need to look for and edit is:

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded

and change to

Format "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" forwarded

 

Once you have done this you will need to edit the vhosts for each of your domains. You should change your logs from:

ErrorLog logs/website.co.uk-error_log
CustomLog logs/website.co.uk-access_log common

to

ErrorLog logs/website.co.uk-error_log
CustomLog logs/website.co.uk-access_log forwarded

Now your done! Restart or reload apache and the changes should take place.

You can perform the following command to view the websites log files live:

tail -f -n 5 /var/log/httpd/website-access.log

tail views the last entries in the log file, -f views the log files live, -n 5 views  5 log entries. You should now load the your website in your browser while this command is running to see the ip address appear.

 

Nginx – CentOS

First you will need to find out the private ip address of your loadbalancer. The easiest way to do this is tail your logs for your domain with tail -f -n 5 /var/log/httpd/website-access.log

You should then be presented with a private net ip, in my case it is 10.189.255.252

You will then need to edit /etc/nginx/nginx.conf and enter the following lines  into your config file (in the http { section) with your load balancers internal ip address in place of mine.

set_real_ip_from 10.189.255.252;
real_ip_header X-Forwarded-For;

Restart nginx and then you can test it by running the tail command and loading your web page to see new access logs. DONE!

Nginx + Varnish

If you are using nginx + varnish the guide above will not work for you. If you view your access logs you will be presented with localhost (127.0.0.1) instead of your loadbalancers IP address. To resolve this issue you will add the following lines to /etc/nginx/nginx.conf:

set_real_ip_from 127.0.0.1;
real_ip_header X-Forwarded-For;

This will change your logs similar to the following example:

Example:

Before – 127.0.0.1 – – [09/Jan/2015:20:40:00 +0000] “GET

After – 10.189.246.5 – – [09/Jan/2015:20:40:25 +0000] “GET

This will change your log files from 127.0.0.1 to the ip address of your load balancer. Now we will need to change the ip address from your load balancer’s to the visitors ip address.

You can do this by editing /etc/varnish/default.conf file and making sure you have the following code in the sub vcl_recv { section:

if (req.restarts == 0) {

if (!req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = client.ip;
}
}

A larger example of this config section:

sub vcl_recv {

if (req.restarts == 0) {
if (!req.http.x-forwarded-for) {
set req.http.X-Forwarded-For = client.ip;
}
}

if (req.request != "GET" &&
req.request != "HEAD" &&
req.request != "PUT" &&
req.request != "POST" &&
....

Restart nginx and varnish and your done!