如何配置 Docker 端口映射使用 Nginx 作为上游代理?

更新二

现在是2015年7月16日,事情又变了。我已经从 Jason Wilder: https://github.com/jwilder/nginx-proxy中发现了这个自动容器,它解决了这个问题,只要停靠码头运行容器。这就是我现在用来解决这个问题的方法。

更新

现在是2015年7月,与联网 Docker 容器相关的事情发生了巨大的变化。现在有许多不同的产品(以各种方式)来解决这个问题。

您应该使用这篇文章来获得对 docker-link 服务发现方法的基本理解,这是最基本的方法,工作得非常好,实际上比其他大多数解决方案需要更少的花里胡哨。它的局限性在于,在任何给定集群中的单独主机上将容器联网是相当困难的,而且一旦联网,容器就不能重新启动,但是它确实提供了一种在同一主机上联网容器的快速且相对简单的方法。这是一个很好的方法来了解你可能用来解决这个问题的软件实际上是做什么的。

此外,你可能还想看看 Docker 的新兴网络,Hashicorp 的领事,Weaveworks 的编织,Jeff Lindsay 的程序员/领事 & gliderlab/注册员,以及 Google 的 Kubernetes。

还有一些 CoreOS 产品,它们利用 etcd、 fleet 和 flannel。

如果你真的想开个派对你可以组织一个集群来运行 MesSphere 或者 Deis 或者 Flynn。

如果你(像我一样)刚开始接触社交网络,那么你应该拿出你的老花镜,在 Wi-Hi-Fi 上点击“用星星画出天空ーー恩雅最好的作品”,然后喝一杯啤酒ーー你要过一段时间才能真正明白你到底想要做什么。提示: 您正试图在集群控制平面中实现服务发现层。这是度过周六晚上的好方法。

这很有趣,但是我希望在投入其中之前,我能够花时间更好地教育自己关于人际网络的一般知识。我最终找到了几篇来自仁慈的数字海洋教程之神的文章: 网络术语和理解入门... 网络。我建议在开始之前先读几遍。

玩得开心!

原文

我似乎不能理解 Docker容器的端口映射。特别是如何将来自 Nginx 的请求传递到同一服务器上监听另一个港口的另一个集装箱。

我得到了一个 Dockerfile,用于 Nginx 容器,如下所示:

FROM ubuntu:14.04
MAINTAINER Me <me@myapp.com>


RUN apt-get update && apt-get install -y htop git nginx


ADD sites-enabled/api.myapp.com /etc/nginx/sites-enabled/api.myapp.com
ADD sites-enabled/app.myapp.com /etc/nginx/sites-enabled/app.myapp.com
ADD nginx.conf /etc/nginx/nginx.conf


RUN echo "daemon off;" >> /etc/nginx/nginx.conf


EXPOSE 80 443


CMD ["service", "nginx", "start"]



然后,api.myapp.com配置文件如下所示:

upstream api_upstream{


server 0.0.0.0:3333;


}




server {


listen 80;
server_name api.myapp.com;
return 301 https://api.myapp.com/$request_uri;


}




server {


listen 443;
server_name api.mypp.com;
    

location / {


proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
proxy_pass http://api_upstream;
    

}


}

还有一个是 app.myapp.com

然后我跑过去:

sudo docker run -p 80:80 -p 443:443 -d --name Nginx myusername/nginx

这一切都很好,但是请求没有传递到其他容器/港口。当我 ssh 进入 Nginx 容器并检查日志时,我没有看到任何错误。

有人帮忙吗?

128045 次浏览

Using docker links, you can link the upstream container to the nginx container. An added feature is that docker manages the host file, which means you'll be able to refer to the linked container using a name rather than the potentially random ip.

@T0xicCode's answer is correct, but I thought I would expand on the details since it actually took me about 20 hours to finally get a working solution implemented.

If you're looking to run Nginx in its own container and use it as a reverse proxy to load balance multiple applications on the same server instance then the steps you need to follow are as such:

Link Your Containers

When you docker run your containers, typically by inputting a shell script into User Data, you can declare links to any other running containers. This means that you need to start your containers up in order and only the latter containers can link to the former ones. Like so:

#!/bin/bash
sudo docker run -p 3000:3000 --name API mydockerhub/api
sudo docker run -p 3001:3001 --link API:API --name App mydockerhub/app
sudo docker run -p 80:80 -p 443:443 --link API:API --link App:App --name Nginx mydockerhub/nginx

So in this example, the API container isn't linked to any others, but the App container is linked to API and Nginx is linked to both API and App.

The result of this is changes to the env vars and the /etc/hosts files that reside within the API and App containers. The results look like so:

/etc/hosts

Running cat /etc/hosts within your Nginx container will produce the following:

172.17.0.5  0fd9a40ab5ec
127.0.0.1   localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.3  App
172.17.0.2  API



ENV Vars

Running env within your Nginx container will produce the following:

API_PORT=tcp://172.17.0.2:3000
API_PORT_3000_TCP_PROTO=tcp
API_PORT_3000_TCP_PORT=3000
API_PORT_3000_TCP_ADDR=172.17.0.2


APP_PORT=tcp://172.17.0.3:3001
APP_PORT_3001_TCP_PROTO=tcp
APP_PORT_3001_TCP_PORT=3001
APP_PORT_3001_TCP_ADDR=172.17.0.3

I've truncated many of the actual vars, but the above are the key values you need to proxy traffic to your containers.

To obtain a shell to run the above commands within a running container, use the following:

sudo docker exec -i -t Nginx bash

You can see that you now have both /etc/hosts file entries and env vars that contain the local IP address for any of the containers that were linked. So far as I can tell, this is all that happens when you run containers with link options declared. But you can now use this information to configure nginx within your Nginx container.



Configuring Nginx

This is where it gets a little tricky, and there's a couple of options. You can choose to configure your sites to point to an entry in the /etc/hosts file that docker created, or you can utilize the ENV vars and run a string replacement (I used sed) on your nginx.conf and any other conf files that may be in your /etc/nginx/sites-enabled folder to insert the IP values.



OPTION A: Configure Nginx Using ENV Vars

This is the option that I went with because I couldn't get the /etc/hosts file option to work. I'll be trying Option B soon enough and update this post with any findings.

The key difference between this option and using the /etc/hosts file option is how you write your Dockerfile to use a shell script as the CMD argument, which in turn handles the string replacement to copy the IP values from ENV to your conf file(s).

Here's the set of configuration files I ended up with:

Dockerfile

FROM ubuntu:14.04
MAINTAINER Your Name <you@myapp.com>


RUN apt-get update && apt-get install -y nano htop git nginx


ADD nginx.conf /etc/nginx/nginx.conf
ADD api.myapp.conf /etc/nginx/sites-enabled/api.myapp.conf
ADD app.myapp.conf /etc/nginx/sites-enabled/app.myapp.conf
ADD Nginx-Startup.sh /etc/nginx/Nginx-Startup.sh


EXPOSE 80 443


CMD ["/bin/bash","/etc/nginx/Nginx-Startup.sh"]

nginx.conf

daemon off;
user www-data;
pid /var/run/nginx.pid;
worker_processes 1;




events {
worker_connections 1024;
}




http {


# Basic Settings


sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 33;
types_hash_max_size 2048;


server_tokens off;
server_names_hash_bucket_size 64;


include /etc/nginx/mime.types;
default_type application/octet-stream;




# Logging Settings
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;




# Gzip Settings


gzip on;
gzip_vary on;
gzip_proxied any;
gzip_comp_level 3;
gzip_buffers 16 8k;
gzip_http_version 1.1;
gzip_types text/plain text/xml text/css application/x-javascript application/json;
gzip_disable "MSIE [1-6]\.(?!.*SV1)";


# Virtual Host Configs
include /etc/nginx/sites-enabled/*;


# Error Page Config
#error_page 403 404 500 502 /srv/Splash;




}

NOTE: It's important to include daemon off; in your nginx.conf file to ensure that your container doesn't exit immediately after launching.

api.myapp.conf

upstream api_upstream{
server APP_IP:3000;
}


server {
listen 80;
server_name api.myapp.com;
return 301 https://api.myapp.com/$request_uri;
}


server {
listen 443;
server_name api.myapp.com;


location / {
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
proxy_pass http://api_upstream;
}


}

Nginx-Startup.sh

#!/bin/bash
sed -i 's/APP_IP/'"$API_PORT_3000_TCP_ADDR"'/g' /etc/nginx/sites-enabled/api.myapp.com
sed -i 's/APP_IP/'"$APP_PORT_3001_TCP_ADDR"'/g' /etc/nginx/sites-enabled/app.myapp.com


service nginx start

I'll leave it up to you to do your homework about most of the contents of nginx.conf and api.myapp.conf.

The magic happens in Nginx-Startup.sh where we use sed to do string replacement on the APP_IP placeholder that we've written into the upstream block of our api.myapp.conf and app.myapp.conf files.

This ask.ubuntu.com question explains it very nicely: Find and replace text within a file using commands

GOTCHA On OSX, sed handles options differently, the -i flag specifically. On Ubuntu, the -i flag will handle the replacement 'in place'; it will open the file, change the text, and then 'save over' the same file. On OSX, the -i flag requires the file extension you'd like the resulting file to have. If you're working with a file that has no extension you must input '' as the value for the -i flag.

GOTCHA To use ENV vars within the regex that sed uses to find the string you want to replace you need to wrap the var within double-quotes. So the correct, albeit wonky-looking, syntax is as above.

So docker has launched our container and triggered the Nginx-Startup.sh script to run, which has used sed to change the value APP_IP to the corresponding ENV variable we provided in the sed command. We now have conf files within our /etc/nginx/sites-enabled directory that have the IP addresses from the ENV vars that docker set when starting up the container. Within your api.myapp.conf file you'll see the upstream block has changed to this:

upstream api_upstream{
server 172.0.0.2:3000;
}

The IP address you see may be different, but I've noticed that it's usually 172.0.0.x.

You should now have everything routing appropriately.

GOTCHA You cannot restart/rerun any containers once you've run the initial instance launch. Docker provides each container with a new IP upon launch and does not seem to re-use any that its used before. So api.myapp.com will get 172.0.0.2 the first time, but then get 172.0.0.4 the next time. But Nginx will have already set the first IP into its conf files, or in its /etc/hosts file, so it won't be able to determine the new IP for api.myapp.com. The solution to this is likely to use CoreOS and its etcd service which, in my limited understanding, acts like a shared ENV for all machines registered into the same CoreOS cluster. This is the next toy I'm going to play with setting up.



OPTION B: Use /etc/hosts File Entries

This should be the quicker, easier way of doing this, but I couldn't get it to work. Ostensibly you just input the value of the /etc/hosts entry into your api.myapp.conf and app.myapp.conf files, but I couldn't get this method to work.

UPDATE: See @Wes Tod's answer for instructions on how to make this method work.

Here's the attempt that I made in api.myapp.conf:

upstream api_upstream{
server API:3000;
}

Considering that there's an entry in my /etc/hosts file like so: 172.0.0.2 API I figured it would just pull in the value, but it doesn't seem to be.

I also had a couple of ancillary issues with my Elastic Load Balancer sourcing from all AZ's so that may have been the issue when I tried this route. Instead I had to learn how to handle replacing strings in Linux, so that was fun. I'll give this a try in a while and see how it goes.

AJB's "Option B" can be made to work by using the base Ubuntu image and setting up nginx on your own. (It didn't work when I used the Nginx image from Docker Hub.)

Here is the Docker file I used:

FROM ubuntu
RUN apt-get update && apt-get install -y nginx
RUN ln -sf /dev/stdout /var/log/nginx/access.log
RUN ln -sf /dev/stderr /var/log/nginx/error.log
RUN rm -rf /etc/nginx/sites-enabled/default
EXPOSE 80 443
COPY conf/mysite.com /etc/nginx/sites-enabled/mysite.com
CMD ["nginx", "-g", "daemon off;"]

My nginx config (aka: conf/mysite.com):

server {
listen 80 default;
server_name mysite.com;


location / {
proxy_pass http://website;
}
}


upstream website {
server website:3000;
}

And finally, how I start my containers:

$ docker run -dP --name website website
$ docker run -dP --name nginx --link website:website nginx

This got me up and running so my nginx pointed the upstream to the second docker container which exposed port 3000.

Just found an article from Anand Mani Sankar wich shows a simple way of using nginx upstream proxy with docker composer.

Basically one must configure the instance linking and ports at the docker-compose file and update upstream at nginx.conf accordingly.

I tried using the popular Jason Wilder reverse proxy that code-magically works for everyone, and learned that it doesn't work for everyone (ie: me). And I'm brand new to NGINX, and didn't like that I didn't understand the technologies I was trying to use.

Wanted to add my 2 cents, because the discussion above around linking containers together is now dated since it is a deprecated feature. So here's an explanation on how to do it using networks. This answer is a full example of setting up nginx as a reverse proxy to a statically paged website using Docker Compose and nginx configuration.

TL;DR;

Add the services that need to talk to each other onto a predefined network. For a step-by-step discussion on Docker networks, I learned some things here: https://technologyconversations.com/2016/04/25/docker-networking-and-dns-the-good-the-bad-and-the-ugly/

Define the Network

First of all, we need a network upon which all your backend services can talk on. I called mine web but it can be whatever you want.

docker network create web

Build the App

We'll just do a simple website app. The website is a simple index.html page being served by an nginx container. The content is a mounted volume to the host under a folder content

DockerFile:

FROM nginx
COPY default.conf /etc/nginx/conf.d/default.conf

default.conf

server {
listen       80;
server_name  localhost;


location / {
root   /var/www/html;
index  index.html index.htm;
}


error_page   500 502 503 504  /50x.html;
location = /50x.html {
root   /usr/share/nginx/html;
}
}

docker-compose.yml

version: "2"


networks:
mynetwork:
external:
name: web


services:
nginx:
container_name: sample-site
build: .
expose:
- "80"
volumes:
- "./content/:/var/www/html/"
networks:
default: {}
mynetwork:
aliases:
- sample-site

Note that we no longer need port mapping here. We simple expose port 80. This is handy for avoiding port collisions.

Run the App

Fire this website up with

docker-compose up -d

Some fun checks regarding the dns mappings for your container:

docker exec -it sample-site bash
ping sample-site

This ping should work, inside your container.

Build the Proxy

Nginx Reverse Proxy:

Dockerfile

FROM nginx


RUN rm /etc/nginx/conf.d/*

We reset all the virtual host config, since we're going to customize it.

docker-compose.yml

version: "2"


networks:
mynetwork:
external:
name: web




services:
nginx:
container_name: nginx-proxy
build: .
ports:
- "80:80"
- "443:443"
volumes:
- ./conf.d/:/etc/nginx/conf.d/:ro
- ./sites/:/var/www/
networks:
default: {}
mynetwork:
aliases:
- nginx-proxy

Run the Proxy

Fire up the proxy using our trusty

docker-compose up -d

Assuming no issues, then you have two containers running that can talk to each other using their names. Let's test it.

docker exec -it nginx-proxy bash
ping sample-site
ping nginx-proxy

Set up Virtual Host

Last detail is to set up the virtual hosting file so the proxy can direct traffic based on however you want to set up your matching:

sample-site.conf for our virtual hosting config:

  server {
listen 80;
listen [::]:80;


server_name my.domain.com;


location / {
proxy_pass http://sample-site;
}


}

Based on how the proxy was set up, you'll need this file stored under your local conf.d folder which we mounted via the volumes declaration in the docker-compose file.

Last but not least, tell nginx to reload it's config.

docker exec nginx-proxy service nginx reload

These sequence of steps is the culmination of hours of pounding head-aches as I struggled with the ever painful 502 Bad Gateway error, and learning nginx for the first time, since most of my experience was with Apache.

This answer is to demonstrate how to kill the 502 Bad Gateway error that results from containers not being able to talk to one another.

I hope this answer saves someone out there hours of pain, since getting containers to talk to each other was really hard to figure out for some reason, despite it being what I expected to be an obvious use-case. But then again, me dumb. And please let me know how I can improve this approach.

@gdbj's answer is a great explanation and the most up to date answer. Here's however a simpler approach.

So if you want to redirect all traffic from nginx listening to 80 to another container exposing 8080, minimum configuration can be as little as:

nginx.conf:

server {
listen 80;


location / {
proxy_pass http://client:8080; # this one here
proxy_redirect off;
}


}

docker-compose.yml

version: "2"
services:
entrypoint:
image: some-image-with-nginx
ports:
- "80:80"
links:
- client  # will use this one here


client:
image: some-image-with-api
ports:
- "8080:8080"

Docker docs