如何避免在为Python项目构建Docker映像时重新安装包?

我的Dockerfile是这样的

FROM my/base


ADD . /srv
RUN pip install -r requirements.txt
RUN python setup.py install


ENTRYPOINT ["run_server"]

每次我构建一个新的映像时，都必须重新安装依赖项，这在我所在的区域可能非常慢。

我想到的cache包的一种方法是用更新的映像覆盖my/base映像，如下所示:

docker build -t new_image_1 .
docker tag new_image_1 my/base

所以下次我用这个Dockerfile构建时，我的/base已经安装了一些包。

但这个解决方案有两个问题:

  1. 并非总是可以覆盖基本映像
  2. 随着更新的图像在基础图像上分层，基础图像变得越来越大

还有什么更好的解决方法呢?

编辑:

关于我机器上的docker的一些信息:

☁  test  docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070
☁  test  docker info
Containers: 0
Images: 56
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Dirs: 56
Execution Driver: native-0.2
Kernel Version: 3.13.0-29-generic
WARNING: No swap limit support
90410 次浏览
小开
最佳答案

试着构建一个类似这样的Dockerfile:

FROM my/base


WORKDIR /srv
ADD ./requirements.txt /srv/requirements.txt
RUN pip install -r requirements.txt
ADD . /srv
RUN python setup.py install


ENTRYPOINT ["run_server"]

Docker将在pip安装期间使用缓存，只要你没有对requirements.txt做任何更改，不管.上的其他代码文件是否被更改。举个例子。

下面是一个简单的Hello, World!程序:

$ tree
.
├── Dockerfile
├── requirements.txt
└── run.py


0 directories, 3 file


# Dockerfile


FROM dockerfile/python
WORKDIR /srv
ADD ./requirements.txt /srv/requirements.txt
RUN pip install -r requirements.txt
ADD . /srv
CMD python /srv/run.py


# requirements.txt
pytest==2.3.4


# run.py
print("Hello, World")

docker build的输出:

Step 1 : WORKDIR /srv
---> Running in 22d725d22e10
---> 55768a00fd94
Removing intermediate container 22d725d22e10
Step 2 : ADD ./requirements.txt /srv/requirements.txt
---> 968a7c3a4483
Removing intermediate container 5f4e01f290fd
Step 3 : RUN pip install -r requirements.txt
---> Running in 08188205e92b
Downloading/unpacking pytest==2.3.4 (from -r requirements.txt (line 1))
Running setup.py (path:/tmp/pip_build_root/pytest/setup.py) egg_info for package pytest
....
Cleaning up...
---> bf5c154b87c9
Removing intermediate container 08188205e92b
Step 4 : ADD . /srv
---> 3002a3a67e72
Removing intermediate container 83defd1851d0
Step 5 : CMD python /srv/run.py
---> Running in 11e69b887341
---> 5c0e7e3726d6
Removing intermediate container 11e69b887341
Successfully built 5c0e7e3726d6

让我们修改run.py:

# run.py
print("Hello, Python")

尝试再次构建，下面是输出:

Sending build context to Docker daemon  5.12 kB
Sending build context to Docker daemon
Step 0 : FROM dockerfile/python
---> f86d6993fc7b
Step 1 : WORKDIR /srv
---> Using cache
---> 55768a00fd94
Step 2 : ADD ./requirements.txt /srv/requirements.txt
---> Using cache
---> 968a7c3a4483
Step 3 : RUN pip install -r requirements.txt
---> Using cache
---> bf5c154b87c9
Step 4 : ADD . /srv
---> 9cc7508034d6
Removing intermediate container 0d7cf71eb05e
Step 5 : CMD python /srv/run.py
---> Running in f25c21135010
---> 4ffab7bc66c7
Removing intermediate container f25c21135010
Successfully built 4ffab7bc66c7

正如你在上面看到的，这次docker在构建过程中使用了缓存。现在，让我们更新requirements.txt:

# requirements.txt


pytest==2.3.4
ipython

下面是docker build的输出:

Sending build context to Docker daemon  5.12 kB
Sending build context to Docker daemon
Step 0 : FROM dockerfile/python
---> f86d6993fc7b
Step 1 : WORKDIR /srv
---> Using cache
---> 55768a00fd94
Step 2 : ADD ./requirements.txt /srv/requirements.txt
---> b6c19f0643b5
Removing intermediate container a4d9cb37dff0
Step 3 : RUN pip install -r requirements.txt
---> Running in 4b7a85a64c33
Downloading/unpacking pytest==2.3.4 (from -r requirements.txt (line 1))
Running setup.py (path:/tmp/pip_build_root/pytest/setup.py) egg_info for package pytest


Downloading/unpacking ipython (from -r requirements.txt (line 2))
Downloading/unpacking py>=1.4.12 (from pytest==2.3.4->-r requirements.txt (line 1))
Running setup.py (path:/tmp/pip_build_root/py/setup.py) egg_info for package py


Installing collected packages: pytest, ipython, py
Running setup.py install for pytest


Installing py.test script to /usr/local/bin
Installing py.test-2.7 script to /usr/local/bin
Running setup.py install for py


Successfully installed pytest ipython py
Cleaning up...
---> 23a1af3df8ed
Removing intermediate container 4b7a85a64c33
Step 4 : ADD . /srv
---> d8ae270eca35
Removing intermediate container 7f003ebc3179
Step 5 : CMD python /srv/run.py
---> Running in 510359cf9e12
---> e42fc9121a77
Removing intermediate container 510359cf9e12
Successfully built e42fc9121a77

注意docker在pip安装期间没有使用缓存。如果它不起作用，请检查docker版本。

Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070
小开

我发现更好的方法是将Python site-packages目录添加为一个卷。

services:
web:
build: .
command: python manage.py runserver 0.0.0.0:8000
volumes:
- .:/code
-  /usr/local/lib/python2.7/site-packages/

这样我就可以安装新的库，而不需要做一个完整的重建。

编辑:忽略这个答案，上面的jkukul的答案对我有用。我的目的是缓存网站文件夹。它看起来应该是这样的:

volumes:
- .:/code
- ./cached-packages:/usr/local/lib/python2.7/site-packages/

但是缓存下载文件夹要干净得多。这也缓存了轮子，所以它正确地完成了任务。

小开

为了最小化网络活动，你可以将pip指向主机上的缓存目录。

运行docker容器时，将主机的pip缓存目录绑定到容器的pip缓存目录中。docker run命令应该看起来像这样:

docker run -v $HOME/.cache/pip-docker/:/root/.cache/pip image_1

然后在Dockerfile中，将需求作为ENTRYPOINT语句(或CMD语句)的一部分安装，而不是作为RUN命令。这很重要，因为(正如评论中指出的)在映像构建期间(当RUN语句执行时)挂载是不可用的。Docker文件应该是这样的:

FROM my/base


ADD . /srv


ENTRYPOINT ["sh", "-c", "pip install -r requirements.txt && python setup.py install && run_server"]
小开

我知道这个问题已经有了一些流行的答案。但是有一种为包管理器缓存文件的新方法。我认为在未来，当BuildKit变得更加标准时，这可能是一个很好的答案。

从Docker 18.09开始，就有了对BuildKit的实验性支持。BuildKit在Dockerfile中增加了对一些新特性的支持，包括用于安装外部卷的实验支架到RUN步骤。这允许我们为$HOME/.cache/pip/这样的东西创建缓存。

我们将使用下面的requirements.txt文件作为示例:

Click==7.0
Django==2.2.3
django-appconf==1.0.3
django-compressor==2.3
django-debug-toolbar==2.0
django-filter==2.2.0
django-reversion==3.0.4
django-rq==2.1.0
pytz==2019.1
rcssmin==1.0.6
redis==3.3.4
rjsmin==1.1.0
rq==1.1.0
six==1.12.0
sqlparse==0.3.0

Python Dockerfile的典型示例可能如下所示:

FROM python:3.7
WORKDIR /usr/src/app
COPY requirements.txt /usr/src/app/
RUN pip install -r requirements.txt
COPY . /usr/src/app

通过使用DOCKER_BUILDKIT环境变量启用BuildKit，我们可以在大约65秒内构建非缓存的pip步骤:

$ export DOCKER_BUILDKIT=1
$ docker build -t test .
[+] Building 65.6s (10/10) FINISHED
=> [internal] load .dockerignore                                                                                                                                          0.0s
=> => transferring context: 2B                                                                                                                                            0.0s
=> [internal] load build definition from Dockerfile                                                                                                                       0.0s
=> => transferring dockerfile: 120B                                                                                                                                       0.0s
=> [internal] load metadata for docker.io/library/python:3.7                                                                                                              0.5s
=> CACHED [1/4] FROM docker.io/library/python:3.7@sha256:6eaf19442c358afc24834a6b17a3728a45c129de7703d8583392a138ecbdb092                                                 0.0s
=> [internal] load build context                                                                                                                                          0.6s
=> => transferring context: 899.99kB                                                                                                                                      0.6s
=> CACHED [internal] helper image for file operations                                                                                                                     0.0s
=> [2/4] COPY requirements.txt /usr/src/app/                                                                                                                              0.5s
=> [3/4] RUN pip install -r requirements.txt                                                                                                                             61.3s
=> [4/4] COPY . /usr/src/app                                                                                                                                              1.3s
=> exporting to image                                                                                                                                                     1.2s
=> => exporting layers                                                                                                                                                    1.2s
=> => writing image sha256:d66a2720e81530029bf1c2cb98fb3aee0cffc2f4ea2aa2a0760a30fb718d7f83                                                                               0.0s
=> => naming to docker.io/library/test                                                                                                                                    0.0s

现在，让我们添加实验头文件，并修改RUN步骤来缓存Python包:

# syntax=docker/dockerfile:experimental


FROM python:3.7
WORKDIR /usr/src/app
COPY requirements.txt /usr/src/app/
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
COPY . /usr/src/app

现在继续执行另一个构建。应该花同样多的时间。但这一次它是在我们的新缓存挂载中缓存Python包:

$ docker build -t pythontest .
[+] Building 60.3s (14/14) FINISHED
=> [internal] load build definition from Dockerfile                                                                                                                       0.0s
=> => transferring dockerfile: 120B                                                                                                                                       0.0s
=> [internal] load .dockerignore                                                                                                                                          0.0s
=> => transferring context: 2B                                                                                                                                            0.0s
=> resolve image config for docker.io/docker/dockerfile:experimental                                                                                                      0.5s
=> CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3                                 0.0s
=> [internal] load .dockerignore                                                                                                                                          0.0s
=> [internal] load build definition from Dockerfile                                                                                                                       0.0s
=> => transferring dockerfile: 120B                                                                                                                                       0.0s
=> [internal] load metadata for docker.io/library/python:3.7                                                                                                              0.5s
=> CACHED [1/4] FROM docker.io/library/python:3.7@sha256:6eaf19442c358afc24834a6b17a3728a45c129de7703d8583392a138ecbdb092                                                 0.0s
=> [internal] load build context                                                                                                                                          0.7s
=> => transferring context: 899.99kB                                                                                                                                      0.6s
=> CACHED [internal] helper image for file operations                                                                                                                     0.0s
=> [2/4] COPY requirements.txt /usr/src/app/                                                                                                                              0.6s
=> [3/4] RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt                                                                                  53.3s
=> [4/4] COPY . /usr/src/app                                                                                                                                              2.6s
=> exporting to image                                                                                                                                                     1.2s
=> => exporting layers                                                                                                                                                    1.2s
=> => writing image sha256:0b035548712c1c9e1c80d4a86169c5c1f9e94437e124ea09e90aea82f45c2afc                                                                               0.0s
=> => naming to docker.io/library/test                                                                                                                                    0.0s

大约60秒。类似于我们的第一个构建。

对requirements.txt做一个小的改变(比如在两个包之间添加一个新行)来强制缓存无效并再次运行:

$ docker build -t pythontest .
[+] Building 15.9s (14/14) FINISHED
=> [internal] load build definition from Dockerfile                                                                                                                       0.0s
=> => transferring dockerfile: 120B                                                                                                                                       0.0s
=> [internal] load .dockerignore                                                                                                                                          0.0s
=> => transferring context: 2B                                                                                                                                            0.0s
=> resolve image config for docker.io/docker/dockerfile:experimental                                                                                                      1.1s
=> CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3                                 0.0s
=> [internal] load build definition from Dockerfile                                                                                                                       0.0s
=> => transferring dockerfile: 120B                                                                                                                                       0.0s
=> [internal] load .dockerignore                                                                                                                                          0.0s
=> [internal] load metadata for docker.io/library/python:3.7                                                                                                              0.5s
=> CACHED [1/4] FROM docker.io/library/python:3.7@sha256:6eaf19442c358afc24834a6b17a3728a45c129de7703d8583392a138ecbdb092                                                 0.0s
=> CACHED [internal] helper image for file operations                                                                                                                     0.0s
=> [internal] load build context                                                                                                                                          0.7s
=> => transferring context: 899.99kB                                                                                                                                      0.7s
=> [2/4] COPY requirements.txt /usr/src/app/                                                                                                                              0.6s
=> [3/4] RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt                                                                                   8.8s
=> [4/4] COPY . /usr/src/app                                                                                                                                              2.1s
=> exporting to image                                                                                                                                                     1.1s
=> => exporting layers                                                                                                                                                    1.1s
=> => writing image sha256:fc84cd45482a70e8de48bfd6489e5421532c2dd02aaa3e1e49a290a3dfb9df7c                                                                               0.0s
=> => naming to docker.io/library/test                                                                                                                                    0.0s

只有大约16秒!

我们得到这个加速是因为我们不再下载所有的Python包。它们由包管理器缓存(在本例中为pip)并存储在缓存卷挂载中。卷挂载提供给运行步骤，以便pip可以重用我们已经下载的包。这发生在任何Docker层缓存之外。

在更大的requirements.txt上的增益应该更好。

注:

  • 这是实验性的Dockerfile语法，应该被这样对待。您目前可能不想在生产环境中使用它进行构建。
  • BuildKit目前还不能在Docker Compose或其他直接使用Docker API的工具下工作。现在在Docker Compose中有1.25.0的支持。看到如何使用docker-compose来启用BuildKit ?
  • 目前还没有用于托管缓存的直接接口。当你执行docker system prune -a时，它会被清除。

希望这些特性能够加入到Docker的构建中，而BuildKit将成为默认配置。如果/当这种情况发生时，我会尝试更新这个答案。

小开 
pipenv install

默认情况下尝试重新锁定。当它这样做时，Docker构建的缓存层不会被使用，因为Pipfile。锁换了。查看文档

对此的解决方案是对Pipfile进行版本化。锁定和使用

RUN pipenv sync

代替。

感谢JFG Piñeiro。


提交答案