并发、工作者和自动伸缩之间的差异

在 /etc/defaults/celeryd配置文件中，我设置了:

CELERYD_NODES="agent1 agent2 agent3 agent4 agent5 agent6 agent7 agent8"
CELERYD_OPTS="--autoscale=10,3 --concurrency=5"

我知道守护进程产生8个芹菜工人，但我完全不确定 autoscale和 concurrency一起做什么。我认为并发是一种指定 worker 可以使用的最大线程数的方法，而自动伸缩是一种工作者在必要时可以向上和向下伸缩子 worker 的方法。

这些任务的有效负载很大(大约20-50kB) ，大约有200-300万个这样的任务，但是每个任务运行不到一秒钟。我看到内存使用激增，因为代理将任务分发给每个工作者，从而多次复制有效负载。

我认为问题在于配置，worker + 并发性 + 自动伸缩的组合是过度的，我想更好地理解这三个选项的作用。

68551

小开

When using --autoscale the number of processes are set dynamically with max/min values which enable the worker to scale according to load and when using --concurrency processes are set statically with a fixed number. So using these two together makes no sense.

Celery --autoscale is responsible for growing and shrinking the pool dynamically based on load. This in turn adds more processes when there is work to do and removes processes when the workload is low. So for example --autoscale=10,3 would give you a maximum of 10 processes and a minimum of 3 processes.

As for --concurrency celery by default uses multiprocessing to perform concurrent execution of tasks. The number of worker processes/threads can be changed using the --concurrency argument and defaults to the number of available CPU's if not set. So for example --concurrency=5 would use 5 processes meaning 5 tasks that can run concurrently.

最佳答案

Let's distinguish between workers and worker processes. You spawn a celery worker, this then spawns a number of processes (depending on things like --concurrency and --autoscale, the default is to spawn as many processes as cores on the machine). There is no point in running more than one worker on a particular machine unless you want to do routing.

I would suggest running only 1 worker per machine with the default number of processes. This will reduce memory usage by eliminating the duplication of data between workers.

If you still have memory issues then save the data to a store and pass only an id to the workers.