OOM Killer 原理


最近线上一台机器的 Redis 进程频繁被系统自动 Kill 掉,syslog(/var/log/syslog)记录了当时的日志:

Aug 19 11:16:31 jsldg kernel: [4836881.950256] Out of memory: Kill process 11063 (redis-cli) score 538 or sacrifice child
Aug 19 11:16:31 jsldg kernel: [4836881.950338] Killed process 11063 (redis-cli) total-vm:4649480kB, anon-rss:3098176kB, file-rss:4kB

这段日志表示 Redis 触发了 OOM(Out of Memory) Killer 机制。

当系统资源不足时,Linux 会调用 out_of_memory 函数,out_of_memory 会选出系统中占用资源过大的进程并 kill 掉,其中 out_of_memory 会调用 select_bad_process 函数,选出占用资源最大的进程。

OOM Kill 机制相关代码在/mm/oom_kill.c:

p = select_bad_process(&points, totalpages, mpol_mask, force_kill);
/* Found nothing?!?! Either we hang forever, or we panic. */
if (!p) {
        dump_header(NULL, gfp_mask, order, NULL, mpol_mask);
        panic("Out of memory and no killable processes...\n");
if (p != (void *)-1UL) {
        oom_kill_process(p, gfp_mask, order, points, totalpages, NULL,
                         nodemask, "Out of memory");
        killed = 1;

当找到占用资源最大的进程后,函数调用 oom_kill_process 来结束进程。

select_bad_process 函数负责选举要被 kill 的进程,首先遍历进程,然后找出资源占用最大的进程。实现如下:

static struct task_struct *select_bad_process(unsigned int *ppoints,
                unsigned long totalpages, const nodemask_t *nodemask,
                bool force_kill)
        struct task_struct *g, *p;
        struct task_struct *chosen = NULL;
        unsigned long chosen_points = 0;

        for_each_process_thread(g, p) {
                unsigned int points;

                switch (oom_scan_process_thread(p, totalpages, nodemask,
                                                force_kill)) {
                case OOM_SCAN_SELECT:
                        chosen = p;
                        chosen_points = ULONG_MAX;
                        /* fall through */
                case OOM_SCAN_CONTINUE:
                case OOM_SCAN_ABORT:
                        return (struct task_struct *)(-1UL);
                case OOM_SCAN_OK:
                points = oom_badness(p, NULL, nodemask, totalpages);
                if (!points || points < chosen_points)
                /* Prefer thread group leaders for display purposes */
                if (points == chosen_points && thread_group_leader(chosen))

                chosen = p;
                chosen_points = points;
        if (chosen)

        *ppoints = chosen_points * 1000 / totalpages;
        return chosen;

其中调用了 oom_badness 函数来给每个进程打分,根据 point 的分数大小决定哪个进程被杀。