Python 原理笔记

Table of Contents

环境:Python 3.5.3

1 使用 LLDB 调试

如,跟踪 locals 函数的实现:

$ Python-3.5.3 lldb ./python
(lldb) target create "./python"
Current executable set to './python' (x86_64).
(lldb) breakpoint set --file bltinmodule.c --line 1467
Breakpoint 1: where = python`builtin_locals + 4 [inlined] builtin_locals_impl at bltinmodule.c.h:460, address = 0x00000000004f34f4
(lldb) run
Process 6648 launched: './python' (x86_64)
Python 3.5.3 (default, Feb 26 2018, 12:50:49)
[GCC 7.3.1 20180130 (Red Hat 7.3.1-2)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> locals()
Process 6648 stopped
* thread #1, name = 'python', stop reason = breakpoint 1.1
    frame #0: 0x00000000004f34f4 python`builtin_locals at bltinmodule.c:1470
   1467	{
   1468	    PyObject *d;
   1469
-> 1470	    d = PyEval_GetLocals();
   1471	    Py_XINCREF(d);
   1472	    return d;
   1473	}
(lldb)

2 字节码

Python3 的执行原理:

  解析 Python 语法 -> 生成字节码 -> Python 虚拟机执行字节码

Python 作为动态语言,解析器并不关心不同的类型,一切都是由 Python 虚拟机来操作。

我很大一部分分析工作,都是在分析字节码的实现,Python/ceval.c 就是字节码解析的实现。

要取字节码,最简单的就是用 dis 模块,如下,取 test.py 的字节码:

python3 -m dis /tmp/test.py
  3           0 LOAD_CONST               0 (<code object hello at 0x7f88f3bb9810, file "/tmp/test.py", line 3>)
              2 LOAD_CONST               1 ('hello')
              4 MAKE_FUNCTION            0
              6 STORE_NAME               0 (hello)
              8 LOAD_CONST               2 (None)
             10 RETURN_VALUE

也可以在交互式中使用 dis 模块:

>>> import dis
>>> dis.dis("print(123)")
  1           0 LOAD_NAME                0 (print)
              2 LOAD_CONST               0 (123)
              4 CALL_FUNCTION            1
              6 RETURN_VALUE

其实字节码就是一堆数字而已,每个数字对应的指令可以到 Include/opcode.h 中查看,Lib/opcode.py 中也有定义。

3 对象

Python 中的一切对象都是 PyObject,定义在 Include/object.h 中:

typedef struct _object {
    _PyObject_HEAD_EXTRA
    Py_ssize_t ob_refcnt;
    struct _typeobject *ob_type;
} PyObject;

3.1 函数

我们取下面代码的字节码:

def hello():
    print("hello world")

字节码如下:

3           0 LOAD_CONST               0 (<code object hello at 0x7f19578f9030, file "/tmp/test.py", line 3>)
            3 LOAD_CONST               1 ('hello')
            6 MAKE_FUNCTION            0
            9 STORE_NAME               0 (hello)
           12 LOAD_CONST               2 (None)
           15 RETURN_VALUE

在 Python/ceval.c 中可看到函数是怎么建立的:

PyObject *qualname = POP(); /* 取函数名 */
PyObject *code = POP(); /* 获得 code 对象,所以说,代码块也是一种 Python 对象 */
PyObject *func = PyFunction_NewWithQualName(code, f->f_globals, qualname);

首先对当前栈帧做了 POP 字节码操作,取到了函数名,然后再做一次 POP 操作取到 code 类型的对象,可以对照上面字节码观察下栈帧内容。

注意上面代码的第二行 code 变量,即便代码本身,它也是一个 Python 对象,通过 Python 可以观察下:

>>> def hello():
    print("hello world")
... ...
>>> print(hello.__code__)
<code object hello at 0x7f6c830bb810, file "<stdin>", line 1>

code 类型定义在 Include/code.h 中:

typedef struct {
  PyObject_HEAD
  int co_argcount;              /* #arguments, except *args */
  int co_kwonlyargcount;        /* #keyword only arguments */
  int co_nlocals;               /* #local variables */
  int co_stacksize;             /* #entries needed for evaluation stack */
  int co_flags;         /* CO_..., see below */
  PyObject *co_code;            /* instruction opcodes */
  PyObject *co_consts;  /* list (constants used) */
  PyObject *co_names;           /* list of strings (names used) */
  PyObject *co_varnames;        /* tuple of strings (local variable names) */
  PyObject *co_freevars;        /* tuple of strings (free variable names) */
  PyObject *co_cellvars;      /* tuple of strings (cell variable names) */
  /* The rest aren't used in either hash or comparisons, except for
     co_name (used in both) and co_firstlineno (used only in
     comparisons).  This is done to preserve the name and line number
     for tracebacks and debuggers; otherwise, constant de-duplication
     would collapse identical functions/lambdas defined on different lines.
  */
  unsigned char *co_cell2arg; /* Maps cell vars which are arguments. */
  PyObject *co_filename;        /* unicode (where it was loaded from) */
  PyObject *co_name;            /* unicode (name, for reference) */
  int co_firstlineno;           /* first source line number */
  PyObject *co_lnotab;  /* string (encoding addr<->lineno mapping) See
                           Objects/lnotab_notes.txt for details. */
  void *co_zombieframe;     /* for optimization only (see frameobject.c) */
  PyObject *co_weakreflist;   /* to support weakrefs to code objects */
} PyCodeObject;

可以看到 code 类型也有很多属性,不妨参考 PyCodeObject 的定义,用 Python 交互式看一看:

>>> hello.__code__.co_name
'hello'
>>> hello.__code__.co_code # 函数字节码也是属性之一
b't\x00\x00d\x01\x00\x83\x01\x00\x01d\x00\x00S'

3.2 对象引用

3.2.1 v=[1,2];v[1]=v 的打印结果

执行:

>>> v = [1, 2]
>>> v[1] = v
>>> v
[1, [...]]

为什么结果会出现 […]?

这个和 Lisp 里处理 list 一样的道理——出现无限循环引用了,见 Objects/listobject.c 中 list_repr 函数的实现,其中一个片段:

/* list 为空的情况下 */
if (Py_SIZE(v) == 0) {
  return PyUnicode_FromString("[]");
 }

/* 检查是否有引用自身 */
i = Py_ReprEnter((PyObject*)v);
if (i != 0) {
  return i > 0 ? PyUnicode_FromString("[...]") : NULL;
 }

打印 list 时,会先调用 Py_ReprEnter 函数检查是否存在无限循环,如果对象出现无限循环,就打印 […]。

如果把这段代码去掉,就无限循环打印了。

3.3 迭代对象

示例代码:

class Hello(object):
    i = 0

    def __iter__(self):
        return self

    def __next__(self):
        self.i += 1

        if self.i > 10:
            raise StopIteration()

        return self.i


hello = Hello()

for i in hello:
    print(i)

如果不需要继续迭代,抛出 StopIteration 异常就可以结束循环。

4 栈帧

每个函数在被调用时,Python 都会创建栈帧。Python 源码中,PyEval_EvalCodeEx 负责创建栈帧,PyEval_EvalCodeEx 实质上是调用 _PyEval_EvalCodeWithName 创建了新的栈帧。相关代码如下:

/* file: Python/ceval.c */

static PyObject *
_PyEval_EvalCodeWithName(PyObject *_co, PyObject *globals, PyObject *locals,
           PyObject **args, int argcount, PyObject **kws, int kwcount,
           PyObject **defs, int defcount, PyObject *kwdefs, PyObject *closure,
           PyObject *name, PyObject *qualname)
{
    PyCodeObject* co = (PyCodeObject*)_co;
    PyFrameObject *f;
    PyObject *retval = NULL;
    PyObject **fastlocals, **freevars;
    PyThreadState *tstate = PyThreadState_GET();
    PyObject *x, *u;
    int total_args = co->co_argcount + co->co_kwonlyargcount;
    int i;
    int n = argcount;
    PyObject *kwdict = NULL;

    if (globals == NULL) {
        PyErr_SetString(PyExc_SystemError,
                        "PyEval_EvalCodeEx: NULL globals");
        return NULL;
    }

    assert(tstate != NULL);
    assert(globals != NULL);

    /* 创建新的栈帧 */
    f = PyFrame_New(tstate, co, globals, locals);
    if (f == NULL)
        return NULL;

    fastlocals = f->f_localsplus;
    freevars = f->f_localsplus + co->co_nlocals;

    /* Parse arguments. */
    if (co->co_flags & CO_VARKEYWORDS) {
        kwdict = PyDict_New();
        if (kwdict == NULL)
            goto fail;
        i = total_args;
        if (co->co_flags & CO_VARARGS)
            i++;
        SETLOCAL(i, kwdict);
    }
    if (argcount > co->co_argcount)
        n = co->co_argcount;
    for (i = 0; i < n; i++) {
        x = args[i];
        Py_INCREF(x);
        SETLOCAL(i, x);
    }
    if (co->co_flags & CO_VARARGS) {
        u = PyTuple_New(argcount - n);
        if (u == NULL)
            goto fail;
        SETLOCAL(total_args, u);
        for (i = n; i < argcount; i++) {
            x = args[i];
            Py_INCREF(x);
            PyTuple_SET_ITEM(u, i-n, x);
        }
    }
    for (i = 0; i < kwcount; i++) {
        PyObject **co_varnames;
        PyObject *keyword = kws[2*i];
        PyObject *value = kws[2*i + 1];
        int j;
        if (keyword == NULL || !PyUnicode_Check(keyword)) {
            PyErr_Format(PyExc_TypeError,
                         "%U() keywords must be strings",
                         co->co_name);
            goto fail;
        }
        /* Speed hack: do raw pointer compares. As names are
           normally interned this should almost always hit. */
        co_varnames = ((PyTupleObject *)(co->co_varnames))->ob_item;
        for (j = 0; j < total_args; j++) {
            PyObject *nm = co_varnames[j];
            if (nm == keyword)
                goto kw_found;
        }
        /* Slow fallback, just in case */
        for (j = 0; j < total_args; j++) {
            PyObject *nm = co_varnames[j];
            int cmp = PyObject_RichCompareBool(
                keyword, nm, Py_EQ);
            if (cmp > 0)
                goto kw_found;
            else if (cmp < 0)
                goto fail;
        }
        if (j >= total_args && kwdict == NULL) {
            PyErr_Format(PyExc_TypeError,
                         "%U() got an unexpected "
                         "keyword argument '%S'",
                         co->co_name,
                         keyword);
            goto fail;
        }
        if (PyDict_SetItem(kwdict, keyword, value) == -1) {
            goto fail;
        }
        continue;
      kw_found:
        if (GETLOCAL(j) != NULL) {
            PyErr_Format(PyExc_TypeError,
                         "%U() got multiple "
                         "values for argument '%S'",
                         co->co_name,
                         keyword);
            goto fail;
        }
        Py_INCREF(value);
        SETLOCAL(j, value);
    }
    if (argcount > co->co_argcount && !(co->co_flags & CO_VARARGS)) {
        too_many_positional(co, argcount, defcount, fastlocals);
        goto fail;
    }
    if (argcount < co->co_argcount) {
        int m = co->co_argcount - defcount;
        int missing = 0;
        for (i = argcount; i < m; i++)
            if (GETLOCAL(i) == NULL)
                missing++;
        if (missing) {
            missing_arguments(co, missing, defcount, fastlocals);
            goto fail;
        }
        if (n > m)
            i = n - m;
        else
            i = 0;
        for (; i < defcount; i++) {
            if (GETLOCAL(m+i) == NULL) {
                PyObject *def = defs[i];
                Py_INCREF(def);
                SETLOCAL(m+i, def);
            }
        }
    }
    if (co->co_kwonlyargcount > 0) {
        int missing = 0;
        for (i = co->co_argcount; i < total_args; i++) {
            PyObject *name;
            if (GETLOCAL(i) != NULL)
                continue;
            name = PyTuple_GET_ITEM(co->co_varnames, i);
            if (kwdefs != NULL) {
                PyObject *def = PyDict_GetItem(kwdefs, name);
                if (def) {
                    Py_INCREF(def);
                    SETLOCAL(i, def);
                    continue;
                }
            }
            missing++;
        }
        if (missing) {
            missing_arguments(co, missing, -1, fastlocals);
            goto fail;
        }
    }

    /* Allocate and initialize storage for cell vars, and copy free
       vars into frame. */
    for (i = 0; i < PyTuple_GET_SIZE(co->co_cellvars); ++i) {
        PyObject *c;
        int arg;
        /* Possibly account for the cell variable being an argument. */
        if (co->co_cell2arg != NULL &&
            (arg = co->co_cell2arg[i]) != CO_CELL_NOT_AN_ARG) {
            c = PyCell_New(GETLOCAL(arg));
            /* Clear the local copy. */
            SETLOCAL(arg, NULL);
        }
        else {
            c = PyCell_New(NULL);
        }
        if (c == NULL)
            goto fail;
        SETLOCAL(co->co_nlocals + i, c);
    }
    for (i = 0; i < PyTuple_GET_SIZE(co->co_freevars); ++i) {
        PyObject *o = PyTuple_GET_ITEM(closure, i);
        Py_INCREF(o);
        freevars[PyTuple_GET_SIZE(co->co_cellvars) + i] = o;
    }

    if (co->co_flags & (CO_GENERATOR | CO_COROUTINE)) {
        PyObject *gen;
        PyObject *coro_wrapper = tstate->coroutine_wrapper;
        int is_coro = co->co_flags & CO_COROUTINE;

        if (is_coro && tstate->in_coroutine_wrapper) {
            assert(coro_wrapper != NULL);
            PyErr_Format(PyExc_RuntimeError,
                         "coroutine wrapper %.200R attempted "
                         "to recursively wrap %.200R",
                         coro_wrapper,
                         co);
            goto fail;
        }

        /* Don't need to keep the reference to f_back, it will be set
         * when the generator is resumed. */
        Py_CLEAR(f->f_back);

        PCALL(PCALL_GENERATOR);

        /* Create a new generator that owns the ready to run frame
         * and return that as the value. */
        if (is_coro) {
            gen = PyCoro_New(f, name, qualname);
        } else {
            gen = PyGen_NewWithQualName(f, name, qualname);
        }
        if (gen == NULL)
            return NULL;

        if (is_coro && coro_wrapper != NULL) {
            PyObject *wrapped;
            tstate->in_coroutine_wrapper = 1;
            wrapped = PyObject_CallFunction(coro_wrapper, "N", gen);
            tstate->in_coroutine_wrapper = 0;
            return wrapped;
        }

        return gen;
    }
    /* 这里执行指令 */
    retval = PyEval_EvalFrameEx(f,0);

fail: /* Jump here from prelude on failure */

    /* decref'ing the frame can cause __del__ methods to get invoked,
       which can call back into Python.  While we're done with the
       current Python frame (f), the associated C stack is still in use,
       so recursion_depth must be boosted for the duration.
    */
    assert(tstate != NULL);
    ++tstate->recursion_depth;
    Py_DECREF(f);
    --tstate->recursion_depth;
    return retval;
}

实际上每个对象作为函数调用时,都是由 function_call(Objects/funcobject.c)完成,实现如下:

static PyObject *
function_call(PyObject *func, PyObject *arg, PyObject *kw)
{
    PyObject *result;
    PyObject *argdefs;
    PyObject *kwtuple = NULL;
    PyObject **d, **k;
    Py_ssize_t nk, nd;

    argdefs = PyFunction_GET_DEFAULTS(func);
    if (argdefs != NULL && PyTuple_Check(argdefs)) {
        d = &PyTuple_GET_ITEM((PyTupleObject *)argdefs, 0);
        nd = PyTuple_GET_SIZE(argdefs);
    }
    else {
        d = NULL;
        nd = 0;
    }

    if (kw != NULL && PyDict_Check(kw)) {
        Py_ssize_t pos, i;
        nk = PyDict_Size(kw);
        kwtuple = PyTuple_New(2*nk);
        if (kwtuple == NULL)
            return NULL;
        k = &PyTuple_GET_ITEM(kwtuple, 0);
        pos = i = 0;
        while (PyDict_Next(kw, &pos, &k[i], &k[i+1])) {
            Py_INCREF(k[i]);
            Py_INCREF(k[i+1]);
            i += 2;
        }
        nk = i/2;
    }
    else {
        k = NULL;
        nk = 0;
    }

    /* 执行函数的代码 */
    result = PyEval_EvalCodeEx(
        PyFunction_GET_CODE(func),
        PyFunction_GET_GLOBALS(func), (PyObject *)NULL,
        &PyTuple_GET_ITEM(arg, 0), PyTuple_GET_SIZE(arg),
        k, nk, d, nd,
        PyFunction_GET_KW_DEFAULTS(func),
        PyFunction_GET_CLOSURE(func));

    Py_XDECREF(kwtuple);

    return result;
}

而最终执行指令的是由 PyEval_EvalFrameEx 负责,这是一个庞大的函数,里面实现了所有的指令。

4.1 globals 和 locals 实现

Python 虚拟机通过frame保存了调用栈的栈帧,每个栈帧保存了当前的上下文。globals 和 locals 函数分别返回所有全局栈帧和局部栈帧的对象。

globals 函数实际上返回的是第一个栈帧内的所有对象:

def test():
    a = 0
    print(locals())

test()
print("=" * 10)
print(globals())
$ /tmp python3 test.py
{'a': 0}
==========
{'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': <_frozen_importlib_external.SourceFileLoader object at 0x7fcba121f3c8>, '__spec__': None, '__annotations__': {}, '__builtins__': <module 'builtins' (built-in)>, '__file__': 'test.py', '__cached__': None, 'test': <function test at 0x7fcba924ee18>}

globals 函数实现在 Python/bltinmodule.c 中:

static PyObject *
builtin_globals_impl(PyObject *module)
{
    PyObject *d;

    d = PyEval_GetGlobals();
    Py_XINCREF(d);
    return d;
}

最关键是调用了 PyEval_GetGlobals 函数,该函数在 Python/ceval.c 中实现:

PyObject *
PyEval_GetGlobals(void)
{
    PyFrameObject *current_frame = PyEval_GetFrame(); /* 这里获取了当前栈的栈帧对象 */
    if (current_frame == NULL)
        return NULL;

    assert(current_frame->f_globals != NULL); /* 栈针里保存了全局对象的指针 */
    return current_frame->f_globals;
}

PyFrameObject 定义在 Include/frameobject.h 中,PyFrameObject 保存了栈帧里所有的对象以及局部对象,locals 函数的实现如下:

PyObject *
PyEval_GetLocals(void)
{
    PyFrameObject *current_frame = PyEval_GetFrame();
    if (current_frame == NULL) {
        PyErr_SetString(PyExc_SystemError, "frame does not exist");
        return NULL;
    }

    if (PyFrame_FastToLocalsWithError(current_frame) < 0)
        return NULL;

    assert(current_frame->f_locals != NULL);
    return current_frame->f_locals;
}

5 从字节码角度分析

测试代码:

hello = 'nihao'

保存为 test.py,然后调用 dis 库查看执行的字节码:

$ /tmp python3 -m dis test.py
  3           0 LOAD_CONST               0 ('nihao')
              2 STORE_NAME               0 (hello)
              4 LOAD_CONST               1 (None)
              6 RETURN_VALUE

字节码操作实现位于 Python/ceval.c 中,我们找到 STORE_NAME 的实现:

TARGET(STORE_NAME) {
  PyObject *name = GETITEM(names, oparg);
  PyObject *v = POP();
  PyObject *ns = f->f_locals;
  int err;
  if (ns == NULL) {
    PyErr_Format(PyExc_SystemError,
                 "no locals found when storing %R", name);
    Py_DECREF(v);
    goto error;
  }
  if (PyDict_CheckExact(ns))
    err = PyDict_SetItem(ns, name, v);
  else
    err = PyObject_SetItem(ns, name, v); /* 将对象存储到当前栈帧中 */
  Py_DECREF(v);
  if (err != 0)
    goto error;
  PyObject_Print(ns, stdout, 0);
  PyObject_Print(f->f_globals, stdout, 0);
  DISPATCH();
}

注意这句:

err = PyObject_SetItem(ns, name, v);

由于我们脚本的变量定义在顶层栈帧中的,所以当前的 f->f_locals 和 f->f_globals 是指向相同的地方,可以把脚本改一下来验证:

hello = 'nihao'
print(globals() == locals())

执行:

$ python3 test.py
True