Can Python pickle lambda functions?

I have read in a number of threads that Python pickle/cPickle cannot pickle lambda functions. However the following code works, using Python 2.7.6:

import cPickle as pickle


if __name__ == "__main__":
s = pickle.dumps(lambda x, y: x+y)
f = pickle.loads(s)
assert f(3,4) == 7

So what is going on? Or, rather, what is the limit of pickling lambdas?

[EDIT] I think i know why this code runs. I forgot (sorry!) i am running stackless python, which has a form of micro-threads called tasklets executing a function. These tasklets can be halted, pickled, unpickled and continued, so i guess (asked on the stackless mailing list) that it also provides a way to pickle function bodies.

77634 次浏览
小开

No, Python can't pickle lambda functions:

>>> import cPickle as pickle
>>> s = pickle.dumps(lambda x,y: x+y)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy_reg.py", line 70, in _reduce_ex
raise TypeError, "can't pickle %s objects" % base.__name__
TypeError: can't pickle function objects

Not sure what you did that succeeded...

小开

Yes, python can pickle lambda functions… but only if you have something that uses copy_reg to register how to pickle lambda functions -- the package dill loads the copy_reg you need into the pickle registry for you, when you import dill.

Python 2.7.8 (default, Jul 13 2014, 02:29:54)
[GCC 4.2.1 Compatible Apple Clang 4.1 ((tags/Apple/clang-421.11.66))] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import dill  # the code below will fail without this line
>>>
>>> import pickle
>>> s = pickle.dumps(lambda x, y: x+y)
>>> f = pickle.loads(s)
>>> assert f(3,4) == 7
>>> f
<function <lambda> at 0x10aebdaa0>

get dill here: https://github.com/uqfoundation

小开

Python can pickle lambdas. We will cover Python 2 and 3 separately as implementation of pickle are different in different Python versions.

  • Python 3.6

In Python 3, there is no module named cPickle. We have pickle instead which also doesn't support pickling of lambda functions by default. Let's see it's dispatch table:

>> import pickle
>> pickle.Pickler.dispatch_table
<member 'dispatch_table' of '_pickle.Pickler' objects>

Wait. I tried looking up dispatch_table of pickle not _pickle. _pickle is the alternative and faster C implementation of pickle. But we haven't imported it yet! This C implementation is imported automatically, if it is available, at the end of pure Python pickle module.

# Use the faster _pickle if possible
try:
from _pickle import (
PickleError,
PicklingError,
UnpicklingError,
Pickler,
Unpickler,
dump,
dumps,
load,
loads
)
except ImportError:
Pickler, Unpickler = _Pickler, _Unpickler
dump, dumps, load, loads = _dump, _dumps, _load, _loads

We are still left with the question of pickling lambdas in Python 3. The answer is you CAN'T with the native pickle or _pickle. You will need to import dill or cloudpickle and use that instead of the native pickle module.

>> import dill
>> dill.loads(dill.dumps(lambda x:x))
<function __main__.<lambda>>
  • Python 2.7

pickle uses pickle registry which is nothing but a mapping from type to the function to use for serializing (pickling) objects of that type. You can see pickle registry as:

>> pickle.Pickler.dispatch


{bool: <function pickle.save_bool>,
instance: <function pickle.save_inst>,
classobj: <function pickle.save_global>,
float: <function pickle.save_float>,
function: <function pickle.save_global>,
int: <function pickle.save_int>,
list: <function pickle.save_list>,
long: <function pickle.save_long>,
dict: <function pickle.save_dict>,
builtin_function_or_method: <function pickle.save_global>,
NoneType: <function pickle.save_none>,
str: <function pickle.save_string>,
tuple: <function pickle.save_tuple>,
type: <function pickle.save_global>,
unicode: <function pickle.save_unicode>}

To pickle custom types, Python provides copy_reg module to register our functions. You can read more about it here. By default, copy_regmodule supports pickling of the following additional types:

>> import copy_reg
>> copy_reg.dispatch_table


{code: <function ipykernel.codeutil.reduce_code>,
complex: <function copy_reg.pickle_complex>,
_sre.SRE_Pattern: <function re._pickle>,
posix.statvfs_result: <function os._pickle_statvfs_result>,
posix.stat_result: <function os._pickle_stat_result>}

Now, type of lambda functions is types.FunctionType. However, the builtin function for this type function: <function pickle.save_global> is not able to serialize lambda functions. Therefore, all third party libraries like dill, cloudpickle, etc override the inbuilt method to serialize lambda functions with some additional logic. Let's import dill and see what it does.

>> import dill
>> pickle.Pickler.dispatch


{_pyio.BufferedReader: <function dill.dill.save_file>,
_pyio.TextIOWrapper: <function dill.dill.save_file>,
_pyio.BufferedWriter: <function dill.dill.save_file>,
_pyio.BufferedRandom: <function dill.dill.save_file>,
functools.partial: <function dill.dill.save_functor>,
operator.attrgetter: <function dill.dill.save_attrgetter>,
operator.itemgetter: <function dill.dill.save_itemgetter>,
cStringIO.StringI: <function dill.dill.save_stringi>,
cStringIO.StringO: <function dill.dill.save_stringo>,
bool: <function pickle.save_bool>,
cell: <function dill.dill.save_cell>,
instancemethod: <function dill.dill.save_instancemethod0>,
instance: <function pickle.save_inst>,
classobj: <function dill.dill.save_classobj>,
code: <function dill.dill.save_code>,
property: <function dill.dill.save_property>,
method-wrapper: <function dill.dill.save_instancemethod>,
dictproxy: <function dill.dill.save_dictproxy>,
wrapper_descriptor: <function dill.dill.save_wrapper_descriptor>,
getset_descriptor: <function dill.dill.save_wrapper_descriptor>,
member_descriptor: <function dill.dill.save_wrapper_descriptor>,
method_descriptor: <function dill.dill.save_wrapper_descriptor>,
file: <function dill.dill.save_file>,
float: <function pickle.save_float>,
staticmethod: <function dill.dill.save_classmethod>,
classmethod: <function dill.dill.save_classmethod>,
function: <function dill.dill.save_function>,
int: <function pickle.save_int>,
list: <function pickle.save_list>,
long: <function pickle.save_long>,
dict: <function dill.dill.save_module_dict>,
builtin_function_or_method: <function dill.dill.save_builtin_method>,
module: <function dill.dill.save_module>,
NotImplementedType: <function dill.dill.save_singleton>,
NoneType: <function pickle.save_none>,
xrange: <function dill.dill.save_singleton>,
slice: <function dill.dill.save_slice>,
ellipsis: <function dill.dill.save_singleton>,
str: <function pickle.save_string>,
tuple: <function pickle.save_tuple>,
super: <function dill.dill.save_functor>,
type: <function dill.dill.save_type>,
weakcallableproxy: <function dill.dill.save_weakproxy>,
weakproxy: <function dill.dill.save_weakproxy>,
weakref: <function dill.dill.save_weakref>,
unicode: <function pickle.save_unicode>,
thread.lock: <function dill.dill.save_lock>}

Now, let's try to pickle lambda function.

>> pickle.loads(pickle.dumps(lambda x:x))
<function __main__.<lambda>>

It WORKS!!

In Python 2 we have two versions of pickle -

import pickle # pure Python version
pickle.__file__ # <install directory>/python-2.7/lib64/python2.7/pickle.py


import cPickle # C extension
cPickle.__file__ # <install directory>/python-2.7/lib64/python2.7/lib-dynload/cPickle.so

Now, let's try to pickle lambda with C implementation cPickle.

>> import cPickle
>> cPickle.loads(cPickle.dumps(lambda x:x))
TypeError: can't pickle function objects

What went wrong? Let's see the dispatch table of cPickle.

>> cPickle.Pickler.dispatch_table
AttributeError: 'builtin_function_or_method' object has no attribute 'dispatch_table'

The implementation of pickle and cPickle is different. Importing dill makes only Python version of pickle work. The disadvantage of using pickle instead of cPickle is that it can be as much as 1000 times slower than cPickle.

I hope this clears all the doubts.

小开

Even though it might be obvious I would like to add an other possible solution. As you probably know lambda functions are just anonymous function declarations. If you don't have many lambdas that are used only once and it wouldn't add much noise to your code you could just name your lambda and pass the name of it (without the parentheses) like this:

import cPickle as pickle


def addition(x, y):
return x+y




if __name__ == "__main__":
s = pickle.dumps(addition)
f = pickle.loads(s)
assert f(3,4) == 7

The name also adds more semantic and you wouldn't need an additional dependency like Dill. But only do that if that outweighs the added noise of the additional function(s).

小开

Install dill

$ pip install dill

Touch a file

touch yeah.p

Now run this python3 script, 

import dill


dill.dump(lambda x:x+1, open('yeah.p', 'wb'))
my_lambda = dill.load(open('yeah.p', 'rb'))
print(my_lambda(2))  # 3
小开

what worked for me (windows 10, python 3.7) was to pass a function instead of a lambda function:

def merge(x):
return Image.merge("RGB", x.split()[::-1])


transforms.Lambda(merge)

instead of:

transforms.Lambda(lambda x: Image.merge("RGB", x.split()[::-1]))

no dill or cPickel needed.


提交答案