8 ms·
You should use highest protocol of pickle, here are the numbers in my machine: In [5]: %timeit pickle.dumps(a) 1 loops, best of 3: 724 ms per loop
by sepeth 11y ago
You should use highest protocol of pickle, here are the numbers in my machine:
In [5]: %timeit pickle.dumps(a)
1 loops, best of 3: 724 ms per loop
In [6]: %timeit pickle.dumps(a, protocol=pickle.HIGHEST_PROTOCOL)
100 loops, best of 3: 12.4 ms per loop
- omginternets 11y agoOh wow! This is more than I dared hope for, many thanks!
- darkxanthos 11y agoWhat's the trade off here?
- AnkhMorporkian 11y agoBackwards compatibility. Anything pickled with HIGHEST_PROTOCOL will be unreadable in anything below Python 2.3.
- robzyb 11y agoSo my Ford Model T won't be able to read it? (That is my humorous way of saying that Python 2.3 is rather old and therefore HIGHEST_PROTOCOL seems like it would be a desirable trade off for most.)
- StavrosK 11y agoYeah, what the hell? Even 2.5 is way too old, I generally only support 2.6+ nowadays. If something breaks compatibility with 2.5, no problem there. Hell, Python 3 has been out for years!
- AnkhMorporkian 11y agoHey, I'm with you. I use 3.4 on almost everything, and 2.7 at worst anywhere else. I weep for those stuck using Python 2.2.
- StavrosK 11y agoNo, I'm agreeing with you. I want to transition to 3.4 completely (mypy looks amazing), but there are some small Django libraries (third-party libs) that aren't yet compatible :(
- AnkhMorporkian 11y agoMypy is indeed amazing, I can't get enough of it. Throw in asyncio and pathlib and you get the reason why I could never go back. You might look into trying to port the django libs by yourself if you have the skill to do so. 2to3 often gets you really far. If you don't, I'd definitely recommend opening a ticket on the project page. I've done that with a few libs I use, and for a couple of them the maintainers just totally forgot about them and got around to doing the conversion just because I asked.
- StavrosK 11y agoI did that already, I just have to find some time to do the conversion. The maintainer was kind enough to assist. How do you use MyPy? I'm particularly worried about two things: 1) If I'm building a library, I can't have MyPy as a requirement, but would still like to use it for the types checks. Is there a way to omit the import when distributing your library? 2) Can it only check parts of an application? Maybe I have a big Django app and don't want it to static-check Django and all the other imports every time, for example.
- AnkhMorporkian 11y agoSorry it took so long for me to get back to you; I only check HN from time to time. 1. No, afaik there's no way to omit the import. Sadly there's little in the way of macros or preprocessors in the python world at this point. 2. I'm fairly sure there's ways to use module stubs, but all my mypy work has dealt with the stdlib which presents no issue.
- Tenzer 11y agoIt might be worth noting that if you pickle data in Python 3 then it actually has protocol versions which aren't compatible with Python 2: https://docs.python.org/3/library/pickle.html#data-stream-format https://docs.python.org/3/library/pickle.html#data-stream-fo....