7 ms·

I find defaultdict, OrderedDict, namedtuple among other data structures/classes in the collections module to be incredibly useful. Another module that's packag

I find defaultdict, OrderedDict, namedtuple among other data structures/classes in the collections module to be incredibly useful. Another module that's packaged with the stdlib that's immensely useful is itertools. I especially find takewhile, cycle, and chain to be incredibly useful building blocks for list-related functions. I highly recommend a quick read. EDIT: functools is also great! Fantastic module for higher-order functions on callable objects. https://docs.python.org/3/library/itertools.html https://docs.python.org/3/library/itertools.html

BerislavLopac 2y ago
ChainMap might be the most underrated bit in the standard library.
- stevesimmons 2y ago
  For anyone wanting some more explanation, ChainMap can be used to build nested namespaces from a series of dicts without having to explicitly merge the names in each level. Updates to the whole ChainMap go into the top-level dict. The docs are here [0]. Some simple motivating applications: - Look up names in Python locals before globals before built-in functions: `pylookup = ChainMap(locals(), globals(), vars(builtins))` - Get config variables from various sources in priority order: `var_map = ChainMap(command_line_args, os.environ, defaults)` - Simulate layered filesystems - etc [0] https://docs.python.org/3/library/collections.html#collections.ChainMap https://docs.python.org/3/library/collections.html#collectio...
  - BerislavLopac 2y ago
    I found it perfect for structured logging, where you might want to modify some details of the logged structures (e.g. a password) without changing the underlying data.
tpoacher 2y ago
also more_itertools ! even less known than itertools, but equally useful.
padthai 2y ago
Why do you use OrderedDict for now that regular dicts are ordered by default?
- judicious 2y ago
  I work with different versions of Python3 (and 2 unfortunately) and some code is still in 3.6, hence I used OrderedDicts.
  - mixmastamyk 2y ago
    3.6 was the first with the new ordered by default dicts, even though wasn't specc'd until 3.7.
    - Izkata 2y ago
      It worked as an accidental implementation detail in CPython from some other optimization, but it wasn't intentional at the time. Because it wasn't intentional and wasn't part of the spec, that code could be incompatible with other interpreters like pypy or jython.
      - ericvsmith 2y ago
        See my comment and the linked email at https://github.com/ericvsmith/dataclasses?tab=readme-ov-file#compatibility https://github.com/ericvsmith/dataclasses?tab=readme-ov-file... for dataclasses and 3.6. I think it's still true.
        raymondh 2y ago
        The reason Guido didn't want 3.6 to guarantee dict ordering was to protect 3.5 projects from mysteriously failing when using code that implicitly relied on 3.6 behaviors (for example, cutting and pasting a snippet from StackOverflow). He thought that one cycle of "no ordering assumptions" would give a smoother transition. All 3.6 implementations would have dict ordering, but it was safer to not have people rely on it right away.
mont_tag 2y ago
My faves are the lru_cache, namedtuples, deques, chainmap, and all of the itertools.
sevensor 2y ago
I mostly migrated to frozen dataclasses from namedtuples when dataclasses became available. I’m curious about your preference for the namedtuple. Is it the lighter weight, the strong immutability, the easy destructing? Or is it that most tuples might as well be namedtuples? Those are the advantages I can think of anyway :)
- sgarland 2y ago
  The main thing I find myself using them for is `_make()`. From the canonical [0] example: import sqlite3 EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade') conn = sqlite3.connect('/companydata') cursor = conn.cursor() cursor.execute('SELECT name, age, title, department, paygrade FROM employees') for emp in map(EmployeeRecord._make, cursor.fetchall()): print(emp.name, emp.title) You could of course accomplish the same with a dictionary comprehension, but I find this to be less noisy. Also, they have `_asdict()` should you want to have the contents as a dict. [0]: https://docs.python.org/3/library/collections.html#collections.namedtuple https://docs.python.org/3/library/collections.html#collectio...
  - judicious 2y ago
    Dictionary comprehensions can be very elegant. List and dictionary comprehensions are very powerful and expressive abstractions. In fact, while not good practice you can pretty much write all Python code inside comprehensions including stuff regarding mutation. This is valid(as in it will run, but highly unidiomatic) code: quicksort = lambda arr: [pivot:=arr[0], left:= [x for x in arr[1:] if x < pivot], right := [x for x in arr[1:] if x >= pivot], quicksort(left) + [pivot] + quicksort(right)][-1] if len(arr) > 1 else arr print(quicksort([1, 33, -4, -2, 110, 5, 88]))
    - hansvm 2y ago
      Sometimes mutations in comprehensions are very expressive. def scan(items, f, initial): x = initial return (x := f(x, y) for y in items) There are lots of other short ways to write `scan`, but I don't think any of them map so clearly to a naive definition of what it's supposed to do.
matsemann 2y ago
I just wish python had some better ergonomics/syntactic sugar working with itertools and friends. Grouping and mapping and filtering and stuff quickly become so unwieldy without proper lambdas etc, especially as the typing is quite bad so after a few steps you're not sure what you even have. Just as recent as today I went to Kotlin to process something semicomplex even though we're a python shop, just because I wanted to bash my head in after a few attempts in python. A DS could probably solve it minutes with pandas or something, but again stringly typed and lots of guesswork. (It was actually a friendly algorithmic competition at work, I won, and even found a bug in the organizer's code that went undetected exactly because of this)
- judicious 2y ago
  I find converting things from map objects or filter objects back to lists to be a bit clunky. Not to mention chaining operations makes it even more clunky. Some syntatic sugar would go a long way.
mturmon 2y ago
I use defaultdict a lot - for accumulators when you're not sure about what is coming. Here's a simplified example: # a[star_name][instrument] = set of (seed, planet index) of visited planets a = defaultdict(lambda: defaultdict(set)) for row in rows: a[row.star][row.inst].add((row.seed, row.planet)) This is a dict-of-dict-of-set that is accumulating from a stream of rows, and I don't know what stars and instruments will be present. Another related tool is Counter (https://docs.python.org/3/library/collections.html#collections.Counter https://docs.python.org/3/library/collections.html#collectio...)
daniel_grady 2y ago
Although it's not part of the standard library, toolz is wonderful for rounding out these modules. https://toolz.readthedocs.io/en/latest/ https://toolz.readthedocs.io/en/latest/

BerislavLopac 2y ago

ChainMap might be the most underrated bit in the standard library.

stevesimmons 2y ago
For anyone wanting some more explanation, ChainMap can be used to build nested namespaces from a series of dicts without having to explicitly merge the names in each level. Updates to the whole ChainMap go into the top-level dict. The docs are here [0]. Some simple motivating applications: - Look up names in Python locals before globals before built-in functions: `pylookup = ChainMap(locals(), globals(), vars(builtins))` - Get config variables from various sources in priority order: `var_map = ChainMap(command_line_args, os.environ, defaults)` - Simulate layered filesystems - etc [0] https://docs.python.org/3/library/collections.html#collections.ChainMap https://docs.python.org/3/library/collections.html#collectio...
- BerislavLopac 2y ago
  I found it perfect for structured logging, where you might want to modify some details of the logged structures (e.g. a password) without changing the underlying data.

tpoacher 2y ago

also more_itertools ! even less known than itertools, but equally useful.

padthai 2y ago

Why do you use OrderedDict for now that regular dicts are ordered by default?

judicious 2y ago
I work with different versions of Python3 (and 2 unfortunately) and some code is still in 3.6, hence I used OrderedDicts.
- mixmastamyk 2y ago
  3.6 was the first with the new ordered by default dicts, even though wasn't specc'd until 3.7.
  - Izkata 2y ago
    It worked as an accidental implementation detail in CPython from some other optimization, but it wasn't intentional at the time. Because it wasn't intentional and wasn't part of the spec, that code could be incompatible with other interpreters like pypy or jython.
    - ericvsmith 2y ago
      See my comment and the linked email at https://github.com/ericvsmith/dataclasses?tab=readme-ov-file#compatibility https://github.com/ericvsmith/dataclasses?tab=readme-ov-file... for dataclasses and 3.6. I think it's still true.
      - raymondh 2y ago
        The reason Guido didn't want 3.6 to guarantee dict ordering was to protect 3.5 projects from mysteriously failing when using code that implicitly relied on 3.6 behaviors (for example, cutting and pasting a snippet from StackOverflow). He thought that one cycle of "no ordering assumptions" would give a smoother transition. All 3.6 implementations would have dict ordering, but it was safer to not have people rely on it right away.

mont_tag 2y ago

My faves are the lru_cache, namedtuples, deques, chainmap, and all of the itertools.

sevensor 2y ago

I mostly migrated to frozen dataclasses from namedtuples when dataclasses became available. I’m curious about your preference for the namedtuple. Is it the lighter weight, the strong immutability, the easy destructing? Or is it that most tuples might as well be namedtuples? Those are the advantages I can think of anyway :)

sgarland 2y ago
The main thing I find myself using them for is `_make()`. From the canonical [0] example: import sqlite3 EmployeeRecord = namedtuple('EmployeeRecord', 'name, age, title, department, paygrade') conn = sqlite3.connect('/companydata') cursor = conn.cursor() cursor.execute('SELECT name, age, title, department, paygrade FROM employees') for emp in map(EmployeeRecord._make, cursor.fetchall()): print(emp.name, emp.title) You could of course accomplish the same with a dictionary comprehension, but I find this to be less noisy. Also, they have `_asdict()` should you want to have the contents as a dict. [0]: https://docs.python.org/3/library/collections.html#collections.namedtuple https://docs.python.org/3/library/collections.html#collectio...
- judicious 2y ago
  Dictionary comprehensions can be very elegant. List and dictionary comprehensions are very powerful and expressive abstractions. In fact, while not good practice you can pretty much write all Python code inside comprehensions including stuff regarding mutation. This is valid(as in it will run, but highly unidiomatic) code: quicksort = lambda arr: [pivot:=arr[0], left:= [x for x in arr[1:] if x < pivot], right := [x for x in arr[1:] if x >= pivot], quicksort(left) + [pivot] + quicksort(right)][-1] if len(arr) > 1 else arr print(quicksort([1, 33, -4, -2, 110, 5, 88]))
  - hansvm 2y ago
    Sometimes mutations in comprehensions are very expressive. def scan(items, f, initial): x = initial return (x := f(x, y) for y in items) There are lots of other short ways to write `scan`, but I don't think any of them map so clearly to a naive definition of what it's supposed to do.

matsemann 2y ago

I just wish python had some better ergonomics/syntactic sugar working with itertools and friends. Grouping and mapping and filtering and stuff quickly become so unwieldy without proper lambdas etc, especially as the typing is quite bad so after a few steps you're not sure what you even have. Just as recent as today I went to Kotlin to process something semicomplex even though we're a python shop, just because I wanted to bash my head in after a few attempts in python. A DS could probably solve it minutes with pandas or something, but again stringly typed and lots of guesswork. (It was actually a friendly algorithmic competition at work, I won, and even found a bug in the organizer's code that went undetected exactly because of this)

judicious 2y ago
I find converting things from map objects or filter objects back to lists to be a bit clunky. Not to mention chaining operations makes it even more clunky. Some syntatic sugar would go a long way.

mturmon 2y ago

I use defaultdict a lot - for accumulators when you're not sure about what is coming. Here's a simplified example: # a[star_name][instrument] = set of (seed, planet index) of visited planets a = defaultdict(lambda: defaultdict(set)) for row in rows: a[row.star][row.inst].add((row.seed, row.planet)) This is a dict-of-dict-of-set that is accumulating from a stream of rows, and I don't know what stars and instruments will be present. Another related tool is Counter (https://docs.python.org/3/library/collections.html#collections.Counter https://docs.python.org/3/library/collections.html#collectio...)

daniel_grady 2y ago

Although it's not part of the standard library, toolz is wonderful for rounding out these modules. https://toolz.readthedocs.io/en/latest/ https://toolz.readthedocs.io/en/latest/