7 ms·
Loading Pydantic models from JSON without running out of memory
- deleted 1y ago[deleted]
- deleted 1y ago[deleted]
- thisguy47 1y agoI'd like to see a comparison of ijson vs just `json.load(f)`. `ujson` would also be interesting to see.
- itamarst 1y agoFor my PyCon 2025 talk I did this. Video isn't up yet, but slides are here: https://pythonspeed.com/pycon2025/slides/ https://pythonspeed.com/pycon2025/slides/ The linked-from-original-article ijson article was the inspiration for the talk: https://pythonspeed.com/articles/json-memory-streaming/ https://pythonspeed.com/articles/json-memory-streaming/
- fjasdfas 1y agoSo are there downsides to just always setting slots=True on all of my python data types?
- itamarst 1y agoYou can't add extra attributes that weren't part of the original dataclass definition: >>> from dataclasses import dataclass >>> @dataclass ... class C: pass ... >>> C().x = 1 >>> @dataclass(slots=True) ... class D: pass ... >>> D().x = 1 Traceback (most recent call last): File "<python-input-4>", line 1, in <module> D().x = 1 ^^^^^ AttributeError: 'D' object has no attribute 'x' and no __dict__ for setting new attributes Most of the time this is not a thing you actually need to do.
- masklinn 1y agoAlso some of the introspection stops working e.g. vars(). If you're using dataclasses it's less of an issue because dataclasses.asdict.
- monomial 1y agoI rarely need to dynamically add attributes myself on dataclasses like this but unfortunately this also means things like `@cached_property` won't work because it can't internally cache the method result anywhere.
- franga2000 1y agoIIRC you can just include a __dict__ slot and @cached_property should start working again. I
- jmugan 1y agoMy problem isn't running out of memory; it's loading in a complex model where the fields are BaseModels and unions of BaseModels multiple levels deep. It doesn't load it all the way and leaves some of the deeper parts as dictionaries. I need like almost a parser to search the space of different loads. Anyone have any ideas for software that does that?
- causasui 1y agoYou probably want to use Discriminated Unions https://docs.pydantic.dev/latest/concepts/unions/#discriminated-unions https://docs.pydantic.dev/latest/concepts/unions/#discrimina...
- jmugan 1y agoYeah, I'm doing that
- enragedcacti 1y agoThe only reason I can think of for the behavior you are describing is if one of the unioned types at some level of the hierarchy is equivalent to Dict[str, Any]. My understanding is that Pydantic will explore every option provided recursively and raise a ValidationError if none match but will never just give up and hand you a partially validated object. Are you able to share a snippet that reproduces what you're seeing?
- jmugan 1y agoThat's an interesting idea. It's possible there's a Dict[str,Any] in there. And yeah, my assumption was that it tried everything recursively, but I just wasn't seeing that, and my LLM council said that it did not. But I'll check for a Dict[str,Any]. Unfortunately, I don't have a minimal example, but making one should be my next step.
- enragedcacti 1y agoOne thing to watch out for while you debug is that the default 'smart' mode for union discrimination can be very unintuitive. As you can see in this example, an int vs a string can cause a different model to be chosen two layers up even though both are valid. You may have perfectly valid uses of Dict within your model that are being chosen in error because they result in less type coercion. left_to_right mode (or ideally discriminated unions if your data has easy discriminators) will be much more consistent. >>> class A(BaseModel): >>> a: int >>> class B(BaseModel): >>> b: A >>> class C(BaseModel): >>> c: B | Dict[str, Any] >>> C.model_validate({'c':{'b':{'a':1}}}) C(c=B(b=A(a=1))) >>> C.model_validate({'c':{'b':{'a':"1"}}}) C(c={'b': {'a': '1'}}) >>> class C(BaseModel): >>> c: B | Dict[str, Any] = Field(union_mode='left_to_right') >>> C.model_validate({'c':{'b':{'a':"1"}}}) C(c=B(b=A(a=1)))
- m_ke 1y agoOr just dump pydantic and use msgspec instead: https://jcristharif.com/msgspec/ https://jcristharif.com/msgspec/
- itamarst 1y agomsgspec is much more memory efficient out of the box, yes. Also quite fast.
- mbb70 1y agoA great feature of pydantic are the validation hooks that let you intercept serialization/deserialization of specific fields and augment behavior. For example if you are querying a DB that returns a column as a JSON string, trivial with Pydantic to json parse the column are part of deser with an annotation. Pydantic is definitely slower and not a 'zero cost abstraction', but you do get a lot for it.
- jtmcivor 1y agoOne approach to do that in msgspec is described here https://github.com/jcrist/msgspec/issues/375#issuecomment-1520301756 https://github.com/jcrist/msgspec/issues/375#issuecomment-15...
- deleted 1y ago[deleted]
- aitchnyu 1y agoCan it do incremental parsing? Cant tell from a brief look.
- jtmcivor 1y agoIIUC: * You still need to load all the bytes into memory before passing to msgspec decoding * You can decode a subset of fields, which is really helpful * Reusing msgspec decoders saves some cpu cycles https://jcristharif.com/msgspec/perf-tips.html#reuse-encoders-decoders https://jcristharif.com/msgspec/perf-tips.html#reuse-encoder... Slides 17, 18, 19 have an example of the first two points https://pythonspeed.com/pycon2025/slides/#17 https://pythonspeed.com/pycon2025/slides/#17
- zxilly 1y agoMaybe using mmap would also save some memory, I'm not quite sure if this can be implemented in Python.
- itamarst 1y agoOnce you switch to ijson it will not save any memory, no, because ijson essentially uses zero memory for the parsing. You're just left with the in-memory representation.
- dgan 1y agoi gave up on python dataclasses & json. Using protobufs object within the application itself. I also have a "...Mixin" class for almost every wire model, with extra methods Automatic, statically typed deserialization is worth the trouble in my opinion
- fidotron 1y agoHaving only recently encountered this, does anyone have any insight as to why it takes 2GB to handle a 100MB file? This looks highly reminiscent (though not exactly the same, pedants) of why people used to get excited about using SAX instead of DOM for xml parsing.
- itamarst 1y agoI talk about this more explicitly in the PyCon talk (https://pythonspeed.com/pycon2025/slides/ https://pythonspeed.com/pycon2025/slides/ - video soon) though that's not specifically about Pydantic, but basically: 1. Inefficient parser implementation. It's just... very easy to allocate way too much memory if you don't think about large-scale documents, and very difficult to measure. Common problem with many (but not all) JSON parsers. 2. CPython in-memory representation is large compared to compiled languages. So e.g. 4-digit integer is 5-6 bytes in JSON, 8 in Rust if you do i64, 25ish in CPython. An empty dictionary is 64 bytes.
- cozzyd 1y agoFunny to see awkward array in this context! (And... do people really store giant datasets in json?!?).
- jfb 1y agoMy sweet summer child
- chao- 1y agoOften the legacy of an engineer (or team) who "did what they had to do" to meet a deadline, and if they wanted to migrate to something better post-launch, weren't allowed to allocate time to go back and do so. At least JSON or CSV is better than the ad hoc homegrown formats you found at medium-sized companies that came out of the 90's and 00's.
- deleted 1y ago[deleted]
- deepsquirrelnet 1y agoAlternatively, if you had to go with json, you could consider using jsonl. I think I’d start by evaluating whether this is a good application for json. I tend to only want to use it for small files. Binary formats are usually much better in this scenario.
- kayson 1y agoHow does the speed of the dataclass version compare?
- scolvin 1y agoPydantic author here. We have plans for an improvement to pydantic where JSON is parsed iteratively, which will make way for reading a file as we parse it. Details in https://github.com/pydantic/pydantic/issues/10032 https://github.com/pydantic/pydantic/issues/10032. Our JSON parser, jiter (https://github.com/pydantic/jiter https://github.com/pydantic/jiter) already supports iterative parsing, so it's "just" a matter of solving the lifetimes in pydantic-core to validate as we parse. This should make pydantic around 3x faster at parsing JSON and significantly reduce the memory overhead.