4 ms·
That's what you call a DOM Parser - the problem with them is, as they serialize all the elements into objects, bigger XML files tend to eat up all of your RAM.
by cHaOs667 1y ago
That's what you call a DOM Parser - the problem with them is, as they serialize all the elements into objects, bigger XML files tend to eat up all of your RAM. And this is where SAX2 parsers come into play where you define tree based callbacks to process the data.
- mort96 1y agoThe solution is simple: don't have XML files that are many gigabytes in size.
- lyu07282 1y agoTell that to wikimedia, I've used libxml's SAX parser in the past to parse 80GB+ xml dumps.
- cHaOs667 1y agoDepending on the XML structure and the servers RAM - it can already happen while you approach 80-100 MB file sizes. And to be fair, in the Enterprise context, you are quite often not in a position to decide how big the export of another system is. But yes, back in 2010 we built preprocessing systems that checked XMLs and split them up in smaller chunks if they exceeded a certain size.
- stuaxo 1y agoSome formats are this and they are historical formats.
- iberator 1y agoA lot of teleco stuff dumps multi-gb stuff of xml hourly. Per BTS. Processing few TB of XML files on one server daily It's doable, just use the right tools and hacks :) Processing schema-less or broken schema stuff is always hilarious. Good times.
- senorrib 1y agoLol I love the upbeat tone here. Helps me deal with my PTSD after working with XML files.