5 ms·

it can be irritating, but you can just as easy parse ", " to "|" or something, by simple string replacing, pre parsing..

by tikumo 12y ago

sunir 12y ago
Think it through. What if there is free text in the field? "How are you, Sally?"
- lignuist 12y ago
  You can replace all commas with a placeholder (e.g. "#COMMA#"), replace the delimiter with a comma, parse the document and then replace all placeholders in the data with ",".
  - Someone 12y ago
    That does not work, unless that first replacement magically ignores the commas that are part of field separators. If you know how to write the code that does that, your problem is solved.
    - lignuist 12y ago
      I was referencing to "What if the character separating fields is not a comma?". And there it clearly works. I used this technique a few times with success. If you find a CSV file that has mixed field separator types, then you probably found a broken CSV file.
      - zAy0LfpBZLC8mAC 12y ago
        No, it doesn't. What if there is #COMMA# in one of the fields?
        lignuist 12y ago
        You just choose a placeholder that does not appear in the data. You could even implement it in a way that a placeholder is automatically selected upfront that does not appear in the data. When it comes to parsing, the thing is that you usually have to make some assumptions about the document structure.
        zAy0LfpBZLC8mAC 12y ago
        What if there is #COMMA, in one of the fields (but no #COMMA#)? Yes, the assumption you have to make is called the grammar, and you better have a parser that always does what the grammar says, and global text replacement is a technique that is easy to get wrong, difficult to prove correct, and completely unnecessary at that.
        lignuist 12y ago
        > What if there is #COMMA, in one of the fields (but no #COMMA#)? What should happen? Since #COMMA is not #COMMA#, it gets not replaced, because it does not match. Please keep in mind, that I replied to suni's very specific question and did not try to start a discussion about general parser theory. In practice, we find a lot of files that do not respect the grammar, but still need to find a way to make the data accessible.
        zAy0LfpBZLC8mAC 12y ago
        What would happen is that you first would replace #COMMA, with #COMMA#COMMA# and then later replace that with ,COMMA# , thus garbling the data. The way to make the data accessible is to request the producer to be fixed, it's that simple. If that is completely impossible, you'll have to figure out the grammar of the data that you actually have and build a parser for that. Your suggested strategy does not work.
        lignuist 12y ago
        I used that strategy for parsing gigabytes of CSVs containing arbitrary natural language from the web - try to get these files fixed, or figure out a grammar for gigabytes of fuzzy data... My approach never failed for me, so telling me that my strategy does not work is a strong claim, where it reliably did the job for me. Your examples are all valid, but what you are describing are theoretical attacks on the method, while the method works in almost all cases in practice. We are talking about two different viewpoints: dealing with large amounts of messy data on one hand and parser theory in an ideal cosmos on the other hand.
- chrismcb 12y ago
  If your delimiter appears in a field, the field needs to be quoted.
mrweasel 12y ago
True, but in my mind picking ", " indicate to me that they don't care or don't know what they're doing. I often run into something similar with XML. I've had more than one partner call or write me saying that the elements in a file are not in the right order. Every single time they've admitted to not actually using an XML parser. Don't do things that screw up the standard tools other developers depend on.
- fhars 12y ago
  Stuff not being in the correct order is a perfectly fine technical reason to reject an XML file if you have a DTD based workflow. It is actually quite difficult to specify that the order of elements is irrelevant (it goes with n!, so allowing six elements exactly once, but in arbitrary order in the contents of So in that case, you might have been the developer that screwed up the standard tools other developers dependan element makes that part of the DTD 720 times longer than specifying a fixed order). So they could have easily played the ball back into your field if they had know what they did...
  - mrweasel 12y ago
    That is much more complex than the issues we've seen. My issue is with say a <person> with the subelements <name> and <email>, it should never make any difference if email or name is first or second.
deleted 12y ago
[deleted]

it can be irritating, but you can just as easy parse ", " to "|" or something, by simple string replacing, pre parsing..

by tikumo 12y ago

it can be irritating, but you can just as easy parse ", " to "|" or something, by simple string replacing, pre parsing..

sunir 12y ago

Think it through. What if there is free text in the field? "How are you, Sally?"

lignuist 12y ago
You can replace all commas with a placeholder (e.g. "#COMMA#"), replace the delimiter with a comma, parse the document and then replace all placeholders in the data with ",".
- Someone 12y ago
  That does not work, unless that first replacement magically ignores the commas that are part of field separators. If you know how to write the code that does that, your problem is solved.
  - lignuist 12y ago
    I was referencing to "What if the character separating fields is not a comma?". And there it clearly works. I used this technique a few times with success. If you find a CSV file that has mixed field separator types, then you probably found a broken CSV file.
    - zAy0LfpBZLC8mAC 12y ago
      No, it doesn't. What if there is #COMMA# in one of the fields?
      - lignuist 12y ago
        You just choose a placeholder that does not appear in the data. You could even implement it in a way that a placeholder is automatically selected upfront that does not appear in the data. When it comes to parsing, the thing is that you usually have to make some assumptions about the document structure.
        zAy0LfpBZLC8mAC 12y ago
        What if there is #COMMA, in one of the fields (but no #COMMA#)? Yes, the assumption you have to make is called the grammar, and you better have a parser that always does what the grammar says, and global text replacement is a technique that is easy to get wrong, difficult to prove correct, and completely unnecessary at that.
        lignuist 12y ago
        > What if there is #COMMA, in one of the fields (but no #COMMA#)? What should happen? Since #COMMA is not #COMMA#, it gets not replaced, because it does not match. Please keep in mind, that I replied to suni's very specific question and did not try to start a discussion about general parser theory. In practice, we find a lot of files that do not respect the grammar, but still need to find a way to make the data accessible.
        zAy0LfpBZLC8mAC 12y ago
        What would happen is that you first would replace #COMMA, with #COMMA#COMMA# and then later replace that with ,COMMA# , thus garbling the data. The way to make the data accessible is to request the producer to be fixed, it's that simple. If that is completely impossible, you'll have to figure out the grammar of the data that you actually have and build a parser for that. Your suggested strategy does not work.
        lignuist 12y ago
        I used that strategy for parsing gigabytes of CSVs containing arbitrary natural language from the web - try to get these files fixed, or figure out a grammar for gigabytes of fuzzy data... My approach never failed for me, so telling me that my strategy does not work is a strong claim, where it reliably did the job for me. Your examples are all valid, but what you are describing are theoretical attacks on the method, while the method works in almost all cases in practice. We are talking about two different viewpoints: dealing with large amounts of messy data on one hand and parser theory in an ideal cosmos on the other hand.
chrismcb 12y ago
If your delimiter appears in a field, the field needs to be quoted.

mrweasel 12y ago

True, but in my mind picking ", " indicate to me that they don't care or don't know what they're doing. I often run into something similar with XML. I've had more than one partner call or write me saying that the elements in a file are not in the right order. Every single time they've admitted to not actually using an XML parser. Don't do things that screw up the standard tools other developers depend on.

fhars 12y ago
Stuff not being in the correct order is a perfectly fine technical reason to reject an XML file if you have a DTD based workflow. It is actually quite difficult to specify that the order of elements is irrelevant (it goes with n!, so allowing six elements exactly once, but in arbitrary order in the contents of So in that case, you might have been the developer that screwed up the standard tools other developers dependan element makes that part of the DTD 720 times longer than specifying a fixed order). So they could have easily played the ball back into your field if they had know what they did...
- mrweasel 12y ago
  That is much more complex than the issues we've seen. My issue is with say a <person> with the subelements <name> and <email>, it should never make any difference if email or name is first or second.

deleted 12y ago

[deleted]