7 ms·

Sure thing :) Here's[0] a .tgz file with 3 files in it that are misidentified by magika but correctly identified by the `file` utility: asp.html, vba.html, unk

Sure thing :) Here's[0] a .tgz file with 3 files in it that are misidentified by magika but correctly identified by the `file` utility: asp.html, vba.html, unknown.woff These are files that were in one of my crawl datasets. [0]: https://poc.lol/files/magika-test.tgz https://poc.lol/files/magika-test.tgz

ebursztein 3y ago
Thank you - we are adding them to our test suit for the next version.
- TomNomNom 3y ago
  Super, thank you! I look forward to it :) I've worked on similar problems recently so I'm well aware of how difficult this is. An example I've given people is in automatically detecting base64-encoded data. It seems easy at first, but any four, eight, or twelve (etc) letter word is technically valid base64, so you need to decide if and how those things should be excluded.
beeboobaa 3y ago
Do you have permission to redistribute these files?
- IvyMike 3y ago
  You are asking what if this guy has "web crawl data" that google does not have? And what if he says no, he does not have permission.
  - beeboobaa 3y ago
    > You are asking what if this guy has "web crawl data" that google does not have? No, I'm asking if he has permission to redistribute these files.
    - timschmidt 3y ago
      Are you attempting to assert that use of these files solely for the purpose of improving a software system meant to classify file types does not fall under fair use? https://en.wikipedia.org/wiki/Fair_use https://en.wikipedia.org/wiki/Fair_use
      - beeboobaa 3y ago
        I'm asking a question. Here's another one for you: Do you believe that all pictures you have ever taken, all emails you have ever written, all code you have ever written could be posted here on this forum to improve someone else's software system? If so, could you go ahead and post that zip? I'd like to ingest it in my model.
        timschmidt 3y ago
        Your question seems orthogonal to the situation. The three files posted seem to be the minimum amount of information required to reproduce the bug. Fair use encompasses a LOT of uses of otherwise copyrighted work, and this seems clearly to be one.
        beeboobaa 3y ago
        I don't see how publicly posting them on a forum is > the minimum amount of information required to reproduce the bug MAYBE if they had communicated privately that'd be an argument that made sense.
        timschmidt 3y ago
        So you don't think that software development which happens in public web forums deserve fair use protection?
        beeboobaa 3y ago
        That's an interesting way to frame "publicly posted someone else's data without their consent for anyone to see and download"
- renewiltord 3y ago
  LOL nice b8 m8. For the rest of you who are curious, the files look like this: <HTML><HEAD> <TITLE>Access Denied</TITLE> </HEAD><BODY> <H1>Access Denied</H1> You don't have permission to access "http://placement.api.test4.example.com/" on this server.<P> Reference #18.9cb0f748.1695037739.283e2e00 </BODY> </HTML> Legend. "Do you have permission" hahaha.

ebursztein 3y ago

Thank you - we are adding them to our test suit for the next version.

TomNomNom 3y ago
Super, thank you! I look forward to it :) I've worked on similar problems recently so I'm well aware of how difficult this is. An example I've given people is in automatically detecting base64-encoded data. It seems easy at first, but any four, eight, or twelve (etc) letter word is technically valid base64, so you need to decide if and how those things should be excluded.

beeboobaa 3y ago

Do you have permission to redistribute these files?

IvyMike 3y ago
You are asking what if this guy has "web crawl data" that google does not have? And what if he says no, he does not have permission.
- beeboobaa 3y ago
  > You are asking what if this guy has "web crawl data" that google does not have? No, I'm asking if he has permission to redistribute these files.
  - timschmidt 3y ago
    Are you attempting to assert that use of these files solely for the purpose of improving a software system meant to classify file types does not fall under fair use? https://en.wikipedia.org/wiki/Fair_use https://en.wikipedia.org/wiki/Fair_use
    - beeboobaa 3y ago
      I'm asking a question. Here's another one for you: Do you believe that all pictures you have ever taken, all emails you have ever written, all code you have ever written could be posted here on this forum to improve someone else's software system? If so, could you go ahead and post that zip? I'd like to ingest it in my model.
      - timschmidt 3y ago
        Your question seems orthogonal to the situation. The three files posted seem to be the minimum amount of information required to reproduce the bug. Fair use encompasses a LOT of uses of otherwise copyrighted work, and this seems clearly to be one.
        beeboobaa 3y ago
        I don't see how publicly posting them on a forum is > the minimum amount of information required to reproduce the bug MAYBE if they had communicated privately that'd be an argument that made sense.
        timschmidt 3y ago
        So you don't think that software development which happens in public web forums deserve fair use protection?
        beeboobaa 3y ago
        That's an interesting way to frame "publicly posted someone else's data without their consent for anyone to see and download"
renewiltord 3y ago
LOL nice b8 m8. For the rest of you who are curious, the files look like this: <HTML><HEAD> <TITLE>Access Denied</TITLE> </HEAD><BODY> <H1>Access Denied</H1> You don't have permission to access "http://placement.api.test4.example.com/" on this server.<P> Reference #18.9cb0f748.1695037739.283e2e00 </BODY> </HTML> Legend. "Do you have permission" hahaha.