7 ms·

Thanks for the feedback -- we will look into it. If you can share with us the list of URL that would be very helpful so we can reproduce - send us an email at m

Thanks for the feedback -- we will look into it. If you can share with us the list of URL that would be very helpful so we can reproduce - send us an email at magika-dev@google.com if that is possible. For crawling we have planned a head only model to avoid fetching the whole file but it is not ready yet -- we weren't sure what use-cases would emerge so that is good to know that such model might be useful. We mostly use Magika internally to route files for AV scanning as we wrote in the blog post, so it is possible that despite our best effort to test Magika extensively on various file types it is not as good on fonts format as it should be. We will look into. Thanks again for sharing your experience with Magika this is very useful.

TomNomNom 3y ago
Sure thing :) Here's[0] a .tgz file with 3 files in it that are misidentified by magika but correctly identified by the `file` utility: asp.html, vba.html, unknown.woff These are files that were in one of my crawl datasets. [0]: https://poc.lol/files/magika-test.tgz https://poc.lol/files/magika-test.tgz
- ebursztein 3y ago
  Thank you - we are adding them to our test suit for the next version.
  - TomNomNom 3y ago
    Super, thank you! I look forward to it :) I've worked on similar problems recently so I'm well aware of how difficult this is. An example I've given people is in automatically detecting base64-encoded data. It seems easy at first, but any four, eight, or twelve (etc) letter word is technically valid base64, so you need to decide if and how those things should be excluded.
- beeboobaa 3y ago
  Do you have permission to redistribute these files?
  - IvyMike 3y ago
    You are asking what if this guy has "web crawl data" that google does not have? And what if he says no, he does not have permission.
    - beeboobaa 3y ago
      > You are asking what if this guy has "web crawl data" that google does not have? No, I'm asking if he has permission to redistribute these files.
      - timschmidt 3y ago
        Are you attempting to assert that use of these files solely for the purpose of improving a software system meant to classify file types does not fall under fair use? https://en.wikipedia.org/wiki/Fair_use https://en.wikipedia.org/wiki/Fair_use
        beeboobaa 3y ago
        I'm asking a question. Here's another one for you: Do you believe that all pictures you have ever taken, all emails you have ever written, all code you have ever written could be posted here on this forum to improve someone else's software system? If so, could you go ahead and post that zip? I'd like to ingest it in my model.
        timschmidt 3y ago
        Your question seems orthogonal to the situation. The three files posted seem to be the minimum amount of information required to reproduce the bug. Fair use encompasses a LOT of uses of otherwise copyrighted work, and this seems clearly to be one.
        beeboobaa 3y ago
        I don't see how publicly posting them on a forum is > the minimum amount of information required to reproduce the bug MAYBE if they had communicated privately that'd be an argument that made sense.
Solvency 3y ago
[flagged]
westurner 3y ago
What is the MIME type of a .tar file; and what are the MIME types of the constituent concatenated files within an archive format like e.g. tar? hachoir/subfile/main.py: https://github.com/vstinner/hachoir/blob/main/hachoir/subfile/main.py https://github.com/vstinner/hachoir/blob/main/hachoir/subfil... File signature: https://en.wikipedia.org/wiki/File_signature https://en.wikipedia.org/wiki/File_signature PhotoRec: https://en.wikipedia.org/wiki/PhotoRec https://en.wikipedia.org/wiki/PhotoRec "File Format Gallery for Kaitai Struct"; 185+ binary file format specifications: https://formats.kaitai.io/ https://formats.kaitai.io/ Table of ': https://formats.kaitai.io/xref.html https://formats.kaitai.io/xref.html AntiVirus software > Identification methods > Signature-based detection, Heuristics, and ML/AI data mining: https://en.wikipedia.org/wiki/Antivirus_software#Identification_methods https://en.wikipedia.org/wiki/Antivirus_software#Identificat... Executable compression; packer/loader: https://en.wikipedia.org/wiki/Executable_compression https://en.wikipedia.org/wiki/Executable_compression Shellcode database > MSF: https://en.wikipedia.org/wiki/Shellcode_database https://en.wikipedia.org/wiki/Shellcode_database sigtool.c: https://github.com/Cisco-Talos/clamav/blob/main/sigtool/sigtool.c https://github.com/Cisco-Talos/clamav/blob/main/sigtool/sigt... clamav sigtool: https://www.google.com/search?q=clamav+sigtool https://www.google.com/search?q=clamav+sigtool https://blog.didierstevens.com/2017/07/14/clamav-sigtool-decode-sigs/ https://blog.didierstevens.com/2017/07/14/clamav-sigtool-dec... : sigtool –-find-sigs "$name" | sigtool –-decode-sigs List of file signatures: https://en.wikipedia.org/wiki/List_of_file_signatures https://en.wikipedia.org/wiki/List_of_file_signatures And then also clusterfuzz/oss-fuzz scans .txt source files with (sandboxed) Static and Dynamic Analysis tools, and `debsums`/`rpm -Va` verify that files on disk have the same (GPG signed) checksums as the package they are supposed to have been installed from, and a file-based HIDS builds a database of file hashes and compares what's on disk in a later scan with what was presumed good, and ~gdesktop LLM tools scan every file, and there are extended filesystem attributes for label-based MAC systems like SELinux, oh and NTFS ADS. A sufficient cryptographic hash function yields random bits with uniform probability. DRBG Deterministic Random Bit Generators need high entropy random bits in order to continuously re-seed the RNG random number generator. Is it safe to assume that hashing (1) every file on disk, or (2) any given file on disk at random, will yield random bits with uniform probability; and (3) why Argon2 instead of e.g. only two rounds of SHA256? https://github.com/google/osv.dev/blob/master/README.md#using-the-scanner https://github.com/google/osv.dev/blob/master/README.md#usin... : > We provide a Go based tool that will scan your dependencies, and check them against the OSV database for known vulnerabilities via the OSV API. ... With package metadata, not (a file hash, package) database that could be generated from OSV and the actual package files instead of their manifest of already-calculated checksums. Might as well be heating a pool on the roof with all of this waste heat from hashing binaries build from code of unknown static and dynamic quality. Add'l useful formats: > Currently it is able to scan various lockfiles, debian docker containers, SPDX and CycloneDB SBOMs, and git repositories Things like bittorrent magnet URIs, Named Data Networking, and IPFS are (file-hash based) "Content addressable storage": https://en.wikipedia.org/wiki/Content-addressable_storage https://en.wikipedia.org/wiki/Content-addressable_storage

TomNomNom 3y ago

Sure thing :) Here's[0] a .tgz file with 3 files in it that are misidentified by magika but correctly identified by the `file` utility: asp.html, vba.html, unknown.woff These are files that were in one of my crawl datasets. [0]: https://poc.lol/files/magika-test.tgz https://poc.lol/files/magika-test.tgz

ebursztein 3y ago
Thank you - we are adding them to our test suit for the next version.
- TomNomNom 3y ago
  Super, thank you! I look forward to it :) I've worked on similar problems recently so I'm well aware of how difficult this is. An example I've given people is in automatically detecting base64-encoded data. It seems easy at first, but any four, eight, or twelve (etc) letter word is technically valid base64, so you need to decide if and how those things should be excluded.
beeboobaa 3y ago
Do you have permission to redistribute these files?
- IvyMike 3y ago
  You are asking what if this guy has "web crawl data" that google does not have? And what if he says no, he does not have permission.
  - beeboobaa 3y ago
    > You are asking what if this guy has "web crawl data" that google does not have? No, I'm asking if he has permission to redistribute these files.
    - timschmidt 3y ago
      Are you attempting to assert that use of these files solely for the purpose of improving a software system meant to classify file types does not fall under fair use? https://en.wikipedia.org/wiki/Fair_use https://en.wikipedia.org/wiki/Fair_use
      - beeboobaa 3y ago
        I'm asking a question. Here's another one for you: Do you believe that all pictures you have ever taken, all emails you have ever written, all code you have ever written could be posted here on this forum to improve someone else's software system? If so, could you go ahead and post that zip? I'd like to ingest it in my model.
        timschmidt 3y ago
        Your question seems orthogonal to the situation. The three files posted seem to be the minimum amount of information required to reproduce the bug. Fair use encompasses a LOT of uses of otherwise copyrighted work, and this seems clearly to be one.
        beeboobaa 3y ago
        I don't see how publicly posting them on a forum is > the minimum amount of information required to reproduce the bug MAYBE if they had communicated privately that'd be an argument that made sense.

Solvency 3y ago

[flagged]

westurner 3y ago

What is the MIME type of a .tar file; and what are the MIME types of the constituent concatenated files within an archive format like e.g. tar? hachoir/subfile/main.py: https://github.com/vstinner/hachoir/blob/main/hachoir/subfile/main.py https://github.com/vstinner/hachoir/blob/main/hachoir/subfil... File signature: https://en.wikipedia.org/wiki/File_signature https://en.wikipedia.org/wiki/File_signature PhotoRec: https://en.wikipedia.org/wiki/PhotoRec https://en.wikipedia.org/wiki/PhotoRec "File Format Gallery for Kaitai Struct"; 185+ binary file format specifications: https://formats.kaitai.io/ https://formats.kaitai.io/ Table of ': https://formats.kaitai.io/xref.html https://formats.kaitai.io/xref.html AntiVirus software > Identification methods > Signature-based detection, Heuristics, and ML/AI data mining: https://en.wikipedia.org/wiki/Antivirus_software#Identification_methods https://en.wikipedia.org/wiki/Antivirus_software#Identificat... Executable compression; packer/loader: https://en.wikipedia.org/wiki/Executable_compression https://en.wikipedia.org/wiki/Executable_compression Shellcode database > MSF: https://en.wikipedia.org/wiki/Shellcode_database https://en.wikipedia.org/wiki/Shellcode_database sigtool.c: https://github.com/Cisco-Talos/clamav/blob/main/sigtool/sigtool.c https://github.com/Cisco-Talos/clamav/blob/main/sigtool/sigt... clamav sigtool: https://www.google.com/search?q=clamav+sigtool https://www.google.com/search?q=clamav+sigtool https://blog.didierstevens.com/2017/07/14/clamav-sigtool-decode-sigs/ https://blog.didierstevens.com/2017/07/14/clamav-sigtool-dec... : sigtool –-find-sigs "$name" | sigtool –-decode-sigs List of file signatures: https://en.wikipedia.org/wiki/List_of_file_signatures https://en.wikipedia.org/wiki/List_of_file_signatures And then also clusterfuzz/oss-fuzz scans .txt source files with (sandboxed) Static and Dynamic Analysis tools, and `debsums`/`rpm -Va` verify that files on disk have the same (GPG signed) checksums as the package they are supposed to have been installed from, and a file-based HIDS builds a database of file hashes and compares what's on disk in a later scan with what was presumed good, and ~gdesktop LLM tools scan every file, and there are extended filesystem attributes for label-based MAC systems like SELinux, oh and NTFS ADS. A sufficient cryptographic hash function yields random bits with uniform probability. DRBG Deterministic Random Bit Generators need high entropy random bits in order to continuously re-seed the RNG random number generator. Is it safe to assume that hashing (1) every file on disk, or (2) any given file on disk at random, will yield random bits with uniform probability; and (3) why Argon2 instead of e.g. only two rounds of SHA256? https://github.com/google/osv.dev/blob/master/README.md#using-the-scanner https://github.com/google/osv.dev/blob/master/README.md#usin... : > We provide a Go based tool that will scan your dependencies, and check them against the OSV database for known vulnerabilities via the OSV API. ... With package metadata, not (a file hash, package) database that could be generated from OSV and the actual package files instead of their manifest of already-calculated checksums. Might as well be heating a pool on the roof with all of this waste heat from hashing binaries build from code of unknown static and dynamic quality. Add'l useful formats: > Currently it is able to scan various lockfiles, debian docker containers, SPDX and CycloneDB SBOMs, and git repositories Things like bittorrent magnet URIs, Named Data Networking, and IPFS are (file-hash based) "Content addressable storage": https://en.wikipedia.org/wiki/Content-addressable_storage https://en.wikipedia.org/wiki/Content-addressable_storage