9 ms·
Extracting AI models from mobile apps
- VectorLock 2y agoExcellent introduction to some cool tools I wasn't aware of!
- powtain-gen1 2y agoWelcome to check out Sam Altman’s January 5, 2025 blog post, “Reflections.” https://web.powtain.com/pow/qao631 https://web.powtain.com/pow/qao631
- do_not_redeem 2y agoCan anyone explain that resize_to_320.tflite file? Surely they aren't using an AI model to resize images? Right?
- smitop 2y agotflite files can contain a ResizeOp that resizes the image: https://ai.google.dev/edge/api/tflite/java/org/tensorflow/lite/support/image/ops/ResizeOp https://ai.google.dev/edge/api/tflite/java/org/tensorflow/li... The file is only 7.7kb, so it couldn't contain many weights anyways.
- raydiak 2y agoExactly. Put another way, tensorflow is not an AI. You can build an AI in tensorflow. You can also resize images in tensorflow (using the traditional algorithms, not AI). I am not an expert, but as I understand, it is common for vision models to require a fixed resolution input, and it is common for that resolution to be quite low due to resource constraints.
- koe123 2y agoProbably not what your alluding to but AI upscaling of images is definitely a thing
- JTyQZSnP3cQGa8B 2y ago> Keep in mind that AI models [...] are considered intellectual property Is it ironic or missing a /s? I can't really tell here.
- Freak_NL 2y agoStandard disclaimer. Like inserting a bunch of 'hypothetically' in a comment telling one where to find some piece of abandoned media where using an unsanctioned channel would entail infringing upon someone's intellectual property.
- SunlitCat 2y agoTo be honest, that was my first thought on reading that headline as well. Given that especially those large companies (but who knows how smaller ones got their training data) got a huge amount of backlash for their unprecedented collection of data all over the web and not just there but everywhere else, it's kinda ironic to talk about intellectual property. If you use one of those AI model as a basis for your AI model the real danger could be that the owners of the originating data are going after you at some point as well.
- ToucanLoucan 2y agoStandard corporate hypocrisy. "Rules for thee, not for me." If you actually expected anything to be open about OpenAI's products, please get in touch, I have an incredible business opportunity for you in the form of a bridge in New York.
- xdennis 2y agoThey got backlash, but (if I'm not mistaken) it was ruled that it's okay to use copyrighted works in your model. So if a model is copyrighted, you should still be able to use it if you generate a different one based on it. I.e. copyright laundry. I assume this would be similar to how fonts work. You can copyright a font file, but not the actual shapes. So if you re-encode the shapes with different points, that's legal. But, I don't think a model can be copyrighted. Isn't it the case that something created mechanically can't be copyrighted? It has to be authored by a person. I find it weird that so many hackers go out of their way to approve of the legal claims of Big AI before it's even settled, instead of undermining Big AI. Isn't the hacker ethos all about decentralization?
- wat10000 2y ago“ Keep in mind that AI models, like most things, are considered intellectual property. Before using or modifying any extracted models, you need the explicit permission of their owner.” That’s not true, is it? It would be a copyright violation to distribute an extracted model, but you can do what you want with it yourself.
- wslh 2y agoIt's also worth noting that there is still no legal clarity on these issues, even if a license claims to provide specific permissions. Additionally, the debate around the sources companies use to train their models remains unresolved, raising ethical and legal questions about data ownership and consent.
- jdietrich 2y agoCircumventing a copy-prevention system without a valid exemption is a crime, even if you don't make unlawful copies. Copyright covers the right to make copies, not the right to distribute; "doing what you want with it yourself" may or may not be covered by fair use. Whether or not model weights are copyrightable remains an open question. https://www.law.cornell.edu/uscode/text/17/1201 https://www.law.cornell.edu/uscode/text/17/1201
- mcny 2y ago>Circumventing a copy-prevention system without a valid exemption is a crime, even if you don't make unlawful copies. Copyright covers the right to make copies, not the right to distribute; "doing what you want with it yourself" may or may not be covered by fair use. Whether or not model weights are copyrightable remains an open question. If that is the law, it is a defect that we need to fix. Laws do not come down from heaven in the form of commandments. We, humans, write laws. If there is a defect in the laws, we should fix it. If this is the law, time shifting and format shifting is unlawful as well which to me is unacceptable. Disclaimer: As usual, I anal.
- dialup_sounds 2y ago
- 23B1 2y ago> hoarding data Laundering IP. FTFY.
- jonpo 2y agoWell done you seem to have liberated an open model trained on open data for blind and visually impaired people. Paper: https://arxiv.org/pdf/2204.03738 https://arxiv.org/pdf/2204.03738 Code: https://github.com/microsoft/banknote-net https://github.com/microsoft/banknote-net Training data: https://raw.githubusercontent.com/microsoft/banknote-net/refs/heads/main/data/banknote_net.csv https://raw.githubusercontent.com/microsoft/banknote-net/ref... model: https://github.com/microsoft/banknote-net/blob/main/models/banknote_net_encoder.h5 https://github.com/microsoft/banknote-net/blob/main/models/b... Kinda easier to download it straight from github. Its licenced under MIT and CDLA-Permissive-2.0 licenses. But lets not let that get in the way of hating on AI shall we?
- cess11 2y ago[flagged]
- jonpo 2y agoYes nothing wrong with cool software or showing people how to use it for useful things. Sorry I'm just kind of sick of the whole 'kool aid', 'rage against AI' thing a lot of people seem to have going on and the way is presented in the post. I have family members with vision impairment helped by this particular app so its a bit personal. Nothing against opening stuff up and understanding how it works etc. I'd just rather see people build/train useful new models and stuff with the open datasets / models already available. I guess AI kind of does pay my bills in a round about way.
- a2128 2y agoSadly companies will hoard datasets and model research in the name of competitive advantage. Obviously with this specific model Microsoft chose to make it open, but this is not always the case, and it's not uncommon to read papers or technical reports saying they trained on an "internal dataset"
- jonpo 2y ago
- nthingtohide 2y agoOne thing I noticed in Gboard is it uses homeomorphic encryption to do federated learning of common words used amongst public to do encrypted suggestions. E.g. there are two common spelling of bizarre which are popular on Gboard : bizzare and bizarre. Can something similar help in model encryption?
- antman 2y agoHad to look it up, this seems to be the paper https://research.google/pubs/federated-learning-for-mobile-keyboard-prediction-2/ https://research.google/pubs/federated-learning-for-mobile-k...
- 1oooqooq 2y agothey have a very "interesting" definition of private data on the paper. it's so outlandish that if you buy their definition, there's zero value on the trained data. heh. they also claim unsuppervisioned users typing away is better than tagged training data, which explain the wild grammar suggestions on the top comment. guess the age of quantity over quality is finally peaking. in the end it's the same as grammarly but without any verification of the interested data, and calling the collection of user data "federation"
- nthingtohide 2y agoactually letting users type whatever they want is good because they are many dialects of english : chinglish, thailish, singlish, hinglish and so on. they have made the system so general that it can handle any quirk users throw at it.
- umeshunni 2y agoHomomorphic, not homeomorphic
- hyperbovine 2y ago`enc(coffee cup) == enc(donut)` would be an interesting guarantee.
- boothby 2y agoIf I understand the position of major players in this field, downloading models in bulk and training a ML model on that corpus shouldn't violate anybody's IP.
- zitterbewegung 2y agoIANAL But, this is not true it would be a piece of the software. If there is a copyright on the app itself it would extend to the model. Even models have licenses for example LLAMA is release under this license [1] [1] https://github.com/meta-llama/llama/blob/main/LICENSE https://github.com/meta-llama/llama/blob/main/LICENSE
- blitzar 2y agoIf I understand the position of major players in this field, copyright itself is optional (for them at least).
- rusk 2y agoThey claim “safe harbour” - if nobody complains it’s fair game
- zitterbewegung 2y agoTrue, I think there has to be a case that sets precedent for this issue.
- Drakim 2y agoIs there a material difference between the copyright laws for software and the copyright laws for images and text?
- boothby 2y agoLLMs are trained on works -- software, graphics and text -- covered by my copyright. What's the difference?
- 2y ago
- avg_dev 2y agopretty cool; that frida tool seems really nice. https://frida.re/docs/home/ https://frida.re/docs/home/ (and a bunch of people seem to be interested in the "IP" note, but I took as, just trying to not get run into legal trouble for advertising "here's how you can 'steal' models!")
- frogsRnice 2y agofrida is an amazing tool - it has empowered me to do things that would have otherwise took weeks or even months. This video is a little old, but the creator is also cracked https://www.youtube.com/watch?v=CLpW1tZCblo https://www.youtube.com/watch?v=CLpW1tZCblo It's supposed to be "free-IDA" and the work put in by the developers and maintainers is truly phenomenal. EDIT: This isn't really an attack imo. If you are going to take "secrets" and shove it into a mobile app, they can't really be considered secret. I suppose it's a tradeoff - if you want to do this kind of thing client-side - the secret sauce isn't so secret.
- Polizeiposaune 2y agoYou wouldn't train a LLM on a corpus containing copyrighted works without ensuring you had the necessary rights to the works, would you?
- deadbabe 2y agoFair use.
- griomnib 2y agoEasy to claim, harder to justify once you start charging money for your subsequent creation. Unless all LLM are a ruthless parody of human intelligence, which they may be, the legal issues will continue.
- dijksterhuis 2y ago*only available in the USA, terms and conditions apply. most other places use fair dealing which is more restrictive https://en.m.wikipedia.org/wiki/Fair_dealing https://en.m.wikipedia.org/wiki/Fair_dealing
- bayindirh 2y agoThe moment you earn money from it, that's not fair use anymore. When I last checked, unlimited access to said models were not free, plus it's not "research" anymore. - Addenda - For the interested parties, the law states the following [0]. Notwithstanding the provisions of sections 17 U.S.C. § 106 and 17 U.S.C. § 106A, the fair use of a copyrighted work, including such use by reproduction in copies or phonorecords or by any other means specified by that section, for purposes such as criticism, comment, news reporting, teaching (including multiple copies for classroom use), scholarship, or research, is not an infringement of copyright. In determining whether the use made of a work in any particular case is a fair use the factors to be considered shall include: 1. the purpose and character of the use, including whether such use is of a commercial nature or is for nonprofit educational purposes; 2. the nature of the copyrighted work; 3. the amount and substantiality of the portion used in relation to the copyrighted work as a whole; and 4. the effect of the use upon the potential market for or value of the copyrighted work. The fact that a work is unpublished shall not itself bar a finding of fair use if such finding is made upon consideration of all the above factors So, if you say that these factors can be flexed depending on the defendant, and can be just waved away to protect the wealthy, then it becomes something else, but given these factors, and how damaging this "fair use" is, I can certainly say that training AI models with copyrighted corpus is not fair use in any way. Of course at the end of the day, IANAL & IANAJ. However, my moral compass directly bars use of copyrighted corpus in publicly accessible, for profit models which undermine many people of their livelihoods. From my perspective, people can whitewash AI training as they see fit to sleep sound at night, but this doesn't change anything from my PoV. [0]: https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factors https://en.wikipedia.org/wiki/Fair_use#U.S._fair_use_factors
- deleted 2y ago[deleted]
- ipsum2 2y agoThis is cool, but only the first part in extracting a ML model for usage. The second part is reverse engineering the tokenizer and input transformations that are needed to before passing the data to the model, and outputting a human readable format.
- rob_c 2y agoIf you can't fix this with a little help from chatgpt or Google you shouldn't be building the models frankly let alone mucking with other people's...
- deleted 2y ago[deleted]
- refulgentis 2y agoThis is a good comment, but only in the sense it documents a model file doesn't run the model by itself. An analogous situation is seeing a blog that purports to "show you code", and the code returns an object, and commenting "This is cool, but doesn't show you how to turn a function return value into a human readable format" More noise, than signal. The techniques in the article are trivially understood to also apply to discovering the input tokenization format, and Netron shows you the types of inputs and outputs. Thanks for the article OP, really fascinating.
- ipsum2 2y agoJust having the shape of the input and output are not sufficient, the image (in this example) needs to be normalized. It's presumably not difficult to find the exact numbers, but it is a source of errors when reverse engineering a ML model.
- refulgentis 2y agoRight, you get it: it's a Frida problem.
- 2y ago
- 1vuio0pswjnm7 2y ago"Keep in mind that AI models, like most things, are considered intellectual property. Before using or modifying any extracted models, you need the explicit permission of their owner." If weights and biases contained in "AI models" are prorietary, then for one model owner to detect infingement by another model owner, it may be necessary to download and extract.
- 1vuio0pswjnm7 2y agoWhere the model owner is not the owner of the training data consider also that weights may be derivative works: https://www.arxiv.org/pdf/2407.13493 https://www.arxiv.org/pdf/2407.13493
- deleted 2y ago[deleted]
- kittikitti 2y agoThis was a great article and I really appreciate it!
- greggawatt 2y ago[dead]
- Fragoel2 2y agoThere's an interesting research paper from a few years ago that extracted models from Android apps on a large scale: https://impillar.github.io/files/ccs2022advdroid.pdf https://impillar.github.io/files/ccs2022advdroid.pdf
- janalsncm 2y agoI’m a huge fan of ML on device. It’s a big improvement in privacy for the user. That said, there’s always a chance for the user to extract your model, so on-device models will need to be fairly generic.
- Zambyte 2y agoMaybe someday we will build a society where standing on the shoulders of giants is encouraged, even when they haven't been dead for 100 years yet.
- andrewfromx 2y agothis would be yellow in https://en.wikipedia.org/wiki/Spiral_Dynamics https://en.wikipedia.org/wiki/Spiral_Dynamics but we are still a mix of orange and green.
- Ekaros 2y agoCan you launder AI model by feeding it to some other model or training process? After all that is how it was originally created. So it cannot be any less legal...
- benreesman 2y agoThere are a family of techniques, often called something like “distillation”. There are also various synthetic training data strategies, it’s a very active area of research. As for the copyright treatment? As far as I know it’s a bit up in the air at the moment. I suspect that the major frontier vendors would mostly contend that training data is fair use but weights are copyrighted. But that’s because they’re bad people.
- qup 2y agoThe weights are my training data. I scraped them from the internet
- benreesman 2y agoThat sentiment is ethically sound and logically robust and directionally consistent with any uniform application of the law as written. But there is a group of people, growing daily in influence, who utterly reject such principles as either worthy or useful. This group of people is defined by the ego necessary to conclude that when the stakes are this high, the decisions should be made by them, that the ends justify the means on arbitrary antisocial behavior (c.f. the behavior of their scrapers) as long as this quasi-religious orgasm of singularity is steered by the firm hand that is willing and able to see it through. That doesn’t distress me: L Ron Hubbard has that. It distresses me that HN as a community refuses to stand up to these people.
- bangaladore 2y agoTo some extent this is how many models are being produced today. Basically its just a synthetic loop of using a previously developed SOTA (was) model like GPT-4 to train your model. This can produce models with seemingly similar performance at a smaller size, but to some extent, less bits will be less good.
- dfefdfdd 2y ago[flagged]
- amolgupta 2y agoFor app developers considering tflite, a safer way would be to host the models on firebase and delete them when their job is done. It comes with other features like versioning for model updates, A/B tests, lower apk size etc. https://firebase.google.com/docs/ml/manage-hosted-models https://firebase.google.com/docs/ml/manage-hosted-models
- hn8726 2y agoThat wouldn't help against the technique explained in the article, would it? Since the model makes it way into the device, it can be intercepted in a similar fashion. I'm not quite sure I understand the firebase feature btw. From the docs, it's pretty much file storage with a dedicated API? I suppose you can use those models for inference in the cloud, but still, the storage API seems redundant.
- biosboiii 2y agoI think the comment author means offering inference via Firebase, with the model never leaving the backend. This works, just like ChatGPT works, but has the downside of 1. You have to pay the computing for every inference 2. Your users can't access it offline 3. Your users will have to use a lot of data from their mobile network operator. 4. Your inference will be slower And since SeeingAI infers the model every second, your and your customers bill will be huge.
- hn8726 2y agoThat's what I thought, but the link doesn't say anything about off-device inference, it's only about storing and retrieving the model. There's just one off-hand note about cloud inference. In any case, yeah you can not download the model to the device at all, but then you have to deal with the other angle - making sure the endpoint isn't abused. Maybe a hybrid approach would work - infer just part of the model (layers?) on the cloud, and then carry on the inference on the device? I'm not familiar with how AI models look like and work like exactly, but I feel like hiding even a tiny portion of the model would make it not usable in practice
- garyfirestorm 2y agoLot of comments here seem to think that there’s no novelty. I disagree. As a new ML engineer I am not very familiar with any reverse engineering techniques and this is a good starting point. Something about ML yet it’s simple enough to follow, and my 17yr old cousin who is ambitious to start cyber security would love this article. Maybe its too advanced for him!
- biosboiii 2y agoThanks a lot :) My general writing style is directed mainly towards my non-technical colleagues, which I wish to inspire to learn about computers. This is no novelty, by far, it is a pretty standard use-case of Frida. But I think many people, even software developers, don't grasp the concept of "what runs on your device is yours, you just dont have it yet". Especially in mobile apps, many devs get sloppy on their mobile APIs because you can't just open the developer tools.
- UnreachableCode 2y agoI'm a mobile developer and I'm new to using Frida and other such tools. Do you have any tips or reading material on how to use things like Frida?
- biosboiii 2y agoI think you are starting off from the perfect direction, being a forward-engineer first, and then a reverse-engineer. The community around Frida is a a) a bit small and b) a bit unorganized/shadowy. You cannot find that many resources, atleast I have not found them. I would suggest you to use Objection, explore an app, enumerate the classes with android hooking list classes or android hooking search classes, then dynamically watch and unwatch them. That is the quickest way to start, when you start developing your own scripts you can always check out code at https://codeshare.frida.re/ https://codeshare.frida.re/. For everything else join the Frida Telegram chat, most knowledge sits there, I am also there feel free to reach out to @altayakkus Oh and btw, I would start with Android, even though iOS is fun too, and I would really really suggest getting a rooted phone/emulator. For the Android Studio Emulator you can use rootAVD (GitHub), just install Magisk Frida. Installing the Frida gadget into APKs is a mess which you wont miss when you go root
- asciii 2y agoThat's pretty cool! I am impressed by the Frida tool, especially to read in the binary and dump it to disk by overwriting the native method. The author only mentions APK for Android, but what about iOS IPA? Is there an alternative method for handling that archive?
- biosboiii 2y agoYeah, you can basically just unzip IPA files. Gaining them is hard though, I have a pathway if you are interested. But the Objective C code is actually compiled, and decompilation is a lot harder than with the JVM languages on Android. My next article will be about CoreML on iOS, doing the same exact thing :)
- asciii 2y ago> My next article will be about CoreML on iOS, doing the same exact thing :) Can't wait - thanks for writing it up!
- 1vuio0pswjnm7 2y ago"Keep in mind that AI models, like most things, are considered intellectual property. Before using or modifying any extracted models, you need the explicit permission of their owner." Is that really true. Is the law settled in this area. Is it the same everywhere or does it vary from jurisdiction to jurisdiction. See, e.g., https://news.ycombinator.com/item?id=42617889 https://news.ycombinator.com/item?id=42617889