13 ms·
Show HN: Apple Notes Liberator – Extract Notes.app Data and Save It as JSON
Hey there!
I just released the first version of a project I’ve been working on solves a very specific problem that perhaps only I have. I welcome any and all feedback, even if you just want to drop in to say that this is a hot piece of garbage!
- frozencell 3y agoThis but for the Mac Library now please (the format transfer are a mess)
- shanecleveland 3y agoI have been adopting Apple Notes more after trying to use a lot of other note methods/apps. Always fall back to simple text in Notes. And I seem to be hearing of more people doing the same. I recently set up a shortcut that works on both my Mac and iPhone to create a new note each day with the date as the title. And then each time I trigger the shortcut that day, it enters the time, location and weather. Easy way to keep a running "diary."
- mdmglr 3y ago"Liberator" implies it was locked down. But notes content has always been easily accessible with AppleScript.
- rhettbull 3y agoVery nice reverse engineering work! I wrote a python library and CLI that does something similar but interacts with Notes via Scripting Bridge: https://github.com/RhetTbull/macnotesapp https://github.com/RhetTbull/macnotesapp
- jamesu 3y agoFor a long while I just copied everything to an imap server and dumped the mailbox, but this looks like an even better solution.
- mkmk 3y agoI consistently struggle with Notes and iMessage sync issues, and I suspect it’s due to being a customer for so many years and making my way through many devices over those years. The UX of the Apple ecosystem is so nice but when it goes sideways you’re kind of out of luck, even if you have applecare. I wish they had some sort of premium “fix my esoteric appleID account issues” service you could pay them for once a decade or so.
- bwbmr 3y agoI agree- long standing issues that have popped up and are never fixed over the years: 1) My iPhone has never properly synced screen time with my other devices (presently my other devices: 2x iPads, MBP) sync fine together. This has persisted across multiple iPhones and major releases of iOS. 2) Apple News has a permanently saved “story” of a specific magazine issue’s table of contents. No idea how it got saved, but since it isn’t actually a story (it’s a ToC instead) the UI option to “unsave” it is greyed out / unavailable. I reported it to a CSR about four years ago, he took down bug reports, I even sent a screen capture demonstrating the bug… it still persists and Apple provides no mechanism to manually clear all saved story data. I can remove it locally from the device from disabling iCloud sync for Apple News -> selecting yes when prompted to remove local data, but it will still persist in iCloud and there is no way to wipe it there… it’s back as soon as I reenable sync.
- amunster 3y agoI had a sync related issue with Messages.app on macOS recently. Escalated through apple support. The solution was to reinstall macOS from scratch. Not pretty but a small price to pay. Now my lingering issue is with search: Calendar events and Reminders prior to the reinstall date do not appear in search results. This has been an issue for years on both macOS and iOS! Much Google searching was no help. I wish that premium fix service included useful forums and…search.
- reaperducer 3y agoI built something similar a while back. But instead of pulling the data out of Notes, I took it from SMTP mail. I don't know if it still works this way, but when I did it, if you used a non-iCloud email address to sync Notes, it saved the data as pain text in email messages that could be easily parsed.
- kello 3y agoWow that is so cool! I had no idea Notes.app worked with IMAP!
- css 3y agoIf anyone ever wanted to do something similar for iMessage data, I built a similar tool for that: https://github.com/ReagentX/imessage-exporter https://github.com/ReagentX/imessage-exporter
- n8henrie 3y agoOh man, excited to see this. Was brainstorming doing something this in rust last week. Beat me to it!
- podviaznikov 3y agoinstead of extracting notes data as JSON my simple tool allows you to liberate your notes by publishing them to the web. https://montaigne.io/ https://montaigne.io/ I use the same amazing library linked in the blog post.
- kello 3y agoOkay this is awesome! 100% cooler than my project!!
- podviaznikov 3y agothank you!
- samwillis 3y agoSuper interesting, particularly the link through to the blog with all the research into how Notes works. A particularly interesting thing that I don't see mentioned anywhere is that Apple Notes uses CRDTs (conflict free replicated data types) to enable both shared notes and multiple concurrent edits. With the CRDTs resolving the conflicts. It looks like the info on the page may be a good place to start looking to how that works: https://www.ciofecaforensics.com/2020/10/20/apple-notes-cloudkit-data/ https://www.ciofecaforensics.com/2020/10/20/apple-notes-clou...
- eddyg 3y agoCRDTs are definitely an “unsung hero” of Apple Notes… and a big reason why it is one Apple’s best “built-in” apps as of late.
- starbugs 3y agoIf they could now implement table resizing and tables that do not render with graphical errors, that would be so nice.
- howinteresting 3y agoA notes app that only works with one 2-trillion-dollar company's hardware and does not have builtin functionality to export notes (e.g. a menu option to do so) cannot reasonably be described as good. One's expectations have to be through the floor for that to be an honest appraisal. Compare it to Obsidian, which just stores data in a bunch of markdown files on disk.
- Kerrick 3y agoI can export to Text and/or PDF on my iPhone from the Notes app.
- howinteresting 3y agoWe're commenting on the website for a project called "Apple Notes Liberator". Obsidian does not need a "liberator".
- Alifatisk 3y agoSo now we know how to export data from the notes.app, messages.app & reminders.app. What’s next?
- CharlesW 3y agoAt https://support.apple.com/en-us/HT204055 https://support.apple.com/en-us/HT204055, Apple provides instructions for a bunch of stuff you might want to export: • iCloud Drive files • Photos and videos • Mail • Contacts • Calendar • Reminders • Safari • Notes • Voice Memos And/or you can use https://privacy.apple.com/ https://privacy.apple.com/, which is basically Apple's version of Google Takeout.
- eviks 3y agoThese are a joke of instructions, per-note app-only export to an awful pdf format
- dangoor 3y agoCool! Thanks for sharing. Also worth noting that there's a Mac app in the app store to export to markdown or html: https://apps.apple.com/us/app/exporter/id1099120373?mt=12 https://apps.apple.com/us/app/exporter/id1099120373?mt=12 (I haven't had a need yet, so I haven't tried it)
- voltaireodactyl 3y agoI have used it several times and it’s fantastic. Exports attachments into properly named media folders and everything.
- oneearedrabbit 3y agoI am excited to see more solutions that work with Apple Notes database directly. This is my go-to application to capture unstructured memos while researching various topics. Last year I got carried away and built a simple site generator for Notes [0]. As a fun exercise I implemented ORM models to query Sqlite database in Ruby: User::Account.server(database: "tmp/test.sqlite").first. folders_dataset.offset(1).first notes. map(&:attachments).flatten. find { |el| el.is_a?(User::ObjectTypes::EmbeddedPublicJpeg) && el.height > 700 && el.summary[/animal/i] }. media_filepath => "Accounts/<uuid>/Media/F669B83A-0C47-408E-9645-015737F3B11F/Pasted Graphic.jpg" Apple Notes' database structure is a fascinating product story. Notes started as a simple note-taking application that stored its content as raw HTML in a few simple tables. However, as the product grew and needed more features like collaboration, they had to make changes and add new columns and tables. They also implemented two flavors of CRDT-like algorithms that evolved over time: one for plaintext chunks and the other one for embedded objects like tables. On top of that, every change that Apple makes must be backward compatible to support legacy devices and systems; but backward-compatibility was not always possible and users had to go through “Migrate your Notes” steps, but it is again a product story. I wish that one day, someone from the development team would write an article about it. [0] https://devlog.notespub.com/2022/08/site-generator-for-apple-notes/ https://devlog.notespub.com/2022/08/site-generator-for-apple...
- sillywalk 3y agoInteresting.. Have you discovered any limitations to Notes e.g. how many total or largest size?
- codetrotter 3y agoThat is awesome! Thank you for sharing your notes on this :)
- wslh 3y agoI would love to have a tool to retrieve the hyperlink of the current note so you can organize the information as a wiki. I know it uses URIs.
- 3y ago
- munk-a 3y agoAs a windows user, while we don't have an OS built-in equivalent, the standard for such tools (in the form of Sublime and the like) is just to operate with temporary buffers that get written out as files on user request. Sublime will write internal data to some persistent storage somewhere and be able to survive sudden crashes and restarts but when you'd like to save the data out to a fixed filepath it's pretty trivial to do so. I do find it quite weird that Apple has managed to cultivate a system where storage in separate files isn't taken as the default.
- simonw 3y agoI wrote a similar tool in Python a few weeks ago, which uses AppleScript to liberate the data and saves it to a SQLite database: https://datasette.io/tools/apple-notes-to-sqlite https://datasette.io/tools/apple-notes-to-sqlite I didn't actually know AppleScript before writing this tool... but it turned out ChatGPT knew it well enough to unblock me and provide me with exactly the code I needed to build the rest of the project! https://til.simonwillison.net/gpt3/chatgpt-applescript https://til.simonwillison.net/gpt3/chatgpt-applescript Apple Notes Liberator creates a copy of the SQLite database and then runs queries against that directly to extract the data. I chose not to do that (despite being all-in on SQLite for everything) because I worry about future changes to the software baking my script - I figured the AppleScript interface was less likely to have breaking changes in the future.
- pjot 3y agoI loved your post about the pelican! https://simonwillison.net/2020/May/21/dogsheep-photos/ https://simonwillison.net/2020/May/21/dogsheep-photos/
- frizlab 3y agoA bit out of context, but did you know you can actually write OSA scripts using JS? In Script Editor there’s a menu where you can change the language.
- tzs 3y agoFrom your article on how you wrote that: > I've been stubbornly refusing to learn AppleScript for nearly twenty years at this point. Thanks to ChatGPT I can continue not to learn it, but I can now use it to solve problems The Apple scripting stuff can also use JavaScript. It might be better to continue not to learn in JavaScript instead of AppleScript, if ChatGPT is as adept at Apple scripting in JavaScript as it seems to be in AppleScript. That way if you want to tweak what ChatGPT gives you it will probably be easier. With AppleScript both interacting with other things and control flow or computation can be weird and confusing. With JavaScript at least the control flow and computation will likely be more normal.
- 3y ago
- kepano 3y agoI am excited to see more ways to export data out of Apple Notes. Long term it would be amazing to see a solution that allows more interoperability with other apps. Depending on what you are looking to achieve there are a few ways to export Apple Notes to Markdown files, but it's only a one-way out solution. From the Obsidian forum: https://forum.obsidian.md/t/import-from-apple-notes-to-obsidian/732 https://forum.obsidian.md/t/import-from-apple-notes-to-obsid...
- pjot 3y agoThis is great! How do you account for updates to a note? I’ve been building an archiver tool as well to convert notes into markdown and post to git using Apple Shortcuts. The interop between almost all iOS data formats (notes, safari, calendar, etc.) the content graph[0] provides is really impressive. [0]: https://support.apple.com/en-me/guide/shortcuts/apd4618db957/ios https://support.apple.com/en-me/guide/shortcuts/apd4618db957...
- ChintanGhate 3y agoI have published an app [Exporter](https://apps.apple.com/us/app/exporter/id1099120373?mt=12 https://apps.apple.com/us/app/exporter/id1099120373?mt=12) that does something similar - exports Apple Notes to HTML and markdown with the attachments. Give it a try, let me know if it solves your issue. Also, why did you select JSON as the output format?
- daydream 3y agoThanks for making your app. I’ve used it more than once and it works well. Many export solutions don’t handle attachments but yours does. So kudos for that. A few years ago I tried to write my own exporter using AppleScript but got stuck by a showstopping bug in Notes AppleScript implementation. Or so it seemed. I hope that bug is fixed now. I just moved on to use your app instead.
- ChintanGhate 3y agoI am glad it's being useful. :) I used AppleScript for the first two years of this app as well (Can't believe it's been 7 years since I started it). But it's hard to handle complex programs, and handing off data to Swift or Objective-C from AppleScript. So I switched to [ScriptingBridge](https://developer.apple.com/documentation/scriptingbridge https://developer.apple.com/documentation/scriptingbridge). It provides better Objective-C APIs to handle Apple Events.
- mark_l_watson 3y agoWorked well for me, thanks.
- kello 3y agoThis is so cool and way cooler than my tool!! I chose JSON because it was the easiest option. And also because it is the format I wanted to support first, but mostly because it was the easiest.
- deleted 3y ago[deleted]
- vszakats 3y agoHere's a Python script that takes the JSON output from https://github.com/threeplanetssoftware/apple_cloud_notes_parser https://github.com/threeplanetssoftware/apple_cloud_notes_pa... (needs master), and rebuilds Notes in Markdown format (or HTML), with links, attachments, lists, etc.: https://gist.github.com/vszakats/5a3bd939721d1dde6142d9ea3b2d1b5f https://gist.github.com/vszakats/5a3bd939721d1dde6142d9ea3b2... And a short, standalone JXA script, that outputs HTML, but loses some data, such as links: https://gist.github.com/vszakats/f24e7700428c1e694e20cee0b1cc542b https://gist.github.com/vszakats/f24e7700428c1e694e20cee0b1c...
- rco8786 3y agoOh neat. I’ve been using Notes extensively since the beginning of this year. Daily journaling. Personal TODO lists, grocery list (shared w my wife), and gym/workout tracking. It works really well and has way more features than people realize. If you’re looking for like a “Notion-lite”, the answer might already be on your phone/MacBook.
- dmd 3y agoI don't understand what value this tool adds over the library it uses under the hood (which includes a command line tool for exporting your notes; I've been using it for years). What is it for?
- kello 3y agoLiterally 0 added value here over the other library. In fact, my tool is a net value subtraction from the other library. The other library has way more features and works great! This was mostly a project for my own learning of how Apple Notes stores its data. I honestly didn't expect it do get any attention!
- divbzero 3y agoNote that Notes.app can also sync with IMAP. IMAP syncing supports basic formatting (bold, italic, underline) but not all features supported by iCloud syncing (attachments, checklists, photos).
- ecliptik 3y agoA feature that a lot of Apple apps have that doesn't seem talked about much is they can use standard protocols and not require iCloud. Mail.app/Reminders.app/Notes.app can use IMAP, Cal.app with CalDAV, and Contacts.app with CardDAV [1]. It doesn't offer a lot of the features iCloud has like advanced formatting or collaboration, but is a useful for simple standards based syncing and portability. I've done this with my iPhone/Macs and Fastmail for the last few years and it works great [2]. 1. https://support.apple.com/guide/iphone/set-up-mail-contacts-and-calendar-accounts-ipha0d932e96/ios https://support.apple.com/guide/iphone/set-up-mail-contacts-... 2. https://www.fastmail.help/hc/en-us/articles/1500000279941 https://www.fastmail.help/hc/en-us/articles/1500000279941
- IggleSniggle 3y agoI…can’t believe I didn’t know this about Notes or Reminders. Thank you for mentioning this.
- kello 3y agoWow! this is so cool! I had no idea about this!
- polyrand 3y agoYes, the IMAP sync protocol was one of my discoveries when trying to turn Apple Notes into a CMS[0]. But like you said, many features are missing. I could do without the collaboration features, but I found that tables didn't work either. A lot of the unsupported features just disappeared from the IMAP-synced note, so you can't recover them later if you move the note back to an iCloud account. [0]: https://ricardoanderegg.com/posts/apple-notes-blog-cms/ https://ricardoanderegg.com/posts/apple-notes-blog-cms/
- donatj 3y agoHmm… I wonder how hard it would be to throw together a little IMAP server just for personal note synchronization. Is note handling a standard part of the protocol or an extension?
- solalf 3y agoAwesome initiative! I wish Apple Notes would support Markdown. This is my go to app for so much of my life, à Notion style editor would be an absolute game changer.
- rhettbull 3y agoI'm the author of python library & command line tool, https://github.com/RhetTbull/macnotesapp https://github.com/RhetTbull/macnotesapp, that will allow you to use Markdown to create (but not yet edit) notes from the Terminal. I use Vim + markdown to create notes all the time: `notes add -m -e` adds a new note using Markdown by opening the default editor.
- yosito 3y agoIt would be really cool to set this up to automatically sync notes to Obsidian.
- akerr 3y agoWhy not just use the built-in Shortcuts to export to your preferred format?
- kello 3y agoYou should absolutely just do this. 100% the better option. I wanted to learn more about how Apple Notes stores its data and see how I could interact with it.
- egeozcan 3y agoI wish someone did this for Whatsapp without having to jailbreak my phone.
- marcjensen 3y agoThanks for creating this. I must say that I don't find things written in Java on HN very often.
- gudh 3y agoI wish there's tool to extract Apple Books' notes too
- adrianhon 3y agoReadwise (https://readwise.io https://readwise.io) has a browser extension + website that will do this, a bit finicky but it gets the job done!
- joe8756438 3y agoThis is cool. I wanted to provide notes export for people using a note taking service I built and landed on a ios shortcut. I think you can accomplish something close or the same? using shortcuts. One obvious benefit is the interface is going to be more approachable than the jar to most people. Shortcuts are also cool because you can build one and package it for others and even define user specific parameters that get filled on install.
- kolinko 3y agoNice! It throws an error that it cannot find the notes folder though. Where is it located usually? Or what's the file name?
- ericzawo 3y agoLove this. I am an Apple Notes absolutist (see: https://pbs.twimg.com/media/FY1rZvNWIAAIRzs.jpg https://pbs.twimg.com/media/FY1rZvNWIAAIRzs.jpg). I have tried Evernote, Workflowy, Dynalist, Notion over the years for marrying personal / professional worlds but always (always!!) come back to Notes. I have dreamed of the ability to have a Notes web app with custom CSS for years and a few other features. Good luck with this project I'll be watching closely
- docmars 3y agoI'm curious — as someone who's had a hard time with Notes' maybe... lack of features and less-than-friendly formatting IMHO (no Markdown, unconventional keyboard shortcuts, layout formatting quirks...) — what about Apple Notes appeals to you more than other popular tools? I feel I may be missing something :)
- swalling 3y agoJust works: fast to load, works offline, I don’t have to think about sync Just notes: It’s not trying to be a programmable networked knowledge engine or a no code app builder or a toaster. Not ugly: You can technically change fonts and crap but it uses clean system styles out of the box, so even as a font snob I don’t get distracted changing formatting instead of writing. Free, or at least really cheap if you have a lot of stuff in iCloud
- docmars 3y agoFair points! Thanks for sharing that :)
- cyrialize 3y agoI've run through a similar thing in the image shown. As someone who recently picked up an MBP I've been loving Apple Notes.
- ithrow 3y agoAnki doesn't belong there though.