7 ms·

My Knowledge Lakehouse

212 points by tabokie 3y ago

fireynis 3y ago
I like the idea, and I do something similar but this has a lot of rules and feels more complex than potentially necessary. I personally use Obsidian.md as the tool for my Zettelkasten method. It provides back links and the like. I also create engineering journals for projects. The only rule I have is to avoid unlinked notes.
dinkleberg 3y ago
There are some good ideas in here. Personally I’ve abandoned all of my complex attempts at productivity and knowledge systems and have recently moved to the ever reliable txt file. You can format it however you want, use fun ascii art, or just plain paragraphs. I’ve got a daily.txt where I prepend a new days section every day and put in whatever I want over the course of the day. I also create other files for courses I’m going through or projects as needed. The nice thing is using my editor I can make a file reference and hit gf and it’ll go right to a referenced file. But this “system” will keep on working regardless of what tools or system I happen to be using.
- ahmedfromtunis 3y ago
  It's clearly a nice and simple way to write data, but what about the other way around? How do you manage to find information when you need it?
  - dinkleberg 3y ago
    Read data? lol it is usually just a write only system (aside reviewing the current info from the week). But since it is all in one place I just do search queries using my editor to find what I need. I don’t use tags, but it would be very simple to just append some #tags in my content and search for those.
  - gkbrk 3y ago
    CTRL-F or grep seem like they would work well
  - wharvle 3y ago
    I use Apple Notes and don’t bother to organize anything unless it’s a whole bunch of related material that I’m going to use together—notes for an rpg session, say. All the rest, I just make sure that I give it a title that I’ll recognize when I see it, and include words that I’ll likely use if I go looking for it with search. Been doing this for years, haven’t lost anything yet. It works so well that time organizing it would have been wasted.
    - falseAss 3y ago
      i've been doing the exactly same thing. Some risk is the data safety - years of log is attached to the icloud account is concerning.
      - wharvle 3y ago
        There exist programs to export them to markdown or what have you. Dunno how well they handle embedded media. I do a lot of copy-pasting screenshots or embedding PDF pages… or entire pdfs. [edit] “why screenshots?” 1) To record gui workflows, walkthrough-style. 2) to record whole screens of values from guis while preserving formatting perfectly (think: cloud dashboard vital stats screens for various resources) 3) to record short message exchanges from ephemeral messaging with all the formatting intact with zero extra effort. (Think: feature discussion in a periodically-cleaned chat channel; I can always turn it into text later if I need to, recording with screenshot is fast) 4) plus now that it’s almost as easy and reliable to copy-paste from images as from regular text, on macOS and iOS, why not?
    - tomcam 3y ago
      Having trouble with Apple notes syncing too slowly. Wife can update the shopping list but I won’t see it for 24+ hours. Unfortunately my life runs on it. Problem may be that my Notes file has become a 15G monstrosity?
      - tensor 3y ago
        I use reminders for shopping lists. They can also be shared. Hopefully they are not the same database as notes!
  - BeetleB 3y ago
    Although I extensively use org mode, I didn't follow the GP's approach precisely because of this searchability problem. These days, however, vector DBs and doing RAG is getting close to changing my mind. I want to do less and less organizing, and more dumping, and use vector search capabilities to find what I need. Haven't set it up yet, but it's on my TODO list.
ukuina 3y ago
> 1.1221 Sometimes the primary tool is not available. An “always-on” secondary should take its place. > 1.1222 Sometimes log from different tools or locations need to be “merged”, “persisted”. These two are the main reasons I've stuck with Google Docs: It's available everywhere, and everything's always in-sync. Google already has all my info of value, so the incremental trust necessary to the threshold of log visibility is minimal. I'll add one more requirement: I don't want to leave an unencrypted on-disk footprint containing my notes. This means I can access the entirety of my notes on any machine and the only thing I have to worry about is keylogging/screenscraping. Google Doc is lacking in many respects, though. Linkability is nonexistent. Docs longer than 100 pages really struggle with latency on mobile. Searchability is bad (!) because you have to open the doc first to see the matches from within. Really wish there was a self-hosted alternative with sync and encrypted storage that didn't result in sync errors. I've tried DEVONthink, Obsidian, LogSeq, Google Keep, Notion, NotesNook, GoodNotes, Samsung Notes, Loop, OneNote, Apple Notes, Org Mode, plain text files, and probably a dozen others... I'd say NotesNook is the best so far, with DEVONthink a close second, but, nothing beats the reliability, omnipresence and privacy of Google Docs.
- zilti 3y ago
  Syncthing + org-mode is the winning combo for me
  - RMPR 3y ago
    With org-roam and roam-ui you have a perfect visualization. It's a bit lacking on mobile, been trying logseq on and off but it's so buggy.
- sebtron 3y ago
  > It's available everywhere Unless you are offline
  - deleted 3y ago
    [deleted]
  - diggan 3y ago
_boffin_ 3y ago
Thanks for posting this. Currently building out something that is consolidating events from numerous sources and then automatically processing of those. This post helped to give me some additional ideas on the processing of those events.
- threecheese 3y ago
  Interesting, and something I’ve been thinking about. What is your architecture? Using any available workflow products for “retrieve/receive event from X, figure out what do, then do that in Y”?
  - _boffin_ 3y ago
    Right now, since it’s currently being built out for a single user, I’m just using NestJS for the backend, TypeScript and Vite for the browser extension, Python for the daemon, Postgres for the DB and ELK for logging and embeddings storage. I have a few sources built out right now and some fun utilities thanks to the browser extension, but in the end, I’m expecting to have 15-20 different data sources that are sent by the browser extension, collected by the daemon, and polled via crons in NestJS. I’m taking ideas i gleaned from the AI-Town paper about taking “memories” and then generating “reflections” and then doing some other stuff on top of that and also building out a knowledge graph of everything i do, read, listen to, etc.. I’m currently figuring out the standardization of the disparate events into memories to then transform into reflections and knowledge graphs. I’d love to hear more about what you’ve been thinking of and the issues you’ve had. Edit: One cool thing you can do with the browser extension for this stuff is to send your cookies and headers to the database and then use in the cron jobs to get the data—think pandora.
csbartus 3y ago
Very cool ideas! Very simple concept, indeed! It helped to pin down my own approach, which is a very similar workflow, with these differences: - The log entries are flashcards with ID - The structure / tree follows the scientific method, aka the glue is not ad-hoc. - So far I need no lists, because I only learn things and I'm not tracking any activity
simonw 3y ago
This year I'm trying a new mechanism based around GitHub Issues (which I've been using for personal knowledge management for a few years already) - I create a new "planner" issue every day, then use that to make notes about what I want to do and what I've got done. Hitting Command+N in Firefox on my Mac opens up my planner issue for the day, or creates it if one doesn't exist yet. I wrote up how that system works (a tiny bit of GitHub Actions + Pages magic) here: https://til.simonwillison.net/github-actions/daily-planner https://til.simonwillison.net/github-actions/daily-planner
civilized 3y ago
This post probably has some kernels of value, but it suffers from the XY problem: it's a note taking system which provides the solution, but not the problem it's solving. Let's invert that and focus on the problem first. Below are the problems my personal system solves, followed by the solutions. My solutions may not fit your situation, but if you see yourself in the problems, maybe you'll be inspired to find the solution that works for you. 1. Formatting easily becomes a distraction, but a little formatting is vital, so I use markdown to write my plans and notes. 2. Mixing streams from different projects is confusing, so each project has its own workspace - either a folder or a distinct prefix on the file name. 3. Work or ideas often become irrelevant for a while as plans change, but I hate feeling like I might lose work that might be valuable later. So I have a "dump" file or folder where I can dump such things. Entries are typically dated so they can be referenced by other files (see below). I typically use a level 1 or 2 markdown section header to delimit entries. 4. I often have a "log" markdown file where I append descriptions of indisputably important project developments (as opposed to maybe-important-later stuff that goes in the dump file). Entries here are dated and delimited similarly to the dump file. 5. There is a "plan" file that I try to keep clean and concise. This is where I go to remember what I have resolved to do next. 6. A common issue with plan files is that they get bloated with discussion of current or past context, as you try to figure out what to do and how to do it. When this happens, the discussion is moved to the "log" or "dump" files and replaced with a "link" describing how to find it again (e.g. "see log date 2024-01-12 for more on this"). 7. There is a master plan file that coordinates all the projects. Bloat in this file is moved down to individual project files and replaced with a link. To recap: markdown; project workspaces; log and dump with dated entries; concise plan file; move bloat out of plan into log or dump and replace with a link; master plan file with similar bloat management strategy. I have used this system for five years. It has developed over time, but all elements have been in use for at least two years. I am a staff data scientist leading a team of a half dozen people that is overhauling the core analytics pipelines and models at a Fortune 100, and 80% of my day is IC work. I have this job not because anyone asked me to do it, but because I proved I could do it on my own time. I work normal hours, parent of two kids, working spouse. I don't have time to putter around. This system is a big part of how I make it work.
sureglymop 3y ago
I just create one markdown file per day. I wrote a custom obsidian plugin that displays the "timeline" as if it was one open file and creates the files in year/month/week subfolders. But I don't use linking and all the other complexity. But if I ever needed it I could. Then it's synced with next cloud and also available on my phone. I like this the most because just as explained, the time line is the most accurate representation of reality. And I can very quickly scroll back to see what I did last week or do full text search. Simple and useful!
- philips 3y ago
  Can you explain the timeline further? I don’t understand- just a list of links?
  - sureglymop 3y ago
    The idea is to create one markdown file per day but to be able to open them all in one editor view. Then one can scroll through all the days conveniently like a timeline. Something like this [0] plugin although I did create my own (more hacky) version for myself. [0]: https://github.com/Quorafind/Obsidian-Daily-Notes-Editor https://github.com/Quorafind/Obsidian-Daily-Notes-Editor
- canadiantim 3y ago
  Why next cloud for sync?
  - sureglymop 3y ago
    Because I already use it for everything else such as contacts (CardDAV), calendar events and tasks (CalDAV) which syncs nicely with my android phone. It's not that hard to host if not many people use the instance, has always worked well for me.
- toddmorey 3y ago
  I love the approach. Have you published the "timeline" plugin or is code available?
hnthrowaway0328 3y ago
I had a revelation that my ideas are worthless and the notes I took are also going to be worthless once I get familiar with X. So I completely stopped taking notes. A to-do list plus a Google Drive for my son and our stuffs are good enough. The only problem is that we are not comfortable to put sensitive information on GD so probably build a NAS for that.
- Zetobal 3y ago
  Can work in a fast paced environment but if you work on projects or ideas with hard numbers for years its not applicable or your brain capacity is off the charts.
  - hnthrowaway0328 3y ago
    Yeah 100%, I forgot to mention the above is just for my private projects so a lot of notes went into as comments. For work we do use Confluence but again the quality of our notes are not great. From my experience the most important notes are the ones that tag a PoC if issue X flares up.
- toddmorey 3y ago
  My experience has been the opposite. I used to envy people with amazing memories like it was a super power. Then I realized if I got into the habit of just writing stuff down, I'd effectively have that same superpower. It's the one thing that I can honestly say has changed my life. I use obsidian.md for notes, which I have on phone and desktop. Nothing fancy. You just need something that's always with you, has great search, and ideally allows you to link notes together.
  - hnthrowaway0328 3y ago
    I don't really have amazing memories. I just found out most of my stuffs are worthless and do not worth being preserved.
  - balder1991 3y ago
    It’s not only about keeping memories but structuring them. Even journaling (about your thoughts and feelings) helps with making sense of what’s on your mind: https://youtu.be/FNJO1pZV-I8?si=Efqm5uAOSp9MSeOq https://youtu.be/FNJO1pZV-I8?si=Efqm5uAOSp9MSeOq
cfr2023 3y ago
An interesting read and a good share. Every time I see that other people are contemplating these subjects, I notice that they hit on many of my own considerations and miss many of the big picture things I consider essential. I think this will evolve into tribalism of mental models for organizing information. More vi vs emacs energy to go around in the decades to come.
riffic 3y ago
this is effectively a very geeked out zettelkasten method. https://en.wikipedia.org/wiki/Zettelkasten https://en.wikipedia.org/wiki/Zettelkasten
moonlion_eth 3y ago
Anytype all day every day
- deleted 3y ago
  [deleted]
- ssivark 3y ago
  I was very excited about this when it launched (in alpha). How has the product matured? Also, does it now have an API for easy import/export of data into the database?
openquery 3y ago
This is the second piece of content I've ever seen using the same numbering system as Wittgenstein's Tractatus, the first of course being the Tractatus itself.
- throwup238 3y ago
  Can't wait for the followup, Knowledgebase Investigations in which he realizes that he has derived no use from his knowledgebase. Water!
dmje 3y ago
Obsidian. Total game changer. Obsidian sync is completely worth paying for as well - firstly it helps support the project but also its slick, so well made, and easier than all the hacked together GitHub / s3 / whatever workarounds. From a content point of view I personally find having structure but not too much structure is the way to go. A bit of yaml if needed but just a gentle “daily notes = meeting notes” (on my work obsidian) and “daily notes = journal” (on my personal one), a few tags but nothing too rigid or cumbersome. Omnisearch is the dogs nuts plugin too, and a couple of others but again, keeping it quite minimal has really worked for me. Just so nice knowing that everything is under my control, using just markdown but with this fantastically powerful framework underneath it. Huge fan.
- bravura 3y ago
  I haven't checked Obsidian in a while, can you explain why you love it so much? I hear it being discussed a lot.
  - dmje 3y ago
    I think it’s the balance of being “just markdown” on one hand - you can literally just point it at a folder and start typing; and the fact that it’s incredibly extendable on the other. The better known plugins like DataView give you really powerful list / sort / search if you want that, but you can also make it look however you want. I’m strangely snobby about my UIs, so being able to take a theme that’s almost right and then tweak the css is a big deal for me. Finally - it’s just a fast and solid bit of engineering. Rarely / (never, actually?) crashes, shortcuts are really powerful, constantly being updated… all manner of great :-)
- rtpg 3y ago
  I really wish that Obsidian Sync had like... 5 megs for free or something. I can't justify paying $8/month for a thing I haven't even started using, and without sync I have a hard time saying "OK let's try this" compared to Joplin or Notion. There's a bunch of cool looking stuff though
  - 3y ago
animal_spirits 3y ago
Anytime a knowledge management system comes into my head I remember this joke: https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F2li4r1riaoe91.jpg https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F2...
- rsync 3y ago
  Well, if you s/apple\ notes/vi/g ...
cobertos 3y ago
Isn't this what Hypercore[0] is? But distributed? It's a append-only log, and then there's higher order abstractions built on top of this log. Hyperstore is a key value store that uses an append only log, hyperdrive is a filesystem, etc... [0]: https://github.com/holepunchto/hypercore https://github.com/holepunchto/hypercore
shrubble 3y ago
I have recently started to play around with the lightweight Emacs mode, "HOWM" which was written about 20 years ago by a Japanese author, who still maintains it today. The howm-mode shows you a summary view, comprised of information (just the title/heading) from the (configurable) last 20 notes you have made. Everything goes under ~/howm/YYYY/MM e.g. ~/howm/2024/01 for notes I am making this month. It has a simple way to enter schedule, todo, deadline, etc. which it shows just below the header that has single-key commands (s for case-insensitive searching, c to create a new note, etc.) for searching, navigation, new notes. The genius of howm is that you can write and have very simple back and forward links to any file, URL, or tag, any kind of text etc., and you write "fragments" that is, whatever the smallest unit of text you want - it just needs a title plus anything else you want to add to it. When you save the file, you hit Ctrl-C plus ",," and are back in the summary view. So you can have very small sized units/fragments, which are very fast to create and save, but due to the forward and back links and the ease of searching (which will collate in a temporary list, everything found), you can "create fragmentarily but view/search collectively" as the documentation puts it. My next step is to author HOWM notes in AsciiDoc or Markdown format which can easily be fed (since each note is a separate text file) into a converter for fancier output. Syncthing easily keeps things in sync between my desktop and laptop. In the very best of FOSS tradition, someone else wrote up a full PDF manual, in English. https://github.com/Emacs101/howm-manual/blob/main/Howm_tutorial_eng.pdf https://github.com/Emacs101/howm-manual/blob/main/Howm_tutor...
accrual 3y ago
Several comments about personal note taking systems. Here's mine. :) function note { mg +-1 "/home/user/notes/$(date "+%Y-%m-%d").txt"; } "mg" is "Micro Emacs clone" and "+-1" opens the file at the bottom for appending, but of course it works equally fine with your choice of $EDITOR. I like this setup because it's plaintext (or markdown if you prefer), easily sorts on filename, and easily greppable. I have a couple years of notes saved this way and it's fun to explore old notes.
devdiary 3y ago
I use the similar kind of structure in Developer Diary. I do it completely offline, no sync.
neighbour 3y ago
Whenever I try one of these apps I'm always impressed by the features but my entire life is in iCloud. Not only that but all the people I connect with for notes/calendars/reminders are also in the Apple ecosystem and they aren't going to switch for me. This means that a lot of the extra features that Notion, Obsidian, etc. offer just aren't valuable for me. It makes me annoyed that Apple's Notes app is so basic.
- pif_ 3y ago
  Maybe NotePlan[0] can sync with iCloud? [0]: https://noteplan.co/ https://noteplan.co/
- faizmokh 3y ago
  You can use iCloud to sync your Obsidian notes across your devices. I find the app experience to be terrible tho.
rickcarlino 3y ago
I hope more people share content like this in HN. Note taking is incredibly personal and as a result, I think the best ideas might be lying dormant and unshared.
fouc 3y ago
I thought Knowledge Lakehouse was going to be a variation of a Memory Palace
up2isomorphism 3y ago
What do all these have to do with lakehouse? (and that is a lame term to begin with anyway)
LAC-Tech 3y ago
I enjoyed this article, though not sure I can be bothered printing books. The approach taken here is essentially event sourcing, btw. It's old and it works.
IOT_Apprentice 3y ago
I’m using capacities.io it is much more accessible than obsidian.
wt__ 3y ago
My approximate journey (7,500 notes, 13% markdown, remainder .txt) (pc, circa 2010) notepad > Evernote > notepad > (mac) > vim > Notational Velocity > NVUltra (still in) beta > "The Archive" (Zettelkasten) > (n)vim - notes synced with Dropbox folder (could just as well use iCloud) - with vim fzf/telescope plugins for search What finally flipped me back to vim was using a setup script to give me a useful set of plugins that did nice markdown highlighting out of the box, and then working out how to do things like auto-indenting, opening URLs in browser, renaming files more easily etc. Slight annoyance: yet to find a way to search both filenames and file contents at the same time Do systems that link from one note to another help me? Apparently not much. Quite a lot of effort to maintain the links, better to express hierarchy in filenames. (still using Omnifocus for todo lists, but it can import/export Taskpaper text format)

212 points by tabokie 3y ago

I like the idea, and I do something similar but this has a lot of rules and feels more complex than potentially necessary. I personally use Obsidian.md as the tool for my Zettelkasten method. It provides back links and the like. I also create engineering journals for projects. The only rule I have is to avoid unlinked notes.

dinkleberg 3y ago

There are some good ideas in here. Personally I’ve abandoned all of my complex attempts at productivity and knowledge systems and have recently moved to the ever reliable txt file. You can format it however you want, use fun ascii art, or just plain paragraphs. I’ve got a daily.txt where I prepend a new days section every day and put in whatever I want over the course of the day. I also create other files for courses I’m going through or projects as needed. The nice thing is using my editor I can make a file reference and hit gf and it’ll go right to a referenced file. But this “system” will keep on working regardless of what tools or system I happen to be using.

ahmedfromtunis 3y ago
It's clearly a nice and simple way to write data, but what about the other way around? How do you manage to find information when you need it?
- dinkleberg 3y ago
  Read data? lol it is usually just a write only system (aside reviewing the current info from the week). But since it is all in one place I just do search queries using my editor to find what I need. I don’t use tags, but it would be very simple to just append some #tags in my content and search for those.
- gkbrk 3y ago
  CTRL-F or grep seem like they would work well
- wharvle 3y ago
  I use Apple Notes and don’t bother to organize anything unless it’s a whole bunch of related material that I’m going to use together—notes for an rpg session, say. All the rest, I just make sure that I give it a title that I’ll recognize when I see it, and include words that I’ll likely use if I go looking for it with search. Been doing this for years, haven’t lost anything yet. It works so well that time organizing it would have been wasted.
  - falseAss 3y ago
    i've been doing the exactly same thing. Some risk is the data safety - years of log is attached to the icloud account is concerning.
    - wharvle 3y ago
      There exist programs to export them to markdown or what have you. Dunno how well they handle embedded media. I do a lot of copy-pasting screenshots or embedding PDF pages… or entire pdfs. [edit] “why screenshots?” 1) To record gui workflows, walkthrough-style. 2) to record whole screens of values from guis while preserving formatting perfectly (think: cloud dashboard vital stats screens for various resources) 3) to record short message exchanges from ephemeral messaging with all the formatting intact with zero extra effort. (Think: feature discussion in a periodically-cleaned chat channel; I can always turn it into text later if I need to, recording with screenshot is fast) 4) plus now that it’s almost as easy and reliable to copy-paste from images as from regular text, on macOS and iOS, why not?
  - tomcam 3y ago
    Having trouble with Apple notes syncing too slowly. Wife can update the shopping list but I won’t see it for 24+ hours. Unfortunately my life runs on it. Problem may be that my Notes file has become a 15G monstrosity?
    - tensor 3y ago
      I use reminders for shopping lists. They can also be shared. Hopefully they are not the same database as notes!
- BeetleB 3y ago
  Although I extensively use org mode, I didn't follow the GP's approach precisely because of this searchability problem. These days, however, vector DBs and doing RAG is getting close to changing my mind. I want to do less and less organizing, and more dumping, and use vector search capabilities to find what I need. Haven't set it up yet, but it's on my TODO list.

ukuina 3y ago

> 1.1221 Sometimes the primary tool is not available. An “always-on” secondary should take its place. > 1.1222 Sometimes log from different tools or locations need to be “merged”, “persisted”. These two are the main reasons I've stuck with Google Docs: It's available everywhere, and everything's always in-sync. Google already has all my info of value, so the incremental trust necessary to the threshold of log visibility is minimal. I'll add one more requirement: I don't want to leave an unencrypted on-disk footprint containing my notes. This means I can access the entirety of my notes on any machine and the only thing I have to worry about is keylogging/screenscraping. Google Doc is lacking in many respects, though. Linkability is nonexistent. Docs longer than 100 pages really struggle with latency on mobile. Searchability is bad (!) because you have to open the doc first to see the matches from within. Really wish there was a self-hosted alternative with sync and encrypted storage that didn't result in sync errors. I've tried DEVONthink, Obsidian, LogSeq, Google Keep, Notion, NotesNook, GoodNotes, Samsung Notes, Loop, OneNote, Apple Notes, Org Mode, plain text files, and probably a dozen others... I'd say NotesNook is the best so far, with DEVONthink a close second, but, nothing beats the reliability, omnipresence and privacy of Google Docs.

zilti 3y ago
Syncthing + org-mode is the winning combo for me
- RMPR 3y ago
  With org-roam and roam-ui you have a perfect visualization. It's a bit lacking on mobile, been trying logseq on and off but it's so buggy.
sebtron 3y ago
> It's available everywhere Unless you are offline
- deleted 3y ago
  [deleted]
- diggan 3y ago

_boffin_ 3y ago

Thanks for posting this. Currently building out something that is consolidating events from numerous sources and then automatically processing of those. This post helped to give me some additional ideas on the processing of those events.

threecheese 3y ago
Interesting, and something I’ve been thinking about. What is your architecture? Using any available workflow products for “retrieve/receive event from X, figure out what do, then do that in Y”?
- _boffin_ 3y ago
  Right now, since it’s currently being built out for a single user, I’m just using NestJS for the backend, TypeScript and Vite for the browser extension, Python for the daemon, Postgres for the DB and ELK for logging and embeddings storage. I have a few sources built out right now and some fun utilities thanks to the browser extension, but in the end, I’m expecting to have 15-20 different data sources that are sent by the browser extension, collected by the daemon, and polled via crons in NestJS. I’m taking ideas i gleaned from the AI-Town paper about taking “memories” and then generating “reflections” and then doing some other stuff on top of that and also building out a knowledge graph of everything i do, read, listen to, etc.. I’m currently figuring out the standardization of the disparate events into memories to then transform into reflections and knowledge graphs. I’d love to hear more about what you’ve been thinking of and the issues you’ve had. Edit: One cool thing you can do with the browser extension for this stuff is to send your cookies and headers to the database and then use in the cron jobs to get the data—think pandora.

csbartus 3y ago

Very cool ideas! Very simple concept, indeed! It helped to pin down my own approach, which is a very similar workflow, with these differences: - The log entries are flashcards with ID - The structure / tree follows the scientific method, aka the glue is not ad-hoc. - So far I need no lists, because I only learn things and I'm not tracking any activity

simonw 3y ago

This year I'm trying a new mechanism based around GitHub Issues (which I've been using for personal knowledge management for a few years already) - I create a new "planner" issue every day, then use that to make notes about what I want to do and what I've got done. Hitting Command+N in Firefox on my Mac opens up my planner issue for the day, or creates it if one doesn't exist yet. I wrote up how that system works (a tiny bit of GitHub Actions + Pages magic) here: https://til.simonwillison.net/github-actions/daily-planner https://til.simonwillison.net/github-actions/daily-planner

civilized 3y ago

This post probably has some kernels of value, but it suffers from the XY problem: it's a note taking system which provides the solution, but not the problem it's solving. Let's invert that and focus on the problem first. Below are the problems my personal system solves, followed by the solutions. My solutions may not fit your situation, but if you see yourself in the problems, maybe you'll be inspired to find the solution that works for you. 1. Formatting easily becomes a distraction, but a little formatting is vital, so I use markdown to write my plans and notes. 2. Mixing streams from different projects is confusing, so each project has its own workspace - either a folder or a distinct prefix on the file name. 3. Work or ideas often become irrelevant for a while as plans change, but I hate feeling like I might lose work that might be valuable later. So I have a "dump" file or folder where I can dump such things. Entries are typically dated so they can be referenced by other files (see below). I typically use a level 1 or 2 markdown section header to delimit entries. 4. I often have a "log" markdown file where I append descriptions of indisputably important project developments (as opposed to maybe-important-later stuff that goes in the dump file). Entries here are dated and delimited similarly to the dump file. 5. There is a "plan" file that I try to keep clean and concise. This is where I go to remember what I have resolved to do next. 6. A common issue with plan files is that they get bloated with discussion of current or past context, as you try to figure out what to do and how to do it. When this happens, the discussion is moved to the "log" or "dump" files and replaced with a "link" describing how to find it again (e.g. "see log date 2024-01-12 for more on this"). 7. There is a master plan file that coordinates all the projects. Bloat in this file is moved down to individual project files and replaced with a link. To recap: markdown; project workspaces; log and dump with dated entries; concise plan file; move bloat out of plan into log or dump and replace with a link; master plan file with similar bloat management strategy. I have used this system for five years. It has developed over time, but all elements have been in use for at least two years. I am a staff data scientist leading a team of a half dozen people that is overhauling the core analytics pipelines and models at a Fortune 100, and 80% of my day is IC work. I have this job not because anyone asked me to do it, but because I proved I could do it on my own time. I work normal hours, parent of two kids, working spouse. I don't have time to putter around. This system is a big part of how I make it work.

sureglymop 3y ago

I just create one markdown file per day. I wrote a custom obsidian plugin that displays the "timeline" as if it was one open file and creates the files in year/month/week subfolders. But I don't use linking and all the other complexity. But if I ever needed it I could. Then it's synced with next cloud and also available on my phone. I like this the most because just as explained, the time line is the most accurate representation of reality. And I can very quickly scroll back to see what I did last week or do full text search. Simple and useful!

philips 3y ago
Can you explain the timeline further? I don’t understand- just a list of links?
- sureglymop 3y ago
  The idea is to create one markdown file per day but to be able to open them all in one editor view. Then one can scroll through all the days conveniently like a timeline. Something like this [0] plugin although I did create my own (more hacky) version for myself. [0]: https://github.com/Quorafind/Obsidian-Daily-Notes-Editor https://github.com/Quorafind/Obsidian-Daily-Notes-Editor
canadiantim 3y ago
Why next cloud for sync?
- sureglymop 3y ago
  Because I already use it for everything else such as contacts (CardDAV), calendar events and tasks (CalDAV) which syncs nicely with my android phone. It's not that hard to host if not many people use the instance, has always worked well for me.
toddmorey 3y ago
I love the approach. Have you published the "timeline" plugin or is code available?

hnthrowaway0328 3y ago

I had a revelation that my ideas are worthless and the notes I took are also going to be worthless once I get familiar with X. So I completely stopped taking notes. A to-do list plus a Google Drive for my son and our stuffs are good enough. The only problem is that we are not comfortable to put sensitive information on GD so probably build a NAS for that.

Zetobal 3y ago
Can work in a fast paced environment but if you work on projects or ideas with hard numbers for years its not applicable or your brain capacity is off the charts.
- hnthrowaway0328 3y ago
  Yeah 100%, I forgot to mention the above is just for my private projects so a lot of notes went into as comments. For work we do use Confluence but again the quality of our notes are not great. From my experience the most important notes are the ones that tag a PoC if issue X flares up.
toddmorey 3y ago
My experience has been the opposite. I used to envy people with amazing memories like it was a super power. Then I realized if I got into the habit of just writing stuff down, I'd effectively have that same superpower. It's the one thing that I can honestly say has changed my life. I use obsidian.md for notes, which I have on phone and desktop. Nothing fancy. You just need something that's always with you, has great search, and ideally allows you to link notes together.
- hnthrowaway0328 3y ago
  I don't really have amazing memories. I just found out most of my stuffs are worthless and do not worth being preserved.
- balder1991 3y ago
  It’s not only about keeping memories but structuring them. Even journaling (about your thoughts and feelings) helps with making sense of what’s on your mind: https://youtu.be/FNJO1pZV-I8?si=Efqm5uAOSp9MSeOq https://youtu.be/FNJO1pZV-I8?si=Efqm5uAOSp9MSeOq

cfr2023 3y ago

An interesting read and a good share. Every time I see that other people are contemplating these subjects, I notice that they hit on many of my own considerations and miss many of the big picture things I consider essential. I think this will evolve into tribalism of mental models for organizing information. More vi vs emacs energy to go around in the decades to come.

riffic 3y ago

this is effectively a very geeked out zettelkasten method. https://en.wikipedia.org/wiki/Zettelkasten https://en.wikipedia.org/wiki/Zettelkasten

moonlion_eth 3y ago

Anytype all day every day

deleted 3y ago
[deleted]
ssivark 3y ago
I was very excited about this when it launched (in alpha). How has the product matured? Also, does it now have an API for easy import/export of data into the database?

openquery 3y ago

This is the second piece of content I've ever seen using the same numbering system as Wittgenstein's Tractatus, the first of course being the Tractatus itself.

throwup238 3y ago
Can't wait for the followup, Knowledgebase Investigations in which he realizes that he has derived no use from his knowledgebase. Water!

dmje 3y ago

Obsidian. Total game changer. Obsidian sync is completely worth paying for as well - firstly it helps support the project but also its slick, so well made, and easier than all the hacked together GitHub / s3 / whatever workarounds. From a content point of view I personally find having structure but not too much structure is the way to go. A bit of yaml if needed but just a gentle “daily notes = meeting notes” (on my work obsidian) and “daily notes = journal” (on my personal one), a few tags but nothing too rigid or cumbersome. Omnisearch is the dogs nuts plugin too, and a couple of others but again, keeping it quite minimal has really worked for me. Just so nice knowing that everything is under my control, using just markdown but with this fantastically powerful framework underneath it. Huge fan.

bravura 3y ago
I haven't checked Obsidian in a while, can you explain why you love it so much? I hear it being discussed a lot.
- dmje 3y ago
  I think it’s the balance of being “just markdown” on one hand - you can literally just point it at a folder and start typing; and the fact that it’s incredibly extendable on the other. The better known plugins like DataView give you really powerful list / sort / search if you want that, but you can also make it look however you want. I’m strangely snobby about my UIs, so being able to take a theme that’s almost right and then tweak the css is a big deal for me. Finally - it’s just a fast and solid bit of engineering. Rarely / (never, actually?) crashes, shortcuts are really powerful, constantly being updated… all manner of great :-)
rtpg 3y ago
I really wish that Obsidian Sync had like... 5 megs for free or something. I can't justify paying $8/month for a thing I haven't even started using, and without sync I have a hard time saying "OK let's try this" compared to Joplin or Notion. There's a bunch of cool looking stuff though
- 3y ago

animal_spirits 3y ago

Anytime a knowledge management system comes into my head I remember this joke: https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F2li4r1riaoe91.jpg https://www.reddit.com/media?url=https%3A%2F%2Fi.redd.it%2F2...

rsync 3y ago
Well, if you s/apple\ notes/vi/g ...

cobertos 3y ago

Isn't this what Hypercore[0] is? But distributed? It's a append-only log, and then there's higher order abstractions built on top of this log. Hyperstore is a key value store that uses an append only log, hyperdrive is a filesystem, etc... [0]: https://github.com/holepunchto/hypercore https://github.com/holepunchto/hypercore

shrubble 3y ago

I have recently started to play around with the lightweight Emacs mode, "HOWM" which was written about 20 years ago by a Japanese author, who still maintains it today. The howm-mode shows you a summary view, comprised of information (just the title/heading) from the (configurable) last 20 notes you have made. Everything goes under ~/howm/YYYY/MM e.g. ~/howm/2024/01 for notes I am making this month. It has a simple way to enter schedule, todo, deadline, etc. which it shows just below the header that has single-key commands (s for case-insensitive searching, c to create a new note, etc.) for searching, navigation, new notes. The genius of howm is that you can write and have very simple back and forward links to any file, URL, or tag, any kind of text etc., and you write "fragments" that is, whatever the smallest unit of text you want - it just needs a title plus anything else you want to add to it. When you save the file, you hit Ctrl-C plus ",," and are back in the summary view. So you can have very small sized units/fragments, which are very fast to create and save, but due to the forward and back links and the ease of searching (which will collate in a temporary list, everything found), you can "create fragmentarily but view/search collectively" as the documentation puts it. My next step is to author HOWM notes in AsciiDoc or Markdown format which can easily be fed (since each note is a separate text file) into a converter for fancier output. Syncthing easily keeps things in sync between my desktop and laptop. In the very best of FOSS tradition, someone else wrote up a full PDF manual, in English. https://github.com/Emacs101/howm-manual/blob/main/Howm_tutorial_eng.pdf https://github.com/Emacs101/howm-manual/blob/main/Howm_tutor...

accrual 3y ago

Several comments about personal note taking systems. Here's mine. :) function note { mg +-1 "/home/user/notes/$(date "+%Y-%m-%d").txt"; } "mg" is "Micro Emacs clone" and "+-1" opens the file at the bottom for appending, but of course it works equally fine with your choice of $EDITOR. I like this setup because it's plaintext (or markdown if you prefer), easily sorts on filename, and easily greppable. I have a couple years of notes saved this way and it's fun to explore old notes.

devdiary 3y ago

I use the similar kind of structure in Developer Diary. I do it completely offline, no sync.

neighbour 3y ago

Whenever I try one of these apps I'm always impressed by the features but my entire life is in iCloud. Not only that but all the people I connect with for notes/calendars/reminders are also in the Apple ecosystem and they aren't going to switch for me. This means that a lot of the extra features that Notion, Obsidian, etc. offer just aren't valuable for me. It makes me annoyed that Apple's Notes app is so basic.

pif_ 3y ago
Maybe NotePlan[0] can sync with iCloud? [0]: https://noteplan.co/ https://noteplan.co/
faizmokh 3y ago
You can use iCloud to sync your Obsidian notes across your devices. I find the app experience to be terrible tho.

rickcarlino 3y ago

I hope more people share content like this in HN. Note taking is incredibly personal and as a result, I think the best ideas might be lying dormant and unshared.

fouc 3y ago

I thought Knowledge Lakehouse was going to be a variation of a Memory Palace

up2isomorphism 3y ago

What do all these have to do with lakehouse? (and that is a lame term to begin with anyway)

LAC-Tech 3y ago

I enjoyed this article, though not sure I can be bothered printing books. The approach taken here is essentially event sourcing, btw. It's old and it works.

IOT_Apprentice 3y ago

I’m using capacities.io it is much more accessible than obsidian.

wt__ 3y ago

My approximate journey (7,500 notes, 13% markdown, remainder .txt) (pc, circa 2010) notepad > Evernote > notepad > (mac) > vim > Notational Velocity > NVUltra (still in) beta > "The Archive" (Zettelkasten) > (n)vim - notes synced with Dropbox folder (could just as well use iCloud) - with vim fzf/telescope plugins for search What finally flipped me back to vim was using a setup script to give me a useful set of plugins that did nice markdown highlighting out of the box, and then working out how to do things like auto-indenting, opening URLs in browser, renaming files more easily etc. Slight annoyance: yet to find a way to search both filenames and file contents at the same time Do systems that link from one note to another help me? Apparently not much. Quite a lot of effort to maintain the links, better to express hierarchy in filenames. (still using Omnifocus for todo lists, but it can import/export Taskpaper text format)