Trivial answer is Google Docs and equivalents. Etherpad is one such equivalent that has some advantages in respecting privacy and self-hosting. Seems likely you’ve considered cloud-based office software and found it lacking, though.
Personal knowledge management software like Roam, Obsidian, Logseq, etc., is generally trying to offer this level of generality. Many of these variants store data in simple data types, allow several different structures (markup languages, tables with formulas, nested lists, embedding of quick drawings, and so on, with OneNote as one of the most flexible on this point), and strong searching and crosslinking. Automation is less well-supported, both in running scripts internally and in interfacing with an API. Collaboration is very poorly supported, but this is a significant development area and several of the major contenders are competing to release it first.
However, the gold standard in general purpose computing has got to be emacs. The only one of these things that emacs org-mode is bad at is real time collaboration. If you want collaboration in emacs you probably can’t make it much faster than pushing and pulling git commits. For everything else, storing your contents in a git repo and accessing them with emacs is highly effective. The downside is that maintaining your emacs configuration in a usable state is difficult and requires significant expertise. Packages and distributions change rapidly and often break each other, so you need to be good at troubleshooting and keep up on the development news. Using a good distro like Doom Emacs or Spacemacs makes things easier, but definitely doesn’t solve the problems. Not to mention the learning curve, which isn’t trivial. It isn’t worth it for most people, but it’s certainly the closest you can get to a fully general computing system using modern software.
Obsidian.md meets many of your requirements (using Dataview and other plugins), but not web API or real-time collaboration (unless you have a good+fast file sync tool and don’t mind a 1-2s update interval).
Obsidian stores notes as markdown text with YAML front matter, and the Dataview plugin allows you to define code blocks that perform queries and format data, using either its query language or embedded Javascript. You can also insert calculated values, and it can use fields from YAML or marked up values in the bodies of notes.
With other plugins you can do drawings (including script-generated ones through an exposed API), kanban boards, mind maps, ebook annotations, etc. Full text search is built in, with a lot of search operators.
Want automation? Macro plugins and user-scripting plugins. Plugins that provide local obsidian:// URLs you can invoke from other programs to fire off commands. Plugins that let you invoke external commands on a note or folder, or launch calculated URIs, on demand. Or write your own plugins in JS. (There’s even a web server plugin, though it’s currently limited to statically serving your repo contents.)
Want collaboration or remote? Use any file sync tool you like. If a file is changed on disk while you’re editing, the changes are detected and merged, and the autosave interval is about 1s, so at least in principle if you had a fast enough file sync tool you could co-edit different parts of the same file. If you’re editing different notes in different panes, the other person’s edits will show up as soon as the file is closed after sync.
None of these parts can compare with something specialized in that area, but a jack of all trades, master of none, is still often better than a master of one. ;-) And it doesn’t hurt that it’s free and extensible.
Hmm, I guess conflict resolution would be garbage, but simultaneous editing is rarely a good experience anyway. Otherwise storing and sharing text files using a file sync service is fairly good compared to other options. Thanks!
Pieces for a general purpose personal computing system. Ideally:
Edit data by hand
Store as plain text
Self-host, access from any device
Write formulas to derive data automatically
Mix and match structured data (markdown, tables, nested lists, whiteboard)
Search and navigate, like any wiki
Automate through a web API and webhooks
Collaborate in real time
Trivial answer is Google Docs and equivalents. Etherpad is one such equivalent that has some advantages in respecting privacy and self-hosting. Seems likely you’ve considered cloud-based office software and found it lacking, though.
Personal knowledge management software like Roam, Obsidian, Logseq, etc., is generally trying to offer this level of generality. Many of these variants store data in simple data types, allow several different structures (markup languages, tables with formulas, nested lists, embedding of quick drawings, and so on, with OneNote as one of the most flexible on this point), and strong searching and crosslinking. Automation is less well-supported, both in running scripts internally and in interfacing with an API. Collaboration is very poorly supported, but this is a significant development area and several of the major contenders are competing to release it first.
However, the gold standard in general purpose computing has got to be emacs. The only one of these things that emacs org-mode is bad at is real time collaboration. If you want collaboration in emacs you probably can’t make it much faster than pushing and pulling git commits. For everything else, storing your contents in a git repo and accessing them with emacs is highly effective. The downside is that maintaining your emacs configuration in a usable state is difficult and requires significant expertise. Packages and distributions change rapidly and often break each other, so you need to be good at troubleshooting and keep up on the development news. Using a good distro like Doom Emacs or Spacemacs makes things easier, but definitely doesn’t solve the problems. Not to mention the learning curve, which isn’t trivial. It isn’t worth it for most people, but it’s certainly the closest you can get to a fully general computing system using modern software.
Thanks!
I wasn’t aware of Etherpad. Other Google Docs equivalents seemed impossible to self-host and extend, so a non-starter.
I agree with your overview:
Etherpad provides collaborative editing, but integrating it with other services will probably take extra work
Logseq has better structure, but worse automation
Emacs can do most things on one computer, but rapid sharing is even harder
Obsidian.md meets many of your requirements (using Dataview and other plugins), but not web API or real-time collaboration (unless you have a good+fast file sync tool and don’t mind a 1-2s update interval).
Obsidian stores notes as markdown text with YAML front matter, and the Dataview plugin allows you to define code blocks that perform queries and format data, using either its query language or embedded Javascript. You can also insert calculated values, and it can use fields from YAML or marked up values in the bodies of notes.
With other plugins you can do drawings (including script-generated ones through an exposed API), kanban boards, mind maps, ebook annotations, etc. Full text search is built in, with a lot of search operators.
Want automation? Macro plugins and user-scripting plugins. Plugins that provide local
obsidian://URLs you can invoke from other programs to fire off commands. Plugins that let you invoke external commands on a note or folder, or launch calculated URIs, on demand. Or write your own plugins in JS. (There’s even a web server plugin, though it’s currently limited to statically serving your repo contents.)Want collaboration or remote? Use any file sync tool you like. If a file is changed on disk while you’re editing, the changes are detected and merged, and the autosave interval is about 1s, so at least in principle if you had a fast enough file sync tool you could co-edit different parts of the same file. If you’re editing different notes in different panes, the other person’s edits will show up as soon as the file is closed after sync.
None of these parts can compare with something specialized in that area, but a jack of all trades, master of none, is still often better than a master of one. ;-) And it doesn’t hurt that it’s free and extensible.
Hmm, I guess conflict resolution would be garbage, but simultaneous editing is rarely a good experience anyway. Otherwise storing and sharing text files using a file sync service is fairly good compared to other options. Thanks!