For a while I have been looking through my journal and other writings of mine over the years, trying to organize it all into a personal knowledge base, piece by piece, and it’s a very slow process. But it feels like your tool could massively speed up the process for me—if I were to split everything I’ve ever written into small chunks using some python scripts, I could set your tool to generate a map of it for me and use it to analyze clusters of related ideas, as a starting point for building a map of my whole mind.
I might end up having to do it all by hand anyway—but with a top-down overview, it would probably be more efficient, at least. Do you plan on developing a version of this which could be downloaded and used locally on one’s own computer, or does it require too much computing power for that to be viable?
The news thing is very intriguing to me. I had the idea a while ago—without the programming knowledge necessary to make it myself, and not knowing anyone who might be able to help—that there should be a web app with a global map which, given keywords or a time frame or both, marks related news events wherever they happened on the map, with each event being clickable to bring up a page of various individual news stories about it, including the known bias / political slant of each—and possibly also a machine-generated summary written to smooth out the various biases as much as possible and give a neutral account. Could your tool be used as part of such a thing?
Do you plan on developing a version of this which could be downloaded and used locally on one’s own computer, or does it require too much computing power for that to be viable?
Partially, yes. Transferring the whole tool into a usable portable app is a lot of troubles, especially the visualization is not so transferable now. So, I mainly think about publishing a piece of code that could be integrated into some existing note systems. It would be relatively easy to extract the part that goes through all documents and runs the clustering and embedding. Then, let’s say, it could add an yaml heading with tags to each of the notes in the format, compatible with the Nested Tags VSCode extension. In theory, we could also adjust the graph visualization extension to show the overview of notes, but it would be trickier. Would it be what you need?
Still need to say that I’d expect that the methods should be tuned to be applicable to notes, so it may not work out of box. In terms of computational power, it shouldn’t be a problem, unless you have 10+ thousands of notes.
Could your tool be used as part of such a thing?
Thanks for the detailed explanation! It’s quite close to the ideal outcome as I see it. However, the unbiased text summary part is close to impossible on the current level of technologies (to my knowledge). Maybe in several years. Then, the map of events requires a good way of extracting facts from the text. I really want to play with it at some point, but it could take a while. But we will be moving in this direction.
Then, let’s say, it could add an yaml heading with tags to each of the notes in the format, compatible with the Nested Tags VSCode extension. In theory, we could also adjust the graph visualization extension to show the overview of notes, but it would be trickier. Would it be what you need?
I have no idea. Unfortunately I am not a programmer and I’m not familiar with any of those things. You probably should explain it in terms of what I can do with it and how rather than talking about specific libraries etc; the most coding I am familiar with is mathematical algorithms in Python.
Sorry for the confusion. The things I listed are not programming libraries, but text editor extensions. VSCode is a very fancy text editor, and the extensions I listed are part of the Foam ecosystem, which allows nice ordering of notes and navigation across them. So if you’re interested, I’d suggest to check out those, as well as the recommended extensions list. May be helpful for your problem by itself.
For a while I have been looking through my journal and other writings of mine over the years, trying to organize it all into a personal knowledge base, piece by piece, and it’s a very slow process. But it feels like your tool could massively speed up the process for me—if I were to split everything I’ve ever written into small chunks using some python scripts, I could set your tool to generate a map of it for me and use it to analyze clusters of related ideas, as a starting point for building a map of my whole mind.
I might end up having to do it all by hand anyway—but with a top-down overview, it would probably be more efficient, at least. Do you plan on developing a version of this which could be downloaded and used locally on one’s own computer, or does it require too much computing power for that to be viable?
The news thing is very intriguing to me. I had the idea a while ago—without the programming knowledge necessary to make it myself, and not knowing anyone who might be able to help—that there should be a web app with a global map which, given keywords or a time frame or both, marks related news events wherever they happened on the map, with each event being clickable to bring up a page of various individual news stories about it, including the known bias / political slant of each—and possibly also a machine-generated summary written to smooth out the various biases as much as possible and give a neutral account. Could your tool be used as part of such a thing?
Partially, yes. Transferring the whole tool into a usable portable app is a lot of troubles, especially the visualization is not so transferable now. So, I mainly think about publishing a piece of code that could be integrated into some existing note systems. It would be relatively easy to extract the part that goes through all documents and runs the clustering and embedding. Then, let’s say, it could add an yaml heading with tags to each of the notes in the format, compatible with the Nested Tags VSCode extension. In theory, we could also adjust the graph visualization extension to show the overview of notes, but it would be trickier. Would it be what you need?
Still need to say that I’d expect that the methods should be tuned to be applicable to notes, so it may not work out of box. In terms of computational power, it shouldn’t be a problem, unless you have 10+ thousands of notes.
Thanks for the detailed explanation! It’s quite close to the ideal outcome as I see it. However, the unbiased text summary part is close to impossible on the current level of technologies (to my knowledge). Maybe in several years. Then, the map of events requires a good way of extracting facts from the text. I really want to play with it at some point, but it could take a while. But we will be moving in this direction.
I have no idea. Unfortunately I am not a programmer and I’m not familiar with any of those things. You probably should explain it in terms of what I can do with it and how rather than talking about specific libraries etc; the most coding I am familiar with is mathematical algorithms in Python.
Sorry for the confusion. The things I listed are not programming libraries, but text editor extensions. VSCode is a very fancy text editor, and the extensions I listed are part of the Foam ecosystem, which allows nice ordering of notes and navigation across them. So if you’re interested, I’d suggest to check out those, as well as the recommended extensions list. May be helpful for your problem by itself.