In the first part of this series I’ll outline the main factors why I’ve decided
to move my digital garden / braindump / Zettelkasten to org-roam and which factors
have facilitated this decision. In the 2nd part (still work in progress) I will expand more
how I’ve built the new brainfck.org using hugo, ox-hugo and org-roam.
Motivation
I’ve been using Tiddlywiki for almost 10 years now and setup different instances for
work, personal stuff and lately as my own personal knowledge management system. I’ve
used it to save highlights, notes, quotes from different sources and organized them
in an useful way. I used daily journaling to handle the daily input of ideas and (web)
articles I’m constantly exposed to. I talked at work about the importance of PKMS
and how Tiddlywiki can increase productivity and contribute to better (mental) health
by using it as a second brain.
Other popular terms: Zettelkasten (slip box in German), memex, braindump or digital garden.
Basically it’s all about giving your a brain a rest and offload information to a medium so
your brain doesn’t have to remember everything:
While I initially started using just one single HTML file for my
tiddlers, I soon switched over to the nodeJS installation. This still
has better benefits like:
Among the many packages I’ve used, stroll has definitely changed the way I interacted with
Tiddlywiki. It allowed me to focus more on the note-taking process by dividing the screen
into 2 columns. This allowed me to work on different tiddlers simultaneously. Still it
took me hours to process my notes and digitize them into Tiddlywiki. I guess the UI kind
of slowed me down, mostly because I’m a keyboard-centric user and don’t use the mouse that
often. Switching between tiddlers, closing them, creating new ones always involved mouse
interaction.
For the same reason I’ve been using VIM for more than a decade and since more than 2 years
I’m happy to consider myself an evil Emacs user. It became not only my primary editor, but
also my RSS feeds reader, mail client, YouTube video player, IDE, API client… I
basically live in Emacs
Here is my config.org
and try to avoid as many context switches as possible.
Personal preferences
ORG mode as lingua franca
After going down the Emacs rabbit hole, I’ve adopted ORG mode as my main file format for writing documents, exporting these to other formats (PDF, markdown, Confluence, Jira and many others), creating diagrams (mainly plantuml), presentations, writing technical documentation and hopefully some day for publishing a whole book. For the note-taking phase I write my notes in ORG mode and create a rudimentary outline sorted by chapters/sections. Usually I use the same structure to create my blog posts from (like I did in the book summaries). Extracting pieces of information for individual tiddlers, however, tends to be a time-intensive process. I’ve managed to use the Tiddlywiki API within Emacs but my Elisp skills are still not good for doing more advanced stuff like:
fetch existing tiddlers, modify body in a new buffer, save new tiddler
when linking text to new/existing tiddler, show list (in the minibuffer) of Tiddlers and if not create new one(s)
show cross-references (e. g. Backlinks) for a specific tiddler
refile specific (ORG) headline to a new tiddler
All these features are some however doable within Tiddlywiki using stroll and streams. But I don’t want to use the web UI anymore since I’m already inside Emacs for the majority of the day 😅
Editing on steroids
At some point I began adopting ORG style syntax for the new tiddlers too:
If you pay attention, there are lots of similarities. That’s why I could easily copy and paste most of the ORG content into the tiddlers. As for the rest (source blocks, quotes, examples, sidenotes etc.) manual conversion (or using ox-tiddly ) was necessary.
It was especially this part that slowed me down in my post-reading process mainly because:
I’m writing my notes in Emacs (using ORG)
converting to full tiddlywiki syntax takes time
in some of blog posts (written in ORG) I wanted to include some content from different tiddlers
I had to convert Tiddlywiki content back to ORG syntax again
This back and forth between ORG/Emacs and Tiddlywiki combined with the fact I was maintaining multiples sources of information (my raw notes in ORG, my own thoughts / processed notes in Tiddlywiki) brought me to org-roam. Not only this, but it also forced me to rethink my note-taking workflow and make adjustments to the whole system.
I’ll explicitly cover org-roam, hugo and ox-hugo in the next part.
Exporting from Tiddlywiki
As I’ve started exporting my notes from Tiddlywiki I soon realized there are 2 options to do so:
David Alfonso has done a great job and put together a repository that helps you with the
export of tiddlers. All you need is to export all your tiddlers bundled as one single HTML and
then follow the instructions in the README.
In my Tiddlywiki root directory I had a tiddlywiki.info with a build step to export all tiddlers:
tiddlywiki . --build index
ls -lh ./output/index.html
1
-rw-r--r-- 1 victor users 6.0M Aug 30 06:09 ./output/index.html
Generate HTML and meta files
Once you have generated your single HTML Tiddlywiki file, clone the repository and copy your
file to wiki.html inside the repository’s root folder. Then you can run make to export your tiddlers.
Afterwards, for each tiddler, you will get:
a HTML file (with the tiddler’s content)
a meta file (containing header information)
As an example (for the “zucker” tiddler):
1
cat ./tmp_wiki/output/zucker.html
1
: <ul><li>Auch Saccharose</li><li>Gehört zur Familie der Saccharide</li><li>Formen<ul><li>Einfachzucker<ul><li><aclass="tc-tiddlylink tc-tiddlylink-resolves"href="#Glukose">Glukose</a></li><li><aclass="tc-tiddlylink tc-tiddlylink-resolves"href="#Fruktose">Fruktose</a></li></ul></li><li>Mehrfachzucker<ul><li>Stärke</li></ul></li></ul></li><li>Haushaltszucker<ul><li>Dissacharid</li><li>Besteht aus 2 Monosacchariden<ul><li>Glukose (Traubenzucker)</li><li>Fruktose (Fruchtzucker)</li></ul></li></ul></li><li><aclass="tc-tiddlylink tc-tiddlylink-resolves"href="#S%C3%BC%C3%9Fstoffe">Süßstoffe</a></li></ul>
The original repository will export by default all tiddlers to markdown. Since pandoc is used
we can also export to ORG mode directly by changing the Makefile.
I’ve created a fork with my own customizations.
Instead of exporting to commonmark we export to markdown first:
#+created: 20220201125456750#+modified: 20220203071728094#+name: Writing a Technical Book in Emacs and Org Mode#+note: Author writes about the workflow itself, importance of pomodoro#+revision: 0#+tags: Bookmark [[ORG Mode]] Writing#+title: Writing a Technical Book in Emacs and Org Mode#+type: text/vnd.tiddlywiki#+url: https://www.kpkaiser.com/programming/writing-a-technical-book-in-emacs-and-org-mode/
So each bookmarks consists of:
a name
a note
an url
a title (usually the same as name)
Now we can easily parse these files and create the desired structure. For this purpose I’ve used this tinny Python snippet:
* [[https://www.kpkaiser.com/programming/writing-a-technical-book-in-emacs-and-org-mode/][Writing a Technical Book in Emacs and Org Mode ]] :Bookmark:[[ORG:Mode]]:Writing: :PROPERTIES:
:CREATED: 20220201125456750
:NOTE: Author writes about the workflow itself, importance of pomodoro
:END:
This way we get a nice ORG mode headline with some properties. Now let’s convert
all available bookmarks and save into one big file:
* 2020-09-14- [[https://www.swr.de/swr2/programm/broadcastcontrib-swr-13438.html][Wie funktioniert Selbstregulierung?]]
- auch in der [[https://www.ardaudiothek.de/wissen/wie-funktioniert-selbstregulierung/80172244][ARD audiothek]]
- Un podcast interesant despre [[https://www.stareanatiei.ro/podcasts/][starea natiei]]
- este si [[https://www.youtube.com/channel/UCtK5Oe8sHjp6WPcwWuHUVpQ][canal youtube]]
- [[https://stackoverflow.com/questions/42531643/amazon-s3-static-web-hosting-caching][how to use caching with S3 static site hosting]]
* 2020-09-15- this site supports now [[https://brainfck.org][TLS/SSL]]
Extract books
This was the most difficult part and I’ll try to explain why. This is how a book tiddler usually looks like (1984):
Basically I wanted to merge every tiddler into one ORG file.
Instead of applying some sed & awk magic, I decided to use Tiddlywikis internal templating
system. The $:/core/templates/static.tiddler.html template for examples defines how a
single tiddler should be exported to its corresponding HTML file:
We can use the same mechanism to define a template for a book tiddler whenever this has to be
exported. But first of all let’s see how a template is used when exporting:
1
2
3
4
5
6
export-books :depspre @echo "Exporting all book tiddlers from $(ORIGINAL_TIDDLYWIKI) to ORG with custom render template"$(NODEJS)$(TIDDLYWIKI_JS)$(WIKI_NAME) --load $(ORIGINAL_TIDDLYWIKI)\
--render [!is[system]tag[Book]][encodeuricomponent[]addprefix[books/]addsuffix[.org]]\
text/plain $$:/vd/templates/render-book
$(NODEJS)$(SAFE_RENAME_JS)$(TW_OUTPUT_DIR)
This is what happens:
we load the single HTML Tiddlywiki file via --load
we use --render
Read more about the RenderCommand.
to export a list of tiddlers
as a filter we use [!is[system]tag[Book]] which means:
We create a list of tiddlers with following filter: [all[current]] (another way to express we just want the current tiddler).
We then create an ORG mode headline consisting of the field title in the current tiddler ({{!!title}}). Then we add
a FINISHED property using the fields finished_year and finished_month
We create a sub-heading called Notes where we add additional sub-nodes. For this to work we create again
a list of tiddlers where we apply the filter: [subfilter<childrenFilter>]. childrenFilter is defined
at the top:
prefix<currentTiddler>
We focus only on the tiddlers which have the currentTiddler as a prefix.
if currentTiddler is 1984, then this will match
1984 - Note 1
1984/Wohlstand
!title<currentTiddler>
This makes sure we don’t match ourself (the currentTiddler)
Also here we create a sub-heading called Quotes and underneath we create additional sub-nodes
for the quotes. As for Notes we have a subfilter (quotesFilter):
it matches all tiddlers which have the currentTiddler’s title as a prefix
AND are tagged by quote.
Put everything together
Now that we have a template let’s have a look at the output:
* 1984 :PROPERTIES:
:FINISHED: 2021-05
:END:** Description
* Theorie und Praxis des oligarchischen Kollektivismus** von Emmanuel Goldstein
** Kapitel 1: Unwissenheit ist Stärke
** Kapitel 3: 1984/Krieg ist Frieden
** Notes
*** 1984/3 Arten von Menschen :note: :PROPERTIES:
:CREATED:
:TAGS: Definition
:END:*** 1984/Aufteilung der Welt :note: :PROPERTIES:
:CREATED:
:TAGS: Stub
:END:*** 1984/Der Große Bruder :note: :PROPERTIES:
:CREATED:
:TAGS:
:END:*** 1984/Doppeldenk :note: :PROPERTIES:
:CREATED:
:TAGS:
:END:*** 1984/Krieg :note: :PROPERTIES:
:CREATED:
:TAGS:
:END:*** 1984/Krieg ist Frieden :note: :PROPERTIES:
:CREATED:
:TAGS:
:END:*** 1984/Kulturelle Integrität :note: :PROPERTIES:
:CREATED:
:TAGS:
:END:*** 1984/Rolle der Partei :note: :PROPERTIES:
:CREATED:
:TAGS:
:END:*** 1984/Versklavung :note: :PROPERTIES:
:CREATED:
:TAGS:
:END:*** 1984/Wohlstand :note: :PROPERTIES:
:CREATED:
:TAGS:
:END:** Quotes
*** 1984 - Note 1 :quote:Krieg ist Frieden, Freiheit ist Sklaverei, Unwissenheit ist Stärke - Ministerium für Wahrheit
*** 1984 - Note 2 :quote:Gedankendelikt hat nicht den Tod zur Folge: Gedankendelikt IST der Tod.
*** 1984 - Note 3 :quote:"Begreifst du denn nicht, dass Neusprech zur ein Ziel hat, nämlich den Gedankenspielraum einzuengen? Zu guter Letzt werden wir
Gedankendelikte buchstäblich unmöglich machen, weil es keine Wörter mehr geben wird, um sie auszudrücken. Jeder
Begriff, der jemals benötigt werden könnte, wird durch exakt ein Wort ausgedrückt sein, dessen Bedeutung streng definiert ist und dessen
sämtliche Nebendeutungen eliminiert und vergessen sind."
*** 1984 - Note 4 :quote:Freiheit bedeutet die Freiheit, zu sagen, dass zwei und zwei vier ist. Gilt dies, ergibt sich alles übrige von selbst.
*** 1984 - Note 5 :quote:Die Massen revoltieren nie aus eigenem Antrieb, und sie revoltieren nie, nur weil sie unterdrückt werden. Solange man ihnen die Vergleichsmaßstäbe entzieht, werden sie nicht einmal merken, dass man sie unterdrückt.
I think that’s pretty good.
And this is the final result.
I intentionally didn’t add content in the Notes section to the sub-nodes. In the next post
I’ll explain how I managed to quickly review my notes using Emacs and some Elisp and add content on the go.