πŸ“š node [[wikilinks amount to a lightweight universal standard for knowledge interoperability]]
πŸ“– stoas
β₯… related node [[agora-protocol]] pulled by user

Agora Protocol

Self: flancia.org/go/agora-protocol

See also: flancia.org/agora (motivation and TLDR), flancia.org/go/agora (early stage implementation), flancia.org/go/agora-roadmap (roadmap)

By: [[D. W.]], [[Flancian]], your name here!

[[Meta, and a note to contributors]] 1

[[Background]] 1

[[Wikilinks]] 2

[[Backlinks]] 2

[[Block references]] 2

[[Multi-user experience]] 2

[[Proposal]] 2

[[Concepts]] 3

[[Multinodes, or constructive conflict handling]] 3

[[Other clients]] 3

[[Data format]] 3

[[Go links]] 4

[[Contexts]] 4

[[Open questions]] 4

[[Possible Applications]] 4

Meta, and a note to contributors

  • Agora is just a working code name; it is a recurrent label that one of the authors has been using as a placeholder for a set of concurrent, only occasionally convergent ideas. Please feel free to propose a different name, or indeed to ignore the name overall.
  • This document will probably make use of unresolved [[wikilinks]] both for the purpose of demonstrating the concept and labelling entities. Eventually this text is expected to move into a reference Agora, and such links will resolve.
  • I tried keeping Background to about one page and Proposal to about one page. Unsure if this is the right proportion/format at all. Open to any and all feedback :)

Background

The Agora is an annotation layer for the internet based on a loosely coupled socially maintained [[distributed knowledge graph]] based on [[wikilinks]] and back references.

Some conventions may be emerging in a new batch of personal knowledge management tools

1

, or amenable to them; for the duration of this document, we'll refer to these tools as Roam-like. These systems have caught on over more legacy apps such as Evernote, because associations can be created in a mesh or graph style between concepts and notes instead of relying on folders and more traditionally hierarchical organizational approaches.

This document seeks to explore common ground and propose a way towards a protocol for cross-tool distributed collaboration including aspects like interop, federation, etc.

Wikilinks

The ubiquity of [[wikilinks]] is a notable innovation over the previous batch of Personal Knowledge Management tools. The [[wikilink]] convention is in wide use not only in wikis proper, but also in all Roam-likes.

At their core, they allow a user to quickly make a relative "wiki-style link" between a note in one place and a note or concept in another. Typically the user will employ a wiki-like convention to begin-- for instance, typing "[["-- at which point an increasingly constrained set of options will be offered in real time as the user continues typing, against a set of notes or concepts that have already been created in their personal knowledge graph.

As of the time of writing, all Roam-likes support [[wikilinks]] with optimistic resolution

2

; following a non-existent [[wikilink]] triggers a new note creation. This encourages link-driven writing and enables users to easily create stub pointers to entries to be filled in later.

Backlinks

Backlinks are core to the note taking and navigation experience in Roam-likes. They are the main innovation over most common Wiki tools previously in widespread use.

Backlinks are usually implemented as a list of incoming edges to the currently focused node.

Block references

Block references are critical to some, but not all, of Roam-likes; notably Roam Research and Athens Research support them. Block references are an instance of block-level transclusion.

Obsidian, Foam and other players either don't support References at all or support only limited forms (e.g. Obsidian can transclude sections only). It is unclear whether this should be a core feature of a protocol aiming to federate between tools.

Multi-user experience

At the time of writing Roam is the only tool with an established multi-user (a.k.a. "multiplayer") experience being developed.

Proposal

We propose to develop Agora, a protocol that

  1. Enables interop between Roam-likes and the rest of the participating internet, in particular the [[fediverse]] and the [[semantic web]].
  2. Offers a common interface for tools targeting Roam-like databases as knowledge graph backends.

Concepts

An Agora hostsa collection of interlinked gardens # 3. Each garden is an instance of a personal knowledge graph such as that produced by any Roam-like targeting a database. For the purpose of this document, assume that distinct users A, B and C all publish their digital gardens in an Agora.

Multinodes, or constructive conflict handling

Whereas in a Garden each node id is expected to be unique, in an Agora conflicts are desired and enable multinoding # 4_._Assume users A and B have nodes in their gardens with a given id, for example Stoa.

Multinoding results in the following behaviour:

  • When a user of the Agora (not necessarily A or B) visits a [[Stoa]], both A and B's nodes will resolve, and be shown one after the other.
  • When user C creates a new node in [[Stoa]], they are made aware of A and B's node (it is shown as related context).

Other clients

One of the most common uses of tools such as Hypothesis is for users to create annotations which are then imported into a wiki-note application for other purposes.

An obvious affordance would be for folks to directly reference their or other's knowledge graphs in their own annotation bodies. Both the Hypothesis app and others in this category are often using a flavor of markdown as the editor syntax, so the applications are already conceptually similar to each other.

Annotation tools could connect to an Agora of the user's choosing so that while an annotation was being created, a direct reference to a concept node could be created inline. The editor would need to be able to query the remote knowledge graph in real time such that candidate concepts could be presented as the user types, just like in a native client. One could imagine this kind of capability becoming social, if some knowledge graphs were exposed publicly and one could query across them and choose amongst them. In this way, if the Hypothesis note were to be exported to another systemβ€” including the remote knowledge app itselfβ€” the direct reference to the graph would automatically work.

Data format

Most Roam-likes, with the notable exception of Roam, store their data as plain Markdown files. This makes their databases amenable to being hosted in common version control systems.

With the exception of block references, Roam notes can be exported to this format in a lossless way.

The Agora v0.5 reference implementation makes use of this data format and is based on git subtrees. This makes it so that each participating user can host their gardens independently, and rely on the Agora exclusively for Agora-enabled flows (such as multinoding and publishing).

Contexts

To be written. See [[distributed knowledge graph]].

Open questions

  • Should [[wikilink]] resolution collapse or maintain plurals and other common variations? Same for go links.

Possible Applications

Go links

Note [[go links]] and [[wikilinks]] have synergies. Go links can be seen as HTTP 302 as a service; they can provide both simple social bookmarking (letting users easily claim "URL space") and be seen as a social knowledge graph client. Making clients resolve also resolve [[wikilinks]] to [[go links]] as published by participating domains (perhaps those in a list controlled by the user, or those of their "friends") would allow users to crowdsource interesting targets.

Wikilinks everywhere

See: anagora.org/go/wikilinks-everywhere.

[[1]]Roam Research; Athens Research; Obsidian; Foam; Notion.

[[2]] Or as Ward Cunningham said: "When you reach the edge of your knowledge, create a new Wiki Page" (h/t Gyuri Lajos for the quotation).

[[3]] The term comes from digital garden, as exposed in https://joelhooks.com/digital-garden.

[[4]] Noding is borrowed from everything2.com, which successfully explored a similar setup.

β₯… related node [[wikilinks-everywhere]] pulled by user

Developed by [[@evan]] Boehs, [[unlicense]]d, powered by [[flancia]]

Wikilinks everywhere is a chrome extension that brings [[wikilinks]], and through it the [[internet]], to life. It can be installed on a large number of browsers and services, and will intelligently determine where [[broken]] wikilinks should go to based on your settings.

Sources

You can order or disable any of these sources at will. When a wikilink (regardless of if it is properly linked already) is detected the extension will go through these sources in that order you provided. The default order is explained below.

Options

Global

  • Error codes:
    • Comma separated HTTP status codes
    • Supports nxx
    • Default: 4xx, 5xx

Source specific

  • Ignore errors
    • Checkbox
    • Should error cases be ignored?
    • Default: Unchecked

Actual sources

Note: The last source is the catchall, no matter what it will ignore errors and faithfully complete the link. This means do nothing has importance. Don't remove or reorder

Existing

If a existing link is already present do nothing

Note: the only difference between the existing source and others is it changes nothing in the browser. By default, if the existing link resolves to global.error, it will move on to the sources below and consider replacing it with them

Note: this might have made no sense. Don't mess with it, probably, it exists for a reason

Prefix

Things like WP:link for wikipedia, or AG:link for [[anagora]], the schema looks like PREFIX:page.

Options

  • Prefixes
    • A table of prefixes and the corresponding url
      • use {page} to have the provided back link, so with the WP prefix the url would be wikipedia.org/wiki/{page}/
      • DON'T INCLUDE THE : IN THE PREFIX

[[IWLEP]]

The user's [[IWLEP]] rules. See the IWLEP page for that config, but it is it looks a lot like your config, instead made by the author of the backlink. Don't worry, it can't be faked *unless the user gets hacked but lots worse things could happen.

Unprefixed

Iterates through all the listed urls in your order until one does not error, or all error.

{page} is the value between the brackets

Options

Nothing

What does it say on the tin?

Concerns

Bandwidth

We use an extraordinarily large amount of bandwidth, and there is not much that we can do. That being said, the whole web is obese so I highly doubt we end up using more than 0.2 of the bandwidth of a standard blog (~1Kb a page so ~15kb a wikilink so ~150kb a page). Below is what we tried

βœ”οΈ Request methods

There is no STATUS request so we had to make due with HEAD. This will only fetch the part of the website that has the page name and useful information. It does not get the page contents, saving bucket loads of bandwidth.

πŸ‘·β€β™€οΈ Local catch

Once implemented we will store the last 1000 pages we checked, so if a wikilink is frequencintly used it will only be loaded once

❌ Catch server

This means storing the status code of every website we checked. I don't really like the idea of storing personal information like that.

Ratelimits

By

Browsers

  • Powercord πŸ”Œ (Discord) Plugin
  • Chrome
  • FireFox πŸ”₯🦊

Wikilinks Everywhere

[[2021-04-14]]

The plan is to get in at least two pomodoros in this to advance it somehow in the direction of a prototype. Will try to build on [[agora ext]].

See also:

By: [[samuel klein]], [[dan whaley]], [[flancian]], your_name_here

Project description

Wikilinks everywhere: a web extension/library/bookmarklet that eagerly or lazily resolves [[wikilinks]] in any web property within a user-chosen context, e.g. an Agora or other distributed knowledge graph.

While using the web, users often need to interact with or refer to their personal knowledge databases (PKB). This means taking notes (about web content or otherwise), and then referring and linking to them (as they can be resolved in their PKB). This process is needlessly cumbersome; users often know precisely which PKB entries they would like to refer to, and can remember their titles, but have no interactive, real time way to insert or resolve these links. Currently they need to go through a multi-step process to fetch them (open database; search for them; select them; get a canonical URL if available; copy/paste it and any relevant information), or they'll flag the term and remember to revisit it later. Instead, a user could type [[wikilink]], with wikilink standing for the title of the note in question, and depend on a web extension to resolve the note reference to canonical URL either eagerly (at the time of writing) or lazily (at read-time).

Use cases

  • As a writer, I know that a KB entry exists, but don't have immediate access to it. So, when I am typing I'll create a stub pointer to a KB entity by typing "[[entity]]". At some point later, those links will be resolved by an asynchronous process.
  • As a writer, I want immediate access to my KB graph, or other public graphs I'm connected to, wherever I am so that as I'm taking notes around the web I can add wikilinks in real time through the wikilink convention "[[entity]]". Real-time KB linking should work w/ type-ahead so that as I'm typing, suggestions of entities that satisfy the characters I've typed (from whichever KB) are presented instantaneously, and I can click to confirm.
  • Unresolved [[wikilinks]] as produced by users through the above use case could be resolved socially; that is, the resolution need not be scoped necessarily to a particular personal knowledge graph, but rather to a set of them. This opens up a second use case: that in which a reader of an unresolved [[wiklink]] created by a third party can choose to resolve the wikilink in a user-specified content; by default, their own knowledge base or user chosen set of knowledge bases (Agora).
    • Note that, if the above use cases prove popular/valuable, unresolved [[wikilinks]] may proliferate within certain social circles and drive adoption of the system.

Applications

  • Ex: resolving wikilinks on hypothes.is notes. [default InstantLinks]
  • Ex: resolving wikilinks in Twitter, Facebook, Google Docs/Etherpad -- anywhere there is user generated content that happens to make use of them.
  • Make this work with [[w:Wikis]], [[d:Wikidata]], [[f:Files]], and {{t:Templates}}
    • Then make subst: work [like _flattening_ for go-links]
  • Make a default resolution pathway for each source, & a default meta-resolution $path (where to look for the pathway config)
    • $path can include localfile, agora:config, metawiki:config, ul:r ...
  • could say: whenever you force flattening of a link [in agora] you could explicitly note this as an addition

Resolution flow

  • The extension should maintain a list of 'wikilink providers', that is, sites that are known to offer wikilink resolution as a service.
  • The user can configure resolution priority for providers.
  • The extension may default to resolving wikilinks to the top priority provider in every case, although it may also choose to do background presence checks to link wikilinks at the top priority provider that already hosts a valid node in the wikilink in question.

[[foo]] -> $base_url/foo

Plan

Side projects

  • Chat
    • Zulip streams could map to Agora nodes
    • Related because it's another type of wikilink integration adapted to a particular website
    • Make use of the magic-hat node-resolution that's trustworthy.
  • Twitter
β₯± context