<!--
 Copyright 2020 Google LLC

 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
 You may obtain a copy of the License at

      http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing, software
 distributed under the License is distributed on an "AS IS" BASIS,
 WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
-->


<div class="subnode">
    <div class="subnode-header">
    <span class="subnode-id">
        <span class="subnode-links">
          <a href="/@flancia.org/about-unicorns-in-south-america" title="Click to see only this subnode.">📓</a> 
            <a href="/raw/garden/flancia.org/posts/about-unicorns-in-south-america.md"><span title="Click to see the raw resource.">posts/about-unicorns-in-south-america.md</span></a> 
            <span title="This resource was contributed to this Agora and seems relevant to the current topic."> by </span>
            <a class="user" href="/@flancia.org"" title="User @flancia.org contributed this.">@<span class="subnode-user">flancia.org</span></a>
        </span>
    </span>

    <span class="subnode-contrib">
        <span class="star-toggle " data-subnode-uri="garden/flancia.org/posts/about-unicorns-in-south-america.md" title="Star this subnode">
☆
</span>
    </span>
</div>
    <!--
.. title: About OpenAI's unsupervised language model, or unicorns in South America
.. slug: about-unicorns-in-south-america
.. date: 2019-02-16 19:32:04 UTC+01:00
.. tags: ml, ai, gpt, unicorns, flancia
.. category: 
.. link: 
.. description: 
.. type: text
<p><hr /></p>>

I was very entertained by [OpenAI&#8217;s recent announcement][1] (and [associated paper][3]) on their text generation model based on [Transformer][2] released earlier this week: &#8220;GPT-2&#8221;. I wanted to jot down some thoughts on it, and probably go off on a tangent with some ideas prompted by it. If you haven&#8217;t read it, I&#8217;d recommend you start by reading the announcement and browsing through the eight examples they provide. Note they are manually selected by a human, so we can safely assume many of the non-selected outputs were significantly less successful in following the prompts.

If some or many of the words in this paragraph don&#8217;t really make sense to you, I encourage you to read the examples and wait for the &#8220;primer&#8221; below; it may be enough to follow along.

Who am I and why does my opinion matter? Well, in short, and to be clear: it probably doesn&#8217;t :). I am a software/devops engineer working in a field not closely related to Machine Learning, who has only recently picked up ML as a hobby of sorts and (still?) doesn&#8217;t understand the state of the art. I also have an (outdated, mostly irrelevant by now) background in linguistics, but no degree. Finally, I have an interest in writing both fiction and non-fiction. My thoughts on this field may (at best) be interesting to others because of this relatively uncommon composite background, but may just as well be irrelevant. So: caveat emptor. But here we go.

![Llamas at Machu Picchu](https://upload.wikimedia.org/wikipedia/commons/0/0c/Llamas_at_Machu_Picchu%2C_Peru.jpg &#8220;User:RedRobot [Public domain]&#8221;)

# The shortest optional primer on ML you may find

I&#8217;m relatively new to ML, so this will not be an authoritative description. If you understood the first paragraph here, and the linked announcement, you can probably skip to the next section. I&#8217;m also not a great technical writer, so if you don&#8217;t understand this paragraph, feel free to skip it. The rest doesn&#8217;t really depend on it that much.

But very, very shortly: ML models are systems that learn from data (they are &#8220;trained&#8221;) and then are able to succeed at tasks that are related to the data they learnt from. Importantly, when such a model works, it can make useful predictions related to the task at hand for new data. Models are usually classified as &#8220;supervised&#8221; or &#8220;unsupervised&#8221; depending on whether the data you give them as input includes information on what the prediction or result you expect from them is. The thing you want to predict is called a &#8220;label&#8221;, so you can put it more succinctly by saying that you usually build supervised models with labelled data and unsupervised models with unlabelled data.

A typical supervised task is predicting quantities. Imagine a spreadsheet where each row represents a house in your neighborhood, and each column has some data about the house in question (these are called &#8220;features&#8221;). For example: area, number of stories, latitude and longitude, and finally price (the &#8220;label&#8221;). With enough rows representing actual houses, you can build a model that then predicts the price of a home not listed in the spreadsheet (say, yours) based on the other features.

A typical unsupervised task is clustering: you give a model a list of items of different kinds, and their features, and the model learns to group the items according to their qualities. Imagine a spreadsheet where each row is a fruit, and you have features such as colour, weight and shape. You don&#8217;t have the name of the fruit for any of these (you could build a supervised model if you did), but even without knowing the name (or knowing that names are somehow relevant to the task) the model can learn to group together heavy, oblong, green fruit (watermelon?) and lighter, roughly spherical, red fruit (apples?).

The model we&#8217;re discussing today is unsupervised, as it learns from a huge database of internet text that doesn&#8217;t include any labels &#8212; any explicit information on how to construct new texts successfully, or about what makes a &#8220;well constructed&#8221; text. But it still manages to learn something that feels significantly more complex than just grouping similar webpages together (which would be closer to the fruit example), or other classical unsupervised problems in this space: it learns how to produce new texts, basically by trying to predict how a text (provided, called prompt) is most likely to continue. Note this is a common approach in Natural Language Processing (NLP for short). [Other articles have been written by people better qualified than myself][10] about how exactly this model fits in the overall development of NLP; consensus seems to be that it&#8217;s an improvement, but not necessarily a breakthrough, and lots of the discussion has actually been about how OpenAI may be trying to gather more publicity that they should by not fully disclosing the model (potentially over-hyping it).

# On the literary merits of the texts produced
I want to start by considering the samples provided as texts, setting aside for a minute both the technical details of how they were generated and the context in which they were produced and released. Reading them, and discussing them with friends, was enjoyable in its own right. The texts themselves take turns at being better and worse than I expected, which is fun. I got the feeling that the model was taking good and wrong turns, and you could tell which were which &#8212; we can tell, but the model can&#8217;t, as it has no feedback overall on whether what it&#8217;s producing makes sense or not. It&#8217;s just trying to produce something that seems likely based on the prompt given and the corpus it was trained on (the internet).

In a sense, and if you allow me to anthropomorphize it for a minute, it sometimes writes like a child. Particularly striking was [sample #6][4], which starts with a homework prompt, and then proceeds to make sense at times and also go off rails at others. You could picture a child with a short attention span and no knack for editing/re-reading turning an essay with some of the same mistakes the model makes. Why is that? One possibility is that it&#8217;s seen many homework papers, and it&#8217;s just remixed them into this. Another is that it picked up on the style of such tasks, and it&#8217;s riffing on it. Without going deeper into how the model works, it&#8217;s hard (for me anyway) to tell; also, many of the most successful ML models score low on explainability &#8212; meaning that you can get them to work, sometimes surprisingly well, but then have no clue when you need to explain how they work on a per-part basis (in the case of a neural network, which has many neurons organized in layers, that would mean knowing how each neuron or each layer is &#8220;processing the input&#8221;). So it may or may not be possible.

Leaving all this aside, I liked picturing GPT-2 as a child &#8212; a child that is in awe of the internet; it has just read all of it and doesn&#8217;t know what to distill of it. Even if &#8220;all&#8221; the model is achieving is remixing existing ideas and expressions from the internet, it could already be an interesting new way to explore the internet and somehow detect patterns in it. At some point it mixes up Lord of the Rings and Star Wars &#8212; is this because of some pre-existing fanfic, or has it noticed how they are in many ways the same? In the Lord of the Rings spoof, where did the turn of phrase &#8220;I take nothing &#8212; but I give my word&#8221; come from? Is it somehow new or is it in common use and I just don&#8217;t know of it?

# On mistakes and creativity 
I have to point out here that the same mistake that ruins an essay or an article may enrich fiction. Essays and articles are grounded in reality, where there is right and wrong (to some extent &#8212; I don&#8217;t want to get too epistemological here). An essay that is untrue is not a good essay; an article may espouse an opinion that you disagree with factually and still be interesting, but probably not in the way the author intended.

Fiction, though &#8212; fiction is not as anchored by truth and falsehood in the same way. Fiction always contains a set of falsehoods, although which falsehoods are allowed is determined by its genre. In genres where worldbuilding is paramount (sci-fi, fantasy) internal consistency is more important than consistency with the real world.

Consider [sample #1][4], about an expedition that finds unicorns in South America. Sure, the prompt is already wacky, which probably helps &#8212; the model may generate a fair amount of wackiness by itself at all times, so having it in the prompt may make it seem more consistent that it usually is. But the model doesn&#8217;t miss a beat &#8212; even several paragraphs after the prompt, it&#8217;s still mostly on-topic, adding relevant details to the overall story: the unicorns originated in Argentina specifically; they have four horns; they not only speak English, but they have a common dialect (&#8220;or dialectic&#8221;).

It has some misses, at least from the point of view of coherency. It mentions &#8220;people who lived there before the arrival of humans&#8221;. And by the last paragraph aliens make an unexpected entrance, although honestly by then it may even add to the charm. Overall I think it works remarkably well as a short story.

Creativity is known to be linked to the ability of making mistakes and learning from them. Even if this model, or its successors, were seen simply as tools for making (somewhat informed) mistakes, they could be useful in aiding creativity. The surrealists were known for making use of mistakes when creating works of art; In writing, the &#8220;[exquisite corpse][6]&#8221; technique even reminds me a bit of Markov chains &#8212; simpler generative models that estimate the probability of some next step in an evolving system based only on the current state. If this or similar models can even just reproduce that level of creativity, they may be useful already. But they may be able to do more.

# Building on this
OK, so let&#8217;s assume we do want to use the model&#8217;s mistakes to aid creativity in some shape or form. How to do so?

A &#8220;Writing Assistant&#8221; is not a huge stretch of the imagination. You write part of a short story in an UI (or your text editor of choice, with some plugin), then you generate a few paragraphs with the model. You review them, keeping the best and ditching the rest; perhaps even rewrite them whole, just keeping some idea. Then you repeat the process next time you are stuck, or feel like the story needs a turn.

We can do better than this, though. We can point out to the Assistant where it went wrong, and where it went right. A mention of erotic asphyxiation in a story for kids? That&#8217;s a no-no, with some nope on top, please. Zombies in a period piece? [It could happen][7], but perhaps not today. The protagonist&#8217;s partner suddenly develops a single-minded passion for mastiffs? Sure, OK, show me what you got. Imagine highlighting sentences and turns of phrase and volunteering critique as feedback (pressing some buttons).

What to do with this feedback from the user? Use it to build a better model, if we can. The feedback is a label, in a sense. My guess is that we could focus on building a separate model, call it an Editor, and have it mediate the production of the Writing Assistant. That may be preferable to trying to incorporate the feedback directly into the Writing Assistant model, as simpler models are easier to understand. In this system, the Writer explores but also makes mistakes &#8212; the Editor points them out and sometimes asks for a redo. The output that reaches the user (the human) is now slightly better, and the process continues.

In ML, having great data is a bit like the Holy Grail. Everybody wants a huge amount of high quality labelled data; if you let it happen, you can end up being overoptimistic about the quality and quantity of your data, and then you run into reality and your model often doesn&#8217;t train well. In reality data is scarce in many fields, and producing the labelled datasets that you want is hard and expensive. If this Assistant was useful enough, and lots of people used it, labelling could keep pouring in. We&#8217;ve set up a virtuous circle in which humans extract value from the act of labelling.

# The future of fiction
So, now for a bit of futurology: it may very well be that this way &#8220;AI&#8221; cracks fiction before other kinds of texts, and way before general AIs happens. There&#8217;s certainly precedent to this idea proper &#8212; Lem&#8217;s amazing [Cyberiad][8] covered this (and much more) back in the 1960s. If the Assistant approach works, though, we could imagine having a path towards this. I&#8217;m skirting over lots of complexity, certainly, but if the quality of the generated results keeps improving based on users&#8217; feedback this could at least presumably and eventually happen. The generated texts could become readable over longer spans, approaching the length of longer stories and eventually novels &#8212; even as the human-provided prompts become shorter and shorter.

In the most extreme case in this line of thought, you could imagine a prompt being just a title &#8212; by which point the human user becomes a reader, not a co-writer. 
Ad-hoc fine tuning could also be worth doing once the general model is good enough that biasing towards styles is feasible. How much would a model that generates half-decent Tolkien be worth to his fans?

![Unicorn llama](https://upload.wikimedia.org/wikipedia/commons/2/29/Unicorn_llama.jpg &#8220;Amberbunting [CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0)]&#8221;)

# Futurology
Thinking about the order in which breakthroughs in AI may happen seems potentially important; it&#8217;s futurology, of course, but AI being a mathematical field it seems that speculation may be sometimes warranted. Sciences seem to advance in part by performing some amount of meta thinking, which we sometimes call philosophy. It *could* be that text generation ends up having a significant role in the advancement of our knowledge in years to come. It could be that it&#8217;s a big thing; some people believe it. It&#8217;s a stretch of the imagination surely, but let&#8217;s suspend disbelief and assume for a second that it&#8217;s a line of thought worth exploring.

You may have noticed I wrote AI between scare quotes in the previous section. I did this because ML is not AI, although it&#8217;s usually seen as a path towards AI. ML may let you build agents that perform specific tasks, mostly around prediction, whereas AI deals with the harder problem of solving general intelligence. But it&#8217;s all in a spectrum, and a set of models that collaborate to produce an output that exhibits a human quality (creativity) may fit the name even if it&#8217;d be hard to argue that the result is generally intelligent. This at least fits well within Minsky&#8217;s vision of the human mind as a [society of agents][8].

So: if writing fiction did indeed fall to this carefully scoped definition of AI, what would that tell us of the probability that general AIs is to follow &#8212; if anything at all?

One way to think about this is to imagine a subsequent breakthrough that allows a new version of the system to produce output that veers away from fiction onto non-fiction; onto writing about the real world. You could imagine the introduction of a new agent, Fact Checker, that works alongside (or after) Editor and pushes back on non-factual or at least non-checkable information. If such a system worked, it could perhaps perform some of the tasks of an entry-level journalist. A human journalist could probably help steer it in the right direction, too. The journalist would collaborate with the AI, as the human did with the first version of the Writer &#8212; as an editor, before the Editor existed. Just one level of abstraction higher. An expert.

Going in a different direction, opinion pieces could be generated in a similar way; just substituting human opinions for facts. Imagine prompting the model algorithm with a topic or high level idea you&#8217;d like to explore: &#8220;gamergate was bad&#8221;. The computer may construct a variety of arguments, some incoherent, some a possible derivation or composition of thinking around the internet. It may thus generate viable ideas that warrant being explored further: &#8220;gamergate is somehow related to the end of democracy&#8221;. Or something inane: &#8220;gamergate has to do with zombie unicorns&#8221;. The user chooses and the model explores.

At the point in which you can generate an AI that produces fact-checked or opinion-based texts, you can probably specialize said AI in different fields. Alongside the Fact Checker could work an Undergrad or Engineer, which is always trained on all the newest papers and can help produce entry-level articles about the latest developments in a field. Guided by a human doing the same job, at first. This human could be said to be an agent in this system &#8212; the Expert. With each iteration, the Expert keeps being promoted to a superior level of abstraction. Hopefully to an interesting ending.

At some point in the process assisted writing may start to feel just like assisted thinking. Perhaps with such a tool we could think of new things; more people would be freed up from relatively mundane aspects of their day jobs and daily lives, and could think about new things with the gained time and focus &#8212; or just be happier. Eventually this could result in improvements in some fields of knowledge. Like AI.


 [1]: https://blog.openai.com/better-language-models/
 [2]: https://arxiv.org/abs/1706.03762
 [3]: https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
 [4]: https://blog.openai.com/better-language-models/#sample1
 [5]: https://blog.openai.com/better-language-models/#sample6
 [6]: https://en.wikipedia.org/wiki/Exquisite_corpse <button class="pull-url autopull" value="https://en.wikipedia.org/wiki/Exquisite_corpse">pull</button>
 [7]: https://en.wikipedia.org/wiki/Pride_and_Prejudice_and_Zombies <button class="pull-url autopull" value="https://en.wikipedia.org/wiki/Pride_and_Prejudice_and_Zombies">pull</button>
 [8]: https://en.wikipedia.org/wiki/The_Cyberiad <button class="pull-url autopull" value="https://en.wikipedia.org/wiki/The_Cyberiad">pull</button>
 [9]: link://slug/minsky
 [10]: http://approximatelycorrect.com/2019/02/17/openai-trains-language-model-mass-hysteria-ensues/
-->
    <div class="subnode-footer">
        
        <span class="wikilink-marker">[[</span><a href="/2020-08-10"><span class="node-name">
        
            commit: 2020-08-10 18:42
        
        </span></a><span class="wikilink-marker">]]</span>
    </div>
</div>