Recitation 6

Prelude to vector symbolic architectures

Bohan Jiang

2026-10-09

Same words, different meaning

The cat chased the dog.
The dog chased the cat.

Same three words. Opposite meaning.

Suppose we are given three word vectors.

$\mathbf{v}_\text{cat} = (2, 1, 0, 0)$
$\mathbf{v}_\text{chased} = (0, 0, 1, 1)$
$\mathbf{v}_\text{dog} = (0, 1, 2, 0)$

The simplest sentence vector adds the word vectors.

Both sentences add up to $(2, 2, 3, 1)$ .

Question: what is the failed case?

Addition ignores order, so any three vectors give the same sum. The sum keeps which words occur but loses who did what to whom.

The words are the same, but cat is the SUBJECT of one sentence and the OBJECT of the other. These roles are the structure that the summation misses.

Attempt 1: glue the words in order

Instead of adding the word vectors, concatenate them in sentence order.

The two sentences now get different vectors. Word order is kept.

Question: does order give us the roles?

Both sentences say that Kay loves Jay. Yet block 1 holds Kay in one and Jay in the other, so compared block by block they look unrelated
The passive has 5 words and the active 3, so the two vectors do not even have the same size

Position is not role.

Side attempt: train a role labeler

Jay is loved
by Kay

→

labeler

→

SUBJECT(loved, Jay)
OBLIQUE(loved, Kay)
PASSIVE(loved)

→

one vector?

A trained role labeler, such as a dependency parser, reads a sentence, labels each word and marks the passive, so it can tell that Kay does the loving

Its output is a set of labels. Labels cannot be added, compared with a cosine, or fed into the vector tools we have built since Session 7

A labeler tells us what to encode. We still need how to encode it in a vector.

From now on the roles are given, by a labeler or by hand, just like the word vectors.

Attempt 2: one slot per role

Reserve one block of the vector for each role, and put the filler in its role’s block. Stack the blocks as rows.

Start with an empty table, one row per role.

Put each sentence’s SUBJECT in the SUBJECT row, cat on the left and dog on the right.

Both verbs are chased. Put its vector in the VERB row.

Put each sentence’s OBJECT in the OBJECT row.

The two hook sentences now get different tables
Every sentence has the same size, one row per role

Question: where does Attempt 2 break?

Jay moves from SUBJECT to OBJECT and Kay from OBLIQUE to ANSWER, while the meaning stays Kay loves Jay. Each construction needs new slots and gives a different table
How many slots do we need? SUBJECT, OBJECT, OBLIQUE, INDIRECT OBJECT, COMPLEMENT, and more. Each new role means redesigning the vector
Mary knows that the cat chased the dog. What goes in the OBJECT slot of knows?

A whole sentence, which is already a table. A slot made for one word vector cannot hold it.

Attempt 3: the slot table is an outer product

Look at the SUBJECT row alone. A slot number is a one-hot vector, and placing a word in that slot is an outer product.

$\mathbf{r}_\texttt{SUBJECT}\,\mathbf{f}_\text{cat}^\top = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix} \begin{pmatrix} 2 & 1 & 0 & 0 \end{pmatrix} = \begin{pmatrix} \color{#b22222}{2} & \color{#b22222}{1} & \color{#b22222}{0} & \color{#b22222}{0} \\ 0 & 0 & 0 & 0 \\ 0 & 0 & 0 & 0 \end{pmatrix} \begin{array}{l} \texttt{SUBJECT} \\ \texttt{VERB} \\ \texttt{OBJECT} \end{array}$

So the whole table of Attempt 2 is a sum of outer products, one per role.

Now let a role be any vector, not only a slot number. This is role-filler binding (Smolensky, 1990) $\mathbf{r} \otimes \mathbf{f} = \mathbf{r}\mathbf{f}^\top$ The filler is the word. The role vector says where it goes.

Why the outer product and not the inner product?

Every pair of positions $(i, j)$ gives a product $a_i b_j$ . That is the full grid
The inner product keeps only the diagonal and adds it into one number. Which positions agreed is lost
The outer product keeps all $n^2$ cells. Nothing is summed away
Cosine and attention are inner products, so they cannot keep a binding

The hook, answered by binding

$\mathbf{S} = \mathbf{r}_\texttt{SUBJECT} \otimes \mathbf{f}_\text{cat} + \mathbf{r}_\texttt{VERB} \otimes \mathbf{f}_\text{chased} + \mathbf{r}_\texttt{OBJECT} \otimes \mathbf{f}_\text{dog}$

Swapping the words swaps the rows. The two sentences are now different matrices
Adding up the rows gives back $(2, 2, 3, 1)$ for both, which is exactly the information the sum kept

Question. Given only $\mathbf{S}$ , how do we read the SUBJECT back out?

Reading a role back

Unbinding: ask for a role

To read the SUBJECT, multiply by the SUBJECT role $\begin{aligned} \mathbf{r}_\texttt{SUBJECT}^\top\, \mathbf{S} &= \underbrace{(\mathbf{r}_\texttt{SUBJECT}^\top\, \mathbf{r}_\texttt{SUBJECT})}_{1}\,\mathbf{f}_\text{cat}^\top + \underbrace{(\mathbf{r}_\texttt{SUBJECT}^\top\, \mathbf{r}_\texttt{VERB})}_{0}\,\mathbf{f}_\text{chased}^\top \\ &\quad + \underbrace{(\mathbf{r}_\texttt{SUBJECT}^\top\, \mathbf{r}_\texttt{OBJECT})}_{0}\,\mathbf{f}_\text{dog}^\top = \mathbf{f}_\text{cat}^\top \end{aligned}$

With one-hot roles this just reads off the SUBJECT row
Roles need not be one-hot. Any orthonormal roles work. With $\mathbf{r}_\texttt{SUBJECT} = (1, 1, 0)/\sqrt{2}$ and $\mathbf{r}_\texttt{VERB} = (1, -1, 0)/\sqrt{2}$ , the matrix rows look scrambled, yet $\mathbf{r}_\texttt{SUBJECT}^\top\, \mathbf{S} = (2, 1, 0, 0)$ is still cat
Orthonormal means each role has length 1 and is perpendicular to the others

Practice

Roles in $\mathbb{R}^2$ are $\mathbf{r}_\texttt{SUBJECT} = (1, 0)$ and $\mathbf{r}_\texttt{OBJECT} = (0, 1)$ . Fillers are $\mathbf{f}_\text{cat} = (1, 2)$ and $\mathbf{f}_\text{dog} = (3, 0)$ . Ignore the verb.

Build the $2 \times 2$ matrix $\mathbf{S}$ for dog chased cat
Compute $\mathbf{r}_\texttt{SUBJECT}^\top\, \mathbf{S}$ . Who is the SUBJECT?

$\mathbf{S} = \mathbf{r}_\texttt{SUBJECT}\mathbf{f}_\text{dog}^\top + \mathbf{r}_\texttt{OBJECT}\mathbf{f}_\text{cat}^\top = \begin{pmatrix} 3 & 0 \\ 0 & 0 \end{pmatrix} + \begin{pmatrix} 0 & 0 \\ 1 & 2 \end{pmatrix} = \begin{pmatrix} 3 & 0 \\ 1 & 2 \end{pmatrix}$
$(1, 0)\,\mathbf{S} = (3, 0) = \mathbf{f}_\text{dog}$ . The SUBJECT is dog

Question: roles that are not perpendicular

Suppose $\mathbf{r}_\texttt{SUBJECT} = (1, 0, 0)$ but $\mathbf{r}_\texttt{OBJECT} = (0.6, 0.8, 0)$ , so the two roles have cosine $0.6$
What do we get back when we ask for the SUBJECT of cat chased dog?

$\begin{aligned} \mathbf{r}_\texttt{SUBJECT}^\top\, \mathbf{S} &= \mathbf{f}_\text{cat}^\top + 0.6\,\mathbf{f}_\text{dog}^\top \\ &= (2, 1, 0, 0) + 0.6\,(0, 1, 2, 0) = (2, 1.6, 1.2, 0) \end{aligned}$

A blend of cat and dog. The OBJECT leaks into the SUBJECT
The leak is exactly the cosine between the roles

Question. A language has many roles and many words. Where can we get many vectors that are nearly perpendicular to each other?

Random vectors as symbols

Random arrows are almost perpendicular

Try the laziest answer first. Pick the vectors at random.

In 2-D, two random arrows can point anywhere relative to each other
In 1000-D, almost every pair lands near 90°, so their cosine is near 0
Near-perpendicular random vectors can stand for different roles without anyone designing them

Random vectors work as roles. What about the fillers?

What about the fillers?

So far cat, chased and dog were toy vectors we were given
Real fillers must carry meaning. Coffee should sit closer to tea than to laptop
They must also live in the same high-dimensional space as the random roles

Question. Can we build meaningful word vectors out of random vectors too?

The random indexing algorithm

Random indexing (Kanerva and Sahlgren, late 1990s) does exactly that.

Give every context a random sparse index vector of length $d$ , mostly zeros with a few $\pm 1$
Give every word a zero vector of length $d$
Walk the corpus. Each time word $w$ appears with context $c$ , add $c$ ’s index vector to $w$ ’s vector

context	cup	brew	morning	screen	battery
index vector	$(1, 1, 0, 0)$	$(0, 1, 0, 1)$	$(1, 0, -1, 0)$	$(0, 0, 1, 1)$	$(0, -1, 1, 0)$

Five contexts get squeezed into 4 numbers. In real use the vectors have $d \approx 1000$ or more and thousands of contexts.

Walking a tiny corpus

Follow coffee as the corpus streams by.

event	add	coffee’s vector
start		$(0, 0, 0, 0)$
coffee, cup	$(1, 1, 0, 0)$	$(1, 1, 0, 0)$
coffee, cup	$(1, 1, 0, 0)$	$(2, 2, 0, 0)$
coffee, brew	$(0, 1, 0, 1)$	$(2, 3, 0, 1)$
coffee, morning	$(1, 0, -1, 0)$	$\mathbf{(3, 3, -1, 1)}$

We never wrote down a count. Each event is one addition
Do the same for tea (cup once, brew twice) and laptop (screen twice, battery once)

The count matrix $P$

Write the tiny corpus as a table. A row is a word, a column is a context, and a cell counts how often the word appeared with that context.

$P \;=\; \begin{array}{r|ccccc} & \text{cup} & \text{brew} & \text{morning} & \text{screen} & \text{battery} \\ \hline \color{#b22222}{\text{coffee}} & \color{#b22222}{2} & \color{#b22222}{1} & \color{#b22222}{1} & \color{#b22222}{0} & \color{#b22222}{0} \\ \text{tea} & 1 & 2 & 0 & 0 & 0 \\ \text{laptop} & 0 & 0 & 0 & 2 & 1 \end{array}$

This is the count matrix from Session 7
The coffee row $(2, 1, 1, 0, 0)$ is the walk on the previous slide. Cup twice, brew once, morning once
Tea and laptop are the two walks you just did

Index matrix times count matrix

The stream computes exactly count matrix $P$ times index matrix $R$ .

$\footnotesize \underbrace{\begin{array}{r|ccccc} & \text{cup} & \text{brew} & \text{morn.} & \text{screen} & \text{batt.} \\ \hline \color{#b22222}{\text{coffee}} & \color{#b22222}{2} & \color{#b22222}{1} & \color{#b22222}{1} & \color{#b22222}{0} & \color{#b22222}{0} \\ \text{tea} & 1 & 2 & 0 & 0 & 0 \\ \text{laptop} & 0 & 0 & 0 & 2 & 1 \end{array}}_{P \;(3 \times 5)} \;\times\; \underbrace{\begin{pmatrix} 1 & 1 & 0 & 0 \\ 0 & 1 & 0 & 1 \\ 1 & 0 & -1 & 0 \\ 0 & 0 & 1 & 1 \\ 0 & -1 & 1 & 0 \end{pmatrix}}_{R \;(5 \times 4)}$

$\footnotesize =\; \underbrace{\begin{array}{r|cccc} \color{#b22222}{\text{coffee}} & \color{#b22222}{3} & \color{#b22222}{3} & \color{#b22222}{-1} & \color{#b22222}{1} \\ \text{tea} & 1 & 3 & 0 & 2 \\ \text{laptop} & 0 & -1 & 3 & 2 \end{array}}_{PR \;(3 \times 4)}$

Each row of $R$ is a context’s index vector, and $P$ says how many times to add it. That gives coffee’s $(3, 3, -1, 1)$
$P$ itself is never stored. Only the small $PR$ is

Did the cosines survive?

pair	cosine of counts ( $5$ numbers)	cosine after random indexing ( $d = 4$ )
coffee, tea	0.73	0.84
coffee, laptop	0.00	-0.24
tea, laptop	0.00	0.07

The ranking survives. Coffee is still much closer to tea than to laptop
But every cosine moved. With $d = 4$ the index vectors are far from perpendicular (cup and brew have cosine $0.5$ )
How large must $d$ be before the movement becomes small?

Johnson–Lindenstrauss in a picture

Each dot is one pair of words. On the red line, the cosine is kept exactly
Johnson–Lindenstrauss (1984) says $n$ points need only about $\log n / \epsilon^2$ random dimensions to keep all distances within a factor $1 \pm \epsilon$
The bound depends on the number of words, not on how many contexts we started with

Roles and fillers are ready

Roles are random vectors. Words are random indexing vectors. Both need around a thousand dimensions to behave like distinct symbols
One binding $\mathbf{r}\mathbf{f}^\top$ is a $d \times d$ table, so $d = 1000$ gives $10^6$ numbers
And Attempt 2 left nesting open. What happens when a sentence sits inside a sentence?

A sentence inside a sentence

Mary knows [that the cat chased the dog]

The inner sentence is a sum of bindings, a $d \times d$ table $\mathbf{S}_\text{inner}$
That table is the OBJECT of knows, so $\mathbf{r}_\texttt{OBJECT} \otimes \mathbf{S}_\text{inner}$ is a cube with $d^3$ numbers
Mary and knows still give $d \times d$ tables, so the outer bindings no longer have the same shape and cannot be added

Each level of embedding multiplies the size by $d$ , so two levels deep is $10^{12}$ . Question. Can we keep the binding but stop squaring the size?

Folding the binding back down

Convolution: multiply every pair, add along the diagonals

Read $\mathbf{a} = (1, 2, 0)$ as $1 + 2x$ and $\mathbf{b} = (1, 0, 1)$ as $1 + x^2$ , then multiply term by term.

Cell $(i, j)$ is $a_i \times b_j$ and lands on the power term $x^{i+j}$ . Same power, same color
Adding each color gives $\mathbf{a} * \mathbf{b} = (1, 2, 1, 2, 0)$
These are the coefficients of the product $1 + 2x + x^2 + 2x^3 + 0x^4$
3 and 3 numbers in, 5 out. Ordinary convolution makes the vector longer

Question. A word has 3 numbers. What can we do to get a vector of length 3?

Solution: Group slots

Keep only 3 slots, and count the powers like hours on a 3-hour clock: 0, 1, 2, then back to 0. So $x^3$ joins slot 0 and $x^4$ joins slot 1.

The same nine products, grouped by slot. It folds last slide’s $(1, 2, 1, 2, 0)$ onto 3 slots, $(1 + 2,\ 2 + 0,\ 1) = (3, 2, 1)$
3 numbers in, 3 out, the same size as a word

Circular convolution

HRR (Plate, 1995) binds with circular convolution. The result has $n$ slots, and $k$ names one of them. $\underbrace{(\mathbf{a} \circledast \mathbf{b})_k}_{\text{slot } k \text{ of the result}} = \sum_{i=0}^{n-1} \underbrace{a_i}_{\text{row } i}\; \underbrace{b_{(k - i) \bmod n}}_{\text{the column that lands in slot } k}$

Each cell shows its slot ${k = (i + j) \bmod 6}$ , the same clock as before with 6 hours
Slot 2 takes one cell per row, in columns 2, 1, 0, then wraps to 5, 4, 3
All 36 products are used, 6 per slot, and 6 numbers come out

Why “circular”?

Put the 6 columns on a clock. Slot 2 walks 2, 1, 0, and one more step back lands on 5, not on $-1$
The subscript $(k - i) \bmod n$ is that clock. Row 3 of slot 2 uses column $(2 - 3) \bmod 6 = 5$
Nothing falls off the edge, so every slot gets exactly $n$ products and the result keeps $n$ numbers

Binding as a walk around the ring

Put $\mathbf{b} = (1, 0, 1)$ on a clock with 3 slots. Each entry $a_i$ of $\mathbf{a} = (1, 2, 0)$ says turn $\mathbf{b}$ clockwise by $i$ slots and scale it by $a_i$ .

Adding the rotated copies gives $\mathbf{a} \circledast \mathbf{b} = (1, 0, 1) + (2, 2, 0) + (0, 0, 0) = (3, 2, 1)$
So the binding is a walk of $\mathbf{b}$ around the ring, weighted by $\mathbf{a}$ . The result stays on a ring of $n$ slots

The hook, answered by HRR

$\mathbf{S} = \mathbf{r}_\texttt{SUBJECT} \circledast \mathbf{f}_\text{cat} + \mathbf{r}_\texttt{VERB} \circledast \mathbf{f}_\text{chased} + \mathbf{r}_\texttt{OBJECT} \circledast \mathbf{f}_\text{dog}$

Every vector has $d = 1024$ random entries, roles included. $\mathbf{S}$ also has just 1024 numbers
The two sentences are now different vectors. They share only the verb binding, about a third of each

Undoing one turn

Binding with $\mathbf{r}$ turned the filler $\mathbf{f}$ forward. Unbinding must turn it back.

$\mathbf{r} = (0, 1, 0)$ is one turn by 1. Turning back 1 is the same as turning by 2 on 3 slots, so ${\mathbf{r}^* = (0, 0, 1)}$
The rule moves the weight on turn $i$ to turn $-i$ . This $\mathbf{r}^*$ is the involution of $\mathbf{r}$

Undoing a mix of turns

Now $\mathbf{r} = (1, 2, 0)$ and $\mathbf{r}^* = (1, 0, 2)$ . Pair up their turns.

from $\mathbf{r}$	then from $\mathbf{r}^*$	weight	net turn
no turn (1)	no turn (1)	1 × 1 = 1	0
turn 1 (2)	back 1 (2)	2 × 2 = 4	0
turn 1 (2)	no turn (1)	2 × 1 = 2	1
no turn (1)	back 1 (2)	1 × 2 = 2	2

Group by net turn. $\mathbf{r}^* \circledast \mathbf{r} = (5, 2, 2)$ is one combined recipe. A perfect undo would be $(1, 0, 0)$
With $\mathbf{f} = (2, 1, 0)$ we get $(12, 9, 6) = 5\,\mathbf{f} + 2\,(\mathbf{f} \text{ turned } 1) + 2\,(\mathbf{f} \text{ turned } 2)$ . Mostly $\mathbf{f}$ , plus blur
Question. Matched pairs add squares. What about the rest when $d$ is large?

Back 1 is a turn by 2 on 3 slots, which is why the last row lands on net turn 2. The bold rows are the matched pairs, a turn and its own reverse. They always add squares, $r_0^2 + r_1^2 + \cdots$ , so they never cancel. The cosine of $(12, 9, 6)$ with $\mathbf{f}$ is 0.91 and with each turned copy 0.66, so with 3 slots the blur is large. Answer to the question, shown on the next slide: with $d$ random numbers each leftover slot gets small, about $1/\sqrt{d}$ , but there are $d - 1$ of them. Together they weigh about as much as the matched part, so the blur does not fade as $d$ grows. Since $\mathbf{r}^* \circledast (\mathbf{r} \circledast \mathbf{f}) = (\mathbf{r}^* \circledast \mathbf{r}) \circledast \mathbf{f}$ , the combined recipe can be applied to $\mathbf{f}$ in one go. Binding with $\mathbf{r}^*$ is circular correlation (8d p. 27, exercise 2).

Unbinding gives a noisy copy

To ask for the SUBJECT, bind the sentence with $\mathbf{r}^*_\texttt{SUBJECT}$ .

Each leftover is small, but there are $d - 1$ of them.
The blur points away from every word, so comparing it with every word we know (a clean-up memory) still picks out cat

How blurred, and how to fix it

Cosine between the recovered filler and the right one, for a single bound pair.

	$d = 64$	$d = 1024$	$d = 4096$
random role vector	0.732	0.709	0.708
unitary role vector	1.000	1.000	1.000

Random roles are stuck near $1/\sqrt{2} \approx 0.707$ for every $d$
Unitary roles leave no leftovers
Question. One binding now comes back exactly, but a sentence is a sum of several bindings into one vector. What happens to each one as we add more?

8d pp. 19 and 20. Why $1/\sqrt{2}$ . The recovered vector is $\mathbf{f}$ plus turned copies whose total length matches $\mathbf{f}$ , and the turned copies are nearly perpendicular to $\mathbf{f}$ , so the cosine is $1/\sqrt{1 + 1}$ . A random role turns and also stretches some directions and squashes others. $\mathbf{r}^*$ undoes the turning but stretches the same directions again, which is the blur. Net turn 2 for $(2, 2, -1)/3$ is also 0. A single turn like $(0, 1, 0)$ is unitary too. Picture for unitary, binding turns the whole space like a globe and stretches nothing. To build a unitary role, take a random vector, set every Fourier magnitude to 1 and keep the phases. Technical reason: unbinding multiplies by $|F(\mathbf{a})|^2$ in the Fourier domain. For Gaussian $\mathbf{a}$ this is exponential with sd equal to its mean, so the cosine tends to $1/\sqrt{2}$ . Unitary means $|F(\mathbf{a})| = 1$ in every frequency bin.

Superposition and the noise floor

With $k$ bindings in one vector, each comes back with cosine $1/\sqrt{k}$ , whatever $d$ is
The noise floor is how high a wrong word appears by chance
A binding can be read back while its signal stays 5 times above the floor. Each dot marks where that stops

Adding bindings lowers the signal, adding dimensions lowers the noise, and capacity is where they meet.

8d pp. 20 to 22. Why $1/\sqrt{k}$ . The $k$ bindings have length 1 and are nearly perpendicular, so by Pythagoras the sum has length $\sqrt{k}$ , and the right filler lines up with only one of them. Lecture check at $d = 1024$ gives 0.576 for 3 items and 0.322 for 10. Why $0.8/\sqrt{d}$ . This is the spread on the slide “Random arrows are almost perpendicular”. A cosine adds $d$ products with random signs, like $d$ coin flips, whose margin is about $\sqrt{d}$ , a fraction $1/\sqrt{d}$ of all flips. The 0.8 is $\sqrt{2/\pi}$ . The dashed lines use the lecture’s measured noise floors (0.1068, 0.0236, 0.0067) times 5, the lecture’s SNR > 5. If the signal sinks to the floor, clean-up can pick a wrong word that happened to score high. The crossing is at about $k = d/16$ , which is the capacity table on the next slide.

Question: how many bindings fit in one vector?

$d$	64	256	1024	4096	16384
bindings retrievable	3	14	71	228	880

Capacity grows linearly in $d$ . Multiply $d$ by 16 and you hold about 12 times more
This is why VSA is also called hyperdimensional computing, with $d \approx 10{,}000$
Distributional vectors use $d \approx 300$ , chosen so classifiers work. VSA dimensions are chosen so symbols do not collide

Practice

Use the clock picture with $n = 3$ . Let $\mathbf{a} = (1, 0, 2)$ and $\mathbf{b} = (0, 1, 1)$ .

Fill in the $3 \times 3$ grid of products $a_i b_j$
Add up the cells with the same $(i + j) \bmod 3$ to get $\mathbf{a} \circledast \mathbf{b}$

Rows are $(0, 1, 1)$ , $(0, 0, 0)$ and $(0, 2, 2)$
Slot 0 gets $0 + 0 + 2 = 2$ , slot 1 gets $1 + 0 + 2 = 3$ , slot 2 gets $1 + 0 + 0 = 1$ . So $\mathbf{a} \circledast \mathbf{b} = (2, 3, 1)$
Check with polynomials. $(1 + 2x^2)(x + x^2) = x + x^2 + 2x^3 + 2x^4$ , and $x^3 = 1$ turns it into $2 + 3x + x^2$

Discussion

Back to the hook

Can the representation tell cat chased dog from dog chased cat?

	operation	tells them apart?	size	unbinding
adding words	$\mathbf{a} + \mathbf{b}$	no	$n$	none
gluing in order	$(\mathbf{a}, \mathbf{b})$	by position only	grows with the sentence	by position
inner product	$\mathbf{a} \cdot \mathbf{b}$	no	1	none
TPR	$\mathbf{a}\mathbf{b}^\top$	yes	$n^2$ and growing	exact
HRR	$\mathbf{a} \circledast \mathbf{b}$	yes	$n$	approximate

TPR pays in size. HRR pays in exactness
Both pay nothing for the roles themselves, since random high-dimensional vectors are already nearly perpendicular

The choice

	reduces by	compares or composes with
Distributional semantics	finding structure (SVD, SGNS, GloVe)	the inner product
VSA	random projection, which finds nothing	a folded outer product

Distributional semantics gets similarity for free but cannot compose
VSA composes but has no graded similarity beyond what you put in

Recitation 6

Same words, different meaning

Question: what is the failed case?

Attempt 1: glue the words in order

Question: does order give us the roles?

Side attempt: train a role labeler

Attempt 2: one slot per role

Question: where does Attempt 2 break?

Attempt 3: the slot table is an outer product

Why the outer product and not the inner product?

The hook, answered by binding

Reading a role back

Unbinding: ask for a role

Practice

Question: roles that are not perpendicular

Random vectors as symbols

Random arrows are almost perpendicular

What about the fillers?

The random indexing algorithm

Walking a tiny corpus

The count matrix PPP

Index matrix times count matrix

Did the cosines survive?

Johnson–Lindenstrauss in a picture

Roles and fillers are ready

A sentence inside a sentence

Folding the binding back down

Convolution: multiply every pair, add along the diagonals

Solution: Group slots

Circular convolution

Why “circular”?

Binding as a walk around the ring

The hook, answered by HRR

Undoing one turn

Undoing a mix of turns

Unbinding gives a noisy copy

How blurred, and how to fix it

Superposition and the noise floor

Question: how many bindings fit in one vector?

Practice

Discussion

Back to the hook

The choice

The count matrix $P$