# A Differentiable Anti-Anti-Aliasing Adventure

Using PyTorch to produce magical moiré patterns

#### Thursday, July 16, 2020 · 5 min read

Another magic trick. Watch these two grids overlap to reveal a secret image! Can you spot the phantom horse?

If my janky CSS animation didn’t work for you, here is another way to see it: simply shrink the top grid slowly until the horse “pops” out. It’s hard to show this effect consistently across displays/browsers because of varying pixel spacings and various browsers’ aggressive anti-aliasing tactics, but here is a screen recording I captured.

Magic trick? Not quite. Graphics folks will immediately recognize this effect
as a curious instance of the dreaded *moiré pattern,* caused by the
interference between two almost-but-not-quite-the-same grids. Researchers work
really hard to escape such “glitchy” artifacts — they are often a result of
sloppily sampling a texture in a grid, leading to jagged edges or moiré streaks
(image below from Wikipedia).

In fact, a lot of
effort has
gone into finding “good” sampling patterns that avoid moirés; for example, in
an odd crossover of sampling theory and anatomy, if you look at the
distribution of photoreceptors in a monkey’s
retina they are neither in a grid
*nor* uniformly random, but rather arranged in a *low-discrepancy* pattern to
avoid exactly this kind of artifact.

But for all the hard work gone into avoiding these patterns, you kind of have to admit that they are beautiful and mysterious. In fact “moiré” gets its name from a desirable form of silk weave that exhibits this phenomenon owing to interference from the weave (photo from Wikipedia again). Similarly, this sculpture (and a lot more art) exploits moiré patterns for their hypnotic beauty.

There are lots of real-world ways you can play with this effect, and indeed soon enough you can’t stop seeing it everywhere you go (this is the joy and frustration of being interested in visual phenomena):

- Point a camera at a checkered shirt; the camera’s pixel grid interferes with the fabric pattern (I’ve been told not to wear checkered shirts when giving talks at certain venues where everything is recorded).
- Align two kitchen strainers and gently bend them to see the patterns change.
- Watch a screen door’s shadow interfere with the screen itself, poke the screen a bit to see the patterns change.
- Watch the wire fences on either side of a pedestrian walkway above a road interfere with each other.
- Look at an office chair from behind, watch the mesh of the back interfere with the mesh of the seat. Depending on their relative curvature you can get some beautiful effects.

These patterns are seemingly random, like marbled paper. But can we control this randomness? Satisfyingly, an on-demand moiré such as the effect above is extremely simple to generate using differentiable programming in PyTorch.

What I am optimizing is a *small distortion* to the image of the grid such that
when the grid and its distortion are superimposed, the resulting image has low
L2 difference from a given target, such as the horse image. What exactly is
this “distortion”? Imagine the image printed on play-dough and deforming it by
stretching and squeezing parts. The result is a warping of the image. To
express this warping, each pixel is assigned a “motion vector” which tells it
where to go. To make sure the motion vectors are smooth and continuous between
adjacent pixels, I actually learn a heavily downsampled set of motion vectors
(even a 12-by-12 array gets good results) and then upsample it to full
resolution with bicubic interpolation.

Here is the wonderful thing: all of these operations are built in to PyTorch and differentiable, apparently to support Spatial Transformer Networks. So, a straightforward 30-line implementation of the above algorithm is all it takes to start “learning” a moiré grid perturbation. Here is a Jupyter notebook with all you need to get started.

How much computation does it take to get good results? See this animation: it converges extremely rapidly, in just 800 iterations or so of plain gradient descent (a couple of minutes on my laptop).

I’m of course not the first person to try this trick; I found references in
2001 and 2013. I think my results look significantly better than those
presented in the paper *Variational Approach to Moire Pattern Synthesis*
(Lebanon and Bruckstein 2001), even though my technique
is much simpler than their gradient-based technique. On the other hand, I think
the more mathematically-motivated results presented in *Target-Driven Moiré
Pattern Synthesis by Phase Modulation* (Tsai & Chuang
2013) are much more impressive, even though they have a
slightly different setup than mine. They also had the resources to print out
their patterns on transparent film and overlay them physically, as well as to
mess with a camera whose Bayer pattern was known! And they suggest using this
trick for cryptography. Very neat.

To understand what they do we need to understand the mechanics of moiré
(instead of leaving it all to SGD!). I’ve always been told that when two
high-frequency signals such as grids are superimposed, they can alias to form
interesting low-frequeny content (famously there is this
comic). But I’ve never been given more detail than
that high-level overview, so I looked it up myself. It’s actually quite
wonderful and principled. Here is my understanding as informed by *A
Generalized Fourier-based Method for the Analysis of 2D Moiré Envelope-forms in
Screen Superpositions* (Amidror 1994), specifically by
Figure 4 reproduced below.

First, we need to come up with a useful characterization of grids. Recall that
you can take a 2D signal (i.e. image) and perform a 2D Fourier transform, which
is *also* an image. Each point in frequency-space represents a set of stripes
(i.e. a sine wave) rotated to match the point’s angle with respect to the
origin (see (a) and (b) in the figure for examples of what I mean). The
frequency of the stripes is given by the distance from the point to the origin,
and the amplitude is given by the point’s brightness (complex-valued
brightnesses let you set phase). To get “sharp” stripes instead of a “smooth”
sine wave gradient, you have to add “harmonics” at regular multiples of that
point (recall the Fourier transform of a square wave if that helps). This is
(d) and (e) in the figure from the paper.

Next, we note that superimposing two sets of stripes is like pointwise
multiplication (if you treat zeros as “occlusion”, then multiplying by zero
always sends the result to zero as desired). Because pointwise multiplication
is convolution in frequency space, the superposition looks like a *lattice*
spanned by the harmonics in frequency space. This lattice is depicted in (f) in
the figure.

The key to the illusion is that this lattice might include points much closer to the origin than either of the harmonics that span it! Because the human visual perceptual system acts as a low-pass filter, the result is that you see a low-frequency “phantom” in the superposition. This is visible in (c) in the figure.

Tsai and Chuang use this exact insight to reverse the moiré effect analytically, with excellent results.

**Open questions:** I think there’s a lot more to do here. In my mind, the holy
grail is a T-shirt you can wear which, *only when photographed* reveals a
secret message. Another neat application would be a wire mesh rolled into a
cylinder such that as you rotated it, the moiré patterns animated into
something cool (perhaps a galloping horse, as a nod to Muybridge). There is
some precedent for this in “barrier-grid” animation (see, e.g. these vinyl
record
covers),
but the benefit of a moiré-based approach would be that the barriers need not
be as thick, allowing for finer detail.

Another direction to go in is audio moiré; that is, could we make two sounds, both above human aural range (20 kHz) such that when played together you hear a perfectly clear, audible voice?

As you might have guessed, I think naïve gradient descent should be able to get us a long way there…