Paul Chiusano's blog

Learning without a gradient (part 2)

Mon, 12 Oct 2020 00:00:00 +0000

Part 1 of this series discussed the difficulties faced by gradient-based approaches to learning:

Learning using gradient-based methods is a lot like trying to find the top of Mount Everest starting from a random point on earth, while blindfolded, by repeatedly sampling only the altitude and slope of the ground you’re standing on, and then using that information to decide where to teleport next… the gradient just isn’t a lot to go on. The local slope at a point in Kansas might help you reach the top of the hill near where you started, but it doesn’t help in locating even the general region where Mount Everest is located.

This fundamental issue of misleading local gradient information arises in higher-dimensional problems too, even though local minima are a lot less common here.

Let’s now look at a class of learning algorithms that don’t need gradient information. Gradient-free learning methods are a big deal: they are more widely applicable (many functions of interest are not differentiable, including hyperparameter search for other learning frameworks), they can in theory be used for training recurrent neural nets where backpropagation is problematic and even in cases where differentiability is possible in principle, learning without a gradient means no fancy frameworks just to build up functions in differentiable form. Instead, you can write the functions being optimized using ordinary code in your favorite programming language, then optimize those functions as a black box.

It might seem like we don’t have much to go on if we aren’t using gradient information… are we reduced to just random search? Or inefficiently estimating the gradient by sampling the output of “nearby” points, as in Evolution Strategies? No, we actually have a lot to go on. Again to build intuition we’ll stick with our original problem of finding the top of Mt. Everest, given only the ability to sample the altitude at any point on the globe.

Algorithm 1

Rather than relying on local gradient information, this is the first algorithms that uses only statistical properties of regions of the search space:

Divide the current search space in half (say, northern and southern hemisphere). Sample random points from both halves, building up a sampled distribution of altitudes for each.
When a statistical test (like perhaps this one) shows with 99% confidence that (say) the northern hemisphere has higher average altitude, go to step 1 but with the northern hemisphere as the current search space.
Repeat, dividing the current search space in half each time, until converging on an arbitrarily small region of space.

That’s it! The current search space gets divided in half each time, and the search proceeds in the half with higher estimated average altitude. It’s analogous to binary search, but using sampling to estimate which half contains the value we’re looking for. Notice this algorithm is not really sensitive to local noisy features of the search space (it won’t stop at the top of a hill in Kansas, for instance), until it reaches the end of the search and is confined to a much smaller region. This property is something that gradient-based search will sometimes fake by altering the step size over time.

One can imagine this algorithm converging to a point which isn’t the global optimum, perhaps finding the top of K2 rather than Mt Everest, but we aren’t going to worry about that: no algorithm can find a needle in a haystack, so being able to converge to any reasonable solution is a win.

The efficiency of this algorithm depends at each stage on how similar the distributions are of the two halves being compared. The more similar the distributions are in reality, the more samples we need to gather in order for our statistical test to determine with confidence which is better.

Armed with this information, let’s build a better algorithm.

Algorithm 2

A problem with Algorithm 1 is it makes an arbitrary choice of how to divide the current search space into two subspaces, and this choice significantly affects performance. Any time important decisions like this are made arbitrarily, that’s a red flag for a learning algorithm. Instead, let’s explore multiple choices in parallel:

For the current region, pick N random hyperplanes dividing the region roughly in half.
Begin sampling points in the current region. If any of the N subdivisions passes the statistical test where one half is better than the other, make that subregion the current region and go to step 1.
Repeat as before.

This is better. Rather than randomly picking one (possibly terrible) way of cutting the space in half, we try multiple ways at the same time and can advance the search as soon as any gives an answer.

Still, we are picking these N hyperplanes randomly. And how did we come up with the number N anyway? Can we do better?

Algorithm 3

Rather than picking N random ways of cutting the space in half, could we just pick the perfect way after gathering enough samples and a pattern reveals itself? Let’s just start gathering samples, and store them and their results in a fancy data structure that will allow us to stop as soon as any way of splitting the space gives statistically different results for the two sub-regions. (This reminds me a bit of support-vector machines)

Seems like a tall order, but if we represent the input to our functions as bit vectors then it becomes doable. In our Mt. Everest finding example, the function being optimized takes 2 floating point numbers (lattitude and longitude) so the bit vector equivalent would be the 128 bits corresponding to those two 64 bit numbers put back to back.

Here’s the algorithm:

Begin sampling points from the current search region, placing them into a set ordered by their output on the function being optimized. After each sample, examine the top 50% of the samples - call these the “winners”. If among the winners, any bit position is mostly 1s, we fix that position to be 1 for subsequent sampling. Likewise if any position is mostly 0s. To be more precise: we will only lock in a bit if there’s a statistically significant difference in the number of 1s vs 0s for that position.
By locking in bits, this constrains the search to a sub-region of the original space. We’re back to step 1, but with a smaller search space.

Aside: I suspect you can use a finger tree with an interesting monoid to make it logarithmic time to determining if there’s a bit position that can be fixed.

So this algorithm learns one bit at a time of an answer, in whatever order is easiest, eventually converging to having all the bits fixed. It uses only the statistics of different regions of the space. Interestingly, the order of the bits is irrelevant: if we reversed the 128 bits of our input or interleaved the two 64 bit numbers everything works the same.

For our running example of finding the top of Mt Everest, it will learn the more significant bits, which roughly pin down regions with higher than average elevation, then learn less significant bits, and so on, until narrowing the region all the way down to a single point.

Algorithm 4

We can simplify Algorithm 3. No fancy data structures needed.

Start with a collection of P bit vectors. (“P” for population size) To insert a bit vector, b, into the population, add it to the population, then evict the bit vector with the worst fitness (might be ‘b’ itself).
At each step, generate a new bit vector by picking a bit at each index in proportion to the current distribution of bits for that index in the population. For instance, if the first index is 80% 1s in the population, you select 1 for that index with 80% probability (you can make it more aggressive with a nonlinear function here). Then insert that bit vector into the population as in 1, possibly displacing another member of the population or just evicting itself if its fitness is worse than everything already in the population.
That’s it, just repeat for as many steps as you like.

This algorithm might not seem all that similar to Algorithm 3, but let’s imagine applying it to our Mt. Everest example. Initially, the algorithm just picks points randomly. After the number of samples exceeds the population count (say, P=1000), it begins discarding samples with lower elevation. Among the samples retained, they may have some bit positions in common. Suppose it’s the bits that indicate “northern hemisphere”. As a result of this pattern, the algorithm does more sampling in the northern hemisphere, which will (perhaps) find more high elevation points in the northern hemisphere, which further increases the weight of those bits. Eventually, those bits are effectively locked in and search proceeds on other remaining unfixed bits.

Surprise!

The algorithms discussed here use the same principles as the genetic algorithm (GA), which are believed to work via this process of progressively locking in bits of the solution. My old labmate, Keki Burjorjee, dubbed this “hyperclimbing”. See this paper by him. (Also, thanks Keki for your neat research and some fun discussion and pointers that shaped this post!)

Also see:

Population-based incremental learning which is very similar to my Algorithm 4 above.
The compact genetic algorithm, which doesn’t require keeping a full population of bit vectors in memory, only the % of 1 (or 0) bits at each bit position. This makes it practical even for large models.

In the compact GA, only the % of 1 (or 0) bits at each position is retained (this is called the “probability vector”). The training looks like:

Generate two models, s1 and s2, given the current probability vector.
Compare s1 and s2 against the objective function to determine a winner. Move the probabilities some small step toward the winning vector.
Repeat.

One further optimization not mentioned in the compact GA paper is that for supervised learning where the objective function is an average of performance over a large training set, comparing two models can be done more efficiently: rather than running each model against the whole training set, instead feed the training set to each model incrementally, stopping as soon as one model is statistically better than the other. And in general, there’s more you can do to be sure you make the most of each evaluation–you don’t want to spend a lot of evaluations on comparing two models which are quite similar in performance, when it’s very easy to find two models in the remaining search space that are quite different in performance.

Discussion

Could these gradient-free learning algorithms be used for (recurrent) neural network training and other forms of supervised learning? It’s quite nice that they don’t require any fancy frameworks or differentiability, just a way of representing the function’s parameters as a bit vector, which isn’t exactly hard.

While there’s been some use of GA-style algorithms for supervised learning (though it’s typically for hyperparameter search), the field seems pretty centralized: a few organizations with access to massive compute resources have picked a standard set of techniques and frameworks, and they get the most impressive results. We may not really know what techniques work best, but because some orgs can throw 100x more compute at the problem using their preferred techniques, they get better results, and the techniques they use thus attract more mindshare.

Next time we will look at when GA-style search does well and consider a new class of adaptive learning algorithm which figures out how to make a problem learnable by GA-style search.

No you should not seriously consider whether we are living in a simulation

Mon, 16 Dec 2019 00:00:00 +0000

I’m reading Superintelligence, a surprisingly dull book given its topic is, well, the possibility of a rampant superintelligent AI destroying civilization and what we might do to prevent that sort of thing. Along the way, the author speculates about how a caged hyperintelligent AI (not joking here) might incorporate reasoning about the the simulation hypothesis into its plans to escape confinement.

The answer to the question of “how do we know we aren’t living in a simulation” is similar to “how do you know the universe wasn’t snapped into existence 5000 years ago?”, “how do you know you aren’t dreaming right now?” or “how do you know you aren’t just a brain in a vat?” On some level, we can’t really know these things with certainty, but we don’t seriously consider these possibilities because we have simpler models that explain reality as well or better.

However, without any judgement, I report that some people take this idea of reality being a simulation pretty seriously. Elon Musk apparently believes we are likely living in a simulation and is rumored to have a team of people trying to figure out a way of breaking out of the simulation. I’ll admit: the first time I read that second article I had to double check that I was not reading The Onion.

Anyway, I want to spend the rest of this post picking apart the simulation hypothesis, but first, let’s hear the argument out! Enter Nick Bostrom:

Many works of science fiction as well as some forecasts by serious technologists and futurologists predict that enormous amounts of computing power will be available in the future. Let us suppose for a moment that these predictions are correct. One thing that later generations might do with their super-powerful computers is run detailed simulations of their forebears or of people like their forebears. Because their computers would be so powerful, they could run a great many such simulations. Suppose that these simulated people are conscious (as they would be if the simulations were sufficiently fine-grained and if a certain quite widely accepted position in the philosophy of mind is correct). Then it could be the case that the vast majority of minds like ours do not belong to the original race but rather to people simulated by the advanced descendants of an original race.

It is then possible to argue that, if this were the case, we would be rational to think that we are likely among the simulated minds rather than among the original biological ones.

Therefore, if we don’t think that we are currently living in a computer simulation, we are not entitled to believe that we will have descendants who will run lots of such simulations of their forebears.

Whoa there! First problem: one can’t just extrapolate from a few decades of computing trends (where computers have indeed become exponentially more efficient) arbitrarily far into the future, without considering that there are theoretical, physical, and practical limits to computing power, even for an advanced civilization. To put things in perspective, we cannot today build computers powerful enough to accurately simulate everything happening in a thimble full of blood; what makes us think we can simulate a planet, a solar system, or a universe?

Something underappreciated here is that the computation happening in the wild, including the computation done by our universe’s laws of physics, tends to be irreducible if we simulate it at sufficient fidelity. (And by all available evidence, our reality is “computed” at high fidelity down to quantum scales, not some vastly simplified approximation like what is done in video games.) It’s only when we consider much simpler idealizations of reality that we’re able to simulate or predict reality using more limited amounts of compute. Stephen Wolfram been beating this drum for a while, though I like to just view it as a consequence of Rice’s theorem or the halting problem: in general, you can’t fast forward the program to know what it does, you have to run it, and that requires energy, space, and time and eventually reaches physical limits. Though much is unknown, it is very possible that our universe cannot be simulated much more efficiently than by building an effectively universe-sized computer. This does not bode well for hypothetical future civilizations planning to simulate our current reality within our current universe.

A more fundamental issue with the simulation hypothesis is it requires speculating about an “outer universe” capable of simulating our current reality. Notice we can’t really speculate about this because nothing we can observe in our current reality places any constraints on such an outer universe. Is it similar size to our own? With similar computing power? Do its laws of physics work similarly? Anything goes here and is consistent with our observations of reality. So we can conjure up the idea that there is an outer reality but can’t actually investigate or test it.

A related objection is we have no idea what sort of activities civilizations in a hypothetical outer universe would engage in, so even if computers powerful enough to simulate our current universe are possible in some outer universe, will creating simulations of our currrent reality be something that is done?

Together, these points establish the simulation hypothesis as having roughly the same logical status as debating how many angels can fit on the head of a pin or what would happen if Superman and Cyclops fired their laser eyes at each other: it is speculation about the properties of things that can’t be disproven or shown to exist.

I find it odd that these presumed creators of the simulation we find ourselves in have left no trace that it is in fact a simulation and everything we observe seems indicative that reality is what it is. Give us a sign, dammit! The arguments one sometimes hears for this lack of evidence (“only when we reach a certain stage of technical advancement will we be able to detect the simulation”) remind me of other unfalsifiable explanations that conspiracy theorists invent: one can always make up reasons for why the thing you believe doesn’t have any observable evidence (or why observations contradict your belief).

But forget all that. We have no need to resort to outer universes simulating our current universe to explain reality, so why introduce that? Let’s all just stop there and get back to (what I sure hope is) reality.

Email display names do not need to be LASTNAME, FIRSTNAME now that computers are a thing

Wed, 11 Dec 2019 00:00:00 +0000

I received an email recently from a guy, let’s call him Robert Smith (name changed to protect the innocent), whose email display name was "Smith, Robert" <robert.smith@somedomain.com>.

As far as I can tell, the LASTNAME, FIRSTNAME convention is a holdover from the pre-computer days where you needed to have the last name at the start for easy visual scanning when searching some directory (like a phone book) by last name. Now that we have computers that can search any which way, there’s no reason to put up with the (weird to read) LASTNAME, FIRSTNAME convention. If you continue to use it, it signals you aren’t really in the loop on all this new-fangled “computer stuff” and is nearly as bad as continued use of an aol or hotmail email address or using terms like “cyberspace” or “information superhighway” non-ironically. 😀

I’m half joking and this might seem like a silly thing, but it’s a little example of this other general phenomenon: whenever new technology is introduced, people sometimes carry forward old ways of working that no longer make sense. We forget to ask ourselves why things were done a certain way and if we should maybe do things differently now.

Question everything!

Is tech adoption really that random? A VHS vs Betamax redux!

Thu, 24 Oct 2019 00:00:00 +0000

I worry a bit that people are adopting an overly fatalist perspective on tech adoption. This perspective has become almost a reflex–we can’t possibly know or predict what technology will end up being widely adopted and often the “worse” technology wins, so just don’t even try!

I acknowledge there is some underlying randomness at work here but it would be a shame if it were totally random, right? Why would you bother trying to create any new technology if its fate will be decided by coin flip? But if it’s not totally random then there must be some underlying principles, so what are they? I’ve started looking at a few examples of this (and would love to hear others in the comments) and I hypothesize a simple rule: if a technology is at least an order of magnitude better for some key factor(s) evaluated by market participants, and not substantially worse for other factors, it either finds a niche or takes over the market. And the corollary of this claim is that if a technology isn’t substantially better for key factors, there’s a certain degree of randomness and path dependence that determines what technologies get adopted.

Let’s look at some examples.

VHS vs Betamax

The way this is usually presented is that Betamax was the better technology (it had higher picture quality, supposedly), but VHS won out, because ‘worse is better’. Geez, I’m disappointed that people are so readily able to accept such a shallow explanation. Implicit assumption: the adoption game is totally random in who it rewards with victory, and maybe it even actively rewards bad quality! But let’s look closer:

The picture quality wasn’t better, at least in any obvious way, here’s a quote from Why VHS was better than Betamax: “Standing in a shop at the time, there was absolutely no visible difference in picture quality, and some reviews had found that VHS’s quality was superior.”
Betamax was more expensive, at least at first
And the kicker: VHS had a 2 hour recording time, long enough to record a movie, while Betamax opted for a smaller, sleeker form-factor that could only record 1 hour!
It’s very much a ‘winner take all’ market due to positive feedback loops (e.g. the more people who own VHS players, the more likely video rental stores are to stock VHS, which leads to more people wanting to own VHS), so the difficulty of catching up increases with time.

There are always multiple factors that go into comparing the utility of products, and different buyers in a market will weight those factors differently, which is why markets can have different niches. In the format wars, we have price, recording time, picture quality, and availability of content in that format, among others. But both VHS and Betamax were in the same ballpark on all these key factors, so they faced direct competition in a market with network effects and VHS had the edge in the factors that mattered to most people. But, as a thought experiment, I hypothesize:

If Betamax picture quality had been miraculously 100x, equivalent to a modern 4k TV, it would have found a niche, even if it were more expensive than VHS and had the same limited recording time. There is always a segment of the market that will pay a premium for obvious quality for some key factors. And most likely, the obvious picture quality would spur huge investment in compensating for the other factors that VHS had an edge in.
If Betamax had a 5 year head start, it might have reached the tipping point where VHS recording time superiority wasn’t enough. (More likely, Betamax could have developed a longer recording time format before VHS had a chance to catch up.)

Consistent with my hypothesis: the market later moved on to DVDs (superior picture and sound quality, no degradation over time, and random access / interactivity), at around the time where these improvements in key factors were sufficient to overcome the inertia of VHS.

Other format wars have worked similarly—formats roughly equivalent in key factors were introduced around the same time, and faced direct competition in a winner-take-all market.

QWERTY vs other keyboard layouts

The QWERTY keyboard layout was a relic from the typewriter era, designed to actively prevent higher typing speeds that would jam the typewriter, and it got carried over to computer keyboards. The trouble is that none of the alternate layouts have been able to show a big enough improvement in typing speed or any other factor. Really, how much better is some super-custom keyboard layout, once you invest all the time in getting fluent with it? Maybe 2x, if that? But you could also improve your typing speed just by learning to touch-type better, which also would carry over to other keyboards you have to use!

On the other hand, as a thought experiment, what if DVORAK were 10x faster than QWERTY? How about 100x faster? At a certain point, the improvement in a key factor becomes sufficient to overcome momentum of the existing technology and find a niche or take over the market. And we have some evidence for this: stenography has existed in parallel alongside QWERTY, because its speed over QWERTY made it more useful than QWERTY for some segment of the market.

RethinkDB vs MongoDB

RethinkDB vs MongoDB is an interesting case. MongoDB beat RethinkDB to market and was faster at first, while RethinkDB took their time and focused on correctness and safety (which many using NoSQL cared less about). Both technologies were in the same ballpark for all factors and faced direct competition.

Time for some wild speculation into human psychology

A funny story is that I myself accepted the usual explanation about VHS vs Betamax without much thought. Then after I started working on Unison people often asked me questions about programming language adoption and I realized how depressing it was for me to accept the premise that it was really just a harsh, cruel, and totally random world out there. According to this premise there was an excellent chance that nothing I worked on would be picked up by anyone else while other technologies I liked a lot less became wildly popular for no apparent reason. While feeling somewhat down in the dumps after one of these conversations I got a bit more curious about VHS vs Betamax… a couple hours of feverish internet research and hope was restored. The world actually makes sense, kinda!!!

The question I asked myself later: why did I so readily accept the earlier explanation of the “harsh cruel and random world”? Like all myths it has an element of truth. In the small, there is a lot of randomness at play, and we should acknowledge this reality. But the world is also not totally random and this fact becomes obvious even on cursory examination. If we’re really being honest, I think this “harsh, cruel, random” explanation is just another way of rationalizing inaction. If the world is really random, you aren’t responsible for how things play out and you might as well not bother doing anything, right? Just collect a paycheck and chill! The people trying to build actual new things are just suckers who are wasting their time.

When you are actually working on a big project, though, this kind of mentality would be super unmotivating, so it’s worth trying to understand whether this mentality reflects reality or if it’s something your brain has just accepted without much examination.

There’s hope!

A happier angle on internet debates about programming tech

Thu, 24 Oct 2019 00:00:00 +0000

I’ve been contemplating internet debates and discussions on programming technology and I think the music composers really have the right idea:

Alice: I woke up this morning with this gorgeous melody in my head. I just can’t stop thinking about it. It’s going to sound amazing for solo cello!

Bob: Wow, cool!! I can’t want to hear it.

Meanwhile, if Bob were a programmer

Bob: Wait, another piece of cello music? Alice, you know, Bach wrote a bunch of cello pieces hundreds of years ago. How is yours going to be different?

Alice: … What, er… Well, um, I guess it’ll be different notes, in different order and duration?

Bob: Okay, sure, some of the notes will be different, but I’m sure it will employ many of the same techniques. Will it have a perfect fifth in it? A major third?

Alice: Well, sure…

Bob: Yeah, see, Bach’s stuff did that too! Seems like pretty much the same thing. Honestly, why even bother? It’s been done before.

Alice: recalling the melody You know what, Bob, it hasn’t been done before you reductionist dweeb, and I’m not going to let you discourage me!

Here’s another variation

Bob: Oh, really? You know, Carol has been writing some stuff for cello, how does your stuff compare?

Alice: Carol’s stuff is pretty nice and I really like her Opus 17 a lot!

Bob: You and Carol should really collaborate since you’re both writing stuff for cello and so you’re both basically working on the same sort of thing.

Alice: Er, well, the thing I have in mind is a bit different than what Carol’s doing.

Bob: I mean, maybe it’s a little different, but it’s just really inefficient to have two people writing separate pieces of cello music, when we could just combine those efforts into one and get cello music that was twice as good!!

Alice: [Takes a breath] Listen, Bob, I really respect what Carol is doing and maybe it would be fun to collaborate together on a piece one day. But right now I want to do my own thing. Also, where did this reductionist idea come from that all these disparate creative efforts can just be combined like amoeba merging into one larger amoeba? It doesn’t work that way! What’s wrong with just being inspired by and having mutual respect for other’s work while still working on our own projects?

Bob: It sounds like you actually think your music is going to be better than Carol’s and that’s really what’s going on here!

Alice: What? If my 5-year old niece draws a picture that she likes, is she doing that because she thinks her drawing skills are better than anyone else? No, that does not even enter into it! She’s creating something that she likes. She doesn’t presume that her feelings about the matter are universal nor that she must convince other people to share her tastes. Some people may share her tastes, others may not, and the world goes round. We don’t need to view every endeavor in this competitive way, like we’re still cavepeople fighting over scraps of meat!

I just want to create

With music, everyone understands there’s an infinite space of possibilities, and it doesn’t really matter if you can concoct some metric which ranks two pieces of music as similar. People can listen to whatever music resonates with them, and we don’t all need to agree. It’s all good!

But with programming artifacts, we aren’t quite sure what to make of them. Programming languages, libraries, and tools are simultaneously creative tools for human expression (think: a paint brush or a violin) but also practical artifacts (think: an I-beam or reinforced concrete).

Bob: I think you’ve hit the nail on the head here. I’m uncomfortable with this wooly “everyone’s perspective is equally valid” take on creative tools. If you are building a skyscraper, you need to choose materials that are up the task and not build it out of toothpicks and marshmallows because just because you like those materials better!!

Alice: Of course. But I’d say programmer culture has this problem that the human expression aspect of programming is barely acknowledged and we therefore feel that decisions MUST be justified in terms of objective and universal statements of fact. Why can’t we just say “I prefer using XYZ and it makes me happy” once in a while?

Bob: Because that way lies madness and vagueness and all that icky liberal arts stuff! Also, how do I justify that to my boss?

Alice: Oh, gosh. It’s like you can’t acknowledge to your boss that you’re a human and part artist, you just have to pretend like you’re this hyperefficient and purely rational machine that will obey any command without thoughts or feelings! The role we have accepted for ourselves is like those badly written Hollywood sci-fi robots that can (somehow) predict exact probabilities of events out to 8 significant digits… because AI. Your happiness and motivation matters and is important… even an unfeeling boss who cared about nothing other than workers’ productivity (sounds dystopian to me) must understand that in a creative and mentally challenging endeavor such as programming it’s better for productivity when people are working with tools they like and which help them find joy in their work!

Alice (cont): And on that subject of vagueness you find uncomfortable, let me try to break down it mathematically for you. If we are being objective about it, most programming tech has a collosal number attributes and different individuals can (quite rationally) place different emphasis on these attributes. We can view this as a partial order where there is not one unique maximal element but many, none of which dominates the others, so there is only a set of tradeoffs. But not only that, because the number of attributes is collosal it is quite difficult to even study this space and come to definitive conclusions about it. People will, quite rightly, tend to make decisions by gut feel. It’s fine to embrace our uncertainty (tell that to your boss). We just don’t need to (nor can we) make definitive universal statements.

Bob: Okay okay, but come on, there ARE some objective statements you can make. Why shy away from that?

Alice: Yes, okay, there are things I believe and that I think have some rationale behind them, but life is just too short to spend time arguing with people about this stuff. If certain ideas really are “the right ones”, I’d rather just build cool things using those ideas and share my enthusiasm in the process. If that draws other people to these same ideas then that’s cool. But also if not that’s no big deal. And it could totally work the other way—maybe I’ll change how I think of things by seeing the cool work that others are doing! Basically, I don’t care about winning internet arguments with people, I just want to create and have fun, and I think it’s great that others are doing the same.

Remarks

Programmers are humans, not maximally productive robots. We like certain things, have different tastes and preferences, and some ideas we just find beautiful or inspiring or motivating or they just give us joy, even independent of practical benefits! Gosh darnit, let’s embrace that and be accepting of our humanity.

Down with internet debates on programming tech!!

Learning how to make a thing learnable is harder than learning it

Mon, 07 Oct 2019 00:00:00 +0000

Here’s something pretty crazy: there are around 10^28 atoms in the human body. An astromically tiny percentage of arrangements of 10^28 atoms in 3D space correspond to a viable human. Most arrangements of atoms would just melt into a puddle!

If you were trying to learn something as complex as a human body, with that many parameters, you’d need to transform the problem to a much simpler one, where the space of possibilities is much much smaller and where more of these possibilities have nonzero fitness. This allows the problem to be learned.

Remarkably, evolution has managed to do this: there is an incredibly complicated mapping from the human genotypes to the human body. Even more crazy, that mapping was itself learned by evolution! This has to be the most impressive thing about evolution - this seemingly dumb process converts what would otherwise be an insanely complex and huge problem space into a much simpler one where that same dumb process can be effective at learning.

Learning is really two things: step 1 is learning how to make the problem learnable, and step 2 is doing the learning. In machine learning, it’s usually the humans that figure out how to make the problem learnable, by applying our smarts to the problem to arrive at some highly constrained representation which is learnable via dumb algorithms. But step 1 is the hard part!

Maybe the reason we often overlook step 1 is that our brains are so good at reducing very complex problems to simpler spaces. We do our thinking in that simpler space and then project results back to the more complex space. For instance, in order to write this post, I definitely don’t think in the space of all possible sequences of characters. Instead I have thoughts which I translate via some complex mapping to actual characters. When I walk, I don’t think about controlling all the gajillions of nerve outputs directly, I execute some very high level movement plan which is translated to nerve outputs via some very complicated mapping. And so on…

In each of these cases, the hard part is actually coming up with this mapping or learning it in the first place!

Learning without a gradient (part 1)

Sat, 06 Jul 2019 00:00:00 +0000

Learning using gradient-based methods is a lot like trying to find the top of Mount Everest starting from a random point on earth, while blindfolded, by repeatedly sampling only the altitude and slope of the ground you’re standing on, and then using that information to decide where to teleport next.

Imagine you’ve got your blindfold on, and you note that the ground slopes upward if you travel northeast. How far should you go in that direction? The blindfold prevents you from just looking off in the distance and observing “oh, it keeps going upward in that direction for a while”. You can only sample the height and slope at the individual points you teleport to.

So maybe you walk 10 feet, and the ground continues sloping upward in that direction. Should you teleport half a mile along that same vector and take another sample? 10 miles? 100 miles? What if you teleport too far and skip right past the highest point? Or maybe you only move 1 foot at a time, but it takes you eons to get anywhere. Should you continue along a vector even if you reach a point where it starts leading downard? Or do you keep some “momentum” and hope that was just a temporary little dip in the landscape?

There aren’t really good answers to these questions and a lot of it comes down to: “I tried XYZ on these problems, and it seemed to work okay” and some rules of thumb for how to discover “good” hyperparameters for your problem.

The gradient just isn’t a lot to go on. The local slope at a point in Kansas might help you reach the top of the hill near where you started, but it doesn’t help in locating even the general region where Mount Everest is located.

Most optimization problems are actually harder than the above analogy lets on. The function being optimized will have more than just two parameters, and the landscape may be very “rugged.” It’s often expensive to compute your current altitude and the current slope, so we settle for an estimate of these things. Extending our analogy, imagine if you didn’t even have an accurate way of measuring current altitude or slope, but instead had lots and lots of unreliable measuring devices whose results you could aggregate (with much effort) to produce more accurate readings.

For instance, in deep learning, when training a network with a gajillion tunable parameters on a training set that’s 100k elements, calculating the actual “altitude” of your current location would involve evaluating the current network output for all 100k elements of the training set. Similarly for estimating the gradient: we can determine what direction to adjust weights to improve average performance for a batch of examples, but we don’t know for certain that improvement on this batch won’t just make performance on the overall training set worse!

You gotta make assumptions, and what, precisely, are they?

A takeaway from the above discussion is that any learning algorithm has to make some assumptions about the space it is optimizing, if it is to be effective. For instance, in our blindfolded Everest-finding example, knowing something about how the structure of the earth’s surface is helpful.

For instance, the earth’s surface altitude may change rapidly at a local scale, but can be quite smooth if we apply a low pass filter and average the altitude of neighboring regions. Mount Everest isn’t just a single spike in the middle of a flatlands, it’s a huge mass of land which must be supported physically, so it (and any other tall mountain peak) tends to be surrounded by regions of similar altitude. This property suggests that keeping some “momentum” in a gradient-based search could be effective, so the search is less sensitive to the (noisy) immediate local gradient signal. But then, what parameters do we use for this momentum?

There’s a famous set of theorems, the “No Free Lunch” (NFL) theorems, which amount to a formal statement about the need for optimization algorithms to make assumptions about problem structure. The first of the NFL theorems states that “any two optimization algorithms are equivalent when their performance is averaged across all possible problems”. So if you consider “all possible problems”, all optimization algorithms, including random search, give equivalent performance! My intuition for the NFL results is that for every problem where your fancy optimization algorithm does well, there’s another problem (in that huge space of all possible problems) where the assumptions your algorithm makes are actively harmful!

The NFL theorems were a pretty explosive result, because up until then people were making all sorts of unqualified statements about their learning algorithm being “better” after getting bettter results on a few benchmarks. NFL makes it clear: no optimization algorithm is better, only better for some kinds of problems.

A common response to NFL is that: “well, we aren’t solving problems randomly chosen from the space of all possible ones; we’re solving real world problems which have quite a bit more structure”.

But the way I like to understand the NFL results is that every optimization algorithm must exploit some assumed “structure” to the functions being optimized. If you can more precisely characterize the structure of the function being optimized, then you can pick the right algorithm for the job, the one that exquisitely exploits this structure to find very good solutions quickly.

As a simple example, in the field of convex optimization a set of highly efficient optimization algorithms have developed which exploit precise assumptions about the function being optimized (namely, that it is a convex function over a convex set).

Of course, many problems of interest aren’t convex, but I think what’s been done for convex optimization should be the gold standard of any research on learning or optimization algorithms: efficient algorithms along with a well-characterized subset of problems for which the algorithms do well (ideally, along with an algorithm to automatically check if a problem is in the subset). We should expect that for any optimization or learning technique, because the alternative is fumbling around in the dark with empirical tinkering and playing the hyperparameter slot machine(“I tried algorithm XYZ with hyperparameters ABC on problem P, and it did terribly, then I switched to hyperparameters DEF and it worked great”).

Also related is star-convexity. It’s been suggested that many deep learning problems are star convex and that this explains the success of stochastic gradient descent.

Next time

For the next post, I’m going to use our same “find Mount Everest while blindfolded” metaphor to explore a class of learning algorithms that doesn’t use gradient information at all.

What is really the root cause of the event-stream vulnerability?

Mon, 03 Dec 2018 00:00:00 +0000

I was trying to make sense of the event-stream vulnerability. My take:

Every JS script gets access to the DOM, a global object from which you can scrape every piece of information on the page. You can explore the DOM just by making up strings at runtime and rooting around in the DOM, so it can’t even be statically known what information JS code accesses. Yikes!
Scripts are just loaded via a JS function call (like require) to which you pass a string. Also a dangerous capability since it means the dependencies of a piece of JS code can’t be statically known.

So you have this global, dynamically accessible object containing all sorts of sensitive information, which every little script on the page gets access to, including the library you expect to be doing only pure computation on the data you explicitly pass it. It’s extremely difficult to secure this: JS access to the DOM is analogous to C’s feature of being able to poke at arbitrary portions of memory, which is difficult to secure for similar reasons.

It’s probably too late for JS, but if you were redesigning a language for the web and had more than 10 days to do it, it would be possible to completely eliminate these possibilities in the first place. Instead we’re stuck with the consequences of a system that’s almost impossible to secure.

People have been quick to blame the maintainer who handed maintenance over to someone who proved to be a bad actor, or other people or process issues. No, those aren’t the root problems. We should expect that people will make mistakes and instead question the system and core tech which allows these mistakes to be amplified and propagated at such enormous scale.

Evolution is a much more interesting learning algorithm than most people realize

Sat, 21 Apr 2018 00:00:00 +0000

Is evolution just another hill-climbing-like algorithm where the gradient is inefficiently estimated via sampling rather than direct differentiation? No. The usual formulations of evolutionary computation (EC) capture little of the algorithmically interesting aspects of evolution, which has led many researchers to wonder (perhaps correctly) whether there’s really any point to EC compared to more rationalized and efficient approaches to ML. This reaction is a bit like deeming human powered-flight a fruitless endeavor after seeing the first ineffectual ornithopers. Let’s not throw the baby out with the bathwater!

Evolution is a meta-learning algorithm that discovers its own heuristics and continually improves the rate at which it learns (something missing from most learning algorithms). It has mechanisms to induce generalization and strategies for smoothing out fitness landscapes to avoid local optima. In combination, its unique features have led to producing artifacts of greater complexity than anything humans have engineered (it produced humans, and all the exquisite complexity of the biosphere). Respect.

Let’s have a closer look:

Evolutionary systems not stuck at equilibrium (more on this in a minute) have an important long-term trend: the speed at which species can adapt to change increases over time. The result of this long-term trend is what we now call “intelligence”. Evolution is deceptive: close up, it seems to find highly specialized, even overfit solutions, but in the long-term, it induces the discovery of quite general solutions. Evolution is an “adaptability” discovery machine.
Biological evolution employs several strategies (often missing from our evolutionary algorithms) to decrease the odds of getting stuck in equilibria. Perhaps the most interesting: it discovers and refines its own heuristics as it learns!

Part 1: Evolution improves its own adaptability

Any evolving ecosystem undergoes regular changes in selection pressure, often due to the coevolving behavior of other organisms. For instance, when a predator evolves new behaviors, the selection pressure on its prey changes.
Thus, speed in adapting to change is selected for, as is the scope of adaptations that can be made by species. A species that can respond to only to a limited number of changes over the course of eons is at a disadvantage to one that can respond to more changes even just a little more quickly. (Example: an organism with only hardwired evolved behaviors cannot respond in time to new behaviors evolved by one of its predators, and is less fit than an organism that can do even a tiny amount of learning to adapt to its predators)
Over evolutionary timescales, we therefore see species evolve an increasing ability to adapt to their environments more quickly. This generates a positive feedback loop: when one species develops an ability to adapt more quickly, it benefits that species but also results in an environment which changes even more quickly, generating further selection pressure on other species to gain more rapid and effective means of adapting to changes.
The long-term trend of this positive feedback loop is what we now call “intelligence”. Intelligence isn’t about any particular skill, it’s about the ability to adapt to an ever-expanding scope of possible changes on shorter and shorter timescales. Provided evolution does not get stuck, intelligence is the result.

This argument can be formalized, but that is the gist: adaptability is selected for (because the rules are changed regularly and species have time to adapt to these changes before going extinct), and this generates a positive feedback loop, where speedier adaptation by one species results in more rapid changes to selection pressure and further increasing the selection pressure on adaptability.

Part 2: Necessary features of the biosphere’s evolutionary algorithm

Here’s a likely incomplete list of necessary features to replicate the biosphere’s unparalleled open-ended evolution:

In an evolving ecosystem, there isn’t one fitness function, there are K fitness functions at any given time, one for each “niche”, and these fitness functions are essentially uncorrelated. Organisms are roughly ranked based on the max of these K fitness functions (since species can move between niches assuming they develop the traits enabling them to compete in the new niche). As K grows large, assuming the functions are uncorrelated, the merged fitness function becomes smooth. This same smoothing of merged fitness functions also plays out within a single organism, where evolved structures are regularly repurposed (put to their maximum good use). Without this, the landscape is often too rugged to avoid bad equilibria.
The set of fitness functions changes over time, at an increasing rate, a strategy to induce general solutions and adaptability. But this feature works in concert with another: selection does not act by instantly killing lower-fitness organisms or species. Lower fitness organisms have time to adapt to their environment (learning) before dying, and over generations, species have time to adapt before going extinct. By having this slack, it generates selection pressure to develop ways of adapting on shorter timescales–either within the lifetime of a single organism (learning), or over fewer generations of a species.
- For instance, crossover is a much more efficient variational operator than mutation, since it generates variation by combining two valid genotypes. When circumstances changed, species that used sexual reproduction were more able to adapt to these changes than counterparts without this means of generating variation.
- If the fitness functions are unchanging or selection has no slack, there’s no selection pressure encouraging adaptability in general, in fact there’s no long-term trends at all, and the system just finds an equilibrium.
The variational operators (encompassing genotype-phenotype mappings) are themselves subject to evolution. We should think of variational operators like heuristics, used to explore the search space. The trick is that these heuristics are wrapped up in the organisms themselves, so when natural selection picks organisms with good survival traits to reproduce, the heuristics which led to those traits are then given further weight (and these heuristics are then further tweaked). In this sense, we can think of evolution as discovering and refining its own heuristics. We can also think of these heuristics as altering the fitness landscape, essentially projecting to a different space that is much smoother.

This last point might be the most important. If the variational operators and genotype-phenotype mapping is static, this limitation eventually overwhelms any smoothing effect of multiple niches. Over time, as the products of evolution become more complex, the odds of random change producing anything useful approach zero—we can’t put an airplane in a giant blender, pulse it a few times, and hope the output is a fighter jet. We would be lucky if the output even functions as an airplane at all! Biological evolution has become increasingly surgical in its variation, matching the complexity of the species it evolves for whom a much smaller number of changes are even viable.

Many uses of evolutionary algorithms have a hardcoded genotype-phenotype mapping (or no such mapping - the mapping is the identity function). Since this is very limited, people often try to engineer the mapping to get better results. But a human engineering this mapping is just “the human doing the learning”, moving the starting line closer to the finish line. Perhaps there’s something to be said for letting an automated procedure handle the “last mile”, but for most interesting domains, coming up with good genotype-phenotype mappings that smooth the search space is exactly the sort of thing we’d like our algorithms to discover for themselves!

The rationalized way of doing something like evolution’s meta-search for heuristics might be to: 1) Generate N heuristics for moving through the search space. 2) Allocate computational resources to each heuristic based on performance (total fitness achieved, for instance) 3) Vary the heuristics, perhaps at a much slower rate than the rest of the system. What isn’t clear to me is how to represent such heuristics in a way that can also be scaled from the very simple to the very complicated. (I wonder: is evolution using the same tricks to evolve “meta-heuristics”, and “meta-meta-heuristics”?)

The features of evolution work in concert, and there are probably other features we are missing. But the cartoon versions of evolution we see in EC, even if they are useful algorithms for some problems, probably aren’t capturing enough to replicate the power of biological evolution.

Remarks

There’s been a tendency in popular accounts of evolution to give a sense of “we’ve got this all figured out, it’s as simple as Darwin said”, so as not to yield any ground to those who would treat every gap in our understanding as evidence that evolution doesn’t work and that in fact, “God did it”. Unfortunately, responses like these have shaped the common understanding of evolution. The simplistic common understanding is insufficient to explain the biosphere’s learning performance (as anyone who has tried building artificial evolutionary systems can attest), but it’s what many people know, including researchers in ML who often dismiss evolutionary methods as cargo-cult biomimicry.

Perhaps it’s time to give evolutionary methods another closer look!

The Fermi paradox, and is "intelligence" the inevitable byproduct of any long-running evolutionary process?

Sat, 14 Apr 2018 00:00:00 +0000

I read a fascinating book recently on the Fermi Paradox, which asks the question: if there are supposedly millions of other civilizations in the galaxy, why haven’t we detected any real evidence of their existence? One class of solutions says: what happened on Earth is a fluke. We’re alone in the galaxy. Gosh, that would be disappointing! Let’s have a closer look at one piece of the puzzle.

One of the terms of the Drake equation is “the probability that intelligent life arises on a planet where life exists”. Put another way, does “intelligence”, however defined, have any special status in evolution or is it just like any other adaptive trait—no different in principle than, say, spiders evolving to spin webs or fish being able to swim? If it’s nothing special, then maybe it is a fluke that intelligence arose on Earth.

Here’s a simple argument for why evolutionary processes will, if they avoid getting stuck (more on that later), develop toward increasing levels of intelligence:

Any evolutionary system (defined as a system where entities reproduce with variation and are subject to natural selection) undergoes change, often due to the evolving behavior of other organisms. For instance, when a predator evolves new behaviors, the selection pressure on its prey changes.
Thus, speed in adapting to change is selected for, as is the scope of adaptations that can be made by species (I’ll call this “adaptability”). A species that can respond to change only over the course of eons is at a disadvantage to one that can respond even just a little more quickly. (Example: an organism with only hardwired evolved behaviors cannot respond in time to new behaviors evolved by one of its predators, and is less fit than an organism that can do even a tiny amount of learning to adapt to its predators)
Over evolutionary timescales, we therefore see species evolve an increasing ability to adapt to their environments more quickly. This generates a positive feedback loop—when one species develops an ability to adapt more quickly, not only does this benefit that species, it also results in an environment which changes even more quickly, generating further selection pressure on other species to gain more rapid and effective means of adapting to changes.
The long-term trend of this positive feedback loop is what we now call “intelligence”. Intelligence isn’t about any particular skill, it’s about the ability to adapt to an ever-expanding scope of possible changes on shorter and shorter timescales. Provided evolution does not get stuck, intelligence is the result.

Let’s look at how this has played out in biological evolution:

The first single-celled organisms were immobile, at the mercy of whatever the conditions of the drop of water they found themselves in. The evolution of motility allowed an organism to adapt more readily to a changing environment—it could move to new food sources, away from predators, etc.
Vision and other forms of sensing served a similar purpose—providing an organism with information it could use to adapt to change on shorter timescales. But with sensory input comes a need to have some ability to perform computation on that sensory input and change behavior in response to that input.
Having an ability to learn new behaviors, rather than merely having the function from sensory input to behavior be hard-wired was a further improvement in the speed and scope of adaptability. Any organism with even a modicum of ability to learn could be at an advantage.
The logical endpoint of this process is investment in a general ability to sense, learn, and implement new behaviors (what we now call “intelligence”), which can respond to change much more rapidly than the eons required to implement changes to the genetic code.
Language and culture is a further improvement in adaptability—it allows for a thing to be learned by just one organism, shared on very short timescales with other members of the species, and retained across generations (rather than being forgotten and relearned by each generation). Written language is a further amplifier of this effect.
In general, cultural and technological evolution increase the adaptability of a species much more efficiently than the slow, generational process of evolving the genetic code.
And so on…

A spider which has evolved over millions of years to be able to spin webs is impressive. But it’s in possession of a specialized piece of hardware. Over time, evolution discovers that a good survival strategy in a changing environment is less specialized, more general purpose hardware on which many pieces of software (learned behaviors) can all run.

Getting stuck

Even if growing intelligence is a trend in evolutionary systems, it’s very possible for evolution to get stuck in equilibria. This might not be obvious, especially if you read popular accounts of evolution where it’s easy to get the impression we have it all figured out.

To see some of the problems, consider a simple island ecosystem with only plants (who feed off sunlight), deer (who eat the plants), and wolves (who eat deer). Even if deer can in theory become more fit by becoming more capable at evading wolves (perhaps by learning to better predict wolf behavior), who is to say that such adaptations are “reachable” according to the variability available to deer? (Deer are not going to develop opposable thumbs, large brains, language, and toolmaking all in one go, and likewise for wolves.)

Other pathologies: the wolves might evolve to be so good at hunting deer that they decimate the deer populaton (which they are dependent on). With the primary food source of wolves decimated, adeptness in hunting deer is no longer selected for, and if the wolf population repopulates, who knows what adaptive traits remain (selection may shift to favoring wolves capable of going longer without food, for instance). Likewise, if deer become so good at evading wolves, the wolf population may decline and the deer may overgraze the plant population, and so on. The system can bounce cyclicly between different states, failing to make any “progress”.

Is it just a fluke that biological evolution seems to have evolved in an open-ended fashion, or do certain kinds of evolutionary systems, like the evolution of earth’s biosphere, have open-endedness as a near inevitability? And if so, how might we engineer such systems? This is a question studied in artificial life and evolutionary computation (which I studied a long time ago). At the time, no one seemed to understand these questions very well at all, and I’m curious if progress has been made.

The way I felt when studying evolutionary computation is that our understanding of evolution was roughly analogous to our understanding of flight before the invention of the airplane. We understand the basic principles at work. We suspect we could replicate the performance of evolution with an extremely detailed simulation of biological evolution (though even this is uncertain—there are questions about whether we’d end up with similar results on Earth if we just “replayed the tape”). But it isn’t clear what aspects are truly essential or if we could substantially simplify our simulations and achieve similar results.

Fermi

I don’t know how likely it is that Earth’s evolutionary process got as far as it did. I suspect there isn’t anything special about it. Time and time again, when we look at biological evolution and wonder how it ever assembled some complicated piece of biological machinery, we learn it found a way (often rediscovering the same complex structure over and over again). Seeing this kind of thing makes me hesitant to conclude that it’s a fluke evolution didn’t get stuck at the prokaryote stage or someplace else along the way to general intelligence. Perhaps there are things about biological evolution that make it unlikely to get trapped in equilibria, which we don’t fully understand yet?

I’ll close with this quote by Richard Dawkins:

Never say, and never take seriously anyone who says, “I cannot believe that so-and-so could have evolved by gradual selection”. I have dubbed this kind of fallacy “the Argument from Personal Incredulity”. Time and again, it has proven the prelude to an intellectual banana-skin experience.

I’d love to hear your thoughts on this.