Learning using gradient-based methods is a lot like trying to find the top of Mount Everest starting from a random point on earth, while blindfolded, by repeatedly sampling only the altitude and slope of the ground you’re standing on, and then using that information to decide where to teleport next.
I was trying to make sense of the event-stream vulnerability. My take:
Is evolution just another hill-climbing-like algorithm where the gradient is inefficiently estimated via sampling rather than direct differentiation? No. The usual formulations of evolutionary computation (EC) capture little of the algorithmically interesting aspects of evolution, which has led many researchers to wonder (perhaps correctly) whether there’s really any point to EC compared to more rationalized and efficient approaches to ML. This reaction is a bit like deeming human powered-flight a fruitless endeavor after seeing the first ineffectual ornithopers. Let’s not throw the baby out with the bathwater!
I read a fascinating book recently on the Fermi Paradox, which asks the question: if there are supposedly millions of other civilizations in the galaxy, why haven’t we detected any real evidence of their existence? One class of solutions says: what happened on Earth is a fluke. We’re alone in the galaxy. Gosh, that would be disappointing! Let’s have a closer look at one piece of the puzzle.
Haskell “enforces” typeclass coherence by asking that you define all the instances for a type in the same module (really, file) where you define that type. Whenever you import anything from that module, you bring all these instances into scope. It “works”, but it’s a kludge - when defining the type
Foo a, you can’t anticipate (nor are you necessarily aware of) all the instances you might wish to give for
Foo. We want an “open universe” where people can after the fact discover common structure and layer it onto existing data types. I feel this is an important property for a language ecosystem.