Vladimir Vapnik: Statistical Learning, VC Theory, and the Nature of Intelligence

The conversation with Vladimir Vapnik, a pioneer in statistical learning, delves into the fundamental nature of reality, learning, and intelligence. Vapnik distinguishes between instrumentalism (theories for prediction) and realism (understanding God's creation), aligning himself with instrumentalism in machine learning, where the goal is prediction rules rather than conditional probabilities for understanding. He emphasizes the "unreasonable effectiveness of mathematics" in revealing simple underlying principles of reality, often surpassing human intuition. Vapnik argues that imagination can be a hindrance in mathematical discovery, advocating for rigorous derivation from axioms rather than speculative interpretations.

A core distinction Vapnik makes is between "strong convergence" and "big convergence" mechanisms in learning, highlighting that current machine learning predominantly uses only the former. He introduces the critical role of "predicates" and "invariants" – exemplified by the "looks like a duck" proverb – as a form of intelligence that significantly reduces the need for vast amounts of training data by narrowing the "admissible set of functions." He criticizes deep learning as "fantasy" and "interpretations" rather than rigorous mathematics, suggesting that optimal solutions often lie in shallow networks, as indicated by the Representer theorem.

Vapnik challenges the current paradigm of deep learning, proposing that problems solved with "zillions of training data" could be tackled with "a hundred times less" by effectively incorporating invariants and predicates. He frames the creation of an "admissible set of functions" with a small VC dimension as the most challenging and crucial problem in learning theory, a task often overlooked in classical approaches. For practitioners, this implies a shift from brute-force data reliance to intelligent feature engineering and predicate generation, focusing on mathematical principles over biological analogies.

The discussion extends to the nature of intelligence itself, questioning whether it resides solely within individuals or is part of a larger, interconnected "world intelligence," citing simultaneous discoveries in science. Vapnik identifies the "how to get predicates" as the ultimate open problem of intelligence, suggesting that understanding "why one teacher is better than another" and how they impart wisdom through "remarks" and "philosophy of reasoning" is key. This redefines the pursuit of Artificial General Intelligence from mere imitation to uncovering the mechanisms of intelligent predicate generation, moving beyond applications to fundamental science.

In philosophy, they distinguish between two positions: positions of instrumentalism, where you're creating theories of prediction and position of realism, where you're trying to understand what God did.

I believe that this article about Unreasonable Effectiveness of Math is that if you look at mathematical structures, they know something about reality.

I think the best human intuition, it is putting in axioms, then it is technical where you have to arrive.

Whatever I did, I exclude any imagination because whatever I saw in machine learning that come from imagination, like features, like deep learning, they're not really one to the problem.

It is not description what's going on. It is interpretation. It is your interpretation. Your vision can be wrong.

It is better than a thousand days of diligent study is one day with a great teacher. But if you'll ask what the teacher does, nobody knows. And that is intelligence.

If it looks like a duck, sleeps like a duck, and quack like a duck, then it is probably a duck.

Mathematics does not know deep learning. Mathematics does not know neurons; it is just functions.

I would say one of the theorems, which is called Representer theorem, it says that optimal solution of mathematical problems, which describe learning, is on a shallow network, not on deep learning.

So, maybe our model of intelligence is only inside of us is incorrect. It may be that they exist with some connection with world intelligence.