Vladimir Vapnik on Predicates, Invariants, and the Philosophical Essence of Intelligence in AI

This podcast episode features Vladimir Vapnik, co-inventor of Support Vector Machines and VC theory, discussing the fundamental differences between engineering intelligence and understanding intelligence. Vapnik argues that engineering, as inspired by Alan Turing's imitation game, focuses on building devices that mimic human behavior, while the science of intelligence seeks to uncover the underlying principles. He introduces the concept of "predicates" – universal, fundamental properties or functions that describe the essence of phenomena, drawing parallels to Vladimir Propp's analysis of folk tales and Plato's theory of forms, where a small world of ideas projects onto a vast world of reality. Vapnik believes that true intelligence lies in discovering these pure ideas or predicates and using them to construct "invariants" from data.

Vapnik elaborates on predicates as functions that capture integral properties of data, such as symmetry in images. He distinguishes between strong and weak convergence in function spaces, explaining that predicates help define an "admissible set of functions" by imposing constraints based on these integral properties. By incorporating known predicates, the search space for learning algorithms can be significantly reduced, leading to more efficient learning from smaller datasets. He contrasts this approach with deep learning, which he views as operating on a much smaller, less rich set of piece-wise linear functions, and suggests that deep learning's success with convolutional networks is an example of implicitly using a single, powerful predicate (translational invariance).

A key challenge Vapnik poses is to achieve record-breaking performance in tasks like digit recognition using significantly fewer examples by explicitly leveraging good predicates. He speculates that the world of useful predicates is small, and the essence of intelligence lies in discovering these "very good predicates" that dramatically reduce the VC dimension of the admissible function set. While he acknowledges that machines can be used to search for predicates, he expresses doubt that they can discover the "smartest" or most profound ones, suggesting this might remain a human endeavor, akin to how critics describe music or Propp analyzed narratives.

The broader implications of Vapnik's perspective point towards a new direction for AI research, one that moves beyond pure imitation and engineering towards a deeper, more philosophical understanding of intelligence. By focusing on the discovery and formalization of universal predicates, AI systems could potentially learn more efficiently, generalize better, and achieve a level of understanding closer to human cognition. This approach emphasizes the importance of abstract, conceptual knowledge – the "world of ideas" – as a guiding principle for building truly intelligent systems, rather than solely relying on data-driven pattern recognition in vast, unconstrained function spaces.

"Engineering is imitation of human activity. You have to make a device which behaves as a human behaves... But to understand what is intelligence about, is quite different problem."

"I believe that intelligence, it is world of ideas. But it is world of pure ideas."

"The combination of ideas and a way to constructing the variant is intelligence, but first of all a predicate."

"I strongly believe in this Plato idea, that exists world of predicate and world of reality and predicate and reality are somehow connected and you have to figure out that."

"So the essence of intelligence is, while only being able to observe the world of things, try to come up with a world of ideas."

"I don't know because it's so complicated. It involves a lot of stuff which I never considered. But I know about digit recognition."

"So you're decreasing, decreasing, and it makes it easier for you to find the function you're looking for. But the most important part, to create a good admissible set of functions."

"So you can, the story is formal story and it's a magical story is that you can use any function you want as a predicate. But some of them are good, some of them are not."

"I hope that there is a standard predicate like, Propp showed. That's what I want to find for digit recognition."

"It is completely new area of what is intelligence about on the level, starting from Plato's idea. What is the world of ideas? And I believe there is not too many."