New algorithm discovers language just by watching videos

The audio-video matching game plays a crucial role in training the DenseAV model. This game involves predicting what is being seen from what is being heard, and vice versa. For example, if the sound of someone saying "bake the cake at 350" is heard, the model should associate it with the visual of a cake or an oven. To succeed in this game across millions of videos, the model has to learn and understand what people are talking about. This process enables the model to learn the meaning of words and the location of sounds, as well as to distinguish between different types of cross-modal connections, such as the difference between the word 'dog' and a dog's bark.

The DenseAV model learns language by predicting what it's seeing from what it's hearing, and vice-versa. It plays an audio-video matching game across millions of videos, which requires the model to understand what people are talking about. For instance, if the sound of someone saying "bake the cake at 350" is heard, the model might expect to see a cake or an oven in the video. By observing which pixels are selected by the algorithm when it hears a specific sound, one can discern what the algorithm thinks a word means. The model also learns to distinguish between different types of cross-modal connections, such as the difference between the word 'dog' and a dog's bark, without human intervention or any knowledge of written language.