points by resters 12 hours ago | 0 comments
The idea that a word is defined by the company it keeps was decades old. Mikolov’s insight was that you could learn it at scale with a very simple model, and that meaning would fall out as geometry. Obvious in hindsight, foundational in practice.
[–] microtonal 11 hours ago
[–] DC-3 10 hours ago
Just FYI: You sang his praises. If you "sang my praises", that means you celebrated yourself.
There is even a paper by Levy & Goldberg from 2014 that shows that skip-gram with negative sampling implicitly factorizes a PPMI word-context matrix.
But Mikolov's formulation is very nice in that it is a simple linear model that allows for a very simple Hogwild optimization that worked very well/fast on multi-core CPUs of the time.