Searching for tutorials and software about Deep Learning and Neural Nets? Be sure to look at my Resource Page!
Looking for Octave? Go to my Easy Octave on Mac page!

Monday, April 20, 2015

Stochastic Music, Stochastic Gradient Descent

Today's notes are rich in the word "stochastic". To me this is an unfortunately popular and potentially venomous word, which needs to be carefully pinned to the page before it can jump out.

However, stochastic music is fun. When I was a student at ENST, we rented a piano, and asked Claude Helffer, an alumnus of Polytéchnique  (where I later got my PhD) to come and play some pieces composed by Iannis Xenakis. 

As Helffer explained, the schemata of the music, and local patterns, are provided by the composer. The interpreting pianist then decides how to repeat the patterns from the "reservoirs de notes" depending on the  duration he intends to perform. Once entered, a "reservoir de notes" is —as I remember it— played to the end. 

Stochastic Gradient Descent —SGD to most of us—is the backbone of much of supervised neural net learning.  Gradient descent is the optimisation method, and "stochastic" means that training patterns from the training ensemble are repeatedly chosen and presented at random rather than in a precise sequence. 

Coincidentally, Léon Bottou is also a Polytéchnicien, and an expert on SGD. I believe he is presently at Facebook. Like Claude Helffer he graduated from Polytéchnique, did some backprop at Orsay, moved to the US, and  joined LeCun at Bell Labs. Bottou authored some of the first net simulators. I met him as a journalist and wrote about a simulator he authored that was marketed by a startup called Neuristique. 

While at Microsoft Research, Bottou penned a cookbook chapter for SGD, called Stochastic Gradient Tricks which I'm looking through at the moment. One of my motivations is to check the FANN code internals, and Léon suggests ways to test such mathematical code by its input-output behavior. 


No comments:

Post a Comment

Hey, let me know what you think of my blog, and what material I should add!