Searching for tutorials and software about Deep Learning and Neural Nets? Be sure to look at my Resource Page!
Looking for Octave? Go to my Easy Octave on Mac page!

Sunday, April 19, 2015

Evaluating the MLP experiment

There are a set of phases in this type of "computer science" work:

  • Thinking vaguely
  • Outlining an experiment
  • Building software
  • Running software, tinkering with the design until happy or just tired
  • Interpreting results
  • Writing up 
Unfortunately, once programming has started, there is often not a lot of time for just thinking and interpreting results. The software design has solidified, and any novel ideas are deferred to "the next experiment". 

In my case, it was my intention to reproduce some existing results, I've done that, and now I've finished one cycle. I've nothing that needs writing up, I can think a bit about what I want to do,  what is feasible, and outline a new experiment.  Here is the structure of the software at the moment:

There are 3 programs:

  1. edconvert which reads Yann Le Cun's preprocessed MNIST data to extract whatever fields I need and write out a file in FANN format. 
  2. edtrain, trains a shallow three layer MLP with stochastic gradient descent
  3. edbulktest computes statistics on the training set of 60K samples and the validation set of 10K.  
Edconvert is only run for data import, but it could presumably be used to do more, like generate batches of data, and even tamper with the data by adding noise or distorting the images. At the moment I don't want to go there, and want to think a bit about edtrain and edbulktest.

To me, the separation between training and testing seems excessive. The learning process has no overview other than mean square error of its global behavior in sample space, nor any chance to learn again on the validation data. If neural nets have in some sense a real world biological inspiration, then I don't think this way of evaluating them is really in accord with the model. I'm not sure that using one differentiable loss or error function in the training and another to compute success statistics is so clever. 

From the speed point of view, a forward computation or a backwards training pass through the net chosen for this problem seems to take on the order of a millisecond on my hardware. This datum informs my immediate experimental choices.


No comments:

Post a Comment

Hey, let me know what you think of my blog, and what material I should add!