Thursday, November 3, 2011

Modeling with Octave

The discipline of machine learning (ML) can be understood as a particular study of data. Data is everything, and it contains patterns and a certain degree of uncertainty. The aim of the field is to identify those and exploit them to solve complex problems. Often algorithms for obtaining such patterns are not easy to implement, involving abstract mathematical concepts and vague descriptions of the internal processes that guide the learning mechanisms. In this particularly discouraging environment there are several useful tools to help with the understanding of the ML techniques. Octave is such a tool.

Octave allows the extensive use of vectorization to exploit parallelism and modeling complex algorithms in a very fourth-generation programming language fashion. I started with Octave due to Standord's ml-class course and I have to say that the experience has been very positive. In one of the exercises we had to implement a handwritten character recognizer, that is, a type of perceptron that were able to recognize handwritten numbers from images. I'm still amazed of the small number of lines of code involved to achieve it. The exercise demanded that the code were fully vectorized, without any loop in the training process (the real difficulty lied here, trust me) and the results were mind-blowing: a very fast response with the beauty of an elegantly small chunk of code. Using C++ I would need thousands of lines of code to do the same, with a lot of arcane crap to handle vectorization (i.e., calls to SSEx instruction set and other fancy stuff), not to mention the difficulties of plotting the results. So it is clear that modeling with Octave (or similar software such as R or Matlab) before implementing the real system with a standard/commercial programming language such as C++ is a great idea.

Do not get me wrong: I really enjoy implementing complex algorithms in C , C++ and Java and I think that to fully understand the complexity of the process of what one is doing implementing it from scratch in a traditional programming language is a must, but modeling first may help a lot.