Machine learning vs Researching brains: the Fibonacci example

Why is there all this buzz about around machine learning (or statistics, as it should be called)?

Because ML allows people, and in particular companies, to find good enough solutions to problems without having to solve those problems completely and in a rigorous way. "I don't need to solve computer vision! I just want to tell the round pieces apart from the square ones!"

The problem is: we need complete and rigorous solutions for science to advance. ML is not enough.

To solve a problem in a rigorous and scientific way, you need to let people study it, compare it with similar problems, try out hypothesis… It can take years.

To solve a problem with ML, all you have to do is describing your problem as a "loss function" problem, find enough examples of the possible inputs and the desired outputs, acquire good enough hardware, let the software run and wait until the result comes up. No long study hours needed.

If you just want to know a decent solution to your problem, then ML allows you to find such approximate solution just by buying computer stuff instead of brains.

The drawback is: the results that ML yield are not as enlightening as those achieved though scientific enquiry. ML will tell you what the answer is, not why that is the answer. This is serious drawback. Without the insights provided by the resulting castle of theories, formulas and explanations, science cannot progress.

Let me illustrate when and how research-based results beat ML-based results with what I call the Fibonacci example.

An "unknown" pleasant sequence of numbers

Suppose you want to predict the next numbers in this sequence:

For instance, the number in position 8 is 21. What is the correct number for position 10? And in position 284?

(Yes, this is the Fibonacci sequence, but let's pretend for a second that we haven't recognized it.)

The machine learning approach

Can you use ML to create a forecast function? Sure you can. It is not that hard for someone who knows the rudiments.

In order to create this forecast function using ML you will need:

1. A big enough set of "examples", in other words, pairs that associate an input to the correct output, say (1, 1), (4, 3), (2, 1), (7, 11).
2. A collection of algorithms that build the forecast function and adjust its internal parameters learning from these examples.
3. Enough time to wait until the ML algorithms process your examples and create the forecast function.

This process must be done only once. Then, every time you want to predict a number in the sequence, you give to the forecast function the position you are interested in and you receive back an answer. This answer is not going to be 100% exact but will be very close to the correct number. (That is in the nature of statistical methods.) Another inconvenience is that calculating this answer will be quite slow.

Do you want your prediction to be more precise? Spend more time (or money) collecting more examples.

Do you want your model to be created using fewer examples? Buy a fancy proprietary library of ML algorithms instead of using a free one.

Do you want to wait less while the forecast function is created or get faster answer? Go and buy more computers or CPU time on AWS.

The main idea here is: Do you want something better? All you have to do is paying more.

But while hardware and software may provide you with better answers, they will not give you any insight on why that answer is right. Don't you want to know why the answer is right?

The research approach

If, instead of spending money, you were willing to do research (or hire a researcher), you may find out that the sequence of numbers you are looking at can be expressed by this simple formula:

` f(n) = f(n-1) + f(n-2) `

This formula is trivial to implement in a computer program and, with the appropriate optimizations in places, can be computed acceptably fast.

Doing research paid off. Here is how. With this formula we will no longer forecast what the value in position 284 is. We will calculate that value. With total precision and the security that the result will always be correct. Provably correct.

In addition, the formula we discovered tells us something deep about the nature of the sequence we studied. For example, that it can be expressed in a recursive way and that it can be calculated using just additions. None of these insights were provided by the forecast function generated by the ML algorithms.

Investing even more in brains

With additional research, you could discover that the simple formula you just found out can be improved and turned into a so called closed-form solution:

` f(n) = floor(1/sqrt(5)((1+sqrt(5))/2)^n+1/2) `

This formula is much more complicated that the previous one and requires more effort to understand it but is also much faster to compute.

Investing more in research paid off even better than in the previous case. Not only we can compute the exact value of any position of this sequence, we can also do it very fast now.

Standing on the shoulder of what?

Let's go back to the original question: why is there all this buzz around machine learning/statistical techniques these days?

Because ML allows you to obtain your tangible result (having a function that predicts the unknown numbers that sequence) by buying stuff instead of doing research.

From an engineering point of view, applying ML techniques and buying stuff makes more sense than doing research. First all all, by carefully applying ML you get good enough results quickly. On top of this, you can get a better results (or the same results but faster) just by spending money, a resource easier to find scarce than good researchers.

The drawback is that the availability of a good enough solution found using ML will often lessen the urge to find a perfect solution via scientific research.

Were an acceptable ML-based Fibonacci forecast function available, would people spend time researching a perfect Fibonacci function? Would basic research on understanding the Fibonacci function be supported?

Preferring ML to classic research does not make sense from the point of view of the advancement of science, even agreed that it does from an engineering point of view. Science advances one discovery at a time, building results on top of previous results ("standing on the shoulders of the giants", as Bernard of Chartres and Isaac Newton put it). Good results obtained via ML are of practical use but give us no ground on top of which other results can be built.

ML solutions are good shot-term deals, not long-term investments. Science needs long-term investments.

On the other hand, doing research can often be fruitless on the short term. It has taken mathematicians more than 400 years to come up with the optimized closed-form for the Fibonacci formula.

I am afraid most business will not support researchers in their slow journey of discovery. Society, and universities above all, should.