Are Octave, Scilab and Matlab Reliable?

Abstract

In a word, no. In this article we show evidence that three platforms frequently used for solving Engineering problems have numerical pitfalls. These platforms are Octave, Scilab and Matlab, running on i386 architecture and three operating systems (Windows, Ubuntu and Mac OS). They were submitted to two comprehensive tests, namely the data sets and functions provided by NIST (National Institute of Standards and Technology), and our proposal of a set of matrices and operations on them. NIST protocol includes the computation of basic univariate statistics (mean, standard deviation and firstlag correlation), linear regression, and extremes of probability distributions. Our set of operations include matrix inversion and the computation of the determinant and eigenvalues. The assessment is made comparing the results computed by the platforms, and assessing the number of correct digits with respect to certified values. Serious pitfalls are identified in seemingly easy tasks as, for instance, the first-lag autocorrelation coefficient. Whenever available, the results are compared with those provided by R, a FLOSS (Free/Libre Open Source Software), whose excellent numerical abilities have been reported elsewhere.