Principal Component Analysis and Randomness Tests for Big Data Analysis

This§book presents the novel approach of analyzing large-sized rectangular-shaped§numerical data (so-called big data). The essence of this approach is to grasp§the "meaning" of the data instantly, without getting into the details§of individual data. Unlike conventional approaches of principal component§analysis, randomness tests, and visualization methods, the authors' approach§has the benefits of universality and simplicity of data analysis, regardless of§data types, structures, or specific field of science.§§First, mathematical preparation is described. The RMT-PCA and the§RMT-test utilize the cross-correlation matrix of time series, C = XX T ,§where X represents a rectangular§matrix of N rows and L columns and X T represents the transverse matrix of X . Because C is§symmetric, namely, C = C T ,§it can be converted to a diagonal matrix of eigenvalues by a similarity§transformation SCS -1 = SCS T using an orthogonal matrix S . When N is§significantly large, the histogram of the eigenvalue distribution can be compared§to the theoretical formula derived in the context of the random matrix theory§(RMT, in abbreviation).§§Then the RMT-PCA applied to high-frequency stock prices in§Japanese and American markets is dealt with. This approach proves its§effectiveness in extracting "trendy" business sectors of the§financial market over the prescribed time scale. In this case, X consists§of N stock- prices of length L , and the§correlation matrix C is an N by N square matrix, whose element at the i -th row and j -th column is the inner product of§the price time series of the length L of the i -th§stock and the j -th stock of the§equal length L .§§Next, the RMT-test is applied to measure randomness of various§random number generators, including algorithmically generated random numbers§and physically generated random numbers.§§The book concludes by demonstrating two application of the§RMT-test: (1) a comparison of hash functions, and (2) stock prediction by means§of randomness.§§