Factor analysis assume the following[1]: x – mu = LF + epsilon
where L is factor loading, and epsilon is error terms(or uniqueness that’s not explained by common latent factor F).

This is basically saying that x is generated from low dimensional F. F is multiplied by L (so that it can be high dimension), and then finally e is added[2].

In R, we use factanal for factor analysis[4]:

> v1 <- c(1,1,1,1,1,1,1,1,1,1,3,3,3,3,3,4,5,6)
> v2 <- c(1,2,1,1,1,1,2,1,2,1,3,4,3,3,3,4,6,5)
> v3 <- c(3,3,3,3,3,1,1,1,1,1,1,1,1,1,1,5,4,6)
> v4 <- c(3,3,4,3,3,1,1,2,1,1,1,1,2,1,1,5,6,4)
> v5 <- c(1,1,1,1,1,3,3,3,3,3,1,1,1,1,1,6,4,5)
> v6 <- c(1,1,1,2,1,3,3,3,4,3,1,1,1,2,1,6,5,4)
> m1 <- cbind(v1,v2,v3,v4,v5,v6)

> factanal(m1, factors=2, rotation=”none”)
Call:
factanal(x = m1, factors = 2, rotation = “none”)

Uniquenesses:
   v1    v2    v3    v4    v5    v6
0.005 0.114 0.642 0.742 0.005 0.097

Loadings:
   Factor1 Factor2
v1  0.853  -0.518
v2  0.804  -0.490
v3  0.598       
v4  0.508       
v5  0.857   0.510
v6  0.796   0.519

               Factor1 Factor2
SS loadings      3.358   1.038
Proportion Var   0.560   0.173
Cumulative Var   0.560   0.733

Test of the hypothesis that 2 factors are sufficient.
The chi square statistic is 23.14 on 4 degrees of freedom.
The p-value is 0.000119

In this output, uniqueness is variability of variable minus communality (This is actually estimation of epsilon). Communality is diagnonal elements in LLT, i.e., amount of variability explained by common factors, i.e., F.

Below uniqueness, loadings express factor loading matrix. For example, v1 = 0.853 * Factor1 – 0.518 * Factor2. This is estimation of L in the equation.

SS loadings are sum of squares of factor loadings. For example, 3.358 = 0.853^2 + 0.804^2 + 0.598^2 + 0.508^2 + 0.857^2 + 0.796^2. This is amount of variance explained by factors. And in this case, 56%(=0.560) of variance is explained by Factor1.

Finally, there’s chi squre fit test to see if 2 factors explain m1. Here, as p-value 0.000119 < 0.05, it does not explain the data very well (as alternative hypothesis is accepted while our H0 is “2 factors are sufficient”). If we plot factor loadings, we can group variables easily based on factor affecting them, e.g.: [code lang="R"] > f = factanal(m1, factors=2, rotation=”none”) > plot(f$loadings) [/code]

In the above, we see there are two groups whose Factor1 is large while their Factor2 is different. This makes grouping difficult. To solve this problem, we can use factor rotation. In the below, I used varimax (this is default of factanal; I’ve specified rotation=”none” in the above on purpose).

> varimax(f$loadings)$loadings

Loadings:
   Factor1 Factor2
v1 0.971   0.228 
v2 0.917   0.213 
v3 0.429   0.418 
v4 0.363   0.355 
v5 0.254   0.965 
v6 0.205   0.928 

               Factor1 Factor2
SS loadings      2.206   2.190
Proportion Var   0.368   0.365
Cumulative Var   0.368   0.733

> plot(varimax(f$loadings)$loadings)

Or, we could have used factanal without specifying rotation to use varimax:

> factanal(m1, factors=2)

Varimax is orthogonal rotation. In other words, it rotates Factor1 and Factor2 axes while keeping their angle right(90 degree). However, there are other rotation like covarimin, quartimin, oblimin(this is popular one) which does not keep the right angle.

References:
1) Wikipedia. http://en.wikipedia.org/wiki/Factor_analysis
2) Machine learning lecture 13 by Andrew Ng. http://www.youtube.com/watch?v=LBtuYU-HfUg&feature=player_detailpage#t=1885s
3) 이용구, 김성수, 김현중, “다변량 분석 입문”, Knou Press.
4) help(factanal) in R.