On lattice reduction algorithms for solving weighted integer least squares problems: comparative study

Decorrelation or reduction theory deals with identifying appropriate lattice bases that aid in accelerating integer search to find the optimal integer solution of the weighted integer least squares problem. Orthogonality defect has been widely used to measure the degree of orthogonality of the reduced lattice bases for many years. This contribution presents an upper bound for the number of integer candidates in the integer search process. This upper bound is shown to be a product of three factors: (1) the orthogonality defect, (2) the absolute value of the determinant of the inverse of the generator matrix of the lattice, and (3) the radius of the search space raised to the power of the dimension of the integer ambiguity vector. Four well-known decorrelation algorithms, namely LLL, LAMBDA, MLAMBDA, and Seysen, are compared. Many simulated data with varying condition numbers and dimensions as well as real GPS data show that the Seysen reduction algorithm reduces the condition number much better than the other algorithms. Also, the number of integer candidates, before and after the reduction process, is counted for all algorithms. Comparing the number of integer candidates, condition numbers, and orthogonality defect reveals that reducing the condition number and the orthogonality defect may not necessarily result in decreasing the number of integer candidates in the search process. Therefore, contrary to the common belief, reducing the orthogonality defect and condition number do not always result in faster integer least squares estimation. The results indicate that LAMBDA and MLAMBDA perform much better in reducing the number of integer candidates than the other two algorithms, despite having a larger orthogonality defect and condition number in some cases. Therefore, these two algorithms can speed up the integer least squares estimation problem in general and the integer ambiguity resolution problem in particular.