Course Notes

Background reading in linear algebra

Background reading in linear and nonlinear least squares

Filtering via SVD

Linear System Identification


Least Squares, Singular Value Decomposition, Error Analysis, and Regularization

  1. Herve Abdi and Lynne J. Williams, ``Principal component analysis'' WIREs Computational Statistics, 2 (July/August 2010) 433-459. DOI: 10.1002/wics.101
  2. Frank Vanden Berghen, ``Levenberg-Marquardt algorithms vs Trust Region algorithms,'' IRIDIA, Universite Libre de Bruxelles e November 12, 2004
  3. James A. Cadzow, ``Signal Processing via Least Squares Error Modeling,'' IEEE ASSP Magazine, October 1990, 12-31.
  4. James A. Cadzow, ``Least Squares, Modeling and Signal Processing,'' Digital Signal Processing, 4 (1994) 2-20.
  5. James A. Cadzow, ``Total Least Squares, Matrix Enhancement, and Signal Processing,'' Digital Signal Processing, 4 (1994) 21-39.
  6. Emmanuel J. Candes, Michael B. Wakin, Stephen P. Boyd, ``Enhancing Sparsity by Reweighted l1 Minimization'' J. Fourier Anal. Appl. 14 (2008) 877–905 DOI 10.1007/s00041-008-9045-x
  7. Roberto Togneri, Estimation Theory for Engineers, 30th August 2005
  8. Maryam Fazel, Ting Kei Pong, Defeng Sun, and Paul Tseng, ``Hankel Matrix Rank Minimization with Applications to System Identification and Realization,'' SIAM Journal on Matrix Analysis and Applications, 34(3) (2013) 946-977.
  9. Jonathan Gillard, ``Cadzow's basic algorithm, alternating projections and singular spectrum analysis,'' Statistics and Its Interface, 3 (2010) 335-343.
  10. Jonathan Gillard, ``Principal Component Analysis,'' DOC493: Intelligent Data Analysis and Probabilistic Inference Lecture 15
  11. Philip E. Gill, ``What's new in active-set methods for nonlinear optimization?,'' Advances in Numerical Computation, Manchester University, July 5, 2011
  12. G.H. Golub, C. Reinsch, ``Singular Value Decomposition and Least Squares Solutions,'' Numer. Math. 14, 403--420 (1970) 403-420.
  13. Gene H. Golub and Charles F. Van Loan, ``An Analysis of the Total Least Squares Problem,'' SIAM J. Numer Anal. 17(6) (December 1980) 883-893.
  14. P.A. Janakiraman and S. Renganathan, ``Recursive Computation of Pseudo-inverse of Matrices,'' Automatica, 18(5) (1982) 631-633.
  15. Jennings, Statistics 512: Applied Linear Models Topic 3
  16. Richard M. Johnson, ``On a Theorem Stated By Eckart and Young,'' Psychometrika, 28(3) (1963) 259-263.
  17. Su-In Lee, Honglak Lee, Pieter Abbeel and Andrew Y. Ng ``Efficient L1 Regularized Logistic Regression,'' 2006, American Association for Artificial Intelligence
  18. Philippe Lemmerling, ``Structured Total Least Squares: Analysis, Algorithms, and Applications,'' Ph.D. dissertation, Katholieke Universiteit Leuven, May 1999.
  19. Chun-Lin Liu, ``A Tutorial of the Wavelet Transform,'' February 23, 2010
  20. Ivan Markovsky, ``Structured low-rank approximation and its applications,'' Automatica 44 (2008) 891 – 909
  21. Ivan Markovskya, Sabine Van Huffel, ``Overview of total least-squares methods,'' Signal Processing 87 (2007) 2283-2302
  22. In Jae Myung, ``Tutorial on maximum likelihood estimation,'' Journal of Mathematical Psychology 47 (2003) 90-100.
  23. Mark Richardson, ``Principal Component Analysis,'' May 2009
  24. Paul A. Samuelson, ``A Note on Alternative Regressions,'' Econometrica, 10(1) (Jan., 1942) 80-83.
  25. Mark Schmidt, Glenn Fung, Romer Rosaless, ``Optimization Methods for l1 - Regularization,'' UBC-TR-2009-19, Original: March 11, 2008. Revised: August 4, 2009
  26. Mark Schmidt, Glenn Fung, Romer Rosaless, ``Fast Optimization Methods for L1 Regularization: A Comparative Study and Two New Approaches,'' ECML 2007.
  27. Mark Schmidt, ``Least Squares Optimization with L1-Norm Regularization,'' CS542B Project Report, December 2005.
  28. Parikshit Shah, Badri Narayan Bhaskar, Gongguo Tang and Benjamin Recht, ``Linear System Identification via Atomic Norm Regularization,'' Proceedings of IEEE CDC,
  29. Jon Shlens , ``A Tutorial on Principal Component Analysis: Derivation, Discussion and Singular Value Decomposition,'' UCSD, 25 March 2003 | Version 1
  30. G.W. Stewart, ``On the Early History of the Singular Value Decomposition,'' SIAM Review, 35(4) (1993) 551-566.
  31. Petre Stoica and Per Ahgren ``Exact initialization of the recursive least-squares algorithm,'' Int. J. Adapt. Control Signal Process. 2002; 16:219-230 (DOI: 10.1002/acs.681).
  32. Tin-Yau Tam ``QR decomposition: History and its Applications,'' presentation, Mathematics and Statistics, Auburn University, December 17, 2010.
  33. Mark K. Transtrum , James P. Sethna, ``Improvements to the Levenberg-Marquardt algorithm for nonlinear least-squares minimization,'' Journal of Computational Physics, preprint, 2012
  34. Mark K. Transtrum, Benjamin B. Machta, and James P. Sethna ``Why are Nonlinear Fits to Data so Challenging?,'' PRL 104, 060201 (2010)
  35. Margaret H. Wright, ``The Interior-Point Revolution in Optimization: History, Recent Developments, and Lasting Consequences,'' Bulletin of the American Mathematical Society, 42(1) (2004) 39-56.

Linear Systems and Linear Filtering

  1. Asad A. Ali , A. M. D’Amato , M. S. Holzel , S. L. Kukreja, and Dennis S. Bernstein, ``Consistent Identification of Hammerstein Systems Using an Ersatz Nonlinearity,'' Proc. American Control Conferences, San Francisco, June 29 - July 01, 2011. 1242-1246.
  2. Khaled Aljanaideh, Benjamin J. Coffer, and Dennis S. Bernstein, ``Closed-Loop Identification of Unstable Systems Using Noncausal FIR Models,'' 2013 American Control Conference (ACC) Washington, DC, USA, June 17-19, 2013, 1672-1677.
  3. Karl Johan Astrom and Richard M. Murray, Feedback Systems: An Introduction for Scientists and Engineers, DRAFT v2.4a (16 September 2006) 2006
  4. Yalcin Bulut, Applied Kalman filter theory, PhD. Dissertation, Northeastern University, August, 2011.
  5. Dennis S. Bernstein and Wasin So, Some Explicit Formulas for the Matrix Exponential, IEEE Trans. Automatic Control, 38(8) (1993): 1228-1232.
  6. Chen, C-T, Introduction to Linear System Theory, Holt, Rienhart, and Winston, 1970
  7. Chen, C-T, Linear Systems Theory and Design, Oxford University Press, 1999.
  8. Natasha Devroye ``Wiener and Kalman Filtering,'' lecture notes, Univ. of Illinois, Chicago, ECE 531: Detection and Estimation, Spring 2011.
  9. Anthony M. Demato, Kenny S. Mitchell, Bruno O. S. Teixeira, and Dennis S. Bernstein, ``Semiparametric Identification of Hammerstein Systems Using Input Reconstruction and a Single Harmonic Input,'' Proc. IEEE Conference on Decision and Control, Dec 15-17, 2010, Atlanta GA. 6365-6370.
  10. Matthew S. Fledderjohn, Matthew S. Holzel, Harish J. Palanthandalam-Madapusi, Robert J. Fuentes, and Dennis S. Bernstein, ``A Comparison of Least Squares Algorithms for Estimating Markov Parameters,'' Proc. American Control Conference, Jun 30-July 02, 2010, Baltimore MD. ThB17.5, 3735-3740.
  11. Michel Gevers ``System Identification without Lennart Ljung: what would have been different?,'' in `Forever Ljung in System Identification', T. Glad and G. Hendeby Eds., Studentlitteratur, Lund, Sweden, 2006. 61-85.
  12. Elmer G. Gilbert, ``Controllability and Observability in Multivariable Control Systems,'' J. SIAM Control, A 2(1) (1963) 128-151.
  13. B. Gopinath, ``On the Identification of Linear Time-Invariant Systems from Input-Output Data,'' The Bell System Technical Journal, 48(5) (1969) 1101-1113.
  14. Jesse B. Hoagg and Dennis S. Bernstein, ``Nonminimum-Phase Zeros,'' IEEE Control Systems Magazine, June 2007, 45-57.
  15. Matthew S. Holzel, Asad A. Ali, and Dennis S. Bernstein, ``Input Richness and Zero Buffering in Time-Domain Identification,'' Proc. American Control Conference, Jun 30-July 02, 2010, Baltimore MD. ThA17.1, 2929-2934.
  16. Thomas Kailath, ``A View of Three Decades of Linear Filtering Theory,'' IEEE Trans. Information Theory, IT-20(2) (1974) 146-181.
  17. R.E. Kalman, ``Mathematical Description of Linear Dynamical Systems,'' J. SIAM Control, A 1(2) (1963) 152-192.
  18. R.E. Kalman and R.S. Bucy, ``A New Approach to Linear Filtering and Prediction Problems ,'' ASME J. Basic Eng'g, 82D(1) (1960) 35-45.
  19. R.E. Kalman and R.S. Bucy, ``New Results in Linear Filtering and Prediction Theory,'' ASME J. Basic Eng'g, 83D(1) (1961) 95-108.
  20. M. Kamrunnahar, B. Huang, D.G. Fisher, ``Estimation of Markov parameters and time-delay/interactor matrix,'' Chemical Engineering Science 55 (2000) 3353}3363
  21. Maria Isabel Ribeiro, ``Kalman and Extended Kalman Filters: Concept, Derivation and Properties,'' Institute for Systems and Robotics, Instituto Superior Tecnico, Av. Rovisco Pais, 1, 1049-001 Lisbon, Portugal, February, 2004.
  22. Anthony J. Tether, ``Construction of Minimal Linear State-Variable Models from Finite Input-Output Data'' IEEE Trans. on Automatic Control, AC15(4) (1970) 427-436.
  23. Tobin H. Van Pelt and Dennis S. Bernstein, ``Least Squares Identification Using mu-Markov Parameterizations,'' Proceedings of the 37th IEEE, Conference on Decision & Control, Tampa, Florida USA December 1998, WM04 14:20, 618-619.
  24. Norbert Wiener, Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications, John Wiley, 1949.

Eigensystem Realization

  1. P.V. Albuquerque, M. Holzel, and D.S. Bernstein, ``On the Equivalence of OKID and Time Series Identification for Markov-Parameter Estimation,'' April 5, 2009
  2. K.F. Alvin, A.N. Robertson, G.W. Reich, K.C. Park, ``Structural system identification: from reality to models,'' Computers and Structures, 81 (2003) 1149–1176
  3. Juan M. Caicedo, ``Practical Guidelines for the Natural Excitation Technique (NExT) and the Eigensystem Realization Algorithm (ERA) for Modal Identification using Ambient Vibration,'' Experimental Techniques, July/August 2011, 52 - 58.
  4. Neil E. Goodzeit and Minh Q. Phan, ``System Identification in the Presence of Completely Unknown Periodic Disturbances,'' Journal of Guidance, Control, and Dynamics, 23(2) (March-April 2000) 251-259.
  5. George H. James III, Thomas G. Carne, and James P. Lauffer, ``The Natural Excitation Technique (NExT) for Modal Parameter Extraction From Operating Wind Turbines,'' Sandia Report SAND92-1666.UC-261, February 1993.
  6. Jer-Nan Juang and Richard S. Pappa, ``An Eigensystem Realization Algorithm for Modal Prameter Identification and Model Reduction,'' J. Guidance, 8(5) (1985) 620-627.
  7. Jer-Nan Juang and Richard S. Pappa, ``Effects of Noise on Modal Parameters Identified by the Eigensystem Realization Algorithm,'' J. Guidance, 9(3) (1986) 294-303.
  8. Jer-Nan Juang, Minh Phan, Lucas G. Horta and Richard W. Longman, ``Identification of Observer/Kalman Filter Markov Parameters: Theory and Experiments,'' J. Guidance, Control, and Dynamics, 16(2) (1993) 320-329.
  9. Jer-Nan Juang and Richard W. Longman, ``Optimized System Identification,'' NASA/TM-1999-209711, 1999.

Subspace System Identification

  1. Huseyin Ackay, ``Frequency domain subspace-based identification of discrete-time power spectra from uniformly spaced measurements,'' Automatica 47(2) (2011) 363-367.
  2. Albert Benveniste and Laurent Mevel ``Nonstationary consistency of subspace methods.'' IEEE Transactions on Automatic Control (2007) 52(6): 974–984.
  3. Rune Brincker and Palle Andersen, ''Understanding Stochastic Subspace Identification,''
  4. Katrien De Cock and Bart De Moor, ``Subspace Identification Methods,'', in "Control Systems, Robotics, and Automation," vol. 1, section 5.5, UNESCO Encyclopedia of Life Support Systems, EOLSS Publishers, Oxford, UK. 2003. 933-979.
  5. Ivan Goethals, Kristiaan Pelckmans, Johan A. K. Suykens, and Bart De Moor ``Subspace Identification of Hammerstein Systems Using Least Squares Support Vector Machines,'' IEEE Trans. Automatic Control, 50(10) (2005) 1509-1519.
  6. Rolf Isermann, Marco Munchhof, Identification of Dynamic Systems An Introduction with Applications, Springer, 2011.
  7. Tohru Katayamaa,, Hidetoshi Kawauchia, Giorgio Piccib ``Subspace identification of closed loop systems by the orthogonal decomposition method,'' Automatica 41 (2005) 863-872.
  8. Karel J. Keesman ``System Identification: An Introduction'' Springer, 2011.
  9. Seth L. Lacy and Dennis S. Bernstein, ``Subspace Identification With Guaranteed Stability Using Constrained Optimization,'' IEEE Transactions on Automatic Control, 48(7) (July 2003) 1259-1263.
  10. Tomas McKelvey, Huseyin Akcay, and Lennart Ljung, ``Subspace-Based Multivariable System Identification from Frequency Response Data,'' IEEE Transactions on Automatic Control, 41(7) (July 1996) 960-979.
  11. Marc Moonen, Bart De Moor, Lieven Vandeberghe, and Joos Vandewalle, ``On- and Off-Line Identification of Linear State Space Models,'' Int. J. Control, 49(1) (1989) 219-232
  12. Eric Moulines, Pierre Duhamel, Jean-Franqois Cardoso, and Sylvie Mayrargue, ``Subspace Methods for the Blind Identification of Multichannel FIR Filters,'' IEEE Transactions on Signal Processing, 43(2) (February 1995)
  13. Harish J. Palanthandalam-Madapusi, Seth Lacy, Jesse B. Hoagg and Dennis S. Bernstein, ``Subspace-Based Identification for Linear and Nonlinear Systems,'' 2005 American Control Conference June 8-10, 2005. Portland, OR, USA ThB02.2 2320-2334.
  14. Bart Peeters and Guido De Roeck, ``Reference-Based Stochastic Subspace Identification for Output-Only Modal Analysis,'' Mechanical Systems and Signal Processing (1999) 13(6), 855-878
  15. Bart Peeters and Guido De Roeck, ``Stochastic System Identification for Operational Modal Analysis: A Review'' Journal of Dynamic Systems, Measurement, and Control, 123 (2001) 659-667. DOI: 10.1115/1.1410370
  16. S. Joe Qin, ``Subspace Identification Methods --- A Tutorial,'' Department of Chemical Engineering The University of Texas at Austin Austin, Texas 78712 February 9, 2004
  17. S. Joe Qin, ``An overview of subspace identification,'' Computers and Chemical Engineering 30 (2006) 1502-1513. doi:10.1016/j.compchemeng.2006.05.045
  18. Chris Rayner, ``Subspace Identification,'' (Tijl De Bie '05), TTT 2009, 14 July 2009.
  19. David Di Ruscio, ``Subspace System Identification: Theory and Applications,'' Telemark Institute of Technology, Porsgrunn, Norway, Lecture notes, January 1995.
  20. Michel Verhaegen and Patrick Dewilde, ``Subspace model identification: Part 1. The output-error state model identification class of algorithms,'' Int'l J. Control, 56(5) (1992) 1187-1210.
  21. Michel Verhaegen and Patrick Dewilde, ``Subspace model identification: Part 2. Analysis of the elementary output-error state-space model identification algorithm,'' Int'l J. Control, 56(5) (1992) 1211-1241.
  22. Michel Verhaegen, ``Subspace model identification: Part 3. Analysis of the ordinary output-error state-space model identification algorithm,'' Int'l J. Control, 58(3) (1993) 555-586.
  23. Michel Verhaegen, ``Identification of the Deterministic Part of MIMO State Space Models given Innovations Form from Input-Output Data,'' Automatica 30(1) (1994) 61-74.
  24. Peter Van Overschee and Bart De Moor ``Subspace Algorithms for the Stochastic Identification Problem,'' Automatica, 29(3) (1993) 649-660.
  25. Peter Van Overschee and Bart De Moor ``N4SID : Subspace Algorithms for the Identification of Combined Deterministic-Stochastic Systems,'' Automatica, 30(1) (1994) 75-93.
  26. Peter Van Overschee and Bart De Moor, ``A Unifying Theorem for Three Subspace System Identification Algorithms,'' Automatica, 31(12) (1995) 1853-1861.
  27. Peter Van Overschee and Bart De Moor, ``Subspace Identification for Linear Systems: Theory - Implementation - Applications,'' Kluwer Academic Publishers, 1996.
  28. Mats Viberg, ``Subspace-based Methods for the Identification Time-invariant Systems,'' Automatica, 31(12) (1995) 1835-1851.


  1. Jaganath Chandrasekar and Dennis S. Bernstein, ``Position Control Using Acceleration- Based Identification and Feedback With Unknown Measurement Bias,'' Journal of Dynamic Systems, Measurement, and Control 130 (January 2008) 014501-1-8
  2. M. De Angelis, H. Lus, R. Betti, R. W. Longman, ``Extracting Physical Parameters of Mechanical Models From Identified State-Space Representations,'' Journal of Applied Mechanics 69 (September 2002) 617-625.
  3. Sven-Erik Rosenow and Palle Andersen, ``Operational Modal Analysis of a Wind Turbine Mainframe using Crystal Clear SSI''
  4. A. Sestieri and W. D’Ambrogio, ``Frequency Response Function Versus Output-Only Modal Testing Identification,'' Proc. IMAC XXXI ,
  5. Jian-Huang Weng, Chin-Hsiung Loh, Jerome P. Lynch, Kung-Chun Lu, Pei-Yang Lin, Yang Wang, ``Output-only modal identification of a cable-stayed bridge using wireless monitoring systems,'' Engineering Structures, 30 (2008) 1820–1831
  6. Pengyu Zhang, ``Experimental Modal Analysis of Star-Spar Buoy Using Eigensystem Realization Algorithm and Stochastic Subspace Identification Methods,'' MS Thesis, Ocean Engineering, University of Rhode Island, 2012.

© 2013-2017 Henri P. Gavin; updated: 2013-11-11; 2015-06-25; 2017-08-30