Skip to main content
SpringerLink
Log in

Approximating Networks and Extended Ritz Method for the Solution of Functional Optimization Problems

  • Published:
Journal of Optimization Theory and Applications Aims and scope Submit manuscript

Abstract

Functional optimization problems can be solved analytically only if special assumptions are verified; otherwise, approximations are needed. The approximate method that we propose is based on two steps. First, the decision functions are constrained to take on the structure of linear combinations of basis functions containing free parameters to be optimized (hence, this step can be considered as an extension to the Ritz method, for which fixed basis functions are used). Then, the functional optimization problem can be approximated by nonlinear programming problems. Linear combinations of basis functions are called approximating networks when they benefit from suitable density properties. We term such networks nonlinear (linear) approximating networks if their basis functions contain (do not contain) free parameters. For certain classes of d-variable functions to be approximated, nonlinear approximating networks may require a number of parameters increasing moderately with d, whereas linear approximating networks may be ruled out by the curse of dimensionality. Since the cost functions of the resulting nonlinear programming problems include complex averaging operations, we minimize such functions by stochastic approximation algorithms. As important special cases, we consider stochastic optimal control and estimation problems. Numerical examples show the effectiveness of the method in solving optimization problems stated in high-dimensional settings, involving for instance several tens of state variables.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. HO, Y. C., and CHU, K. C., Team Decision Theory and Information Structures in Optimal Control Problems, IEEE Transactions on Automatic Control, Vol. 17, pp. 15-28, 1972.

    Google Scholar 

  2. SAGE, A. P., Optimum Systems Control, Prentice-Hall, New York, NY, 1968.

    Google Scholar 

  3. MäKILä, P. M., and TOIVONEN, H. T., Computational Methods for Parametric LQ Problems: A Survey, IEEE Transactions on Automatic Control, Vol. 32, pp. 658-671, 1987.

    Google Scholar 

  4. PARISINI, T., and ZOPPOLI, R., Neural Approximations for Multistage Optimal Control of Nonlinear Stochastic Systems, IEEE Transactions on Automatic Control, Vol. 41, pp. 889-895, 1996.

    Google Scholar 

  5. PARISINI, T., SANGUINETI, M., and ZOPPOLI, R., Nonlinear Stabilization by Receding-Horizon Neural Regulators, International Journal of Control, Vol. 70, pp. 341-362, 1998.

    Google Scholar 

  6. PARISINI, T., and ZOPPOLI, R., Neural Approximations for Infinite-Horizon Optimal Control of Nonlinear Stochastic Systems, IEEE Transactions on Neural Networks, Vol. 9, pp. 1388-1408, 1998.

    Google Scholar 

  7. ALESSANDRI, A., BAGLIETTO, M., PARISINI, T., and ZOPPOLI, R., A Neural State Estimator with Bounded Errors for Nonlinear Systems, IEEE Transactions on Automatic Control, Vol. 44, pp. 2028-2042, 1999.

    Google Scholar 

  8. BEARD, R. W., and MCLAIN, T. W., Successive Galerkin Approximation Algorithms for Nonlinear Optimal and Robust Control, International Journal of Control, Vol. 71, pp. 717-743, 1998.

    Google Scholar 

  9. SJöBERG, J., ZHANG, Q., LJUNG, L., BENVENISTE, A., GLORENNEC, P. Y., DELYON, B., HJALMARSSON, H., and JUDITSKY, A., Nonlinear Black-Box Modeling in System Identification: A Unified Oûerview, Automatica, Vol. 31, pp. 1691-1724, 1995.

    Google Scholar 

  10. LESHNO, M., LIN, V. YA., PINKUS, A., and SCHOCKEN, S., Multilayer Feedforward Networks with a Nonpolynomial Activation Function Can Approximate Any Function, Neural Networks, Vol. 6, pp. 861-867, 1993.

    Google Scholar 

  11. GIROSI, F., Regularization Theory, Radial Basis Functions, and Networks, From Statistics to Neural Networks: Theory and Pattern Recognition Applications, Edited by J. H. Friedman, V. Cherkassky, and H. Wechsler, Computer and Systems Sciences Series, Springer Verlag, Berlin, Germany, pp. 166-187, 1993.

    Google Scholar 

  12. PARK, J., and SANDBERG, I. W., Universal Approximation Using Radial-Basis-Function Networks, Neural Computation, Vol. 3, pp. 246-257, 1991.

    Google Scholar 

  13. GELFAND, I. M., and FOMIN, S. V., Calculus of Variations, Prentice Hall, Englewood Cliffs, New Jersey, 1963.

    Google Scholar 

  14. SIRISENA, H. R., and CHOU, F. S., Convergence of the Control Parametrization Ritz Method for Nonlinear Optimal Control Problems, Journal of Optimization Theory and Applications, Vol. 29, pp. 369-382, 1979.

    Google Scholar 

  15. ATTOUCH, H., Variational Convergence for Functions and Operators, Pitman Publishing, London, England, 1984.

    Google Scholar 

  16. PINKUS, A., n-Widths in Approximation Theory, Springer Verlag, Berlin Heidelberg, Germany, 1985.

    Google Scholar 

  17. BARRON, A. R., Universal Approximation Bounds for Superpositions of a Sigmoidal Function, IEEE Transactions on Information Theory, Vol. 39, pp. 930-945, 1993.

    Google Scholar 

  18. GIROSI, F., JONES, M., and POGGIO, T., Regularization Theory and Neural Networks Architectures, Neural Computation, Vol. 7, pp. 219-269, 1995.

    Google Scholar 

  19. GIULINI, S., and SANGUINETI, M., On Dimension-Independent Approximation by Neural Networks and Linear Approximators, Proceedings of the International Joint Conference on Neural Networks, Como, Italy, pp. 283-288, 2000.

  20. KůRKOVá, V., and SANGUINETI, M., Comparison of Worst Case Errors in Linear and Neural Network Approximation, IEEE Transactions on Information Theory, Vol. 48, 2002.

  21. KůRKOVá, V., and SANGUINETI, M., Bounds on Rates of Variable-Basis and Neural-Network Approximation, IEEE Transactions on Information Theory, Vol. 47, pp. 2659-2665, 2001.

    Google Scholar 

  22. KAINEN, P. C., KůRKOVá, V., and VOGT, A., Approximation by Neural Networks Is Not Continuous, Neurocomputing, Vol. 29, pp. 47-56, 1999.

    Google Scholar 

  23. DEVORE, R., HOWARD, R., and MICCHELLI, C., Optimal Nonlinear Approximation, Manuskripta Mathematica, Vol. 63, pp. 469-478, 1989.

    Google Scholar 

  24. KUSHNER, H. J., and YIN, G. G., Stochastic Approximation Algorithms and Applications, Springer Verlag, New York, NY, 1997.

    Google Scholar 

  25. ERMOLIEV, YU., and WETS, J. B., Editors, Numerical Techniques for Stochastic Optimization, Springer Verlag, Heidelberg, Germany, 1988.

    Google Scholar 

  26. PAPAGEORGIOU, M., Applications of Automatic Control Concepts to Traffic Flow Modeling and Control, Lecture Notes in Control and Information Sciences, Springer Verlag, New York, NY, 1983.

    Google Scholar 

  27. MESSNER, A., and PAPAGEORGIOU, M., Motorway Network Control ûia Nonlinear Optimization, Proceedings of the 1st Meeting of the EURO Working Group on Urban Traffic and Transportation, Landshut, Germany, pp. 1-24, 1992.

  28. PARISINI, T., and ZOPPOLI, R., A Receding-Horizon Regulator for Nonlinear Systems and a Neural Approximation, Automatica, Vol. 31, pp. 1443-1451, 1995.

    Google Scholar 

  29. CHEN, V. C. P., RUPPERT, D., and SHOEMAKER, C. A., Applying Experimental Design and Regression Splines to High-Dimensional Continuous-State Stochastic Dynamic Programming, Operations Research, Vol. 47, pp. 38-53, 1999.

    Google Scholar 

  30. BAGLIETTO, M., CERVELLERA, C., PARISINI, T., SANGUINETI, M., and ZOPPOLI, R., Approximating Networks, Dynamic Programming, and Stochastic Approximation, Proceedings of the American Control Conference, Chicago, Illinois, pp. 3304-3308, 2000.

  31. ALESSANDRI, A., PARISINI, T., and ZOPPOLI, R., Neural Approximations for Nonlinear Finite-Memory State Estimation, International Journal of Control, Vol. 67, pp. 275-302, 1997.

    Google Scholar 

  32. CHAN, Y. T., HU, A. G. C., and PLANT, J. B., A Kalman Filter Based Tracking Scheme with Input Estimation, IEEE Transactions on Aerospace and Electronic Systems, Vol. 15, pp. 237-244, 1978.

    Google Scholar 

  33. AIDALA, V. J., Kalman Filter Behavior in Bearings-Only Tracking Applications, IEEE Transactions on Aerospace and Electronic Systems, Vol. 14, pp. 29-39, 1979.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Communications, Computer, and System Sciences, University of Genova, Genova, Italy

    R. Zoppoli (Professor)

  2. Department of Communications, Computer, and System Sciences, University of Genova, Genova, Italy

    M. Sanguineti (Research Associate)

  3. Department of Electrical, Electronic Engineering and Computer Engineering, DEEI-University of Trieste, Trieste, Italy

    T. Parisini (rofessor)

Authors
  1. R. Zoppoli
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. M. Sanguineti
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. T. Parisini
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Zoppoli, R., Sanguineti, M. & Parisini, T. Approximating Networks and Extended Ritz Method for the Solution of Functional Optimization Problems. Journal of Optimization Theory and Applications 112, 403–440 (2002). https://doi.org/10.1023/A:1013662124879

Download citation

  • Issue Date:

  • DOI: https://doi.org/10.1023/A:1013662124879

Search

Navigation