The current paper discusses the applicability of Gaussian process regressions, also known as Kriging models, in the context of structural and reliability analysis. Due to their flexibility these models appear in the field of structural analysis in many forms. Applications to approximate limit state functions, replace the computational expensive codes that solves the dynamic of complex systems, or replicate stochastic fields can be identified. Due to this fact, a discussion on the different parameters that depend on the implementation procedure chose to use these model is presented in the current paper. Design of experiments, polynomial approximation, correlation function, hyperparameters convergence and estimation function are the main global variables analysed. When implementing a Gaussian regression or Kriging model, the user is faced with the choice of these before any further progress. The discussion presented complements previous works on the implementation of such models in the sense that it focus on the structural analysis application and on how these parameters influence the accuracy. It is shown that depending on the approximation, significant advantage can be taken from understanding these major variables. Different examples are presented to support the understanding of the problem and the main conclusions on the applicability of the Gaussian regression models as surrogates for structural analysis are drawn.