A Study of the Effectiveness of Model Selection Criteria for Multilevel Analysis
Abstract
This research aimed to study the effectiveness of model selection criteria for multilevel analysis. Both criteria were Akaike’s Information Criteria (AIC) and Bayesian Information Criteria (BIC). The simulation was applied by Monte Carlo technique. The conditions for simulation were 1) populations were having normal distribution; 2) populations were having negative skewness and platykurtic distributions; 3) populations were having positive skewness and leptokurtic distributions; 4) independent variables were divided into two variables for each level; 5) intraclass correlation coefficient were 0.10 and 0.20; and 6) the sample size was divided into four sizes for each level (level 1, the group sizes were 5, 15, 30 and 50; level 2 , the number of groups were 15, 30, 50 and 100.). Each condition was simulated with 1,000 data set. The results revealed that when the number of groups was small, the effectiveness of model selection criteria was considerable low. Whereas the higher the number of groups, the better effectiveness of criteria. Furthermore, when the number of groups was 100, it was found in most cases that the model selection of BIC yielded better effective than AIC when estimating parameter with Restricted Maximum Likelihood (RML). Keywords : Multilevel analysis, Hierarchical Linear Models, AIC, BICReferences
Akaike, H. (1973). Information Theory and an Extension of the Maximum Likelihood Principle. In 2nd International Symposium on Information Theory. (pp. 267-281). N.N. Petrov and F. Csaki, eds. Akademiai Kiado, Budapest.
Boonsuk, N. & Jitthavech, J. (2014). Criterion and Test Statistics for Selecting Multiple Linear Regression Models without Full Model. KMUTT Research and Development Journal, 37(2), 227-238. (in Thai)
Burstein, L. (1980). Analysis of Multilevel Data in Education Research and Evaluation. In Berliner (Ed) Review of Research Association.
Bates, D., Machler, M., Bolker, B.M. & Walker S.C. (2015). Fitting Linear Mixed-Effects Models Using Ime4. Journal of Statistical Software, 67(1), 1-48, Doi: 10.18637/jss.v067.i01.
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. New Jersey: Lawrence Erlbaum Associates.
Goldstein, H. (1987). Multilevel Models in Educational and Social Research. Oxford: Oxford University Press.
Goldstein, H. (1995). Multilevel statistical models. (2nd ed). Newyork-Toronto: John Wiley & Sons.
Hox, J.J. (1998). Multilevel modeling: When and why. In I. Balgerjahn, R. Mathar, & M. Schader (Eds.), Classification, data analysis, and data highways. (pp. 147-154). New York: Springer Verlag.
Hox, J.J. (2010). Multilevel Analysis Techniques and Applications. (2nd ed). New York: Routledge.
Kajornsin, K. (2012). Hierarchical Linear Modeling. Bangkok: P.S. Print. 327 p. (in Thai)
Kanjanawasee, S. (2007). MULTI-LEVEL ANALYSIS. (4th ed). Bangkok: Chulalongkorn University Printing House. (in Thai)
Kreft, I.G.G. (1996). Are multilevel techniques necessary? An overview, including simulation studies. Unpublished Report, California State University, Los Angeles. Retrieved July 16, 2018, from http://eric.ed.gov.
Kreft, I.G.G & de Leeuw, J. (1998). Introducing multilevel modeling. Newburg Park, CA: Sage.
Longford, N.T. (1993). Random coefficient models. Oxford, UK: Clarendon Press.
Maas, C.J.M., & Hox, J.J. (2004). Robustness of multilevel parameter estimates against small sample sizes. Department of Methodology and Statistics, Utrecht University.
Muller, S., Scealy J.L. & Welsh, A.H. (2013). Model Selection in Linear Mixed Models. Statistical Science, 28(2), 135-167 DOI: 10.1214/12-STS410
Montgomery, D.C., Peak, E.A., & Vining, G.G. (2006). Introduction to Linear Regression Analysis. (4th ed). New York: Wiley.
Na Bangchang, K. & Jitthavech, J. (2013). A Variable Selection in Multiple Linear Regression Models Based on Tabu Search. KKU Science Journal, 41(1), 250-261. (in Thai)
Ramberg, J.S., Dudewicz, E.J., Tadikamalla, P.R., & Mykytka, E.F. (1979). A probability distribution and its uses in fitting data. Technometrics, 21, 201-214.
Raudenbush, S.W., & Bryk, A.S. (2002). A Hierarchical Linear Mode: Applications and Data Analysis Methods. (2nd ed). California: Sage Publications.
Seenoi, P. & Jitthavech, J. (2010). Test Statistics for Selecting Multiple Linear Regression Models. Burapha Science Journal, 15(2), 47-56. (in Thai)
Snijder, T. & Bosker, R. (1999). Multilevel analysis. London: Sage Publications. Inc.
Sawa, T. (1978). Information Criteria for Discriminating among Alternative Regression Model. Journal of Econometrica, 46, 1273-1282.
Songthong, M. (2013). The Study of Efficiency of Parameter Estimation Methods in Multilevel Analysis of Small Sample Groups. Burapha Science Journal, 18(1), 116-124. (in Thai)
Taesombut, S. (2005). Regression analysis. (3rd ed.), Bangkok: Kasetsart University Press. 294 p. (in Thai)
Wiratchai, N. & Pitiyanuwat, S. (2000). Nature of Educational Sciences and Educational Research Methodology.
Journal of Research Methodology: JRM, 13(2), 34-72. (in Thai)
Boonsuk, N. & Jitthavech, J. (2014). Criterion and Test Statistics for Selecting Multiple Linear Regression Models without Full Model. KMUTT Research and Development Journal, 37(2), 227-238. (in Thai)
Burstein, L. (1980). Analysis of Multilevel Data in Education Research and Evaluation. In Berliner (Ed) Review of Research Association.
Bates, D., Machler, M., Bolker, B.M. & Walker S.C. (2015). Fitting Linear Mixed-Effects Models Using Ime4. Journal of Statistical Software, 67(1), 1-48, Doi: 10.18637/jss.v067.i01.
Cohen, J. (1988). Statistical Power Analysis for the Behavioral Sciences. New Jersey: Lawrence Erlbaum Associates.
Goldstein, H. (1987). Multilevel Models in Educational and Social Research. Oxford: Oxford University Press.
Goldstein, H. (1995). Multilevel statistical models. (2nd ed). Newyork-Toronto: John Wiley & Sons.
Hox, J.J. (1998). Multilevel modeling: When and why. In I. Balgerjahn, R. Mathar, & M. Schader (Eds.), Classification, data analysis, and data highways. (pp. 147-154). New York: Springer Verlag.
Hox, J.J. (2010). Multilevel Analysis Techniques and Applications. (2nd ed). New York: Routledge.
Kajornsin, K. (2012). Hierarchical Linear Modeling. Bangkok: P.S. Print. 327 p. (in Thai)
Kanjanawasee, S. (2007). MULTI-LEVEL ANALYSIS. (4th ed). Bangkok: Chulalongkorn University Printing House. (in Thai)
Kreft, I.G.G. (1996). Are multilevel techniques necessary? An overview, including simulation studies. Unpublished Report, California State University, Los Angeles. Retrieved July 16, 2018, from http://eric.ed.gov.
Kreft, I.G.G & de Leeuw, J. (1998). Introducing multilevel modeling. Newburg Park, CA: Sage.
Longford, N.T. (1993). Random coefficient models. Oxford, UK: Clarendon Press.
Maas, C.J.M., & Hox, J.J. (2004). Robustness of multilevel parameter estimates against small sample sizes. Department of Methodology and Statistics, Utrecht University.
Muller, S., Scealy J.L. & Welsh, A.H. (2013). Model Selection in Linear Mixed Models. Statistical Science, 28(2), 135-167 DOI: 10.1214/12-STS410
Montgomery, D.C., Peak, E.A., & Vining, G.G. (2006). Introduction to Linear Regression Analysis. (4th ed). New York: Wiley.
Na Bangchang, K. & Jitthavech, J. (2013). A Variable Selection in Multiple Linear Regression Models Based on Tabu Search. KKU Science Journal, 41(1), 250-261. (in Thai)
Ramberg, J.S., Dudewicz, E.J., Tadikamalla, P.R., & Mykytka, E.F. (1979). A probability distribution and its uses in fitting data. Technometrics, 21, 201-214.
Raudenbush, S.W., & Bryk, A.S. (2002). A Hierarchical Linear Mode: Applications and Data Analysis Methods. (2nd ed). California: Sage Publications.
Seenoi, P. & Jitthavech, J. (2010). Test Statistics for Selecting Multiple Linear Regression Models. Burapha Science Journal, 15(2), 47-56. (in Thai)
Snijder, T. & Bosker, R. (1999). Multilevel analysis. London: Sage Publications. Inc.
Sawa, T. (1978). Information Criteria for Discriminating among Alternative Regression Model. Journal of Econometrica, 46, 1273-1282.
Songthong, M. (2013). The Study of Efficiency of Parameter Estimation Methods in Multilevel Analysis of Small Sample Groups. Burapha Science Journal, 18(1), 116-124. (in Thai)
Taesombut, S. (2005). Regression analysis. (3rd ed.), Bangkok: Kasetsart University Press. 294 p. (in Thai)
Wiratchai, N. & Pitiyanuwat, S. (2000). Nature of Educational Sciences and Educational Research Methodology.
Journal of Research Methodology: JRM, 13(2), 34-72. (in Thai)
Downloads
Published
2019-01-21
Issue
Section
Research Article
