MeMa - Methodology Matters
Seminar Cycle

Prof. Jaak Billiet (Emeritus professor KU Leuven - CeSO)
Some reflections on measurement and theoretical validity in comparative research with cross-country or/and over time survey data

20 May 2016, 10.30
Room A
Graduate School in Social and Political Sciences
Via Pace, 10 - Milan

Abstract

Social scientists are interested in societies, institutions, and the social networks in which individuals interact. They believe that these contexts influence characteristics, attitudes, choices and the behaviour of individuals. There is a long-standing tradition of cross-nation research, especially in comparative political analysis in which political systems (e.g. nations) are analysed as cases or used as context. In the most general terms, comparative social research refers to research designs by which data from different societies or data from particular societies at different time periods are collected and compared (Allardt, 1990: 183).
The nature of research questions that are subject to comparative quantitative analysis has been drastically changed due to the availability of comparable micro data collected in many countries (e.g. ESS, ISSP, EVS, SHARE). In interaction with this evolution in available data, statistical analysis of hierarchical data has expanded enormously. After nearly two decades of engagement in the collection and analysis of large scale survey data, I became more and more concerned about the gap between the expansion, and accessibility of sophisticated statistical tools for analysis, and a relative stagnation in theoretical and conceptual development. I have the impression that the low explanatory power of hierarchical explanatory models are often more related to conceptual (theoretical) weakness than to the tools for analysis and model testing that have been used.
I will reflect on several of the conceptual problems that are at various stages of a research project: operationalisation, measurement, measurement equivalence, and assumptions behind model building and inference. The leading idea behind my examples is by my reckoning the crucial distinction between measurement validity and theoretical (or concept) validity. Much methodological work is restricted to the assessment of measurement validity and thereby disregarding the assumptions and theoretical concepts behind the measured concepts. I will try to give a flavour of this proposition by considering a handful of examples derived from different stages in the research process: the measurement of a construct; equivalence testing; the bridging assumptions (or the micro-macro link in hierarchical models); the omission of the social context in making inferences.

References
Billiet, J. (2013). Quantitative methods with survey data in comparative research. In: P. Kenneth (Ed.) (pp. 264-300). A Handbook of Comparative Social Policy. (2nd Edition). Cheltenham: Edward Elgare.
Billiet, J., Meuleman, B., Davidov, E. (2015). Some methodological challenges of cross-national social research: conceptual and measurement validity. In: Sztabinsky P., Domanski H., Sztabinsky F. (Eds.), Hopes and anxieties. Six waves of the European Social Survey (pp. 99-120). Frankfurt am Main: Peter Lang.