How to Validate Your Model Empirically (Part 1)

In the past two years, I put a lot of effort into empirically testing models and had many failed attempts. Some were just boring instead of wrong, but they helped me learn the nature of modeling and led to better attempts. This post shares tips on how to empirically test or validate models based on experience in cultural evolution and complex systems.

There are two ways to validate a model empirically: validate the assumptions or validate the results. If resources are limited, it can be better to start from one end.

Part 1: Testing the results

In a perfect world, we measure every component in the model, estimate parameters, calibrate the model, and then test whether the simulated pattern matches reality. Epidemiology models during COVID-19 are a good example of this kind of calibration with daily data.

In the social sciences, we rarely get timely measures for every component. We can still measure some components, make informed assumptions from prior empirical research, and test simulated results against real-world data. Engineers and public health researchers often do this; it is less common in the social sciences.

Sometimes we cannot recover the model process in the empirical world because it is too long, too short, or too complex to observe. In these cases, we can make qualitative predictions about how independent variables affect dependent variables. Experiments can help by controlling the independent variable.

Cross-cultural studies can also support modeling results. We can use model predictions to guide new cross-cultural tests of hypotheses.

One hard problem is validating self-censorship models: it is easy to observe what people say, but difficult to measure what they do not say. Two approaches are to compare discussion across events with different risk and norm salience, or to measure how people use alternative language to avoid censorship.

Source: Self-made Modeler