Blog | Qiankun Zhong

How to Validate Your Model Empirically (Part 1)

Tue, 24 Sep 2024 00:00:00 +0000

In the past two years, I put a lot of effort into empirically testing models and had many failed attempts. Some were just boring instead of wrong, but they helped me learn the nature of modeling and led to better attempts. This post shares tips on how to empirically test or validate models based on experience in cultural evolution and complex systems.

There are two ways to validate a model empirically: validate the assumptions or validate the results. If resources are limited, it can be better to start from one end.

Part 1: Testing the results

In a perfect world, we measure every component in the model, estimate parameters, calibrate the model, and then test whether the simulated pattern matches reality. Epidemiology models during COVID-19 are a good example of this kind of calibration with daily data.

In the social sciences, we rarely get timely measures for every component. We can still measure some components, make informed assumptions from prior empirical research, and test simulated results against real-world data. Engineers and public health researchers often do this; it is less common in the social sciences.

Sometimes we cannot recover the model process in the empirical world because it is too long, too short, or too complex to observe. In these cases, we can make qualitative predictions about how independent variables affect dependent variables. Experiments can help by controlling the independent variable.

Cross-cultural studies can also support modeling results. We can use model predictions to guide new cross-cultural tests of hypotheses.

One hard problem is validating self-censorship models: it is easy to observe what people say, but difficult to measure what they do not say. Two approaches are to compare discussion across events with different risk and norm salience, or to measure how people use alternative language to avoid censorship.

Source: Self-made Modeler

One Tiny Step Further in Model Learning

Fri, 05 Feb 2021 00:00:00 +0000

In the past several months, I have been trying to develop a model with my advisor Cristina Moya to capture both the identity-driven polarization and the biased social learning process of public discussion dynamics. The model-developing process is challenging but it has been surprisingly helpful for me to understand the aesthetics of a good model and the meaning of true emergence. I summarized a few tips for model learners who are stuck.

Read a lot of models

We all know the importance of reading other models but when you’re new in the modeling world, it is hard to know what to read. My suggestion is to start with one book in your field that summarizes the classical models you need to know (in my case, it’s Culture and the Evolutionary Processes, the only book I brought with me during the summer vacation). Question-driven reading is more effective. For example, I’m interested in both identity signaling and information transmission and thus I read about recent signaling models, coordination models, social learning models, and their empirical applications. Further on, I realized that I don’t have to be bounded to the topic but focus on the process, so I started to read models that can generate runaway processes. A large amount of literature is fundamental to cultivating a “sense of modeling”.

Understand the real mechanisms

Sometimes we need to focus on specific aspects when reading other people’s papers. The most important thing is not to see how realistic their assumptions are (other modelers may disagree) but to understand which mechanisms produce their results. Sometimes, modeling papers might make bigger claims than they actually can because the mechanisms are not explicated clearly. I often think this is a great advantage of modeling compared to verbal theories. Because once the model is written, you will be able to understand the weirdest result produced through it.

Annotated literature reviews are great, but have you tried annotated model reviews? 😉

Make analogies

Once you understand the mechanisms and you’re eager to adapt and extend, it’s time to make analogies from one abstract domain to another abstract domain. You need to resist the temptations of just writing down inspirations and philosophies from the models you just learned and started to think about the meanings of specific variables and processes. Sometimes it gets really weird when the contexts are very far from each other. In my case, it’s when I want to map sexual selection into sender-receiver interaction, or when I think of the actual cost to fitness when people chase misinformation. How can they work? Isn’t that too arbitrary? But actually, this is the space you can use your creativity to write more context-related process or connect it with field-specific theories.

Do both the analytical solution and computational simulation (the drift can do much)

The next step is to write up the model! There are plenty of materials out there to help you write down a mathematical model (check out Mathematical Models of Social Evolution and a friend’s recommendation Matrix Population Models) or code up a computational model (Introduction to ABM course). In my opinion, computational models are necessary for understanding social and evolutionary processes. In my project, I didn’t realize that any tiny drift can lead to the instability of one group of the population before I did agent-based modeling, and now it has become a core question in understanding identity-driven processes. The computational models not only add stochasticity but also test whether the system is resilient to stochasticity. Results possible in numerical simulation might not be possible in computational simulation.

Explain to people who don’t model

Pretty much the Feynman method. Or better, present to an academic audience who don’t model. The field of communication I interact with is mostly empirical. The advantage of presenting your model to an empirical audience is that you can learn what you don’t know about your model and get a lot of specific questions with specific examples. I believe these empirical questions are necessary, especially for you to think further about the applicability of your model and how likely your model is telling some obvious fact that does not require a modeling mind. To do this type of presentation, you need to at least prepare a concise answer to the question “Why do you need to build a model”? Because you will get that question for sure.

System Robustness and Censorship

Mon, 13 Apr 2020 00:00:00 +0000

I have an idea that comes from a biological phenomenon of system robustness. If the system does not detect and correct errors, accumulated error reduces performance. If the system is too sensitive to errors, it cannot remain robust.

During COVID-19, many governments used information control. If the news agency is an error-detection mechanism, then to maintain robustness and stability, the system may reduce sensitivity and control how much information people can access. Otherwise, panic could lead to hoarding and preempting resources.

If people do not get enough information, they cannot correctly perceive how dangerous the situation is, which can speed contagion and increase damage. Over time, the government may lose trust, which also reduces robustness.

This suggests a need to model information control and its influence on individual behavior and collective patterns. The key question is the equilibrium amount of information a government will allow releasing to maintain robustness.

Source: Self-made Modeler

Shall We Play this Game?

Wed, 01 Apr 2020 00:00:00 +0000

This machine learning idea raises a question: when does an algorithm understand the logic of a game? It is the same question I ask when watching people play video games: is there a point between getting familiar with a skill and understanding the logic?

One view is through information theory. The golden mean process shows that block entropy can reveal structure at certain lengths. At length 2, the system realizes the logic: there shall not be consecutive 0s.

Another view is through learning. Social learning is puzzling because it is hard to tell if we do what our peers do or if we are learning consequences they already discovered. I want to build a model where agents follow local rules but can generalize after moving across environments, using memory and adaptation.

A third view is humanistic. We often say a musician shows sophistication in how they play Bach, and creativity emerges from repeated practice. How do we examine creativity? It might be best left to creative people.

Bourdieu might say all social processes are status games. If you understand the game, do you learn how to escape it?

Source: Self-made Modeler