Questions & explanations
1. Compare the latent factor models learned by matrix factorization and Variational Autoencoders in terms of interpretability.
Matrix factorization learns deterministic latent factors for each user and item, which can sometimes be interpreted as features like 'action vs. drama' or 'mainstream vs. niche'. However, these factors are not always easy to interpret. Variational Autoencoders (VAEs) learn a probabilistic distribution over latent factors, meaning each user's embedding is a mean and variance vector. This adds uncertainty but makes interpretation harder because the latent space is continuous and not aligned with human concepts. VAEs also learn a non-linear mapping from the latent space to interactions, so the meaning of each latent dimension is less clear. In practice, matrix factorization factors are slightly more interpretable, but both are often used as black-box models.
2. Give an example of how a policy gradient method can be used to personalize problem difficulty.
A policy gradient method directly learns a policy (a probability distribution over actions) without using a value function. In personalizing problem difficulty, the policy might output probabilities for easy, medium, or hard problems based on the student's state. The agent samples an action (e.g., medium difficulty) and receives a reward (e.g., +1 if the student solves it correctly and shows improvement). The policy is updated to increase the probability of actions that led to higher rewards. For example, if giving a hard problem when the student is struggling leads to frustration and low reward, the policy will reduce the chance of selecting hard problems in similar states. This allows the system to adapt difficulty continuously.
3. How does Q-learning work in the context of a tutoring system?
Q-learning is a model-free RL algorithm that learns a Q-value for each state-action pair, representing the expected total future reward. In a tutoring system, the state could be the student's knowledge profile, and actions are possible teaching moves (e.g., give a hint, present a problem). The agent starts with random Q-values and updates them using the Bellman equation after each interaction. For example, if the agent gives a hint and the student then answers correctly, the Q-value for that state-hint pair increases. Over many students, the Q-values converge to optimal values, and the agent can choose the action with the highest Q-value in each state. This enables the system to make decisions that maximize long-term learning.
4. Compare randomized controlled trials (RCTs) and observational studies for evaluating personalization.
Randomized controlled trials (RCTs) randomly assign students to personalized or standard instruction, which eliminates confounding and provides the strongest causal evidence. However, RCTs can be expensive, impractical, or unethical (e.g., denying beneficial treatment). Observational studies use existing data without randomization, so they are cheaper and more feasible, but prone to bias from confounding variables. Causal inference methods like propensity score matching or IV can reduce bias in observational studies, but they rely on strong assumptions. For example, an RCT might show that personalized math software improves scores by 10%, while an observational study might show 15% due to selection bias. Both have trade-offs.
5. Compare Thompson Sampling and UCB in terms of handling uncertainty.
Thompson Sampling handles uncertainty by sampling from a posterior distribution, which naturally incorporates both the mean and variance of the estimate. It is stochastic, meaning it can randomly explore even when one item appears best. UCB handles uncertainty by adding a deterministic exploration bonus based on the number of times an item has been chosen. Thompson Sampling tends to be more efficient in practice because it explores in a more informed way, especially with complex reward distributions. UCB is simpler and works well for stationary problems. Both algorithms balance exploration and exploitation, but Thompson Sampling can be more robust to non-stationary environments because it updates posteriors continuously.
6. Compare Q-learning and policy gradient methods in terms of handling continuous action spaces.
Q-learning typically handles discrete action spaces because it needs to compute Q-values for each action. For continuous actions (e.g., choosing a difficulty level on a scale from 1 to 10), Q-learning would require discretization, which can lose precision. Policy gradient methods can naturally handle continuous action spaces by outputting parameters of a probability distribution (e.g., mean and variance of a Gaussian). For example, the policy can directly output a difficulty value like 7.3. Policy gradient methods also tend to be more stable for stochastic policies. However, Q-learning with function approximation can also be adapted for continuous actions using techniques like DDPG (Deep Deterministic Policy Gradient).
7. How does an LSTM-based model improve upon a simple RNN for knowledge tracing?
An LSTM (Long Short-Term Memory) network improves upon a simple RNN by using gating mechanisms to better capture long-term dependencies. In knowledge tracing, a student's earlier performance can influence later outcomes, but simple RNNs suffer from vanishing gradients and forget information over long sequences. LSTM's forget gate, input gate, and output gate control what information to keep or discard. For example, if a student mastered a skill early on, the LSTM can retain that knowledge even after many other exercises. This allows the model to more accurately predict performance on later questions that depend on earlier skills. LSTMs are therefore more effective for modeling student learning over many interactions.
8. How does a recurrent neural network model the change in a student's knowledge over time?
A recurrent neural network (RNN) models knowledge change by updating a hidden state vector at each time step. The hidden state represents the student's current knowledge. When the student answers a question, the RNN takes the previous hidden state and the current interaction (e.g., question features and correctness) as input, and outputs a new hidden state. This update is learned from data to reflect learning or forgetting. For example, a correct answer might increase the hidden state's value for related skills, while an incorrect answer might decrease it. The RNN can also incorporate the time elapsed between interactions to model forgetting curves. The final hidden state is used to predict future performance.
9. How does Thompson Sampling balance exploration and exploitation?
Thompson Sampling is a Bayesian algorithm that balances exploration and exploitation by sampling from a posterior distribution of each item's reward probability. For each item, the algorithm maintains a belief (e.g., a Beta distribution) about how likely the user will like it. At each step, it draws a random sample from each item's distribution and recommends the item with the highest sample. This naturally explores items with high uncertainty (wide distributions) and exploits items with high estimated reward. For example, if a new movie has a broad posterior, it might occasionally get a high sample and be recommended. Over time, as the algorithm gathers data, the posteriors narrow, and exploitation dominates.
10. How does propensity score matching help estimate the causal effect of a personalized tutoring program?
Propensity score matching reduces bias by pairing students who received the personalized tutoring (treatment group) with similar students who did not (control group), based on their probability of receiving treatment (propensity score). The propensity score is estimated using observed characteristics like prior grades, demographics, and engagement. For example, a student with high prior grades might have a high propensity to receive tutoring. Matching creates pairs of students with similar propensity scores, so that the only difference is the tutoring. Then, the difference in outcomes between the matched pairs estimates the causal effect. This mimics a randomized experiment when randomization is not possible.
11. Why is it important to use a reward function that captures long-term learning rather than just immediate correct answers?
Using only immediate correct answers as rewards can lead to short-sighted policies, like always giving easy problems to get high immediate reward. However, the goal is long-term learning, which may require challenging problems that cause temporary failures. A reward function that includes measures of learning gain (e.g., pre-test to post-test improvement) or retention over time encourages the agent to choose actions that build durable knowledge. For example, the agent might learn that a difficult problem now leads to better performance on future tests. This aligns the RL agent's objective with the true educational goal. Without long-term rewards, the system might optimize for engagement rather than learning.
12. Compare DKT with traditional Bayesian Knowledge Tracing (BKT) in terms of handling multiple skills.
Bayesian Knowledge Tracing (BKT) typically models each skill independently using a separate hidden Markov model. It assumes that skills do not interact, which is unrealistic. Deep Knowledge Tracing (DKT) uses a single recurrent neural network that can learn relationships between different skills. For example, DKT can capture that mastering addition helps with multiplication, while BKT treats them separately. DKT also does not require manually specifying skill tags; it can learn latent skill representations from data. However, DKT is less interpretable than BKT, as the hidden state does not directly correspond to known skills. BKT is simpler and works well when skills are well-defined and independent.