Questions & explanations
1. Compare log loss and hinge loss; when would you use each?
Log loss is used for probabilistic classification, outputting well-calibrated probabilities. Hinge loss is used for support vector machines and outputs a margin-based score, not a probability. Log loss penalizes all mistakes proportionally to confidence, while hinge loss only penalizes mistakes that violate the margin. Hinge loss often leads to better accuracy in some tasks, but log loss gives probability estimates that can be useful for ranking or threshold selection. For example, if you need probabilities for decision-making, use log loss; if you just want the best class boundary, hinge loss might work better. In practice, the choice depends on whether calibrated probabilities are important.
2. What is an instrumental variable (IV) in feature selection?
An instrumental variable is a variable that helps estimate the true causal effect of a feature on the target when there may be hidden confounding. For example, in studying how advertising affects sales, the ad spend might be correlated with other factors. If you use an instrument like the cost of TV ads (which affects ad spend but not directly sales), you can isolate the effect. In feature selection, IV methods identify features that have a causal influence rather than just correlation. This is useful to avoid selecting features that are only correlated due to a common cause. IV must satisfy two conditions: it is correlated with the feature and affects the target only through the feature.
3. What is a common method to align image patches with text tokens in a multimodal Transformer?
One common method is to use a contrastive loss, where the model learns to pull together matching image-text pairs and push apart non-matching ones. For example, in CLIP (Contrastive Language-Image Pretraining), the Transformer processes text and image separately but then compares their output embeddings. In a single-stream multimodal Transformer, alignment happens through cross-attention layers, where the attention weights show which image patch corresponds to which word. Training with a masked language modeling objective (predicting missing words) also forces alignment, because the model uses image context to guess the word. These methods help the model link visual and textual concepts.
4. Give an example of a task where a long-range transformer works better than a standard transformer.
A standard transformer with full attention cannot process a whole book because it would need too much memory. A long-range transformer like Longformer can handle a long research paper of 20,000 words. It reads all the text at once and answers questions about details from the introduction and conclusion together. For instance, if the paper says in the beginning 'we use method A' and later says 'results show 90% accuracy', the model can connect those sentences. Standard transformers would need to cut the paper into short pieces and lose the full context. Long-range transformers keep the whole picture, so they are better for document classification or long-document question answering.
5. Compare how a multimodal Transformer combines modalities versus a separate model for each modality.
A separate model for each modality (e.g., an image model and a text model) processes each type of data independently, and their outputs are combined later. This can miss fine-grained interactions, because the image model doesn't know the text while analyzing. A multimodal Transformer processes all modalities together from the start, allowing cross-modal attention. For example, while reading the word 'cat', the model can directly attend to image patches that look like a cat. This joint processing often gives better performance on tasks that need close alignment, like visual question answering. However, it requires more memory because the combined sequence is longer.
6. How do normalizing flows transform a simple distribution into a complex one?
Normalizing flows apply a sequence of invertible transformations to a simple base distribution, like a standard normal. Each transformation warps the space a little, making the distribution more complex. For example, an initial Gaussian blob is squeezed, stretched, and rotated through the layers. The final distribution can model multimodal or structured data like images. Because each step is invertible, you can compute the probability of any point by mapping it back through the layers. The overall transformation is learned by training to maximize the likelihood of real data. This way, the flow learns to shape the base distribution into the target data distribution.
7. What does it mean for a clustering label to be scale-variant?
A clustering label is scale-variant when the clustering result changes if you multiply the values of a feature by a constant. For example, if you have data with heights in meters and you change them to centimeters, the groups found by a scale-variant method can become different. This happens because methods like k-means use distance calculations that are sensitive to feature scales. In contrast, scale-invariant methods, like those using correlation, give the same labels no matter how you scale the features. Knowing this helps you choose a method that fits your data. If your features are measured in different units, you may prefer a scale-invariant method.
8. What is a scale co-occurrence matrix (SCM) for texture features?
A scale co-occurrence matrix captures how often pairs of pixel intensities occur together at a given distance and scale in an image. It extends the gray-level co-occurrence matrix (GLCM) by considering multiple scales, meaning you look at the image at different resolutions. For example, you can compute co-occurrence at a fine scale (small pixel offsets) and a coarse scale (larger offsets). This gives texture features that are robust to changes in image size or viewing distance. The matrix entries count how many times a particular pair of gray levels appears at a certain spatial relationship. From the matrix, you derive statistics like contrast or energy.
9. Compare space mapping with feature selection. When would you use each?
Space mapping changes the representation of existing features, while feature selection removes some features entirely. Use space mapping when you believe the current representation is not suitable for your model, such as when data is not linearly separable or when features are on different scales. Use feature selection when you have many irrelevant or redundant features that could cause overfitting. For instance, in text classification, you might use space mapping (TF-IDF) to transform word counts, but also feature selection to remove very rare words. Sometimes both are applied together: first select important features, then map them to a better space.
10. Compare instrumental variables approach with correlation-based feature selection.
Correlation-based feature selection picks features that are highly correlated with the target. But correlation does not mean causation; a confounder could make both feature and target move together. IV selection focuses on causal relationships: it selects features that have a causal effect on the target. For instance, ice cream sales and drowning are correlated due to summer heat. Correlation selects ice cream sales, but IV would not because there is no instrument that isolates a causal effect. IV is harder to apply because you need a valid instrument, but it gives more trustworthy features for decision-making. Correlation is simpler but can mislead.
11. How does instrumental variables estimation help in feature selection for regression models?
In regression, features may be correlated with the error term (endogenous), leading to biased coefficients. IV estimation uses the instrument to extract the part of the feature that is not correlated with the error. For feature selection, you can test whether the instrumented feature has a significant effect on the target. Only features with a significant causal effect are kept. This reduces false positives from spurious correlations. For example, in demand forecasting, price is endogenous. You can use cost shifters as instruments to select price as a predictive feature only if its causal effect is real. The process helps build more reliable models.
12. Compare Laplace and Gaussian mechanisms: when would you use each?
Both mechanisms add noise to achieve differential privacy, but they use different distributions. The Laplace mechanism uses a sharp distribution and satisfies pure differential privacy (epsilon-DP). The Gaussian mechanism uses a flatter distribution and satisfies a relaxed version called (epsilon, delta)-DP, which allows a tiny chance of failure. Laplace is simpler and exact but can require more noise for high-dimensional queries. Gaussian is often used in machine learning because it works better with many queries and allows a small delta. You choose Laplace when you need strict privacy, and Gaussian when you need better accuracy with composition.