Questions & explanations
1. You want to test if the average word length in two corpora is different. How can you use a permutation test?
A permutation test (also called a randomization test) works by shuffling the group labels. First, compute the observed difference in mean word length between the two corpora. Then, combine all words from both corpora into one pool. Randomly assign them to two groups of the same sizes as the original corpora. Compute the difference in means for this shuffled assignment. Repeat this many times (e.g., 10,000 times). This creates a distribution of differences under the null hypothesis that the two corpora come from the same population. If the observed difference is extreme compared to this distribution (e.g., only 5% of shuffled differences are as large), you reject the null hypothesis. The p-value is the proportion of shuffled differences as extreme as the observed.
2. Compare on-the-fly rescoring with shallow fusion in the context of language model integration.
On-the-fly rescoring and shallow fusion are both methods to combine an external language model with an acoustic model during decoding. In shallow fusion, the language model scores are added to the acoustic model scores at each step, and the decoder uses the combined score to guide search. On-the-fly rescoring typically refers to re-ranking a list of hypotheses generated by a base decoder, but it can also be done incrementally. Shallow fusion is a specific technique where the language model is interpolated directly into the beam search. Both improve accuracy, but shallow fusion is more tightly integrated, while on-the-fly rescoring can be applied as a post-processing step. The choice depends on the system architecture and real-time requirements.
3. How can you find the most similar document to a query using vector-space models?
First, represent the query as a vector using the same TF-IDF weighting as the documents. Then compute the cosine similarity between the query vector and each document vector. The document with the highest cosine similarity is the most similar. For example, if the query is 'cat dog', its vector might be [1,1,0,0] (assuming same vocabulary). Compare to Doc1 and Doc2: cosine with Doc1 = (1*1+1*1+0*1+0*0)/(sqrt(2)*sqrt(3)) = 2/(1.414*1.732)=2/2.45=0.816; with Doc2 = 2/(1.414*1.732)=0.816 as well. So both are equally similar. In practice, you would rank all documents by similarity and return the top ones. This is how search engines work: they convert queries and documents to vectors and find nearest neighbors.
4. Give a simple example of Viterbi decoding for a CRF with two labels (A and B) and a sequence of three words.
Suppose we have two labels, A and B, and three words. The emission scores for each word-label pair are: word1: A=0.8, B=0.2; word2: A=0.3, B=0.7; word3: A=0.6, B=0.4. Transition scores: A->A=0.9, A->B=0.1, B->A=0.2, B->B=0.8. Start scores: A=0.5, B=0.5. Viterbi starts at word1: best score for A = 0.5+0.8=1.3, for B=0.5+0.2=0.7. For word2: score for A = max(1.3+0.9+0.3, 0.7+0.2+0.3)=max(2.5,1.2)=2.5 from A; for B = max(1.3+0.1+0.7, 0.7+0.8+0.7)=max(2.1,2.2)=2.2 from B. For word3: A = max(2.5+0.9+0.6, 2.2+0.2+0.6)=max(4.0,3.0)=4.0 from A; B = max(2.5+0.1+0.4, 2.2+0.8+0.4)=max(3.0,3.4)=3.4 from B. Best final label is A (score 4.0). Backtrack: word3 A from word2 A, word2 A from word1 A, so sequence is A-A-A.
5. How is a Bayesian credible interval different from a frequentist confidence interval?
A Bayesian credible interval is a direct probability statement about the parameter. For example, a 95% credible interval means there is a 95% probability that the true parameter lies within that interval, given the data and prior. A frequentist confidence interval means that if you repeated the study many times, 95% of the intervals would contain the true parameter. It does not say anything about a specific interval. In practice, for simple cases, they may be numerically similar, but they are interpreted differently. Bayesian intervals are often easier to explain: 'We are 95% sure the true value is between X and Y.' They also incorporate prior information, which can be useful when data are sparse.
6. Why use resampling methods instead of a traditional t-test for corpus data?
Corpus data often violate assumptions of traditional tests like the t-test. For example, word frequencies are not normally distributed; they follow a skewed distribution. Also, corpus observations may not be independent because words come from the same text. Resampling methods like bootstrap and permutation tests do not require normality or independence assumptions. They rely on the data itself to estimate the sampling distribution. This makes them more robust. Additionally, they can handle small sample sizes and unusual statistics (e.g., median, ratio). However, they are computationally intensive, but modern computers make them feasible. So for corpus linguistics, resampling is often preferred.
7. What is topic modeling and what does Latent Dirichlet Allocation (LDA) do?
Topic modeling is a way to discover hidden topics in a collection of documents. Latent Dirichlet Allocation (LDA) is a popular topic model. It assumes each document is a mix of topics, and each topic is a mix of words. For example, a topic about 'sports' might have high probabilities for words like 'ball', 'game', 'team'. LDA learns these topics automatically from the word co-occurrence patterns in the documents. It uses a probabilistic process: for each document, it picks a distribution over topics; for each word, it picks a topic from that distribution, then picks a word from that topic's word distribution. The result is a set of topics each represented by a list of words with probabilities.
8. Compare zero-shot translation with pivot-based translation (using an intermediate language). Which is better and why?
Zero-shot translation is often better than pivot-based translation because it avoids the double translation errors. In pivot-based translation, you translate from language A to a pivot language (like English) and then to language B. Each step can lose meaning or add mistakes. Zero-shot translation does it in one step, so it can preserve more of the original meaning. For example, translating French to German via English might change 'Je t'aime' to 'I love you' then to 'Ich liebe dich', which is fine, but for complex sentences, errors multiply. Zero-shot models are trained to map directly, so they can be more fluent. However, pivot-based can be more reliable if the zero-shot model is weak.
9. How does cross-lingual transfer learning relate to multilingual models like mBART or mT5?
Multilingual models like mBART (a model trained on many languages at once) are built on cross-lingual transfer learning. They are pretrained on text from 50 or more languages, so they learn shared patterns across languages. When used for a low-resource language, they already have some knowledge of that language if it was included in pretraining. Even if not, they can still transfer from similar languages. For example, mT5 (a text-to-text model) can be fine-tuned on a low-resource language like Uyghur, using its knowledge from related Turkic languages. This is more powerful than using just one high-resource language because the model sees many language families and can generalize better.
10. How do you evaluate the quality of topics from LDA?
Evaluation can be intrinsic or extrinsic. Intrinsic measures include topic coherence (e.g., using word co-occurrence statistics) and topic diversity (how distinct topics are from each other). A common coherence measure is the average pairwise similarity of top words using external word embeddings or pointwise mutual information. Higher coherence means the words are more related. Extrinsic evaluation involves using topics for a downstream task, like document classification or information retrieval, and measuring performance. Human evaluation is also important: ask people to rate whether a topic is coherent or to label topics. No single metric is perfect, so it's best to combine several.
11. How is TF-IDF calculated for a word in a document?
TF-IDF stands for Term Frequency-Inverse Document Frequency. Term frequency (TF) is the number of times the word appears in the document, often divided by the total words in the document to normalize. Inverse document frequency (IDF) is log(total number of documents / number of documents containing the word). Then TF-IDF = TF * IDF. For example, if a word appears 3 times in a 100-word document, TF = 3/100 = 0.03. If the word appears in 10 out of 1000 documents, IDF = log(1000/10) = log(100) = 2 (if using natural log, about 4.6). So TF-IDF = 0.03 * 2 = 0.06. This value is high if the word is frequent in the document but rare overall. It helps identify important words for that document.
12. Compare cross-lingual transfer learning with training a model from scratch for a low-resource language. Which is better and why?
Cross-lingual transfer learning is usually much better than training from scratch for a low-resource language. Training from scratch needs thousands of sentence pairs to learn even basic translations, which is hard to get for small languages. Transfer learning uses a model already trained on a big language pair, so it needs only a few hundred or thousand examples to adapt. For example, to translate English to a language with only 10,000 sentence pairs, a from-scratch model might give poor results, while a transferred model can be quite accurate. The transferred model also trains faster because it starts with good knowledge. So, transfer learning saves data, time, and computing power.