Questions & explanations
1. Give an example of a weakly-acyclic set of existential rules for data exchange.
Consider a source database with Employee(name, dept) and a target with Person(name, worksIn) and Department(dept, mgr). A weakly-acyclic rule: 'if Employee(x,y) then exists z Person(x,z)' creates a Person with a new worksIn? Actually let's do: 'if Employee(x,y) then Person(x,y)' (no existential). Another: 'if Employee(x,y) then exists w Department(y,w)'. This creates a Department with a new manager. The rule from Department to Person: 'if Department(y,w) then Person(w, y)' might create a cycle if Person also creates Department. But if the rule is designed so that only certain positions create new values, it can be weakly-acyclic. Typically, the chase terminates because new values are only introduced in non-cyclic positions.
2. Explain a scenario where data reconciliation would catch a problem that a unit test on the pipeline code would not.
A unit test checks if the code logic is correct with small synthetic data, but it may not test with real-world data shapes and edge cases. For example, the pipeline code might handle NULL values correctly in tests, but a source system sometimes sends unexpected NULLs in a column that the pipeline treats as zero, causing sums to be wrong. Data reconciliation against the actual source would catch that the target sum doesn't match. Also, reconciliation detects issues from configuration changes, infrastructure failures, or data that violates schema assumptions—things unit tests don't cover. So both tests are needed: unit tests verify logic, reconciliation verifies actual output.
3. How does ICA differ from PCA?
PCA finds components that are uncorrelated and maximize variance. ICA finds components that are statistically independent, meaning the value of one component gives no information about the others. PCA uses second-order statistics (covariance), while ICA uses higher-order statistics to achieve independence. PCA yields orthogonal components, but ICA components are not necessarily orthogonal. A simple example: mixing two voice recordings, PCA cannot separate them because voices are correlated in amplitude? Actually, PCA may not separate if they are non-Gaussian? ICA successfully separates. In summary, PCA is for dimensionality reduction, ICA for source separation.
4. Compare PCA with factor analysis.
Both reduce dimensionality, but they differ in goal. PCA focuses on explaining total variance by creating new variables (components) that are linear combinations of observed variables. Factor analysis focuses on explaining covariances among observed variables using unobserved latent factors. In PCA, components are exact mathematical transformations; in factor analysis, factors are inferred and may not exactly recreate the data. PCA is often used for data compression; factor analysis for identifying underlying constructs. Interpretation differs: PCA components are weighted sums, while factor loadings represent correlations with latent factors.
5. How does sharding help improve data pipeline performance?
Sharding splits a large dataset into smaller, independent pieces called shards, which can be processed in parallel. For instance, a table of 1 billion customer records might be sharded by customer ID range (0-200M, 200M-400M, etc.). Each shard is stored on a separate server or processed by a separate pipeline instance. This allows the pipeline to handle huge data without hitting a single server's limits. Sharding also improves scalability: adding more shards increases throughput. However, it adds complexity in querying across shards and maintaining data distribution. It's commonly used in databases and big data systems like Hadoop or Spark.
6. Why are weakly-acyclic TGDs a standard choice for data exchange systems?
Weakly-acyclic TGDs are standard because they balance expressiveness and tractability. They allow many natural schema mappings that involve cycles, like a person managing a department while the department employs that person. Yet they guarantee that the chase will terminate, so we can compute a finite target instance. Data exchange systems like Clio use weakly-acyclic rules. They also support query answering through rewriting into Datalog or SQL. Because the chase terminates, we can answer queries by running the rewritten program. This makes weakly-acyclic TGDs a practical and well-studied choice for real data exchange scenarios.
7. How is logistic regression different from linear regression in terms of the outcome variable?
The key difference is the type of outcome variable. Linear regression is for continuous outcomes, like temperature or price, which can take any numerical value. Logistic regression is for categorical outcomes, often binary (0/1). Because of this, logistic regression uses a different link function (sigmoid) to ensure predictions stay between 0 and 1. Linear regression assumes the outcome is normally distributed around the line, while logistic regression assumes the outcome follows a binomial distribution. The interpretation of coefficients also differs: in logistic regression, coefficients affect log-odds, not the mean directly.
8. Compare factor analysis with PCA in terms of modeling error.
PCA does not model error separately; it treats all variance as common and tries to capture it in principal components. Factor analysis explicitly separates common and unique variance. PCA is a simple transformation that maximizes variance, while factor analysis is a model with assumptions about underlying causal structure. In PCA, the components explain all variance, but in factor analysis, the factors only explain the common variance. This makes factor analysis better for understanding latent constructs, while PCA is better for pure data reduction. Factor analysis can also produce more interpretable results if the model fits.
9. How do wrapper methods differ from filter methods?
Filter methods evaluate features independently using statistical tests like correlation or ANOVA, without using a model. Wrapper methods use a specific model and its performance to judge feature subsets. Filter methods are faster and scale to large datasets. Wrapper methods can capture interactions between features because the model learns from combinations. However, wrapper methods risk overfitting to the training data. For example, a filter method ranks features by chi-square, while a wrapper method trains a decision tree and tests different feature sets. The choice depends on computational budget and need for interactions.
10. Compare weakly-acyclic TGDs with acyclic TGDs.
Acyclic TGDs require no cycles in the dependency graph, while weakly-acyclic TGDs allow cycles as long as certain positions are safe. Acyclic rules are simpler and always terminate, but they are less expressive. Weakly-acyclic rules can capture more complex mappings, such as when you need to create a person and then a department for that person, and then the department refers back to the person. In data exchange, weakly-acyclic rules are often sufficient and allow efficient query answering. However, checking weak acyclicity is more involved than checking acyclicity. Both are important classes for practical data integration.
11. What is a linear existential rule?
A linear existential rule is a rule that has exactly one atom in its body. This makes it the simplest type of existential rule. For example, 'if Student(x) then exists y Enrolled(x,y)' is linear. Linear rules are easy to handle because there is no join between body atoms. They allow query answering to be rewritten into Datalog efficiently. Because of their simple structure, linear rules have nice properties like finite chase. They are often used in data integration when we need to map attributes from one source to another. Linear rules are a subclass of guarded rules, as the single body atom naturally guards all variables.
12. Give an example of backward elimination with five features.
Start with model using all five features, accuracy 92%. For each feature, remove it and retrain. Removing feature A gives 91%, B gives 91.5%, C gives 90%, D gives 91.8%, E gives 91.2%. The smallest drop is from removing D (92% to 91.8%, drop 0.2%). So remove D. Now with four features (A,B,C,E), accuracy 91.8%. Again try removing each: remove A gives 90.5%, B gives 91%, C gives 89%, E gives 91.5%. Smallest drop from removing E (0.3%). Remove E. Now with three features (A,B,C) accuracy 91.5%. Next removal: remove A gives 89%, B gives 88%, C gives 87%. All drops are large (over threshold 0.5%). So stop with features A,B,C.