Purdue CERIAS · Published 2026-09-11

2026-09-09 CERIAS - Reasoning About Interventions in Data and Machine Learning Systems

privateReview: unreviewedConfidence: high
fairnessinfluence functionspipelinescausal DAGdata valuationinfluence-functionsdata-valuationcausal-dagpipeline-repairmulti-armed-banditsdistribution-shiftselective-data-expansion
Open on YouTube ↗

Summary

Overview

  • Speaker: Dr. Romila Pradhan
  • Channel: Purdue CERIAS
  • Main topic: Reasoning about interventions in data and machine learning systems for fairness, robustness, and pipeline repair.
  • Purpose: To introduce principled methods for reasoning about data and pipeline interventions to ensure fairness, accuracy, and reliability in machine learning and data science systems. Dr. Romila Pradhan from Purdue University presents research from the Responsible Data Science Lab on reasoning about interventions in data and machine learning systems. The talk focuses on two main papers: DataSift (selective data expansion using multi-armed bandits and influence functions to improve model fairness without sacrificing accuracy) and PipeLens (identifying minimal structural and parameter interventions to repair malfunctioning data science pipelines using data profiles, causal DAGs, and historical execution logs).

Topic Map

Introduction and Motivation

  • Explanation: Overview of algorithmic decision-making across critical domains (healthcare, criminal justice, hiring, finance) and the challenges of black-box models, bias, and unexpected system behavior.
  • Key claims:
    • Algorithmic decision-making is widely used in high-stakes domains.
    • Complex ML models are often opaque and prone to unexpected bias and errors.
    • Three pillars of modern decision-making systems are Explainability, Fairness, and Robustness.
  • Examples:
    • Amazon's scrapped AI recruiting engine showing bias against women.
    • Racial bias in algorithms used to manage population health.
  • Terminology:
    • Explainability
    • Fairness
    • Robustness
    • Algorithmic Recourse
  • Why it matters: Ensures stakeholders across the pipeline (developers, practitioners, businesses, users) can diagnose, explain, and redress algorithmic harms.

DataSift: Selective Data Expansion for Fairness

  • Explanation: Addressing model bias through goal-oriented selective data expansion using multi-armed bandits and influence functions.
  • Key claims:
    • Naive expansions (adding all data or random data) fail to improve fairness and often degrade it.
    • Uniform representation in added data helps fairness but can hurt accuracy.
    • Combining multi-armed bandit reward scoring with influence functions efficiently identifies valuable training data.
  • Examples:
    • Adult income dataset classification task predicting income levels across demographic groups.
  • Terminology:
    • DataSift
    • Multi-Armed Bandit
    • Influence Functions
    • Statistical Parity
    • Data Valuation
  • Why it matters: Demonstrates that 'the right data is more important than more data' when repairing unfair machine learning models.

PipeLens: Repairing Malfunctioning Data Science Pipelines

  • Explanation: Identifying minimal structural and parameter interventions to restore performance in malfunctioning data science pipelines.
  • Key claims:
    • Pipelines optimized for initial data distributions fail when data changes over time.
    • PipeLens models pipelines as causal directed acyclic graphs (DAGs) connecting data profiles to task utility.
    • Optimistic and exhaustive search strategies efficiently rank pipeline repairs using historical runs and data profiles.
  • Examples:
    • Entity resolution pipeline for matching book records experiencing language assumption violations.
  • Terminology:
    • PipeLens
    • Causal DAG
    • Data Profiles
    • Parameter Intervention
    • Structural Intervention
    • Entity Resolution
  • Why it matters: Provides automated, efficient pipeline repair without requiring full pipeline re-execution across large search spaces.

Key Points

More data does not solve fairness

  • Explanation: Adding arbitrary or random data to a training set often increases model accuracy while degrading demographic fairness.
  • Evidence: Experimental results comparing naive data addition against DataSift show accuracy gains accompanied by drops in fairness metrics.
  • Practical implication: Data practitioners must carefully select and augment targeted data points using valuation techniques rather than scaling dataset size blindly.

Pipelines require joint reasoning over data and components

  • Explanation: Most debugging tools look only at component parameters or data quality in isolation, missing the interaction between data distributions and pipeline steps.
  • Evidence: Entity resolution pipeline failures caused by language changes between input data and block matching assumptions.
  • Practical implication: Systems should model data science pipelines as causal graphs linking data profiles to downstream task utility.

Frameworks, Models & Processes

DataSift Framework

  • How it works: Partitions data pools into clusters, uses a multi-armed bandit with a fairness-accuracy reward scoring function, and employs influence functions to select the top-K valuable data points for augmentation.
  • Components:
    • Data pool clustering
    • Multi-armed bandit selection
    • Reward scoring function
    • Influence functions
  • When to use: When an ML model exhibits demographic bias and you need to selectively expand training data to restore fairness without hurting accuracy.

PipeLens Framework

  • How it works: Models pipeline execution as a causal DAG where components and parameters affect intermediate data profiles and downstream utility, using data profiles and historical logs to rank repairs.
  • Components:
    • Data profile generator
    • Causal DAG representation
    • Global module catalog
    • Optimistic and exhaustive search
  • When to use: When a data science or ML pipeline experiences performance degradation due to distribution shifts or component mismatches.

Examples & Case Studies

Adding all available data or random data to correct income classification bias resulted in 85% accuracy but a drop in fairness (fairness metric degradation).

  • Illustrates: Naive data expansion fails to balance accuracy and fairness.
  • Lesson: Only judiciously selected data (DataSift) improves fairness while preserving accuracy.

An entity resolution pipeline matching book records failed when input data contained French titles instead of English, dropping F1-score from 87% to 39%.

  • Illustrates: Pipeline malfunction caused by language assumption violations.
  • Lesson: Inserting a language translator component and updating blocking attributes repaired the pipeline successfully.

Actionable Takeaways

  • Immediate:
    • Avoid naive data expansion when attempting to mitigate model bias.
    • Model data science pipelines as causal graphs to trace how parameter or structural changes impact downstream utility.
  • Strategic:
    • Combine reinforcement learning (multi-armed bandits) with influence functions for efficient data selection.
    • Leverage data profiling and historical logs to navigate large pipeline configuration search spaces without full re-execution.
  • Questions to investigate:
    • How do these intervention techniques scale to deep neural network architectures with non-convex loss functions?
    • What are the privacy implications of computing influence functions across sensitive training records?

Claims Worth Verifying

  • DataSift achieves fairness thresholds using only ~6% additional data compared to 20%+ for baseline methods. (empirical)
  • PipeLens identifies valid pipeline repairs in 10-11 iterations across a 50M+ search space. (empirical)

Notable Quotes

"Sophisticated ML models have been shown to be highly accurate for many applications only until there is an unexpected or undesired behavior." (at 2:12) "More data or better representation does not resolve fairness; the right data is more important." (at 38:21)

Compressed Summary

  • DataSift uses multi-armed bandits and influence functions for goal-oriented selective data expansion.
  • PipeLens repairs malfunctioning pipelines using causal DAGs, data profiles, and historical execution logs.
  • Both frameworks emphasize that targeted interventions on data and pipelines outperform brute-force re-training.
  • Keywords: fairness, influence functions, pipelines, causal dag, data valuation
  • Core insight: Principled interventions on training data and pipeline structures can restore model fairness and pipeline performance efficiently using data profiles and influence estimation.

Core insights

6
Empirical Resultmedium noveltystrong evidence

Naive data expansion — adding all available data or random data — can raise accuracy while degrading fairness, so bias mitigation is not a data-volume problem; it is a data-selection problem requiring valuation-scored, targeted augmentation.

Why it matters

It directly contradicts the reflexive engineering instinct to fix a weak or biased model by scaling the training set, and it implies that any automated data-acquisition or synthetic-data-generation loop needs a scoring/selection gate rather than a volume target.

Generalization

Any pipeline that treats dataset size as a proxy for quality (augmentation, synthetic data, retrieval corpus growth, agent memory accumulation) risks optimizing one metric while silently regressing another; selection mechanisms, not ingestion rate, should gate additions.

Naive expansions (adding all data or random data) fail to improve fairness and often degrade it.
Open source video
Adding all available data or random data to correct income classification bias resulted in 85% accuracy but a drop in fairness (fairness metric degradation).
Open source video
Mechanismhigh noveltymoderate evidence

DataSift combines a multi-armed bandit reward loop with influence functions to select a top-K set of data points for augmentation, i.e., influence estimates act as a within-pool valuation signal while the bandit handles the explore/exploit cost of choosing which clusters to expand.

Why it matters

It is a concrete composition pattern for turning an expensive, per-item attribution method (influence functions) into a practical selector by wrapping it in a cheap reward-driven search over grouped candidates — a reusable architecture for any ranked-selection-under-budget problem.

Generalization

Ranked selection over a large candidate pool where scoring is expensive can be decomposed into cheap group-level exploration (bandit) plus expensive within-group ranking (attribution/valuation), reducing scoring calls.

Combining multi-armed bandit reward scoring with influence functions efficiently identifies valuable training data.
Open source video
uses a multi-armed bandit with a fairness-accuracy reward scoring function, and employs influence functions to select the top-K valuable data points for augmentation
Open source video
Architecturehigh noveltymoderate evidence

PipeLens reframes pipeline repair by representing the pipeline as a causal DAG that links intermediate data profiles to downstream task utility, so diagnosis reasons jointly about data distributions and component parameters instead of inspecting either in isolation.

Why it matters

This relocates the responsibility for failure reasoning from per-component debugging tools into a pipeline-level graph abstraction, which is the abstraction an automated repair agent would need to propose and rank interventions.

Generalization

For compound systems, correctness reasoning belongs at the level of an explicit dependency graph over intermediate artifacts (data/state), not at the level of individual stages; stage-local debugging structurally cannot see cross-stage assumption violations.

Most debugging tools look only at component parameters or data quality in isolation, missing the interaction between data distributions and pipeline steps.
Open source video
Systems should model data science pipelines as causal graphs linking data profiles to downstream task utility.
Open source video
Practicemedium noveltymoderate evidence

Repair candidates can be ranked using historical execution logs and data profiles without re-executing the full pipeline, making search tractable over large configuration spaces via optimistic and exhaustive strategies.

Why it matters

Directly affects cost and latency of any automated repair/self-healing loop: a cheap proxy ranking stage before expensive execution turns an intractable search into an ordered evaluation, which is the same pattern needed for agent config/tool retry search.

Generalization

Self-repair and configuration-search loops should maintain an artifact of past runs plus cheap observational signals (profiles/telemetry) and use them to prioritize candidates before paying full evaluation cost.

Optimistic and exhaustive search strategies efficiently rank pipeline repairs using historical runs and data profiles.
Open source video
Provides automated, efficient pipeline repair without requiring full pipeline re-execution across large search spaces.
Open source video
Mechanismmedium noveltystrong evidence

The repair objective is a minimal intervention, and the demonstrated fix was structural rather than parametric: inserting a new component (language translator) alongside a parameter update (blocking attributes) recovered a pipeline whose F1 collapsed from 87% to 39% due to an unstated language assumption.

Why it matters

It shows the repair space must be heterogeneous — adding/shifting stages as well as tuning parameters — and that silent assumptions embedded in a component (e.g., that input is English) are a distinct failure class that only surfaces under distribution shift.

Generalization

Long-lived systems accumulate unstated input assumptions; robustness work should include mechanisms to detect assumption violations and to synthesize structural fixes, not only parameter retuning.

An entity resolution pipeline matching book records failed when input data contained French titles instead of English, dropping F1-score from 87% to 39%.
Open source video
Inserting a language translator component and updating blocking attributes repaired the pipeline successfully.
Open source video
Tradeoffmedium noveltymoderate evidence

There is a real, acknowledged tension in adding data for fairness: uniform demographic representation in the added data improves fairness but can cost accuracy, so fairness-focused data augmentation must be optimized jointly rather than as a single objective.

Why it matters

It flags that fairness-oriented selection cannot be treated as a hard constraint or a re-weighting afterthought; the reward/scoring function has to balance two metrics, which is exactly where specification gaming and metric collapse occur.

Generalization

Multi-objective selection loops (fairness, accuracy, cost, safety) require an explicit reward design and monitoring, because optimizing the primary metric alone predictably degrades the secondary one.

Uniform representation in added data helps fairness but can hurt accuracy.
Open source video

Deep dives

5

Two-tier selection: bandit-gated group exploration over expensive influence-based ranking

Research question

How should a ranked-selection-under-budget system decompose scoring into cheap group-level exploration plus expensive within-group valuation so that top-K quality is preserved while the number of valuation calls is minimized, and how does the optimal group granularity shift with pool size and heterogeneity?

Why

The same cost structure recurs in synthetic-data generation, retrieval-corpus growth, agent memory accumulation, and tool-retry search; if the decomposition holds, it becomes a reusable default architecture for gating additions rather than a single-paper trick.

Combining multi-armed bandit reward scoring with influence functions efficiently identifies valuable training data.
Open source video
uses a multi-armed bandit with a fairness-accuracy reward scoring function, and employs influence functions to select the top-K valuable data points for augmentation
Open source video
Source video

Pipeline-level causal DAG as the substrate for an automated repair agent

Research question

What minimum graph schema over intermediate data profiles, component choices, and task utility is required for a repair agent to propose and rank interventions end-to-end, and which failure classes remain undiagnosable when reasoning stays stage-local?

Why

It relocates correctness reasoning from per-component debugging tools into the artifact an automated repair loop actually needs, and it bounds what logs-plus-per-tool diagnostics can ever detect.

Most debugging tools look only at component parameters or data quality in isolation, missing the interaction between data distributions and pipeline steps.
Open source video
Systems should model data science pipelines as causal graphs linking data profiles to downstream task utility.
Open source video
Source video

Silent assumption violations and the heterogeneous intervention space (structural vs parametric)

Research question

Can a detector over intermediate data profiles classify a failure as an unstated-assumption violation versus parameter drift before repair search begins, and does restricting the intervention space to parameters predictably fail on the former class?

Why

Long-lived systems accumulate unstated input assumptions, and a repair loop whose search space is parameter-only is structurally incapable of fixing the failure mode that produced the largest observed collapse (F1 87% to 39%).

An entity resolution pipeline matching book records failed when input data contained French titles instead of English, dropping F1-score from 87% to 39%.
Open source video
Inserting a language translator component and updating blocking attributes repaired the pipeline successfully.
Open source video
Source video

Cheap proxy ranking from retained execution history as the economics of self-repair

Research question

How often does a proxy ranking built from historical runs and data profiles place the true best repair in the top-K relative to exhaustive execution, and how fast does that hit rate degrade when the current shift type is absent from the logs?

Why

It decides whether auto-remediation is economically viable in large configuration spaces, and it makes execution history a durable asset rather than disposable telemetry — with a coverage caveat for genuinely novel failures.

Optimistic and exhaustive search strategies efficiently rank pipeline repairs using historical runs and data profiles.
Open source video
Provides automated, efficient pipeline repair without requiring full pipeline re-execution across large search spaces.
Open source video
Source video

Validity and leakage of influence-based valuation when scores leave the training loop

Research question

Do influence-function-based interventions remain a faithful ranking signal under non-convex deep losses, and how much membership or attribute information leaks when influence scores are exposed by a data-valuation service?

Why

The selection loop is only as good as the fidelity of its valuation signal, and the same scores that rank data points are an attack surface once a service or agent exposes them outside the training loop.

How do these intervention techniques scale to deep neural network architectures with non-convex loss functions?
Open source video
Combining multi-armed bandit reward scoring with influence functions efficiently identifies valuable training data.
Open source video
Source video

Article ideas

4

Your Bias Fix Is a Selection Problem, Not a Volume Problem

Because naive and random expansion can raise accuracy while degrading fairness, teams that instrument dataset size as their quality proxy are shipping silent metric regressions; every augmentation path — human-sourced or synthetic — needs a scored selection gate before data enters the training set.

Angle

Trace the failure from the reflex ('add more data') to the measured outcome (85% accuracy with fairness degradation), then specify the gate: candidate pool, scoring function, top-K admission, and monitoring of the secondary metric.

Source video

Stop Debugging Stages, Start Debugging the Graph

Stage-local inspection cannot see cross-stage assumption violations, so any agent that proposes and ranks repairs needs an explicit graph over intermediate data profiles and downstream utility as a first-class artifact — logs and per-tool diagnostics are structurally insufficient, not merely incomplete.

Angle

Argue from the structure of the failure (component parameter vs data quality in isolation) to the required abstraction, then sketch the artifact: nodes for profiles and components, edges to task utility, and interventions as edges of the graph.

Source video

Rank Before You Run: Cheap Proxy Evaluation Is What Makes Self-Repair Affordable

Automated repair over large configuration spaces only pencils out when a cheap ranking stage derived from retained execution history and data profiles precedes expensive execution, which forces a design decision most teams have not made: execution history is a durable asset, not disposable telemetry.

Angle

Cost-model the two-phase loop (proxy rank, then execute), then stress the assumption with novel shift types that past runs do not cover and quantify the hit-rate decay.

Source video

Assumption Debt: The Failure Class Parameter Tuning Cannot Reach

Long-lived pipelines accumulate unstated input assumptions, and an intervention space restricted to parameters is provably unable to repair them — robustness work must include detection of assumption violations and synthesis of structural interventions, with minimality defined over a mixed edit space rather than a single one.

Angle

Use the 87%-to-39% F1 collapse as the case study, contrast parameter-only, structural-only, and mixed search spaces, and argue for minimality criteria that cover adding a stage as well as retuning one.

Source video

Project ideas

4

Selector Gate Harness

beyond-evals

Across at least three tabular datasets and two model families, bandit-plus-influence top-K selection reaches a target fairness improvement with strictly fewer valuation calls and no larger accuracy drop than random, uniform-representation, or volume-based expansion at matched compute.

Proof of concept

Build a candidate pool expander with group-level bandit reward (joint fairness-accuracy) wrapping within-group influence ranking for top-K; run it head-to-head against random addition, uniform-representation addition, and add-all baselines at matched augmentation budget.

Measurement

Number of influence/valuation calls and wall-clock to target fairness; Pareto frontier of (fairness metric, accuracy); post-selection accuracy and fairness on a held-out distribution shift.

Source video

PipeLens-mini Repair Ranker

movement-lab

Cheap proxy ranking from historical runs and data profiles places the true best repair in the top-3 far above chance and cuts full pipeline executions by more than 80% versus exhaustive search, but its hit rate collapses when the current shift type is absent from the logs.

Proof of concept

Stand up a small pipeline graph (ingest, entity resolution, classifier), inject faults including an unstated-language assumption break, log data profiles and historical runs, rank candidate structural and parametric repairs with an optimistic estimator, then verify only top-K candidates by execution.

Measurement

Top-K hit rate of the best repair, number of full executions saved, F1 recovery versus exhaustive search, and hit-rate degradation on held-out shift types not present in logs.

Source video

Influence-Leak Audit

gatehouse

Influence scores exposed by a data-valuation service leak membership and attribute information above a chance baseline, and score aggregation plus calibrated noise reduces that leakage at a bounded cost to selection quality.

Proof of concept

Compute influence scores for a trained model, expose the top-K ranked points and their scores through a mock valuation service, train a membership-inference attacker against the exposure, then compare against aggregation and DP-noised score variants.

Measurement

Membership-inference AUC and attribute-inference accuracy against the exposed scores; change in fairness-accuracy outcome of top-K selection under each mitigation.

Source video

Assumption-Debt Fault Library

new

Parameter-only intervention search fails to recover utility on assumption-violation faults while mixed structural search recovers it, and a classifier over intermediate data profiles can separate assumption-violation from parameter-drift faults before repair search starts.

Proof of concept

Assemble a fault library over one real pipeline (schema drift, unit change, language/locale assumption, upstream distribution shift), run three search spaces (parametric-only, structural-only, mixed), and train a profile-based fault classifier used to route the repair search.

Measurement

Utility recovery per fault class, number of interventions applied per repair, re-failure rate under repeated shifts, and classifier accuracy in routing faults to the correct search space.

Source video

Architectural implications

3

Pipeline failure diagnosis is modeled as a causal graph over data profiles and downstream utility rather than as per-stage inspection.

Before

Debugging inspects each component's parameters or data quality in isolation.

After

An explicit causal DAG connects data profiles, component/parameter choices, and task utility so interventions can be traced end-to-end.

Consequence

A repair-capable agent needs a graph/state model of the pipeline as a first-class artifact, not just logs and per-tool diagnostics.

Source video

Data augmentation becomes a scored, ranked selection loop instead of a bulk ingestion step.

Before

More data is added indiscriminately (all data or random data) to improve the model.

After

Candidate points are clustered, scored with a fairness-accuracy reward, and ranked by influence to select top-K.

Consequence

Data curation shifts from a volume pipeline to a query-and-rank service that can be invoked as an automated remediation action.

Source video

Repair search is two-phase: cheap ranking from history and data profiles, then expensive execution.

Before

Candidate repairs require full pipeline re-execution to evaluate.

After

Optimistic and exhaustive search rank candidates using historical runs and data profiles before execution.

Consequence

Auto-remediation loops become economically viable in large configuration spaces, but depend on retained execution history as a durable asset.

Source video

Tradeoffs and failure modes

3

Fairness-oriented data augmentation

Benefit

Uniform representation in added data improves fairness.

Cost or risk

It can hurt accuracy; naive expansion can instead raise accuracy while degrading fairness, so neither volume nor uniformity alone works.

Uniform representation in added data helps fairness but can hurt accuracy.
Open source video
Source video

Influence-function-based data valuation

Benefit

Efficiently identifies valuable training data for targeted augmentation.

Cost or risk

Open concerns about computing influence functions at scale and their privacy implications, plus unclear behavior on deep networks with non-convex loss functions.

How do these intervention techniques scale to deep neural network architectures with non-convex loss functions?
Open source video
Source video

Automated pipeline repair

Benefit

Finds minimal structural and parameter interventions and avoids full re-execution across large search spaces.

Cost or risk

Effectiveness shown on a specific entity-resolution case (F1 87%→39% recovered); generality and guarantees of minimal repairs across pipeline types remain unestablished in the summary.

An entity resolution pipeline matching book records failed when input data contained French titles instead of English, dropping F1-score from 87% to 39%.
Open source video
Source video

Open questions

5

Do influence-function-based interventions remain valid for deep neural networks with non-convex loss functions?

Why unresolved

The summary raises this explicitly as a question to investigate, indicating the demonstrated results are not shown to extend to deep architectures.

Research direction

Benchmark DataSift-style selection on deep models, comparing influence approximations against empirical retraining deltas to measure ranking fidelity under non-convexity.

Source video

What are the privacy implications of computing influence functions over training data?

Why unresolved

The summary lists it as an open question without any findings.

Research direction

Analyze and measure membership/attribute leakage from influence scores exposed by a data-valuation service, and design aggregation or DP-noise mitigations.

Source video

Can a multi-armed bandit reward that balances fairness and accuracy be specified without inducing metric gaming or collapse?

Why unresolved

The summary states uniform representation helps fairness but can hurt accuracy and that naive expansions degrade fairness, but does not describe how the reward function avoids these failure modes.

Research direction

Ablate reward formulations across datasets and measure out-of-distribution generalization of both fairness and accuracy after selection.

Source video

How robust is historical-log-based repair ranking when the failure is novel, i.e., when past runs contain no example of the new distribution shift?

Why unresolved

The summary shows logs and data profiles are used to rank repairs, but the entity-resolution failure arose from a new input-distribution change, and no coverage guarantee is given.

Research direction

Stress-test repair ranking with held-out shift types absent from logs and quantify degradation versus exhaustive re-execution.

Source video

What minimality guarantees hold for structural versus parameter interventions, and when is adding a component preferable to retuning parameters?

Why unresolved

The summary describes both intervention types and one mixed fix (add translator + update blocking attributes) but provides no criterion for choosing between them.

Research direction

Compare search strategies that restrict the intervention space to structural, parametric, or mixed, and measure repair cost, correctness, and re-failure rate.

Source video

Key claims

8
causalVerification needed

Adding arbitrary or random data to a training set can increase model accuracy while degrading demographic fairness.

Evidence

Adding all available data or random data to correct income classification bias resulted in 85% accuracy but a drop in fairness (fairness metric degradation).

Question

On which datasets, model families, and fairness metrics does naive data addition reliably degrade fairness rather than improve it?

Source video
causalVerification needed

Combining multi-armed bandit reward scoring with influence functions efficiently identifies valuable training data.

Evidence

Combining multi-armed bandit reward scoring with influence functions efficiently identifies valuable training data.

Question

What is the measured efficiency gain (scoring calls, wall-clock, accuracy/fairness per unit compute) versus random or uniform selection?

Source video
comparativeVerification needed

Most debugging tools look only at component parameters or data quality in isolation, missing the interaction between data distributions and pipeline steps.

Evidence

Most debugging tools look only at component parameters or data quality in isolation, missing the interaction between data distributions and pipeline steps.

Question

Do joint data-profile/parameter debugging methods demonstrably diagnose failures that per-stage tools miss?

Source video
causalVerification needed

A pipeline's failure can be attributed to a violated input assumption (English-language titles) that a structural intervention plus a parameter update reverses.

Evidence

Inserting a language translator component and updating blocking attributes repaired the pipeline successfully.

Question

Does the translator-plus-blocking fix generalize to other assumption violations, or is it specific to this entity-resolution pipeline?

Source video
causalVerification needed

Repair candidates can be ranked using historical runs and data profiles without full pipeline re-execution.

Evidence

Optimistic and exhaustive search strategies efficiently rank pipeline repairs using historical runs and data profiles.

Question

How often does the cheap proxy ranking place the true best repair in the top-K relative to exhaustive execution?

Source video
opinionVerification not requested

Data practitioners must select and augment targeted data points using valuation techniques rather than scaling dataset size blindly.

Evidence

Data practitioners must carefully select and augment targeted data points using valuation techniques rather than scaling dataset size blindly.

Source video
comparativeVerification needed

Uniform representation in added data improves fairness but can hurt accuracy.

Evidence

Uniform representation in added data helps fairness but can hurt accuracy.

Question

How large is the accuracy penalty of uniform-representation augmentation across datasets and classifier families?

Source video
causalVerification needed

PipeLens can identify minimal structural and parameter interventions to restore performance in malfunctioning pipelines.

Evidence

Identifies minimal structural and parameter interventions to restore performance in malfunctioning data science pipelines.

Question

In what sense are the found interventions minimal, and are they minimal with respect to cost, edit distance, or utility recovery?

Source video

Connections

5