Arnav Podichetty

the pipeline, end to end

1. 01Profile every column and null rate, emit a per-column keep/drop/standardize spec, then apply it deterministically. Cleaning policy lives in spec files, not buried in code.
2. 02Restore utility that rolls a merged table back to baseline schema and de-duplicates keys, so a bad merge is recoverable instead of fatal.
3. 03Merge layer that left-joins cleaned tables into patient-level datasets, skipping columns already present so reruns stay idempotent.
4. 04Cohort exclusions, outcome attachment, and a stratified train/test split into model-ready artifacts.
5. 05EDA over missingness, class balance, and target-column lineage, feeding an explicit feature-triage decision.
6. 06Preprocess and train: one-hot encoding, KNN imputation, scaling, an XGBoost baseline with balanced sample weights, Optuna tuning, SHAP attributions, and a serialized bundle.
7. 07Inference and validation: reload the bundle, confirm it reproduces training metrics on held-out test, sweep decision thresholds, and predict for a single patient or a batch.
8. 08Model comparison: three classifiers tuned on an identical budget, ranked on identical metrics, logged to MLflow, winner serialized.

data foundation

• Profiled, spec’d, and cleaned 24 clinical trial source streams into 72 inspectable CSV artifacts: 1,868 profiling records, 1,868 cleaning rules, and 38,512 standardized rows across 24 cleaned tables totalling 1,167 columns.
• Normalized inconsistent yes/no conventions into machine-readable binaries and stamped import metadata during cleaning for provenance.
• Read multi-sheet Excel workbooks and CSVs through pandas, keeping every intermediate layer (profiles, specs, cleaned tables) on disk as plain CSV so any step stays inspectable and tool-agnostic.
• Produced 641×657 train and 275×657 test sets after cohort exclusions and outcome attachment.

reliability & recovery

• Deduplicated multi-row-per-patient sources before every left join, so one-to-many tables couldn’t silently multiply the cohort.
• Made merges idempotent by skipping columns already present, so reruns stopped bloating the schema.
• Aligned the second cohort’s patient IDs against the first with a deterministic key offset, so joins failed loudly rather than missing quietly.
• Stripped pipeline metadata columns before final output so process artifacts never leaked into the feature table.
• Wrote a rollback utility that truncates a merged file to baseline schema and de-duplicates by key, which made integration safe to iterate on.
• Validated merged output on shape, key duplication, and overlap counts rather than trusting the join.

modeling & evaluation

• XGBoost baseline with balanced sample weights, evaluated on confusion matrix, ROC AUC, and per-class precision, recall, and F1.
• Tuned with Optuna across 50 trials per model, scored inside a manual stratified 3-fold CV loop that recomputes class weights per fold. Tuning against a single split would have inflated the numbers.
• Compared XGBoost, LightGBM, and logistic regression from the same preprocessing, on the same splits, metrics, and tuning budget, so the choice was evidence rather than preference. The leaderboard landed around 0.72–0.76 ROC AUC depending on model class.
• SHAP attributions on the winning model, handling LightGBM’s list-shaped output against the direct arrays the others return.
• Threshold sweeps plotting precision, recall, and F1 against the decision cutoff, the trade-off clinicians actually have to pick a point on.
• Logged every run, parameter, and metric to MLflow on a SQLite backend, so the leaderboard is reproducible rather than a screenshot.
• Serialized preprocessor, imputer, scaler, model, and metadata together as joblib bundles, so inference never depends on notebook state.

leakage & governance

• Triaged pre- against post-procedure features and dropped identifier, post-outcome, free-text, and near-constant columns. Adverse-event fields correlate hard with the outcome, for exactly the wrong reason.
• Traced the target column’s lineage through the merges to confirm it still meant upstream what it claimed to mean.
• Quantified ~40% missingness across model inputs and documented an imputation strategy rather than letting library defaults decide it silently.

the ramp

The co-op ran on a structured mentorship plan: two weeks of pure skill building, four working the pipeline under guidance, then three contributing code. Each week closed on one build rather than an exercise.

• Wk 1–2NumPy and pandas fluency, then logistic regression, random forest, XGBoost, and CatBoost trained from scratch on a public cardiac dataset. Judged on AUROC, Brier score, and calibration rather than accuracy, because a 6% event rate makes accuracy meaningless.
• Wk 3–4Clinical grounding in aortic stenosis and the TAVI procedure, a read-only tour of the codebase, then the first production-style script: a CLI data-quality checker emitting a JSON report on missingness, class balance, and constant columns.
• Wk 5–6XGBoost trained on real patient outcomes with Optuna tuning and strict NaN-label handling, then SHAP over the test set and a written clinical-validity review checking the top features against domain expectations and for leakage artifacts.
• Wk 7A three-layer Keras MLP with batch norm, dropout, and AdamW, benchmarked head-to-head against the XGBoost baseline to see where deep learning does and doesn’t pay on tabular clinical data.
• Wk 8Traced the team’s multi-task architecture end to end: a BiLSTM with multi-head attention over the temporal stream, fused with a static encoder into several task-specific prediction heads. Wrote the explainer document for it.
• Wk 9Ran the full pipeline, then contributed a stacking meta-learner over out-of-fold predictions from the tree models, comparing logistic, ridge, and elastic-net combiners against a plain averaging baseline.

ai-assisted documentation

• A second workstream building GenAI workflows for SOP summarization, definition extraction, and requirements generation over dense internal process documents.
• Cut manual review effort and made the generated artifacts consistent enough to reuse, turning language-heavy inputs into structured, actionable deliverables.

handoff

• Wrote a pipeline map and run instructions so the next person could rerun the whole thing without re-deriving it.
• Maintained a cleaning tracker with per-dataset status, priority ranking, and merge guidance.
• Documented scope boundaries explicitly, separating what was finished from what was groundwork, so the report-out didn’t overstate model maturity.

stack