MIPCandy: A Modular PyTorch Framework for Medical Image Processing

MIPCandy is Project Neura's modular PyTorch framework for medical image processing, co-first-authored with Tianhao Fu during my research internship at UTMIST (University of Toronto Machine Intelligence Student Team). Our paper is available on arXiv (2602.21033).

The Problem

Existing medical image processing tools fall into two extremes: low-level component libraries (like MONAI and TorchIO) that require substantial integration effort, and rigid monolithic pipelines (like nnU-Net) that resist modification. Researchers either spend weeks assembling infrastructure or struggle to customize opinionated workflows.

What MIPCandy Does

MIPCandy bridges this gap. A researcher can obtain a fully functional segmentation workflow by implementing a single method (build_network()), while retaining granular control over every component. The framework spans data loading, training, inference, and evaluation -- all usable out of the box with researched defaults, yet fully customizable.

Four guiding principles shape the design:

1. PyTorch-native: All components are standard nn.Module subclasses, composable with existing PyTorch utilities.
2. Opt-in and Incremental: No module assumes other framework components are present. Individual pieces can be adopted independently.
3. Composition over Inheritance: The LayerT system enables runtime configuration (swapping convolution, normalization, or activation layers) without subclassing.
4. Minimal API Surface: The common case requires no configuration.

Key Contributions

• LayerT Configuration System: A deferred module descriptor that stores a module type with constructor arguments, instantiated at runtime. This enables swapping convolution, normalization, and activation layers across an entire architecture without subclassing.
• Dataset Inspection: The inspect() function automatically computes per-case foreground bounding boxes, class distributions, and intensity statistics, deriving optimal ROI shapes for patch-based training.
• Validation Score Prediction: Quotient regression fits a rational function to validation trajectories, estimating maximum achievable score, optimal stopping epoch, and time to completion.
• Bundle Ecosystem: Self-contained packages (model + trainer + predictor) following a consistent pattern. Shipped bundles include U-Net, UNet++, V-Net, CMUNeXt, MedNeXt, and UNETR.
• Training Transparency: Auto-generated metric curve plots, worst-case prediction previews (2D overlays and 3D PyVista meshes), and per-case validation tables highlighting underperforming cases.
• Multi-Frontend Tracking: Pluggable experiment tracking with Weights & Biases, Notion, and MLflow, supporting simultaneous use of multiple frontends.
• Training Infrastructure: Deep supervision, exponential moving average, full state recovery, k-fold cross-validation, and early stopping -- each enabled with a single flag.

My Role

As co-first author and research intern at Project Neura, I contribute to the framework's core architecture, bundle development, and the research presented in the paper.

Case Studies

The paper demonstrates MIPCandy through two case studies:

• 2D Skin Lesion Segmentation (PH2 Dataset): A minimal 8-line implementation using the U-Net bundle, producing model checkpoints, per-epoch metrics, training curves, and worst-case prediction previews.
• 3D Volumetric Segmentation (BraTS 2021): Automatic ROI computation via inspect(), RandomROIDataset with foreground oversampling, deep supervision, and multiclass segmentation with 3D visualization.