Skip to main content
CUDA-specific pip/conda wheel version does not match the base image’s CUDA toolkit.
PropertyValue
SeverityWarning
CategoryCorrectness
DefaultEnabled
Auto-fixYes (suggestion)

​
Description

Detects RUN instructions where pip, uv, or conda installs reference a CUDA version that does not match the base image’s CUDA toolkit version. A mismatch can cause:
  • Silent fallback to CPU execution
  • Runtime CUDA errors or build failures
  • Subtle performance degradation
The rule supports four detection paths:
  1. pip/pip3 package suffixes — torch==2.0.0+cu118
  2. pip/pip3/uv index URLs — --index-url https://download.pytorch.org/whl/cu118
  3. uv --torch-backend — --torch-backend cu118
  4. conda/mamba/micromamba — pytorch-cuda=11.8 or cudatoolkit=11.8

​
Why this matters

  • Silent CPU fallback — PyTorch may load but silently use CPU instead of GPU when the CUDA wheel version doesn’t match the available CUDA runtime
  • Runtime crashes — mismatched CUDA versions can produce cryptic CUDA error messages at runtime
  • Copy-paste bugs — the most common real-world pattern is upgrading the base image CUDA version but forgetting to update the pip --index-url suffix

​
Examples

​
Violation

# Base provides CUDA 12.1, but pip installs CUDA 11.8 wheels
FROM nvidia/cuda:12.1.0-devel-ubuntu20.04
RUN pip install --index-url https://download.pytorch.org/whl/cu118 torch

# Base provides CUDA 12.4, but --extra-index-url uses CUDA 11.8
FROM nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
RUN pip install --extra-index-url https://download.pytorch.org/whl/cu118 xformers

# Base provides CUDA 12.2, but conda installs CUDA 11.8 pytorch
FROM nvidia/cuda:12.2.0-devel-ubuntu22.04
RUN conda install -y pytorch pytorch-cuda=11.8 -c pytorch -c nvidia

# Base provides CUDA 12.4, but uv uses CUDA 11.8 backend
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04
RUN uv pip install --torch-backend cu118 torch

​
No violation

# Exact match: CUDA 11.8 base with cu118 wheels
FROM nvidia/cuda:11.8.0-base-ubuntu22.04
RUN pip install --index-url https://download.pytorch.org/whl/cu118 torch

# Forward-compatible: cu121 wheel on CUDA 12.6 base (older minor is fine)
FROM nvidia/cuda:12.6.0-runtime-ubuntu22.04
RUN pip install --index-url https://download.pytorch.org/whl/cu121 torch

# Non-CUDA base: rule does not apply
FROM ubuntu:22.04
RUN pip install --index-url https://download.pytorch.org/whl/cu121 torch

​
Version compatibility

The rule uses NVIDIA’s forward compatibility guarantee:
  • Same major, wheel minor at or below base minor — OK (forward-compatible)
  • Same major, wheel minor above base minor — Mismatch (wheel needs newer CUDA)
  • Different major — Always a mismatch

​
CUDA suffix mapping

SuffixCUDA version
cu11811.8
cu12112.1
cu12412.4
cu12612.6
cu12812.8

​
Fix suggestions

The rule offers two fix alternatives:
  1. Update the wheel/index to match the base image — preferred when the base image has a higher CUDA version (the common case: base was upgraded but pip URL was not)
  2. Update the base image to match the wheel — preferred when the wheel targets a newer CUDA version than the base
Both fixes use FixSuggestion safety — verify the target wheel or image tag exists before applying.

​
Applicability

This rule fires when:
  • The base image is nvidia/cuda:* (or docker.io/nvidia/cuda:*)
  • The CUDA version can be parsed from the image tag
  • A CUDA version reference is found in a RUN instruction
  • In multi-stage builds, stages that inherit from a CUDA-based parent stage (FROM builder where builder uses nvidia/cuda:*) also trigger the rule. In this case, only the “update wheel/index” fix is offered — the “update base image” fix is skipped since the FROM line references a stage name, not an image tag.
It does not fire on:
  • Non-NVIDIA base images
  • Digest-only or ARG-based image tags (version cannot be determined)
  • pip installs without CUDA suffixes or index URLs

​
Configuration

This rule has no rule-specific options. 
[rules.tally."gpu/cuda-version-mismatch"]
severity = "warning"

​
References