tally/gpu/cuda-version-mismatch

​

Description

• Silent fallback to CPU execution
• Runtime CUDA errors or build failures
• Subtle performance degradation

1. pip/pip3 package suffixes — torch==2.0.0+cu118
2. pip/pip3/uv index URLs — --index-url https://download.pytorch.org/whl/cu118
3. uv --torch-backend — --torch-backend cu118
4. conda/mamba/micromamba — pytorch-cuda=11.8 or cudatoolkit=11.8

​

Why this matters

• Silent CPU fallback — PyTorch may load but silently use CPU instead of GPU when the CUDA wheel version doesn’t match the available CUDA runtime
• Runtime crashes — mismatched CUDA versions can produce cryptic CUDA error messages at runtime
• Copy-paste bugs — the most common real-world pattern is upgrading the base image CUDA version but forgetting to update the pip --index-url suffix

​

Examples

​

Violation

# Base provides CUDA 12.1, but pip installs CUDA 11.8 wheels
FROM nvidia/cuda:12.1.0-devel-ubuntu20.04
RUN pip install --index-url https://download.pytorch.org/whl/cu118 torch

# Base provides CUDA 12.4, but --extra-index-url uses CUDA 11.8
FROM nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04
RUN pip install --extra-index-url https://download.pytorch.org/whl/cu118 xformers

# Base provides CUDA 12.2, but conda installs CUDA 11.8 pytorch
FROM nvidia/cuda:12.2.0-devel-ubuntu22.04
RUN conda install -y pytorch pytorch-cuda=11.8 -c pytorch -c nvidia

# Base provides CUDA 12.4, but uv uses CUDA 11.8 backend
FROM nvidia/cuda:12.4.0-runtime-ubuntu22.04
RUN uv pip install --torch-backend cu118 torch

​

No violation

# Exact match: CUDA 11.8 base with cu118 wheels
FROM nvidia/cuda:11.8.0-base-ubuntu22.04
RUN pip install --index-url https://download.pytorch.org/whl/cu118 torch

# Forward-compatible: cu121 wheel on CUDA 12.6 base (older minor is fine)
FROM nvidia/cuda:12.6.0-runtime-ubuntu22.04
RUN pip install --index-url https://download.pytorch.org/whl/cu121 torch

# Non-CUDA base: rule does not apply
FROM ubuntu:22.04
RUN pip install --index-url https://download.pytorch.org/whl/cu121 torch

​

Version compatibility

• Same major, wheel minor at or below base minor — OK (forward-compatible)
• Same major, wheel minor above base minor — Mismatch (wheel needs newer CUDA)
• Different major — Always a mismatch

​

CUDA suffix mapping

​

Fix suggestions

1. Update the wheel/index to match the base image — preferred when the base image has a higher CUDA version (the common case: base was upgraded but pip URL was not)
2. Update the base image to match the wheel — preferred when the wheel targets a newer CUDA version than the base

​

Applicability

• The base image is nvidia/cuda:* (or docker.io/nvidia/cuda:*)
• The CUDA version can be parsed from the image tag
• A CUDA version reference is found in a RUN instruction
• In multi-stage builds, stages that inherit from a CUDA-based parent stage (FROM builder where builder uses nvidia/cuda:*) also trigger the rule. In this case, only the “update wheel/index” fix is offered — the “update base image” fix is skipped since the FROM line references a stage name, not an image tag.

• Non-NVIDIA base images
• Digest-only or ARG-based image tags (version cannot be determined)
• pip installs without CUDA suffixes or index URLs

​

Configuration

[rules.tally.gpu.cuda-version-mismatch]
severity = "warning"

​

References

• PyTorch Previous Versions
• NVIDIA CUDA Docker Hub
• uv PyTorch integration
• NVIDIA CUDA Compatibility