GPU & AI Cloud Glossary

VRAM (video RAM) is the dedicated memory on a GPU, used to hold model weights, activations, and intermediate tensors during computation. Unlike system RAM, VRAM is physically on the GPU die and connected to the compute cores via a high-bandwidth memory bus. The amount of VRAM directly limits the maximum model size you can load and the batch sizes you can run without spilling to slower system memory.

Why it matters when comparing providers: When comparing providers, VRAM per GPU determines which models fit on a single card and how many concurrent requests a node can serve — a key factor in per-token cost.

TFLOPS (tera floating-point operations per second) measures raw arithmetic throughput — how many trillion multiply-add operations a GPU can execute each second at a given numeric precision. Published TFLOPS figures are theoretical peaks; real workloads achieve a fraction of peak due to memory bandwidth limits and kernel efficiency. Precision matters: an H100 delivers roughly 2× the FP16 TFLOPS of an A100, but the gap narrows or widens depending on the operation mix.

Why it matters when comparing providers: TFLOPS per dollar is a common first-pass metric for comparing GPU value, but it must be read alongside memory bandwidth and VRAM — a high-TFLOPS GPU bottlenecked on memory will underperform on large-batch inference.

Floating-point precision formats describe how many bits are used to represent each number during computation. FP32 (32-bit) is the traditional training standard. FP16 (16-bit) halves memory use and roughly doubles throughput on hardware with native FP16 support, with minimal quality loss when used with loss scaling. FP8 (8-bit), supported on H100 and newer GPUs, halves memory and bandwidth requirements again versus FP16, enabling larger batch sizes and lower cost per token at the expense of slightly more careful calibration.

Why it matters when comparing providers: Providers that expose FP8 inference can serve the same model at lower cost or higher throughput than FP16-only providers — relevant when comparing per-token prices for the same model across APIs.

NVLink is NVIDIA's high-speed GPU-to-GPU interconnect, offering substantially higher bandwidth than PCIe for multi-GPU communication. In an 8-GPU NVLink node, all GPUs can exchange data at full NVLink bandwidth simultaneously, which is critical for tensor-parallel and pipeline-parallel training where gradient and activation tensors must move between GPUs on every forward and backward pass. PCIe-only multi-GPU nodes have a fraction of the cross-GPU bandwidth.

Why it matters when comparing providers: For large-model training or inference that spans multiple GPUs, NVLink nodes are significantly faster than PCIe-only alternatives — a meaningful differentiator when comparing node types across providers.

ECC (error-correcting code) memory detects and corrects single-bit memory errors in real time, preventing silent data corruption. Data center GPUs like the A100 and H100 ship with ECC enabled by default. Consumer GPUs (RTX series) typically lack ECC or offer it only in limited form. A single uncorrected bit flip in a model weight or gradient can silently corrupt a training run or produce wrong inference results.

Why it matters when comparing providers: Providers running consumer GPUs without ECC carry a higher risk of silent data corruption for long training jobs — relevant when evaluating lower-cost marketplace offerings against data center hardware.

How fast data moves between the GPU chip and its memory, measured in GB/s. Higher bandwidth usually means faster text generation, because the model is read from memory on every token.

A part of modern NVIDIA GPUs built specifically for the matrix math AI models use.

NVIDIA's software platform for running programs on its GPUs. Most AI software supports CUDA first, which is why NVIDIA cards are the default choice for AI work.

Splitting one model across several GPUs, used when a model is too large for a single card's VRAM.

On-demand pricing means you pay a fixed hourly (or per-second) rate for a GPU instance with no upfront commitment and no minimum term. The instance is available immediately and you can terminate it at any time. On-demand rates are the highest per-hour price a provider charges, reflecting the flexibility premium — you are not committing capacity in advance.

Why it matters when comparing providers: On-demand rates are the baseline for comparison across providers. They reflect the true cost of flexible, uncommitted GPU access and are the most relevant figure for short or unpredictable workloads.

Spot instances (also called preemptible or interruptible instances) offer unused GPU capacity at a discount versus on-demand rates, in exchange for the possibility of interruption when the provider reclaims the capacity. Interruption can happen with little or no notice. The discount varies by provider and GPU model. Workloads must be designed to checkpoint and resume, or to tolerate restarts, to use spot effectively.

Why it matters when comparing providers: Spot pricing can substantially reduce GPU costs for fault-tolerant workloads like training with checkpointing, but the interruption risk makes it unsuitable for latency-sensitive inference or jobs without restart logic.

Reserved instances (also called committed-use or reserved capacity) let you pay upfront or commit to a fixed term — typically one to three years — in exchange for a lower effective hourly rate than on-demand. The GPU capacity is guaranteed for the term. Some providers offer shorter-term reservations (monthly or quarterly) at a smaller discount. Breaking a reservation early typically forfeits the prepayment or incurs a penalty.

Why it matters when comparing providers: For stable, predictable workloads running continuously, reserved pricing can significantly reduce total cost versus on-demand — but the commitment risk means it is only appropriate when utilization is reliably high.

Per-second billing charges you only for the exact seconds a GPU instance is running, rather than rounding up to the nearest hour or minute. If a job finishes in 90 seconds, you pay for 90 seconds. Hourly billing rounds up to the full hour regardless of actual usage. Per-minute billing is a middle ground. The difference matters most for short, frequent jobs where hourly rounding would add significant overhead cost.

Why it matters when comparing providers: For workloads with many short jobs — batch inference, fine-tuning runs, or interactive experiments — per-second billing can meaningfully reduce waste compared to hourly-billed providers.

Egress fees are charges for data transferred out of a cloud provider's network — to the public internet, to another provider, or to your own infrastructure. Ingress (data coming in) is typically free. Egress rates vary widely across providers and are often not prominently advertised alongside compute pricing. For workloads that produce large outputs — model checkpoints, generated datasets, inference results — egress can become a significant line item.

Why it matters when comparing providers: When comparing total cost across providers, egress fees must be included alongside compute rates — a cheaper GPU hourly rate can be offset by higher data transfer costs for output-heavy workloads.

A setup where the provider starts a GPU only when your code runs and bills for the seconds it actually uses. No idle machine to manage, usually with a per-second price.

Providers that do not charge for data leaving their network. Worth checking before moving large datasets.

Tokens are the units LLMs process and generate — roughly corresponding to word fragments, punctuation, or short character sequences depending on the tokenizer. A common rule of thumb is that one token is approximately four characters or three-quarters of a word in English, but this varies by language and model. LLM API pricing is denominated in tokens: you pay separately for input tokens (your prompt) and output tokens (the model's response), with output typically priced higher.

Why it matters when comparing providers: Token counts determine your API bill. Understanding how your prompts and responses tokenize helps you estimate costs accurately and compare per-token rates across providers for the same model.

The context window is the maximum number of tokens a model can process in a single request — the combined length of your input prompt plus the generated output. Tokens beyond this limit are either truncated or cause an error. Larger context windows allow longer documents, multi-turn conversations, and more complex prompts, but processing long contexts requires more VRAM and compute, which is why some providers charge more for long-context requests.

Why it matters when comparing providers: When comparing providers for document processing, long-context retrieval, or multi-turn agents, verify both the advertised context window and whether the provider charges a premium for long-context usage.

Quantization reduces the numeric precision of a model's weights — for example, converting FP16 (16-bit) weights to 8-bit integers (INT8) or 4-bit integers (INT4). This shrinks the model's memory footprint and can increase inference throughput, at the cost of a small reduction in output quality. The quality impact varies by model, quantization method, and task; well-calibrated quantization is often imperceptible for general use, but may degrade performance on precision-sensitive tasks.

Why it matters when comparing providers: Providers serving quantized models can offer lower per-token prices or higher throughput than those serving full-precision weights — worth checking when a provider's price for a given model is notably lower than competitors.

Latency is the time it takes to generate a response for a single request — typically measured as time-to-first-token (TTFT) and tokens per second for that one user. Throughput is the total number of tokens the system can serve across all concurrent users per unit of time. These two metrics trade off against each other: batching multiple requests together increases throughput but adds latency for each individual request. Providers optimize their serving infrastructure differently depending on their target use case.

Why it matters when comparing providers: For interactive applications (chat, copilots), low latency matters most. For batch processing or high-volume pipelines, throughput per dollar is the relevant metric. A provider optimized for one may not be the best choice for the other.

Open weights means the trained weight files are publicly downloadable — you can run the model on your own infrastructure without going through the original developer's API. This is distinct from open-source in the strict sense: an open-weights model does not necessarily carry an OSI-approved open-source license. License terms vary significantly by model and release; some restrict commercial use, some restrict certain applications, and some are permissive for most uses. Always review the specific license before deploying commercially. Closed models, by contrast, are available only through the provider's API — the weights are never released, so self-hosting is not an option.

Why it matters when comparing providers: Open-weights models can be self-hosted on rented GPU compute, eliminating per-token API costs at scale — but require you to manage infrastructure, and license terms must be verified before commercial deployment.

The type of AI model behind chatbots and text APIs, such as the GPT, Claude, Gemini, and Llama families.

The internal numbers a model learns during training, counted in billions. Bigger models are often more capable but need more VRAM and cost more to run.

A model design where only part of the model runs per token.

The legal terms for using a model. Some open-weight licenses are permissive (MIT, Apache 2.0); others restrict commercial use or large deployments. Always check before a commercial project.

Running a trained model to get answers, as opposed to training or fine-tuning it.

The expensive process that creates a model from data. Fine-tuning is a lighter version that adapts an existing model to your data.

Input tokens the provider has seen before in the same form and can reuse at a steep discount. Long, repeated prompts benefit most.

A discounted API mode where requests are processed within hours instead of seconds, often at around half price. Good for large jobs that are not time-sensitive.

The caps a provider sets on requests per minute and tokens per minute for your account. They decide how much load one API key can carry.

A shared endpoint runs your requests on the provider's pooled capacity. A dedicated deployment reserves specific hardware for you.