Essential Guide: Quantization - Comprehensive Guide

👋 Sign in for the ability to sort posts by relevant, latest, or top.

Huilin Ye

Oct 30

FP8 Cut Our TPOT by 25%. At High Concurrency, It Also Changed the Answers.

#llm #vllm #quantization #benchmarking

3 min read

amrit

Oct 26

Why T Quant is the Secret Weapon for Mobile AI in 2026

#diffusionllm #quantization #mobileai #ai

6 min read

GINIGEN AI

Oct 19

LFM2.5-24B-A2B on Your Own Hardware: The RAM, KV Cache and Bandwidth Budget

#ai #machinelearning #edgeai #quantization

11 min read

amrit

Oct 12

How One Startup Cut Diffusion‑LLM Inference Costs by 45 % – No Calibration Needed

#ai #quantization #diffusionllm

6 min read

Muhammad Tahir

Oct 12

Model Distillation & Quantization: Running 8B Parameter Models at a Fraction of Cloud Cost

#modeldistillation #quantization #vllm #awq

10 min read

xbill for Google Developer Experts

Oct 30

Four GPUs, One Quantization Ladder: Gemma 4 E2B From a 4 GB Laptop to an NVIDIA L4

#gemma #llamacpp #quantization #nvidia

11 min read

Jon Handler

Oct 8

From 1.3 TB to 66 GB: On-Disk Vector Search in OpenSearch

#vectorsearch #airetrieval #opensearch #quantization

8 min read

xbill for Google Developer Experts

Oct 29

Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4

#gemma #llamacpp #aws #quantization

11 min read

Jon Handler

Oct 6

Native Quantization: Let OpenSearch Service Compress Your Vectors

#vectorsearch #airetrieval #opensearch #quantization

6 min read

GINIGEN AI

Oct 6

4-bit GGUF Quality for MoE Models: Why Only 3B of 180B Params Fire, and How to Prove Parity

#ai #llm #quantization #edgeai

7 min read

xbill for AWS Community Builders

Oct 29

Performance vs Quality: Gemma 4 E2B Quantization on an NVIDIA L4 and a Tesla T4

#gemma #llamacpp #aws #quantization

11 min read

GINIGEN AI

Oct 6

Running a 180B Model on a Laptop With No GPU: How 4-bit GGUF Keeps Full Accuracy

#ai #llm #edgeai #quantization

6 min read

Reno Lu

Oct 2

Qwen3.8-27B on one RTX 3090, and the setting its README says chat clients should leave at default

#vllm #localllm #speculativedecoding #quantization

4 min read

Jon Handler

Oct 2

1.3 TB of RAM for 100M Vectors? Shrink Them Before They Hit OpenSearch

#vectorsearch #airetrieval #opensearch #quantization

8 min read

Includes a tricky broadcast K/V debugging story

xbill for Google Developer Experts

Oct 24

Gemma 4 on a 4 GB GPU in Plain PyTorch: From Out of Memory to Faster Than llama.cpp

#gemma #pytorch #cuda #quantization

13 min read

👋 Sign in for the ability to sort posts by relevant, latest, or top.