QuantizedModelLoader¶

Loader for quantized models saved by OneComp.

On macOS, load_quantized_model() places the model on MPS when available (CUDA > MPS > CPU via get_default_device()). Use Transformers generate() for inference; vLLM requires Linux with an NVIDIA GPU. See the macOS / MPS guide.

QuantizedModelLoader ¶

Loader for quantized models saved by onecomp (GPTQ, DBF, OneBit, etc.).

load_quantized_model `classmethod` ¶

load_quantized_model(save_directory: str, *, torch_dtype: Optional[dtype] = None, device_map: str = 'auto', trust_remote_code: bool = True, local_files_only: bool = True) -> Tuple[Any, Any]

Load a quantized model and tokenizer from a safetensors directory.

The directory must contain: - config.json (with quantization_config) - tokenizer files - model.safetensors (quantized layers: qweight/scales for GPTQ, scaling0/bp for DBF)

Quantization parameters (quant_method, bits, group_size, etc.) are read from config.json and quantized layers are reconstructed directly from the safetensors state_dict. No quantization_results.pt is needed.

For models saved with post-processing modifications (e.g. LoRA adapters), use :meth:load_quantized_model_pt instead.

Parameters:

Name	Type	Description	Default
`save_directory`	`str`	Path to the saved model directory.	required
`torch_dtype`	`Optional[dtype]`	Model dtype (default: torch.float16).	`None`
`device_map`	`str`	Device placement (default: "auto").	`'auto'`
`trust_remote_code`	`bool`	Passed to from_pretrained.	`True`
`local_files_only`	`bool`	Passed to from_pretrained.	`True`

Returns:

Type	Description
`Tuple[Any, Any]`	(model, tokenizer)

Example

model, tokenizer = QuantizedModelLoader.load_quantized_model("./tinyllama_gptq3")

load_quantized_model_pt `classmethod` ¶

load_quantized_model_pt(save_directory: str, *, device_map: str = 'auto', local_files_only: bool = True, allow_unsafe_deserialization: bool = False) -> Tuple[Any, Any]

Load a quantized model and tokenizer saved as a PyTorch .pt file.

Use this method to load models saved by :meth:Runner.save_quantized_model_pt, which preserves custom module types (e.g. LoRAGPTQLinear from LoRA post-processing).

The directory must contain: - model.pt (serialized with torch.save) - Tokenizer files

.. warning:: This method deserializes model.pt with torch.load(..., weights_only=False). Because PyTorch .pt checkpoints use Python's pickle, a maliciously crafted model.pt can execute arbitrary code during deserialization (CWE-502). weights_only=False is required here because the .pt format preserves full custom module objects (e.g. LoRAGPTQLinear) that cannot be reconstructed from tensors alone. Only load model.pt files that you produced yourself or obtained from a fully trusted source. For untrusted or third-party models, prefer the safetensors-based :meth:load_quantized_model, which does not execute code.

Parameters:

Name	Type	Description	Default
`save_directory`	`str`	Path to the saved model directory.	required
`device_map`	`str`	Device placement (default: `"auto"`). Set to `""` or `None` to skip device placement.	`'auto'`
`local_files_only`	`bool`	Passed to `AutoTokenizer.from_pretrained`.	`True`
`allow_unsafe_deserialization`	`bool`	Must be explicitly set to `True` to acknowledge the unsafe-deserialization risk described above and permit loading. Defaults to `False`, in which case this method raises before any code can be executed.	`False`

Returns:

Type	Description
`Tuple[Any, Any]`	(model, tokenizer)

Raises:

Type	Description
`ValueError`	If `allow_unsafe_deserialization` is not `True`.

Example

model, tokenizer = QuantizedModelLoader.load_quantized_model_pt( ... "./quantized_model_lora", ... allow_unsafe_deserialization=True, # trusted source only ... )

Convenience Functions¶

The top-level aliases provide shortcuts for both formats:

from onecomp import load_quantized_model, load_quantized_model_pt

# Load a safetensors model (standard quantized, no LoRA)
model, tokenizer = load_quantized_model("./saved_model")

# Load a PyTorch .pt model (post-processed, e.g. LoRA-applied)
# Requires explicit opt-in: the .pt loader uses torch.load(weights_only=False),
# which can execute code from a malicious file (CWE-502). Only enable this for
# model.pt files from a fully trusted source.
model, tokenizer = load_quantized_model_pt(
    "./saved_model_lora", allow_unsafe_deserialization=True
)

Unsafe deserialization (.pt loader)

load_quantized_model_pt() loads model.pt with torch.load(..., weights_only=False). Because PyTorch .pt checkpoints use Python pickle, a maliciously crafted model.pt can execute arbitrary code during loading (CWE-502). The method refuses to load unless you pass allow_unsafe_deserialization=True. Only opt in for models you produced yourself or obtained from a fully trusted source. For untrusted or third-party models, prefer the safetensors-based load_quantized_model(), which does not execute code.

QuantizedModelLoader¶