oobabooga/text-generation-webui
1
0
Fork 0
mirror of https://github.com/oobabooga/text-generation-webui.git synced 2026-04-05 06:35:15 +00:00
Code Issues Projects Releases Packages Wiki Activity

ExLlamaV3: Add kv cache quantization (#6903)

Browse source
This commit is contained in:
oobabooga 2025-04-25 21:32:00 -03:00 • committed by GitHub
parent d4b1e31c49
commit d4017fbb6d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
4 changed files with 32 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

2
modules/loaders.py
View file

@ -13,6 +13,7 @@ loaders_and_params = OrderedDict({
'cache_type',
'tensor_split',
'extra_flags',
'streaming_llm',
'rope_freq_base',
'compress_pos_emb',
'flash_attn',
@ -49,6 +50,7 @@ loaders_and_params = OrderedDict({
],
'ExLlamav3_HF': [
'ctx_size',
'cache_type',
'gpu_split',
'cfg_cache',
'trust_remote_code',