oobabooga/text-generation-webui
1
0
Fork 0
mirror of https://github.com/oobabooga/text-generation-webui.git synced 2026-02-24 16:46:36 +01:00
Code Issues Actions Packages Projects Releases Wiki Activity

Make exllamav3 safer as well

Browse source
This commit is contained in:
oobabooga 2025-04-18 06:17:58 -07:00
parent 03544d4fb6
commit 1c4a2c9a71
1 changed files with 3 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

3
modules/exllamav3_hf.py
View file

@ -153,6 +153,9 @@ class Exllamav3HF(PreTrainedModel):
else:
self.past_seq = seq_tensor
if torch.cuda.is_available():
torch.cuda.synchronize()
loss = None
if labels is not None:
# Shift so that tokens < n predict n