Add transformers integration section #1904
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SunMarc
approved these changes
Mar 14, 2024
Co-authored-by: Marc Sun <[email protected]>
Co-authored-by: Marc Sun <[email protected]>
Co-authored-by: Marc Sun <[email protected]>
Co-authored-by: Marc Sun <[email protected]>
dacorvo
reviewed
Mar 15, 2024
quanto-introduction.md
Outdated
| ) | ||
| ``` | ||
|
|
||
| You can quantize the weights and / or activations in int8, float8, int4 or int2 by simply passing the correct argument in `QuantoConfig`. The activations can be either in int8 or float8 - note for float8 you need to have hardware that is compatible with float8 precision, otherwise quanto will silently upcast the weights and activations to torch.float32 or torch.float16 (depending on the original data type of the model) when we perform the matmul |
There was a problem hiding this comment.
What you describe here also happens if activations are not quantized, because mixed mm are not supported by any hardware. Note also that pytorch will raise an exception if you try to use float8 on MPS device.
There was a problem hiding this comment.
Makes sense, ! I added something in the lines you suggested
quanto-introduction.md
Outdated
|
|
||
| Quanto is device agnostic, meaning you can quantize your model regardless if you are on CPU / GPU / MPS (Apple Silicon) therefore you can run quantized models on any of these devices. | ||
|
|
||
| Quanto is also torch.compile friendly, you can quantize a model with quanto and call `torch.compile` to the model to compile it for faster generation. |
There was a problem hiding this comment.
This won't work if there is some dynamic quantization is involved, ie:
- QAT,
- quantized activations.
QTensor cannot be created from inside a dynamo graph optimum-quanto#46
So basically for frozen models it is ok (I think this is always the case in transformers).
There was a problem hiding this comment.
I added few explanations of the edge cases in b778f8d ! Also note one can change the activation type with transformers integration, so users should keep them to None for torch.compile
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
cc @dacorvo @SunMarc