This matches my subjective impressions of testing 8bit vs 4bit Llama. I don’t notice much degradation of capability at 8bit, but definitely do notice at 4bit.
This matches my subjective impressions of testing 8bit vs 4bit Llama. I don’t notice much degradation of capability at 8bit, but definitely do notice at 4bit.