LLM Resources

LLM QuantizationPermalink

• 4-bit transformers

    from transformers import AutoModelForCausalLM
  
    model = AutoModelForCausalLM.from_pretrained("facebook/opt-350m", load_in_4bit=True, device_map="auto")
  

• Intro to 8-bit transformers multiplication