Items tagged gpt3, andrejkarpathy in 2023
Filters: Year: 2023 × gpt3 × andrejkarpathy × Sorted by date
2 results
The most dramatic optimization to nanoGPT so far (~25% speedup) is to simply increase vocab size from 50257 to 50304 (nearest multiple of 64). This calculates added useless dimensions but goes down a different kernel path with much higher occupancy. Careful with your Powers of 2.
— Andrej Karpathy # 4th February 2023, 12:08 am
nanoGPT. “The simplest, fastest repository for training/finetuning medium-sized GPTs”—by Andrej Karpathy, in about 600 lines of Python. # 2nd January 2023, 11:27 pm