Winston Smith
@smithwinst0n.bsky.social
Graduate Student, ML, CV, Robotics
Reposted by Winston Smith
One of the physics of llm papers studied that and found you need a certain amour of repetitions of a factoid before it’s memorized. Repetition can be either multi epochs or just the same fact in another document. Number of needed repeats is also related to model size.
December 13, 2024 at 4:27 PM
One of the physics of llm papers studied that and found you need a certain amour of repetitions of a factoid before it’s memorized. Repetition can be either multi epochs or just the same fact in another document. Number of needed repeats is also related to model size.
Very interesting approach, added to my read list!
December 12, 2024 at 9:33 AM
Very interesting approach, added to my read list!