[Source: TomperWomper on Deviantart]
didn’t they technically build him together, but onna’s memories were erased or something?
Nope, she remembers everything she did with Elsa, she only forgot about her powers. That’s why Anna also get surprised when meeeting Olaf. She remembers he is identical to the snowman they built as children