Yes or No

When I was a little kid, questions like "so you don't want this?" used to confuse me. Should I say yes or no? I usually replied "yes," meaning yes, you are correct, I don't want it, but you are suppose to say no meaning no, I do not want it.