He thought you were the one person in the world with as much reason to hate the Cullens as he does. Sam feels sort of, betrayed that you would just let them back into your life like they never hurt you.