You can totally go back and kill your grandfather. The universe won’t stop you; it doesn’t care if things don’t make sense. Bang, he’s dead. You’ll never be born.
So when you travel forward to the time you came from, you’re out of place. It’s not that you were never born–of course you were born; you’d remember it if your brain had been wired for it back then–but you were never born here. You arrive at a world in which you’ve never existed. You are utterly alone, with no birth, friends or connection to the world. No one exists who can look at you and say, “Oh hi, I know you.”
So don’t go back and kill your grandfather. You wouldn’t like it.
Though it seems a great premise for a sci-fi about assassins.