The Dursleys are not what they seem. When the Potters are murdered and a baby is left on their doorstep, they act. (If you haven't read 'Agents of the Dragon King', this won't make sense.)
What if some squibs were more powerful than wand users believed? What if the Dursleys were cursed to hate Harry? (Dark Lord Dumbledore, BAMF goblins, more or less competent adults) Edits finally complete (I hope)
Mother Magic took her children to an Earth-like planet in a far galaxy. She didn't bother to tell them exactly where or when. (Couldn't resist a sequel.)