When me and my buddy use this phrase, it's usually used as shorthand for saying 'x character completely blanked out for a hot second' typically for when trying to figure out what (typically absurd thing) was just said to them, or maybe make sense of the situation they were thrown into. Rarely do...
Magic is generally used as a descriptor from someone outside of the realm of understanding of a thing or phenomena, in my mind at least. Since wizards, sorcerers, and the like know what they're doing, would they consider what they're doing as magic? What would they even call it if they didn't...