I agree that it almost certainly isn't legal or enforceable. But how many debut and midlist writers do you know who have the resources to take it to court to find out? Big publishers with their millions in revenue and teams of lawyers can still hold these clauses over the individual writer's head.
Or they can use such clauses as an excuse to cancel a contract. I remember the story of Kiana Davenport, traditionally published award winning writer, who signed a contract in 2010 for a book with a Big 5 imprint. It wasn't going to be published until 2012 and she needed some income, so she self published a collection of short stories that had already been rejected by publishers many times (including by the publisher she was contracted with). However, when the publisher found out they went "ballistic" and demanded that she take the volume down and attempt to delete all record of it from the internet. (heh) She refused (the short stories had nothing to do with the civil war era love story she was contracted with them for) and so they cancelled the contract and demanded she repay the advance.
So let me give you what I think was the takeaway on that Davenport case...Although I should note that the contract was never made public so we are all "guessing" on what it said. But here goes.
I don't think she violated non-compete...what she probably violated was a nasty little clause that is added to the indemnity section. Usually that section say things like I attest that I wrote this book and no one else can claim they did, blah, blah, blah...but recently they've been inserting.... "I attest that this work will be my next published work." For my first contract, I didn't fight that clause because I had no other book that was coming out before my trad books were. But for my second contract...I had another book waiting in the wings (that I planned to self-publish) so I got that struck.
My guess is she didn't read her contract carefully and that clause snuck in. Given that she signed it...the publisher had every right (although it was a crappy thing to do) to make her abide by the contract. If she wasn't willing to abide by what she signed...they had every right to cancel the contract...and if the contract said the advance is refundable if the book isn't published (and many do say that) then again the publisher followed what the contract said. I felt bad for Kiana, but I still am 99.9% sure that it's her own fault for signing something that she didn't realize the repercussions of.
Now, this exact sort of thing isn't likely to happen anymore, since publishers are much more accepting of authors self publishing these days. But it's an example of the kinds of lengths that publishers can go to thanks to the language they put in their contracts. And you can't believe them when they say things like "oh, well, that's just standard boilerplate, but we would never exercise it, trust us". Whenever you sign a contract you need to read it in light of what's the worst case scenario that can come from it. And if that worst case is something you don't want to have to deal with, then you need to get it changed.
I do think such clauses still are being put in...and yes I think the publisher hopes you don't notice...but the point you make is 100% correct - you have to look at "worst case scenaios" and make sure that you aren't boxing yourself in because you didn't pay attention to what you signed. If you sign a "bad" contract - you have only yourself to blame...and if you sign a contract...you should expect them to enforce it.