As others have said the best option is to make these people not human. It would be difficult for readers to accept that an entire human society could be short-sighted enough to kill off all the people who know how to farm. Human raiders would either enslave or vassalize farmers. An orc or goblin or other ficitonal being on the other hand could be capable of this.
The other option is to have the raiders be part of some sort of cult. There are many real-world examples of cult members doing very strange and self-harmful things. Perhaps the raiders believed that their deity/deities would come down to earth and give them free lasagna for the rest of their lives if they killed off all the farmers.
The other option is to have the raiders be part of some sort of cult. There are many real-world examples of cult members doing very strange and self-harmful things. Perhaps the raiders believed that their deity/deities would come down to earth and give them free lasagna for the rest of their lives if they killed off all the farmers.