Here’s a theory. What if your allergy is really an inability to digest the food.. Perhaps from a low metabolism, you’re not making the enzymes to process it well etc. So when you eat this food, you don’t digest it well and you don’t get the full benefit from it. So your body wants it all the more to try to get those nutrients? Keeps hitting the “we need dairy” button because it’s not working well? Kind of like how our remote control here has not been working well lately and everyone’s mashing the crap out of the buttons to change the channel.
And perhaps eating said food will eventually help you process it. I know Wheat used to make me hurt, after I had been off it for a while. Like hurt really bad, like miserable. I started eating it and kept eating it and actually in about a week I was fine eating it and I eat it every day now. I don’t get sore near as easily as I used to in general either. Like I hauled off and started riding by bike over 10 miles and didn’t get much soreness at all. I used to get sore at any increase in physical activity.