Note (I'm adding this comment to a few of my markets): I was hoping to do regular early tests of this but it's too far back on my backlog right now. I'm still committing to resolving this properly at the end of the year, however.

Mixtral has just pulled ahead of gpt3.5. When I have time I'll see if I can find a quantized model I can run that doesn't do more than 2% worse on the Winograd Schema Challenge.

Mixtral is currently tied. Can you run it (at least quantized)?

@ProjectVictory I noticed that tie, yeah. I'm not sure how to deal with the case of quantized models. EDIT: see below

@ProjectVictory
This is what I'm thinking of doing:

For a quantized model to be eligible, it cannot differ more than 2% from the original model's score on the Winograd Schema Challenge.