@traders I'm resolving this based on personal judgment to save time. If you disagree, please DM me and I'll refund you.

@singer presumably this is deepseek? I think you shouldn't refund people who disagree with you but that's just me

@Bayesian it's not any model in particular. it's been so long since I used gpt3.5 that I really can't remember how good it was compared to the local models I've been using. I offered to refund because I resolved based on vibes

Note (I'm adding this comment to a few of my markets): I was hoping to do regular early tests of this but it's too far back on my backlog right now. I'm still committing to resolving this properly at the end of the year, however.

Mixtral has just pulled ahead of gpt3.5. When I have time I'll see if I can find a quantized model I can run that doesn't do more than 2% worse on the Winograd Schema Challenge.

Mixtral is currently tied. Can you run it (at least quantized)?

@ProjectVictory I noticed that tie, yeah. I'm not sure how to deal with the case of quantized models. EDIT: see below

@ProjectVictory
This is what I'm thinking of doing:

For a quantized model to be eligible, it cannot differ more than 2% from the original model's score on the Winograd Schema Challenge.