There is now a topic with a bunch of mirrored questions from the 2024 ACX prediction contest on Metaculus. This question asks what strategy for predicting these questions will get the best Brier score on all binary questions in this topic as of January 31st.
User submissions are valid, with a few caveats:
It must be somewhat reasonable to compute the prediction on January 31st.
Submissions must be described in 240 characters or less.
New submissions after the 31st will not be accepted (I will close the market on the 31st, disable new submissions, and then reopen).
Referencing other predictions is allowed, as long as there is provably no way that it could lead to a paradox.
If there is a tie for first I will resolve to % equally among the winners
I suggest an adversarial mindset.
1,000
3.00Did not get to assembling my spreadsheet for this earlier this year, with my defense I was busy.
Here is the spreadsheet now.
Note, there seems to be some missing data: Zvi did not answer the US Congress LLMs question, and Metaculus did not answer the crewed artemis question (awkwardly delay was announced before the 31st, making the question essentially irrelevant).
I think the fairest thing to do is:
Give metaculus a 0.00 prediction on the artemis one, since I feel dqing the question is effectively saying that the obvious result will happen
Give zvi the metaculus prediction for the day his post came out on the LLM question, since his predictions were based on this information.
@traders LMK if there is any objection to this, or if any of my values are wrong.
Edit also note that for convenience, I am largely rounding to nearest percent.
@BoltonBailey I'd argue that the Artemis question is no longer part of the "2024 ACX prediction contest", and should be ignored here too.
The contest was intended as a fixed point in time, not one where you needed to go back and update, that's why it was removed.
@DanielTilkin Well, the criteria says "binary questions in the topic" not "questions that were a part of the contest". Ultimately it will not affect the scores too much either way.
@BoltonBailey Yeah, I guess the criteria does disagree with the description. I agree though that I don’t expect it to make a difference.
Looks like these were Jack, Manifold, and Zvi at the end of January when this closed.
I'm writing my prediction post now. I want to clarify how it's going to work so you can set this market correctly (I'm going to link to it because I think it's fun).
First I predict each question blind meaning I look at financial markets but not prediction markets.
Second I look at Manifold and make trades (which are in, so if you want to know what side I'm on, you know, if it's M10 or less I'm just tracking mostly)
Third is the number I'm settling on and entering into Metaculus.
The third number is the one you should use for Briar score here, although of course someone is welcome to also put in a market option for the blind ones. In which case, it's going to heavily depend on the few places I messed up and got a wildly off initial answer!
I won't trade on this market unless it gets a lot more liquidity, more fun to watch it play out, but I will say that if there was more liquidity I would definitely make some trades.
@ZviMowshowitz 😃 Ok, thanks for letting us know!
@Joshua I agree with Zvi that the Metaculus prediction is the one that makes the most sense for the answer you gave, so I'm editing it to clarify that. If you want to add the other ones you can do that too, or I might add them myself.
@Joshua the Manifold Calibration says that we're overconfident on average, but I don't think it says by how much?
Or am I missing that somewhere?
Lol if this and the individual manifold and metaculus ones are the top three then this is just the same as the top-three non-meta ones averaged.
@BoltonBailey Ok quick rules update, if there is a tie for first I resolve to prob equally among the winners that seems most sensible.
@BoltonBailey I guess you could say non-meta is anything that doesn't reference the market itself? But then I could make a side market that references this market and have the side market reference this one. Maybe "nothing with this market in the closure of things it recursively could reference"?
@BoltonBailey options which don't reference any other options in this market directly or indirectly are non-meta, and non-other is just ruling out "Other" from which answers are split.