Resolves YES if:

• None of the scenarios in NO, NA, or 50% section occur.

Resolves NO if any of the following occur:

• An unreplicable and closed LLM is verified to be created by Facebook before the end of 2024.

• No LLM-based product is verified to exist before 2024, and there is no strong evidence of an LLM-based product existing.

• There is strong evidence that LLM-based research papers are suppressed for publication due to competitive, safety, or other concern.

Resolves 50% if:

• Strong evidence arises that an LLM is created by Facebook before 2024 ends that will be replicated or opened. In this case, the market will be extended to June 2025 pending replication or opening.

• There is conflicting strong evidence for a resolution criterion.

Definitions

• An "affiliate of X" is an employee, owner, investor, contractor, or vendor signing an NDA of X. Customers bound by a Terms of Service are not considered affiliates.

• "Strong evidence" means a press release, journalist article, interview with an affiliate. Strong evidence can be refuted by strong evidence to the contrary, creating "conflicting strong evidence".

  • Example: If a whistleblower says, "My LLM research paper was rejected for publication by the censorship board!", Facebook may publish a refutation saying "No such paper exists" or "The paper exists and we were planning to release it, but it was pending review" and later releasing the paper(if the paper is not released on time, the refutation is rejected).

• "Verify" means either personally using it, observing someone I believe to be a non-affiliate using it, or seeing a journalist report access.

• "LLM" means an equivalence class of models(likely loadable in PyTorch) related to each other by computation of less than 1 week by an enthusiast with 4 or fewer A100 GPUs. A distilled model is the same LLM as the original. A finetuned or RLHF'd model may be the same LLM as the original. If the entire model is trainable in less than 1 week by an enthusiast even having no weight-relation to the original, then the concept of distinct models becomes meaningless but this market will consider them to be the same model and thus a replication. An LLM is considered created at the moment it finishes training.

• "LLM-based product" means a product such as chat assistant, article summarizer, spelling and grammar checker, translator, or more that is based on a general-purpose language model architecture such as GPT or otherwise. It doesn't need to be equally as capable as a state-of-the-art model, but must be at least as general. (e.g. GPT-3 may write bad code, but it knows that it should write code, and the code it writes has some correlation with the input) An product that only does a limited set of tasks would not be evidence of an LLM existing for this market. An LLM counts as an LLM-based product for this market, to simplify resolution criteria.

• "Replicable" means that an academic research paper is published with sufficient detail that someone is able to write public code implementing the algorithm. It is not necessary to replicate the original full-size model, but it is expected that smaller models are trained with similar performance on benchmarks as much as possible. The intention is that someone with an unlimited budget could replicate the work. If the paper comes with an NDA, it is disqualified - anyone must be able to read the paper. If the code comes with a non-free copyright license, but it was possible for the author to write free code, then it counts as evidence of replicability.

• "Open" means that model weights and code are released. Code is optional for Facebook to release, though someone must write code able to load the weights and run the model in the timeframe. Model weights must be released under a license allowing research use. If they are non-commercial or even NDA'd, it is allowed as long as any research university is able to obtain a copy of the weights. If the weights are leaked(e.g. by torrent), then this is evidence sufficient for verification of openness as long as the license would allow for research use, even if the verifier hasn't obtained a license.