Source: arxiv | Overall 6.4/10 | Corroboration: 1
Signal 9.4
Novelty 4.0
Impact 2.0
Confidence 9.5
Actionability 6.5
Summary: arXiv:2606.00093v2 Announce Type: replace Abstract: Whether a rubric-based LLM judge can replace human annotation is decided by its measured agreement with human labels.
- What happened: We treat the choices as a measurement protocol that fixes what the reported number estimates before any metric is computed, assemble the relevant results into a single.
- Why it matters: Under exclusion, accuracy over all cases is pinned down only to a worst-case interval as wide as the uncovered fraction.
- What to do: Validate with one small internal benchmark and compare against your current baseline this week.
Deep
Context
Submission history From: Delip Rao [view email] [v1] Mon, 25 May 2026 07:31:44 UTC (269 KB) [v2] Fri, 31 Jul 2026 04:03:56 UTC (615 KB) Ancillary-file links: Ancillary files (details): Current browse context: cs.CL Change to browse by: References & Citation...
What's new
arXiv:2606.00093v2 Announce Type: replace Abstract: Whether a rubric-based LLM judge can replace human annotation is decided by its measured agreement with human labels.
Key details
- Yet the same verdicts can support wildly varying agreement numbers, depending on seemingly minor choices: the judgment scale, the retained cases, the handling of abstentions and invalid outputs, and the pooling of verdicts across items and rubric criteria.
- The statistics that settle these choices are established, but in psychometrics, econometrics, and corpus annotation rather than in the evaluation practice that needs them.
- We treat the choices as a measurement protocol that fixes what the reported number estimates before any metric is computed, assemble the relevant results into a single source-attributed analysis, and apply it to three published LLM-judge evaluations.
- For non-degenerate binary verdicts, Pearson's $r$, Spearman's $\rho$, Kendall's $\tau_b$, the phi coefficient, and the Matthews correlation coefficient are exactly the same statistic, so reporting several repeats one number under different names.
Results & evidence
- arXiv:2606.00093v2 Announce Type: replace Abstract: Whether a rubric-based LLM judge can replace human annotation is decided by its measured agreement with human labels.
- Cohen's $\kappa$ differs from them only through a marginal-mismatch factor in $(0,1]$ and shares their asymptotic variance when judge and human assign the positive verdict equally often.
- On a rubric benchmark carrying per-criterion human labels, protocol choice alone moves reported accuracy from $0.551$ to $0.899$ and carries $\kappa$ across zero, without altering a single verdict.
Limitations / unknowns
- Generalization outside curated tasks is still unclear.
Next-step validation checks
- Reproduce one claim with a public baseline and fixed evaluation settings.
- Check robustness on out-of-distribution or long-context cases.
- Track whether independent teams report matching results.
Source: hackernews | Overall 6.2/10 | Corroboration: 1
Signal 8.6
Novelty 4.0
Impact 4.8
Confidence 7.5
Actionability 3.5
Summary: Quickstart | Configurations | MacOS | Example notebooks | FAQ AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card โ.
- What happened: You can even run 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on ~12GB, and Kimi K3 (2.8T) โ the largest open-source model released to date โ on under 4GB, because sparse.
- Why it matters: Quickstart | Configurations | MacOS | Example notebooks | FAQ AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB.
- What to do: Track for corroboration and benchmark data before adopting.
Deep
Context
Quickstart | Configurations | MacOS | Example notebooks | FAQ AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card โ without quantization, distillation, or pruning.
What's new
Quickstart | Configurations | MacOS | Example notebooks | FAQ AirLLM dramatically reduces inference memory usage, letting 70B large language models run on a single 4GB GPU card โ without quantization, distillation, or pruning.
Key details
- You can even run 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on ~12GB, and Kimi K3 (2.8T) โ the largest open-source model released to date โ on under 4GB, because sparse MoE models stream one expert at a time rather than a whole layer.
- - Best AI Game Sprite Generator - Best AI Facial Expression Editor - Bloome โ build & run AI agent teams in the cloud, zero setup [2026/07] Kimi K3 (2.8T) support: the largest open-source model runs on a single card in 3.72GB of VRAM, measured end to end on...
- Per-expert streaming loads only the experts a token actually routes to.
- K3 brings three requirements of its own: pip install compressed-tensors flash-attn (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and transformer...
Results & evidence
- You can even run 405B Llama 3.1 on 8GB, DeepSeek-V3 (671B) on ~12GB, and Kimi K3 (2.8T) โ the largest open-source model released to date โ on under 4GB, because sparse MoE models stream one expert at a time rather than a whole layer.
- - Best AI Game Sprite Generator - Best AI Facial Expression Editor - Bloome โ build & run AI agent teams in the cloud, zero setup [2026/07] Kimi K3 (2.8T) support: the largest open-source model runs on a single card in 3.72GB of VRAM, measured end to end on...
- K3 brings three requirements of its own: pip install compressed-tensors flash-attn (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and transformer...
Limitations / unknowns
- Generalization outside curated tasks is still unclear.
Next-step validation checks
- Reproduce one claim with a public baseline and fixed evaluation settings.
- Check robustness on out-of-distribution or long-context cases.
- Track whether independent teams report matching results.
Source: github | Overall 7.7/10 | Corroboration: 1
Signal 10.0
Novelty 5.1
Impact 7.8
Confidence 7.0
Actionability 6.5
Summary: AI agents running research on single-GPU nanochat training automatically One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other.
- What happened: AI agents running research on single-GPU nanochat training automatically One day, frontier AI research used to be done by meat computers in between eating, sleeping.
- Why it matters: It modifies the code, trains for 5 minutes, checks if the result improved, keeps or discards, and repeats.
- What to do: Validate with one small internal benchmark and compare against your current baseline this week.
Deep
Context
Instead, you are programming the program.md Markdown files that provide context to the AI agents and set up your autonomous research org.
What's new
AI agents running research on single-GPU nanochat training automatically One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other fun, and synchronizing once in a while using sound wave interconnect in the ri...
Key details
- Research is now entirely the domain of autonomous swarms of AI agents running across compute cluster megastructures in the skies.
- The agents claim that we are now in the 10,205th generation of the code base, in any case no one could tell if that's right or wrong as the "code" is now a self-modifying binary that has grown beyond human comprehension.
- This repo is the story of how it all began.
- The idea: give an AI agent a small but real LLM training setup and let it experiment autonomously overnight.
Results & evidence
- The agents claim that we are now in the 10,205th generation of the code base, in any case no one could tell if that's right or wrong as the "code" is now a self-modifying binary that has grown beyond human comprehension.
- It modifies the code, trains for 5 minutes, checks if the result improved, keeps or discards, and repeats.
Limitations / unknowns
- Generalization outside curated tasks is still unclear.
Next-step validation checks
- Reproduce one claim with a public baseline and fixed evaluation settings.
- Check robustness on out-of-distribution or long-context cases.
- Track whether independent teams report matching results.