0x20b8A

Sharing

Things I've read recently at the intersection of binary analysis and AI, and why I think they matter. Added to when something is actually worth adding.

0x2100A

.note

measurement

We can't yet measure what we claim to measure

arXiv:2607.07738 · Jul 2026 benchmark

REFORGE: Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming

Koller and Schmidt argue the real obstacle to evaluating LLMs on binaries isn't model capability — it's whether you can reliably align a compiled function back to its source under optimization. Their pipeline tracks that provenance from C through compilation and DWARF, and the numbers are uncomfortable: the share of functions that can be evaluated with high confidence drops from 87.2% to 65.9% as optimization levels rise.

The consequence is worse than a lower score. If harder functions silently drop out of your evaluable set, unpaired comparisons make models look more robust to optimization than they are. That's survivorship bias sitting inside a benchmark, and it's the kind of bug you can't see from the leaderboard.

AIWare 2026 dataset

REBench: A Procedural, Fair-by-Construction Benchmark for LLMs on Stripped-Binary Types and Names

The complementary move: instead of auditing existing evaluations, build a standard one. REBench consolidates datasets that RE papers had each been assembling independently, spanning x86, x64, ARM 32-bit, and MIPS 32-bit at optimization levels O0 through O3, normalized through a single decompiler.

Worth knowing about even if you don't use it. A field where every paper brings its own dataset can't compare results, and "we beat the baseline" means very little when the baseline was measured somewhere else.

0x2180A

.note

human factors

The analyst is still the interesting variable

NDSS 2026 · Best Paper user study

Decompiling the Synergy: An Empirical Study of Human-LLM Teaming in Software Reverse Engineering

A team from EURECOM, Arizona State, and Padua ran what I believe is the first controlled study of AI assistance in reverse engineering: a survey of 153 practitioners to establish how the community actually uses LLMs, then a hands-on experiment on stripped CTF-style binaries. It won Best Paper at NDSS 2026.

The framing I keep coming back to is that the useful skill isn't avoiding the model or trusting it — it's being able to interrogate its output and notice when it's confidently wrong. That's a claim about training reverse engineers, not about training models, and almost nobody in this space is designing for it.

0x2200A

.note

threat landscape

Two readings of the same year

Check Point Research · Jul 2026 industry report

Annual AI Security Report 2026

Check Point's thesis is that AI has crossed from assistant to operator — it now does hands-on work inside live intrusions rather than helping prepare them. The report's opening case study walks through one operator breaching nine Mexican government agencies with coding agents, and separately documents an 88,000-line command-and-control framework built in under a week.

For someone doing binary analysis the detail that matters is that the AI involvement is often invisible in the finished artifact. If the malware you're staring at was generated rather than written, provenance signals we've relied on for attribution get considerably weaker.

Google Threat Intelligence Group · May 2026 threat intel

Adversaries Leverage AI for Vulnerability Exploitation, Augmented Operations, and Initial Access

Read this next to the report above, because it pulls in the other direction on one specific point: GTIG observed that through early 2026 threat actors had not achieved breakthrough capabilities against the core safety logic of frontier models. What they did instead was fall back on ordinary supply chain tactics.

I find the pairing more informative than either report alone. Capability is scaling on the operational side — automation, volume, speed — while the direct model-subversion story is less dramatic than the discourse suggests. Those are different problems and they want different defenses.