web browser (Servo)
git clone https://git.lucas.co/cce-browser.git
fix: dodge the iris aux-map deadlock and recover from one unasked
The 2026-10-05 hang was a Mesa iris lock-order inversion: WebKit's two
Skia GPU painting threads took the aux-map mutex and bufmgr->lock in
opposite orders. Still present on Mesa main.
- main() sets INTEL_DEBUG=noccs before anything starts. Without CCS the
aux map is never used, so neither half of the cycle runs. Page
processes inherit it through bubblewrap; no measurable cost scrolling
an image-heavy page in a scale-2 shadow. CCE_BROWSER_CCS=1 opts out.
- A hung active tab whose page processes all sit idle for 5s is taken
for a deadlock: the process is stopped and the page loaded again. A
busy process (a spinning script) is left to the prompt, as is a hang
the person chose to wait on, and a tab recovered in the last 2 min.
- A page blocked in alert() or an auth challenge no longer counts as
hung through the ping sent with the click that opened it.
examples/wpe_crash.rs covers all three (SIGSTOP stands in for a
deadlock: unanswering and idle).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CLAUDE.md | 22 ++++-
examples/wpe_crash.rs | 75 ++++++++++++++++-
src/main.rs | 28 +++++++
src/wpe/host.rs | 220 ++++++++++++++++++++++++++++++++++++++++++++------
4 files changed, 317 insertions(+), 28 deletions(-)
diff --git a/CLAUDE.md b/CLAUDE.md
index b22b37a..0e2f3ce 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -259,7 +259,27 @@ against the real engine. Points that are choices:
Every page press therefore also sends a no-op script in a private world
(`ping`), and a ping unanswered for `HANG_GRACE` (3s) is a hang. A one-shot
GLib timeout wakes the loop when the grace runs out, since a hung page
- sends nothing that would.
+ sends nothing that would. A pending `alert()` or auth challenge is never a
+ hang — the page is blocked on *us* — and answering one forgets the ping
+ that was waiting behind it.
+- **A deadlock is recovered without asking** (`watch_deadlock`). A hung page
+ and a spinning script look alike from here; the difference is CPU: the
+ 2026-10-05 deadlock sat at zero, a script burns a core. WebKit says nothing
+ about which process serves which tab, so every WPEWebProcess under the
+ browser is sampled from `/proc`, and only if *all* of them stay under 5% of
+ a core for `DEADLOCK_WATCH` (5s) is the process stopped and the page loaded
+ again — a plain load, so a form post is not resubmitted. Busy, waited on
+ ("Wait"), or already recovered within `AUTO_RECOVER_GAP` (2 min) is left to
+ the prompt, so a page that deadlocks on every load cannot reload-loop.
+- **The deadlock itself is a Mesa iris bug, still on Mesa `main`**: a lock-order
+ inversion between the aux-map mutex and `bufmgr->lock`, hit by WebKit's two
+ Skia GPU painting threads allocating textures at once (one adds an aux
+ mapping and needs a new table page; the other reuses a cached BO and unmaps
+ its old aux range). `main()` therefore sets `INTEL_DEBUG=noccs` before
+ anything starts (`disable_intel_ccs`): without CCS the aux map is never
+ used. Page processes inherit it through bubblewrap. Measured in a scale-2
+ shadow scrolling 24 large images: no difference in frame rate or CPU.
+ `CCE_BROWSER_CCS=1` turns compression back on.
## The chrome is hand-rolled
diff --git a/examples/wpe_crash.rs b/examples/wpe_crash.rs
index 3f67d53..d6c5e12 100644
--- a/examples/wpe_crash.rs
+++ b/examples/wpe_crash.rs
@@ -10,6 +10,10 @@
//! * the dead tab shows the error page **under its own URL**, so the URL
//! bar and the saved session still mean the real page;
//! * a reload from there fetches the real page again;
+//! * a page that spins is never stopped without asking, but a process that
+//! is hung *and idle* — what a deadlock looks like, simulated with SIGSTOP —
+//! is stopped and reloaded on its own;
+//! * a page waiting in `alert()` is not taken for hung;
//! * a WebProcess that dies outright (SIGKILL) gets the crash page.
//!
//! `cargo run --release -p cce-browser --example wpe_crash`
@@ -39,6 +43,10 @@ const HANG: &str = "<!doctype html><title>hang</title><p>spinning\
<script>setTimeout(() => { for (;;) {} }, 300)</script>";
#[cfg(feature = "wpe")]
const FINE: &str = "<!doctype html><title>recovered</title><p>fine";
+/// Opens an alert on the first click.
+#[cfg(feature = "wpe")]
+const ALERT: &str = "<!doctype html><title>alert</title><p>click me\
+<script>addEventListener('mousedown', () => alert('hi'), {once: true})</script>";
/// The first request gets the hanging page, every later one the fine one.
#[cfg(feature = "wpe")]
@@ -52,12 +60,16 @@ fn serve() -> u16 {
let Ok(mut s) = stream else { continue };
let mut buf = [0u8; 4096];
let n = s.read(&mut buf).unwrap_or(0);
- if !String::from_utf8_lossy(&buf[..n]).starts_with("GET /page") {
+ let req = String::from_utf8_lossy(&buf[..n]).to_string();
+ let body = if req.starts_with("GET /alert") {
+ ALERT
+ } else if req.starts_with("GET /page") {
+ served += 1;
+ if served == 1 { HANG } else { FINE }
+ } else {
let _ = write!(s, "HTTP/1.1 404 Not Found\r\nContent-Length: 0\r\nConnection: close\r\n\r\n");
continue;
- }
- let body = if served == 0 { HANG } else { FINE };
- served += 1;
+ };
let _ = write!(
s,
"HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{}",
@@ -135,6 +147,15 @@ fn main() {
settle(&mut host, 4000);
check("unanswered input reports a hang", host.active_unresponsive(), String::new());
+ // Well past the deadlock watch: a spinning page burns a core, so it is
+ // left to the person to stop.
+ settle(&mut host, 7000);
+ check(
+ "a busy page is never stopped unasked",
+ host.active_unresponsive() && host.title().as_deref() == Some("hang"),
+ format!("{:?}", host.title()),
+ );
+
host.wait_unresponsive();
check("waiting quiets the question", !host.active_unresponsive(), String::new());
@@ -157,6 +178,52 @@ fn main() {
settle(&mut host, 4000);
check("a live page is not reported", !host.active_unresponsive(), String::new());
+ // A deadlock, simulated: every page process frozen, so nothing answers
+ // and nothing burns CPU.
+ let frozen = web_processes();
+ for &pid in &frozen {
+ unsafe { libc_kill(pid, 19) }; // SIGSTOP
+ }
+ host.mouse_move(130.0, 130.0);
+ host.mouse_button_ui(MouseButton::Left, true, 130.0, 130.0);
+ host.mouse_button_ui(MouseButton::Left, false, 130.0, 130.0);
+ settle(&mut host, 4000);
+ check("an idle hang is reported first", host.active_unresponsive(), String::new());
+ settle(&mut host, 6000);
+ let title = host.title().unwrap_or_default();
+ check(
+ "then stopped and reloaded on its own",
+ title == "recovered" && !host.active_unresponsive(),
+ format!("{title:?}"),
+ );
+ check("under the same URL", host.url().as_ref() == Some(&page), String::new());
+ // Whatever was frozen and not stopped (the spare) goes back to work.
+ for &pid in &frozen {
+ unsafe { libc_kill(pid, 18) }; // SIGCONT
+ }
+ settle(&mut host, 500);
+
+ // A page blocked in alert() is waiting on the chrome, not hung — even
+ // though the ping sent with the click that opened it goes unanswered.
+ let alert = url::Url::parse(&format!("http://127.0.0.1:{port}/alert")).unwrap();
+ host.load(alert);
+ settle(&mut host, 1500);
+ host.mouse_move(140.0, 140.0);
+ host.mouse_button_ui(MouseButton::Left, true, 140.0, 140.0);
+ host.mouse_button_ui(MouseButton::Left, false, 140.0, 140.0);
+ settle(&mut host, 4500);
+ check(
+ "a page in alert() is not hung",
+ host.pending_dialog().is_some() && !host.active_unresponsive(),
+ format!("dialog={:?}", host.pending_dialog().map(|d| d.message)),
+ );
+ host.respond_dialog(true, None);
+ check("nor right after it is answered", !host.active_unresponsive(), String::new());
+ settle(&mut host, 1000);
+ check("and it is still the same page", host.title().as_deref() == Some("alert"), format!("{:?}", host.title()));
+ host.load(page.clone());
+ settle(&mut host, 1500);
+
let victims = web_processes();
// SIGKILL rather than SIGSEGV: JavaScriptCore installs its own SEGV
// handler, and a sent one is not a fault it will die of.
diff --git a/src/main.rs b/src/main.rs
index 463fb68..a3490bb 100644
--- a/src/main.rs
+++ b/src/main.rs
@@ -4037,7 +4037,35 @@ impl Application for BrowserApp {
}
}
+/// Turn off Intel's CCS compression for this process and everything it
+/// spawns.
+///
+/// Mesa's iris driver can deadlock two threads against each other through
+/// the aux map CCS needs: one adds a mapping and wants the buffer manager's
+/// lock, the other reuses a cached buffer under that lock and wants the aux
+/// map's. WebKit's two GPU painting threads hit exactly that and froze a
+/// page for good (2026-10-05; CLAUDE.md, "A dead or hung page"). Without CCS
+/// the aux-map code never runs. The cost is uncompressed surfaces — more
+/// memory bandwidth on the iGPU.
+///
+/// Set first, before anything has started a thread or opened a GPU device:
+/// WebKit's page processes inherit it through their sandbox, and the chrome's
+/// own Vulkan device reads it too. `CCE_BROWSER_CCS=1` keeps compression, for
+/// measuring what this costs.
+fn disable_intel_ccs() {
+ if std::env::var_os("CCE_BROWSER_CCS").is_some_and(|v| v == "1") {
+ return;
+ }
+ let current = std::env::var("INTEL_DEBUG").unwrap_or_default();
+ if current.split(',').any(|f| f.trim() == "noccs") {
+ return;
+ }
+ let value = if current.is_empty() { "noccs".to_string() } else { format!("{current},noccs") };
+ std::env::set_var("INTEL_DEBUG", value);
+}
+
fn main() {
+ disable_intel_ccs();
env_logger::init();
// A read-only look at what a Raindrop sync would do (RAINDROP-SYNC.md).
// Ahead of the instance hand-off: it is a tool, not a launch, and must
diff --git a/src/wpe/host.rs b/src/wpe/host.rs
index 576d336..72e9b56 100644
--- a/src/wpe/host.rs
+++ b/src/wpe/host.rs
@@ -55,6 +55,76 @@ struct TabState {
/// When the outstanding [`WebKitHost::ping`] went out, if one has not
/// been answered yet.
ping_since: Cell<Option<std::time::Instant>>,
+ /// The process is being stopped as a deadlock, so its termination should
+ /// reload the page rather than show the error page.
+ auto_reload: Cell<bool>,
+ /// When this tab was last recovered that way.
+ last_auto: Cell<Option<std::time::Instant>>,
+}
+
+/// How long every page process must sit idle while the active tab is
+/// unresponsive before the hang is taken for a deadlock and recovered
+/// without asking.
+const DEADLOCK_WATCH: std::time::Duration = std::time::Duration::from_secs(5);
+
+/// A tab is recovered automatically at most this often. A page that
+/// deadlocks on every load is left to the prompt instead of reloading in a
+/// loop.
+const AUTO_RECOVER_GAP: std::time::Duration = std::time::Duration::from_secs(120);
+
+/// A deadlock in progress: when the watch began, on which tab, and every
+/// page process's CPU time at that moment.
+struct DeadlockWatch {
+ tab: Rc<TabState>,
+ since: std::time::Instant,
+ ticks: std::collections::HashMap<i32, u64>,
+}
+
+/// CPU time (user + system, in `/proc` clock ticks) of every WPEWebProcess
+/// below this one — they sit under bubblewrap, so not as direct children.
+///
+/// WebKit has no API that says which process serves which tab, so the
+/// watch reads all of them. That is what makes it conservative: one busy
+/// process anywhere is enough to leave the hang to the prompt.
+fn web_process_ticks() -> std::collections::HashMap<i32, u64> {
+ let me = std::process::id() as i32;
+ let stat = |pid: i32| std::fs::read_to_string(format!("/proc/{pid}/stat")).ok();
+ // The fields after the parenthesized command name, which may hold spaces.
+ let fields = |s: &str| -> Vec<String> {
+ s.rsplit_once(')').map_or_else(Vec::new, |(_, rest)| {
+ rest.split_whitespace().map(str::to_string).collect()
+ })
+ };
+ let mut out = std::collections::HashMap::new();
+ for entry in std::fs::read_dir("/proc").into_iter().flatten().flatten() {
+ let Ok(pid) = entry.file_name().to_string_lossy().parse::<i32>() else { continue };
+ let comm = std::fs::read_to_string(format!("/proc/{pid}/comm")).unwrap_or_default();
+ if comm.trim() != "WPEWebProcess" {
+ continue;
+ }
+ let Some(s) = stat(pid) else { continue };
+ let f = fields(&s);
+ let mut parent = f.get(1).and_then(|p| p.parse::<i32>().ok());
+ let mut ours = false;
+ for _ in 0..4 {
+ match parent {
+ Some(p) if p == me => {
+ ours = true;
+ break;
+ }
+ Some(p) if p > 1 => {
+ parent = stat(p).and_then(|s| fields(&s).get(1).and_then(|p| p.parse().ok()));
+ }
+ _ => break,
+ }
+ }
+ // utime and stime are fields 14 and 15; `f` starts at field 3.
+ let ticks = |i: usize| f.get(i).and_then(|t| t.parse::<u64>().ok()).unwrap_or(0);
+ if ours {
+ out.insert(pid, ticks(11) + ticks(12));
+ }
+ }
+ out
}
/// How long a page may leave a ping unanswered before it counts as hung —
@@ -302,6 +372,8 @@ pub struct WebKitHost {
/// press-drag-release over text selected nothing (and dragged nothing)
/// while every move went out with an empty mask.
held_buttons: Cell<WPEModifiers::Type>,
+ /// A hang being watched to see whether it is a deadlock.
+ deadlock_watch: Option<DeadlockWatch>,
}
unsafe fn cstr(s: &str) -> CString {
@@ -457,6 +529,7 @@ impl WebKitHost {
spare: None,
window_focused: false,
held_buttons: Cell::new(0),
+ deadlock_watch: None,
};
// The account watcher's channel, in its own script world. Both
// halves are registered here, once, on the shared content
@@ -864,6 +937,7 @@ impl WebKitHost {
if let Some(p) = &mut self.poll {
p.sync();
}
+ self.watch_deadlock();
// Nothing has drawn the last frame yet, so reading another would be
// copying over a picture that was never shown. Leave the buffer held:
// the engine's next frame supersedes it and hands it back unread.
@@ -982,26 +1056,38 @@ impl WebKitHost {
continue;
}
if let Some(reason) = tab.state.terminated.take() {
- // Loading anything respawns a WebProcess; this loads the
- // error page under the dead page's URL. Reload — the chrome's
- // or the page's link — then fetches the real one.
- log::warn!(
- "web process for {} terminated (reason {reason})",
- tab.url.as_ref().map_or("<no url>", |u| u.as_str())
- );
- unsafe {
- let html = cstr(&terminated_page(tab.url.as_ref(), reason));
- let uri = tab.url.as_ref().map(|u| cstr(u.as_str()));
- webkit_web_view_load_alternate_html(
- tab.webview,
- html.as_ptr(),
- uri.as_ref().map_or(std::ptr::null(), |u| u.as_ptr()),
- // The page's own URL as the base too: without one the
- // error page is `about:blank` to itself, so its
- // Reload link — refused outright when the dead page
- // was a `file:` — had nowhere real to go.
- uri.as_ref().map_or(std::ptr::null(), |u| u.as_ptr()),
+ // Stopped as a deadlock: just load the page again. A plain
+ // load, not a reload, so a page that came from a form post
+ // is not posted twice.
+ let reload = tab.state.auto_reload.take().then_some(tab.url.as_ref()).flatten();
+ if let Some(u) = reload {
+ log::warn!("reloading {u} after stopping its deadlocked web process");
+ unsafe {
+ let c = cstr(u.as_str());
+ webkit_web_view_load_uri(tab.webview, c.as_ptr());
+ }
+ } else {
+ // Loading anything respawns a WebProcess; this loads the
+ // error page under the dead page's URL. Reload — the chrome's
+ // or the page's link — then fetches the real one.
+ log::warn!(
+ "web process for {} terminated (reason {reason})",
+ tab.url.as_ref().map_or("<no url>", |u| u.as_str())
);
+ unsafe {
+ let html = cstr(&terminated_page(tab.url.as_ref(), reason));
+ let uri = tab.url.as_ref().map(|u| cstr(u.as_str()));
+ webkit_web_view_load_alternate_html(
+ tab.webview,
+ html.as_ptr(),
+ uri.as_ref().map_or(std::ptr::null(), |u| u.as_ptr()),
+ // The page's own URL as the base too: without one the
+ // error page is `about:blank` to itself, so its
+ // Reload link — refused outright when the dead page
+ // was a `file:` — had nowhere real to go.
+ uri.as_ref().map_or(std::ptr::null(), |u| u.as_ptr()),
+ );
+ }
}
}
tab.title = tab.state.title.borrow().clone();
@@ -1036,10 +1122,96 @@ impl WebKitHost {
/// The active tab's WebProcess has stopped answering, and the person has
/// not already chosen to wait on it.
pub fn active_unresponsive(&self) -> bool {
- self.tabs.get(self.active).is_some_and(|t| {
- let ping_overdue = t.state.ping_since.get().is_some_and(|at| at.elapsed() >= HANG_GRACE);
- (t.state.unresponsive.get() || ping_overdue) && !t.state.hang_waived.get()
- })
+ self.tabs.get(self.active).is_some_and(|t| self.hung(t) && !t.state.hang_waived.get())
+ }
+
+ /// The tab's process has stopped answering — by WebKit's timer or an
+ /// overdue ping. Never while a page dialog or auth challenge is up: the
+ /// process is blocked on *us* then, and a ping sent just before it
+ /// opened goes unanswered for as long as it stays open.
+ fn hung(&self, t: &Tab) -> bool {
+ let p = self.prompts.borrow();
+ if p.dialog.is_some() || p.auth.is_some() {
+ return false;
+ }
+ let ping_overdue = t.state.ping_since.get().is_some_and(|at| at.elapsed() >= HANG_GRACE);
+ t.state.unresponsive.get() || ping_overdue
+ }
+
+ /// Recover a deadlocked page without asking.
+ ///
+ /// A deadlock and a busy page look alike from here — both stop
+ /// answering — but a deadlocked process burns no CPU (the 2026-10-05
+ /// one sat at zero, every thread parked on a lock) and a spinning script
+ /// burns a whole core. So while the active tab is hung, every page
+ /// process's CPU time is sampled; if none of them has used more than a
+ /// sliver of it across `DEADLOCK_WATCH`, the process is stopped and the
+ /// page loaded again. Anything busier is left to the prompt, as is a
+ /// hang the person chose to wait on, and a tab recovered within
+ /// `AUTO_RECOVER_GAP`.
+ fn watch_deadlock(&mut self) {
+ let state = self
+ .tabs
+ .get(self.active)
+ .filter(|t| self.hung(t) && !t.state.hang_waived.get())
+ .map(|t| t.state.clone());
+ let Some(state) = state else {
+ self.deadlock_watch = None;
+ return;
+ };
+ if state.last_auto.get().is_some_and(|at| at.elapsed() < AUTO_RECOVER_GAP) {
+ return;
+ }
+ let watching = self.deadlock_watch.as_ref().filter(|w| Rc::ptr_eq(&w.tab, &state));
+ let Some(watch) = watching else {
+ self.deadlock_watch = Some(DeadlockWatch {
+ tab: state,
+ since: std::time::Instant::now(),
+ ticks: web_process_ticks(),
+ });
+ return;
+ };
+ let elapsed = watch.since.elapsed();
+ if elapsed < DEADLOCK_WATCH {
+ return;
+ }
+ let now = web_process_ticks();
+ // 5% of one core, at /proc's 100 ticks a second. A process that
+ // appeared mid-watch is measured from zero, which counts its whole
+ // startup against it — on the side of not stopping anything.
+ let budget = (elapsed.as_secs_f64() * 100.0 * 0.05) as u64;
+ let idle = !now.is_empty()
+ && now.iter().all(|(pid, t)| {
+ t.saturating_sub(watch.ticks.get(pid).copied().unwrap_or(0)) <= budget
+ });
+ if !idle {
+ // Busy: a script, most likely. Watch again from here, so a page
+ // that stops spinning and then deadlocks is still caught.
+ self.deadlock_watch = Some(DeadlockWatch {
+ tab: state,
+ since: std::time::Instant::now(),
+ ticks: now,
+ });
+ return;
+ }
+ self.deadlock_watch = None;
+ log::warn!(
+ "{} has not answered in {:.0}s and no page process is running; \
+ stopping it as deadlocked",
+ self.tabs[self.active].url.as_ref().map_or("<no url>", |u| u.as_str()),
+ (elapsed + HANG_GRACE).as_secs_f64(),
+ );
+ state.auto_reload.set(true);
+ state.last_auto.set(Some(std::time::Instant::now()));
+ self.stop_unresponsive();
+ }
+
+ /// A dialog or auth challenge was answered: whatever ping was waiting
+ /// behind it was waiting on the person, not on a hung page.
+ fn forget_ping(&self) {
+ if let Some(t) = self.tabs.get(self.active) {
+ t.state.ping_since.set(None);
+ }
}
/// Ask the active page's main thread for an answer, to learn whether it
@@ -1275,6 +1447,7 @@ impl WebKitHost {
let Some((dialog, pending)) = self.prompts.borrow_mut().dialog.take() else {
return;
};
+ self.forget_ping();
unsafe {
if pending.prompt_default.is_some() {
// A cancelled prompt must return null, not "" — a page
@@ -1301,6 +1474,7 @@ impl WebKitHost {
let Some((request, _)) = self.prompts.borrow_mut().auth.take() else {
return;
};
+ self.forget_ping();
unsafe {
match credentials {
Some((user, password)) => {