Posted to tcl by juliannoble2 at Sun Jul 12 18:09:06 GMT 2026view raw

  1. ## Summary
  2.  
  3. When the console *server* (conhost.exe) serving a Tcl 9 process dies without the client
  4. being terminated, a script with a `chan event stdin readable` handler is never notified:
  5. the driver detects the error internally, but the event it queues for the interp thread is
  6. discarded before `Tcl_NotifyChannel`, so no readable event, no EOF, no read error ever
  7. reaches script level. Meanwhile the driver's console reader thread busy-loops on the
  8. persistent error with no wait, and its continuous watcher-nudging keeps the interp
  9. thread's event loop cycling at zero block time. The process cannot even implement its own
  10. defence via fileevents - only a poll (see Workaround) can see the condition.
  11.  
  12. ## Real-world triggers (legitimate application scenarios)
  13.  
  14. The reproduction below force-kills a conhost to make the trigger deterministic, but the
  15. same client-side state - console server gone, client alive - arises in ordinary use.
  16. Exposed application profile: any long-running Tcl program that reads console stdin via a
  17. fileevent - a REPL, a monitor accepting operator commands, a server with console control
  18. input - since the trigger is entirely outside the application's control:
  19.  
  20. - **A user kills a hung console window via Task Manager.** Killing "Console Window Host"
  21. (conhost.exe) is common folk-advice for a stuck window; TerminateProcess delivers no
  22. CTRL_CLOSE_EVENT, so the client survives against a dead console. (Identical mechanism
  23. to the scripted repro - verified.)
  24. - **conhost.exe crashes.** The host is an ordinary user-mode process; when it dies of its
  25. own bug the client is orphaned exactly as above.
  26. - **A terminal emulator crashes.** A clean close runs ConPTY/tab teardown that terminates
  27. the client tree, but a *crashed* terminal (Windows Terminal or third-party) never runs
  28. that cleanup; its console server goes away ungracefully. (Plausible-by-mechanism;
  29. untested here - the classic-conhost case is the verified repro.)
  30. - **Automation/CI harnesses that kill their console host.** A harness that launches work
  31. under a hidden console (Start-Process -WindowStyle Hidden, scheduled tasks, agent
  32. runners) and later "cleans up" by killing the launcher or host process - without a job
  33. object that covers the clients - leaves the worker tclsh orphaned. This is how the bug
  34. was first hit in production use: an interactive shell under a hidden console outlived
  35. its host and was found 37 minutes later with ~1550 cpu-seconds accumulated, discovered
  36. only because it polluted a test harness's process-liveness checks.
  37. - **Session teardown edge cases.** Any path where the session infrastructure tears down
  38. the console host before (or without) terminating its clients produces the same state.
  39.  
  40. In every case the failure compounds: each event adds one immortal spinner, and the
  41. Observability section explains why nobody notices them.
  42.  
  43. ## Steps to reproduce
  44.  
  45. Two files; the driver is one command. The payload writes a 1-second heartbeat so liveness
  46. of the process and its event loop is provable independently of CPU sampling.
  47.  
  48. repro.tcl:
  49.  
  50. ```tcl
  51. #stdin fileevent + 1s heartbeat. Optional argv 0: heartbeat file path.
  52. if {[llength $argv]} {
  53. set hbf [lindex $argv 0]
  54. } else {
  55. set hbf [file join [file dirname [info script]] heartbeat.txt]
  56. }
  57. chan configure stdin -blocking 0
  58. chan event stdin readable {
  59. set data [read stdin]
  60. puts stderr "readable fired: [string length $data] bytes eof=[chan eof stdin]"
  61. flush stderr
  62. if {[chan eof stdin]} {
  63. puts stderr "eof - exiting"
  64. exit 0
  65. }
  66. }
  67. proc heartbeat {} {
  68. set f [open $::hbf a]
  69. puts $f "[clock format [clock seconds] -format %H:%M:%S] alive - pid [pid]"
  70. close $f
  71. after 1000 heartbeat
  72. }
  73. heartbeat
  74. puts stderr "armed - pid [pid]"
  75. flush stderr
  76. vwait ::forever
  77. ```
  78.  
  79. run_repro.ps1 (PowerShell; run from the directory containing repro.tcl):
  80.  
  81. ```powershell
  82. param([string]$Tclsh = 'tclsh90.exe') # a Tcl 9.0.x tclsh
  83. $dir = Join-Path $env:TEMP ("deadconsole_" + (Get-Date -Format HHmmss_fff))
  84. New-Item -ItemType Directory $dir | Out-Null
  85. $err = Join-Path $dir err.txt
  86. $hb = Join-Path $dir heartbeat.txt
  87. # 1. launch the payload under its OWN dedicated CLASSIC conhost (bypasses Windows Terminal)
  88. $con = Start-Process -WindowStyle Hidden -PassThru conhost.exe -ArgumentList `
  89. "$env:SystemRoot\System32\cmd.exe /c `"$Tclsh $PWD\repro.tcl $hb 2> $err`""
  90. # 2. wait for the payload to arm and report its pid
  91. $tclpid = $null
  92. foreach ($i in 1..20) {
  93. Start-Sleep -Milliseconds 500
  94. if ((Test-Path $err) -and ((Get-Content $err -Raw) -match 'armed - pid (\d+)')) {
  95. $tclpid = [int]$Matches[1]; break
  96. }
  97. }
  98. if (-not $tclpid) { throw "payload never armed (see $err)" }
  99. "conhost $($con.Id), tclsh $tclpid armed; baselining..."
  100. $p = Get-Process -Id $tclpid; $c0 = $p.CPU; Start-Sleep 3; $p.Refresh()
  101. "idle baseline: $($p.CPU - $c0) cpu-sec over 3s (expect ~0)"
  102. # 3. force-kill the console server (TerminateProcess - no CTRL_CLOSE_EVENT is generated)
  103. Stop-Process -Force -Id $con.Id
  104. Start-Sleep 2
  105. $p = Get-Process -Id $tclpid -ErrorAction SilentlyContinue
  106. if (-not $p) { "tclsh terminated - not reproduced"; Get-Content $err; exit 1 }
  107. $c1 = $p.CPU; Start-Sleep 10; $p.Refresh()
  108. "SURVIVED. cpu rate: $(($p.CPU - $c1) / 10) cores over 10s"
  109. $p.Threads | Sort-Object {$_.TotalProcessorTime} -Descending |
  110. Select-Object -First 3 Id, ThreadState, WaitReason, TotalProcessorTime
  111. "stderr trail (no 'readable fired'/'eof' will have appeared):"; Get-Content $err
  112. "heartbeat tail (still ticking - event loop alive, timers fire, only the fileevent is lost):"
  113. Get-Content $hb -Tail 3
  114. "kill the specimen when done: Stop-Process -Force -Id $tclpid"
  115. ```
  116.  
  117. ### Important: kill methods that do NOT reproduce (triage note)
  118.  
  119. The bug needs the console server to die *without* anything terminating the client. All of
  120. the following are healthy paths in which the client is deliberately terminated, so they
  121. will NOT show the bug:
  122.  
  123. - Closing a console/terminal window manually: clients receive CTRL_CLOSE_EVENT and are
  124. force-terminated after the grace period.
  125. - Closing or killing a Windows Terminal tab (or its OpenConsole/ConPTY host): the terminal
  126. tears down the client tree on pseudoconsole close.
  127. - Running the repro from an interactive shell inside Windows Terminal and killing "a
  128. conhost": the tclsh shares the tab's ConPTY; it has no dedicated classic conhost, and
  129. killing the tab's host lands in the previous bullet.
  130.  
  131. The driver above forces the reproducing topology: a dedicated classic conhost.exe as the
  132. console server, force-killed by pid.
  133.  
  134. ## Observed
  135.  
  136. - The tclsh survives and consumes ~1.85 cores indefinitely when it is the only orphan
  137. (one thread permanently Running - the console reader thread, hammering SetEvent - plus
  138. the interp thread woken continuously).
  139. - err.txt shows only "armed - pid NNN": the readable handler NEVER fires after the console
  140. dies. No eof, no error - script level is completely blind.
  141. - heartbeat.txt keeps ticking every second: the process and its event loop (timer events)
  142. are fully alive; only the channel notification is lost.
  143. - Orphans accumulate. Concurrent orphans serialize on the kernel wake/handoff syscalls, so
  144. per-orphan burn drops but total grows: measured 4 concurrent orphans at ~1.15 cores each
  145. (4.6 cores aggregate); 8 concurrent produced a clear thermal (fan) response.
  146.  
  147. ## Expected
  148.  
  149. The dead console is reported to the script (readable event whose read then yields EOF or
  150. an error), and no driver thread busy-waits. The process can then close the channel and
  151. exit or continue headless, idling at ~0 CPU.
  152.  
  153. ## Observability (why this ships unnoticed)
  154.  
  155. On a 32-logical-processor workstation a single orphan is ~3-6% total CPU with no pegged
  156. core anywhere: the scheduler migrates the spinning threads across LPs, so per-LP graphs
  157. show only a diffuse ripple, and per-process CPU displays that are frequency-normalized
  158. (Windows 11 "Processor Utility") under-report a load that is almost entirely kernel
  159. wait/wake handoff and never drives clocks up - the affected processes were observed
  160. showing 0 in Task Manager's Details CPU column while Get-Process/TotalProcessorTime showed
  161. them burning more than a core each. The bug was originally found only because an orphan
  162. polluted process-liveness checks in a test harness, not because anyone saw CPU load.
  163.  
  164. ## Analysis (win/tclWinConsole.c, 9.1b1 sources; same code in 9.0.x)
  165.  
  166. When the console host dies, ReadConsoleW fails (observed GetLastError: ERROR_BROKEN_PIPE)
  167. and the reader thread records it in handleInfoPtr->lastError. Two defects then interact:
  168.  
  169. **Defect 1 - the error notification is produced but never delivered.**
  170. ConsoleSetupProc and ConsoleCheckProc both treat `lastError != ERROR_SUCCESS` as "input
  171. ready" (block time 0 / queue a ConsoleEvent). But ConsoleEventProc computes the notify
  172. mask ONLY from ring-buffer state:
  173.  
  174. ```c
  175. if ((chanInfoPtr->watchMask & TCL_READABLE)
  176. && RingBufferLength(&handleInfoPtr->buffer)) {
  177. mask = TCL_READABLE;
  178. }
  179. ```
  180.  
  181. With the buffer empty and only lastError set, mask stays 0 and Tcl_NotifyChannel is never
  182. called. The queued event is discarded - and CheckProc immediately queues another. The
  183. interp thread therefore spins through Tcl_DoOneEvent at zero block time, queueing and
  184. discarding console events forever, and the registered fileevent script never runs.
  185. (ConsoleInputProc would correctly surface the error/EOF if a read ever happened - it
  186. never does, because the script is never notified.)
  187.  
  188. **Defect 2 - the reader thread's persistent-error path has no wait.**
  189. In ConsoleReaderThread's main loop, the "notify clients" branch runs when
  190. `inputLen > 0 || handleInfoPtr->lastError != 0`. For the data case the buffer drains and
  191. the thread subsequently blocks in SleepConditionVariableSRW. For the persistent-error case
  192. nothing resets lastError (ConsoleInputProc clears it only on a read, and for
  193. ERROR_INVALID_HANDLE not even then), so every iteration takes the branch:
  194. WakeAllConditionVariable + NudgeWatchers + `continue` - no wait anywhere on the path. The
  195. thread hard-loops calling SetEvent (via Tcl_ThreadAlert in NudgeWatchers) until the channel
  196. is closed (numRefs == 1) - which never happens, because of Defect 1. One full core, plus
  197. the wakeups feeding the Defect-1 spin on the interp thread.
  198.  
  199. Related contributing quirk: ConsoleDataAvailable returns -1 on PeekConsoleInputW failure
  200. and its caller treats the result as boolean, so a dead console reads as "data available",
  201. which is what triggers the failing ReadConsoleChars and sets lastError in the first place.
  202. That happens to be load-bearing for eventually noticing the error at all, but the -1/true
  203. conflation looks unintended.
  204.  
  205. ## Suggested fix
  206.  
  207. 1. ConsoleEventProc: also set `mask = TCL_READABLE` when
  208. `(chanInfoPtr->watchMask & TCL_READABLE) && handleInfoPtr->lastError != ERROR_SUCCESS`
  209. (matching the condition SetupProc/CheckProc already use). The script then gets the
  210. readable event, the read surfaces EOF/error, and a well-behaved script closes the
  211. channel, which lets the reader thread exit.
  212. 2. ConsoleReaderThread: after notifying watchers of an error with an empty private buffer,
  213. wait on the condition variable (as the no-data path does) instead of continuing the
  214. loop - the error is persistent; re-announcing it in a hot loop serves nothing. This
  215. protects processes whose scripts never read/close (e.g. no fileevent registered) from
  216. the one-core spin as well.
  217.  
  218. Either half alone materially improves matters; both together make the failure mode clean:
  219. console dies -> readable -> eof on read -> script closes/exits, no busy-wait meanwhile.
  220.  
  221. ## Workaround (script level, until fixed)
  222.  
  223. Fileevents cannot see the condition, so poll: `chan configure stdin -inputmode` performs a
  224. live GetConsoleMode and errors iff the console is gone; on failure close the console
  225. channel(s) - closing is what stops the reader-thread spin - and exit or continue headless.
  226. A 5-second poll made orphans of the reporting application exit within seconds of console
  227. death in testing, with no observable effect on live-console interactive behaviour.
  228.  

Add a comment

Please note that this site uses the meta tags nofollow,noindex for all pages that contain comments.
Items are closed for new comments after 1 week