src/genro_asgi_multiworker_spa/spa_app.py¶
Source from this local checkout, regenerated when the reader rebuilds.
Line links use #L<number>; a GitHub line range opens its first line.
1 # Copyright 2025 Softwell S.r.l.2 #3 # Licensed under the Apache License, Version 2.0 (the "License");4 # you may not use this file except in compliance with the License.5 # You may obtain a copy of the License at6 #7 # https://www.apache.org/licenses/LICENSE-2.08 #9 # Unless required by applicable law or agreed to in writing, software10 # distributed under the License is distributed on an "AS IS" BASIS,11 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.12 # See the License for the specific language governing permissions and13 # limitations under the License.14 15 """The mountable SPA front: one door, and no state at all.16 17 **The pool is born with the server, not with the object.** An application reaches18 its own configuration only once a server holds it, so the vertex is built at19 startup out of what the recipe wrote under this front's code. There is ONE way to20 build it, and the tests come through the same door as production.21 ``commander_class`` is a class attribute and not a kwarg: a Python type does not22 travel in a recipe, and whoever wants another pool subclasses this front, which is23 what the recipe already names.24 25 **A pool belongs to the application that owns it.** The chain is server →26 applications → this front → its ORCHESTRATION → its commander → its groups →27 their workers, and the recipe says it in that shape: the pool's words are this28 class's own grammar and are written under29 ``applications.<code>.orchestration.commander``. That node is REQUIRED — a spa30 front declared without it, or with it and no commander under it, does not boot:31 a front with no pool answers every request with a raise, and the recipe is asked32 for the node instead. Several fronts on one server are legitimate,33 each with its own orchestration. ``memory_max_percent`` stays a share of the34 MACHINE, so apportioning it between two pools is the installation's own business;35 an installation that over-declares is caught by the machine's alarm line, past36 which nothing grows.37 38 **Serving is a two-stage demux.**39 Stage 1 reads the FIRST segment of the (already mount-relative) path: not one of40 ``internal_roots`` — the app's own first-level roots — and the path belongs to the41 hosted site. Stage 2 resolves the FULL path in the app's own router: the node42 exists, so the request is served natively; a structural miss under a claimed root43 belongs to the site after all and falls through. ``resolves_natively`` asks the44 router WITHOUT auth filters, so a route that exists but would be denied still45 answers its own 403 natively and never leaks the site behind it.46 47 **The forward is one line.** Everything the pool does — the identity, the wait of48 a user between two homes, the placement, the wire — happens inside49 ``SpaCommander.serve_request``. This front never names a group, a worker or a50 wire, and keeps no state of its own. It mints nothing either: the cookie carries51 the hosted site's own connection id, read off the answer and written back on the52 way out, so the identity the browser holds and the identity the site keeps are53 one and the same.54 55 **The pool is configurable while it runs.** ``orchestration.profile_name`` names56 a stored profile the boot must find, ``env_settings`` is what the installation57 fixed by environment — a constructor kwarg and no word of any grammar — and the58 effective configuration of the one group is composed as59 defaults ⊕ recipe ⊕ profile ⊕ env, the two immutable levels kept apart, so every60 later apply recomposes instead of stacking. A profile governs exactly ONE group:61 with zero or several the boot fails and a hot apply reads 409. With62 ``orchestration.control_enabled`` on, ``OrchestrationControl`` mounts under63 ``_orchestration`` — LAST, once the pool is actually up, so a boot that failed64 leaves the router untouched — and the three routes reach the vertex's own apply;65 with the gate off that path belongs to the hosted site like any other.66 67 **What comes back out.** The site's own answer, rebuilt from the reply. A refusal68 — nobody could take this user, or he stayed between two homes longer than this69 front is willing to wait — is a polite **503** carrying the ``Retry-After`` the70 vertex composed, because only the vertex knows when the machine will have decided71 again. A failure — the site broke inside a healthy process, or the wire is gone —72 is a **502**: the site is this gateway's upstream and its breakage is not the73 client's fault. Both answer with a GENERIC line: the real text is written to the74 log, where the sysop looks for it, and never handed to a browser, because an75 exception's text carries the inside of the house.76 """77 78 from __future__ import annotations79 80 import asyncio81 import logging82 import os83 from typing import TYPE_CHECKING, Any, NoReturn84 85 from genro_bag import BagResolver86 from genro_builders.builder import element87 from genro_routes import RoutingClass, route88 89 from genro_asgi.asgi_endpoint import BufferedAsgiEndpoint90 from genro_asgi.application import ApplicationGrammar91 from genro_asgi.channel.frame import Frame92 from genro_asgi.http_record import HttpRecord93 from genro_asgi.transport_limits import FrameTooLarge, HttpBodyTooLarge, http_max_body_size94 from genro_asgi.config.handler import ConfigError95 from genro_asgi.exceptions import HTTPBadRequest, HTTPException, HTTPForbidden, HTTPNotFound96 from genro_asgi.lifespan import FatalBootError97 from genro_asgi.middleware.base import cookie_value98 from genro_asgi.orchestration_profile_store import (99 OrchestrationProfileContentError,100 OrchestrationProfileNameError,101 OrchestrationProfileNotFoundError,102 OrchestrationProfileStore,103 )104 from genro_asgi.response import Response105 from genro_asgi.wsx_payload import SerializedWsxPayload106 from genro_asgi.routed_application import RoutedApplication107 from genro_asgi.server import QUITTING, REFUSED_RETRY_AFTER_SECONDS, RUNNING108 from .inspector_section import INSPECTOR_ENV_VAR, InspectorSection109 from .orchestration import AssignmentRefused, SiteFailedRequest, SpaCommander110 from .orchestration.group_policy import GroupPolicy, GroupPolicyError111 from .orchestration.spa_commander import SingleGroupRequired112 113 if TYPE_CHECKING:114 from pathlib import Path115 116 from genro_asgi.types import Receive, Scope, Send117 118 #: The routing cookie. Its value is the hosted site's OWN connection id, never a119 #: number of ours: one identity space, so the chain from the cookie to the worker120 #: translates nothing.121 SPA_CONNECTION_ID_COOKIE = "spa_connection_id"122 123 #: How long that cookie lives. The same 24 hours the site gives its own connection124 #: cookie (``CONNECTION_TIMEOUT * 24``, gnrwebpage_proxy/connection.py): ours must125 #: not die first, or a browser the site still recognises would come back with no126 #: connection named and be routed anonymous for the rest of the day.127 CONNECTION_COOKIE_MAX_AGE = 24 * 3600128 129 #: How long a request may spend, in total, waiting for a user who is between two130 #: homes. It is NOT derived from the beat: a move is an evict and an install,131 #: milliseconds when things are well, and past a few seconds something is wrong132 #: and the polite refusal is the honest answer.133 REQUEST_HOLD_MAX_SECONDS = 5.0134 135 #: The app's own root the runtime configuration answers under. It is claimed136 #: ONLY when the gate is on: a first-level root of this front is a root the137 #: hosted site loses, and a machine nobody reconfigures must not lose it.138 ORCHESTRATION_ROOT = "_orchestration"139 140 #: The front's internal root for what a page asks of its channel. The reserved141 #: segment is the core's (the server answers ``/_wsx/ping`` itself); under an142 #: application's mount it is this front's own control surface.143 WSX_ROOT = "_wsx"144 145 #: The words that used to hang on the application element and now live on the146 #: ``orchestration`` node. A recipe still writing one of them is refused by name.147 MOVED_APPLICATION_WORDS = ("profiles_path", "profile_name", "orchestration_control")148 149 #: What the two refusals say out loud. The inside of the house — which user, which150 #: worker, what the site raised — goes to the log and not through the wire.151 ERR_503_TEXT = "server busy"152 ERR_502_TEXT = "the site could not answer this request"153 154 155 class SpaApplicationGrammar(ApplicationGrammar):156 """The words this front adds to a recipe, all under one node: ``orchestration``.157 158 They live HERE and not in the site dialect because a pool belongs to the159 application that owns it: the chain is server → applications → this front →160 ITS ORCHESTRATION → its commander → its groups → their workers, and the161 recipe says it in that shape::162 163 front = applications.application(app_class=SpaApplication, code="spa",164 mount="")165 orchestration = front.orchestration(profiles_path="/var/spa/profiles",166 profile_name="busy_hours",167 control_enabled=True)168 commander = orchestration.commander(frozen_users_path="/var/spa/frozen",169 instance_dir="/run/genro-asgi")170 commander.groups(default="standard").group(name="standard")171 172 ``orchestration`` is REQUIRED: a spa front IS its pool, so one declared173 without the node is an incomplete configuration and the server does not174 start. Wanting no pool is declaring no spa front. Nothing of the pool hangs175 on the application element any more — a recipe writing ``profiles_path``,176 ``profile_name`` or ``orchestration_control`` there is refused by name, with177 the new path in the message.178 179 ``env_settings`` is no word of any grammar: it is a dict the Python recipe180 composes at runtime out of the environment it has already read, and it181 travels as a plain constructor kwarg of the application — the last level of182 the overlay, above anything a recipe or a profile may say.183 """184 185 @element(sub_tags="commander[0:1]", node_label="orchestration")186 def orchestration(187 self,188 profiles_path: str | BagResolver | None = None,189 profile_name: str | None = None,190 control_enabled: bool = False,191 ) -> None:192 """The whole orchestration of this front: its own three words, then the pool.193 194 ``profiles_path`` is the folder the stored profiles are read from — the195 same one the ``_sysop`` archive writes — and ``profile_name`` the profile196 the boot must find and put in force. Named without a folder, or named and197 not there, or there and invalid: the server does not start.198 199 ``control_enabled`` opens the runtime configuration under the front's200 ``_orchestration`` root — apply, reload, status. Off, that root is never201 claimed and the path belongs to the hosted site.202 203 The node MUST carry a ``commander``: a profile and a control surface204 with no pool to act on address nothing, and the boot says so instead of205 starting half-configured. The node itself is not optional either — a spa206 front declared without it does not start.207 """208 209 @element(parent_tags="commander", sub_tags="group", collection_key="name")210 def groups(self, default: str = None) -> None:211 """Collection of worker groups, each labelled by its ``name`` — stable paths212 ``applications.<code>.orchestration.commander.groups.<name>``. A group is213 the workers built from ONE grammar: the same child, the same policies.214 215 The optional ``default`` ELECTS the group that receives whoever arrives216 with no past; omitted, the first group declared is the one. Unlike217 ``applications.default``, which is a redirect destination and elects218 nothing, this one decides where a newcomer is born."""219 220 @element(parent_tags="groups", sub_tags="")221 def group(222 self,223 name: str = None,224 memory_max_percent: float | BagResolver = None,225 worker_max_number: int | BagResolver = None,226 worker_memory_max_percent: float | BagResolver = None,227 worker_memory_admission_percent: float | BagResolver = None,228 restart_occupancy_max_percent: float | BagResolver = None,229 cpu_close_percent: float | BagResolver | None = None,230 cpu_admission_close_percent: float | BagResolver = None,231 cpu_admission_reopen_percent: float | BagResolver = None,232 cpu_offload_percent: float | BagResolver | None = None,233 cpu_retirement_quiet_seconds: float | BagResolver | None = None,234 cpu_heating_seconds: float | BagResolver | None = None,235 cpu_cooling_seconds: float | BagResolver | None = None,236 worker_admission_interval_seconds: float | BagResolver | None = None,237 worker_min_life_seconds: float | BagResolver = None,238 worker_max_users: int | BagResolver = None,239 user_idle_freeze_minutes: float | BagResolver = None,240 entry_module: str = None,241 executable: str | BagResolver = None,242 worker_class: str = None,243 main_threadpool_size: int | BagResolver = None,244 aux_threadpool_size: int | BagResolver = None,245 worker_kwargs: dict = None,246 engine_factory: str = None,247 engine_kwargs: dict = None,248 ) -> None:249 """One group of workers: its own policies, and the identity of its child.250 251 ``name`` is the collection key and names the group's workers too252 (``<name>_0001``), so it is short — a worker's name is its socket's.253 254 **Nothing here says how many workers there are.** The group brings its255 reception into being at boot and then grows on demand and shrinks by256 waste, so the count is a reading and never a setting —257 ``worker_max_number`` included: it says how many workers the quota is258 SIZED FOR (the per-worker ceiling becomes quota / that number, 6 when259 nothing is declared), and caps nothing. It replaces the bridge-era260 RAM-share-over-workers derivation with one intuitive count of slots;261 an explicit ``worker_memory_max_percent`` wins over it.262 263 The POLICIES: ``memory_max_percent`` is this group's share of the264 server's concession and ``worker_memory_max_percent`` what ONE worker may265 hold of that share (the same word one rung down — the cascade is machine,266 concession, quota, worker); ``worker_memory_admission_percent`` is how full a worker267 gets before it stops admitting and ``restart_occupancy_max_percent`` where268 a process is replaced instead of kept; ``cpu_close_percent`` is the temperature, shared onto the survivors, under269 which a worker is a closure candidate (unset, the reopen threshold itself)270 and ``worker_min_life_seconds`` the age before which a worker is no271 closure candidate; ``worker_admission_interval_seconds`` is how long after272 admitting a user a worker is skipped by the placement, so its load shows273 in the temperature first; ``user_idle_freeze_minutes`` is the silence past274 which the group parks a user in the freezer. ``cpu_admission_close_percent`` (experimental,275 off when omitted) turns on soft CPU admission: a worker above it is276 closed to NEW users and reopens below ``cpu_admission_reopen_percent``. CPU277 samples do not fork processes. When a concrete arrival finds no open278 worker that can admit it, placement creates one worker and assigns that279 same user. ``cpu_offload_percent`` (off when omitted; requires280 ``cpu_admission_close_percent`` and sits above it) makes a CPU-closed worker past281 it slim itself: one active user per beat — the least busy — is parked282 in the freezer, and his next request lands on an open worker or births283 one. ``cpu_retirement_quiet_seconds`` is how long the CPU must284 stay silent — no blocking or reopening — before retirement judges285 again: the quiet of the GROUP, distinct from the age of one worker,286 restarted whole by every CPU admission transition. ``cpu_heating_seconds``287 (1 s) and ``cpu_cooling_seconds`` (5 s) are the two time constants of the288 filter every CPU judge reads the 100 ms temperature through: a worker289 heats up fast and cools down slowly, so one idle sample never reopens it.290 291 The IDENTITY of the child: ``entry_module`` (what ``python -m`` runs),292 ``executable`` (the interpreter — a group is how two versions of a site293 live side by side), ``worker_class`` (the ``module:Class`` the child294 loads), the two thread pool sizes, and ``worker_kwargs``, the grammar that295 class is built with. The two paths are the installation's and are declared296 once, on ``commander``.297 298 The BIRTH of the child: ``engine_factory`` is the ``module:Class`` of the299 class that builds the one expensive thing all this group's workers share,300 and ``engine_kwargs`` what that class is built with. Declared, the group301 runs a template process that builds it once and forks every worker out of302 it, so the cost is paid once instead of once per worker. Omitted, the group303 spawns its workers the ordinary way and has no template at all.304 """305 306 @element(parent_tags="orchestration", sub_tags="groups[0:1]", node_label="commander")307 def commander(308 self,309 frozen_users_path: str | BagResolver = None,310 instance_dir: str | BagResolver = None,311 memory_max_percent: float | BagResolver = None,312 machine_memory_alarm_percent: float | BagResolver = None,313 orchestration_log_path: str | BagResolver = None,314 orchestration_log_max_bytes: int | BagResolver = None,315 orchestration_log_backup_count: int | BagResolver = None,316 user_expiry_hours: float | BagResolver = None,317 guest_expiry_hours: float | BagResolver = None,318 cpu_temperature_sample_seconds: float | BagResolver | None = None,319 ) -> None:320 """The SPA pool: the vertex's own policies, and the groups under it.321 322 The two PATHS of the installation, declared once here and shared by every323 group: ``frozen_users_path`` is the freezer root — the vertex reads what a324 worker wrote there, so it is one root for the whole machine — and325 ``instance_dir`` holds the sockets.326 327 ``cpu_temperature_sample_seconds`` is the cadence of commander-side,328 traffic-independent worker CPU measurement. CPU admission, placement and329 offload read this channel through each group's filter; omit it for the330 100 ms default.331 332 ``memory_max_percent`` is what this server may hold OF THE MACHINE (the333 concession; omitted, all of it), and every percentage below is a share of334 it. ``machine_memory_alarm_percent`` is the health line of the whole335 machine, past which nothing grows. The freezer's own storage answers to no336 key: under a tenth free the log says so and the machine asks for more.337 338 ``orchestration_log_path`` (+ ``_max_bytes`` / ``_backup_count``) is the339 file every order lands on — who decided, what, on whom, with which numbers340 and how it ended; omitted, the rows stay on the logger.341 342 ``user_expiry_hours`` / ``guest_expiry_hours`` are the ages a FROZEN user343 is kept for before the machine forgets him whole. A guest is shorter: he344 is a browser, not a person the machine knows.345 346 Technical times — the beat, the patience of a departure, the cadences —347 are module constants and not grammar: an installation tunes policies, not348 clocks.349 """350 351 352 class OrchestrationControl(RoutingClass):353 """The runtime configuration of one pool: apply, reload, read.354 355 Mounted under ``_orchestration`` only when the front's gate is on. The three356 routes carry no logic of their own: they name what the caller asked for and357 hand it to the front, which owns the translation of the vertex's refusals.358 """359 360 def __init__(self, application: SpaApplication) -> None:361 self.application = application362 363 @route(openapi_method="post")364 async def apply(365 self, body_data: dict[str, Any] | None = None366 ) -> dict[str, Any]:367 """Put the body in force as the profile level: an inline configuration.368 369 Args:370 body_data: the setpoints, written the way a stored profile writes371 them. Nothing stored stays active afterwards.372 373 Returns:374 The payload of the apply, as the vertex composed it.375 """376 return await self.application.apply_settings(377 profile=self.application.body_profile(body_data), source="inline"378 )379 380 @route(openapi_method="post")381 async def reload(382 self, body_data: dict[str, Any] | None = None383 ) -> dict[str, Any]:384 """Read a stored profile off the disk again and put it in force.385 386 Args:387 body_data: optionally ``{"name": ...}`` — the profile to read, which388 becomes the active one; without it the active profile is reread.389 390 Returns:391 The payload of the apply, as the vertex composed it.392 393 Raises:394 HTTPException: 400 — no name was given and no profile is active, so395 there is nothing to reload.396 """397 asked = self.application.body_profile(body_data).get("name")398 name = asked or self.application.orchestration_commander.active_profile399 if name is None:400 raise HTTPBadRequest(401 "nothing to reload: no name was given and no profile is active"402 )403 return await self.application.apply_settings(profile_name=name, source="profile")404 405 @route()406 async def status(self) -> dict[str, Any]:407 """What configuration is in force right now; no lock is taken to answer."""408 return self.application.settings_status409 410 411 class WebsocketOperations(RoutingClass):412 """What a WORKER calls on the vertex to reach a browser: the push.413 414 Attached under the commander's own operations by the front, because the415 delivery needs both halves of the machine — the registry of live sockets,416 which is the server's, and ``page_connection_map``, which is the vertex's.417 418 Args:419 application: the front, which holds the server and the commander.420 """421 422 def __init__(self, application: SpaApplication) -> None:423 self.application = application424 425 @route()426 async def send(427 self, page_id: str, path: str, data: Any = None, cid: str | None = None428 ) -> dict[str, Any]:429 """Write one message of the site onto the socket that page speaks on.430 431 Args:432 page_id: the page to address.433 path: what the client routes the message on.434 data: the payload, as the TYTX string the worker put on the lane —435 the lane is JSON and carries no date, Decimal or bytes of its436 own. The frontend forwards this explicit serialized value.437 cid: the connection the worker believes that page belongs to.438 439 Returns:440 ``{"delivered": bool}`` — written to a socket, or nobody there.441 442 The page is validated against ``page_connection_map`` before anything443 is written: a page the fold has already dropped is not there any more,444 and a page whose connection is another one is not this caller's to445 write to. Fire and forget (W-12): delivered means written to the446 socket, never executed by the page.447 """448 application = self.application449 owner = application.commander.page_connection_map.get(page_id)450 if owner is None or (cid is not None and owner != cid):451 return {"delivered": False}452 server = application.server453 return {"delivered": await server.send_serialized_message(454 page_id, path, SerializedWsxPayload(data)455 )}456 457 458 class WsxControl(RoutingClass):459 """What a page asks of its channel, mounted under ``_wsx`` on the front.460 461 Sibling of ``OrchestrationControl``: a routing class under an internal root462 of the front, whose routes carry no logic of their own. ``openchannel`` is463 the only command for now, and it is the one a page MUST send before any464 other message of its own reaches the worker.465 """466 467 def __init__(self, application: SpaApplication) -> None:468 self.application = application469 # The neutral ``fields`` block is what tells ``bind_kwargs`` that a470 # handler declared ``_request``; it is the pydantic plugin that fills471 # it, so this surface arms it on its own router, as the console does.472 self.route.plug("pydantic")473 474 @route()475 async def openchannel(476 self, parameters: dict[str, Any] | None = None, _request=None477 ) -> dict[str, Any]:478 """Open the channel of one page: validate it, then write it on its row.479 480 Args:481 parameters: how it wants to be served on it; ``sequential`` asks for482 one call at a time.483 _request: the live request, injected by ``bind_kwargs``. Left484 unannotated so it stays out of the schema; it is where the485 connection AND the page are read from — a client does not get486 to say which connection it is, and the page it names travels in487 the envelope's own field, never in the payload.488 489 Returns:490 What the client reads as the answer of the command.491 492 Raises:493 HTTPForbidden: this page is not this connection's. The vertex knows494 which connection every page belongs to — the birth of a page495 rode the reply of the request that created it — so the check496 costs nothing and never goes down to the worker.497 HTTPBadRequest: the message names no page, or the request carries498 no connection at all.499 500 The page is bound to the socket by the CONNECTION, not here, and only501 because this answered 200: whoever holds the socket does the binding,502 whoever holds the pool decides (owner, 2026-09-07).503 """504 page_id = _request.scope.get("genro.page_id")505 if not page_id:506 raise HTTPBadRequest("openchannel names no page: put it in the envelope's page_id")507 cid = self.application.request_cid(_request.scope)508 if cid is None:509 raise HTTPBadRequest("this connection carries no cookie")510 commander = self.application.commander511 if commander.page_connection_map.get(page_id) != cid:512 raise HTTPForbidden(f"page {page_id!r} is not this connection's")513 reply = await commander.serve_wsx_request(514 cid,515 {"wsx": {"cid": cid, "page_id": page_id, "parameters": parameters}},516 hold_timeout=REQUEST_HOLD_MAX_SECONDS,517 )518 return {"channel": reply.get("result")}519 520 521 class SpaApplication(RoutedApplication):522 """A single-page-application front backed by the new user-sticky pool."""523 524 #: The words this front adds to a recipe, read back under its own code.525 forwards_payloads = True526 527 grammar = SpaApplicationGrammar528 529 @property530 def handshake_cookie(self) -> str | None:531 """The connection cookie a websocket handshake must carry to reach this front.532 533 Returns:534 The name of the SPA's own connection cookie.535 536 Every message on that socket is a request of the user the cookie names,537 so a socket opened without one could never be served: the handshake is538 accepted and closed 1008, and the browser reads why. The first live539 probe found this property unimplemented and the socket left open for540 ever (#70).541 """542 return SPA_CONNECTION_ID_COOKIE543 544 #: The pool this front builds. A subclass names another one — a vertex that545 #: can grow its own machine, say — and the recipe names the subclass.546 commander_class: type[SpaCommander] = SpaCommander547 548 def __init__(self, *, env_settings: dict[str, Any] | None = None, **kwargs: Any) -> None:549 """Build the front; the whole orchestration is read at startup.550 551 Args:552 env_settings: the setpoints the installation fixed by environment,553 the strongest level of the overlay. It is a runtime dict and no554 word of any grammar, so it travels HERE and nowhere else.555 556 Raises:557 ConfigError: the recipe wrote one of the three words that moved558 under ``orchestration`` on the application element.559 """560 self.refuse_moved_words(kwargs)561 self._commander: SpaCommander | None = None562 self._channel_mounted = False563 self._logger = logging.getLogger(__name__)564 #: Where the named profiles are read from at boot; the vertex is given565 #: the same folder and reads them itself from there on. Written by the566 #: boot out of the orchestration node, so it is None until then.567 self.profiles_path: str | Path | None = None568 #: Which profile the boot puts in force, and the front's own answer to569 #: "what was active" until the first apply moves it.570 self.profile_name: str | None = None571 #: Whether the three configuration routes exist at all. The boot reads572 #: it, and mounts them only once the pool is up.573 self.control_enabled = False574 #: Whether THIS front already put them on its router. It tells a second575 #: boot (a retry, a stop and start) from a root somebody else claimed.576 self._control_mounted = False577 #: The environment's own level, kept as its own dict for good: every578 #: apply recomposes recipe ⊕ profile ⊕ env instead of stacking.579 self.env_settings = dict(env_settings or {})580 super().__init__(**kwargs)581 self._forward_slots = asyncio.Semaphore(16)582 583 def refuse_moved_words(self, kwargs: dict[str, Any]) -> None:584 """Refuse, by name, the words that moved under ``orchestration``.585 586 Args:587 kwargs: the constructor kwargs, which are the attributes the recipe588 wrote on the application element.589 590 Raises:591 ConfigError: naming every word found and the path it now lives at.592 The bare ``TypeError`` of an unexpected kwarg would say the word593 is unknown, when it is known and has moved.594 """595 moved = [word for word in MOVED_APPLICATION_WORDS if word in kwargs]596 if moved:597 raise ConfigError(598 f"{', '.join(moved)}: no longer written on the application element — "599 f"the orchestration of a spa front is one subtree, so these live on "600 f"applications.<code>.orchestration "601 f"(orchestration_control is now control_enabled)"602 )603 604 @property605 def commander(self) -> SpaCommander:606 """The pool this front owns.607 608 Raises:609 RuntimeError: it is not built yet — the vertex is born at startup,610 out of a configuration that only a mounted application can read.611 """612 if self._commander is None:613 raise RuntimeError(614 f"{type(self).__name__} has no pool yet: it is built when the server starts"615 )616 return self._commander617 618 @property619 def internal_roots(self) -> set[str]:620 """The app's OWN first-level roots, minus ``index``.621 622 Recomputed per access from the STRUCTURAL router view623 (``forbidden=True``): a route hidden by a plugin filter is still a624 claimed root, so it can never fall through to the hosted site.625 """626 nodes = self.route.nodes(lazy=True, forbidden=True)627 return (set(nodes.get("entries", {})) | set(nodes.get("routers", {}))) - {"index"}628 629 def resolves_natively(self, path: str) -> bool:630 """Whether ``path`` resolves to an EXISTING node in the app's router.631 632 Resolution runs with NO auth filters, so the answer is purely "does this633 node exist?". Only a genuine ``not_found`` is a miss; an existing node a634 filter would deny still answers True and stays native.635 """636 return bool(self.route.node(path).error != "not_found")637 638 async def on_startup(self) -> None:639 """Read the orchestration, build the pool, bring it up, then open the door.640 641 The vertex is born HERE and not in the constructor: its words live under642 ``applications.<code>.orchestration``, and an application reaches its own643 subtree only once a server has it. The three words of the node are read644 first and land on this front, so what follows composes on them.645 646 The order is the point. Everything that can refuse runs BEFORE anything647 is mounted — the node, the composition, the vertex, its start — and648 ``OrchestrationControl`` goes on the router last, only when the pool is649 actually up, with the inspector's own mount after it.650 A boot that fails leaves the router exactly as it was, and651 leaves this front holding NO vertex: a start that raises, and a mount that652 raises after it, both take the half-built pool back down first, so the653 next startup builds a new one instead of guarding a broken one.654 655 A front declared with no ``orchestration`` node is an INCOMPLETE656 configuration and does not boot: a spa front without a pool answers every657 request with a raise, so the recipe is asked for the node instead of the658 server pretending to serve. A front WITH the node and no commander under659 it is refused for the same reason: a profile and a control surface with660 nothing to act on address nothing. Wanting no pool at all is declaring no661 spa front, not declaring one and leaving it hollow.662 663 A startup on a front whose pool is already up does nothing: the same664 object is never given a second vertex while the first is running.665 666 Acts on this application — the vertex is born here — and, through667 ``SpaCommander.start``, on the machine: the base group's reception is668 launched and awaited, then the beat starts. A reception that would not669 start leaves its group broken and the front serves polite refusals until670 the group's own round brings one up: a process that fails to start can be671 a passing thing, and the machine knows how to heal it.672 673 A named profile or an environment level is composed onto the recipe674 BEFORE the vertex is built, so the pool is born already effective. What675 the composition refuses — a profile that is not there, one the schema676 rejects, a machine with no single group to give the setpoints to — is677 said once on this module's logger and raised as ``FatalBootError``, the678 one exception the lifespan does not swallow: the server does not start,679 because a pool nobody could configure as asked is not the pool that was680 asked for.681 """682 if self._commander is not None:683 return684 handler = self.server.config685 orchestration = handler.orchestration_kwargs(self.code)686 if orchestration is None:687 self.refuse_boot(688 "it declares no 'orchestration' node — a spa front is its pool, so "689 "the recipe must write applications.<code>.orchestration with a "690 "commander under it"691 )692 self.profiles_path = orchestration.get("profiles_path")693 self.profile_name = orchestration.get("profile_name")694 self.control_enabled = bool(orchestration.get("control_enabled", False))695 commander_kwargs = handler.commander_kwargs(self.code)696 if commander_kwargs is None:697 self.refuse_boot("its orchestration declares no commander")698 if (699 self.control_enabled700 and not self._control_mounted701 and ORCHESTRATION_ROOT in self.internal_roots702 ):703 self.refuse_boot(704 f"the {ORCHESTRATION_ROOT!r} root is already claimed by its own router, "705 f"so the runtime configuration has nowhere to mount"706 )707 try:708 groups, recipe_settings = self.boot_group_settings(handler.group_kwargs(self.code))709 except Exception as refused:710 self.refuse_boot(f"the pool cannot be configured: {refused}", refused)711 commander = self.commander_class(712 **commander_kwargs,713 groups=groups,714 profiles_path=self.profiles_path,715 recipe_settings=recipe_settings,716 env_settings=self.env_settings,717 active_profile=self.profile_name,718 )719 self._commander = commander720 try:721 await commander.start()722 except asyncio.CancelledError:723 # Whoever cancelled this boot gets its cancellation back: the pool is724 # taken down all the same, and nothing is turned into a boot failure.725 await self.take_pool_down(commander)726 raise727 except Exception as broken:728 await self.take_pool_down(commander)729 self.refuse_boot(f"the pool could not be brought up: {broken}", broken)730 try:731 self.mount_control()732 self.mount_channel_control()733 except Exception as broken:734 await self.take_pool_down(commander)735 self.refuse_boot(736 f"the runtime configuration could not be mounted and the pool was "737 f"taken back down: {broken}",738 broken,739 )740 self.mount_inspector()741 742 async def take_pool_down(self, commander: SpaCommander) -> None:743 """Undo a boot that could not finish: stop the pool, and let go of it.744 745 Args:746 commander: the vertex the boot built and could not bring all the way747 up. It is passed rather than read back, because letting go is748 exactly what this does.749 750 A pool that refuses to stop is said on this module's logger and nothing751 more: it must never replace the reason the boot failed, which is what the752 caller raises next. Either way the front ends holding no vertex, so the753 next startup builds a new one instead of finding this one.754 """755 try:756 await commander.stop()757 except Exception:758 self._logger.exception(759 "Front %s: the pool refused to go back down after a failed boot", self.code760 )761 finally:762 self._commander = None763 764 def refuse_boot(self, reason: str, cause: Exception | None = None) -> NoReturn:765 """Say the boot failed once, on this module's logger, and make it fatal.766 767 Args:768 reason: what is wrong, in the front's own words.769 cause: the exception underneath, when there is one — the tests read770 it back off ``__cause__``.771 772 Raises:773 FatalBootError: always. It is the one exception the lifespan does not774 swallow on startup, so uvicorn receives ``lifespan.startup.failed``.775 """776 failure = f"Front {self.code}: {reason}, the server does not start"777 self._logger.error(failure)778 raise FatalBootError(failure) from cause779 780 def mount_control(self) -> None:781 """Put the runtime configuration on the router, if the recipe asked for it.782 783 The LAST mutation of the boot, when the pool is up: the root is claimed784 only by a front that can actually answer on it. That the root is FREE was785 established before anything was built, so what is left here cannot refuse786 — and a startup after a shutdown finds the branch this method already put787 there and does nothing.788 """789 if not self.control_enabled or self._control_mounted:790 return791 self.route.add_branches(792 {"name": ORCHESTRATION_ROOT, "instance": OrchestrationControl(self)}793 )794 self._control_mounted = True795 796 def mount_inspector(self) -> None:797 """Put the pool's own watching page on ``_server``, if the environment asked.798 799 The last thing the boot does, and outside the mount above on purpose: a800 second ``inspector`` is a deliberate refusal, and wrapping it in that801 ``except`` would tell it as a failure of the runtime configuration.802 803 The section reads THIS pool, so it is the front that attaches it and not804 the ``_server`` app that hosts it: a core carrying no orchestration must805 not import one to look at it. A server has ONE orchestrated application806 (owner, 2026-09-07), so a mount already there means a second front is807 booting on the same server — the state that decision declares808 impossible, and the boot says so instead of watching one pool of two.809 810 Raises:811 FatalBootError: ``inspector`` is already attached.812 """813 if not os.environ.get(INSPECTOR_ENV_VAR):814 return815 server_app = self.server.applications.get("_server")816 if server_app is None:817 self._logger.warning(818 "Front %s: %s is set but no '_server' application is mounted, "819 "so the inspector has nowhere to go",820 self.code,821 INSPECTOR_ENV_VAR,822 )823 return824 if "inspector" in server_app.sections:825 self.refuse_boot(826 "the 'inspector' section is already attached, so a second front is "827 "booting on this server and a server has one orchestrated application"828 )829 server_app.attach_section(InspectorSection(self), name="inspector")830 831 def mount_channel_control(self) -> None:832 """Claim ``_wsx`` on this front's router: the channel commands of a page.833 834 No gate: a front that hosts pages hosts their channel too, and the835 command is refused per page anyway — a page that is not this836 connection's gets a 403 whether the root is there or not. Called once,837 when the pool is up, beside the orchestration root.838 839 The other half goes the other way: ``websocket`` is attached under the840 commander's own operations, which is how a worker reaches a browser.841 Not lazily — a worker may call it as soon as it is up.842 """843 if self._channel_mounted:844 return845 self.route.add_branches({"name": WSX_ROOT, "instance": WsxControl(self)})846 self.commander.commander_dispatcher.add_branches(847 [{"name": "websocket", "instance": WebsocketOperations(self)}]848 )849 self._channel_mounted = True850 851 def boot_group_settings(852 self, groups: dict[str, dict[str, Any]]853 ) -> tuple[dict[str, dict[str, Any]], dict[str, Any]]:854 """Compose the one group's effective setpoints, and keep the recipe's apart.855 856 Args:857 groups: the groups the recipe wrote, as ``{name: kwargs}`` — only the858 words it actually wrote, so the library defaults are not there.859 860 Returns:861 The same map with that group's setpoints replaced by the effective862 ones (defaults ⊕ recipe ⊕ profile ⊕ env, materialized by863 ``GroupPolicy``), and the recipe's own setpoint level on its own —864 the vertex recomposes every later apply from it. A machine with no865 single group and nothing to overlay is handed back untouched: it is866 the composition that has always been legitimate.867 868 Raises:869 SingleGroupRequired: there is an overlay to compose and this machine870 has zero or several groups, so nothing says whose setpoints871 these are.872 OrchestrationProfileNotFoundError: the named profile is not there, or no folder873 was declared to look for it in.874 OrchestrationProfileNameError, OrchestrationProfileContentError:875 the file could not be read as a profile.876 GroupPolicyError: the composed settings are invalid, carrying every877 violation found.878 """879 if len(groups) != 1:880 if self.profile_name is None and not self.env_settings:881 return groups, {}882 raise SingleGroupRequired(883 f"Front {self.code}: a profile governs exactly one group, this recipe "884 f"declares {len(groups)} ({sorted(groups)})"885 )886 name, group_kwargs = next(iter(groups.items()))887 recipe_settings = {888 key: value for key, value in group_kwargs.items() if key in GroupPolicy.SETPOINTS889 }890 profile: dict[str, Any] = {}891 if self.profile_name is not None:892 if self.profiles_path is None:893 raise OrchestrationProfileNotFoundError(894 f"profile {self.profile_name!r} was named and no profiles folder "895 "was declared"896 )897 profile = OrchestrationProfileStore(self.profiles_path).read(self.profile_name)898 policy = GroupPolicy.from_settings({**recipe_settings, **profile, **self.env_settings})899 structural = {900 key: value for key, value in group_kwargs.items() if key not in GroupPolicy.SETPOINTS901 }902 return {name: {**structural, **policy.to_settings()}}, recipe_settings903 904 @property905 def orchestration_commander(self) -> SpaCommander:906 """The pool, when it is in a state to be reconfigured.907 908 Raises:909 HTTPException: 503 — the pool is not built yet, or the server has910 left RUNNING: a machine on its way out takes no new911 configuration, and the caller is told to come back.912 """913 if self._commander is None or self.server.state != RUNNING:914 raise HTTPException(915 503,916 "the pool is not in a state to be reconfigured",917 headers=[(b"retry-after", str(REFUSED_RETRY_AFTER_SECONDS).encode())],918 )919 return self._commander920 921 def body_profile(self, body_data: Any) -> dict[str, Any]:922 """The request body as a JSON object, or the one 400 that leaves no audit.923 924 Args:925 body_data: what the request layer hydrated — a dict when the body926 was a JSON object, the raw text when it could not be parsed.927 928 Returns:929 The object, an absent body reading as an empty one.930 931 Raises:932 HTTPException: 400 — the body is not a JSON object. It is refused933 HERE, before the vertex is asked anything, which is what keeps a934 malformed body out of the orchestration log.935 """936 if body_data is None:937 return {}938 if not isinstance(body_data, dict):939 raise HTTPBadRequest("the body of a configuration must be a JSON object")940 return body_data941 942 async def apply_settings(943 self,944 *,945 profile: dict[str, Any] | None = None,946 profile_name: str | None = None,947 source: str,948 ) -> dict[str, Any]:949 """Ask the vertex for a new effective configuration, and answer its refusals.950 951 Args:952 profile: the profile level given inline.953 profile_name: the stored profile to read as that level instead.954 source: who asked — the word that reaches the audit and the answer.955 956 Returns:957 The payload the vertex composed: ``outcome``, ``source``,958 ``active_profile``, ``generation``, ``changed_settings`` and959 ``effective_settings``.960 961 Raises:962 HTTPException: 400 the settings or the stored file are invalid — the963 violations are the message —, 404 the named profile is not964 there, 409 this machine has no single group, 503 the pool is not965 in a state to be reconfigured.966 """967 commander = self.orchestration_commander968 try:969 return await commander.apply_group_settings(970 profile=profile, profile_name=profile_name, source=source971 )972 except SingleGroupRequired as several:973 raise HTTPException(409, str(several)) from several974 except OrchestrationProfileNotFoundError as missing:975 raise HTTPNotFound(str(missing)) from missing976 except (977 GroupPolicyError,978 OrchestrationProfileNameError,979 OrchestrationProfileContentError,980 ) as rejected:981 raise HTTPBadRequest(str(rejected)) from rejected982 983 @property984 def settings_status(self) -> dict[str, Any]:985 """What configuration is in force: the profile, the generation, the record.986 987 Returns:988 ``active_profile``, ``generation``, ``last_apply`` and the989 ``effective_settings`` the one group is running on. Nothing is990 locked to read it: the swap that writes those four never yields the991 loop, so what is read here is one apply's picture and never a mix.992 993 Raises:994 HTTPException: 409 this machine has no single group, 503 the pool is995 not in a state to answer.996 """997 commander = self.orchestration_commander998 try:999 group = commander.configured_group1000 except SingleGroupRequired as several:1001 raise HTTPException(409, str(several)) from several1002 return {1003 "active_profile": commander.active_profile,1004 "generation": commander.configuration_generation,1005 "last_apply": commander.last_apply,1006 "effective_settings": group.policy.to_settings(),1007 }1008 1009 async def on_shutdown(self) -> None:1010 """Take the pool down with the server: with a photo, or dry.1011 1012 A server that is QUITTING gets the soft quit — every user parked in the1013 reboot directory and the vertex's own item beside them. Any other way1014 out is dry, which is what every start that is not the deliberate liturgy1015 expects to find (F2).1016 1017 The front lets go of the vertex whatever it answered, so the pool it held1018 is never handed to a second startup: that one reads the configuration1019 again and builds a new one.1020 """1021 commander = self._commander1022 if commander is None:1023 return1024 try:1025 if self.server.state == QUITTING:1026 await commander.quit()1027 else:1028 await commander.stop()1029 finally:1030 # The front lets go whatever the vertex answered: a pool that failed1031 # to go down cleanly is still not this front's any more, and the next1032 # startup builds a new one instead of finding a dead one.1033 self._commander = None1034 1035 async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:1036 """Demultiplex: the app's own router, else the hosted site."""1037 path = str(scope.get("path", "/"))1038 first_segment = path.strip("/").split("/")[0]1039 if first_segment in self.internal_roots and self.resolves_natively(path):1040 await super().__call__(scope, receive, send)1041 else:1042 await self.forward_request(scope, receive, send)1043 1044 async def forward_request(self, scope: Scope, receive: Receive, send: Send) -> None:1045 """Admit before buffering: at most sixteen bounded HTTP bodies per front."""1046 async with self._forward_slots:1047 await self._forward_request(scope, receive, send)1048 1049 async def _forward_request(self, scope: Scope, receive: Receive, send: Send) -> None:1050 """Serve a site path through the pool: pack, hand over, translate, answer.1051 1052 The cookie is read ONCE here and travels down as it came — None included,1053 which is a browser the site has never named. Nothing is minted: the1054 identity is the site's to give, and it gives it while serving. What comes1055 back names the connection the request settled on, and that is what the1056 cookie is written with.1057 """1058 carried = self.request_cid(scope)1059 local_response = True1060 try:1061 http = await self.pack_http(scope, receive, carried)1062 reply = await self.commander.serve_request(1063 carried, http, hold_timeout=REQUEST_HOLD_MAX_SECONDS1064 )1065 except (FrameTooLarge, HttpBodyTooLarge):1066 response = Response(content="Request too large", status_code=413)1067 except AssignmentRefused as refusal:1068 self._logger.warning("Front %s: %s", self.code, refusal)1069 response = self.busy_response(refusal)1070 except ConnectionError as gone:1071 self._logger.warning("Front %s: %s", self.code, gone)1072 response = self.wire_lost_response(gone)1073 except SiteFailedRequest as failure:1074 self._logger.error("Front %s: %s", self.code, failure)1075 response = self.gateway_response(failure)1076 else:1077 response = self.build_response(reply, carried)1078 local_response = False1079 if local_response and scope.get("method") == "WSK":1080 result = await BufferedAsgiEndpoint(response).serve(scope, b"")1081 response = Response(content=result["body"], status_code=result["status"],1082 headers=result["headers"])1083 await response(scope, receive, send)1084 1085 def busy_response(self, refusal: AssignmentRefused) -> Response:1086 """The polite 503: come back when the machine will have decided again.1087 1088 No cookie goes out with it: the site never served this request, so there1089 is no connection to name — and a refusal must not overwrite the one the1090 browser already holds.1091 """1092 headers = [("content-type", "text/plain; charset=utf-8")]1093 if refusal.retry_after is not None:1094 headers.append(("retry-after", str(int(refusal.retry_after))))1095 return Response(content=ERR_503_TEXT, status_code=503, headers=headers)1096 1097 def wire_lost_response(self, gone: ConnectionError) -> Response:1098 """What a dead wire means depends on why the process on the other end left.1099 1100 A server that is quitting killed that wire on purpose, so the answer is1101 the polite 503 a refusal gets — the browser is told to come back, not1102 that something upstream broke. A wire that died while the server was1103 running IS a breakage, and reads 502.1104 """1105 if self.server.state == RUNNING:1106 return self.gateway_response()1107 return Response(1108 content=ERR_503_TEXT,1109 status_code=503,1110 headers=[1111 ("content-type", "text/plain; charset=utf-8"),1112 ("retry-after", str(REFUSED_RETRY_AFTER_SECONDS)),1113 ],1114 )1115 1116 def gateway_response(self, failure: SiteFailedRequest | None = None) -> Response:1117 """What the caller reads when the worker answered a failure.1118 1119 Args:1120 failure: what the worker said, when the caller is to be told; the1121 502 with the fixed text is the answer without it.1122 1123 Returns:1124 The refusal the worker named, with ITS words, when it named a1125 status — a page that never opened its channel is a 409 and the1126 browser must read why (#70). Everything else is the 502 of an1127 upstream that broke, whose reason stays in the log alone: what1128 failed inside the site is nobody's business out here.1129 """1130 if failure is not None and failure.status is not None:1131 return Response(1132 content=failure.cause,1133 status_code=failure.status,1134 headers=[("content-type", "text/plain; charset=utf-8")],1135 )1136 return Response(1137 content=ERR_502_TEXT,1138 status_code=502,1139 headers=[("content-type", "text/plain; charset=utf-8")],1140 )1141 1142 def build_response(self, reply: Frame, carried: str | None) -> Response:1143 """The outer response, rebuilt from the site's own answer.1144 1145 The cookie is written when the connection the site settled on is not the1146 one the browser sent: a first visit, and the replacement the site makes1147 whenever the connection its own cookie names does not validate. A request1148 that reused the connection it arrived with names none, and nothing is1149 written.1150 """1151 result = HttpRecord().decode_response(reply.payload)1152 response = Response(1153 content=result["body"],1154 status_code=int(result.get("status", 200)),1155 headers=[(str(name), str(value)) for name, value in result.get("headers") or []],1156 )1157 settled = reply.info.get("connection_id")1158 if settled is not None and settled != carried:1159 response.set_cookie(1160 SPA_CONNECTION_ID_COOKIE,1161 settled,1162 max_age=CONNECTION_COOKIE_MAX_AGE,1163 path="/",1164 httponly=True,1165 samesite="lax",1166 )1167 return response1168 1169 def request_cid(self, scope: Scope) -> str | None:1170 """The connection this request carries, or ``None`` when it carries none."""1171 return cookie_value(scope, SPA_CONNECTION_ID_COOKIE)1172 1173 async def read_body(self, receive: Receive) -> bytes:1174 """Drain the whole request body off the ASGI receive channel."""1175 body = bytearray()1176 while True:1177 message = await receive()1178 chunk = message.get("body", b"") or b""1179 if len(body) + len(chunk) > http_max_body_size():1180 raise HttpBodyTooLarge("request body exceeds buffered transport limit")1181 body.extend(chunk)1182 if not message.get("more_body", False):1183 break1184 return bytes(body)1185 1186 async def pack_http(1187 self, scope: Scope, receive: Receive, cid: str | None1188 ) -> Frame:1189 """Pack routing facts and a bounded opaque HTTP record for the child.1190 1191 The headers are forwarded as they came: our cookie is ours to route on1192 and the hosted site has no use for it — it reads its own, and the value1193 in both is its own connection id anyway.1194 1195 ``page_id`` and ``reply_path`` join the routing info only when the scope carries1196 them — a message born on a websocket does, a real HTTP request does not1197 — and the endpoint writes them into the hosted scope under their genro names.1198 """1199 info = {"format": "http", "cid": cid}1200 for key in ("genro.page_id", "genro.reply_path"):1201 if key in scope:1202 info[key.split(".")[1]] = scope[key]1203 return Frame(method="CALL", path="/site" + str(scope.get("path", "/")),1204 info=info, payload=HttpRecord().encode_request(scope, await self.read_body(receive)))