# Runbook How to run, check and ship the Kapitano Logistic backend. The dispatch algorithm's configuration, Redis operations, fallback behaviour and cost levers are added in task T9.2. --- ## 1. Local services | Service | Local setup | Needed by | |---|---|---| | PHP 8.4 | `C:\xampp\php\php.exe` | everything | | MariaDB 10.4 (MySQL-compatible) | XAMPP | the application; `php artisan migrate` | | A `kapitano_test` database | `CREATE DATABASE kapitano_test CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;` | the parallel-assignment test only; see below | | Redis 8.10 | portable build in `C:\Users\DELL\redis\Redis-8.10.1-Windows-x64-msys2` | dispatch (GEO index, assignment lock), `RedisCheckCommandTest` | | Laravel Reverb | `php artisan reverb:start` (a plain process, like Redis) | the route-plan repaint over websockets; `BROADCAST_CONNECTION=log` works without it | | Firebase service account | JSON in `storage/app/firebase/` (gitignored), path in `FIREBASE_CREDENTIALS`; needs the CA bundle below | push to captains, `php artisan firebase:test` | ### Starting Redis It is a plain process, not a Windows service, so it has to be started again after a reboot. The msys2 build reads a Windows absolute path as a relative one, so start it **from its own folder**: ```powershell Set-Location C:\Users\DELL\redis\Redis-8.10.1-Windows-x64-msys2 .\redis-server.exe kapitano-local.conf ``` `kapitano-local.conf` binds to `127.0.0.1:6379` with no persistence. Then confirm, from the project folder: ```powershell php artisan dispatch:redis-check ``` Every row must say `OK`. When Redis is down, `RedisCheckCommandTest` reports itself **skipped** — never leave it that way before a merge. ### The parallel-assignment test `tests/Feature/Dispatch/Concurrency/ParallelAssignmentTest.php` is the only test that does not use the in-memory SQLite database, because it spawns eight real PHP processes and they have to meet somewhere. It uses the `mysql_concurrency` connection (`DB_CONCURRENCY_*` in `phpunit.xml`, CI variables in the pipeline) and **runs `migrate:fresh` on `kapitano_test`**, so point it at a database you are happy to lose. Create it once: ```powershell & C:\xampp\mysql\bin\mysql.exe -u root -e "CREATE DATABASE IF NOT EXISTS kapitano_test CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" ``` Without it — or without Redis — both tests skip with the reason printed, so the suite stays green on a machine that has neither. It costs about 27 s when it does run. **A skip is not a pass.** These are the tests that prove two dispatchers cannot overfill one captain, and they are the easiest in the suite to lose silently: stop MariaDB and they go quiet and green. If they have been skipping, nothing has been checking the capacity race. #### When they fail with "Tablespace ... exists. Please DISCARD the tablespace" ``` SQLSTATE[HY000]: General error: 1813 Tablespace for table '`kapitano_test`.`sessions`' exists. ``` `migrate:fresh` dropped the tables but an `.ibd` file was left behind on disk — which happens when `mysqld` is killed while the test is migrating. InnoDB then refuses to create a table whose tablespace file it can still see, and **every run afterwards fails at the same table**, which looks like a broken test rather than a broken data directory. `DROP DATABASE` does not fix it either: it removes the tables, fails to remove the directory ("Directory not empty") and leaves the orphan exactly where it was. Delete the file with the server stopped, then recreate the database: ```powershell & C:\xampp\mysql\bin\mysqladmin.exe -u root --protocol=tcp -h 127.0.0.1 shutdown Remove-Item -Recurse -Force C:\xampp\mysql\data\kapitano_test # start mysqld again, then: & C:\xampp\mysql\bin\mysql.exe -u root -e "CREATE DATABASE kapitano_test CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;" ``` Safe because the database holds nothing but what `migrate:fresh` puts there. Check first that the directory contains only the orphan, and never confuse it with `kapitano_logistic`, which is the real one. #### Stray replication files in the data directory Check for these if MariaDB behaves oddly after being killed: ```powershell Get-ChildItem C:\xampp\mysql\data | Where-Object { $_.Name -like 'master-*' -or $_.Name -like 'mysql-relay-bin*' } ``` Thirteen of them accumulated on this machine, with names that are **URL-encoded lines from MariaDB's own error log** — `master-2026@002d09@002d20...ready@0020for@0020connections.info`. MariaDB reads every `master*.info` file on start as a multi-source replication connection, so each one spawns an I/O and an SQL thread that immediately fails with *"Invalid (empty) username when attempting to connect to the master server"* and is retried. Every restart added more. Nothing in this project uses replication. Stop the server and delete them: ```powershell Stop-Process -Name mysqld -Force Get-ChildItem C:\xampp\mysql\data | Where-Object { $_.Name -like 'master-*' -or $_.Name -like 'mysql-relay-bin*' -or $_.Name -eq 'multi-master.info' } | Remove-Item -Force ``` A clean start then logs no `Added new Master_info` lines at all, which is how to tell it worked. **This is worth fixing and it is *not* the cause of the hang below.** Both were cleared and the parallel tests still wedged the server, so do not stop looking here. #### Unsolved: the server stops accepting connections while the parallel tests run Reproduced four times on 2026-09-20. The signature is specific and worth recognising: - established connections keep working — a session opened beforehand can still run `SHOW PROCESSLIST` throughout; - **new** connections hang for ever, including a bare `PDO` connect with no database selected; - `mysqld` sits under 1 s of CPU, so it is blocked rather than busy; - it only happens while `tests/Feature/Dispatch/Concurrency` is running, which spawns eight PHP processes against `kapitano_test`. Ruled out so far: **name resolution** (reproduced with `--skip-name-resolve`), **connection exhaustion** (`Threads_connected` was 6, the limit is 151), **ephemeral port exhaustion** (18 `TIME_WAIT` of 16384 available), and the two data-directory faults above. Until it is understood, those four tests cannot be run on this machine, and **a skip is not a pass** — they are what proves two dispatchers cannot overfill one captain. CI runs them against a real MariaDB service container, which is where to check they are still green. ### Captain capacity drift `drivers.active_orders` is what an assignment reserves against, and it has to be — the reservation happens **before** the order row exists, which is what stops two dispatchers filling the same captain. The orders table is the truth about what a captain is carrying; the counter is how that truth is enforced atomically. Everything that assigns or ends a delivery keeps the two in step. Anything that writes an in-progress order another way does not, and the store feed is such a door. **Drifting high is the dangerous direction.** Too low lets a captain take one order more than they should, which the next delivery corrects. Too high locks that captain out of *every* assignment, silently, until somebody looks. ```powershell php artisan dispatch:reconcile-capacity # report only php artisan dispatch:reconcile-capacity --fix # correct them ``` `--fix` runs hourly from the scheduler. Each correction is a `Log::warning` carrying both numbers and whether the captain had been locked out — the correction is routine, needing one is not, and that log is where the cause gets found. A captain **being assigned at that moment is skipped**, not corrected: between reserving the capacity and writing the order the counter is ahead of the orders table on purpose, and "correcting" it there would hand the same capacity out twice. The reconciler takes the same per-captain lock the assignment takes, so it can tell the two apart, and catches the skipped ones on the next run. ### Websockets (Reverb) Route plans are pushed to the captain app and the dispatch screens over private channels. Reverb is the websocket server; it is a plain process, like Redis, and has to be started again after a reboot: ```powershell php artisan reverb:start ``` `BROADCAST_CONNECTION=reverb` in `.env`, with `REVERB_APP_ID` / `REVERB_APP_KEY` / `REVERB_APP_SECRET` beside it — `php artisan reverb:install` generates a set. Any values work as long as the server and the clients agree. The key reaches browsers and is public by design; the secret is not. On a machine with no websocket server, set `BROADCAST_CONNECTION=log`: broadcasts are written to the log instead of thrown away, and nothing else changes. The tests force it to `null`. `config/broadcasting.php` is deliberately **not** published — Laravel ships a `reverb` connection in its own default config, and a published copy would be one more file to keep in step for no gain. Channel authorisation lives in `routes/channels.php`, wired by `withBroadcasting()` in `bootstrap/app.php` on `auth:driver,admin`. ### Route deviation (reroute) Every accepted GPS ping is measured against the polyline of the captain's current route plan. Past `DISPATCH_REROUTE_DEVIATION_M` (250 m) **continuously** for `DISPATCH_REROUTE_DEVIATION_S` (30 s), the captain is taken to have gone their own way and one `RecomputeRoutePlan` job is queued with trigger `deviation`. One ping back near the line and the clock starts over. The episode lives in Redis at `dispatch:deviation:{driverId}` (15-minute TTL) and carries the plan version it belongs to. That is what makes it fire **once**: the state no longer matches after a replan, so the new plan gets a clean clock and a captain off route on every ping does not queue a recompute every few seconds. - **Nothing here calls the routing engine.** It decodes a polyline it already has. The engine call happens inside the recompute, for a captain who really has deviated. - **A degraded plan has no polyline**, so deviation is unanswerable there and the detector says no. Same for a captain carrying nothing. - **The check never fails a ping.** It runs through the same best-effort wrapper as the Redis writes beside it in `DriverLocationService::update()`; a failure is reported and the ping is still accepted. To make it quieter without touching code, raise `DISPATCH_REROUTE_DEVIATION_M` (a city with poor GPS) or `DISPATCH_REROUTE_DEVIATION_S` (captains who legitimately detour). Setting the distance absurdly high effectively turns it off; there is no separate switch, because a route nobody re-plans is worse than one re-planned too often. ### When the map service is down Nothing above the routing layer throws: a timeout, a refusal or a missing key all come back as `RoutingUnavailable`, and the dispatch code carries on with straight-line estimates. What that costs depends on whether the captain already has a route. | Situation | What the captain gets | |---|---| | No plan yet | A plan built from straight lines: right stops, right order, estimated times, **no polyline**. `degraded = true`. | | A plan, same stops left | **The plan they already have**, untouched — same version, real polyline — with `degraded` set to true on the row. | | A plan, stops changed | A rebuilt plan from straight lines, published as a new version, `degraded = true`. | The middle row is T7.5 and the one worth knowing: replacing a measured route with a straight-line one mid-delivery takes the line off the captain's map and gains nothing, so the route stays and only its honesty about the times changes. The flag is what tells a dispatcher the arrival times are ageing. `RecomputeRoutePlan` then asks again by itself — **5 s, 30 s, 120 s**, then it gives up with a `Log::warning` ("A route plan is still degraded after the last retry"). Nothing else would ever notice the outage ended: the next recompute otherwise waits for an order to move or the captain to leave the route. The flag clears on its own, because a successful recompute publishes a fresh version that is simply not degraded. The retry is the job **releasing itself**, not dispatching a new one. Laravel keeps a job's uniqueness lock across a release, so a captain still has at most one replan in flight and triggers arriving during the wait coalesce into it. If you ever change this to a fresh `dispatch()`, it will be silently swallowed as a duplicate. To see it in the log, watch for `Keeping the current route plan through a maps outage` — one line per captain per recompute, with the driver, the plan and the engine that failed. ### Every value the algorithm runs on All in `config/dispatch.php`, all env-backed. **Nothing in `app/Services/Dispatch` hardcodes one of these** — `scripts/check-architecture.php` fails the build if a literal matching a config value appears there, which is how they stay changeable. The three marked ✱ can additionally be overridden from the dashboard at runtime (`dispatch_settings` table, `DispatchSettingKey`); everything else is config only, on purpose — they are values somebody should have to deploy to change. | Key | Default | Sensible range | What it decides | |---|---|---|---| | `MAX_ACTIVE_ORDERS` | 2 | 1–3 | How many orders one captain carries. Raising it raises batching and risk together. | | `RADIUS_STEPS_KM` | 5,8,12 | — | The geo filter expands through these until Top N is found. | | `TOP_N` | 7 | 5–10 | How many captains reach the routing matrix. **This is the cost dial** — elements scale with it. | | `DETOUR_INDEX` | 1.3 | 1.2–1.5 | Straight line × this = road estimate before routing. | | `ETA_ESTIMATE_SPEED_KMH` | 30 | measure it | Straight-line km/h when nothing else answers. See below — **still a placeholder**. | | `HANDOFF_BUFFER_PREPAID_MIN` ✱ | 3 | 2–5 | Handing over a prepaid parcel. | | `HANDOFF_BUFFER_COD_MIN` ✱ | 5 | 4–8 | The same with cash to collect. | | `GPS_AGING_MIN` / `GPS_STALE_MIN` | 1.5 / 3 | — | Badge, then exclusion. A stale captain leaves the list. | | `ROUTING_TIMEOUT_MS` | 1000 | 800–2000 | How long a dispatcher waits before the list degrades. | | `SUGGESTION_CACHE_TTL_S` | 75 | 60–90 | How long an identical list is reused. Raising it cuts cost and staleness together. | | `MAX_BATCH_DETOUR_MIN` ✱ | 6 | 5–7 | The most a batch may delay the customer already on board. | | `TIE_BREAK_BAND_MIN` | 2 | 1–3 | Adjusted ETAs within this are tied; longest idle wins. | | `REROUTE_DEVIATION_M` / `_S` | 250 / 30 | — | Off route by this much, for this long, means a replan. | | `ASSIGN_LOCK_TTL_S` | 30 | — | How long one dispatcher holds a captain while deciding. | | `PING_MOVING_S` / `PING_STATIONARY_S` | 8 / 45 | — | What the captain app is told about its next ping. | | `MOVING_SPEED_MPS` | 2 | — | Above this a captain counts as moving. | | `BUDGET_FILTER_MS` / `BUDGET_ASSEMBLY_MS` | 100 / 200 | — | Measured, never enforced: over budget is recorded, not abandoned. | | `AT_STORE_RADIUS_M` / `AT_STORE_DROPOFF_KM` | 100 / 3 | — | When a captain counts as already at this store. | | `BATCH_REJECTED_POLICY` | `rank_lower` | or `exclude` | What happens to a captain whose batch is refused. | | `GPS_HISTORY_RETENTION_DAYS` | 30 | — | How long GPS history is kept. 0 keeps everything. | | `MAPS_UNIT_PRICE` | 0.005 | your contract | Turns the element counter into money on the KPI screen. Decides nothing. | Every key is prefixed `DISPATCH_` in `.env`. `.env.example` lists them all, commented. ### The cost levers, in the order worth pulling The maps bill is `elements × unit price`, and elements are consumed by the matrix, which is `candidates × pickups` per suggestion. In descending order of effect: 1. **`TOP_N`** — the matrix is N×K. Going from 7 to 5 is a 30 % cut in the largest line item. 2. **`SUGGESTION_CACHE_TTL_S`** — a cache hit costs nothing. Two orders from the same store within the window share one matrix. 3. **Traffic-awareness.** Already off for the matrix and on only for the single chosen route, which is the cheap arrangement. Turning it on for the matrix multiplies the element price. 4. **The field mask.** `GoogleRoutesEngine` asks for ten fields and no more. Adding tolls or fuel-efficient routing moves every call to a dearer SKU — the tests assert the exact mask because a comment would not have stopped it. 5. **Two-wheeler routing and waypoint optimisation** are never requested. Both are priced above standard, and the algorithm gets the same answers from `detour_index` and `StopSequencer`. Watch `routing_elements_per_assignment` and `maps_cost_per_delivery` on the KPI screen. Per delivery rather than per order, because two orders on one route cost barely more to plan than one. ### First contact with the real Google Routes API (2026-09-21) Run against a **Maps Demo Key**, which covers `computeRoutes` but **not** `computeRouteMatrix`. Three things came out of it, and two were bugs. #### 1. A real bug the simulation could never have caught `computeRoutes` answered **400 INVALID_ARGUMENT**: ``` Unknown name "waypoint" at 'origin': Cannot find field. ``` The two endpoints of the same API disagree about the shape of a point: | Endpoint | Shape | |---|---| | `computeRouteMatrix` | `origins: [ { "waypoint": { "location": … } } ]` — the origin *carries* a waypoint | | `computeRoutes` | `origin: { "location": … }` — the origin **is** a waypoint | `GoogleRoutesEngine` used one helper for both. Every route call would have failed on the first day with a real key. The simulation never reads the payload, and the engine test asserted the field mask, `travelMode` and the intermediates count — but not the shape of `origin`. Fixed, with both shapes now asserted explicitly. **The lesson is the general one:** a test double that never validates its input cannot catch a contract error. The only thing that finds these is a real call. #### 2. `ROUTING_TIMEOUT_MS = 1000` is too short for a cold call The first attempt failed with `cURL error 28: Resolving timed out after 1008 ms` — **both** calls, before Google was ever reached. The budget covers DNS resolution and the TLS handshake, not just the round trip. With the timeout raised to 15 s the same calls answered in 1.9 s and 5.8 s. On a warm connection in a datacentre 1 s is plausible. From a cold start — a fresh container, or the first request after a DNS TTL expires — it is not. Worth watching `routing_fallback_rate` for a spike after every deploy, and considering whether the budget should exclude connection setup. #### 3. What the demo key can and cannot do ``` matrix: 403 BILLING_DISABLED — computeRouteMatrix needs billing enabled route: 200 — computeRoutes works ``` So with a demo key the system runs in a genuinely useful half-state: **route plans are real** and **suggestion lists are degraded**, which is exactly what the design promises and it announces it (`ranking_degraded: true`, `degraded_reason: routing_error`). Verified end to end with `dispatch:demo`: both plans came back `drawn by google`, and the stored polyline decoded to **250 points over 772 characters** — a real road shape, against the six points (one per stop) the simulation produces. That is the first time the deviation detector has had real road geometry to measure against, and the first realistic test of the FCM payload size that made the push carry a pointer rather than the plan. #### The numbers, simulation versus reality Between the two Riyadh points `dispatch:routing-check` uses: | | Distance | Time | Implied speed | |---|---|---|---| | Simulation | 6.0 km | 12.0 min | 30 km/h (by definition) | | Google, first call | 5.5 km | 8.3 min | ≈ 40 km/h | | Google, second call | 5.5 km | 11.0 min | ≈ 30 km/h | **The same pair of points, asked twice, answered 8.3 and 11.0 minutes.** The distance was identical both times; only the traffic differed. That gap is the finding, and it is not the one the first call suggested. A single sample read as *"the real speed is 40, so the configured 30 is pessimistic"* — the second sample lands on 30 exactly. **The spread between two readings of one route is wider than the distance from either to the configured value**, so neither is evidence the setting is wrong. What it does show is that the quantity `ETA_ESTIMATE_SPEED_KMH` stands in for is **not a constant**, and no single number can be right for both calls. That is an argument for reading the ETA-accuracy KPIs over a period rather than retuning the constant, and for `dispatch:measure-speed` over a window wide enough to cover the working day — not for changing the 30. `DETOUR_INDEX = 1.3` overstated the road distance on both calls (5.5 km actual against the simulation's 6.0 km), which is the one reading the two calls agree on. Still one pair of points. #### To repeat this Put a key in `.env`, set `DISPATCH_ROUTING_ENGINE=google`, `php artisan config:clear`, then `php artisan dispatch:routing-check`. It reports the matrix and the route separately, so a partial key shows up as exactly that. The engine is left on `fake` by default: with the matrix refused, every suggestion list would be permanently degraded, which is worse for the ranking than the simulation. ### Switching engines ``` DISPATCH_ROUTING_ENGINE=fake # the simulation: no key, no network, deterministic DISPATCH_ROUTING_ENGINE=google # needs DISPATCH_GOOGLE_MAPS_KEY, Routes API enabled DISPATCH_ROUTING_ENGINE=osrm # needs DISPATCH_OSRM_BASE_URL, self-hosted, free ``` Then `php artisan config:clear` and `php artisan dispatch:routing-check`. That command fails if the configured engine fell back to straight lines, which is the failure that otherwise looks exactly like success. **Google needs exactly one product: the Routes API.** Two endpoints (`:computeRouteMatrix` and `:computeRoutes`). Not Geocoding — orders arrive with coordinates. Not Places, not Roads, not the legacy Directions or Distance Matrix APIs. Restrict the key to the Routes API and to the server's address; it is a server-side key that never reaches a browser. ### Quota guards Nothing above the routing layer throws, so a quota exhaustion looks like a slow afternoon rather than an incident. The things that make it visible: - `routing_fallback_rate` on the KPI screen. A rate climbing off zero is the first sign. - The `dispatch` log channel records every failed call with its mode and elapsed time. - `dispatch:routing-check` turns "is it working" into one command with an exit code, so it can go in a deploy script or a cron. If the quota does run out, the system keeps dispatching on straight-line estimates and says so on every list. That is the designed behaviour, not a degradation to be fixed in a hurry — but the ranking is worse while it lasts, so it is worth an alert rather than a dashboard nobody reads. ### The ETA speed, and why it must be measured `DISPATCH_ETA_ESTIMATE_SPEED_KMH` is the speed every ETA falls back to when there is no route plan and no routing answer. It is not a display setting: it decides **which captain wins an order** (adjusted ETA), **whether a batch is refused** for exceeding the detour limit, and **what time the customer is told**. ```powershell php artisan dispatch:measure-speed # last 30 days of GPS history php artisan dispatch:measure-speed --days=7 # a single week ``` It reports and never writes. To adopt the figure, set it and clear the config cache: ```powershell # .env DISPATCH_ETA_ESTIMATE_SPEED_KMH= php artisan config:clear ``` **It is a straight-line speed, not a road speed.** Every caller divides a haversine distance by it, so it has to already absorb bending roads, junctions and one-ways — it is always lower than what a captain's speedometer reads. Do **not** set it from an average of the `speed_mps` the phones report: that is a road speed, it runs about a third high, and every ETA in the system would come out optimistic. The command measures displacement over elapsed time, which is the right thing. What it throws away, and what each would have done to the figure: windows under `speed_sample_window_s` (a traffic light would dominate), gaps over `speed_sample_max_gap_s` (a closed app would read as a crawl), displacement under `speed_sample_min_meters` (parked at a store is real time but not travel), speeds over `speed_sample_max_kmh` (a GPS jump, not a car). It reports the **median**; the mean is shown beside it only so you can see the difference. Under 200 samples it says the sample is thin and recommends nothing. **As of 2026-09-20 it has never been run against real driving, so the 30 km/h default is still a placeholder** — read Epic 8's ETA-accuracy KPIs with that in mind until it has a real answer. ### Testing against a running server Every other suite drives Laravel's test kernel: fast, transactional, in-process. It covers the application and stops where the application stops — it never sees a web server parse a request, a real `.env` resolve, the real Redis database, the real queue connection, or the middleware stack as PHP assembles it outside a harness. `LiveServerE2ETest` talks to a real server over real HTTP and asserts only what comes back on the wire. ```powershell php artisan serve --port=8001 php artisan queue:work # the route plan is a queued job $env:E2E_BASE_URL = 'http://127.0.0.1:8001' $env:E2E_ADMIN_EMAIL = 'super-admin@kapitano-logiistic.com' $env:E2E_ADMIN_PASSWORD = '' php artisan test --filter=LiveServerE2ETest ``` **It is opt-in and skips with the reason printed when `E2E_BASE_URL` is unset**, because it is not hermetic: it writes to whatever database that server is pointed at. Point it at a machine you are happy to seed. What it covers: the header and token contract, the CORS preflight from the dashboard origin, eligibility (a captain at capacity, one with stale GPS, one on a break and one offline must not be offered), batching, ranking by adjusted ETA rather than distance, assignment through to the captain reading their own route, a GPS ping, a 409 for a captain at capacity, and the KPI screen. **Three things worth knowing before it confuses you:** **A test process cannot reach the server's database.** `phpunit.xml` forces `DB_CONNECTION=sqlite`, `DB_DATABASE=:memory:`, `REDIS_DB=15` and a null broadcaster onto this process, and a subprocess inherits all of it — so `db:seed` from a test cheerfully seeds an in-memory database and reports success while the server sees nothing. The suite clears those variables for every subprocess it runs, which is what makes them behave like an operator's own shell. **The captains have to be put on the live map, and the suite does it through the API.** The seeder deliberately does not write Redis: a captain's live position is filled by GPS pings, and faking it would skip the freshness and ordering rules. So each scenario captain's seeded position **and its age** are replayed through `POST /api/driver/location`. The age is the point — C5's fix is minutes old, and a stale captain must drop out of the list. **Two things cannot go over HTTP, and the suite says so rather than pretending.** The scenario is seeded by an artisan subprocess, and the captain's token is minted the same way, because captain sign-in is an SMS code and the code is stored hashed — a harness can neither receive it nor read it back. Everything *asserted* still goes over the wire. Preparation runs once per process rather than per test. Doing it per test took eight minutes to say what it now says in two. ### Showing the algorithm without a Google key ```powershell php artisan dispatch:demo --fresh ``` Runs the whole pipeline on `FakeRoutingEngine` and prints what it decided: the ranked list with the arithmetic, which captain was nearest and which one won, the route, a second order joining it, and the first customer's delay against the batch limit. Needs Redis; needs no network. Use it to demonstrate the system, and as a smoke test after changing anything in dispatch. Full walkthrough in `scripts/demo.md`. Always pass `--fresh` unless you mean to continue a previous run. It re-seeds the worked example **and clears the replan locks** — `RecomputeRoutePlan` is unique per captain for two minutes, and a lock left by an earlier run makes the next one assign the order and silently produce no route. ### Readiness checks Three commands answer "is this environment actually wired up", for the three pieces the test suite necessarily stops short of. Run them after a deploy, after changing `.env`, and on a fresh machine. ```powershell php artisan dispatch:redis-check # PING, GEOADD, GEOSEARCH, SET NX EX php artisan dispatch:broadcast-check # the websocket server accepts a broadcast php artisan dispatch:routing-check # the map service answers a matrix and a route ``` **`dispatch:broadcast-check`** publishes one frame on a channel nothing subscribes to. It catches the failure that is silent in the worst direction: with a mismatched `REVERB_APP_KEY` nothing errors, no request fails, nothing appears in the log, and captains simply stop being told their route changed — the first anyone hears of it is a dispatcher asking why the map is stale. A `null` or `log` driver is warned about rather than failed, because `log` is a reasonable choice on a machine with no websocket server. If it fails, check in this order: the server is running (`php artisan reverb:start`), `REVERB_HOST`/`REVERB_PORT`, the three `REVERB_APP_*` credentials, then `php artisan config:clear`. **`dispatch:routing-check`** asks for a matrix and a route between two real Riyadh points and prints the minutes and kilometres. Nothing above the routing layer throws, so an engine that is down looks exactly like one that is working with worse numbers — a configured engine that fell back to straight lines therefore **fails** the command. With `DISPATCH_ROUTING_ENGINE=fake` it says plainly that this is the simulation and not a map, so a green check is never mistaken for "Google is wired up". ### What the tests reach, and what they do not Worth knowing before trusting a green suite: | Layer | Covered by tests | How | |---|---|---| | HTTP, guards, permissions, middleware | yes | the E2E suite drives real requests | | Route plan build, versioning, recompute | yes | unit + feature + E2E | | The broadcast payload and channels | yes | a recording broadcast **driver**, not `Event::fake()` | | Channel authorisation and its signature | yes | the real Reverb driver over `/api/broadcasting/auth` | | The FCM message and its data types | yes | a `Messaging` double behind the real channel | | **The websocket socket itself** | no | `dispatch:broadcast-check` | | **A real map service** | no | `dispatch:routing-check`, and `RUN_REAL_MAPS=1` for the Google suite | `Event::fake()` is deliberately **not** used in the E2E suite. It stops an event before `ShouldBroadcast` is consulted, so `broadcastOn()`, `broadcastAs()` and `broadcastWith()` never run — a mistake in any of them would pass every test while no phone in the field repaints. ### Dispatch KPIs and the metrics log `GET /api/dashboard/dispatch/kpis` (permission `dispatch.view`) reports how the algorithm is performing. `?from=YYYY-MM-DD&to=YYYY-MM-DD`; both default to the last seven days, and a window reaching further back than the counters' retention is **refused** rather than answered with the part that still exists. The numbers come from two places, and the split matters when one of them looks wrong: | Source | Answers | Where it lives | |---|---|---| | Redis daily counters | *how many* — every rate and volume | `dispatch:metrics:{date}:{metric}`, 45-day TTL | | The Suggestion Log | *how long* — the two latency percentiles | `order_suggestions`, in the database | A counter is cheap enough to increment on the request path; a latency percentile needs the individual measurements, which a counter has thrown away. Neither can do the other's job. **A rate with nothing in its denominator comes back `null`, not `0`.** A quiet Sunday has no batch rate. If a screen shows `0 %` there, the screen is wrong, not the API. Everything also lands in `storage/logs/dispatch-YYYY-MM-DD.log`, one JSON object per line, kept 14 days. That is the file to grep when **one** delivery went wrong — it carries the order, the captain, the suggestion and the timings that a counter cannot: ```powershell Select-String -Path storage\logs\dispatch-*.log -Pattern '"order_id":1234' ``` The nine metrics: `suggestion.generated`, `suggestion.chosen`, `suggestion.chosen_first`, `assignment.made`, `assignment.batched`, `assignment.failed`, `routing.answered`, `routing.fallback`, `routing.elements`, `routeplan.recomputed`, `delivery.completed`. **Reading them honestly.** `routing.elements` is what the map service bills for and is the only one measured in something other than events. Cost is reported **per delivery** rather than per order, because two orders on one route cost barely more to plan than one. `DISPATCH_MAPS_UNIT_PRICE` turns elements into money and is a placeholder until a real contract says otherwise — it changes the reported cost and nothing else; no part of the algorithm reads it. **If a counter reads zero and you expect otherwise**, check Redis first (`dispatch:redis-check`): the metrics layer reports a Redis failure and carries on rather than failing the assignment it was measuring, so a broken counter is silent by design. The log channel is written first for the same reason, so the lines will still be there. ### When a Composer command hangs `composer require` can reach *"Loading composer repositories with package information"* and sit there at 0% CPU indefinitely. It is almost always a **poisoned cache**, not the network: ```powershell composer clear-cache ``` Before blaming connectivity, check it: `composer diagnose` and `composer show --all` both answer in seconds when the network is fine, and they did while an install was hanging for half an hour. Two other traps on this machine — a background job started in a PowerShell session dies when that session exits, so long installs need `Start-Process`; and `Set-Content -Encoding utf8` writes a BOM, which makes `composer.json` invalid JSON. ### Scheduler (GPS history) `dispatch:flush-gps` writes the GPS points waiting in Redis (`captains:gps_history`) to `driver_location_history` every 30 seconds; `dispatch:flush-gps --prune` also deletes history older than `gps_history_retention_days` (30; 0 keeps everything) every day at 03:00. Both are defined in `routes/console.php`. - Locally, keep a scheduler running in its own terminal: `php artisan schedule:work`. - On a server, one cron line runs everything, the 30-second task included: `* * * * * php /path/to/artisan schedule:run >> /dev/null 2>&1`. - Without a scheduler the points only wait in Redis. The local Redis keeps nothing on disk, so points not yet flushed are lost when it restarts. - To flush by hand: `php artisan dispatch:flush-gps` (it prints how many points it wrote). If the insert fails the points are put back in Redis and the command exits with an error. ### Push notifications (Firebase) `FIREBASE_CREDENTIALS` points at the service account JSON, by default `storage/app/firebase/captain-app-43cfa.json`. **That file is a secret**: `storage/app/` is gitignored and it must never be committed. A relative path is resolved against the project root, so the same value works on a server. Run `php artisan config:clear` after changing it. ```powershell php artisan firebase:test # credentials + a simulated token; delivers nothing php artisan firebase:test --token= # dry run against a real device php artisan firebase:test --token= --deliver # the phone actually buzzes ``` Without `--token` the command uses a fake one. Firebase has to authenticate the service account before it can look at the token, so "service account accepted, token invalid" is the expected pass — it proves the credentials without touching anyone's phone. It keeps three outcomes apart, and only the last one is a pass: | Outcome | Exit | Meaning | |---|---|---| | Firebase could not be reached | 1 | Network, proxy or missing CA bundle. **Nothing is proven about the credentials.** | | Firebase refused the service account | 1 | Wrong project, revoked key, or a skewed server clock (`invalid_grant`). | | Service account accepted | 0 | The credentials work; a rejected simulated token is expected here. | Those three look alike in the SDK and are easy to collapse into a false "all good" — a refused service account and an invalid token both come back as `InvalidMessage`, and only the report's own target flag and the OAuth wording separate them. To check that the check still works, break it on purpose: ```powershell $env:FIREBASE_HTTP_CLIENT_PROXY='http://127.0.0.1:9'; php artisan firebase:test # must fail: unreachable Remove-Item Env:\FIREBASE_HTTP_CLIENT_PROXY # copy the JSON, change client_email to one that does not exist, then: $env:FIREBASE_CREDENTIALS=''; php artisan firebase:test # must fail: invalid_grant Remove-Item Env:\FIREBASE_CREDENTIALS ``` Delete that copy afterwards — it still carries the real private key. **XAMPP's PHP has no CA bundle**, and without one every Firebase call dies with `unable to get local issuer certificate`. `C:\xampp\php\php.ini` now sets both of these; a fresh machine needs the same two lines, and Apache has to be restarted after adding them: ```ini curl.cainfo = "C:\xampp\apache\bin\curl-ca-bundle.crt" openssl.cafile = "C:\xampp\apache\bin\curl-ca-bundle.crt" ``` Dead tokens prune themselves: when Firebase reports one as unknown (app uninstalled, data cleared, token rotated) or malformed, `DeleteExpiredNotificationTokens` deletes that row. A quota error or an FCM outage never deletes — the device is still out there holding the token. **One notification per captain, never two.** Four things hold that line: - `fcm_token` is unique across the whole `device_tokens` table. A token identifies a handset, not a person, so registering one that another captain still holds **moves** it to whoever registered last — the captain actually carrying that phone. Signing out does not release it (`logout()` only revokes Sanctum tokens), which is why the transfer happens on registration. - A captain with two devices is pushed once per device, in a single batch. The unique index makes a repeated token impossible, so one device is never addressed twice in one send. - `FIREBASE_HTTP_CLIENT_TIMEOUT` (30s) stays below the queue's `retry_after` (90s). kreait sets no timeout of its own, so without this a hung request could outlive the retry window and be picked up by a second worker while the first was still sending. Observed calls take 2–5s; 30s leaves room for a slow network without ever approaching 90s. - Assignment is guarded by the state machine inside a transaction (`pending → assigned` is the only way in), so a double-clicked "assign" is refused rather than dispatching `OrderAssigned` a second time. Listeners are registered once. Check with `php artisan event:list`: anything both discovered and registered by hand in a service provider runs twice for every event. Delivery needs a **running queue worker**, because the notifications are sent from queued listeners. Run it with `--tries=1`: ```powershell php artisan queue:work --tries=1 ``` FCM has no idempotency key, so a message that reached Google but failed afterwards would be delivered a second time by a retry. With `--tries=1` the job goes to `failed_jobs` instead, where it can be inspected — a missing notification is recoverable, a duplicated one is not. ### Managing push from the back office Every push writes one row per device to `push_deliveries`: who it was for, what they were shown, and Firebase's answer (`sent` with Google's message id, `failed` with the error, or `no_device` when the captain had no registered handset). The back office reads it under `/api/dashboard/push` (OpenAPI tag *Dashboard — Push Notifications*): | Endpoint | Permission | Use | |---|---|---| | `GET deliveries` | `push.view` | "Did captain X get it?" — filter by `driver_uuid`, `broadcast_uuid`, `status`, `notification`, dates | | `POST deliveries/{uuid}/resend` | `push.send` | Send a logged push again to the recipient's *current* devices | | `GET stats` | `push.view` | Counts per outcome, failure rate, and the thresholds in force | | `POST test` | `push.send` | Push a real test notification to one captain and report Firebase's answer | | `GET devices` | `push.view` | Which captains can be reached, and in which language | | `DELETE devices/{uuid}` | `push.manage` | Remove a lost or handed-over phone | | `GET broadcasts`, `GET broadcasts/{uuid}` | `push.view` | Broadcast history and one broadcast's progress | | `POST broadcasts` | `push.broadcast` | Message all approved captains, those on duty, or chosen ones | | `GET failed-jobs`, `POST …/retry`, `DELETE …` | `push.manage` | The push jobs that ran out of attempts | `operations-manager` holds all four permissions, `support` can only view. Sending to a phone is rate limited: 30 tests/resends and 5 broadcasts a minute per admin. A test push and a resend report what Firebase answered, read back from the log — never "sent" just because a device exists. **Broadcasts** are never sent inside the request. They are resolved to their captains (approved and active only), cut into chunks of `PUSH_BROADCAST_CHUNK_SIZE` (50) and dispatched as one job batch; the batch's completion marks the broadcast completed. Each chunk job is attempted once and times out at 60 s — far inside the queue's 90 s `retry_after`, so no chunk is ever picked up twice. The same admin sending the same text to the same audience within 120 s gets 409 instead of a second fleet-wide push. Captains without a device are skipped, so `target_count` can be larger than the devices reached. **Language**: a device stores the `Accept-Language` its app registered with, and `Driver` implements `HasLocalePreference`, so every push — built on a queue worker whose locale is the default — is translated into the captain's language. **Settings** (retention, health thresholds, chunk size, icon) are deploy configuration in `config/push.php` / `.env`, not dashboard settings: they are reviewed and deployed like code. `stats` shows the thresholds so admins can see what "unhealthy" means. Two scheduled commands keep it useful (`routes/console.php`): - `push:prune` (daily 03:30) deletes rows older than `PUSH_RETENTION_DAYS` (30; 0 keeps all). - `push:health` (hourly) exits non-zero, logs a warning, and notifies every admin's inbox (group *System*, at most once per `PUSH_HEALTH_ALERT_COOLDOWN_DAYS`) when at least `PUSH_HEALTH_MINIMUM_FAILURES` (5) pushes failed and they are at least `PUSH_HEALTH_FAILURE_PERCENT` (25%) of the last `PUSH_HEALTH_WINDOW_HOURS` (24). When a captain reports a missing notification: check `devices` (no row → the app never registered), then `deliveries` for that captain (`failed` → read `error`; `no_device` → same as no row), `POST test` to prove the handset now, and `resend` the notification they missed. #### Production checklist Server side — all required, none optional: - [ ] `php artisan migrate`, then `db:seed --class=PermissionSeeder` and `--class=RoleSeeder`. - [ ] A supervised worker: `php artisan queue:work --tries=1` under Supervisor/systemd, restarted on failure. Broadcasts and every order notification go through it. - [ ] `php artisan queue:restart` on **every deploy** — a running worker keeps the old code in memory and would run new jobs against it. - [ ] The scheduler cron line, for `push:prune` and `push:health`. - [ ] `APP_URL` set to the public URL, and `PUSH_NOTIFICATION_ICON_URL` to a public image (or left empty for no image). - [ ] Rotate the service account key if it was ever shared outside the secrets store, and keep the JSON outside the web root with read access for the PHP user only. Outside this repository — push silently never appears without them: - [ ] The Android app creates the notification channel `high_importance_channel` (`BaseFcmNotification::ANDROID_CHANNEL`) at start. Android 8+ drops a push naming a channel the app never created. - [ ] An APNs authentication key is uploaded in the Firebase console (Project settings → Cloud Messaging) for iOS delivery. - [ ] The captain app sends `Accept-Language` when registering its token, and re-registers when the user changes language. - [ ] One real handset proven end to end: `php artisan firebase:test --token= --deliver`. --- ### Store integration (Feature 08) Orders arrive from the store over `POST /api/integration/v1/orders` and every lifecycle move is sent back as a signed webhook. Two things have to be running for that second half to work at all. **A queue worker is not optional.** Webhooks are queued; without a worker the events sit in the jobs table and the store is told nothing, while the orders themselves flow normally — so the failure is invisible from our side and total from theirs. **And it is monitored, so that failure is no longer invisible.** `integration:health` runs every fifteen minutes from the scheduler and raises a dashboard notification (deduplicated to one a day, so an afternoon-long outage is one alert rather than thirty): ```powershell php artisan integration:health # exit 0 healthy, 1 when something is wrong php artisan integration:health --notify # also writes the dashboard notification ``` It separates three silences because they need different actions: | Row | Means | Do | |---|---|---| | **Stalled > 15m** | deliveries sitting `pending` nobody picked up | start the queue worker | | **Overdue retries** | a retry was due and did not run | same — the worker is not turning | | **Dropped** | every retry exhausted | the URL or secret is wrong, or the store is down; replay from the dashboard once fixed | The exit code is what a cron or an uptime check should watch; a fresh delivery is deliberately **not** an alarm, because a brief backlog is normal and an alert that cries wolf gets muted. ```powershell php artisan queue:work --tries=1 ``` **The scheduler** runs `integration:prune` nightly at 03:40. Both integration tables keep raw payloads — that is what makes a dispute answerable — so without it they grow without bound. #### Onboarding a store ```powershell php artisan integration:issue-token kapitano-marketplace --create --name="Kapitano Marketplace" ``` It prints the bearer token **once** (Sanctum stores a hash) alongside both signing secrets. Hand them over on a secure channel, then set the callback URL on the row. `--rotate-secrets` replaces both secrets; traffic signed with the old ones is refused from that moment, so agree a window first. #### Driving the whole thing locally Four terminals. Nothing here is a mock except the store itself. ```powershell php artisan serve --host=127.0.0.1 --port=8123 # 1. the application php artisan queue:work --tries=1 # 2. the worker that sends webhooks php artisan integration:webhook-sink kapitano-marketplace --port=8129 # 3. the store's receiver php artisan integration:simulate-store kapitano-marketplace --token= --retry # 4. the push ``` Set the client's `webhook_url` to `http://127.0.0.1:8129` first. The sink prints each event with its sequence and whether the signature verified — a `*** REJECTED ***` line means the two sides disagree about the secret or the body, which is the single most common integration fault. `--retry` sends the same payload twice under one `Idempotency-Key`: the second call must answer **200** with `Idempotency-Replayed: true` and create nothing. #### When the store says it never heard about an order ```powershell # What did we send, and what came back? GET /api/dashboard/integration/webhooks?rows=25&page=1&order_uuid= # Everything currently failing GET /api/dashboard/integration/webhooks?rows=25&page=1&status=dropped # Send one again, from a clean slate POST /api/dashboard/integration/webhooks//replay ``` `failed` means more attempts are coming; **`dropped` means we have stopped** — six attempts over about two and a half hours are exhausted and nothing further will happen without a replay. That distinction is the whole reason the two statuses are separate. #### Turning intake off without cutting anybody off `INTEGRATION_ACCEPT_ORDERS=false` makes intake answer **503** while leaving every token valid, so a bad release can be isolated and the store's own retries pick the orders back up afterwards. To pause one client rather than all of them, set `store_clients.is_active = false` — their credential survives, so onboarding does not restart. #### `TRUSTED_PROXIES` is a security setting, not a convenience Behind nginx, `$request->ip()` is the proxy's address unless the proxy is trusted. Two things break quietly: the store's IP allowlist passes **everything**, and every IP-keyed rate limiter — including the OTP and admin sign-in ones that predate this feature — collapses into one shared bucket. Set it on any deployment that sits behind a proxy. ### Store portal (Feature 09) The shop's own staff, as opposed to its servers: `/api/store-portal`, guard `store_user`. Plan and decisions in `docs/feature-09-store-portal.md`. #### Onboarding a store, end to end 1. The store applies at `POST /api/store-portal/auth/register`. That creates an **application** — `store_clients.status = pending`, `is_active = false`, **no API token**. Nothing can flow. 2. The owner can sign in straight away and reach `/settings` only. This is deliberate: they set their callback URL and run `POST /settings/webhook/test`, which sends a real signed `integration.ping` through the real queue and the real ledger, so their receiver is proved before the approval rather than after it. 3. An admin with `store_clients.review` approves at `POST /api/dashboard/store-clients/{uuid}/approve`. That switches the client on and moves every `pending` person at it to `active`. 4. Approval does **not** mint the API token. Issue it separately: `php artisan integration:issue-token `. #### The three questions support gets asked | "…" | Where to look | |---|---| | "Where has our application got to?" | `GET /api/dashboard/store-clients?status=pending` — needs `store_clients.view` | | "Did you ever tell us the order was delivered?" | The store can answer this themselves now: `GET /api/store-portal/webhooks`. Our side is `/api/dashboard/integration/webhooks` | | "What are we owed this month?" | `GET /api/store-portal/payments/statement`. Read the assumptions it returns before quoting a figure — see below | #### The statement's one soft spot There is no `amount_collected` column anywhere in this system. The statement assumes a delivered cash order yielded exactly its `amount_to_collect`, which is true unless a captain took a different amount at the door — something nothing currently records. The API returns that caveat in `assumptions` on every response and the screen must show it. **Do not quote a cash figure in a dispute without saying so**; recording the real amount is a captain-app change that has not been made. #### Key rotation is self-service now, and it is safe A store owner can rotate both secrets from `/settings/secrets/rotate`. For `INTEGRATION_SECRET_OVERLAP_HOURS` (default 24) afterwards we accept **either** secret inbound, and every outbound callback carries **two `v1` values** in one signature header. A store whose verifier reads only the first `v1` will start rejecting our callbacks the moment somebody rotates — that requirement is written into `docs/store-api-ar.md` §8.4 and is the one thing this feature asks of their side. #### Two settings that can lock somebody out - **`allowed_ips`** (owner-editable) gates the **machine** API, not the portal. A store that types the wrong address locks its own servers out but can still sign in and fix it. - **Suspension** (`POST /api/dashboard/store-clients/{uuid}/suspend`, needs `store_clients.manage`) revokes every portal token immediately and takes effect on the next request — it does not wait for a token to expire, because Sanctum tokens do not. #### A callback URL is refused unless it is public `StoreWebhookUrlRule` demands https, a public host and the standard port, and rejects credentials in the URL. This is not fussiness: we make the request from inside our own network, so `https://169.254.169.254/...` would turn a settings field into a reader for the server's cloud credentials. `INTEGRATION_ALLOW_PRIVATE_WEBHOOK_URLS=true` switches the range check off and belongs **only** on a developer machine running the `--sink` harness. The rule cannot resolve host names, so a domain pointing at a private address would pass it. That half is covered separately by `WebhookUrlGuard`, which runs **immediately before each callback is sent**: it resolves the host and refuses if *any* address it resolves to is private, loopback or link-local. A refusal is written into the delivery ledger as an ordinary failure, so the store sees why nothing arrived. Still not airtight — the answer can change between that check and the socket opening, and only pinning the resolved address into the connection closes it. It turns a trivially exploitable hole into a race. #### Settlements The statement is derived from the orders and can only ever agree with itself. The settlements are what actually changed hands, and the pair is what makes a reconciliation possible. ``` GET /api/dashboard/store-clients/{uuid}/settlements/preview?from=&to= # total the period up POST /api/dashboard/store-clients/{uuid}/settlements # draw up a draft POST /api/dashboard/settlements/{uuid}/settle # mark it paid ``` Three things to know before touching one: - **Preview first.** The figures reach a human before they become a payment, instead of being retyped off another screen. The amounts you then post are *yours* — they are allowed to differ from the computed ones (an adjustment, a rounding, a dispute settled halfway) and are recorded as sent. - **A draft is invisible to the store.** They see a settlement once it is settled. - **Settled is final.** The figures are frozen and the row cannot be edited or re-settled. Correct a mistake with *another* settlement, never by rewriting one both sides are holding. `settlements.view` reads the ledger; `settlements.manage` moves money in it. They are separate on purpose. #### Staff at a store An owner adds colleagues at `POST /api/store-portal/staff`. **No password is set by anybody** — the new account has one nobody knows, and the invitee's first act is to ask for a recovery code on the sign-in screen, which goes to the phone number the owner typed. So: - get the phone number right, or they cannot get in; - an owner never holds a colleague's credentials; - the role is always `staff`. An owner cannot mint another owner, which is how a removed employee would otherwise keep their access. Turning somebody off revokes their sessions at once. An owner cannot turn off their own account. --- ## 2. Checks to run before pushing The same checks CI runs, in the same order: ```powershell vendor/bin/pint.bat --test # formatting; drop --test to fix php scripts/check-architecture.php # repository/service rules + dispatch-settings rule php artisan test --compact # Unit, Feature and E2E suites ``` - `php artisan test --testsuite=E2E` runs only the full-system dispatch scenarios (`tests/Feature/Dispatch/E2E`); the Feature suite excludes that folder. Until task T9.1 adds the first scenario, that command alone answers "No tests found" (exit 1); the full run and CI are not affected, because they run every suite together. - Tests that touch the dispatch location layer use Redis **database 15** (`REDIS_DB` in `phpunit.xml`) and flush it before each test; they skip, with the reason, when Redis is down. Never point `REDIS_DB` for tests at a database holding real data. - Run **one** test run at a time on this machine: a full suite started while another PHP test run was going hung twice, while the suite alone finishes in about 70 s. If a run seems stuck, stop the `php.exe` processes running `artisan test` / `phpunit` and start it again alone. - Coverage needs a coverage driver (PCOV or Xdebug), which the local XAMPP PHP does not have; CI measures it. With a driver installed: ```powershell php vendor/bin/phpunit --coverage-clover build/coverage/clover.xml php scripts/check-coverage.php build/coverage/clover.xml app/Services/Dispatch 80 ``` --- ## 3. Continuous integration (GitLab) Defined in `.gitlab-ci.yml`. Pipelines run for merge requests, and for branch pushes that have no open merge request. | Stage | Job | What fails it | |---|---|---| | lint | `lint` | any file Pint would change; any architecture-check **error** (warnings pass) | | test | `test` | Redis not answering `dispatch:redis-check`; any failing test; line coverage of `app/Services/Dispatch` below `DISPATCH_COVERAGE_MINIMUM` (80) | - **Image:** `php:8.4-cli` plus `exif gd intl pcov pdo_mysql zip` (installed with `install-php-extensions`), Composer dependencies cached on `composer.lock`. - **Services:** `mysql:8.0` (alias `mysql`, database `kapitano_test`) for the parallel-assignment test, `redis:7-alpine` (alias `redis`). - **Databases in tests:** every test runs on in-memory SQLite (`phpunit.xml`) except the concurrency test, which uses the `mysql_concurrency` connection (`DB_CONCURRENCY_*`; local defaults in `phpunit.xml`, CI values in the job variables). - **Routing:** `DISPATCH_ROUTING_ENGINE=fake`; the Google engine is exercised only against recorded fixtures, never the real API. - **Reports:** the JUnit report shows per-test results on the merge request; the Clover file is kept as an artifact for 14 days; the job's coverage figure is the dispatch module's. To block merges on a red pipeline: GitLab → Settings → Merge requests → *Pipelines must succeed*. ### When a job fails | Symptom | Likely cause | |---|---| | `lint` lists files | run `vendor/bin/pint.bat` locally and commit the result | | `dispatch-settings` error | a number from `config/dispatch.php` written in `app/Services/Dispatch` or `app/Repositories/Dispatch`; read it through `DispatchSettings` | | `dispatch:redis-check` fails in `test` | the Redis service did not start; retry the job | | coverage below the minimum | new dispatch code without tests; the job log prints the measured figure | --- ## 4. Architecture notes The dispatch code follows the `create-laravel-feature` skill (repository → service → thin controller, DTOs, enums, ar/en translations, `scripts/check-architecture.php`). Where it cannot, the exception is deliberate and recorded here: | Where | Exception | Why | |---|---|---| | `app/Repositories/Dispatch/{RedisGateway,GeoIndex,CaptainGpsStore}` | data-access classes that do not extend `BaseRepository` and have no interface binding | Redis has no Eloquent model behind it. They still live with the repositories, so no service talks to Redis directly; their names avoid the `Repository` suffix, which the checker reserves for `BaseRepository` subclasses. | | `DispatchSettingRepository::overrides()` | `Cache::remember` on one key instead of `Cacheable` + `CacheableRepository` | the tagged contract needs a store that supports tags; the `file` and `database` stores used locally do not, and one key needs no tag to be forgotten. | | `DispatchScenarioSeeder` | writes through models, not repositories | the scenario needs `forceFill` on the capacity counter and `updateOrCreate` keyed on (order, sequence), which no repository exposes; `OrderSeeder` does the same. | | `App\Listeners\Dispatch\RefreshCaptainEffectiveLocation` | a listener that does not implement `ShouldQueue` | it writes one Redis point; the next suggestion list may be built a moment later and must see the captain's real state, and a queue worker is not always running. Failures are reported, never thrown, so an order's move never fails because of it. | | Dispatch features | written by hand, not generated with `scaffold.php` | the scaffolder produces a CRUD feature; these are algorithm services. The checker and Pint enforce the same layout. | | Store integration (Feature 08) | written by hand, not generated with `scaffold.php` | same reason, confirmed by a dry run: it wanted a public `/api/integration-requests` CRUD resource with its own `Mobile/` controller, route file and translations. The idempotency ledger must never be served, and the endpoints belong to the existing `Integration` feature and its route group. | | `App\Http\Controllers\Integration\` | a third audience folder beside `Mobile/` and `Dashboard/` | the rule is that controllers are grouped by who calls them, not that there are only ever two callers. The store is neither the captain app nor the back office, and its routes carry their own guard. | | `App\Repositories\Integration\SignatureNonceStore` | a data-access class that does not extend `BaseRepository` and has no interface binding | the same arrangement as the dispatch Redis classes: there is no Eloquent model behind Redis. It uses the connection's own `set()` rather than `RedisGateway`, because only the GEO commands need to be sent raw. | | `App\Http\Middleware\Integration\EnsureStoreAbility` | duplicates Sanctum's `CheckAbilities` instead of using it | Sanctum throws `MissingAbilityException`, which extends `AuthorizationException`, and Laravel's `prepareException()` rewrites it into a plain `AccessDeniedHttpException` **before** any renderable callback runs — so no entry in `ApiExceptionHandler::$handlers` can reach it and the store would get an untyped 403 with no `missing_ability` key to branch on. | | The store API's middleware order | the guard runs **first**, then active / IP / signature | the signature is verified against that client's secret and the address against that client's allowlist, so both have to know who is calling. Verifying before the guard would need a key id in the header, which this contract does not have. | | `OrderRepository::nextSequentialNumber()` | reads a locked counter row rather than the highest existing number | two callers arriving together read the same highest number and the loser hits the unique index, which becomes a 500 the store is entitled to retry — and therefore a duplicate delivery. Proved by mutation: with the old read restored, five of eight parallel processes fail with `1062`. | | `orders.source` / `orders.store_client_id` | not fillable; written by `OrderRepository::createWithProvenance()` | they describe the caller, never the payload, so nothing arriving over HTTP may set them — the same discipline `drivers.active_orders` has. | | `App\Services\Dispatch\Routing\RoutingEngine` | the interface lands in T3.2, ahead of the implementations it belongs with in T4.1 | T3.2 has to prove a negative — the geo filter never calls routing — and a spy needs something to implement. Contract only: the array shapes it declares are replaced by the typed `MatrixResult` / `RouteResult` when T4.1 builds `FakeRoutingEngine`. | | `DriverRepository::eligibleForDispatchAmong()` | answers with plain rows, not Eloquent models | hydrating 500 captains to read three columns off each of them measured at 67 ms of the geo filter's 100 ms budget, and nothing downstream calls a method on them. Every other read on this repository still returns models. Measured, not assumed: `CandidateFinderPerformanceTest` prints the cost of each phase on every run. | | `CaptainGpsStore::capturedAtTimestamps()` | reads many hashes with a Lua `EVAL` | there is no Redis command that reads one field from a list of hashes, and `RedisGateway` sends raw commands — which the GEO commands require — so it cannot pipeline. One script is one round trip; the alternative was 500. | | `fake_routing_speed_kmh`, `fake_routing_latency_ms` | dispatch settings that are **not** in `DispatchSettings::effective()` | they describe the simulation, not the business the algorithm runs on. `effective()` is what the dashboard settings screen shows, and a dispatcher should not be offered a number that stops meaning anything the day the Google keys arrive. | | `PolylineEncoder::PRECISION` | carries the checker's `@dispatch-literal` escape | five decimals is the polyline format's own precision. It collides with a `radius_steps_km` step by pure coincidence, and renaming the format's constant to satisfy a dispatch rule would be the tail wagging the dog. | | `RoutingFailedException` | a plain `RuntimeException`, not an `ApiException` | a maps outage is not the client's mistake. It must never reach the dispatcher as an error page; the caller catches it, ranks by straight-line distance and flags the list degraded. | | `RoutingEngineManager` | returns `RoutingUnavailable` instead of throwing | above the routing layer, no road time is a normal situation with a defined behaviour — rank by the straight-line estimate and mark the list degraded — and a situation with defined behaviour should not travel as an exception a caller might forget to catch. | | `RoutingEngineManager` | does **not** enforce the timeout it measures | PHP cannot interrupt a call that is already blocking. The deadline is enforced inside the engine (`Http::timeout` from `routing_timeout_ms`); the manager times the call and flags a late one (`over_budget`) without discarding the answer. | | `App\Events\Dispatch\RoutingCallCompleted` | was emitted with no listener from T4.3 until T5.5 | the metrics that consume it are task T8.1, which depends on work not yet done. Emitting from the day the calls existed means those metrics will measure history rather than starting at zero. Since T5.5 it also feeds `DispatchCallTally`, so it is no longer listener-less. | | `SuggestionCache` / `SuggestionCacheStore` | the task named one service; storage is split into a repository | cache access belongs to repositories here, as with `OtpRepository`. The payload is plain JSON because `cache.serializable_classes` is `false`, and cells are keyed by captain id so a reordered shortlist never receives another captain's road time. | | `App\Listeners\Dispatch\ForgetCaptainSuggestions` | a listener that does not implement `ShouldQueue`, and fires on every order move rather than only on assignment | the next list may be built a moment later and must not describe a captain who has since changed. A delivery turns a captain idle as surely as an assignment makes them busy. Redis failures are reported, never thrown. | | `RemainingEtaProvider` | an interface with one implementation, bound in `DispatchServiceProvider` | today every remaining delivery is estimated from distance; task T7.1 adds the implementation that reads the captain's route plan. The calculator never changes, and the rule that the remaining delivery never calls a routing engine is held by a spy test. | | `HandoffBuffer` | uses the payment method of the order the captain is **carrying**, not the new order | the buffer is time at the current customer's door, spent before the captain can leave for the new store. | | `Ranker` | measures the tie band from the fastest captain in a group, not from the previous captain | comparing neighbours chains: at a 2-minute band, 8, 10, 12 and 14 minutes would become one group and a captain six minutes slower would rank as tied with the fastest. Anchoring each group on its own fastest member keeps nearness from accumulating. | | `AtStoreStackingDetector` | reads orders with an uncollected pickup directly instead of working from the geo shortlist | a carrying captain sits on the map at their drop-off, which may be outside the pickup radius, so the shortlist can miss exactly the captain who will be at the store. "Left the store" is the pickup row's `picked_up_at`; being physically in the shop is not required. | | `BatchCompatibilityService` | prices every stop order from **one matrix** per busy captain instead of a route call per order | the task wrote one `getRoute` per permutation, which is up to 90 billed calls for one captain. The matrix is traffic-unaware; the chosen captain's real route is drawn once at assignment. The best insertion is the shortest total route — "least delay to the current customer" would always pick "finish first", and the detour rule could never reject. | | `DispatchCallTally` | a `scoped` service that learns the routing and cache counts from **events**, not from return values | `routing_elements` and `cache_hit` are facts about calls two and three layers below `SuggestionService`. Returning them would mean widening the signature of every method in between so the top could report numbers it never uses. The events already existed for T8.1; one per-request tally listens to them. | | `SuggestionService` | **measures** the phase budgets and never enforces them | a phase that overruns is recorded next to its budget in the Suggestion Log so it can be found afterwards. Abandoning it would hand the dispatcher half a list, which helps nobody; the answer to a slow phase is to fix it, not to truncate the screen. | | The suggestion endpoint vs. the assignment endpoint | between T5.5 and T6.1 the list offered busy captains while the assignment refused them | stacking an order needs the atomic capacity reservation T6.1 owns; without it two dispatchers could overfill one captain. **T6.1 closed this** — the two ends now agree and a busy captain can be given the order. | | `AssignmentService` | reserves capacity on the `drivers.active_orders` **counter**, while the suggestion list's eligibility counts **real orders** | the reservation happens before the order row is written, so a subquery over `orders` would still see the old count and let two dispatchers through — the counter is what makes the check atomic. The two agree while every assignment goes through this service and every ending through `OrderService::transition()`. **Known risk:** an order feed that writes in-progress orders directly would drift them, and a counter drifting high silently makes a captain un-assignable. A reconciliation command is proposed but not yet decided. | | `AssignmentService` | re-checks eligibility **inside** the reserving `UPDATE` rather than before it | anything checked beforehand is a fact about the past. A captain who goes offline while the dispatcher reads the list simply fails to match the statement, which is the only check that cannot be raced. | | `AssignmentLock` | guards an assignment that is already safe without it | the conditional `UPDATE` is what prevents over-assignment; the lock exists so the dispatcher who loses a race reads "somebody else is confirming this captain" rather than "full", which is a different and more useful thing to know. Released in a `finally`, with the TTL only as a backstop for a process that dies. | | `AssignmentLockStore::release()` | compares the token in a Lua script instead of a plain `DEL` | a lock that expired and was re-taken by another dispatcher must not be deleted by the one who timed out, or both end up inside the same captain's assignment. The read and the delete have to be one step. | | `App\Exceptions\ApiExceptionHandler` | matches an exception by exact class **and then** by `instanceof` | the exact-match-only version turned every `ApiException` subclass into a 500, which hid T6.1's deliberate 409. Exact matches still take priority, so a subclass can still claim its own handler. | | `ParallelAssignmentTest` | races the capacity reservation **with the lock deliberately left out**, in a second test beside the endpoint one | with the lock in play, most children are refused as `locked` before the database is ever asked, so the test measures the lock and not the statement under it. The first version passed with `WHERE active_orders < :max` deleted; splitting the two is what makes the conditional `UPDATE` actually tested. Verified by mutation: broken, both fail 3 runs out of 3. | | `ParallelAssignmentTest` | the endpoint test **retries** a child refused as `locked` | without it the number of winners is whatever the scheduler decided, and an assertion on an exact count is flaky. Retrying on `locked` is also what the API documents a client should do, so the test exercises the documented behaviour rather than working around it. | | `App\Listeners\Dispatch\BuildRoutePlanOnAssignment` | builds the plan **after** the assignment commits, not inside it, and does not implement `ShouldQueue` | building a plan calls a routing engine, and the assignment holds a Redis lock on the captain: a slow maps call inside that window would make other dispatchers see `locked` for as long as Google took. Not queued yet because a worker is not always running and a captain holding an order with no route is worse than a short wait — T7.2 moves it into a job serialised per captain. Failures are logged, never thrown: the order is already assigned and that is not in question. | | `RoutePlanBuilder` | prices candidate sequences from **one matrix** and draws **one route**, rather than asking the engine to draw each candidate | the same trap `BatchCompatibilityService` avoids: drawing every permutation is a billed call per permutation. Sequencing is arithmetic — precedence plus added-up legs — and only the winner is worth a route. A captain with one single-pickup order has one legal sequence, so no matrix is asked for at all. | | `RecomputeRoutePlan` | keeps its `ShouldBeUnique` lock in the **Redis** cache store via `uniqueVia()`, not the default store | the lock is what makes a burst of triggers coalesce into one plan, and it only works if every worker shares it. The file and database stores would serialise workers on one machine and nothing across several. `phpunit.xml` forces `REDIS_CACHE_DB` to the test database so a test run never writes locks into a developer's own cache. | | `RoutePlanService::guardCustomerPromises()` | **logs** a breach of `max_batch_detour_min` instead of throwing, though the task list said "throw + alert" | the guard should be unreachable: T5.2 refuses such a batch before the order is assigned, so a breach means two of our own services disagree. Throwing would withhold the route and leave the captain driving one that does not include the order already in their hands — punishing the captain and the new customer for our bug. The plan is published and the disagreement is made loud. | | `OrderService::markPickedUp()` | stamps **all** of an order's pickups as collected | `order_pickups.picked_up_at` previously had no writer anywhere in `app/`, so a route could never shrink and the builder would send a captain back to a store they had emptied. Until the captain app can confirm one store at a time, an order picked up means all its stores were. | | The route-plan reorder endpoint | identifies stops by **name** (`pickup:{id}`) and requires the plan `version` | a plan can be recomputed between the screen being drawn and the reorder arriving, so a position would point at a different place. The version makes a stale reorder a `409` rather than something applied to a route the dispatcher never saw. | | `RoutePlanUpdatedNotification` | does **not** extend `BaseFcmNotification`, and sends no `notification` block at all | that base exists to give every *visible* push the same title, icon, channel and priority — exactly what a route repaint must not have. A route is recomputed every time a stop is completed, so a visible push would buzz a captain a dozen times an hour about work they are already doing, and they would learn to swipe away the one that matters. | | The route push | carries the **version**, not the plan | FCM caps a message at 4 KB and a route with a polyline can exceed it; an oversized message is rejected outright, so the captain would get nothing rather than a truncated route. The phone fetches `GET /api/driver/route-plan` — the same call it makes on open, so there is one way to read a route instead of two that can disagree. Every value in the data payload is a string because FCM rejects anything else with a 400 from Google. | | `RoutePlanUpdated` | dispatched from `RoutePlanService::publish()` rather than from the triggers | this is what makes "a version is never pushed twice" true: several triggers coalesce into one new version (T7.2), and the announcement belongs to the version, not to the asking. | | The broadcasting auth route | one endpoint on `auth:driver,admin` rather than one per guard | a captain subscribing to their own route and an admin subscribing to the dashboard are the same question asked by two audiences. `routes/channels.php` checks the user's *type*, so an admin token reaching a captain's channel is refused as firmly as another captain's. | | `RoutePlanBuilder` | writes a **degraded plan** rather than none when the engine fails, and also when the captain has no GPS point | the stop order is arithmetic, not cartography, so it survives an outage intact; what is lost is the polyline and the confidence in the minutes. A captain holding an order and no plan would be worse than one holding a rough plan. With no GPS there is no first leg to measure, so the route is built from the first stop and flagged rather than withheld. | **Routing engines.** `DISPATCH_ROUTING_ENGINE` picks the implementation (`fake`, `google`, `osrm`) and `DispatchServiceProvider` resolves it. All three resolve, but only `fake` and `google` answer: `osrm` is a contract with no client behind it yet and refuses loudly on every call, naming whether the URL is missing or the engine simply is not built. It never falls back to an estimate, so a deployment cannot silently ship invented road times. Test fixtures live in `tests/Fixtures/Dispatch/`. **Going live on Google Maps** is a configuration change and nothing else: set `DISPATCH_GOOGLE_MAPS_KEY` and `DISPATCH_ROUTING_ENGINE=google`. The key sits in `config/dispatch.php` (`google.key`) rather than `config/services.php` so that `DispatchSettings` stays the one door to everything the algorithm runs on, and it is never included in `effective()` — that array is what the dashboard settings endpoint returns. **No test ever calls Google.** `phpunit.xml` forces `RUN_REAL_MAPS=0`, so a key sitting in a developer's environment cannot turn a test run into billed traffic. To exercise the real API on purpose, set `RUN_REAL_MAPS=1` in your own shell with a real key; one test then runs and every other keeps answering from the recorded fixtures. Two things are never requested, from the cost report §12: two-wheeler routing and toll fields. A test asserts their absence in both the request body and the field mask, because an invoice is a slow way to discover a widened mask. **The dispatch benchmark is sensitive to what else the machine is doing.** `CandidateFinderPerformanceTest` asserts that 500 captains are narrowed in under 100 ms, judged on the median of nine runs. On a quiet machine that median sits near 45 ms; with another test run or a build in progress it has been measured at 121 ms and failed. It prints every sample and a per-phase breakdown, so compare the phases before suspecting the algorithm: if `geosearch` and `capturedAtTimestamps` both grew by the same factor, the machine is busy, not the code. Run it alone before drawing a conclusion. ### 4.1 The rest of the codebase (pass of 2026-09-16) The skill was applied to every file, including the ones the dispatch work never touched. Three things were brought into line: | What | Change | |---|---| | `app/DTOs/DeviceTokenData.php` | moved to `app/DTOs/DeviceToken/DeviceTokenData.php` — a DTO belongs in a feature folder. | | `OtpService` held its own `Cache::` calls | the storage moved to `app/Repositories/Otp/OtpRepository`, bound in `RepositoryServiceProvider`. The service decides what a code means; the repository decides where it sits. The cache key format is unchanged. | | `Dashboard\Order\OrderController` took `Order $order` | the two routed actions now take the UUID as a string and let `OrderService` resolve the record, like every other action. Route-model binding is an implicit query in a controller and skips the repository. | The `base-repository` rule of `scripts/check-architecture.php` was narrowed to the repositories that inject a model. `BaseRepository` is built around an Eloquent model it takes in its constructor, so a store with no table behind it — `OtpRepository`, and the Redis classes above — cannot inherit it. Two tests in `tests/Unit/Scripts/ArchitectureCheckScriptTest.php` hold both halves of that rule. Deliberately left alone: | What | Why | |---|---| | `OtpStatus`, `DriverSignInStatus`, `AdminAuthStatus` have no `label()` / `options()` | the skill asks for those on a set that is cast on a model and offered in a select. These three are outcomes returned by a service and turned into an HTTP status and a message; they are never cast, never validated with `Rule::enum`, never rendered in a select. Adding the pair would mean en + ar keys nothing reads. | | `routes/.php` rather than the skill's `routes/api/.php` | all eight feature route files already follow the flat convention and the checker's `route-loading` rule accepts it. Moving them would touch every route file for no behavioural gain. | | `app/Services/BaseJsonService.php` caches directly | nothing extends it — the class is unused. Worth deleting rather than refactoring; left for whoever owns it to confirm. | | `app/Services/Dashboard/DashboardStatsService.php` caches directly | uncommitted work belonging to a teammate. It needs the same `Cache::` → repository move as `OtpService`, by whoever owns the branch. |