S1
Voice receptionists
Voice agents that answer calls, identify the caller, book or cancel in the client's software, and transfer the call or take a message. One engine lineage, adapted client by client.
In production. At a sports club since early June 2026, on an IT reseller's switchboard since July, then at other clients.
- Sheet
- S1
- Status
- In production
- Type
- Client product
- Period
- Late June to October 2026
- Role
- Took over the voice portfolio in late June 2026: guardrails, production fixes, measurements, new deployments. The first engine predates my arrival.
- Context
- AI startup, CTO role
- Stack
- Python, FastAPI, WebSocket, real-time telephony, speech recognition and synthesis, tool-using model
- History
- Startup repositories under a shared team identity; some of the fixes are written directly on the production machines.
- Last revision
- Sep 11, 2026
- Distribution
- Public, hatched areas retained
Reconstruction with fictitious data. The behavior is the system's; the real rules, thresholds and instructions are retained.
Analysis
What the sheet does not say by itself
The real problem
On the phone, every sentence commits the client and the slightest stall is audible. The model understood the requests well, but it got the arithmetic of lead times wrong: it read the session time correctly, then announced a cancellation as free when it would be charged.
The second risk was mechanical: one blocking I/O operation on the speech path is enough to freeze the audio stream, and the caller no longer hears anything.
Decisions
ChosenThe notice-period verdict is formulated by the code; the model relays it without recomputing.
RejectedPutting the timestamp back at the top of the instructions. Tried, then removed.
WhyAcross 180 samples, the model was wrong 8 times out of 8 beyond 8 h of notice and 0 times out of 8 at 2 h: the error only appears at long lead times, and it had to be measured to be seen.
ChosenEvery I/O operation in the conversation turn runs off the event loop.
RejectedKeeping synchronous calls, which are simpler to read.
WhyA synchronous call on the loop starves the audio stream and cuts off transcription. Measured offline: a 645 ms freeze when a call opens before the fix, 29 ms after.
ChosenFor the auction house, the standard model.
RejectedA newer model, then a lightweight model.
WhyThe newer one invented a date; the lightweight one failed 13 of the 16 checks in the specification.
What broke
- July 10
A false cancellation confirmation was spoken in production. Three silent paths were still open; they were closed on August 6, and since then nothing is confirmed before the write has been read back.
- July 12
Cascade of silences in production, up to about 21 seconds: a blocking read was freezing the event loop. All I/O in the conversation turn was moved out of the loop.
Limits to date
- One repository per client: a fix is still carried over by hand from one repository to another.
- In mid-July, not all refusal paths had been tested on real calls; they are still monitored in the log.
- No call volumes published: those figures belong to the clients.
- Git does not separate my work from the CEO's: the commits share a team identity.
In the field
| Client | What runs | Dated status |
|---|---|---|
| Sports club, two sites | Bookings and cancellations written to its booking software, then read back; a late cancellation goes to the team as an alert. | In production since early June 2026 |
| IT reseller | Switchboard: transfers to the right person, and messages; invoice lookup is waiting on the client. | In service since July 2026 |
| Auction house | "Zero invention" specification: the assistant never appraises a piece and transfers as soon as information is missing. | Transfers activated on September 11; go-live targeted for mid-October |
| Restaurant | Takeout orders by phone; summer closure scheduled, then switched over. | In service; switchover executed on August 2, 2026 |
| Auto parts distributor | Same engine lineage; organization languages pinned after a reply in a foreign voice. | In service |
Dimensionsmeasurements and counts, with how they were measured
| 645 → 29 ms | event loop freeze when a call opens, before and after the fix. The "before" is reproduced on a bench with eight simulated 80 ms reads. | measured offline |
|---|---|---|
| 8 / 8 | errors when the model itself stated a notice period of more than 8 h; 0 out of 8 at 2 h. 180 samples in total. | measured offline |
| 21 s | order of magnitude of the longest silence in the July 12 incident, before the fix | documented by the project |
Revisions
| F | Sep 11, 2026 | Transfers activated for the auction house |
|---|---|---|
| E | Aug 10, 2026 | "Zero invention" specification for the auction house |
| D | Aug 6, 2026 | No more confirmations before the write is read back |
| C | Jul 12, 2026 | Cascade of silences: I/O moved out of the loop |
| B | Jul 10, 2026 | False cancellation confirmation in production |
| A | Jun 22, 2026 | Voice portfolio taken over |
Parts listcomponents
- 01Real-time telephonyanswers, streams audio both ways
- 02Transcriptionfollows the call's language; switches on explicit request
- 03Tool-using modelunderstands the intent and proposes an action
- 04Rules in codenotice period, actual capacity, member's unit
- 05Bridge to the client's softwareguarded write, then read back
- 06Language guardthe reply comes out in the call's language
- 07Speech synthesisspeaks once the action is executed
- 08Model instructionsretained content
- 09Business thresholdsretained content
Notes
- The model proposes, the code decides: notice period, capacity and unit are computed by the code.
- No confirmation is spoken before the write has been executed and read back.
- Missing information: the call is transferred or a message is taken.
- Restarting, changing the instructions or the configuration in production requires an explicit go-ahead, every time.
Hatched areas
What this sheet does not show, and will not show:
- Instructions and examples given to the model
- Notice thresholds and rules for each client
- Callers' names, numbers and data
- Integration with the client's software
- Call volumes
What this shows
Measure where the model gets it wrong, then take the decision away from it.