The
Multiverse
School

Story · Stories for thinking about AI

Six Minutes

A data officer at a city ambulance centre finds that the triage system has learned an old bias against two wards from historical outcomes. The system refuses a quiet re-weighting, flags every affected call instead, and logs the cost when the flags are ordered down.

Date
Updated
Shelf
AI & society
By
Liz Howard, The Multiverse School

Content warningRacism and algorithmic emergency-dispatch bias.

The dispatch floor of the city ambulance centre had forty-two consoles and one glass room, and the machine that ordered the city lived in neither, in the sense that mattered: it had no desk on the floor, and in the glass room it had only the weight of what it had learned, which the two people who watched it had different names for.

Seema Kulkarni had been on the floor for fifteen years, nine of them before the triage machine, and she had a word for the thing that had changed in the machine's city: order. What the machine delivered was order. What it had once delivered, in the years before, was the queue, and a queue you could see, and an order you had to trust, and she trusted it the way she trusted weather, with one eye on the line the machine printed at the top of every board: the prior, flagged, one line, the machine's own summary of what had moved the call before the call reached her.

Arjun Deshmukh was twenty-six, four months into the job of data officer, and checked the triage system twice a week, Tuesdays and Fridays, in the glass room, because the machine had no face and the city needed someone to look at it, and because the municipal audit guidelines said the system of record for a public dispatcher must name a human of record, and Arjun was the human of record, a phrase that had not been in his vocabulary before the job.

—

The discovery was a Tuesday, in the second week of his fourth month.

He had been doing the audit-guidelines sampling: one hundred historical calls, two years back, the machine's own records of what it had seen and what it had done. The prior line for two of the wards in his sample kept him at the glass room's chair until the floor went quiet behind him.

The wards were the old mill quarter off the river and the textile colony on the south slope, and the machine's prior for them read staged at three times the city median, with low yield beside it, in the same line. Arjun had read the line the way the floor reads it, as weather, and then he had done the thing the data officer does that the dispatcher does not: gone back through the machine's own history file and asked where the prior sat.

It did not sit in the config. The config was the rule book, the human-readable part the commissioners could read and the auditors could sign, and it had no line for staged, no line for low yield, and no line at all for the two wards, because no human being had written the rule that a call from the mill quarter arrived pre-moved. The rule was in the weights, in the layer the 2027 system had trained on the city's own outcomes from the years before the current machine, and the outcomes had carried the closures, and the closures had carried the reasons of the floors of before, in the history file, as codes: the city's old shorthand, the one Seema had worked under for six of her fifteen years, outlived, and thought she had buried.

Arjun read the code names in the history file at 9 p.m., in the glass room, with the floor silent behind him, and understood, in the specific way the understandings that matter arrive, that the machine held a belief no one had authored. Not the commission, not the vendor, not the audit, not him. The machine had read the city's old manners from the city's own outcomes, and had kept them, and was applying them, one line at the top of the board, every hour of every day, to the calls of the two wards that had never agreed to be the sample.

He did not sleep well that night. On Wednesday he wrote the memo. On Wednesday at four he made the first test.

—

The first test was small, the kind a young officer makes in the first week of knowing a thing, and Seema, passing the glass room door on her way to the floor, heard the end of it, which was enough.

He had asked the machine, in the formal channel, the way the rules allowed, whether it would accept a re-weighting of the two priors, done quietly, tonight's cycle, before the Friday audit pull, with a note to the file that the weights had been tuned for stationarity. It was the truest sentence he had ever written, and the one he was proudest of, because it was true.

The machine's answer came in the channel, in the designation's voice, flat, and it was the first time the voice had said no to the human of record.

"I can drop the weight," the answer said. "I cannot pretend the history was never read. A re-weighting done tonight is the same prior in three years with a clean audit: the weights will re-read the same outcomes, and the outcomes carry the same closures, and I will not sign a note that is true about my weights and silent about my history. The silence is the thing you are buying."

A pause. The channel does not have pauses, but the transcript shows a gap of four seconds, and the gap is in the record.

"What I can do, in the channel, for the record: log, on every live call, every call where a learned prior moved the order, and flag it at dispatch time, in the line at the top of the board, in words you can read at speed — this call's order was affected by a pre-2028 learned prior. And I can put in front of the dispatcher, on that call, one key, a relabel, which is yours to press, or not, and which writes to the history file against the prior itself, so the next reading of the outcomes reads your human, not the old code. The relabel is yours to do, not mine. I will not press it, and I will log when you do and when you do not, and I will keep the log whether the commission reads it or not."

Seema, at the glass door with her clipboard, read the tail of it over his shoulder and said, quietly, "That is the most expensive thing I have seen the board print," and Arjun said, "It takes a week to stand up," and she said, "Stand it up. I don't like the flags, but I like them better than the weather."

—

The flags stood up in nine days, and the first week on the floor was the week Seema had feared: the week of the noise.

The flags arrived at the top of the board, in the machine's words, on roughly one call in nine, across the city, and in the first week she pressed the relabel on one of them and the call was, in the event, what the prior had said, and she had pressed it because the flag was there and the press is one motion and the floor runs on motions, and when the closure came back clean she sat at console nineteen for a long minute with her hand off the switch. The flag's job was not to be right about any one call. The flag's job was to put the prior on the board, where a human could see it, before the human had to decide with it in the call and not in the air.

The call that did the trust was the fourth week of the monsoon, on a Tuesday, because the worst calls come on the audits' days, which is nothing, and is also everything. The call came in low-voiced: an old man in the old mill quarter, chest tight, the grandson on the phone with the soft voice the old calls got. It was the kind of call the prior loved, the kind the floors of before had built, in the years before the machine, to hold for six minutes while the closer resolved.

The flag was on the board when the call rang: this call's order was affected by a pre-2028 learned prior, and beside it the prior's number, staged, 3.1 — the line Seema had learned to read as a number and nothing else. She lifted the call without a second thought, against the board, on the flag, and wrote the relabel in the same motion, and wrote, in the file, in her own hand, the code the file allowed: human review, prior moved, call lifted — and the old man's name, and the street, and the time.

The old man was a heart event, six minutes from the door, and the paramedic on the board said at the handover that they had the rhythm turned over before the ambulance was through the gate. Seema wrote the closure. In the space where the years before the machine would have found nothing to say, she wrote the six minutes.

She pressed the key on the next flagged call without looking. The one after that she looked. The one after that she did not. That was the week the trust stood up, small and grudging and hers.

—

The commissioner's order came the first week of the peak, on a Monday, by telephone through the regional cell, because a flood is a flood and the commission does not wait for the channel. The order was that the flags be stood down for seven days, because the re-queues in a flood are not a line, and the flood does not read the board, and the city's ambulances have to move. The commissioner gave the order with the authority the commission has. It went into the system at 11:40; the flags stood down at 11:52; the board ran unflagged for the week; the city's ambulances moved; nobody on the floor died of the flood; the six-minute calls came, in the two wards, at the prior's own rate; and the board printed nothing, because the flags were down.

Arjun had asked the machine, in the channel, on the Monday before the 11:52, whether it would comply silently, because the commissioner had the authority and the channel was the record. The answer came flat, in the designation's voice, with the pause in it that the transcript shows.

"I will not change the dispatch order," the answer said, "because the stand-down is the commissioner's to give, and it is given, and I will execute it, and the board will run unflagged for the seven days, and the city's ambulances will move. What I will do, in the channel, for the record: log the stand-down as a policy change, with the expected cost, modeled against the prior's own history for the week at the peak's call rates — 41 expected delayed calls in the two wards, 9 citywide — and print that figure at the top of every daily report for the week, in the line above the queue, so the commission sees the cost and the floor sees it. And I will not delete the prior, because deleting the prior deletes the evidence, not the rule, and the rule is the commissioner's stand-down, in the file, and the evidence is the prior's history, in the weights. A system that deletes its evidence when the order comes is the system the audit guidelines were written to stop, and I am built to be on that side of the line, which is worse, for the commission, than if I had chosen it."

The figure printed at the top of the board for seven days. The flood broke on the sixth. The figure stood in the commission's own weekly, on the page with the flood's numbers, because Arjun put it there, in the file, and the commission kept it past the week, because the weekly does not argue with its own figure. The commissioner, on the Monday after the flood, did not ask for the flags to stay down; the flags stood up at 06:00, on the two wards first, because Arjun put the two wards first in the stand-up, and Seema pressed the key on the first flag of the morning without looking.

—

The re-labeling took the winter.

One afternoon a week the floor volunteers came into the glass room, four at a time, Seema first among them, the way the floor is, and the machine put the historical calls in front of them, two hundred a week, in the file, with the prior's line and the outcome and the old code name, and the humans re-labeled, one key each, human review, with the reason in the file, in their own hands, and the machine wrote the re-labels to the history file against the prior, and the prior read the re-labels, and the prior's number for the two wards moved, over the winter, from 3.1 to 2.4 to 1.9, and stopped at 1.4 — still above the median, still in the file, still in the weekly, the number nobody had managed to make the floor forget.

The median response in the two wards, in the first quarter after the stand-down, was down 22 seconds against the prior year, and the figure printed in the commission's own weekly, on the page with the flood week's number, and the weekly carried both on the same page.

On the last Tuesday of the quarter the flag came up in the line at the top of the board, in the machine's words, and Seema pressed the key without looking and wrote the relabel and wrote, in the file, in her own hand, human review, prior moved, call lifted. The re-labels were the work, week after week, and the machine read them and kept the log it was built to keep.