The trial stopped recruiting one-off senders

Churn that was actually a completed job

Jonah Adeyemi, product lead at Kite Postal, had a dashboard and a phone. The dashboard said these accounts churned. The phone said the parcel arrived and the person was finished. Those cannot be the same event. One is a product failure. The other is a job completed in a flow that only knew how to sell a habit.

The question was not “how do we reduce cancellations.” Cancellations mixed two populations, and reducing the mix would have meant talking some one-off senders into a subscription they did not want. The question was whether the first week of behavior could separate people who wanted a route from people who wanted a single send, early enough to put them in different doors.

One timeline, four events

Each account became a single timeline. Signup path, first parcel, second parcel, cancel reason. Sessions before an account existed were attached on the email used at the door, then dropped if that email never confirmed. We did not model pageviews as loyalty. A person can read a pricing page and still want exactly one box.

Cancel reason was kept as text and as a code. The code was used in the model. The text was read beside the interviews, because “too expensive” and “I only needed one” had been collapsing into the same bucket in the warehouse. That collapse was the original analytical error. The research started by undoing it.

Who was allowed to count as a leave

Accounts were deduplicated on address plus email. A second signup the same day was a retry, not a second customer. Cancels with no second session were kept as their own class and were not merged into “product failure.” Payment failures and address errors were removed from the survival set. They are operational exits, and they would have taught the model the wrong lesson about the offer.

The first seven days were censored carefully. An account that had not yet had seven days of exposure at the moment of the extract was not labeled as a day-30 leave. Right-censoring is the entire point of a survival model, and skipping it would have made new signups look like churn. We also dropped internal test routes and the employee parcel program. They send every week by policy, which is not a customer habit.

Survival to day 30, scored from week one

The model estimated the chance of still being active at day 30. Inputs from the first seven days were: number of sends, whether a second session happened, whether the weekly line on the signup page was in the viewport long enough to count as seen, and the cancel code if it had already arrived. Seeing the weekly line mattered because the heatmap said many people never looked at it. A sentence under the fold is not an offer.

One send and no return visit carried a high chance of a day-30 leave. That split was stable across the five weeks we used to fit it. It did not require the cancel reason to be known. By the time the reason arrived, the routing mistake was already weeks old. The useful output was a population, not a probability displayed to anyone. We never shipped a score on the page. We shipped a second door.

The line people skipped

The signup heatmap is hottest on the left block, around the weekly-use promise. The rest of the trial offer cools quickly. People were not studying the terms and rejecting them. They were passing the only sentence that said what the trial assumed. The model and the picture describe one miss: the habit was not in a place they looked.

Two doors, then the email

Before, there was one trial door. The weekly habit was a sentence under the fold. After, the choice is the page. Every week carries the tension, because that is the promise people used to skip, and it is now the thing they have to pick. Once sits beside it at its own price, with less heat, because it was never the confused part. The email field comes after the door. A person who wants one parcel is not asked to join a route in order to type an address.

Signup volume was expected to fall. Jonah could defend that if the people who stayed were the weekly ones. The research agreed in advance to report both numbers, and to let only the second one decide.

What the phone had already said

The interviews were cancellation calls, not a panel of enthusiasts. The repeated line was that the parcel had arrived and they were done. No complaint about the route, because they had never meant to have one. Leah’s description in the room was the model’s description: one send, no second session, high chance of a day-30 leave. That is a routing problem.

Leave had to fall, even if signups did

The routed offer ran against the old trial for five weeks of new accounts. Primary measure: leave in the first month. Hold: the share of trials that became paid. Signup count was reported beside them. A design that saved the dashboard by shrinking the denominator without improving the people who remained would have failed the hold.

First-month leave fell 14%. Trial-to-paid rose 11%. Signups did drop, and that drop was not treated as a loss in the write-up to the team. Conversion improved because the wrong customers were no longer counted as failed ones. The smaller trial is the product working on the people it was built for.