Contents
What this modal is
It is the dark "Welcome to your Advanced trial" pop-up that appears the moment someone creates a Carepatron account and starts their 14-day trial. It shows a hero image, lists eight features (reminders, calendar sync, custom branding, white labeling, unlimited storage, workflows, premium support, and 1:1 onboarding), and has one button: "Get Started".
Two things about who actually sees it:
- Since late May, only half of new signups see it. The other half is a holdout group for an A/B test that is still running.
- A branch added in April means people who have not enabled payments can instead be shown a different first modal: a payments pitch ("15 minutes to start taking card payments", first $1,000 in processing fees free, book a 15-minute call).
People clearly engage with it. Of everyone it is shown to, about 86% view it and about 23% click "Get Started". Those are strong numbers for an in-app message.
The open question was whether it makes people more likely to become paying customers. The A/B test (numbers from the aligned window, 30 May to 9 July; I did not re-pull fresh figures because the shape of the result would not change):
| Group | Sent | Became paying | Rate |
|---|
| Saw the modal | 1,863 | 34 | 1.83% |
| Saw nothing (holdout) | 1,791 | 26 | 1.45% |
That looks like a 26% relative lift, but it is not statistically significant (p = 0.38). To prove a gap this small you would need about 17,000 people per group. At roughly 45 people per group per day, the test would need to run for over a year. So the honest read is: no evidence of harm, a mildly positive lean, and a test that can never give a definitive answer on this metric.
What is wrong with it
- The button goes nowhere. "Get Started" opens an empty web address and then closes the modal. Nearly 1 in 4 people click it and simply land back where they already were. The click is tracked, but the user gets nothing.
- The copy is a feature dump. Eight features in one sentence, no single clear action. Welcome messages work best when they point at one first win.
- Half the audience sees nothing, for no benefit. The holdout keeps running but cannot reach a conclusion, so it is only costing reach.
- The payments branch is a heavy first ask. For people without payments enabled, the very first thing the product says to them is "book a 15-minute sales call". That is a big ask at second zero.
- Cosmetic issues. The hero image is 998px wide inside a 650px modal, its alt text says "Get Started" instead of describing the image, and the dark theme does not match the product interface.
Recommendations, ranked by impact
- Point the button somewhere. Make "Get Started" open the setup dashboard or checklist. This is the cheapest win: the 23% who already click would finally be taken to a next step.
- End the holdout and show the modal to everyone. It is directionally positive, costs nothing, and the test cannot conclude. Ending it doubles the modal's reach immediately.
- Rewrite the copy around one action. Replace the eight-feature list with a single next step, and personalize it with first name and profession from the signup questions.
- Judge future changes on a faster measure. Instead of "became paying within 35 days" (too rare to test against), compare day-one activation or setup-checklist completion in PostHog. Those happen often enough to give an answer in weeks, not years.
- Move the payments pitch to second position. Let everyone get the welcome first; show the payments offer after, once they have seen the product.
- Tidy the visuals. Right-size the hero image, fix the alt text, and consider matching the in-product look instead of the dark theme.
What I have done already
- Reviewed the live template and the campaign wiring in Customer.io. Nothing in the live campaign was changed.
- Pulled the A/B result from the 9 July analysis.
- Created a Todoist task ("Improve welcome modal in-app notification, CIO campaign 50, action 343") holding all six recommendations, tagged @deep.