Technical team session · September 2026
The respondent
platform, rebuilt.
Where we stand, what is actually wrong, and four routes forward — from a fortnight of cleanup to a full rebuild.
Part one
Where we are
How the platform reaches people today, and the single measurement that governs everything else.
The job, in one line
We ask people a question, and we need them to answer it
Everything else — the databases, the servers, the apps — exists to serve that loop. Worth holding on to, because most of what follows is about machinery that has grown up around it.
The three boxes in the middle are the interesting part. We built three separate ways to reach people, and we still run all three. This session is largely about whether we need to.
The whole business depends on one thing: that a question reaches someone, and they answer it.
The measurement that governs everything
Does the notification arrive?
Response rates vary enormously by region — the Middle East, the Sahel and Türkiye have completely different phone, connectivity and network realities. So response rate is the wrong thing to design against. Notification delivery is the right one, and it is close to binary.
If people have to remember to open something and check, the answer rate collapses to a few percent. That gap — and nothing else — is why we still run a native app and the social channels alongside the Chatapp.
Why we ended up with three
Each channel solves one problem and creates another
Mobile app
Notifications always arrive. Richest features.
But: people won't install it. The download step is where most drop out — so it only ever works for an already-committed group. One customer left.WhatsApp & Facebook
Notifications always arrive, and people are already there. Nothing to install.
But: Meta changes the rules and things break or get blocked. And from 1 October it starts costing money where it used to be free.Chatapp
Most flexible, nothing to install, entirely ours to change.
But: it can't reliably notify anyone. Browser permission prompts are so abused by news sites that people dismiss them without reading.So we run a workaround: notify through WhatsApp or Facebook, then send a link into the Chatapp. It works — but it means every conversation depends on Meta, and we maintain three separate pieces of software to do one job.
Part two
What is actually wrong
Four structural problems and four deadlines. Some of them are not what we thought they were.
Problem one · cost
We were right that storage dominates. We were wrong about which storage.
The belief going in
“About half our cloud bill is storing all those documents.”
What the bill actually shows
Document storage costs about four dollars a month — under a fifth of one percent. The folder we assumed held them is completely empty.
Where it really goes
Roughly a quarter of the entire bill is two large disks plus their backups, belonging to one machine: the system that keeps a personal copy of everything in step across every device.
The question was never where to put the documents. It is whether we should be making a personal copy for every person at all.
Problem two · complexity
We give every respondent a private copy of everything
When a survey goes out, we don't send one questionnaire to a thousand people. We create a thousand private copies, one per person, and keep each in step with that person's phone.
It was a reasonable choice when it was made — it let the app work offline. But it means cost grows with how many people we reach and how often anything changes, rather than with how much we actually store.
And the irony
We are paying for an offline-first system that does not actually let anyone answer offline. Neither the app nor the Chatapp can hold an answer and send it later. Checked in the code, twice.
This one design choice drives most of the cost and most of the complexity. It is the thing Option B exists to undo.
Problem three · dependency
Our most reliable channel is the one we control least
Why it keeps breaking
It isn't bad luck. Meta retires each version of its interface on a fixed schedule — roughly three releases a year, each switched off about two years later. On top sit template approvals, quality ratings and number blocking, any of which can stop us without a single change on our side.
And now it costs money
From 1 October, replies inside a conversation stop being free, with no free monthly allowance. Verified in Meta's own documentation — several vendor blogs claim an allowance that does not exist.
What doesn't help
Switching to a different Meta-owned channel, or putting a reseller in front. Both change who absorbs the disruption, not whether it happens.
What does
Keeping at least one non-Meta channel actually live and carrying traffic, so switching is tested rather than theoretical. Telegram is free at our volumes with no forced upgrade cycle — though it was blocked nationwide in India for six days this June, so it is a second channel, never a sole one.
We can't remove this dependency, and shouldn't try — it is where people are. We can make it survivable.
Problem four · maintenance
Three pieces of software doing one job — and one is frozen
What that yearly cost actually buys
Every year one colleague — the one with a MacBook, because Apple requires it — upgrades the framework, raises the Android and iPhone version targets, and republishes to both stores. It takes real time, and a user would not notice a single difference afterwards. It is pure permission to keep existing.
And this year's round is overdue
Google Play raised its baseline on 31 August and the app is below it. Nobody loses access and existing installs keep working — but no new version can be published until the round is done, and the extension window closes 1 November.
So the real question isn't what the app costs — it's whether we want to owe this every year, for one customer, forever. Note this also rests on one person and one laptop.
Not optional, and not waiting for us
Four deadlines are already running
WhatsApp starts charging
Replies inside a conversation stop being free. We need to decide how our messages are classified, because that sets the rate we pay.
The sync disks fill up
Both are 91.5% full and growing steadily. Needs a decision or more space, whatever else we choose.
The foundations are end-of-life
The operating system under the sync machine stopped getting security updates in June 2024. Half our small automated jobs run on retired versions too.
The app's update window
The last date an extension can be requested, before the app is stuck at its current version indefinitely.
Only the second of these is a rebuild question. The other three need an owner this week regardless of which option we choose.
Separate from everything else on this agenda
Two security items that need owners
These have nothing to do with which option we pick. They surfaced while reading the code, and would be wrong to leave in a research document.
A key that should be secret, isn't
The Chatapp's signing key is stored in plain text, is the same in test and production, and is delivered inside the app itself — so anyone who opens the Chatapp can read it. On the face of it, that means login links could be forged.
Passwords in the change history
Several live credentials, including production certificates, are committed into our code history. Deleting the files doesn't help — every copy anyone ever took still contains them. These need replacing, not removing.
I haven't tested whether the first is exploitable — that would mean attacking our own service. But given our certification and who our respondents are, it deserves a look today, ahead of anything else in this deck.
Part three
The good news
The foundations turned out to be in far better shape than the symptoms suggested.
What we found when we looked properly
Most of what we need has already been built
We already have a delivery hub
The part of our system that works out who needs to know about this is well built, and it has a deliberate socket for plugging in new ways of delivering. It already drives both the private copies and the push notifications.
We were going to propose building this. We wrote it years ago.The Chatapp is already modern
It doesn't use the private-copy system at all. It asks for what it needs and sends answers straight back — exactly the design we'd move everything else towards.
One of our three clients has already made the journey.The real data is safe and separate
Our main database is the authoritative record and always has been. The private copies are a distribution mechanism, not our records — so changing how we distribute doesn't put data at risk.
Telling detail: nobody ever set up backups for the copies.This changes the question. Not “how do we rebuild the sync system”, but “how do we plug a new delivery method into the hub we already have, and switch the old one off” — a staged swap, with a way back at every step.
Part four
What we decide
Three questions, four options, and one thing that has to be measured before we can honestly choose.
Everything else follows from these
Three decisions
Do we keep making private copies?
Our surveys don't need it: one person answers, nobody edits anyone else's work, answers are never revised. The machinery solves a problem we don't have.
Do we keep the mobile app?
One customer, and one of two things keeping the old system alive. Now a commercial and product question rather than a technical one.
How exposed to Meta do we accept being?
Not a yes or no. It's whether that exposure sits behind a boundary with a tested alternative, or runs through the middle of everything as it does today.
The four options are combinations of these answers, and they stack: B includes A, C includes B. We can commit to one and decide the next later.
Stabilise
SafestStop the bleeding, buy time, change no architecture.
What we do
Add disk space. Switch off the duplicate test environment overnight. Delete what's running for nothing. Update the retired components. Close the security items.
What it buys
The disk deadline goes away. Roughly 13% off the monthly bill. Security stops degrading. No product risk at all.
2–4 weeks, one person.
What it doesn't fix
Everything structural — three codebases, the Meta exposure, the app's cost. And the disks fill up again. A reprieve, not a cure.
This isn't really a choice — it's hygiene. Every other option starts here. The only question is whether we stop here.
Swap the delivery method
Best valueA + stop using private copies to deliver things. Plug a new delivery method into the hub we already have, move each client across, then switch the old one off.
Why it's safer than it sounds
The socket already exists and was built for this. Old and new can run side by side, one client at a time, with a way back at every step. Nothing about how we notify people changes.
What it buys
Removes the cost driver permanently rather than deferring it. Retires the end-of-life machine and its 1.8 TB. The Chatapp and both chat channels need no changes at all.
Where the work is
Rebuilding a dozen pieces of reporting logic written years ago with no tests and no documentation. They must be recreated and checked against live behaviour — careful work rather than clever work.
Start with the Webapp half — it has no customer attached, so it can begin immediately, and it proves the whole approach before we go near the app.
One app for everyone
Needs proving firstA + B + retire the mobile app, so the Chatapp becomes the only thing respondents use. Only defensible if notifications still arrive.
How we'd keep notifications working
On Android — 82–93% of our respondents — we can wrap the Chatapp in a thin shell that delivers notifications exactly as the app does, over the same underlying service, while keeping one codebase. A genuine like-for-like swap.
On iPhone, people must add the Chatapp to their home screen once. After that, notifications go through the same Apple service our app uses, with no loss of quality.
The risk, plainly
That home-screen step is a real cliff, and we cannot prompt for it — only show instructions. If the remaining customer's panel is iPhone-heavy, this gets materially worse. It has to be measured, not assumed.
What it does to the yearly round
The Android shell is thin — no framework upgrade treadmill, and far less to redo each year. On iPhone there is no store listing at all, so Apple leaves the critical path entirely, and with it the MacBook dependency.
What else is missing
One real gap: the field where a respondent enters the reference their points system needs. Rewards and vouchers are not in use and out of scope — and the points integration already works for the Chatapp.
Until we have run one notification pilot, this is a hypothesis. A good one — but not yet a plan.
Rebuild the estate
Most radicalC + finish the modernisation somebody already started. Forty machines become a handful of managed services.
What it buys
The maintenance burden largely disappears — no operating systems to patch, no end-of-life surprises, no duplicate test estate running all night. For a team our size that is the real prize, not the hosting bill.
The case against
9–18 months — long enough that our priorities will change underneath it. Every rewritten service can regress, and some date from 2017 with nobody left who remembers them. It would absorb most of the team's capacity for a year while the business still needs new features.
The honest version
If C is done well, what's left is mostly tidiness. The strongest form of D isn't a project — it's a standing rule that anything we touch gets modernised, letting the old estate die by attrition.
The real argument for D is that attrition never actually happens. Teams say “we'll modernise as we go” and then don't — and our own history supports the worry: the Chatapp has been on the right architecture for years, and nobody finished the move.
Side by side
The four options against what we care about
| A · Stabilise | B · Swap delivery | C · One app | D · Rebuild | |
|---|---|---|---|---|
| Time | 2–4 weeks | Months | Months, plus a pilot | 9–18 months |
| Risk to notifications | None | None | Must be proven | Must be proven |
| Fixes the disk deadline | Defers it | Yes | Yes | Yes |
| Removes the cost driver | No | Yes | Yes | Yes |
| Reduces Meta exposure | No | No | Yes | Yes |
| Stops the app's upkeep | No | No | Yes | Yes |
| Needs the customer's agreement | No | To finish | Yes | Yes |
| Can we change our mind? | Fully | Largely | Partly | No |
A is hygiene. B is the best value and cannot harm what pays us. C is B plus a pilot we haven't run. D is C plus a year we probably shouldn't spend.
If we choose A → B → C
What the first ninety days look like
This week — none of it commits us to an option
- Decide how our WhatsApp messages are classified, before 1 October
- Add disk space to the sync machine
- Replace the exposed keys
- Pull the real cloud bill so we can check these numbers
- Ask the app customer what they actually need
Weeks 2–12
- Clear the waste and update the retired components
- Start the Webapp on the new delivery method
- Run one notification pilot — one group, one region, measuring delivery
- Bring a non-Meta channel live so switching is tested, not theoretical
Before going near the app
Agree a written end date with the customer. Drifting indefinitely is how migrations like this fail — not by going wrong, but by never finishing.
If we leave this session having only started this week's list, that is a good session. None of it is irreversible, and all of it makes the eventual choice better informed.
For the room
What we still need to find out
1. Will a home-screen Chatapp notify as reliably as the app?
Everything about Option C rests on this, and nothing else answers it. One group, one region, measure delivery.
2. How many people are on which version of the app?
This sets the real timeline for B — and it's a database query, not an estimate. We already hold the answer.
3. How iPhone-heavy is that customer's panel?
Android has a clean swap. iPhone needs the home-screen step. This ratio decides whether C is comfortable or risky.
4. Do we believe we'll modernise as we go?
If not, D deserves a serious hearing rather than a polite one. That is a judgement about us, not about the technology.
Three of those four can be answered this week, and two without leaving the building.
Over to the room
Where would you like to start?
Technical detail — architecture, migration inventory, method and corrections — follows in the appendix.
Appendix
The technical detail
Everything the main deck deliberately left out — for the engineering discussion, not the decision.
Appendix · architecture as it actually is
The real shape of the system
uPublish is a message bus, not a document generator
Flow is webservice → uPublish action → subscriber resolves audience → publisher delivers. 18 actions, 10 subscribers, 4 publishers behind a one-method IPublisher interface with a factory. Two publishers live: CouchDB documents, and push to APNs/FCM/web-push.
CouchDB is transport, not the system of record
MySQL is authoritative — question_push is the assignment table, answer and its satellites hold answers. One database per user, u<id>. No backup job exists for CouchDB; MySQL and Redis both have one.
usync is a stack, not a database
CouchDB 1.6.1 behind nginx + an HHVM request handler + a beanstalkd worker replaying writes into MySQL. Loop prevention is an nginx header map; the in-code guard compares a header array to an integer and can never fire.
Coupled clients
webapp — live as uwebapp-v3, pull-only, six views, change-feed-driven UI — and mobapp, frozen and below target SDK. chatapp is not coupled. e-learning and control-center are retired.
Write path
Clients mutate over REST → webservice → MySQL, then POST /sync-requests to uPublish refreshes documents. Client-side save() is an optimistic local cache write. No conflict resolution exists and none is needed — documents have a single writer.
Points integration
AnswerModel::_executeWebhook fires on every answer and skip, reads external-url off the focus group, posts with user data from Redis. Server-side and channel-agnostic — already works for Chatapp. Owner-side config UI exists in webapp.
Appendix · the migration surface
Estimate from this, not from my number
My Option B estimate moved four times in two days — 4–8 weeks, 8–14 weeks, 3–5 months, then back down on finding the publisher seam. Each move followed a fact, but a figure that unstable should not be planned against. The inventory is what's solid.
| Server-side CouchDB design views | 12 |
| …plus a drifted dev copy to reconcile | 11 views |
| Document types | 16 |
| uPublish actions | 16 |
| Replication clients | 2 |
| Coupled repos | ~18 |
| …needing code changes | ~13 |
The hard part
The views are minified JavaScript inside PHP string literals, with no tests and no reference implementation. Worst is the reduce that is the only definition anywhere of “which question does this respondent see next, and how far through are they”. Rebuild and verify against live output.
Rollback risk
The legacy sync box may not be rebuildable — CouchDB 1.6.1 + HHVM + beanstalkd on Ubuntu trusty, pulling GPG keys from a keyserver network dead since 2019. If it can't rebuild, there is no rollback target.
Three tickets, unrelated to the decision
The user-deletion Lambda never removes orphaned usync databases — on the disk that's 91.5% full. Mobapp's Sentry config silences every design-doc error. usync_host sits in six repos' CI and is read by zero lines of source.
Appendix · how this was put together
Method, and what I got wrong on the way
Measured
Read-only sweep of the production AWS account — every instance, volume, snapshot, bucket, table and function; nothing changed. Code read directly with a read-only token across all 81 repos, cross-checked against the running estate and against product-development/docs, which are accurate and current.
Estimated
Every euro figure is public list price applied to observed inventory, not our bill — Cost Explorer wasn't available. Database instance sizes are inferred from memory metrics and are 26% of the estimate: the single largest uncertainty.
Corrections the team made
- I said only the app used the private copies. The Webapp does too — that came from Rik, not from me
- I proposed building a delivery service. It already exists — uPublish, once Rik explained the flow
- I called rewards the biggest gap. They're not in use; the real gap is one field
- I treated response rate as the metric. Delivery is the metric, and it's near-binary
Still unknown
Nothing was observed at runtime — no view-access statistics, so we don't know which of the 12 views are hot. The most likely undiscovered consumers are outside the GitHub org: console-created Lambdas, BI tooling, partner integrations, operator scripts.