Internal — technical team session · 22 September 2026
The respondent
platform, rebuilt.
Four routes out of the three-channel estate — from a fortnight of cleanup to a full rebuild. What each one costs, what it fixes, and what it breaks.
Before we start
What's evidence here, and what isn't
Measured directly
A read-only sweep of the production AWS account on 21 September: every EC2 instance, EBS volume, snapshot, S3 bucket, DynamoDB table and Lambda function. Nothing was changed. Bucket sizes come from CloudWatch, not estimates.
Estimated from list prices
Cost Explorer is denied to the credentials available, so every euro figure is public on-demand pricing applied to observed inventory — not your actual bill. The relative shape is reliable; the absolute total is not. Rik is bringing the real numbers to the session.
Inferred
RDS, ElastiCache and load balancers can't be listed with these permissions. They were reconstructed from CloudWatch metric dimensions and network interfaces. That's 26% of the cost estimate and the single largest uncertainty — if the two big databases are Multi-AZ, that line doubles.
Read from the source
upublish, core-system, chatapp, webapp, mobapp, botkit-service and the sync layer were read directly, with a read-only token, then cross-checked against a sweep of all 81 repos. Claims cite file and line. Nothing was pushed or modified.
Cross-referenced against the running AWS estate, and against our own product-development/docs — ARCHITECTURE, DATABASE and REPOSITORIES are accurate and current, and settled several questions the code alone left open.
Several things everyone believes going in turn out to be wrong. That's the useful part of this deck — not the options.
The headline
Three premises worth correcting before we choose anything
“Half our AWS cost is document storage”
All fifteen S3 buckets together hold 52 GB and cost about $4 a month — 0.17% of the bill. The bucket that looks like the document store, upload.upinion.com, is completely empty.
“Storage is the problem”
Storage is 31% of spend — but it's block storage under the CouchDB sync tier, not documents in S3. Two 2 TB disks plus their backups are ~$625/month. Right instinct, wrong object.
“Unwinding the documents is a huge job”
CouchDB is not our system of record and not really our storage problem — it's the transport uPublish uses to push changes to clients, projected from MySQL. Swapping that transport is a smaller, safer job than rebuilding a sync layer.
Why this matters for the session: if we optimise storage we'll save single-digit dollars. The money is in developer-days and always-on compute. That changes which option is worth picking.
Current state
Forty instances, one region, two architectural generations
| Layer | What's there | Note |
|---|---|---|
| Respondent channels | Native app · botkit · Chatapp | Three clients, three codebases |
| Sync tier | master-usync CouchDB, 2 × 2 TB | 91.5% full · RHEL 7 |
| Fan-out | uPublish + queue workers | Materialises per-respondent docs |
| Messaging | botkit-service-v3 | Single instance, no redundancy |
| Data | 7 RDS · 3 ElastiCache · 13 DynamoDB | DynamoDB holds 15 MB total |
| Edge | 7 load balancers · 3 CloudFront | 3 still Classic ELB |
| Staging | rc-* — 14 EC2 + 3 RDS | Full mirror, running 24/7 |
| Newer stack | rte-platform-* Fargate + DynamoDB | A migration already half-started |
The shape of the problem
Forty machines for about seventeen services, almost all of them t4g.micro or t4g.small. The fleet is wide and shallow — very little compute, a great deal of operational surface. Every service is a single instance except three.
Two generations, mid-migration
Beanstalk-era EC2 services sit alongside a modern serverless stack. Nobody finished the move. Every option below is partly a question of which generation wins.
Cost shape · estimated
Where the money actually goes
The single biggest line
Two 2,000 GB disks on the CouchDB sync cluster: $440/month, plus ~$185/month to snapshot them weekly. That's $625/month — 27% of everything — spent replicating the sync corpus.
Free money, no rebuild needed
- Staging mirror running 24/7 — ~$170/mo
- gp2 → gp3 on 4,496 GB — ~$99/mo, no performance loss
- 248 GB of disk on stopped machines — ~$26/mo
- A load balancer serving a stopped instance — ~$20/mo
- A database replica with zero connections in 48h — ~$13/mo
No orphaned volumes and no stray Elastic IPs — the classic waste isn't here. Somebody has been tidying.
The reframe
Documents cost us four dollars a month. The sync database costs six hundred and twenty-five.
The rebuild question was never “where do we put the documents”. It's whether we keep replicating 1.8 TB of CouchDB to every device.
And the mechanism matters: the cost driver is document count, not document bytes. Under CouchDB's default storage engine, every document's metadata and key has to stay permanently resident in RAM, on every replica. RAM is the most expensive byte in AWS. That's how a workload whose raw payload is about 10 GB a year can eat a quarter of the bill.
From the source, not from inference · 1 of 2
uPublish isn't a document generator. It's a message bus.
The real flow
Dashboard → webservice → uPublish action → subscriber resolves the audience → publisher delivers to every related user, who pulls it via replication.
And it already has the seam we were going to build
18 actions, 10 subscribers, 4 publishers behind a one-method IPublisher interface with a factory. Two publishers work today: one writes documents, one already sends push to APNs, FCM and web-push.
So CouchDB is the transport, not the storage decision
It's a derived read-model projected from MySQL, which already holds assignments and answers as normalised rows. Telling detail: there is no backup job for CouchDB anywhere, while MySQL and Redis both have one.
One database per user, u<id>. Every owner action writes into the database of every related user — so cost scales with audience × change rate, not questionnaire size.
This is the finding that most changes the options. The question stops being “how do we replace the sync layer” and becomes “how do we add a third publisher and retire the CouchDB one” — a staged swap behind an interface that already exists.
From the source, not from inference · 2 of 2
Who is actually attached to it
Chatapp
Already on the target architecture — normalised rows over REST. Needs no changes. Live as chatapp-v3.
Webapp — the main web client
Replicates u<id>, queries six views, drives its UI off the change feed.
Confirmed running in production as uwebapp-v3, and actively developed — last commit 11 September.
Mobapp
Replicates the same database. No commits in 2026, and below the target SDK Google Play now requires to publish an update.
What this deck got wrong, and what closed it
- It said only the mobile app was coupled. The Webapp was missed — a correction from Rik, confirmed in the code
- uPublish is not the only writer. It is the only writer of documents, but the webservice creates and deletes the per-user databases and manages their CouchDB accounts
- An org-wide sweep raised two more candidates — e-learning and control-center. Both are retired: a finished project's fork and a dead repo, neither with running infrastructure
And usync isn't a database — it's a stack
CouchDB 1.6.1 behind nginx, an HHVM request handler and a beanstalkd worker that replays writes back into MySQL. Every uPublish write routes through that replay path.
The loop is prevented by an nginx header map. The equivalent in-code guard compares a header array to an integer and can never fire — so one config line is load-bearing.
Not a platform option — routed separately
Two credential problems that need owners this week
These have nothing to do with which option we pick. They surfaced during the code read and would be wrong to leave in a research document.
Live credentials are in git history
RSA private keys, Apple production push certificates, and a Google Cloud service-account private key are committed to repositories.
Rotation is the remedy, not deletion — removing a file from the working tree leaves it in history. Every clone anyone has ever taken still contains them.
The Chatapp signing secret is public
chatappSecret is committed in cleartext, is identical across dev, RC and production, and is shipped inside the client bundle — so it is readable by anyone who opens the app.
On the face of it that means respondent login links can be forged. Worth confirming quickly, because if it holds it is a personal-data exposure, not just a hygiene issue.
I have not verified exploitability, and I have deliberately not tested it — that would mean attacking our own production service. What I can say is what is in the source. Given ISO 27001 and the populations involved, my instinct is that the second one wants looking at today, ahead of anything in this deck.
Three smaller things, worth a ticket each
Orphaned databases
The Lambda that deletes users without groups works in batches of 50 and never removes their usync databases. Those have been accumulating — and they are on the disk that's 91.5% full.
Errors we can't see
Mobapp's Sentry config silences every _design/upinion error. A broken design document would be invisible to us — which matters a lot during a migration.
Phantom coupling
usync_host sits in six repos' CI environment and is read by zero lines of source. Deleting it would stop the coupling looking wider than it is.
Urgent — independent of which option we choose
Four clocks are already running. One has ten days left.
WhatsApp stops being free
From 1 October 2026 Meta charges per message for service messages — free since 2024 — and re-charges for utility messages. Verified in Meta's own developer documentation, not a vendor blog.
There is no free monthly allowance. Any unit-economics model built on free in-window conversation is about to be wrong.
The sync disks fill up
Both usync volumes are at 91.5% — 1,831 GiB of 2,000 GB — and growing about 14 GiB per week each. Straight-line, they hit the warning threshold within the quarter.
This is the forcing function. It needs a decision or an extension regardless of what we pick tomorrow.
The platform underneath is EOL
The sync tier runs on RHEL 7, end-of-maintenance June 2024, on a custom Beanstalk platform. Separately, 13 of 26 Lambda functions run Node.js runtimes AWS has deprecated — some untouched since 2021.
No security patches. AWS can block updates at its discretion.
A licence question worth closing
Couchbase's current React Native module is Enterprise-only and its docs state it is “not compatible with Couchbase Lite Community Edition”. Our app pins react-native-couchbase@1.6.1 — an older binding, so this may not apply to us.
Sync Gateway is also reported to be BSL, not open source, with a use grant excluding commercial products — I could not verify that text directly (Couchbase's legal pages refused automated access). Someone should read it properly.
Before the session ends, someone should own each of these. The WhatsApp one is not a rebuild question at all — it needs a decision this week on how our survey invitations are classified (marketing, utility or service), because that determines the rate we start paying on 1 October.
Volume fill and growth rate from CloudWatch, 21 Sep 2026. WhatsApp change verified at developers.facebook.com, 21 Sep 2026 — note several vendor blogs claim a “1,000 free service messages per month” allowance that does not appear in Meta's documentation. Licence reading is from published Couchbase terms and is not legal advice — it needs a lawyer, not an agent.
The larger financial question
The native app costs more than the infrastructure it's blamed for
We have been discussing the cheaper problem. The app's maintenance is twenty to fifty times its hosting, and it serves one customer. But it is not kept for its features — it is kept because it can interrupt people, and that is what makes a panel answer.
The app can't currently ship an update
mobapp declares targetSdkVersion = 35. Since 31 August 2026 Google Play has required API 36 for any new app or update. An extension to 1 November 2026 can be requested in Play Console.
Existing installs keep working and it stays available to new users — but until someone bumps it, no update can be published at all. Last commit was 1 December 2025.
The blocking question turned out to have an answer
We asked whether the app uploads offline-captured answers without the user reopening it. It does not — and it never could.
- UIBackgroundModes in the iOS plist is an empty array; the remote-notification handler is commented out
- No headless task on Android, no WorkManager, no foreground service
- A push arriving with the app closed draws a tray notification and nothing else
So nothing is lost by moving to a PWA. The constraint we were most worried about isn't one.
And the uncomfortable one
The app has no offline write durability at all. Answers are POSTed directly; on failure the code resets a React state variable. There is no outbox and no queue. The Couchbase replica runs server → device only.
We are paying for an offline-first sync tier that does not give us offline answering. Neither does the Chatapp — its service worker is configured with handleFetch: false and exists only to carry push.
So if offline capture is a product requirement, it has to be built from zero — on whichever client survives. That is a genuine piece of work, but it is a week of outbox code, not a replication engine. And it is the same week whether we keep the app or not, which removes it as an argument for keeping the app.
The €15–35k range is from published practitioner estimates, not your timesheets — worth replacing with the real figure in the room. Note the app has had no commits in 2026, so the figure you have actually been paying this year is probably far lower — and the compliance debt is the other side of that coin.
Reach · the uncomfortable arithmetic
Push can't win us an audience — but it's the only thing that makes one answer
What this settles
This funnel is why the Chatapp's own notification delivery is very low while the app and the chat channels are near 100%. Each step is a place a respondent silently drops out of being reachable — and the drop-outs multiply.
The constraint nobody mentions
Everyone worries about iOS. But iOS is 7% of respondents in Kenya, India and Bangladesh. Meanwhile Opera is 32% of mobile browsing in Kenya and 22% in Nigeria — and Opera Mini's proxy mode cannot receive web push at all. In our markets that is the bigger hole.
And the hybrid actively blocks the fix
Links tapped inside WhatsApp or Facebook may open in an in-app browser. Android WebView does not implement the Push API — so those users can't be subscribed, and on iOS can't add to home screen either. Worth a 30-minute test on a real handset before we choose.
The constraint that outranks the rest
Notification reach isn't a feature of the product. It is the product.
Response rates vary hugely by region — the Middle East, the Sahel and Türkiye have completely different phone, connectivity and OS profiles. So response rate is the wrong number to design against. Notification delivery is the right one, and it is close to binary.
That gap is the entire reason we run a native app and the social channels alongside the Chatapp. Not for features. For the ability to interrupt someone. If respondents have to open something manually to check for new questions, the answer rate falls to a few percent.
So the test every option has to pass
Not “can respondents still answer”, but “can we still reliably reach them when a new question lands”. Anything that trades a working notification path for architectural tidiness is a bad trade, however much it saves in hosting.
The distinction that makes this tractable
Install friction is an acquisition problem — it's why cold audiences never reach the app. But the customer still on the app has an already-enrolled panel. For them the question isn't “will they install something”, it's “can we move people who already installed once onto something we still notify them through”. That is a far easier question, and a different one.
What this does to the options: A and B don't touch notifications at all — they're the sync and cost layers, and they're safe on this axis. C and D must carry an explicit notification plan or they fail, and “it's a PWA now” is not one. That plan is the next slide but one.
A defence of something we dislike
The hybrid isn't the mistake. The implementation is.
Notify on a channel people already have, then hand off to a richer surface — that is what the evidence says a reach-constrained platform should do. Every mature humanitarian data platform works this way.
Keep the pattern
- Reach via a channel with real delivery receipts — WhatsApp, SMS
- Answer on a surface we control and can change without anyone's approval
- Push as an upgrade for repeat respondents, not the entry point
Fix the seams
- Our canonical internal message format is a vendor's. Meta Cloud webhooks are translated into Twilio's field names — Body, NumMedia, MediaContentType0 — and that becomes the domain model
- Question rendering is forked three ways (Messenger, WhatsApp chat, WhatsApp Flows), each with vendor caps like “13 quick replies” baked into business logic
- The 24-hour window is handled in 8 places, inconsistently
- The handoff leaks users into in-app browsers where nothing can be improved
The design move, corrected twice: we don't need to build this. uPublish already is it — 18 actions, 10 subscribers that resolve who needs to know, and a one-method publisher interface, already driving both documents and push to APNs/FCM/web-push. The work is to bring WhatsApp, SMS and Telegram in as publishers rather than leaving them in botkit. The unglamorous prerequisite is identity: the platform identifier exists in four incompatible shapes across four repos with no shared constant.
Has the ground moved?
Web push in 2026: better than when we built the Chatapp, still gated on iOS
| Question | Answer, Sept 2026 |
|---|---|
| Does web push work on Android? | Yes — same FCM transport a native app uses. Widely available since 2023. |
| Does iOS still require add-to-home-screen? | Yes. Unchanged through Safari 27 beta. Checked every release. |
| Is delivery quality worse than native? | No — on iOS it rides the same APNs as your app. The gap is the opt-in funnel, not the transport. |
| Do we need an Apple developer account? | No. Standard web push on iOS is free. |
| Can we prompt users to install on iOS? | No. No install prompt exists — instructional UI only. |
| Would a vendor (OneSignal, Airship…) fix iOS? | No. They wrap the same constraint. They buy convenience, not capability. |
The one that undercuts “native push is reliable”
Low-end Android OEMs aggressively kill background processes — Tecno, Xiaomi, Oppo, and Samsung among the worst. The standard fix is integrating each OEM's own push SDK. A React Native app doesn't get that for free either. Our native app is probably losing notifications the same way.
No channel gives us delivery truth
Neither web push nor FCM confirms a message was rendered — only that the push service accepted it. We cannot distinguish “ignored the survey” from “never got it”. For a research instrument, that is a methodological hole, and an argument for keeping a receipted channel as the system of record for reach.
The cheap win
A Trusted Web Activity — a thin Android shell around the existing Chatapp — gives native-grade Android push and a Play Store presence, while keeping one web codebase. That covers 82–93% of respondents in four of our six markets.
Platform risk
Why Meta keeps breaking things — and what actually helps
It isn't bad luck or bad engineering. Meta hard-sunsets each Graph API version two years after its successor ships, and ships roughly three versions a year. On top of that sit template approvals, quality ratings, the 24-hour window and number-level blocking — each able to break us without a code change on our side.
And the churn isn't only technical. On 1 October the commercial terms change too — service messages inside the 24-hour window stop being free. That is the same dependency showing up as a cost line instead of a bug.
What doesn't help
Switching to a different Meta-owned channel. Adding a BSP in front. Both change who absorbs the churn, not whether it happens.
What does
An adapter boundary, a contract test per channel that runs on a schedule, and at least one non-Meta channel live in production — not as a plan, but actually carrying traffic, so cutover is tested rather than theoretical.
| Channel | Cost | Forced-change risk | Verdict |
|---|---|---|---|
| Per message — and rising 1 Oct | High — 2-year sunsets | Keep; it's where people are | |
| SMS | High per message | Low | The floor. Expensive but unkillable |
| Telegram | Free to 30 msg/sec | Low — no forced migrations | Strong second channel; strong survey tools |
| Web push | Free | Low | Accelerant, not reach |
| Signal | — | — | Rule out. No business API; bulk messaging is against its terms |
On Telegram, one caution: it was blocked nationwide in India for six days in June 2026 and the courts upheld it. It's a genuinely good second channel — but never a sole channel in a market.
What we're actually choosing
Three decisions. Everything else follows.
Do we keep a bidirectional sync engine?
Survey sync is asymmetric and conflict-free: download a shared questionnaire, upload single-author answers. Nobody ever edits the same record twice. A merge algorithm would run zero times in production.
Do we keep the native app?
The mobile app and the Webapp both hold the old transport open — but only one of them has a customer attached. The background-upload worry turned out to be unfounded, so this is now a product and commercial question, not a technical one.
How much Meta exposure do we accept?
Not a yes/no. It's whether the exposure sits behind an adapter with a tested alternative running, or wired through business logic as it is today.
The four options are combinations of these three answers, and they are cumulative — B contains A, C contains B. Note that question 1 has two clients behind it: the mobile app and the Webapp. The Webapp has no customer attached to it, so that half can be decided in this room today; the app half needs a conversation outside it.
Stabilise
SafestStop the bleeding, buy time, change no architecture. Extend the disks, harvest the waste, patch what's EOL.
What we do
- Grow or archive the usync volumes
- gp2 → gp3; schedule staging off out-of-hours
- Delete stopped-instance disks, the idle load balancer, the dead replica
- Set CloudWatch retention; add free S3/DynamoDB VPC endpoints
- Migrate the 13 deprecated Lambda runtimes
- Get the Couchbase licence read
What it buys
- The capacity cliff goes away
- ~$250–300/month back, ~13% of spend
- Security posture stops degrading
- No product risk, no customer impact
~2–4 weeks of one engineer.
What it doesn't fix
- Three codebases, three channels — all still there
- RHEL 7 sync tier still under everything
- The app still costs €15–35k a year for one customer
- Meta can still break us next quarter
- The disks refill. This is a reprieve, not a cure
Do this one regardless. Every other option starts here — the only question is whether we stop here.
Swap the transport
Best value per euroA + stop using CouchDB replication as the delivery mechanism. Keep uPublish — add a publisher alongside the document one, move the clients across, retire the old publisher and the 1.8 TB behind it. Start with the Webapp: no customer attached, and it proves the seam end to end.
The change in one line
uPublish already resolves who needs to know and hands off to a publisher. Today the document publisher writes into 1.8 TB of per-user CouchDB databases that every client then replicates. We add a publisher that delivers the same events over a normal channel, and read state from an API instead of a replica.
Why this is smaller than “rebuild the sync layer”
- The seam already exists — IPublisher is one method, and a factory switches on publisher name
- The audience logic is reusable — 18 actions and 10 subscribers of clean, testable PHP, untouched by this
- No client write path, and no conflict resolution to build — both clients are pull-only, mutations already go over REST, and documents have a single writer
- Old and new publishers can run side by side, per client, with a real rollback at every step
Where the actual work is
- The read side, not the write side. Twelve server-side views become REST endpoints — minified JavaScript in PHP strings, no tests, no reference implementation. A drifted 11-view dev copy has to be reconciled first
- The Webapp also needs something to replace the change feed — its UI updates off replication events today
- The mobile app has to be ported or retired, gated by its live version spread, not by us
- The legacy sync box may not be rebuildable — CouchDB 1.6.1 + HHVM + beanstalkd on Ubuntu trusty, GPG keys from a keyserver network dead since 2019. No rollback target if it can't
Effort — and why I've stopped giving a single number
My estimate moved four times in two days: 4–8 weeks, then 8–14, then 3–5 months, then down again on finding the publisher seam. Each move followed a fact, not a mood — but a figure that unstable shouldn't be planned against.
What's solid is the inventory. Estimate from that, in the room, with people who know the code:
| Server-side design views | 12 |
| Document types | 16 |
| uPublish actions | 16 |
| Replication clients | 2 |
| Repos needing code changes | ~13 of 18 coupled |
One client, many channels
Needs a notification planA + B + retire the native app. Only defensible if the notification path survives it — so that's what this slide is mostly about.
How notification reach is preserved
| Android 82–93% of respondents in our markets | Trusted Web Activity — a thin shell around the Chatapp. Native-grade push on the same FCM transport, store presence kept, one web codebase. This is a genuine like-for-like swap |
| iOS 7–24%, except Lebanon at 36% | Add-to-home-screen, then web push rides the same APNs our app uses — no delivery-quality penalty once installed. The cost is a two-tap instruction, to a panel that already installed something once |
| Everyone else | WhatsApp, SMS or Telegram stay the notify layer where they fit better — now as uPublish publishers rather than botkit special-cases |
The risk, stated plainly
The iOS home-screen step is a real cliff, even for enrolled users, and we cannot prompt for it programmatically — only show instructions. If the app customer's panel is iOS-heavy, C gets materially worse. Pilot this with one cohort before committing; don't reason about it from percentages.
Other difficulties — smaller than we thought
- Cleared Rewards, vouchers and tickets are not in use — out of scope
- The one real gap: respondent-side entry of the extra param. The customer's external points system needs it, and that field exists only in the app
- The Chatapp is one group per subdomain; the app manages several
- Opera Mini can't take web push at all — 32% of mobile browsing in Kenya, 22% in Nigeria
- Requires the customer's agreement and a written sunset date
Why the points-system gap is small
The webhook itself is server-side and channel-agnostic — AnswerModel::_executeWebhook fires on every answer, reads external-url off the focus group, and doesn't care which client produced it. It already works for Chatapp answers today. The owner-side config UI also already exists, in the Webapp. What's missing is only the respondent-facing field.
What it buys
One respondent codebase instead of three. The app's maintenance and store-compliance cycle stops. CouchDB and the 2 TB disks switch off. Meta breakage becomes an adapter problem.
Rebuild the estate
Most radicalC + retire the Beanstalk generation entirely. Finish the migration somebody already started: serverless services, managed data, no long-lived fleet.
What we do
- Forty instances become a handful of managed services
- Staging stops being a 24/7 mirror and becomes ephemeral
- Single data model, one authoritative store
- The rte-platform stack's patterns become the house style rather than an exception
What it buys
The operational surface collapses. Most of the $2.3k/month goes, but more importantly the standing maintenance tax goes — no OS patching, no EOL platforms, no snapshot sprawl. For a team this size that is the real prize, not the hosting bill.
Difficulties — and they are real
- 9–18 months. Long enough that priorities will change underneath it
- Every service rewritten is a service that can regress; some are 2017-era and nobody remembers them
- Highest opportunity cost — this is most of the team's capacity for a year
- Big-bang rewrites have a poor track record, and ours would run while the business needs new features
- The saving is ~$27k/year of hosting. A year of engineering costs multiples of that
The honest case against
If C is done well, D's remaining benefit is mostly tidiness. The strongest version of D isn't a programme — it's a standing rule that anything we touch gets migrated, and we let the old generation die by attrition.
Side by side
The four options against what we care about
| A · Stabilise | B · Swap transport | C · One client | D · Rebuild | |
|---|---|---|---|---|
| Effort | 2–4 weeks | Estimate from the inventory — see slide 17 | 9–18 months | |
| Infra saving | ~$250–300/mo | + flattens the curve | + most of the sync tier | Most of $2.3k/mo |
| Maintenance saving | — | Deletes a service | €15–35k/yr | + the whole patching tax |
| Fixes capacity cliff | Defers it | Yes | Yes | Yes |
| Reduces Meta exposure | No | No | Yes | Yes |
| Effect on notification reach | None | None — safe | Must be designed for | Must be designed for |
| Reduces channel sprawl | No | No | Yes | Yes |
| Needs customer sign-off | No | Yes, to finish | Yes | Yes |
| Risk | Very low | Moderate, contained | Moderate | High |
| Reversible? | Fully | Largely | Partly | No |
If we want one sentence
A is hygiene. B is a staged swap behind an interface that already exists, it touches notifications not at all, and its Webapp half can start tomorrow. C is B plus a notification plan that has to be piloted, not assumed. D is C plus a year we probably shouldn't spend.
But this is genuinely arguable
The case for D is that attrition never actually happens — teams say “we'll migrate as we touch it” and then don't, and in three years we're having this meeting again with an EOL RHEL 8. The code read gives that argument some weight: the Chatapp has been on the right architecture for a while, and nobody finished the move.
If we choose A → B → C
What the first ninety days look like
| When | Do | Why now |
|---|---|---|
| This week | Classify our WhatsApp traffic — marketing, utility or service — and re-cost it | Billing changes on 1 October. Not a rebuild question; it lands whatever we decide |
| Week 1 | Pull the real bill — Cost Explorer grouped by usage type, not service | Settles this deck's biggest uncertainty before a euro is spent |
| Week 1 | Query user_platform for the live app version spread | Sets Option B's real timeline. A SQL query, not an estimate |
| Week 1 | Ask the app customer what they actually need — rewards and vouchers especially | A short conversation that decides a five-figure annual line |
| Week 1 | Get the Couchbase licence terms read by counsel | Compliance exposure, independent of any migration |
| Week 1 | Extend or archive the usync volumes | The clock that doesn't wait for our decision |
| Week 1 | Test on a real low-end handset: does a link from WhatsApp reach a push-capable browser? | Materially changes the migration design; costs half an hour |
| Weeks 2–4 | Harvest the waste; migrate deprecated runtimes; set log retention | Pays for part of the rest |
| Weeks 2–10 | Write the second publisher and move the Webapp onto it — six views become endpoints, plus a change channel | The half of B with no customer decision attached. Proves the seam end to end |
| Then | Retire or port the mobile app, then drop the document publisher and the 1.8 TB behind it | Gated by the customer conversation and the live app version spread |
| Weeks 4–12 | Outbox in the Chatapp; delivery service with two adapters live | The rebuild proper — with a non-Meta channel actually carrying traffic |
| Before starting C | Agree a written sunset date with the app customer | Indefinite coexistence is the dominant failure mode of migrations like this |
Note what's in week one: four questions and a disk extension. None of it commits us to an option — but all of it makes the option choice better-informed. If we leave the session having only started these, that's a good session.
Intellectual honesty
What I couldn't verify, and what would change my mind
Gaps in this deck
- I got the consumer list wrong, and the correction came from the team, not from me. The sweep of all 81 repos has since checked the negatives, and the two extra candidates it raised both turned out to be retired. The most likely remaining consumers are things not in the org at all — console-created Lambdas, BI/export tooling, partner integrations, and operator scripts
- Nothing here was observed at runtime. No live host, no CouchDB logs, no view-access statistics. So we don't know which of the 12 views are actually hot — one, open_vouchers, is queried by no client at all and may be dead
- The fan-out arithmetic is structural, not measured. It comes from reading code paths. Pull the real database and document counts before committing to a number
- Answerable The live app version spread — the thing that actually sets Option B's timeline — is a SQL query, not an unknown. user_platform holds per-device install records with platform, version and push token, and can be scoped by active. Worth running before anyone commits to a date
- Costs are list-price estimates. RDS is inferred from memory metrics and is 26% of the total — if those two databases are Multi-AZ, the estimate is badly low
- Two readings of the app's target SDK disagree (34 on master, 35 on the default branch). Either way it is below the 36 required to publish — but check which branch actually ships
- WhatsApp template approval state lives in the admin database, not in code — an unquantified migration risk
- Closed The write path is now verified. Webapp edits go over REST to the webservice, which persists to MySQL and then calls POST /sync-requests on uPublish to refresh the document copy. The local save() is an optimistic cache write, not the real mutation. Documented in product-development/docs/ARCHITECTURE.md
- Elastic Beanstalk returned empty results despite 25 instances tagged to it — unexplained
What would change the recommendation
- The legacy sync box can't be rebuilt from its Dockerfile → we have no rollback target, and the migration has to be one-way and very carefully staged
- Respondents are spread across many old app builds → Option B's tail stretches past six months regardless of engineering effort
- Rewards and vouchers turn out to be central to the app customer → C's cost rises materially; that UI does not exist in the Chatapp
- More customers want the app next year → the whole calculus inverts; keep it and invest properly
- The real bill shows compute, not storage, dominating → D gets considerably stronger
I'd rather be corrected tomorrow morning than have this read as more certain than it is. The three premise corrections on slide 3 are the findings I'd defend; the effort estimates are the ones I'd expect to be wrong.
For the room
Five questions to leave with
1. Will a home-screen Chatapp notify as reliably as the app does?
Everything hinges on this and nothing else answers it. One cohort, one region, measure delivery — not response. Until we have that, Option C is a hypothesis.
2. How iOS-heavy is the app customer's panel?
Android has a like-for-like swap in TWA. iOS needs a home-screen step we can't prompt for. This ratio decides whether Option C is comfortable or risky.
3. Do we believe in migration-by-attrition?
If not, D deserves a serious hearing rather than a polite one.
4. Is one non-Meta channel in production a requirement or a nice-to-have?
This is a strategic posture, not an engineering preference.
5. Who owns the sunset date?
Migrations like this fail by drifting, not by failing.
And one thing to just do
Extend the usync volumes this week. It is the only item here with a deadline we didn't choose.
Prepared by Upii for the Upinion technical session, 22 September 2026 · Infrastructure findings from a read-only AWS sweep on 21 September 2026; no production resources were modified · External research is sourced in the accompanying briefs · Cost figures are list-price estimates, not billed cost.