Observability (New Relic APM)

Issue #631. TailSpend reports to New Relic from the service Lambdas, the web SPA, the desktop renderer, and the iOS and Android apps, so a slow or failed user action can be followed from the client into the Lambda that served it. It is additive: Pino JSON logs, API Gateway access logs, request ids, and the CloudWatch cost and abuse alarms (MonitoringStack, cost controls runbook) are unchanged and remain the source of truth for logs.

Everything is off unless configured, per stage and per client. With no configuration the Lambdas, the web/desktop bundles and the mobile JS behave exactly as before.

Enabling a Stage

Roll out to beta first, then prod, through the normal deploy and EAS release workflows. One New Relic account (free plan) serves every stage; stages are separated by entity name.

1. New Relic setup (once). Create, in the New Relic UI:

   • a browser app per stage named exactly TailSpend Web (<stage>) (copy/paste method; note its application id and browser license key);
   • a separate browser app per stage named TailSpend Desktop (<stage>) for the Electron renderer (note its application id and browser license key);
   • mobile apps per stage named exactly TailSpend iOS (<stage>) and TailSpend Android (<stage>) (note each application token);
   • a User API key for uploads and NerdGraph.

   The dashboard and alert NRQL rely on these names.

2. AWS SSM (per stage, same tree as the OAuth config):

   /tailspend/<stage>/deployment-environment/NEW_RELIC_ACCOUNT_ID             String        enables Lambda APM + browser config
   /tailspend/<stage>/deployment-environment/NEW_RELIC_LICENSE_KEY            SecureString  ingest key, read by the Lambda extension at runtime
   /tailspend/<stage>/deployment-environment/NEW_RELIC_BROWSER_APPLICATION_ID String        browser app id
   /tailspend/<stage>/deployment-environment/NEW_RELIC_BROWSER_LICENSE_KEY    String        browser key (public: it ships in the page)
   /tailspend/<stage>/deployment-environment/NEW_RELIC_DESKTOP_APPLICATION_ID String        desktop browser app id
   /tailspend/<stage>/deployment-environment/NEW_RELIC_DESKTOP_LICENSE_KEY    String        desktop browser key (public)
   

   The license key is never in a template or function configuration: the extension reads it via NEW_RELIC_LICENSE_KEY_SSM_PARAMETER_NAME, and each function may read only its own stage's parameter (decrypted only through SSM).

3. GitHub — repository (or deploy-<stage> environment) secret NEW_RELIC_USER_API_KEY: uploads web source maps and applies dashboards and alerts. Optional variable NEW_RELIC_ENABLED=false is the kill switch.

4. EAS environment variables (per environment: development, preview → beta, production → prod):

   cd apps/mobile
   npx eas env:create --environment preview --name EXPO_PUBLIC_NEW_RELIC_IOS_APP_TOKEN --value <token> --visibility plaintext
   npx eas env:create --environment preview --name EXPO_PUBLIC_NEW_RELIC_ANDROID_APP_TOKEN --value <token> --visibility plaintext
   npx eas env:create --environment preview --name NEWRELIC_USER_API_KEY --value <user key> --visibility sensitive
   

   Use sensitive, not secret, for the user key: EAS never exposes secret-visibility variables to build scripts, so the agent's upload steps would silently skip.

5. Deploy beta (Deploy workflow), then ship a new native build (EAS Build) — the agent is a native module, so an OTA update cannot add it. Then verify (see Acceptance checks) before repeating for prod.

How It Works

Lambdas

ServiceLambda attaches New Relic's Node.js 24 ARM64 layer (slim variant, pinned in lib/config/new-relic.ts), sets the function handler to the layer's ESM wrapper, and points NEW_RELIC_LAMBDA_HANDLER at our index.handler. Every service Lambda — API routes, the scan worker, the scan reconciler and the recurring-expenses tick — goes through this construct. The layer's extension sends telemetry straight to New Relic at the end of each invocation; nothing goes through CloudWatch.

• Invocations (AwsLambdaInvocation) carry stage, release (commit), requestId (the API Gateway id, the join key to the access log) and route (e.g. POST /groups/{groupId}/expenses).
• Errors (AwsLambdaInvocationError). Only unexpected failures are APM errors: the 5xx path in mapDomainError, unexpected scan-pipeline failures, reconciler and per-row recurring-expense failures. Validation errors, 4xx domain errors and quota denials (429) are ignored by status code (NEW_RELIC_ERROR_COLLECTOR_IGNORE_ERROR_CODES) and never noticed.
• Dependencies. Every AWS SDK call is a span named AWS/<Client>/<Command>, added by traced() at client construction. The agent's own AWS SDK instrumentation does not apply (see Compatibility).
• Custom spans Custom/scan/read-text (Textract) and Custom/scan/structure (Bedrock) split a scan's time.
• Log correlation. When the agent is active, every Pino line gains trace.id and span.id, so a trace can be found from a CloudWatch log line (filter trace.id = "…") and vice versa. Logs are not forwarded to New Relic.
• Async scans. The scan route (and the reconciler on redispatch) puts the current trace context in the worker's invocation payload (traceHeaders), so the worker continues the request's trace. EventBridge Scheduler recovery checks carry no trace context; there the scanJobId attribute (on the worker, reconciler and mobile receipt.scan event) is the correlation key. The worker records scanOutcome (completed, failed, skipped:<reason>, settled). CloudWatch also records scans.job_submitted with the opaque attempt/job IDs and scans.job_completed with the job ID. Mobile submission failures include the operation (preparing, requesting-upload, uploading, submitting, or accepting-result), API error code and attempt ID when available. These help distinguish a lost client response from a failed paid job after the short result retention expires, without recording receipt text or amounts.

The serverless extension buffers sparse invocations until its batch threshold or execution-environment shutdown. A quiet worker's telemetry can arrive well after the request completes. Correlate by scanJobId and traceId before concluding that collection failed. Dashboards and alerts scope Lambda invocations, errors and spans by aws.lambda.arn; serverless spans do not carry appName.

Web

main.tsx reads the build-time config (monitoring-config.ts) and, only when present, lazily loads the agent chunk (~19 kB gzipped) without delaying first render. The agent is composed from individual features — Ajax, JS errors, metrics, page view, page timing and soft navigations. Session replay, session traces, logs and user-action events are deliberately excluded.

Distributed tracing headers (traceparent, tracestate, newrelic) go only to the API origin; API Gateway's CORS allowlist admits them (apps/infra/lib/config/cors.ts, asserted by api-integration-tests). Each build sets the release (commit) and stage attribute, and the deploy workflow uploads every dist/assets/*.js.map against the URL it is served from under release tailspend-web@<commit>.

Desktop

Web and desktop share @repo/browser-monitoring, with separate config and agent entry points so the agent is loaded asynchronously before first render. The desktop renderer explicitly selects platform: "desktop"; the web config still refuses to start inside Electron, preventing accidental web reporting. The same feature allowlist, credential-query redaction, disabled cookies and disabled request/response payload capture apply to both clients. Tracing headers go only to the configured API origin. Desktop also removes local installation directories from file:// URLs so OS usernames do not appear in telemetry; bundle filenames, stack line/column and HashRouter routes remain available.

npm run env:pull --workspace=desktop -- --stage <stage> reads the account id and the two NEW_RELIC_DESKTOP_* SSM parameters above into public VITE_NEW_RELIC_* variables. Missing parameters disable monitoring as a unit; SSM permission/network failures fail the command. There is no fallback to the web application id. NEW_RELIC_ENABLED=false skips those reads and generates an unmonitored build. Desktop Release passes the repository variable through and regenerates this config before building the installers.

Telemetry carries platform=desktop, stage, and the checked-out commit as the application version and tailspend-desktop@<commit> release. The renderer's CSP pins connect-src to its configured API and Cognito origins plus https://bam.nr-data.net; agent scripts are bundled locally under script-src 'self'. Packaged file:// pages and HashRouter use the same agent features as web. Desktop source-map uploads and main-process/native crash monitoring remain follow-ups; renderer errors are reported without uploaded desktop source maps.

Mobile

The newrelic-react-native-agent Expo config plugin (in app.base.json) applies the Android Gradle plugin, the iOS dSYM / source-map build phase and android/app/newrelic.properties on every expo prebuild --clean, so nothing depends on hand-edited native projects. startMonitoring() runs at the top of the root layout and starts the agent only when the platform's EXPO_PUBLIC_NEW_RELIC_*_APP_TOKEN is set and the native module exists:

• Expo Go / web: inert. Expo Go has no native module; the web build uses the no-op telemetry.web.ts.
• Development builds: report like release builds if the development EAS environment has tokens; leave them unset to keep dev data out.
• Maestro builds swap in the no-op module (metro.config.js).

Session attributes: buildNumber, easUpdateId, easChannel, runtimeVersion. TailSpendAction events (expense.create, settlement.create, receipt.scan) record outcome, durationMs and counts only — never labels, amounts, names or receipt text.

Source maps and symbols. The JS bundle of a store build is identified by its native app version: the agent's build steps upload the iOS main.jsbundle.map and dSYMs, and the Android JS map (and R8 mapping when minification is enabled), using NEWRELIC_USER_API_KEY and the app token. An OTA update reports its update id instead (jsBundleId()), and the EAS Update workflow uploads that update's maps under the update id via apps/mobile/scripts/upload-update-sourcemaps.mjs. Expo OTA hash filenames are normalized to index.bundle in reported stacks, with matching index.bundle.map uploads; Hermes bytecode offsets are preserved.

See 11 October validation results for captured source resolution, measured Lambda overhead and remaining gates.

Data and Privacy

Never sent: auth tokens, request or response bodies, receipt images or text, expense labels, amounts, participant names, emails.

• Lambda: request.headers.*, request.parameters.* and response.headers.* are excluded (headers and query strings can carry invite tokens). Only opaque ids are added as attributes.
• Web: no request/response payloads, no cookies or local-storage session id, code/state/token/invite query values redacted, no session replay.
• Mobile: response-body capture off; the agent's console mirroring is undone after start-up so console output stays on the device.

Cost and Sampling

Target: the free plan — 100 GB ingest per month and one full-platform user (everyone else uses free basic access). No paid feature is enabled.

• Volume. A Lambda invocation sends one transaction event plus its spans (a few kB). Logs are not forwarded (NEW_RELIC_EXTENSION_SEND_FUNCTION_LOGS and application-log forwarding are off), so CloudWatch log volume is not duplicated. Interaction traces (mobile) and session replay/traces (web) are off. The agents' default adaptive sampling applies to traces.

• Usage alert. The prod policy alerts at 80 GB month-to-date (NrMTDConsumption); the dashboard's Usage page breaks ingest down by source.

• When the allowance runs out, New Relic stops accepting data until the monthly reset; the apps are unaffected (agents drop data they cannot send) and CloudWatch alarms keep working.

• AWS overhead. The layer adds cold-start time (the slim layer is used to keep this down) and the extension's end-of-invocation flush adds billed duration. Measure on beta before prod, comparing a week before and after:

  # CloudWatch Logs Insights over /aws/lambda/tailspend-beta-*
  filter @type = "REPORT"
  | stats avg(@initDuration), pct(@initDuration, 95), avg(@billedDuration), pct(@billedDuration, 95) by bin(1d)
  

  Record the results in the PR that enables prod. The extension's outbound HTTPS has no NAT cost (the functions are not in a VPC).

Alerts

apps/infra/scripts/apply-new-relic.ts runs after every deploy and upserts the stage's dashboard TailSpend (<stage>) and alert policy TailSpend (<stage>) (edits made in the UI are overwritten). The split with CloudWatch avoids double notification:

One-time: in New Relic, create a workflow that notifies the operator for issues from the TailSpend (prod) policy (and beta, if wanted). The script does not manage notification destinations.

When an alert fires: open the dashboard page for the signal, pick an example trace or error, and use its requestId / trace.id to find the matching CloudWatch log lines.

Compatibility and Known Gaps

Validated while building this (October 2026):

Gaps to keep in mind:

• Same-version store builds. The agent registers an embedded bundle's map under the app version, and the first map for a version wins. Rebuilding the same version (only the build number changes) keeps the first build's map, so bump expo.version for store releases that change JS.
• Cognito trigger Lambdas (AuthStack) are not instrumented.
• Desktop main process is not monitored.
• Recovery checks from EventBridge Scheduler start new traces (correlate by scanJobId).
• Release env on Lambdas. Deployments resolve TAILSPEND_RELEASE from the last backend-affecting commit (scripts/service-release.mjs) instead of the moving deployment SHA. Client source/docs/CI-only merges retain this tag and avoid metadata-only Lambda configuration updates. Infrastructure, shared build and lock changes conservatively advance it; frontend release IDs still use the deployment commit. Local callers can set NEW_RELIC_SERVICE_RELEASE, with GITHUB_SHA retained as the fallback.
• The NerdGraph script and the dashboard queries cannot be exercised in CI; run the first apply against beta and check its output.

Acceptance Checks

On beta, after enabling and deploying:

1. Web → API trace. Create an expense on the web app. In New Relic, Browser → TailSpend Web (beta) → AJAX shows the request; its trace continues into tailspend-beta-core-api-app (the app API now uses the REST core handler).
2. Native → API trace. Repeat on an iOS and an Android build; the MobileRequest links to the Lambda trace; a TailSpendAction event with name = 'expense.create' appears.
3. Receipt scan. Scan a receipt on mobile; the worker transaction continues the POST …/scan-jobs trace, has scanOutcome = 'completed', and shows the Custom/scan/* spans. receipt.scan carries the same scanJobId.
4. Recurring job. After the next hourly tick, the dashboard's recurring widget shows ran / failed.
5. Controlled failures. Throw in the browser console (setTimeout(() => { throw new Error("nr-test") })) and confirm the error resolves to source. On a test build of each platform, trigger a JS error and a native crash (the agent's crashNow() API) and confirm both are symbolicated. For the backend, invoke tailspend-beta-scan-job-worker from the Lambda console with {}: the validation error appears on its transaction. Pick any API log line in CloudWatch and confirm its trace.id opens the matching trace.
6. OTA. Publish a preview EAS update and confirm a JS error from it resolves to source (the workflow logs the uploaded update id).
7. Disabled path. Deploy beta with NEW_RELIC_ENABLED=false and confirm functions have no layer and the app works; re-enable.

Rollback

Troubleshooting

• No Lambda data. Check the function has the layer and NEW_RELIC_LAMBDA_HANDLER=index.handler, and look for [NR_EXT] lines by temporarily setting NEW_RELIC_EXTENSION_LOGS_ENABLED=true. A missing or unreadable license key parameter is the usual cause.
• Browser traces not linking. The preflight must allow traceparent, tracestate and newrelic (cors.test.ts in the integration suite checks this).
• Minified web errors. Check the deploy log of "Upload Web Source Maps To New Relic" and that the error's releaseIds match the commit.
• Mobile maps not applied. NEWRELIC_USER_API_KEY must be sensitive, not secret; check the build log for New Relic: lines.

Automated Beta Rollback Check

Run Deploy with stage beta and validate_new_relic=true to deploy without backend/web monitoring, verify the deployed state, run integration tests, restore monitoring, and verify/test again. The restoration job runs even when the disabled deployment or its tests fail. Do not cancel a running CDK deployment. The complete manual cycle shares main's pipeline concurrency group, so it waits for active main deployments and keeps new main pipelines from changing beta between its test jobs. Runtime verification checks both Service and CoreApi application Lambdas. The override is beta-only and leaves stored GitHub/EAS/SSM configuration unchanged.

This check does not disable mobile monitoring. Its workflow result must pass before recording the live rollback acceptance check as complete.

Follow-up validation on 12 October 2026

The runtime checker covers application Lambdas in both Service and CoreApi stacks, ignoring CDK helper functions. The entire manual off/test/on/test run shares the main pipeline's concurrency group, including restoration after cancellation/failure, so a normal pipeline cannot interleave with the temporary beta disable.

Backend bundles include compact linked source maps (sourcesContent: false) and NODE_OPTIONS=--enable-source-maps. The maps retain TypeScript file/line mappings without embedding dependency source text. A local minified-bundle probe resolves an exception to the original TypeScript line even after the original file is deleted. New Relic remains enabled; its Node APM source mapping uses Node's runtime maps, so the Browser source-map upload API cannot replace these maps. See cold-start measurements. Deployment and a New Relic backend-error probe are still required before marking live error resolution complete.

Lambda-specific dashboard/alert ARN filters include trailing qualifiers. This fixes the recurring-tick query when telemetry uses an ARN with a qualifier rather than the bare function ARN. People read-cost widgets and attributes are described in notifications.md.

Previously observed iOS/Android beta OCR scans have matching scanJobId values across mobile receipt.scan actions and completed worker spans. Both platforms also report beta API requests and expense.create actions. These observations do not by themselves complete native crash symbolication or the repeated rollback acceptance gate.

iOS simulator native-crash limitation — 13 October 2026

The release-mode 1.2.1 simulator app and dSYM share UUID 1DDDC07E-2186-3B0B-A0E0-A38161C7C604; the official New Relic symbol tool accepted the map upload. A controlled crashNow() probe reached New Relic as occurrence 8BB395BC-96E3-493E-B723-66B83955B1F8, after the regular app was rebuilt and relaunched. All temporary diagnostic code was removed.

The report's symbolicated flag was true, but NerdGraph returned 24 raw-address frames and no named frames. New Relic explicitly states that iOS simulator crashes cannot be symbolicated. Simulator capture, matching UUIDs and accepted uploads do not complete supported-device native symbolication acceptance. On 16 October the user explicitly deferred physical-device native symbolication proof. This limitation no longer blocks the simulator-scoped release; supported-device proof remains a tracked follow-up. Browser/mobile JavaScript source maps and request/scan tracing are separate checks.

Android native-crash evidence — 13 October 2026

A release-mode 1.2.1 diagnostic build called the installed New Relic Android SDK from a temporary native-activity hook on an isolated emulator. Occurrence 9feda524-1f1d-4d9c-8287-c753b66b6f25 reached TailSpend Android (beta) with obfuscated=false. NerdGraph returned 12 frames, including com.newrelic.agent.android.NewRelic.crashNow() at NewRelic.java:774 and TailSpend's MainActivity.onCreate$lambda$0() at MainActivity.kt:26.

Both generated native files were restored, all diagnostic code was removed, and the normal app was rebuilt/reinstalled. The EAS-signed push-validation app is separate and retains version 1.2.1/build 21; its Google sign-in, OS opt-in and push/lifecycle checks remain pending.

Deployed beta validation — 13 October 2026

The serialized beta off/test/on/test run passed at fd14e9fa8022a2128387e32535b7bb789ead23d4. Both disabled/enabled configurations verified 16 Service/CoreApi application Lambdas and the deployed web bundle. Each integration phase passed 77 tests across 21 files, with two skipped. Core API monitoring was independently rechecked after restoration, including its wrapper/layer and NODE_OPTIONS=--enable-source-maps.

The controlled scan-worker probe was rejected before repository access or domain writes. New Relic stored its TypeScript stack at apps/services/src/handlers/scans/process-scan-job.ts:35:37; request 03886aad-92cd-4414-b55c-012bdca940cc, trace a91e98436fe01c195be5dd3a3198bc0b. The error initially took several minutes to appear; query stackTrace on AwsLambdaInvocationError, rather than error.stack.

Creating the synthetic New Relic web trace test expense in the isolated New Relic Web Validation beta group returned HTTP 201. Browser AJAX, the POST /groups/{groupId}/expenses Core API invocation and DynamoDB spans share trace b31ea4e0ac0b98ccd044b6f24ea9dec9; request 43d97f36-a218-45fc-9a10-6cd82f601c5f. The existing London group was not changed.

The deployed beta dashboard includes qualified-ARN recurring filters and both People widgets. The post-restoration 14:00 UTC recurring tick carries the current release, with ran=0, skipped=0, failed=0; no due templates is valid chart data. A real iOS People request for one group measured six SDK read commands, three read units and 73.88 ms; the beta integration baseline for one group measured six commands, three units and 18.04 ms. These are small-group measurements, not proof of cost at the 50-group cap.

On iOS 1.2.1, authenticated test pushes displayed in the foreground, background and after termination. Foreground/background/cold-start taps opened notification history; a missing expense showed Expense not found. and a payload for a different recipient did not navigate. Transport probes bypass domain dispatch and do not create history. Account switching, logout/deletion/reinstall, denied permission, full domain flows and Android delivery remain separate gates.

Android beta testing exposed simultaneous /oauth2/token requests from the native browser completion and callback route: one succeeded and one failed with HTTP 400 for each one-use code. Both callers now share a PKCE-validated completion, including late duplicate callbacks. Startup waits for persisted-session bootstrap; logout/new sign-in invalidates an unfinished completion. Regression tests cover concurrency, cold callbacks, invalid state and superseded sessions. The final native version is 1.2.3, giving the final JavaScript a fresh source-map version after intermediate validation builds; native acceptance is still required on that build.

Android beta 1.2.3/build 22 now has successful foreground/background display and taps, a verified process-absent cold-start tap after preserving-data APK replacement, logout device removal and signed-out tap gating. Four notification.open actions reached New Relic; all four selected Android provider receipts were successful. iOS 1.2.3 permission denial/recovery and its recovery push also passed; the temporary group mute was restored and verified empty. A two-group web People request measured 11 read commands, 5.5 read units and 159.68 ms. These remain small-group measurements, not a claim about cap-scale performance.

Android logout recovery exposed a separate navigator render loop. The corrected 1.2.4 build uses an always-mounted navigator with protected private screens, plus the one-use OAuth completion fix. The new session-expiry/callback Maestro regression awaits native CI, and fresh rebuilt-binary acceptance remains required. Provider receipts and direct pushes do not satisfy the domain outbox, preference or account-lifecycle acceptance gates. Production Client Release and production push remain held/off; the iOS simulator named-frame scope decision was resolved on 16 October by explicitly deferring physical-device proof.

Final 1.2.5 release-mode simulator/emulator binaries compile and install. Fresh Android 1.2.4 sign-in reached the beta dashboard; New Relic recorded exactly one token HTTP 200 and no token-request error, confirming the one-use code fix. Fresh Android 1.2.5 sign-in also reached the dashboard. The callback recovery button now targets the public /sign-in route explicitly. The correctly ad-hoc signed iOS 1.2.5 simulator restored its beta session and repeated foreground banner/display/tap plus process-terminated cold-start display/tap; both Expo receipts succeeded. The earlier unsigned local simulator build lacked keychain entitlements and was replaced with the correctly signed validation build; no signing-policy change is shipped. Temporary Android signing keys/password files were removed after every build.

Final Android 1.2.5 logout and callback recovery returned to the sign-in screen without the prior navigator crash. Fresh sign-in was repeated afterward. Both stage push switches were independently rechecked absent/default-off, and Client Release remains disabled_manually.

Final Android 1.2.5 missing-expense push rendered Expense not found. A subsequent FRIEND_REQUESTED test naming a fabricated different recipient was ignored, leaving the app on the prior expense screen.

Release readiness — 16 October 2026

The user accepted simulator crash capture and deferred physical-device iOS native symbolication proof. Android named app/SDK frames, backend source resolution, web/API distributed tracing, rollback checks and measured overhead retain their previous evidence. No named iOS simulator frames are claimed.

PR #678 is merged. Its native auth/callback CI passed, and Main Pipeline run 38531264362 passed all deployment and acceptance stages. A later pipeline failed Expo Doctor after new SDK patch releases; the 1.2.6 candidate aligns those dependencies and passes all 21 Doctor checks plus the full workspace build locally.

Live beta API acceptance passed expense creation/edit/deletion and structured mentions through the outbox, actor exclusion, history retention while muted, device-token ownership transfer, People workflows and disposable account deletion. The deletion test also verifies device rows and token claims are absent afterward. These checks do not replace fresh native build/device acceptance or authorize claiming production sending and Client Release have been restored.

Real beta worker push validation additionally confirmed Android join-request and expense tap destinations, personal group-mute/category suppression with history retained, and resumed sends after restoring settings. Both native devices received the join-request alert. Fresh 1.2.6 build/device acceptance remains required; production release and sending have not been restored.

Beta push is enabled for the acceptance pilot. The user confirmed iOS navigation from Notifications into the validation group. Production sending and Client Release remain held pending fresh native candidate/device acceptance.