From Midnight Patches to Minutes: Rewiring MyOrbit’s 5 TB Pipeline at Home

From Midnight Patches to Minutes: Rewiring MyOrbit’s 5 TB Pipeline at Home

5/18/2025

Late 2021. Jakarta was still half‑asleep behind masks, and my “office” was a dining‑table‑plus‑laptop wedged between the fridge and a stack of baby bottles. Telkom Indonesia is the country’s largest telecom operator, and MyOrbit is its prepaid 4G/LTE home‑internet modem line.

My project manager had the uncanny talent of declaring, “Sorry, I have another meeting,” just as every call with our grumpy Telkom stakeholder turned prickly. That morning Skype lit up:

Boss: “Revan, you’ve been crushing it on our team, so I’m shifting you to the MyOrbit project. The senior who wrote the original—back when it was a tiny API—has been pulled onto a strategic initiative, and the scope here has exploded. Telkom’s numbers look all wrong. They need fresh eyes—yours.”
PM (private chat): “I might have to jump to another client call—can you lead today’s review?”

A promotion wrapped in a landmine. The legacy API once worked for a simpler flow; now it drowned in complexity. Gigabytes made it sweat; terabytes made it cry. Telkom’s project lead didn’t hide his irritation—every sync failure cost him another midnight report.

A new Telkom analyst—soft‑spoken, organised, curious—joined our call. Finally, a friendly face. Together we sketched the modem journey on a shared draw.io canvas:

ORDER → PICKUP → ONPROGRESS → DELIVERED → REGISTERED → ACTIVATED

Events landed out of order, spelled four different ways, or never showed at all. No wonder activation read 0.8 %.

The three promises I muttered after that call

Move the data fast enough that Finance wouldn’t flinch, make the loads smart enough to heal themselves so I could sleep, and enforce one event vocabulary—DELIVERED means delivered, no exceptions.

While the stakeholder sighed and my PM vanished, I opened a blank draw.io sheet. BigQuery Data Transfer Service could haul bulk data, a tight Cloud VPN would keep RDS private, Apache Beam on Dataflow could enrich rows mid‑stream, and Cloud Composer would conduct retries and alerts. If both sides signed off, Telkom’s execs might finally sip kopi without frowning at charts.

Convincing everyone took a week of joint calls and diagram tweaks, but by Friday evening the plan was stamped “approved.” I scrapped the brittle Lumen API—straight into the bin—and began wiring a fresh set‑up with Google’s console, the gcloud CLI, and an embarrassingly large mug of coffee. Alone.

Building the River From a Dining‑Room Desk

Here’s the architecture I mocked up (cleaned for public viewing).

Before a single line of configuration, I spent two days spelunking the BigQuery Data Transfer Service docs. Could DTS really move MySQL data for free? A quick PoC on a staging snapshot said yes. That win triggered the bigger hurdle: we’d need a site‑to‑site VPN—DTS won’t talk to a private RDS otherwise.

I jumped on a call with our DevOps lead. He owned the AWS side; Telkom’s infra team owned Google Cloud. Three meetings, two firewall rule audits, and a fresh draw.io diagram later, the tunnel was up and a lonely /32 IP was allow‑listed on both ends.

Mid‑sprint a new requirement slammed in: Telkom wanted store, witel, and agency metadata from another API. The old script couldn’t handle multi‑source joins. I reached for Apache Beam on Dataflow and Cloud Composer to keep my sanity.

The day Security finally said “approved”

After two firewall reviews and one politely tense call, Telkom’s security team blessed our Cloud VPN plus networkAttachment. That single /32 IP became our secret handshake—RDS stayed private, DTS had a doorway, and the grumpy stakeholder cracked his first smile.

The free freight train

Pointing DTS at RDS felt almost too simple: tick encryption = FULL, set the hourly schedule, and watch the first load grab only the newest slice.

Before this, catching a gap meant shuffling an API parameter by hand—start_date=YYYY‑MM‑DD&end_date=YYYY‑MM‑DD—and babysitting it overnight while rows trickled in. After this change, Composer dutifully crawled backward through the calendar, replaying any day that looked thin. It took three caffeine‑soaked days before every missing partition turned green. The mood when the last back‑fill job succeeded was pure bodily relief—like finally clearing a stubborn clog.

Finance still loved the price tag: $0 ingestion.

Giving each row a passport

Beam became my side‑kick. As soon as RAW partitions landed, Dataflow streamed them, called the Warehouse Stock API, and stamped an in_warehouse flag. Thousands of HTTP calls per second—no sweat—and Telkom finally knew where every modem was hiding.

Keeping the band in time

Cloud Composer watched for new partitions like a stage manager. If DTS sneezed, Airflow reran the act; if Dataflow lagged, I got a failure ping in Skype via an Airflow webhook long before any angry email chain woke up.

Seeing the heartbeat — Cloud Logging

Every job, retry, and grumpy exception funneled into Cloud Logging. A quick log-based metric told me at a glance whether yesterday’s load moved the right number of rows, and Error Reporting grouped stack traces so I could squash the noisy ones once and for all. After wiring alerts, the pipeline felt less like code and more like a living, monitored service.

Telling the story—Looker Studio (Telkom BI Team)

My remit stopped at a clean ENRICHED dataset. Telkom’s BI crew wired Looker Studio to those tables and built dashboards execs now refresh obsessively. Watching their first live demo felt like passing a relay baton and seeing the anchor sprint to the finish.

Deployment Day (still in pyjamas, PM “in another meeting”)

08:00 — VPN goes green—I exhale; the tunnel lives.
09:15 — First DTS run: five terabytes humming through the VPN—knuckles white on my mug.
11:06 — Boss on Skype: “Progress?”—“Half‑way, zero errors.”
13:12 — Full load done; Dataflow flying—my kopi is stone‑cold, but tastes like victory.
15:00 — Looker tiles flip—Activation 0.8 % → 34 % → 61 % → 76 %. Analyst drops a 🎉; stakeholder mutters “Finally.” I fist‑pump under the desk, realising I skipped lunch.

Impact (Measured in Calm Emails and Saved Weekends)

A week after go‑live, Skype stayed eerily silent. No 2 a.m. “data gap” pings, no frantic Excel patch jobs. Telkom’s analysts began mornings with a quick glance at Looker and moved on. I reclaimed weekends; they reclaimed sanity.

The PM re‑appeared just in time for the success call. I just smiled and poured another coffee.

Lessons Breathing Steam from My Mugs of Home‑Brew

Over the months I learned that managed services aren’t a luxury—sometimes they’re your only lifeline. A handful of well‑placed configs beat thousands of heroic lines of code. I also discovered how a single misspelled status can torpedo a KPI, and that the best alert is the one that fires while you can still fix things in your pyjamas. Most of all, I felt the mood in every call shift the moment data matched reality; nothing soothes a grumpy room faster than a dashboard you can trust.

Next quest? Maybe Datastream if Telkom craves sub‑minute freshness—ideally from a real office instead of my dining‑table command centre.

Need Beam snippets or more remote‑firefighting tales? Ping me—always happy to share.