Meta Ads Sync Pipeline

Context

The reporting dashboard is only as good as the data underneath it, and that data lives in Meta. More than a hundred advertising accounts have to be pulled in on a schedule, each one belonging to a different client and each one authenticated with its own credentials.

Nobody should have to press a button for this to happen, and nobody should have to find out from a client that yesterday's numbers are missing.

Problem

A hundred accounts is the number where the naive approach stops working. Loop over the accounts, call the API, write the rows — and the first account with an expired token, a permissions change or a rate limit takes the entire run down with it. Everything after it in the loop simply never happens, and the failure is silent until someone notices a gap in a report.

There is a second, quieter problem. Credentials are per-account, so any shared, module-level API client becomes a piece of mutable state that concurrent workers fight over. Two tasks running at once can end up making calls with each other's credentials, which is both a correctness bug and a data-isolation one.

What I built

• Celery Beat schedules the recurring sync; Celery workers do the fetching and writing.
• Credentials for each advertising account are held in encrypted multi-credential storage using Fernet symmetric encryption, so the database never holds a usable access token in plaintext.
• A client factory builds a fresh, fully-configured Meta Marketing API client per credential at the point of use.
• Every account is synced inside its own error boundary, with the outcome recorded so a failed account is visible rather than merely absent.
• Worker and Beat both run as Systemd services on DigitalOcean, so they restart with the machine and are managed the same way as everything else on the box.

# apps/campaigns/tasks.py
# One Celery task per campaign, run concurrently by the workers.

sync_meta_campaign.delay(campaign.id)

# apps/campaigns/services/meta_api.py
# Each task builds its own Meta API instance. There is no process-global
# client, so two workers running at once can never end up sharing a session
# or, far worse, each other's credentials.

# Results go back in bulk rather than row by row.
Model.objects.bulk_update(
    rows,
    update_fields=["spend", "leads", "impressions", "clicks", "updated_at"],
    batch_size=1000,
)

Technical decisions

Why a client factory rather than one shared client

A shared client is a single object holding one set of credentials. The moment two workers process two accounts concurrently, whichever task configured the client last decides which account the other task is really querying. That is a race condition whose symptom is not a crash but wrong data attributed to the wrong client — the worst possible failure in a multi-tenant system, because it looks like a successful run.

Why isolation at the credential boundary, not the task boundary

Wrapping the whole sync task in a try block would keep the scheduler alive, but it would not save the run — the exception still aborts the loop, and every account after the failing one is skipped. Isolating per credential means the unit of failure matches the unit of work: one account's expired token costs exactly that account's data for that cycle, and the other ninety-nine complete normally.

It also makes the failures legible. Because each credential's outcome is recorded separately, "which accounts did not sync today, and why" is a question the system can answer directly, instead of one that has to be reconstructed from a stack trace.

Why encrypted credential storage

Long-lived access tokens for a hundred client ad accounts are, collectively, the most sensitive thing in the database. Encrypting them at rest with Fernet means a database dump — a backup copied somewhere careless, a restore into a staging box — is not the same thing as handing over live access to every client's advertising account.

# apps/campaigns/services/meta_api.py
# A campaign's budget lives somewhere different depending on its type.
# This resolves one true daily figure.

def resolve_campaign_daily_budget(mc, adset_map):
    """Returns (daily_budget, source).

    CBO on   -> the campaign-level daily budget
    lifetime -> None, because there is no per-day figure to report
    CBO off  -> the sum of the enabled ad sets' daily budgets
    """
    camp_daily = mc.get("daily_budget")
    if camp_daily:
        # Meta sends paise; the dashboard works in rupees
        return (Decimal(str(camp_daily)) / 100, "campaign")

    if mc.get("lifetime_budget"):
        return (None, "lifetime")

    summed = adset_map.get(mc["id"])
    if summed and summed > 0:
        return (summed, "adset")

    return (None, "adset")

Results

• Schedule: a nightly run, covering a rolling three-day window so late-attributed conversions are picked up rather than missed
• Volume per run: roughly 2,500 ad-level rows, plus about 650 campaign-level rows a day
• Unit of failure: one credential. A failed account costs that account's data for that cycle and nothing else
• Accounts whose failure can abort the run: none
• Credentials held in plaintext: none — storage is Fernet-encrypted per account
• Services under Systemd on DigitalOcean: two — the Celery worker and Beat, both restarting with the machine

What I would do differently

I would never read identity out of a campaign name again. The first version of the sync took the project from the second segment of the name, so the same event name used by two different clients collapsed into a single row. An alias table fixed it, but a name is a label people edit, not a key — that should have been the design from the start.