Data Retrieval - Large Buys

This template provides a step-by-step example for efficiently retrieving data at scale using the Carbon Arc Python SDK. It demonstrates how to:

Batch requests across multiple entity groups (e.g., brands, companies, locations)
Handle pagination to retrieve complete data assets
Combine and normalize paginated results into a single consolidated DataFrame

The final DataFrame can then be exported or used for downstream analysis in notebooks, dashboards, or external tools. This approach is useful for working with larger data assets or automating multi-entity retrievals.

Setup Your Environment

# Required dependencies
import os
import pandas as pd
from dotenv import load_dotenv
from carbonarc import CarbonArcClient

# Load environment variables and authenticate
load_dotenv()
API_AUTH_TOKEN = os.getenv("API_AUTH_TOKEN")
client = CarbonArcClient(token=API_AUTH_TOKEN)
print("Client initialized successfully")

Define Entities, Dates, and Insight

# Define entity batches to split large workloads
entity_sets =[
        {"carc_id": 52002, "representation": "company"},
        {"carc_id": 670, "representation": "company"},
        {"carc_id": 56338, "representation": "company"},
        {"carc_id": 826, "representation": "company"},
        {"carc_id": 60220, "representation": "company"},
        {"carc_id": 66928, "representation": "company"},
        {"carc_id": 64874, "representation": "company"},
        {"carc_id": 2558, "representation": "company"},
        {"carc_id": 66932, "representation": "company"},
        {"carc_id": 64127, "representation": "company"},
        {"carc_id": 8367, "representation": "company"},
    # Add more sets as needed
]
# Define date range and insight
date_range = {"start_date": "2021-01-01", "end_date": "2025-06-30"}
insight_id = 626  # Replace with your Insight ID this is Card Spend in Card - US Complete Panel

Check Price

price = client.explorer.check_framework_price(framework)
print(price)

Download Framework Data in Batches

unioned_df = pd.DataFrame()
entity_set_counter = 1

for entity_set in entity_sets:
    # Step 1: Build and purchase the framework
    framework = client.explorer.build_framework(
        entities=entity_set,
        insight=insight_id,
        filters={
            "location_resolution": "us",
            "date_resolution": "month",
            "date_range": date_range,
            "transaction_method": "All"  # Customize as needed
        },
        aggregate="sum"
    )
    order = client.explorer.buy_frameworks([framework])
    framework_id = order["frameworks"][0]

    # Step 2: Paginate and collect all data
    all_data = []
    page = 1
    while True:
        response = client.explorer.get_framework_data(framework_id=framework_id, page=page)
        data_page = response.get("data", [])
        if not data_page:
            break
        all_data.extend(data_page)
        print(f"Retrieved page {page} with {len(data_page)} records")
        page += 1

    # Step 3: Append results to unioned DataFrame
    df = pd.DataFrame(all_data)
    unioned_df = pd.concat([unioned_df, df], ignore_index=True)
    entity_set_counter += 1

Setup Your Environment

Define Entities, Dates, and Insight

Check Price

Download Framework Data in Batches

Need Help