Skip to main content

Block Data Delivery

For clients who have purchased block row-level data, Carbon Arc provides two access methods:

  1. Iceberg REST Catalog — Query data directly using industry-standard Iceberg table format (Recommended)
  2. Amazon S3 — Direct file access via AWS S3 buckets

Both methods provide access to the same underlying data. We recommend using Polaris for new integrations as it provides a modern, query-ready interface without the need to manage file ingestion pipelines.

Choosing a Delivery Method

Both methods deliver the same underlying data. Polaris is recommended for all new integrations; S3 is maintained for existing pipelines.

DimensionIceberg REST Catalog (Polaris)Amazon S3
Access modelQuery tables in place via an Iceberg REST catalogRead raw files from a dedicated S3 bucket
Setup effortPoint your engine at the catalog with credentialsBuild and maintain a file-ingestion pipeline
ETL requiredNone — query directlyYes — you load Full + Incremental drops yourself
Data freshnessAlways the latest committed snapshotAs of the last drop you have ingested
Incremental updatesChangelog table (update_id, event_timestamp){drop_date}/Incremental/ folders
Full refreshesaction = 'FULL_REFRESH' rows in the changelog{drop_date}/Full/ folders
Point-in-time / historyNative Iceberg time travel, last 45 days (as-of timestamp or snapshot)Manual reconstruction from Full + Incrementals (available beyond the 45-day window)
Schema evolutionHandled automaticallyHandled manually in your pipeline
Compatible toolsSnowflake, Databricks, ClickHouse, Spark, Trino/Starburst, and other Iceberg-compatible enginesAny S3-capable tool
CredentialsPolaris Client ID / Secret (OAuth)AWS IAM principal (ARN) granted read access
Best forNew integrations and query-ready analyticsExisting S3 workflows and custom file processing
Recommendation

If you're starting fresh, use Polaris. You skip building an ingestion pipeline, always see the latest data, and get time travel and changelog-based incremental updates for free. Choose S3 only when you have an existing file-based pipeline you need to keep.

Manage Block access from the SDK

The Carbon Arc Python SDK can discover Block datasets, request evaluation access, track the approval lifecycle, and register S3 ARNs programmatically. See Block for Devs.

Iceberg REST Catalog (Polaris)

Overview

Carbon Arc provides access via an Iceberg REST Catalog, allowing you to connect your data platform directly to Carbon Arc's block data warehouse. This is the recommended approach for new clients as it offers:

  • No ETL required — Query tables directly without building ingestion pipelines
  • Always up-to-date — Access the latest data without managing incremental updates
  • Industry standard — Compatible with Snowflake, Databricks, ClickHouse, Spark, Trino, and more
  • Schema evolution — Automatic handling of schema changes

Connection Details

ParameterValue
Catalog URIhttps://bulk.apps.carbonarc.co/api/catalog
Warehousebulk
Auth ScopePRINCIPAL_ROLE:ALL
OAuth Token Endpointhttps://bulk.apps.carbonarc.co/api/catalog/v1/oauth/tokens

Credentials

Your Client ID and Client Secret will be provided via a secure 1Password link after purchase. Keep these credentials secure and do not share them.

Platform Connection Guides

Select your data platform below for specific connection instructions:

Snowflake Integration

Step 1: Create Catalog Integration

CREATE OR REPLACE CATALOG INTEGRATION carbon_arc
  CATALOG_SOURCE = POLARIS
  TABLE_FORMAT = ICEBERG
  REST_CONFIG = (
    CATALOG_URI = 'https://bulk.apps.carbonarc.co/api/catalog'
    WAREHOUSE = 'bulk'
    ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS
  )
  REST_AUTHENTICATION = (
    TYPE = OAUTH
    OAUTH_CLIENT_ID = '<your_client_id>'
    OAUTH_CLIENT_SECRET = '<your_client_secret>'
    OAUTH_ALLOWED_SCOPES = ('PRINCIPAL_ROLE:ALL')
  )
  ENABLED = TRUE;

Step 2: Create Linked Database

CREATE DATABASE carc
  LINKED_CATALOG = (
    CATALOG = 'carbon_arc'
  );

Step 3: Query Data

Once connected, you can query tables directly:

SELECT * FROM carc.sloth.app_performance_data_daily LIMIT 100;
note

Replace <your_client_id> and <your_client_secret> with the credentials provided via 1Password.

Querying Your Data

Once your platform is connected, Block tables behave like any other tables in your engine. A few conventions to know:

  • Table naming — tables are addressed as carc.<namespace>.<table>, where carc is the linked database/catalog you created, <namespace> is the data family (e.g. sloth), and <table> is the feed (e.g. app_performance_data_daily).
  • Always filter on dt — every data table is partitioned by the dt date column. Including a dt predicate lets the engine prune partitions, which makes queries dramatically faster and cheaper.
  • Companion changelog — each data table has a {table}_changelog table in the same namespace (see Tracking Data Updates).

Explore what you have access to:

SHOW SCHEMAS IN DATABASE carc;
SHOW TABLES IN SCHEMA carc.sloth;

Run a basic query (note the dt filter for partition pruning):

SELECT *
FROM carc.sloth.app_performance_data_daily
WHERE dt >= DATE '2026-04-01'
  AND dt <  DATE '2026-05-01'
LIMIT 100;
Best Practice

Start narrow. Select only the columns you need and always bound dt to the window you care about — Block tables can hold billions of rows, and partition pruning on dt is the single biggest lever on query cost and speed.

Tracking Data Updates with Changelog Tables

Every client-facing data table has a companion changelog table in the same namespace, named {table_name}_changelog. Carbon Arc writes a row to the changelog each time a new partition is written to the data table, so you can drive incremental ingestion from a single audit stream instead of scanning the data table itself.

If you have access to a data table, you automatically have access to its changelog — no additional setup is required.

Schema

ColumnTypeDescription
update_idSTRINGUnique identifier for the update event
event_timestampTIMESTAMPUTC timestamp when the partition was written
actionSTRINGFULL_REFRESH (reinstatement) or INCREMENTAL (daily drop)
drop_partitionSTRINGThe drop_partition value written to the data table
dtDATEDate partition column — always include in filters for efficient querying

Example Changelog Tables

Data TableChangelog Table
dalmatian.clickstream_datadalmatian.clickstream_data_changelog
sloth.app_performance_data_dailysloth.app_performance_data_daily_changelog
  1. Persist a cursor — track the maximum event_timestamp you have processed so far.
  2. Poll the changelog — on each run, read new rows where event_timestamp > <cursor>, filtered on dt for partition pruning.
  3. Handle INCREMENTAL rows — re-read only the listed drop_partition values from the data table and merge them into your downstream store.
  4. Handle FULL_REFRESH rows — the upstream vendor data was fully reinstated. Truncate your local copy of the table and re-ingest it from scratch.
  5. Advance the cursor — persist the new maximum event_timestamp once ingestion succeeds.

Example Queries

Fetch all updates since your cursor:

SELECT update_id, event_timestamp, action, drop_partition
FROM carc.sloth.app_performance_data_daily_changelog
WHERE dt >= DATE '2026-04-01'
  AND event_timestamp > TIMESTAMP '2026-04-14 00:00:00'
ORDER BY event_timestamp;

Find the most recent full refresh (use this as a hard reset point):

SELECT MAX(event_timestamp) AS last_full_refresh
FROM carc.sloth.app_performance_data_daily_changelog
WHERE dt >= DATE '2026-01-01'
  AND action = 'FULL_REFRESH';

List every incremental partition written since the last full refresh:

SELECT drop_partition, MIN(event_timestamp) AS written_at
FROM carc.sloth.app_performance_data_daily_changelog
WHERE dt >= DATE '2026-01-01'
  AND action = 'INCREMENTAL'
  AND event_timestamp > (
    SELECT MAX(event_timestamp)
    FROM carc.sloth.app_performance_data_daily_changelog
    WHERE dt >= DATE '2026-01-01'
      AND action = 'FULL_REFRESH'
  )
GROUP BY drop_partition;
tip

A single update event can produce multiple changelog rows — one per drop_partition written. Group by update_id if you need to collapse them back into a single event.

Working with update_id (the "Drop ID")

update_id uniquely identifies a single update event — one publish, or "drop," of data to a table. Some teams refer to it as the Drop ID; it is the update_id column already present in every changelog table.

Key things to know:

  • One event, possibly many rows. A single drop can write multiple partitions. Each written partition produces its own changelog row, and all of them share the same update_id (one row per drop_partition). Group by update_id to collapse a drop back into a single logical event.
  • Type is per-event. The action value (FULL_REFRESH or INCREMENTAL) applies to the whole update_id.
  • It maps to an S3 delivery. The same drop delivered over S3 appears under a {drop_date}/Full/ or {drop_date}/Incremental/ path, so update_id is the Polaris equivalent of an S3 drop folder.

Use it for exactly-once ingestion. Persist the set of update_ids you have already processed and skip them on replay. This makes your pipeline idempotent even if a run restarts or the changelog is re-read.

Fetch the rows for one specific drop. Look up the drop's drop_partition value(s) from the changelog, then filter the data table on them:

-- 1) Which partitions did this drop write?
SELECT drop_partition
FROM carc.sloth.app_performance_data_daily_changelog
WHERE update_id = '<update_id>';

-- 2) Read exactly those partitions from the data table
SELECT *
FROM carc.sloth.app_performance_data_daily
WHERE drop_partition IN ( /* values from step 1 */ );

Collapse multi-partition drops into one event per update_id:

SELECT
  update_id,
  MIN(event_timestamp)         AS event_timestamp,
  MAX(action)                  AS action,
  COUNT(*)                     AS partitions_written,
  ARRAY_AGG(drop_partition)    AS drop_partitions
FROM carc.sloth.app_performance_data_daily_changelog
WHERE dt >= DATE '2026-04-01'
GROUP BY update_id
ORDER BY event_timestamp;
tip

When contacting support about a specific delivery, include the update_id. It uniquely identifies the drop across both Polaris and S3 and lets us trace it end to end.

Time Travel (Point-in-Time Queries)

Because Block tables use Apache Iceberg, every write creates an immutable snapshot of the table. Time travel lets you query a table exactly as it existed at an earlier point in time — either at a wall-clock timestamp or at a specific snapshot ID.

What it's for:

  • Reproducibility — rerun a report or model against the exact data that was available on a past date and get identical results.
  • Point-in-time / as-of analysis — reconstruct what was known as of date T, avoiding lookahead bias in backtests and research.
  • Auditing & debugging — compare the table before and after a specific update to see precisely what changed.

Finding a point to travel to

You can travel to any timestamp, but the most useful anchors are the update events recorded in the changelog. The event_timestamp of a changelog row is the moment that snapshot was committed, so you can read a point-in-time from the changelog and use it directly in a time-travel query.

To browse the raw snapshot history, query the Iceberg metadata tables:

-- Snapshot IDs and commit times
SELECT snapshot_id, committed_at, operation
FROM carc.sloth."app_performance_data_daily$snapshots"
ORDER BY committed_at DESC;

Querying as of a point in time

-- As of a timestamp
SELECT *
FROM carc.sloth.app_performance_data_daily
  AT(TIMESTAMP => '2026-04-14 00:00:00'::timestamp_tz)
WHERE dt >= DATE '2026-04-01';

FOR TIMESTAMP AS OF (and its equivalents) resolves to the snapshot that was current as of that instant — i.e. the most recent snapshot committed on or before the timestamp.

Time travel is limited to the last 45 days

Carbon Arc retains Iceberg snapshots for 45 days. Older snapshots are expired as part of routine table maintenance, so time-travel queries can only reach back 45 days. To reproduce a state older than that, use the S3 point-in-time reconstruction approach, or track update_ids (and the data you need) as you ingest.

Amazon S3

Overview

Block data is delivered to dedicated S3 buckets with a standardized folder structure. Your AWS IAM user or role is granted read-only access to the bucket containing your purchased data assets.

Bucket Access

After purchase, you'll receive:

  • Bucket ARN: The S3 bucket location (e.g., arn:aws:s3:::carc-ext-{dataset})
  • IAM Access: Your AWS principal is granted read access to the bucket

Delivery Structure

Data is organized into two delivery patterns:

Incremental Updates

For ongoing data updates, we follow a standardized incremental delivery pattern:

AttributeValue
Path Structure{drop_date}/Incremental/[data_files]
ContentContains only new records received from vendor

Example path:

s3://carc-ext-sloth/20260203/Incremental/sloth_app_performance_data_daily/

Full Reinstatement Deliveries

Complete data reinstatements are delivered when upstream data is updated:

AttributeValue
Path Structure{drop_date}/Full/[data_files]
ContentComplete data asset including all historical data ingested to date

Example paths:

s3://carc-ext-sloth/20260129/Full/sloth_app_performance_data_daily/
s3://carc-ext-sloth/20260129/Full/sloth_app_performance_data_monthly/

Delivery Completion: the _SUCCESS File

Each S3 drop includes a _SUCCESS marker file that Carbon Arc writes only after every data file in that drop has finished uploading. It is the signal that a delivery is complete and safe to consume.

Because a drop is made up of many objects and S3 has no folder-level "commit," a drop folder can appear on the bucket while files are still being written. Ingesting before the drop is finished can silently pull a partial delivery (missing files or records). Always gate ingestion on the _SUCCESS marker.

How to use it:

  • Wait for the marker. Treat a drop folder as not-ready until its _SUCCESS object exists. Only then begin reading the data files.
  • Trigger off the marker (recommended). Fire your ingestion pipeline from an S3 event notification on the _SUCCESS object key rather than on the data files — that way you never start mid-write.
  • Readers skip it automatically. Most Spark/Hadoop-based readers ignore files beginning with _ when reading a directory of Parquet, so the marker won't interfere with your reads.

Example (Incremental drop):

s3://carc-ext-sloth/20260203/Incremental/sloth_app_performance_data_daily/
    ├── <data files>.parquet
    ├── ...
    └── _SUCCESS          <-- present only when the drop is complete
tip

Pair this with the point-in-time reconstruction below — only fold a drop into your baseline once its _SUCCESS marker is present, so a backtest never sees a half-written drop.

When is Data Reinstated?

We generally reinstate data on the first Monday of the month if there are changes to the ontology or significant upstream data corrections.

Data Consumption Guidelines

Recommended Approach:

  1. Start with Full Reinstatement — Always consume the most recent Full reinstatement as your baseline
  2. Append Incrementals — Apply the Incremental deliveries that occurred after the latest Full refresh
  3. Re-ingest on New Full — When a new Full reinstatement is available, delete your existing ingested data and re-ingest the complete Full reinstatement
Best Practice

Monitor the S3 bucket for new Full directories. When one appears, schedule a complete re-ingestion to ensure data consistency.

Reconstructing a Point-in-Time Snapshot (as of time T)

S3 delivery has no query engine, so there is no automatic time travel. You can still reconstruct the exact state of a table as of any date T from the Full and Incremental drops on the bucket. This is the S3 equivalent of a Polaris time-travel query, it is the right approach for backtests that must avoid lookahead bias, and it is how you reach point-in-time states older than the 45-day Polaris time-travel window.

Drop folders are named by date ({drop_date}, formatted YYYYMMDD), which is what makes the reconstruction deterministic:

  1. Pick a baseline. Find the most recent Full reinstatement whose drop_date is on or before T.
  2. Layer on increments. Apply every Incremental drop whose drop_date is after that Full and on or before T, in ascending drop_date order.
  3. Ignore the future. Skip any drop with drop_date later than T — that data would not have been known at time T.

Example — reconstruct sloth_app_performance_data_daily as of T = 2026-02-04:

# Most recent Full on or before T  -> baseline
s3://carc-ext-sloth/20260129/Full/sloth_app_performance_data_daily/

# Incrementals after 20260129 and on/before 20260204 -> apply in order
s3://carc-ext-sloth/20260203/Incremental/sloth_app_performance_data_daily/
s3://carc-ext-sloth/20260204/Incremental/sloth_app_performance_data_daily/

# Anything dated after 20260204 -> ignore for this point-in-time view

Load the baseline first, then merge the incrementals in date order; the result matches what the table contained at end of day T.

tip

If you can use Polaris, prefer a FOR TIMESTAMP AS OF query instead — it produces the same point-in-time view without downloading and merging files. See Time Travel.

Available Datasets

The specific tables and feeds available depend on your purchased data package.

info

Contact your Carbon Arc representative for the complete schema documentation for your purchased data assets.

Support

For questions about block data access or connection issues: