Connect GCP to Xplorr: Service Account and BigQuery Billing Export Setup

Connect GCP to Xplorr

Xplorr reads your GCP costs from the Cloud Billing export in BigQuery, your recommendations from the Recommender API, and your resources from the Cloud Asset API. It signs in as a service account you create, using its JSON key. Every call is a read: nothing is created, modified or deleted in your project.

Prefer Terraform or a gcloud script? See Set it up with code.

So there are two parts to the setup:

1. Turn on the Cloud Billing export to BigQuery, if it is not on already
2. Create a service account with read roles, and give Xplorr its key

One Xplorr cloud account is one GCP project: the project that holds the billing export dataset.

Before you start

• You need the member or admin role in Xplorr. Viewers cannot connect accounts. Whoever connects an account owns it and controls who else in the organization can see it.
• In GCP you need Billing Account Administrator (or Billing Account Costs Manager) on the billing account to turn on the export, and permission to create service accounts and grant IAM roles in the project.
• Billing export data lags by up to 24 to 48 hours. Today’s costs appear later. This is Google’s refresh rate, not Xplorr’s.

Step 1: Turn on the BigQuery billing export

Skip this if Billing → Billing export → BigQuery export already shows standard usage cost export going to a dataset.

1. Create a dataset

   In BigQuery, create a dataset in the project you will connect, for example billing_export.

   Choose the location with care. A multi-region dataset (US or EU) receives billing data from the start of the previous month when you first turn the export on. A single-region dataset (such as europe-west1) only receives data from the day you turn it on. Remember the location: Xplorr needs it if it is anything other than US.

   bq mk --dataset --location=US YOUR_PROJECT_ID:billing_export

2. Point the export at it

   Go to Billing → Billing export → BigQuery export, edit Standard usage cost, and select the project and dataset. Save.

   Turn on Detailed usage cost as well if you want per-resource costs. The standard export has no resource column, so without the detailed table per-resource costs stay empty while everything else works.

3. Wait for the first table

   Google creates a table named gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX (your billing account ID with dashes turned into underscores) once data starts to arrive, which can take a few hours. The detailed export writes gcp_billing_export_resource_v1_XXXXXX_XXXXXX_XXXXXX. Connect after the table exists, because the connect check queries it.

Step 2: Create the service account

1. Create it

   Go to IAM & Admin → Service Accounts → Create service account. Name it xplorr-reader or anything you like.

2. Grant roles on the project

   Role	Needed for	Required?
   BigQuery Data Viewer (roles/bigquery.dataViewer)	Reading the billing export tables. You can grant it on just the billing export dataset instead of the project (add the FOCUS or Carbon Footprint export datasets too if you use them)	Yes
   BigQuery Job User (roles/bigquery.jobUser)	Running the queries	Yes
   Recommender Viewer (roles/recommender.viewer)	Idle VM, idle disk, idle Cloud SQL, machine type and committed use recommendations	Yes, for recommendations
   Cloud Asset Viewer (roles/cloudasset.viewer)	The full resource inventory	Yes, for inventory
   Compute Viewer (roles/compute.viewer)	Committed use discounts, and the inventory fallback	Recommended
   Storage Bucket Viewer (roles/storage.bucketViewer)	Listing Cloud Storage buckets in the inventory fallback, used only when the Cloud Asset API is denied	Recommended
   Logs Viewer (roles/logging.viewer)	Admin Activity audit logs. Every sync checks they are readable and Cloud Accounts shows a warning when they are not; anomaly root cause reads them	Recommended

   And on the billing account (not the project), if you want the features it powers:

   Role	Needed for	Required?
   Billing Account Viewer (roles/billing.viewer)	Spend based committed use recommendations, and the Cloud Billing budget shown beside invoice reconciliation (read through the Budget API). Set billingAccountId in the credentials too	Optional

   Also enable the Recommender API (recommender.googleapis.com) and the Cloud Asset API (cloudasset.googleapis.com) in the project, plus the Cloud Billing Budget API (billingbudgets.googleapis.com) if you grant Billing Account Viewer.

   Missing Recommender access is easy to miss

   The connect check and the cost sync only need the two BigQuery roles, so the account shows Active without the others. Without Recommender Viewer, the idle resource and rightsizing lookups are denied and skipped, and GCP shows no such recommendations. Committed use discount recommendations are the exception: a missing permission there is named on the Data health and Commitments pages. Without Cloud Asset Viewer, Xplorr falls back to listing Compute Engine instances, disks, addresses, networks and Cloud Storage buckets, which needs Compute Viewer and Storage Bucket Viewer (roles/storage.bucketViewer), and misses every other resource type.

3. Create a JSON key

   Open the service account, go to Keys → Add key → Create new key, choose JSON, and download it. Keep it safe: it grants read access to your billing data.

With the gcloud CLI

PROJECT_ID="your-project-id"
gcloud config set project "$PROJECT_ID"

gcloud services enable recommender.googleapis.com cloudasset.googleapis.com

gcloud iam service-accounts create xplorr-reader \
  --display-name="Xplorr reader"
SA="xplorr-reader@${PROJECT_ID}.iam.gserviceaccount.com"

for ROLE in roles/bigquery.dataViewer roles/bigquery.jobUser \
            roles/recommender.viewer roles/cloudasset.viewer roles/compute.viewer \
            roles/storage.bucketViewer roles/logging.viewer; do
  gcloud projects add-iam-policy-binding "$PROJECT_ID" \
    --member="serviceAccount:${SA}" --role="$ROLE"
done

# Optional: spend based CUD recommendations and the invoice budget hint
BILLING_ACCOUNT_ID="000000-000000-000000"
gcloud services enable billingbudgets.googleapis.com
gcloud billing accounts add-iam-policy-binding "$BILLING_ACCOUNT_ID" \
  --member="serviceAccount:${SA}" --role="roles/billing.viewer"

gcloud iam service-accounts keys create xplorr-key.json --iam-account="$SA"

Step 3: Connect it in Xplorr

1. Open the form

   In Xplorr go to Infrastructure → Cloud Accounts and click Connect account. Set Provider to Google Cloud Platform.

2. Fill in the fields

   • Account name: whatever you want to call it in Xplorr

   • GCP project ID: the project that holds the billing export dataset

   • Credentials (JSON): one JSON object that wraps your key file. The Fill template button gives you a starting point; the full shape is:

     {
       "projectId": "your-project-id",
       "billingDataset": "billing_export",
       "billingTable": "gcp_billing_export_v1_XXXXXX_XXXXXX_XXXXXX",
       "bigqueryLocation": "EU",
       "keyfileJson": {
         "type": "service_account",
         "project_id": "your-project-id",
         "private_key_id": "...",
         "private_key": "PASTE_THE_PRIVATE_KEY_FROM_YOUR_KEY_FILE",
         "client_email": "[email protected]",
         "client_id": "..."
       }
     }

     Paste the whole downloaded key file as the value of keyfileJson.

3. Test, then connect

   Test connection runs the check below without saving anything. Then click Connect account.

The credential fields

Field	Required	Default	What it is
projectId	Yes		The project that holds the export dataset. Queries run in, and are billed to, this project
keyfileJson	Yes		The service account key file, as a JSON object or a string
billingDataset	No	billing_export	The export dataset. datasetId is accepted as the same thing
billingTable	No	gcp_billing_export_v1_*	The standard export table. tableId is accepted as the same thing. The default wildcard matches any standard export table in the dataset
bigqueryLocation	Yes, unless the dataset is in US	US	The dataset’s location, for example EU or europe-west1. It must match the dataset exactly
resourceBillingTable	No	Derived from billingTable	The detailed export table, for per-resource costs. Only needed if it does not follow Google’s naming
regions	No	["us-central1", "us-east1", "europe-west1"]	Up to three locations to read recommendations from
billingAccountId	No		Your billing account ID, for spend based committed use recommendations and the budget shown beside invoice reconciliation. Needs Billing Account Viewer (roles/billing.viewer) granted on the billing account, not the project

EU and regional datasets need bigqueryLocation

Xplorr runs every query in location US unless you say otherwise. If your export dataset is in EU or a single region, leave out bigqueryLocation and the connect check fails with a “not found” error for a dataset that plainly exists.

The Fill template button uses datasetId and tableId and a sample table name of gcp_billing_export_v1. Replace that with your real table name, or delete the tableId line to use the wildcard.

You can also paste the key file itself as the whole JSON, with billingDataset, billingTable and bigqueryLocation added at the top level beside type. In that form the dataset must live in the key’s own project_id.

What happens when you click Connect

1. Xplorr checks the credentials before saving anything by running the real cost query against your export table for the last three days. It has 20 seconds; past that you get an error saying the account was not added and nothing was saved.
2. What the check proves: the key is valid, the service account can read the dataset and run BigQuery jobs, and the project, dataset, table and location are right. What it does not prove: Recommender or Cloud Asset access. Those are first used by the first sync, and fail quietly (see above).
3. If the project is already connected to your organization, you get a message saying so rather than a duplicate.
4. On success the account is saved and the first sync starts straight away, backfilling 90 days, or as far back as your export has data. It usually takes 10 to 30 minutes, and you get an email when it finishes. After that, every account syncs once a day at 02:00 UTC.

The status column shows Active once the account is connected. A failed sync shows Error with the reason underneath it.

Who can do what

Action	Who
Connect an account	Member or admin
Sync or delete an account	An organization admin, or whoever connected it
See an account’s data	Admins see everything; members and viewers see only accounts they have been granted

Troubleshooting

“GCP permission denied” The service account is missing BigQuery Data Viewer or BigQuery Job User, or the grant has not propagated yet. Wait a few minutes and try again.

Connect fails with “Could not connect with the provided credentials” Click Test connection, which shows Google’s own error. A “Not found: Dataset” or “Not found: Table” message is the usual one. Check, in order: bigqueryLocation matches the dataset’s location; the project, dataset and table names are right; and the export has created its first table. bq ls billing_export lists the tables.

“GCP credentials are incomplete” projectId or keyfileJson is missing from the JSON.

“GCP key file is invalid or expired” The key was truncated when pasting, or it has been deleted or disabled. Create a new key.

Active, costs arrive, but no recommendations Recommender Viewer is missing, or the Recommender API is not enabled. Also check that regions covers where your resources run.

Active, but Inventory is thin or empty Cloud Asset Viewer is missing, or the Cloud Asset API is not enabled, so only the Compute Engine and Cloud Storage fallback ran.

Per-resource costs are empty The detailed usage cost export is off, or its table name does not follow Google’s pattern. Turn it on, or set resourceBillingTable.

Setup checklist

• Standard usage cost export to BigQuery on (and detailed, for per-resource costs)
• The export table exists in the dataset
• Service account has BigQuery Data Viewer (on the project or the billing export dataset), BigQuery Job User, Recommender Viewer and Cloud Asset Viewer (plus Compute Viewer, Storage Bucket Viewer and Logs Viewer)
• Optional: Billing Account Viewer on the billing account, and billingAccountId in the credentials
• Recommender API and Cloud Asset API enabled
• Credentials JSON has projectId, keyfileJson, and bigqueryLocation if the dataset is not in US
• Account shows Active in Infrastructure → Cloud Accounts
• Costs on the dashboard, recommendations and inventory present

FAQ

Is the access read only? Yes. Every role above is a viewer role, and Xplorr makes no write calls.

How often does Xplorr sync? Once a day at 02:00 UTC. You can also sync any account on demand with the Sync button on its row, or through the trigger_sync MCP tool.

Will the queries cost me money? They run as BigQuery jobs in your project and are billed at your normal BigQuery rates. Billing export tables are usually small, so the cost is usually small too.

Can I connect several projects? Costs come from the billing export, which covers every project on the billing account. If several projects export to one dataset, one connection covers their costs. Recommendations and inventory, though, are read for the connected project only, so connect other projects separately if you want those for them.

Set it up with code

The public Xplorrio/cloud-onboarding repository, release v0.0.1, creates the service account xplorr-reader, grants the read-only roles in Step 2, enables the APIs Xplorr calls and writes the Credentials (JSON) the Connect account form asks for. The console shows the same quick starts under Set up with code on the Connect account form, pre-filled with the project ID you typed. Turn on the billing export first (Step 1): Google offers no API for it, so no template can.

Terraform

provider "google" {
  project = "my-project-id"
}

module "xplorr" {
  source = "git::https://github.com/Xplorrio/cloud-onboarding.git//gcp/terraform?ref=v0.0.1"

  project_id         = "my-project-id"
  billing_dataset_id = "billing_export"
}

output "xplorr_connect_form" {
  value = module.xplorr.xplorr_connect_form
}

output "key_create_command" {
  value = module.xplorr.key_create_command
}

Apply it, then create the key yourself so it never lands in Terraform state, and merge it into xplorr-credentials.json:

terraform init
terraform apply
$(terraform output -raw key_create_command)
(umask 077 && terraform output -json xplorr_connect_form \
  | jq --slurpfile key xplorr-key.json '. + {keyfileJson: $key[0]}' > xplorr-credentials.json)

project_id is the only required input; billing_dataset_id defaults to billing_export. Set billing_account_id to also grant Billing Account Viewer. The GCP README lists every input.

gcloud (Cloud Shell)

From Cloud Shell:

git clone --depth 1 --branch v0.0.1 https://github.com/Xplorrio/cloud-onboarding.git cloud-onboarding && ./cloud-onboarding/gcp/gcloud.sh --project my-project-id

The script is safe to run again. It prints the command that creates the key file; run it, then run the script again to write xplorr-credentials.json, or pass --create-key to have the script create the key. Add --billing-account 000000-000000-000000 for Billing Account Viewer, and --location EU --create-dataset to create the export dataset. ./cloud-onboarding/gcp/gcloud.sh --help lists every option.

Then, in Connect account:

Xplorr field	Value
GCP project ID	projectId
Credentials (JSON)	The contents of xplorr-credentials.json: projectId, billingDataset, billingTable (left out when the wildcard is used), bigqueryLocation and keyfileJson

Delete xplorr-key.json and xplorr-credentials.json once the account is connected. Every template also has an xplorr_principal trust mode; it needs keyless onboarding, which is not available yet, so keep the default.

Related guides

• Connect AWS
• Connect Azure
• Data Health: check that the first sync returned data
• Budget Alerts: set up spend notifications after connecting