Configurations
Config
Default configuration
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Name |
String |
The name for this configuration. Connectors reference the configuration with this name. |
x |
|
Connection |
The connection types that can be provided to this configuration. |
x |
||
Expiration Policy |
Configures the minimum amount of time that a dynamic configuration instance can remain idle before the runtime considers it eligible for expiration. This does not mean that the platform will expire the instance at the exact moment that it becomes eligible. The runtime will actually purge the instances when it sees it fit. |
|
Connection Types
Connection
This class (as it's name implies) provides connection instances and the funcionality to disconnect and validate those connections.
All connection related parameters (values required in order to create a connection) must be declared in the connection providers.
This particular example is a PoolingConnectionProvider which declares that connections resolved by this provider will be pooled and reused. There are other implementations like CachedConnectionProvider which lazily creates and caches connections or simply ConnectionProvider if you want a new connection each time something requires one.
</div>Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Project ID |
String |
Project Identity |
x |
|
Service Account Key (JSON) File |
String |
The service authentication credentials |
|
|
Reconnection |
When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn't pass after exhausting the associated reconnection strategy |
|
||
Pooling Profile |
Characteristics of the connection pool |
|
Associated Operations
Operations
Cancel Job
<bigquery:cancel-job>
CancelJob operation sends a job cancel request. If the location of the job is not "US" or "EU", the jobId must specify the job location. It logs 'Job Not Found', if specified Job does not exist. It won't log/return anything on successful execution.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Job Name |
String |
Google BigQuery job name |
x |
|
Job Location |
String |
Google BigQuery job location |
|
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:JOB_FAILURE_EXCEPTION
-
BIGQUERY:JOB_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Create Dataset
<bigquery:create-dataset>
CreateDataset operation creates a new empty dataset. A dataset is a grouping mechanism that holds zero or more tables.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Dataset Name |
String |
Dataset identity. Dataset ID must be alphanumeric (plus underscores, dashes, and colons) and must be at most 1024 characters long. |
x |
|
Table Life Time |
Number |
Default lifetime of all tables in the dataset, in milliseconds. Table expiration should not be less than 3600000ms. |
|
|
Description |
String |
User-friendly description for the dataset |
|
|
Friendly Name |
String |
User-friendly name for the dataset |
|
|
Labels |
Object |
List of labels associated with labels |
|
|
Location |
String |
Geographic location where the dataset should reside |
|
|
ACL |
Array of One of: |
Dataset's access control configuration |
|
|
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:DATASET_NOT_EMPTY
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:DATASET_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Create Job
<bigquery:create-job>
CreateJob operation create and start an asynchronous job.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Job Config |
One of: |
Google BigQuery job configuration |
x |
|
Job Info |
Google BigQuery create job options configuration |
x |
||
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:JOB_FAILURE_EXCEPTION
-
BIGQUERY:JOB_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Create Table
<bigquery:create-table>
CreateTable operation creates a new table in existing dataset.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Table Info |
Table information configuration to create a table |
x |
||
Table Fields |
Array of Table Field Configuration |
Table Schema details , Field Type , Field Name , Field Description , Field Mode , Sub-fields. |
|
|
Table Option |
Create table information options |
|
||
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:TABLE_NOT_FOUND
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Delete Dataset
<bigquery:delete-dataset>
DeleteDataset operation deletes the dataset specified by datasetId. It logs 'Dataset Not Empty' if deleting non-empty dataset is disabled and Dataset is still in use. It logs 'Dataset Not Found', if specified dataset does not exist. It won't log/return anything on successful execution.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Dataset Id |
String |
Google BigQuery dataset Id |
x |
|
Delete Non Empty Dataset |
Boolean |
Delete a dataset even if non-empty |
true |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:DATASET_NOT_EMPTY
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:DATASET_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Delete Table
<bigquery:delete-table>
DeleteTable operation deletes the specified table from the dataset. This operation logs 'Table Not found', if specified table does not exist. And it won't log/return anything on successful execution.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Table |
Table Configuration with tableId and datasetId |
x |
||
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
For Configurations.
Throws
-
BIGQUERY:TABLE_NOT_FOUND
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Get Dataset
<bigquery:get-dataset>
GetDataset operation returns the dataset specified by datasetId.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Dataset Id |
String |
Google BigQuery dataset identity |
x |
|
Field |
Get dataset options configuration |
|
||
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:DATASET_NOT_EMPTY
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:DATASET_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Get Job
<bigquery:get-job>
GetJob operation returns information about a specific job.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Job Name |
String |
Google BigQuery job name |
x |
|
Job Location |
String |
Google BigQuery job location |
|
|
Job Fields |
Job information options configurations |
|
||
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:JOB_FAILURE_EXCEPTION
-
BIGQUERY:JOB_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Get Query Result
<bigquery:get-query-result>
GetQueryResult operation returns the results of a query job.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Job Name |
String |
Google BigQuery job name |
x |
|
Job Location |
String |
Google BigQuery job location |
|
|
Job Result Options |
Job query result options configurations |
|
||
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:JOB_FAILURE_EXCEPTION
-
BIGQUERY:JOB_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Get Table
<bigquery:get-table>
GetTable operation returns the specified table from dataset.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Dataset Id |
String |
Google BigQuery dataset identity. |
x |
|
Table Id |
String |
Google BigQuery table identity. |
x |
|
Table Option |
Get table information options |
|
||
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:TABLE_NOT_FOUND
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Insert All
<bigquery:insert-all>
InsertAll operation streams data into BigQuery one record at a time without running the load job.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Rows Data |
Array of Object |
Row data to be inserted |
x |
|
Ignore Unknown Values |
Boolean |
Whether to accept rows that contain values that do not match the schema. |
true |
|
Skip Invalid Rows |
Boolean |
Whether to insert all valid rows of a request, even if invalid rows exist. |
false |
|
Template Suffix |
String |
The destination table is treated as a base template. |
|
|
Table Id |
String |
Table identity. |
x |
|
Dataset Id |
String |
Dataset identity. |
x |
|
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
List Dataset
<bigquery:list-dataset>
ListDataset operation returns partial information on each dataset: DatasetId, FriendlyName and GeneratedId. To get complete information use "Get Dataset" operation.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
List Dataset Option |
List dataset options configuration |
|
||
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
Array of Dataset |
Attributes Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:DATASET_NOT_EMPTY
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:DATASET_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
List Job
<bigquery:list-job>
ListJob operation lists all jobs that you started in the specified project.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Job Info |
Google BigQuery job options configuration |
|
||
Page Token |
String |
Specifies the token from which to start listing jobs |
|
|
Page Size |
Number |
Specifies the maximum number of jobs returned per page |
|
|
State |
Enumeration, one of:
|
State of the job (DONE, PENDING, RUNNING) |
|
|
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
Array of Job |
Attributes Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:JOB_FAILURE_EXCEPTION
-
BIGQUERY:JOB_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
List Table
<bigquery:list-table>
ListTable operation list the tables in the specified dataset. This operation returns partial information on each table: TableId, FriendlyName, Generated and type, which is part of Definition.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Dataset Id |
String |
Google BigQuery dataset identity. |
x |
|
List Table Options |
List table options configurations |
|
||
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
Array of Table Information |
Attributes Type |
For Configurations.
Throws
-
BIGQUERY:TABLE_NOT_FOUND
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
List Table Data
<bigquery:list-table-data>
ListTableData operation list the content of a table in rows.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Dataset Id |
String |
Google BigQuery dataset identity |
x |
|
Table Id |
String |
Google BigQuery table identity |
x |
|
Page Size |
Number |
Specifies the maximum number of rows returned per page |
|
|
Page Token |
String |
Page token of the request. |
|
|
Start Index |
Number |
Start row index of the table. |
|
|
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Query
<bigquery:query>
Runs the query associated with the request, using the given JobId or using an internally-generated random JobId.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Query Configuration |
Google BigQuery query configuration |
x |
||
Job Information |
Google BigQuery job options configuration |
x |
||
Job Name |
String |
Google BigQuery job name |
|
|
Job Location |
String |
Google BigQuery job location |
|
|
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:JOB_FAILURE_EXCEPTION
-
BIGQUERY:JOB_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Update Dataset
<bigquery:update-dataset>
UpdateDataset operation updates information in an existing dataset.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Field |
Update dataset options configuration |
|
||
Dataset Name |
String |
Dataset identity. Dataset ID must be alphanumeric (plus underscores, dashes, and colons) and must be at most 1024 characters long. |
x |
|
Table Life Time |
Number |
Default lifetime of all tables in the dataset, in milliseconds. Table expiration should not be less than 3600000ms. |
|
|
Description |
String |
User-friendly description for the dataset |
|
|
Friendly Name |
String |
User-friendly name for the dataset |
|
|
Labels |
Object |
List of labels associated with labels |
|
|
Location |
String |
Geographic location where the dataset should reside |
|
|
ACL |
Array of One of: |
Dataset's access control configuration |
|
|
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:EXECUTION
-
BIGQUERY:DATASET_NOT_EMPTY
-
BIGQUERY:CONNECTIVITY
-
BIGQUERY:DATASET_NOT_FOUND
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Update Table
<bigquery:update-table>
UpdateTable operation updates information in an existing table.
Parameters
Name | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
String |
The name of the configuration to use. |
x |
|
Table Info |
Table information configuration to update table |
x |
||
Table Fields |
Array of Table Field Configuration |
Table Schema details , Field Type , Field Name , Field Description , Field Mode , Sub-fields. |
x |
|
Table Option |
Table options configurations |
x |
||
Target Variable |
String |
The name of a variable on which the operation's output will be placed |
|
|
Target Value |
String |
An expression that will be evaluated against the operation's output and the outcome of that expression will be stored in the target variable |
#[payload] |
|
Reconnection Strategy |
A retry strategy in case of connectivity errors |
|
Output
Type |
For Configurations.
Throws
-
BIGQUERY:TABLE_NOT_FOUND
-
BIGQUERY:EXECUTION
-
BIGQUERY:CONNECTIVITY
-
MULE:CONNECTIVITY
-
BIGQUERY:RETRY_EXHAUSTED
Types
Reconnection
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Fails Deployment |
Boolean |
When the application is deployed, a connectivity test is performed on all connectors. If set to true, deployment will fail if the test doesn’t pass after exhausting the associated reconnection strategy |
||
Reconnection Strategy |
The reconnection strategy to use |
Reconnect
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Frequency |
Number |
How often (in ms) to reconnect |
||
Count |
Number |
How many reconnection attempts to make |
Reconnect Forever
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Frequency |
Number |
How often (in ms) to reconnect |
Pooling Profile
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Max Active |
Number |
Controls the maximum number of Mule components that can be borrowed from a session at one time. When set to a negative value, there is no limit to the number of components that may be active at one time. When maxActive is exceeded, the pool is said to be exhausted. |
||
Max Idle |
Number |
Controls the maximum number of Mule components that can sit idle in the pool at any time. When set to a negative value, there is no limit to the number of Mule components that may be idle at one time. |
||
Max Wait |
Number |
Specifies the number of milliseconds to wait for a pooled component to become available when the pool is exhausted and the exhaustedAction is set to WHEN_EXHAUSTED_WAIT. |
||
Min Eviction Millis |
Number |
Determines the minimum amount of time an object may sit idle in the pool before it is eligible for eviction. When non-positive, no objects will be evicted from the pool due to idle time alone. |
||
Eviction Check Interval Millis |
Number |
Specifies the number of milliseconds between runs of the object evictor. When non-positive, no object evictor is executed. |
||
Exhausted Action |
Enumeration, one of:
|
Specifies the behavior of the Mule component pool when the pool is exhausted. Possible values are: "WHEN_EXHAUSTED_FAIL", which will throw a NoSuchElementException, "WHEN_EXHAUSTED_WAIT", which will block by invoking Object.wait(long) until a new or idle object is available, or WHEN_EXHAUSTED_GROW, which will create a new Mule instance and return it, essentially making maxActive meaningless. If a positive maxWait value is supplied, it will block for at most that many milliseconds, after which a NoSuchElementException will be thrown. If maxThreadWait is a negative value, it will block indefinitely. |
||
Initialisation Policy |
Enumeration, one of:
|
Determines how components in a pool should be initialized. The possible values are: INITIALISE_NONE (will not load any components into the pool on startup), INITIALISE_ONE (will load one initial component into the pool on startup), or INITIALISE_ALL (will load all components in the pool on startup) |
||
Disabled |
Boolean |
Whether pooling should be disabled |
Expiration Policy
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Max Idle Time |
Number |
A scalar time value for the maximum amount of time a dynamic configuration instance should be allowed to be idle before it’s considered eligible for expiration |
||
Time Unit |
Enumeration, one of:
|
A time unit that qualifies the maxIdleTime attribute |
Dataset
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Acl |
Array of Acl |
Dataset's access control configuration. | ||
Creation Time |
Number |
The time when this dataset was created, in milliseconds since the epoch. | ||
Dataset Id |
The dataset identity. | |||
Default Table Lifetime |
Number |
The default lifetime of all tables in the dataset, in milliseconds. | ||
Description |
String |
A user-friendly description for the dataset. | ||
Etag |
String |
The hash of the dataset resource. | ||
Friendly Name |
String |
A user-friendly name for the dataset. | ||
Generated Id |
String |
The service-generated id for the dataset. | ||
Labels |
Object |
A map for labels applied to the dataset. | ||
Last Modified |
Number |
The time when this dataset or any of its tables was last modified, in milliseconds since the epoch. | ||
Location |
String |
The geographic location where the dataset should reside. | ||
Self Link |
String |
URL that can be used to access the resource again. |
Acl
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Entity |
The entity for this ACL. | |||
Role |
String |
The role specified by this ACL. |
Entity
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Type |
String |
Types of BigQuery entities. |
Dataset Id
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Dataset |
String |
Dataset Identity | ||
Project |
String |
Project Name |
Job
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
The job's configuration | |||
Etag |
String |
The hash of the job resource. | ||
Generated Id |
String |
The service-generated id for the job. | ||
Job Id |
The job identity. | |||
Self Link |
String |
URL that can be used to access the resource again. | ||
Statistics |
Information about the job, including starting time and ending time of the job. | |||
Status |
The status of this job. | |||
User Email |
String |
The email address of the user who ran the job. |
Job Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Type |
String |
Type of a BigQuery Job. |
Job Id
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Job |
String |
The job's id. | ||
Location |
String |
The job's location. | ||
Project |
String |
Job's project id. |
Job Statistics
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Creation Time |
Number |
The creation time of the job in milliseconds since epoch. | ||
End Time |
Number |
The end time of the job in milliseconds since epoch. | ||
Start Time |
Number |
The start time of the job in milliseconds since epoch. |
Job Status
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Error |
The final error result of the job. | |||
Execution Errors |
Array of Big Query Error |
All errors encountered during the running of the job. | ||
State |
String |
The state of the job. |
BigQuery Error
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Location |
String |
Location where the error occurred, if present. | ||
Message |
String |
A human-readable description of the error. | ||
Reason |
String |
Short error code that summarizes the error. |
Job Info Options
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Configuration |
Boolean |
Job Configuration |
true |
|
E Tag |
Boolean |
Job eTag |
false |
|
Id |
Boolean |
Job Identity |
true |
|
Job Reference |
Boolean |
A job reference is a fully qualified identifier for refering to a job. |
true |
|
Self Link |
Boolean |
A URL that can be used to access the resource again. You can use this URL in Get or Update requests to the resource. |
false |
|
Statistics |
Boolean |
Information about the job, including starting time and ending time of the job. |
true |
|
Status |
Boolean |
The status of this job. |
true |
|
User Mail |
Boolean |
The email address of the user who ran the job. |
true |
Table Information
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Creation Time |
Number |
The time when this table was created, in milliseconds since the epoch. | ||
Description |
String |
Table Description | ||
E Tag |
String |
A hash of the page of results. | ||
Encryption Configuration |
Custom encryption configuration (e.g., Cloud KMS keys). | |||
Expiration Time |
Number |
The time when this table expires, in milliseconds since the epoch. If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed. | ||
Friendly Name |
String |
The user-friendly name for the table. | ||
Generated Id |
String |
The service-generated id for the table. | ||
Labels |
Object |
A map for labels applied to the table. | ||
Last Modified Time |
Number |
Table last modified time | ||
Num Bytes |
Number |
The size of the table in bytes | ||
Num Long Term Bytes |
Number |
The number of bytes considered "long-term storage" for reduced billing purposes. | ||
Num Rows |
Number |
Number of rows in a table | ||
Self Link |
String |
An URL that can be used to access the resource again. | ||
Table Id |
The table identity. | |||
Table Definition |
Table Definition |
Encryption Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Kms Key Name |
String |
KMS key name for encryption configuration |
Table Id
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Dataset |
String |
Dataset Identity | ||
Project Id |
String |
Project's user-defined id. | ||
Table |
String |
Table Identity |
Table Definition
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Schema |
The table's schema. | |||
Type |
String |
The table's type. |
Schema
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Fields |
Array of Field |
The fields in the current table schema. |
Field
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Description |
String |
The field description. | ||
Mode |
String |
The field mode. | ||
Name |
String |
The field name. | ||
Sub Fields |
Array of Field |
The list of sub-fields if field type is a RECORD. | ||
Type |
String |
The field type. |
Table Info Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Table |
Details of table(Table Name, Dataset Name) |
x |
||
Description |
String |
Table Description |
||
Kms Key Name |
String |
Creates a EncryptionConfiguration object for KMS key name |
||
Expiration Time |
Number |
The time when this table expires, in milliseconds since the epoch. If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed. |
||
Friendly Name |
String |
The user-friendly name for the table. |
||
Labels |
Object |
The labels associated with the table. Can be used to organize and group your tables. |
Table Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Table |
String |
Table identity. |
x |
|
Dataset |
String |
Dataset identity. |
x |
Table Field Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Field Name |
String |
The table field name |
x |
|
Field Type |
String |
The type of the table field Supported types are : BOOLEAN, BYTES, DATE, DATETIME, FLOAT, INTEGER, RECORD, STRING, TIME, TIMESTAMP |
x |
|
Field Description |
String |
The Table field description |
||
Field Mode |
String |
Mode for a Table field. NULLABLE fields can be set to null, REQUIRED fields must be provided. REPEATED fields can contain more than one value. |
||
Field Sub Fields |
Array of Table Field Configuration |
Sub field of the field name. Type of the parent field must be RECORD to add subfield. |
Table Info Options
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Creation Time |
Boolean |
The time when this table was created, in milliseconds since the epoch. |
true |
|
Description |
Boolean |
Table Description |
true |
|
E Tag |
Boolean |
A hash of the page of results. |
true |
|
Expiration Time |
Boolean |
The time when this table expires, in milliseconds since the epoch. If not present, the table will persist indefinitely. Expired tables will be deleted and their storage reclaimed. |
true |
|
Friendly Name |
Boolean |
The user-friendly name for the table. |
true |
|
Id |
Boolean |
An opaque ID of the table. |
true |
|
Last Modified Time |
Boolean |
Table last modified time |
true |
|
Location |
Boolean |
The geographic location where the table should reside. Possible values include EU and US. The default value is US. |
true |
|
Num Bytes |
Boolean |
The size of the table in bytes |
true |
|
Num Rows |
Boolean |
Number of rows in a table |
true |
|
Schema |
Boolean |
Table Schema |
true |
|
Self Link |
Boolean |
A URL that can be used to access the resource again. You can use this URL in Get or Update requests to the resource. |
true |
|
Streaming Buffer |
Boolean |
true |
||
Time Partitioning |
Boolean |
Configures time-based partitioning for the table. |
true |
|
Type |
Boolean |
The type of table. |
true |
|
View |
Boolean |
The view definition of table |
true |
Dataset Info Option
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Access |
Boolean |
It defines dataset access for one or more entities |
true |
|
Creation Time |
Boolean |
Creation time of Dataset |
true |
|
Dataset Reference |
Boolean |
A reference that identifies the dataset. |
true |
|
Description |
Boolean |
Dataset description |
true |
|
E Tag |
Boolean |
Dataset eTag |
true |
|
Friendly Name |
Boolean |
Dataset friendly name |
true |
|
Labels |
Boolean |
Dataset Labels |
true |
|
Last Modified Time |
Boolean |
Dataset last modified time |
true |
|
Location |
Boolean |
The geographic location where the dataset should reside. Possible values include EU and US. The default value is US. |
true |
|
Self Link |
Boolean |
A URL that can be used to access the resource again. You can use this URL in Get or Update requests to the resource. |
true |
|
Id |
Boolean |
Dataset Identity |
true |
Table Result
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Next Page |
The next page of result. | |||
Next Page Token |
String |
The token of next page of result. | ||
Schema |
It represents the schema for a Google BigQuery Table or data source. | |||
Total Rows |
Number |
Total number of rows in the complete result set, which can be more than the number of rows in the first page of results returned by getValues(). | ||
Values |
Array of Array of Field Value |
Represent rows in returned query result (table row). |
Field Value
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Attribute |
Enumeration, one of:
|
The attribute of this Field Value. | ||
Boolean Value |
Boolean |
This field's value as a Boolean. | ||
Bytes Value |
Binary |
This field's value as a byte array. | ||
Double Value |
Number |
This field's value as a Double. | ||
Long Value |
Number |
This field's value as a long. | ||
Null |
Boolean |
true if this field's value is null, false otherwise. | ||
Numeric Value |
Number |
This field's value as a BigDecimal. | ||
Record Value |
Array of Field Value |
This field's value as a list of FieldValue. | ||
Repeated Value |
Array of Field Value |
This field's value as a list of FieldValue. | ||
String Value |
String |
This field's value as a String. | ||
Timestamp Value |
Number |
This field's value as a long, representing a timestamp in microseconds since epoch (UNIX time). | ||
Value |
Any |
This field's value as an Object. |
Job Query Results Option
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Max Wait Time |
Number |
How long to wait for the query to complete, in milliseconds, before the request times out and returns. Note that this is only a timeout for the request, not the query. |
||
Page Size |
Number |
Number of records to be displayed on a page. |
||
Page Token |
String |
Specify the page token from which to start listing datasets |
||
Start Index |
Number |
Zero-based index of the starting row. |
Insert All Response
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Insert Errors |
Object |
All insertion errors as a map whose keys are indexes of rows that failed to insert. |
Page Attribute
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Has Next Page |
Boolean |
Used for paging results. When this field is true, it indicates additional results are available. | ||
Page Token |
String |
Specify the page token from which to start listing datasets |
List Dataset Option
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
All |
Boolean |
List all datasets, even hidden ones. |
false |
|
Page Size |
Number |
Specify the maximum number of datasets returned per page |
x |
|
Page Token |
String |
Specify the page token from which to start listing datasets |
List Table Options
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Page Size |
Number |
Specify the maximum number of datasets returned per page |
||
Page Token |
String |
The page token from which to start listing datasets |
Query Job
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Query String |
String |
A query string, following the BigQuery query syntax, of the query to execute. |
x |
|
Allow Large Results |
Boolean |
Sets whether the job is enabled to create arbitrarily large results. If true the query is allowed to create large results at a slight cost in performance. If true Destination Table must be provided. |
false |
|
Default Dataset |
String |
Specifies the default datasetId and projectId to assume for any unqualified table names in the query. If not set, all table names in the query string must be qualified in the format 'datasetId.tableId'. |
||
Dry Run |
Boolean |
If set to true, BigQuery doesn’t run the job. Instead, if the query is valid, BigQuery returns statistics about the job such as how many bytes would be processed. If the query is invalid, an error returns. The default value is false. |
false |
|
Flatten Results |
Boolean |
Sets whether nested and repeated fields should be flattened. If set to false AllowLargeResults must be true. By default results are flattened. |
true |
|
Maximum Billing Tier |
Number |
Limits the billing tier for this job. Queries that have resource usage beyond this tier will fail (without incurring a charge). If unspecified, this will be set to your project default. |
||
Named Parameters |
Array of Named Parameter Configuration |
the query parameters to a set of named query parameters to use in the query. Named parameters are denoted using an @ prefix, e.g. @myParam for a parameter named "myParam". SELECT * FROM <table> WHERE <column_name>=@myParam in query |
||
Positional Query Parameter Values |
Array of Query Parameter Value Configuration |
Sets the query parameters to a list of positional query parameters to use in the query. Positional parameters are denoted in the query with a question mark (?). SELECT * FROM <table> WHERE <column_name>=? |
||
Priority |
Enumeration, one of:
|
a priority for the query. If not specified the priority is assumed to be INTERACTIVE. Query is executed as soon as possible and count towards the concurrent rate limit and the daily rate limit. |
||
External Table Definitions |
BigQuery’s external tables are tables whose data reside outside of BigQuery But can be queried as normal BigQuery tables. |
|||
Use Legacy Sql |
Boolean |
UseLegacySql must be set to false to use query parameters |
false |
|
Use Query Cache |
Boolean |
Whether to look for the result in the query cache. The query cache is a best-effort cache that will be flushed whenever tables in the query are modified. Moreover, the query cache is only available when Destination Table is not set. |
true |
|
User Defined Functions |
Array of User Defined Function Configuration |
For legacy SQL, specify user defined function here. For Standard SQL, specify user defined function in Query string. |
||
Create Disposition |
Enumeration, one of:
|
Whether the job is allowed to create new tables CREATE_IF_NEEDED :- Configures the job to create the table if it does not exist. CREATE_NEVER :- Configures the job to fail with a not-found error if the table does not exist. Create Disposition |
||
Destination Table |
Sets the table where to put query results. If not provided a new table is created. This value is required if Allow Large Results is set to true. |
|||
Schema Update Option |
Enumeration, one of:
|
Specifies options relating to allowing the schema of the destination table To be updated as a side effect of the load or query job. Schema Update Option |
||
Write Disposition |
Enumeration, one of:
|
Action that should occur if the destination table already exists. WRITE_APPEND :- Configures the job to append data to the table if it already exists. WRITE_EMPTY :- Configures the job to fail with a duplicate error if the table already exists. WRITE_TRUNCATE :- Configures the job to overwrite the table data if table already exists. Write Disposition |
||
Kms Key Name |
String |
Creates a EncryptionConfiguration object for KMS key name |
Named Parameter Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Key |
String |
A parameter given to a query. | x |
|
Named Parameter |
Contains the type of the parameter and value of the parameter. | x |
Query Parameter Value Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Array Type |
Boolean |
The type of the array's elements, if this is an array. | false |
|
Type |
Enumeration, one of:
|
The type of the parameter. | x |
|
Values |
Array of String |
The values of the parameter. | x |
External Table Definition Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Table Name |
String |
Name of the table | x |
|
Source Uris |
Array of String |
The fully-qualified URIs that point to your data in Google Cloud Storage. Each URI can contain one '*' wildcard character that must come after the bucket's name. Size limits related to load jobs apply to external data sources, plus an additional limit of 10 GB maximum size across all URIs. | x |
|
Table Schema |
The Schema for the external data | x |
||
Format |
Enumeration, one of:
|
The source format of the external data | x |
Table Schema
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Table Fields |
Array of Table Field Configuration |
Table field contains Datatype , Field Mode , Field Name , Field Type , Field Description to configure Table Schema |
User Defined Function Configuration
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Type |
Enumeration, one of:
|
Type of user-defined function. User defined functions can be provided inline as code blobs ({@link #INLINE}) or as a Google Cloud Storage URI ({@link #FROM_URI}). | x |
|
Content |
String |
If type is INLINE, it returns a code blob. If type is FROM_URI, it returns a Google Cloud Storage URI (e.g. gs://bucket/path). | x |
User Acl
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
String |
User’s email address. |
x |
||
User Role |
Enumeration, one of:
|
Role specified to user |
x |
Domain Acl
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Domain |
String |
The domain name. |
x |
|
Role |
Enumeration, one of:
|
Role specified to domain |
x |
View Acl
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Project |
String |
Project IDs must contain 6-63 lowercase letters, digits, or dashes. IDs must start with a letter and may not end with a dash. |
x |
|
Dataset |
String |
Dataset identity. Dataset ID must be alphanumeric (plus underscores, dashes, and colons) and must be at most 1024 characters long. |
x |
|
Table |
String |
Table IDs must be alphanumeric (plus underscores) and must be at most 1024 characters long. |
x |
|
Role |
Enumeration, one of:
|
Role specified to view |
x |
Copy Job
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Source Tables |
Array of Table Configuration |
Contains Table and Dataset name. It will copy the data from one or more source tables to a destination table. |
x |
|
Destination Table |
Data will get copied to a destination table. |
x |
||
Create Disposition |
Enumeration, one of:
|
Whether the job is allowed to create new tables CREATE_IF_NEEDED :- Configures the job to create the table if it does not exist. CREATE_NEVER :- Configures the job to fail with a not-found error if the table does not exist. Create Disposition |
||
Write Disposition |
Enumeration, one of:
|
Action that should occur if the destination table already exists. WRITE_APPEND :- Configures the job to append data to the table if it already exists. WRITE_EMPTY :- Configures the job to fail with a duplicate error if the table already exists. WRITE_TRUNCATE :- Configures the job to overwrite the table data if table already exists. Write Disposition |
||
Kms Key Name |
String |
KMS key name for encryption configuration |
Extract Job
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Destination Uris |
Array of String |
The list of fully-qualified Google Cloud Storage URIs where the extracted table should be written. |
x |
|
Source Table |
The table to export. |
x |
||
Compression |
String |
the compression value of exported files. |
||
Field Delimiter |
String |
The delimiter used between fields in the exported data. |
||
Format |
String |
The exported files format. |
||
Print Header |
Boolean |
Whether an header row is printed with the result. |
false |
Load Job
Field | Type | Description | Default Value | Required |
---|---|---|---|---|
Source Uris |
Array of String |
Fully-qualified URIs that point to source data in Google Cloud Storage |
x |
|
Destination Table |
Destination table to load the data into |
x |
||
Autodetect |
Boolean |
Automatic inference of the options and schema for CSV and JSON sources |
true |
|
Format Option |
Enumeration, one of:
|
Source format and parsing options of the external data |
||
Ignore Unknown Values |
Boolean |
BigQuery should allow extra values that are not represented in the table schema |
false |
|
Max Bad Records |
Number |
Maximum number of bad records that BigQuery can ignore when running the job |
||
Null Marker |
String |
String that represents a null value in a CSV file |
||
Destination Table Schema |
Schema for the destination table |
|||
Create Disposition |
Enumeration, one of:
|
Whether the job is allowed to create new tables Create Disposition |
||
Kms Key Name |
String |
Creates a EncryptionConfiguration object for KMS key name |
||
Write Disposition |
Enumeration, one of:
|
Action that should occur if the destination table already exists Write Disposition |
||
Allow Fixed Relaxtion |
Boolean |
Allow relaxing a required field in the original schema to nullable when schema of the destination table to be updated as a side effect of the load job |
true |
|
Allow Field Addition |
Boolean |
Allow adding a nullable field to the schema when schema of the destination table to be updated as a side effect of the load job |
true |