Skip to main content

clusters

Operations on a clusters resource.

Overview​

Nameclusters
TypeResource
Iddatabricks_workspace.compute.clusters

Fields​

NameDatatype
autotermination_minutesinteger
aws_attributesobject
cluster_idstring
cluster_namestring
cluster_sourcestring
creator_user_namestring
default_tagsobject
disk_specobject
driver_instance_sourceobject
driver_node_type_idstring
enable_elastic_diskboolean
enable_local_disk_encryptionboolean
init_scripts_safe_modeboolean
instance_sourceobject
last_state_loss_timeinteger
node_type_idstring
num_workersinteger
spark_context_idinteger
spark_versionstring
start_timeinteger
statestring
state_messagestring
terminated_timeinteger
termination_reasonobject

Methods​

NameAccessible byRequired ParamsDescription
getSELECTcluster_id, deployment_nameRetrieves the information for a cluster given its identifier. Clusters can be described while they are running, or up to 60 days after they are terminated.
listSELECTdeployment_nameReturn information about all pinned and active clusters, and all clusters terminated within the last 30 days. Clusters terminated prior to this period are not included.
createINSERTdeployment_nameCreates a new Spark cluster. This method will acquire new instances from the cloud provider if necessary. This method is asynchronous; the returned
deleteDELETEdeployment_nameTerminates the Spark cluster with the specified ID. The cluster is removed asynchronously. Once the termination has completed, the cluster will be in a
updateUPDATEdeployment_nameUpdates the configuration of a cluster to match the partial set of attributes and size. Denote which fields to update using the
editREPLACEdeployment_nameUpdates the configuration of a cluster to match the provided attributes and size. A cluster can be updated if it is in a
changeownerEXECdeployment_nameChange the owner of the cluster. You must be an admin and the cluster must be terminated to perform this operation. The service principal application ID can be supplied as an argument to
permanentdeleteEXECdeployment_namePermanently deletes a Spark cluster. This cluster is terminated and resources are asynchronously removed.
pinEXECdeployment_namePinning a cluster ensures that the cluster will always be returned by the ListClusters API. Pinning a cluster that is already pinned will have no effect. This API can only be called by workspace admins.
resizeEXECdeployment_nameResizes a cluster to have a desired number of workers. This will fail unless the cluster is in a
restartEXECdeployment_nameRestarts a Spark cluster with the supplied ID. If the cluster is not currently in a
startEXECdeployment_nameStarts a terminated Spark cluster with the supplied ID. This works similar to
unpinEXECdeployment_nameUnpinning a cluster will allow the cluster to eventually be removed from the ListClusters API. Unpinning a cluster that is not pinned will have no effect. This API can only be called by workspace admins.

SELECT examples​

SELECT
autotermination_minutes,
aws_attributes,
cluster_id,
cluster_name,
cluster_source,
creator_user_name,
default_tags,
disk_spec,
driver_instance_source,
driver_node_type_id,
enable_elastic_disk,
enable_local_disk_encryption,
init_scripts_safe_mode,
instance_source,
last_state_loss_time,
node_type_id,
num_workers,
spark_context_id,
spark_version,
start_time,
state,
state_message,
terminated_time,
termination_reason
FROM databricks_workspace.compute.clusters
WHERE deployment_name = '{{ deployment_name }}';

INSERT example​

Use the following StackQL query and manifest file to create a new clusters resource.

/*+ create */
INSERT INTO databricks_workspace.compute.clusters (
deployment_name,
data__kind,
data__cluster_name,
data__spark_version,
data__use_ml_runtime,
data__is_single_node,
data__node_type_id,
data__autoscale,
data__num_workers,
data__data_security_mode,
data__runtime_engine,
data__enable_elastic_disk,
data__aws_attributes,
data__cluster_log_conf,
data__init_scripts,
data__spark_conf,
data__spark_env_vars,
data__driver_node_type_id,
data__ssh_public_keys,
data__custom_tags,
data__single_user_name,
data__autotermination_minutes,
data__enable_local_disk_encryption,
data__policy_id,
data__apply_policy_default_values,
data__instance_pool_id,
data__driver_instance_pool_id,
data__docker_image,
data__workload_type,
data__clone_from
)
SELECT 
'{{ deployment_name }}',
'{{ kind }}',
'{{ cluster_name }}',
'{{ spark_version }}',
 {{ use_ml_runtime }},
 {{ is_single_node }},
'{{ node_type_id }}',
'{{ autoscale }}',
 {{ num_workers }},
'{{ data_security_mode }}',
'{{ runtime_engine }}',
 {{ enable_elastic_disk }},
'{{ aws_attributes }}',
'{{ cluster_log_conf }}',
'{{ init_scripts }}',
'{{ spark_conf }}',
'{{ spark_env_vars }}',
'{{ driver_node_type_id }}',
'{{ ssh_public_keys }}',
'{{ custom_tags }}',
'{{ single_user_name }}',
 {{ autotermination_minutes }},
 {{ enable_local_disk_encryption }},
'{{ policy_id }}',
 {{ apply_policy_default_values }},
'{{ instance_pool_id }}',
'{{ driver_instance_pool_id }}',
'{{ docker_image }}',
'{{ workload_type }}',
'{{ clone_from }}'
;

UPDATE example​

Updates a clusters resource.

/*+ update */
-- replace field1, field2, etc. with the fields you want to update        
UPDATE databricks_workspace.compute.clusters
SET field1 = '{{ value1 }}',
field2 = '{{ value2 }}', ...
WHERE deployment_name = '{{ deployment_name }}';

REPLACE example​

Replaces a clusters resource.

/*+ update */
-- replace field1, field2, etc. with the fields you want to update
REPLACE databricks_workspace.compute.clusters
SET field1 = '{ value1 }',
field2 = '{ value2 }', ...
WHERE deployment_name = '{{ deployment_name }}';

DELETE example​

Deletes a clusters resource.

/*+ delete */
DELETE FROM databricks_workspace.compute.clusters
WHERE deployment_name = '{{ deployment_name }}';