pipelines

Operations on a pipelines resource.

Overview

Name	pipelines
Type	Resource
Id	databricks_workspace.deltalivetables.pipelines

Fields

Name	Datatype
name	string
cluster_id	string
creator_user_name	string
latest_updates	array
pipeline_id	string
run_as_user_name	string
spec	object
state	string

Methods

Name	Accessible by	Required Params	Description
get	SELECT	pipeline_id, deployment_name
listpipelines	SELECT	deployment_name	Lists pipelines defined in the Delta Live Tables system.
create	INSERT	deployment_name	Creates a new data processing pipeline based on the requested configuration. If successful, this method returns the ID of the new pipeline.
delete	DELETE	pipeline_id, deployment_name	Deletes a pipeline.
update	UPDATE	pipeline_id, deployment_name	Updates a pipeline with the supplied configuration.
stop	EXEC	pipeline_id, deployment_name	Stops the pipeline by canceling the active update. If there is no active update for the pipeline, this request is a no-op.

SELECT examples

pipelines (listpipelines)

SELECT
name,
cluster_id,
creator_user_name,
latest_updates,
pipeline_id,
run_as_user_name,
spec,
state
FROM databricks_workspace.deltalivetables.pipelines
WHERE deployment_name = '{{ deployment_name }}';

pipelines (get)

SELECT
name,
cluster_id,
creator_user_name,
latest_updates,
pipeline_id,
run_as_user_name,
spec,
state
FROM databricks_workspace.deltalivetables.pipelines
WHERE pipeline_id = '{{ pipeline_id }}' AND
deployment_name = '{{ deployment_name }}';

INSERT example

Use the following StackQL query and manifest file to create a new pipelines resource.

pipelines

/*+ create */
INSERT INTO databricks_workspace.deltalivetables.pipelines (
deployment_name,
data__continuous,
data__name,
data__clusters,
data__libraries,
data__storage
)
SELECT 
'{{ deployment_name }}',
'{{ continuous }}',
'{{ name }}',
'{{ clusters }}',
'{{ libraries }}',
'{{ storage }}'
;

Manifest

- name: your_resource_model_name
  props:
  - name: continuous
    value: false
  - name: name
    value: Wikipedia pipeline (SQL)
  - name: clusters
    value:
    - label: default
      autoscale:
        min_workers: 1
        max_workers: 5
        mode: ENHANCED
  - name: libraries
    value:
    - notebook:
        path: /Users/username/DLT Notebooks/Delta Live Tables quickstart (SQL)
  - name: storage
    value: /Users/username/data

UPDATE example

Updates a pipelines resource.

/*+ update */
-- replace field1, field2, etc. with the fields you want to update        
UPDATE databricks_workspace.deltalivetables.pipelines
SET field1 = '{{ value1 }}',
field2 = '{{ value2 }}', ...
WHERE pipeline_id = '{{ pipeline_id }}' AND
deployment_name = '{{ deployment_name }}';

DELETE example

Deletes a pipelines resource.

/*+ delete */
DELETE FROM databricks_workspace.deltalivetables.pipelines
WHERE pipeline_id = '{{ pipeline_id }}' AND
deployment_name = '{{ deployment_name }}';