Skip to main content

schema_management

Schema management utilities for the Pod.

Classes​

SchemaGenerationHooks​

class SchemaGenerationHooks():

Prefect hooks for schema generation.

Static methods​

on_schema_worker_completion​

def on_schema_worker_completion(tsk: Task, run: TaskRun, state: State) ‑> None:

Log completion of schema worker task.

on_schema_worker_failure​

def on_schema_worker_failure(tsk: Task, run: TaskRun, state: State) ‑> None:

Log failure of schema worker task.

SchemaManagement​

class SchemaManagement():

Schema management utilities for the Pod.

Static methods​

is_prefect_server_healthy​

def is_prefect_server_healthy() ‑> bool:

Check if the Prefect server is up and running.

resume_suspended_flow​

async def resume_suspended_flow(    flow_run_id: Optional[UUID], datasource_containers: list[DatasourceContainer],) ‑> None:

Resume a suspended flow run with updated datasource configs.

Arguments

  • flow_run_id: ID of the flow run to resume.
  • datasource_containers: List of all datasource containers which will be filtered to those with schema types that are not "full".

schema_worker​

async def schema_worker(    datasource_config: MinimalDatasourceConfig,    schema_generation_config: MinimalSchemaGenerationConfig,    schema_upload_config: MinimalSchemaUploadConfig,    hub_config: HubConfig,) ‑> tuple[str, str]:

Process each record in the dataset with the ability to cancel.

Arguments

  • datasource_config: The datasource config to generate schema for.
  • schema_generation_config: The schema generation config.
  • schema_upload_config: The schema upload config.
  • hub_config: The hub config.

Returns The datasource name and the schema as a JSON string.

suspend_active_flow​

async def suspend_active_flow(flow_name: str) ‑> Optional[uuid.UUID]:

Suspend the most recent active run of a flow.

Arguments

  • flow_name: Name of the flow to suspend.

Returns ID of the suspended flow run if successful, otherwise None.