Skip to main content

schema_manager

Per-pod schema lifecycle manager.

Owns schema creation, validation, and configuration migration. Wraps the class-level SchemaManagement utilities (Prefect-based background generation, hub metadata upload) and holds per-pod configuration (hub, keys, DP settings) so callers don't need to pass them on every call.

Responsibilities:

  • setup_schema: Generate or validate a BitfountSchema for a datasource.
  • _schema_config_migration: Normalise legacy nested-by-datasource-name config formats into the flat structure BitfountSchema expects.
  • Prefect flow lifecycle (start/stop/health-check) for background schema generation.
  • Public metadata retrieval for hub upload.

Classes​

SchemaManager​

class SchemaManager(    *,    hub: BitfountHub,    pod_public_key: RSAPublicKey,    access_manager_public_key: RSAPublicKey,    pod_dp: DPPodConfig | None,    username: str | None = None,    secrets: APIKeys | RefreshableJWT | dict[SecretsUse, APIKeys | RefreshableJWT] | None = None,    prefect_unavailable: bool = False,):

Per-pod schema lifecycle: creation, validation, background generation.

Holds instance-specific config (hub, keys, DP settings) and delegates to the static SchemaManagement helpers. Used by DatasourceManager during datasource loading and by Pod for Prefect flow control.

Variables​

  • prefect_unavailable : bool - Whether Prefect is unavailable for background schema generation.

Static methods​

is_prefect_server_healthy​

def is_prefect_server_healthy() ‑> bool:

Check if the Prefect server is up and running.

Methods​

get_public_metadata​

def get_public_metadata(    self,    datasource_container: DatasourceContainer,    ds_add_number_of_records: bool = False,) ‑> PodPublicMetadata:

Get public metadata for a datasource container.

Arguments

  • datasource_container: The container of the datasource to get the metadata for.
  • ds_add_number_of_records: Whether the number of records for the datasource should be retrieved and included in the metadata.

setup_schema​

def setup_schema(    self,    datasource_name: str,    datasource: BaseSource,    data_config: PodDataConfig,    schema: BitfountSchema | None = None,    force_schema_update: bool = False,) ‑> BitfountSchema:

Generate or validate schema for a datasource.

The schema will be generated if:

  • schema update was forced
  • no schema is provided
  • schema is not populated
  • datasource name does not match the table name
  • force stypes do not have appropriate type

Only the first batch of data is used to generate the schema in this method.

start_prefect_flow​

def start_prefect_flow(    self, base_datasources: dict[str, DatasourceContainer], pod_name: str,) ‑> None:

Start background schema generation via Prefect.

Arguments

  • base_datasources: The datasources to generate schemas for.
  • pod_name: Owning pod, carried down only so the flow run can tag itself pod:<pod_name>. Schema generation is pod-wide rather than per-datasource, so there is no task_hash to identify it by and the pod name is the whole of its identity.

stop_background_schema_generation​

def stop_background_schema_generation(self) ‑> None:

Stop any running background schema generation (Prefect flow).