Skip to main content

utils

Utility functions for HuggingFace data.

Module​

Functions​

get_data_factory_dataset​

def get_data_factory_dataset(    datasource: BaseSource,    data_split: DataSplit,    selected_cols: List[str],    selected_cols_semantic_types: Mapping[_SemanticTypeValue, List[str]],    batch_transforms: Optional[List[Dict[str, _JSONDict]]],    labels2id: Optional[Dict[str, int]] = None,    target: Optional[Union[str, List[str]]] = None,) ‑> Tuple[_BaseHuggingFaceDataFactory, Union[_IterableHuggingFaceDataset, _HuggingFaceDataset]]:

Get the HuggingFace data factory and dataset for the given datasource.