Skip to main content

datasplitters

Classes for splitting data.

Classes

DatasetSplitter

class DatasetSplitter():

Parent class for different types of dataset splits.

Ancestors

Static methods


create

def create(    splitter_name: str, **kwargs: Any,)> DatasetSplitter:

Create a DataSplitter of the requested type.

splitter_name

def splitter_name()> str:

Returns string name for splitter type.

Methods


create_dataset_splits

def create_dataset_splits(    self, data: pd.DataFrame,)> Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]:

Returns indices for data sets.

get_filenames

def get_filenames(    self,    datasource: Union[FileSystemIterableSource, _DataViewFromFileIterableSource],    split: DataSplit,)> List[str]:

Returns a list of filenames for a given split.

Only used for file system sources.

Arguments

  • datasource: A FileSystemIterableSource object.
  • split: The relevant split to return filenames for.

Returns A list of filenames.

get_split_query

def get_split_query(self, datasource: DatabaseSource, split: DataSplit)> str:

Modifies the datasource SQL query to return a split of the data.

Only used for database sources.

Arguments

  • datasource: A DatabaseSource object.
  • split: The relevant split to return from the query.

Returns The modified SQL query.

PercentageSplitter

class PercentageSplitter(    validation_percentage: int = 10,    test_percentage: int = 10,    shuffle: bool = True,    time_series_sort_by: Optional[Union[List[str], str]] = None,):

Splits data into sets based on percentages.

The default split is 80% of the data is used training, and 10% for each validation and testing, respectively.

Arguments

  • validation_percentage: The percentage of data to be used for validation. Defaults to 10.
  • test_percentage: The percentage of data to be used for testing. Defaults to 10.
  • time_series_sort_by: A string/list of strings to be used for sorting time series. The strings should correspond to feature names from the dataset. This sorts the dataframe by the values of those features ensuring the validation and test sets come after the training set data to remove potential bias during training and evaluation. Defaults to None.
  • shuffle: A bool indicating whether we shuffle the data for the splits. Defaults to True.

Variables

  • static shuffle : bool
  • static test_percentage : int
  • static time_series_sort_by : Union[List[str], str, ForwardRef(None)]
  • static validation_percentage : int

Static methods


create

def create(    splitter_name: str, **kwargs: Any,)> DatasetSplitter:

Inherited from:

DatasetSplitter.create :

Create a DataSplitter of the requested type.

splitter_name

def splitter_name()> str:

Class method for splitter name.

Returns The string name for splitter type.

Methods


create_dataset_splits

def create_dataset_splits(    self, data: pd.DataFrame,)> Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]:

Create splits in dataset for training, validation and test sets.

Arguments

  • data: The dataframe type object to be split.

Returns A tuple of arrays, each containing the indices from the data to be used for training, validation, and testing, respectively.

get_filenames

def get_filenames(    self,    datasource: Union[FileSystemIterableSource, _DataViewFromFileIterableSource],    split: DataSplit,)> List[str]:

Inherited from:

DatasetSplitter.get_filenames :

Returns a list of filenames for a given split.

Only used for file system sources.

Arguments

  • datasource: A FileSystemIterableSource object.
  • split: The relevant split to return filenames for.

Returns A list of filenames.

get_split_query

def get_split_query(self, datasource: DatabaseSource, split: DataSplit)> str:

Modifies the datasource SQL query to return a split of the data.

caution

This method will only work for databases that support the LIMIT ... OFFSET syntax. Notably, Microsoft SQL Server does not support this syntax.

caution

It is strongly recommended that you sort the data as part of the SQL query in order to ensure the splits are random. This is because for iterable datasets, the splits are simply taken in order from TRAIN to TEST.

Similarly, time_series_sort_by is ignored and a warning logged if True. If you want to sort by time series, you should do this as part of the SQL Query.

Arguments

  • datasource: A DatabaseSource object.
  • split: The relevant split to return from the query.

Returns The modified SQL query.

SplitterDefinedInData

class SplitterDefinedInData(    column_name: str = 'BITFOUNT_SPLIT_CATEGORY',    training_set_label: str = 'TRAIN',    validation_set_label: str = 'VALIDATION',    test_set_label: str = 'TEST',    infer_data_split_labels: bool = False,):

Splits data into sets based on value in each row.

The splitting is done based on the values in a user specified column.

Arguments

  • column_name: The column name for which contains the labels for splitting. Defaults to "BITFOUNT_SPLIT_CATEGORY".
  • training_set_label: The label for the data points to be included in the training set. Defaults to "TRAIN".
  • validation_set_label: The label for the data points to be included in the validation set. Defaults to "VALIDATION".
  • test_set_label: The label for the data points to be included in the test set. Defaults to "TEST".

Variables

  • static column_name : str
  • static infer_data_split_labels : bool
  • static test_set_label : str
  • static training_set_label : str
  • static validation_set_label : str

Static methods


create

def create(    splitter_name: str, **kwargs: Any,)> DatasetSplitter:

Inherited from:

DatasetSplitter.create :

Create a DataSplitter of the requested type.

splitter_name

def splitter_name()> str:

Class method for splitter name.

Returns The string name for splitter type.

Methods


create_dataset_splits

def create_dataset_splits(    self, data: pd.DataFrame,)> Tuple[numpy.ndarray, numpy.ndarray, numpy.ndarray]:

Create splits in dataset for training, validation and test sets.

Arguments

  • data: The dataframe type object to be split.

Returns A tuple of arrays, each containing the indices from the data to be used for training, validation, and testing, respectively.

get_filenames

def get_filenames(    self,    datasource: Union[FileSystemIterableSource, _DataViewFromFileIterableSource],    split: DataSplit,)> List[str]:

Inherited from:

DatasetSplitter.get_filenames :

Returns a list of filenames for a given split.

Only used for file system sources.

Arguments

  • datasource: A FileSystemIterableSource object.
  • split: The relevant split to return filenames for.

Returns A list of filenames.

get_split_query

def get_split_query(self, datasource: DatabaseSource, split: DataSplit)> str:

Modifies the datasource SQL query to return a split of the data.

Arguments

  • datasource: A DatabaseSource object.
  • split: The relevant split to return from the query.

Returns The modified SQL query.