Skip to main content

background_file_counter

Background file counting utility that works with any FileSystemIterableSource.

Module​

Functions​

get_background_file_count​

def get_background_file_count(datasource: "'FileSystemIterableSource'") ‑> Optional[int]:

Get the background file count for a datasource.

serialize_datasource_configs​

def serialize_datasource_configs(    datasource: "'FileSystemIterableSource'",) ‑> Dict[str, Any]:

Create serializable data from datasource configuration.

Arguments

  • datasource: The datasource to serialize

Returns Serialized datasource configuration

start_background_file_counting​

def start_background_file_counting(datasource: "'FileSystemIterableSource'") ‑> None:

Start background file counting for a datasource.

stop_background_file_counting​

def stop_background_file_counting(datasource: "'FileSystemIterableSource'") ‑> None:

Stop background file counting for a datasource.

Classes​

BackgroundFileCounter​

class BackgroundFileCounter(datasource: FileSystemIterableSource):

Non-intrusive background file counter for FileSystemIterableSource.

Arguments

  • datasource: The FileSystemIterableSource to count files for.

The BackgroundFileCounter class allows counting files in a FileSystemIterableSource while a task is executing in a different process. Using threads instead of processes is discouraged because it can block execution within the app, as the Flask server operates in a single-threaded mode, making file counting a blocking task.

Methods​

get_count​

def get_count(self) ‑> Optional[int]:

Get current file count (None if counting not complete).

is_counting_complete​

def is_counting_complete(self) ‑> bool:

Check if counting is complete.

start_counting​

def start_counting(self) ‑> None:

Start background file counting.

stop_counting​

def stop_counting(self) ‑> None:

Stop background file counting.