Skip to main content

Referencing a model

Bitfount currently supports referencing models from the following providers:

Bitfount-hosted models​

Models hosted in Bitfount are referenced inside an algorithm that requires a model (such as bitfount.ModelInference) via the model.bitfount_model block:

  • model_ref: the identifier of the model in the Bitfount Hub (excluding the username)
  • model_version: integer version to pin. Otherwise, the latest version will be used. It is recommended to pin the version to avoid unexpected changes to the model as well as to avoid access issues.
  • username: model owner/namespace

Hyperparameters for the model can be set separately within the model block:

  • hyperparameters: arguments to pass to the model constructor. For instance batch size is a commonly set hyperparameter. The specific hyperparameters accepted will differ between different models. If you have access to view the code, you can check the constructor arguments for the model to determine its hyperparameters.
task:
  algorithm:
    - name: bitfount.ModelInference
      model:
        bitfount_model:
          model_ref: CatDogImageClassifier
          model_version: 3
          username: research-user
        hyperparameters:
          batch_size: 8
tip

Bitfount models can be used for inference, evaluation and fine-tuning tasks and can be toggled between public and private. Models from the Hugging Face model hub however must be made public in order to be used in Bitfount. More information about uploading your models to the Bitfount Hub can be found here.

Hugging Face models​

Hugging Face models are invoked via dedicated Bitfount algorithms that are specific to the model type. You pass a model_id (e.g., google/vit-base-patch16-224) in the algorithm arguments. Do not use the model block for Hugging Face models. Confusingly, Hugging Face refers to these model types as tasks.

Available task types​

The task types defined by Hugging Face and supported by Bitfount can be found below with their corresponding Bitfount algorithm:

info

Hugging Face models are currently only supported for inference tasks within Bitfount.

tip

Make sure to choose a model that is compatible with the algorithm you are using. The links above will take you to the Hugging Face model hub filtered for that specific task type.

Image segmentation example​

task:
  algorithm:
    - name: bitfount.HuggingFaceImageSegmentationInference
      arguments:
        model_id: CIDAS/clipseg-rd64-refined
        dataframe_output: true
        batch_size: 1
  data_structure:
    select:
      include:
        - image_path

Image-text-to-text example​

The image-text-to-text task type enables the use of vision-language models that take both an image and a text prompt as input and generate a text response. This is useful for tasks such as medical image captioning or visual question answering. A notable example is MedGemma, a medical vision-language model.

task:
  algorithm:
    - name: bitfount.HuggingFaceImageTextGenerationInference
      arguments:
        model_id: google/medgemma-1.5-4b-it
        max_new_tokens: 500
        prompt_template: "Describe the findings in this medical image given the following clinical notes: {context}"
  data_structure:
    select:
      include:
        - image_path
        - clinical_notes
tip

The prompt_template argument is optional. If provided, the {context} placeholder will be replaced by the value from the context column. If omitted, the context column value is used as the prompt directly. See the API documentation for the full list of configuration options.

TIMM models​

TIMM is a popular PyTorch image models library that provides a collection of the latest pretrained image models. Originally developed independently by Ross Wightman, it has now been brought under the Hugging Face umbrella.

TIMM models are supported by Bitfount for both inference and fine-tuning tasks via the bitfount.TIMMInference and bitfount.TIMMFineTuning algorithms respectively. The model is specified in the same way as Hugging Face models via the model_id argument.

TIMM fine-tuning example​

Hyperparameters for the model can be set separately within the args block. The full list of hyperparameters can be found here. As mentioned in the timm documentation, the variety of training args is large and not all combinations of options (or even options) have been fully tested.

task:
  protocol:
    name: bitfount.ResultsOnly
  algorithm:
    - name: bitfount.TIMMFineTuning
      arguments:
        model_id: bitfount/RETFound_MAE
        labels:
          - "0"
          - "1"
          - "2"
          - "3"
          - "4"
        args:
          epochs: 1
          batch_size: 32
          num_classes: 5

TIMM inference example​

task:
  protocol:
    name: bitfount.InferenceAndCSVReport
  algorithm:
    - name: bitfount.TIMMInference
      arguments:
        model_id: bitfount/RETFound_MAE
        num_classes: 5
    - name: bitfount.CSVReportAlgorithm

MONAI models​

MONAI (Medical Open Network for AI) is a PyTorch-based framework specialising in deep learning for medical imaging. Bitfount supports running inference using pre-trained models from the MONAI Model Zoo via the bitfount.MONAIBundleInference algorithm.

MONAI models are referenced by their bundle name from the MONAI Model Zoo. The algorithm downloads the specified bundle and runs inference using the bundle's pre-trained weights and preprocessing pipeline.

Hardware Recommendation

MONAI models can be computationally intensive. Running on CPU can be very slow, so CUDA-enabled GPU hardware is strongly recommended. Note that MPS (Apple Silicon) is not supported for MONAI models.

MONAI inference example​

task:
  protocol:
    name: bitfount.InferenceAndCSVReport
  algorithm:
    - name: bitfount.MONAIBundleInference
      arguments:
        bundle_name: wholeBrainSeg_Large_UNEST_segmentation
        nifti_output: true
        batch_size: 1
    - name: bitfount.CSVReportAlgorithm
  data_structure:
    select:
      include:
        - image_path

For the full list of configuration options, see the MONAIBundleInference API documentation.