← All blog
Blog

Introducing FedOps VLM Framework


What is FedOps VLM Framework?

FedOps VLM Framework is an open, cloud-native federated learning (FL) platform purpose-built for Vision-Language Models (VLMs). It bridges the gap between large-scale multimodal AI and privacy-preserving distributed training — enabling researchers and practitioners to fine-tune powerful VLMs across decentralized edge devices without ever centralizing raw data.

Built on top of the FedOps ecosystem (PyPI fedops), the framework combines:

  • Flower — for robust FL communication and client–server orchestration
  • PEFT / LoRA / QLoRA — for parameter-efficient fine-tuning of large VLMs on resource-constrained devices
  • FedMAP (Parameter-Aware Federated Aggregation) — a novel component-aware weighted aggregation theorem that achieves tighter convergence bounds than flat averaging when LoRA and projection parameters have different learning dynamics.

The result is a full-stack FL solution that works from a web portal download all the way to a Kubernetes-managed cloud FL server — reducing the barrier to federated VLM research to just a few configuration steps.


Core Concepts

Federated Learning for VLMs

Traditional VLM fine-tuning requires pooling sensitive data (medical images, private conversations, user photos) into a central server. FedOps VLM Framework flips this: the data never leaves the device. Each client trains a local adapter (LoRA / QLoRA), and only lightweight parameter updates are sent to the server for aggregation.

Plugin-Based Architecture

The framework is plugin-based — models, datasets, and aggregation strategies are swappable components. You pick your combination from the web portal, download a ready-to-run codebase, and start training. No boilerplate wiring required.

Supported Models

Model Parameter Size Quantization
OneVision 0.5B QLoRA (4-bit)
PhiVA 3.8B QLoRA (4-bit)

Custom models can be added via models.py

Supported Datasets

Dataset HuggingFace ID Domain Task Size
VQA-RAD flaviagiammarino/vqa-rad Medical radiology Visual Question Answering 1793 train, 451 test
VQAv2 HuggingFaceM4/VQAv2 General vision Visual Question Answering Large-scale, diverse image-question pairs
PathVQA flaviagiammarino/path-vqa Pathology Visual Question Answering ~19.7K train — histology & microscopy, yes/no + open-ended
SLAKE mdwiratathya/SLAKE-vqa-english Multi-organ medical Visual Question Answering ~4.9K train, 1K test — brain, chest, abdomen (X-ray, CT, MRI)
ChartQA HuggingFaceM4/ChartQA Chart & figure reasoning Visual Question Answering ~18K train — bar charts, line graphs, pie charts
DocVQA HuggingFaceM4/DocVQA Document understanding Visual Question Answering ~10K train — scanned business & industrial documents

Custom datasets can be added via data_preparation.py

Aggregation Strategies

Strategy Description
FedAvg Standard weighted average of client parameters
FedMAP Component-aware quadratic programming aggregation — separate λ weights for LoRA adapters vs. projection layers

Architecture Overview

image.png

Client side runs locally (laptop, workstation) using:

  • client_main.py — FL training loop with PEFT adapters
  • client_manager_main.py — heartbeat and connection management
  • data_preparation.py — local dataset loading + preprocessing

Server side runs in FedOps’ managed Kubernetes environment:

  • server_main.py — global aggregation logic
  • conf/config.yaml — FL rounds, learning rate, strategy config
  • parameter_shapes.json — model architecture metadata for aggregation routing

🚀 Product Release Announcement

Introducing FedOps VLM Framework — Federated Fine-Tuning for Vision-Language Models, Made Simple

We are excited to announce the public release of the FedOps VLM Framework, the first end-to-end federated learning platform specifically designed for multimodal Vision-Language Models.

Why We Built This

As VLMs like OneVision and PhiVA move from research labs into real-world applications — medical diagnosis, assistive technology, on-device AI — the challenge of fine-tuning them on private, distributed data becomes critical. Existing FL frameworks were built for CNNs and simple classifiers. They lack the tooling to handle:

  • Billion-parameter models with heterogeneous adapter architectures
  • Multi-component parameter spaces (vision encoder, language model, connector projector)

FedOps VLM Framework was built from the ground up to solve these problems.

What’s Included

  • Web portal codebase generator — select your model, dataset, and strategy; download a fully configured project in seconds
  • Managed K8s FL server — no infrastructure setup; launch a server pod from the portal with one click
  • PEFT-native client — LoRA and QLoRA adapters out of the box; tested on consumer-grade GPUs
  • FedMAP Theorem implementation — component-aware aggregation with real-time CGH monitoring

Get Started in 5 Minutes

  1. Visit https://gachon-cclab.github.io/fedopsMultimodal/
  2. Select your model + dataset + strategy → Download codebase
  3. conda create -n fedops_multimodal_env python=3.12.7 && conda activate fedops_multimodal_env
  4. bash setup.sh && python client_main.py
  5. Spin up your server pod at https://ccl.gachon.ac.kr/fedops/task

Full setup guide: see https://www.notion.so/giai/How-to-Use-FedOps-VLM-Framework-34e5dfbe76bb8019aa84eafccec768a5 in this repository.

Who Is This For?

  • ML researchers working on privacy-preserving multimodal learning
  • Healthcare AI teams needing to fine-tune VQA models across hospital silos
  • Financial Firm teams needing to fine-tune VQA models across Business & law docs silos
  • Any privacy sensitive data teams needing to fine-tune VQA models across their sensitive data silos
  • FL practitioners looking for a production-ready alternative to vanilla Flower setups for large models.

Resource URL
Web Portal https://gachon-cclab.github.io/fedopsMultimodal/
Server Management https://ccl.gachon.ac.kr/fedops/task
FedOps PyPI https://pypi.org/project/fedops/

FedOps VLM Framework is developed and maintained by Gachon CCLab.