Introducing FedOps VLM Framework
What is FedOps VLM Framework?
FedOps VLM Framework is an open, cloud-native federated learning (FL) platform purpose-built for Vision-Language Models (VLMs). It bridges the gap between large-scale multimodal AI and privacy-preserving distributed training — enabling researchers and practitioners to fine-tune powerful VLMs across decentralized edge devices without ever centralizing raw data.
Built on top of the FedOps ecosystem (PyPI fedops), the framework combines:
- Flower — for robust FL communication and client–server orchestration
- PEFT / LoRA / QLoRA — for parameter-efficient fine-tuning of large VLMs on resource-constrained devices
- FedMAP (Parameter-Aware Federated Aggregation) — a novel component-aware weighted aggregation theorem that achieves tighter convergence bounds than flat averaging when LoRA and projection parameters have different learning dynamics.
The result is a full-stack FL solution that works from a web portal download all the way to a Kubernetes-managed cloud FL server — reducing the barrier to federated VLM research to just a few configuration steps.
Core Concepts
Federated Learning for VLMs
Traditional VLM fine-tuning requires pooling sensitive data (medical images, private conversations, user photos) into a central server. FedOps VLM Framework flips this: the data never leaves the device. Each client trains a local adapter (LoRA / QLoRA), and only lightweight parameter updates are sent to the server for aggregation.
Plugin-Based Architecture
The framework is plugin-based — models, datasets, and aggregation strategies are swappable components. You pick your combination from the web portal, download a ready-to-run codebase, and start training. No boilerplate wiring required.
Supported Models
| Model | Parameter Size | Quantization |
|---|---|---|
| OneVision | 0.5B | QLoRA (4-bit) |
| PhiVA | 3.8B | QLoRA (4-bit) |
Custom models can be added via
models.py
Supported Datasets
| Dataset | HuggingFace ID | Domain | Task | Size |
|---|---|---|---|---|
| VQA-RAD | flaviagiammarino/vqa-rad |
Medical radiology | Visual Question Answering | 1793 train, 451 test |
| VQAv2 | HuggingFaceM4/VQAv2 |
General vision | Visual Question Answering | Large-scale, diverse image-question pairs |
| PathVQA | flaviagiammarino/path-vqa |
Pathology | Visual Question Answering | ~19.7K train — histology & microscopy, yes/no + open-ended |
| SLAKE | mdwiratathya/SLAKE-vqa-english |
Multi-organ medical | Visual Question Answering | ~4.9K train, 1K test — brain, chest, abdomen (X-ray, CT, MRI) |
| ChartQA | HuggingFaceM4/ChartQA |
Chart & figure reasoning | Visual Question Answering | ~18K train — bar charts, line graphs, pie charts |
| DocVQA | HuggingFaceM4/DocVQA |
Document understanding | Visual Question Answering | ~10K train — scanned business & industrial documents |
Custom datasets can be added via
data_preparation.py
Aggregation Strategies
| Strategy | Description |
|---|---|
| FedAvg | Standard weighted average of client parameters |
| FedMAP | Component-aware quadratic programming aggregation — separate λ weights for LoRA adapters vs. projection layers |
Architecture Overview

Client side runs locally (laptop, workstation) using:
client_main.py— FL training loop with PEFT adaptersclient_manager_main.py— heartbeat and connection managementdata_preparation.py— local dataset loading + preprocessing
Server side runs in FedOps’ managed Kubernetes environment:
server_main.py— global aggregation logicconf/config.yaml— FL rounds, learning rate, strategy configparameter_shapes.json— model architecture metadata for aggregation routing
🚀 Product Release Announcement
Introducing FedOps VLM Framework — Federated Fine-Tuning for Vision-Language Models, Made Simple
We are excited to announce the public release of the FedOps VLM Framework, the first end-to-end federated learning platform specifically designed for multimodal Vision-Language Models.
Why We Built This
As VLMs like OneVision and PhiVA move from research labs into real-world applications — medical diagnosis, assistive technology, on-device AI — the challenge of fine-tuning them on private, distributed data becomes critical. Existing FL frameworks were built for CNNs and simple classifiers. They lack the tooling to handle:
- Billion-parameter models with heterogeneous adapter architectures
- Multi-component parameter spaces (vision encoder, language model, connector projector)
FedOps VLM Framework was built from the ground up to solve these problems.
What’s Included
- Web portal codebase generator — select your model, dataset, and strategy; download a fully configured project in seconds
- Managed K8s FL server — no infrastructure setup; launch a server pod from the portal with one click
- PEFT-native client — LoRA and QLoRA adapters out of the box; tested on consumer-grade GPUs
- FedMAP Theorem implementation — component-aware aggregation with real-time CGH monitoring
Get Started in 5 Minutes
- Visit https://gachon-cclab.github.io/fedopsMultimodal/
- Select your model + dataset + strategy → Download codebase
conda create -n fedops_multimodal_env python=3.12.7 && conda activate fedops_multimodal_envbash setup.sh && python client_main.py- Spin up your server pod at https://ccl.gachon.ac.kr/fedops/task
Full setup guide: see https://www.notion.so/giai/How-to-Use-FedOps-VLM-Framework-34e5dfbe76bb8019aa84eafccec768a5 in this repository.
Who Is This For?
- ML researchers working on privacy-preserving multimodal learning
- Healthcare AI teams needing to fine-tune VQA models across hospital silos
- Financial Firm teams needing to fine-tune VQA models across Business & law docs silos
- Any privacy sensitive data teams needing to fine-tune VQA models across their sensitive data silos
- FL practitioners looking for a production-ready alternative to vanilla Flower setups for large models.
Links
| Resource | URL |
|---|---|
| Web Portal | https://gachon-cclab.github.io/fedopsMultimodal/ |
| Server Management | https://ccl.gachon.ac.kr/fedops/task |
| FedOps PyPI | https://pypi.org/project/fedops/ |
FedOps VLM Framework is developed and maintained by Gachon CCLab.