DeDLOC enables internet-based collaborative pretraining of sahajBERT Bengali

AI Impact Summary

DeDLOC enables decentralized, internet-based collaborative training across volunteer GPUs/CPUs to pretrain sahajBERT for Bengali, adapting to heterogeneous bandwidth and intermittent connectivity. It replaces centralized parameter servers with adaptive gradient aggregation and decentralized All-Reduce, potentially reducing upfront hardware costs while improving fault tolerance and scalability for large language model pretraining. Technical details include gradient partitioning by connection speed and selective inbound connectivity, supported by the Hivemind library to orchestrate distributed learning.