2023: Open LLMs Shift to Data-Focused Approach

AI Impact Summary

2023 marked a significant shift in the open Large Language Model (LLM) landscape, driven by a move beyond simply scaling model size. The release of models like BLOOM, OPT, and GLM-130B demonstrated a focus on training data volume, mirroring the findings of DeepMind's Chinchilla research. This trend highlights a more efficient approach to LLM development, prioritizing data quality and quantity over solely increasing model parameters, and offers a more accessible path for researchers and developers with limited compute resources.