Switch to Hugging Face Inference Endpoints for ML inference — latency gains, higher cost

AI Impact Summary

The organization is migrating CPU-based ML inference from AWS ECS/Fargate to Hugging Face Inference Endpoints, aiming to simplify deployment by leveraging HF Hub as the model registry and HF tooling. They report notable latency improvements on CPU endpoints and highlight a straightforward deployment path via GUI, REST API, or the hugie CLI, but acknowledge a 24-50% higher ongoing cost compared to their previous ECS setup. Key considerations include potential vendor lock-in, unaccounted data ingress/ECR costs, and opportunities to host multiple models per endpoint, along with gaps in tooling (e.g., Terraform provider) that will shape future automation and governance requirements.