Data Engineer – AI

Job Description: • Define and drive the technical vision for data platforms that support AI-powered features in Crossplane and Upbound Spaces • Lead the design of data pipelines that transform infrastructure and data into training datasets for ML models • Architect vector search and RAG systems that leverage Crossplane Control Planes & Upbound Marketplace as a knowledge store • Build data infrastructure that processes resources, extensions, and compositions for semantic search • Establish frameworks for collecting, processing, and analyzing infrastructure configuration data • Design data pipelines that handle Crossplane-specific data • Create infrastructure for indexing and searching Upbound Marketplace content, documentation, and community patterns • Develop metrics and monitoring for AI features integrated with Upbound's control plane architecture • Design data systems that power AI agents for infrastructure provisioning & operations, helping users generate and optimize Crossplane compositions • Create feature engineering platforms that extract signals from control plane operations, resource status, and reconciliation patterns • Implement data infrastructure for training models that predict infrastructure failures, optimize resource allocation, and suggest configuration improvements • Drive the development of knowledge graph representations of infrastructure dependencies and relationships Requirements: • 10+ years of software/data engineering experience with at least 4 years in technical leadership roles • Proven track record building data platforms that support production systems at scale • Deep expertise in both traditional data engineering (Spark, Airflow, data lakes) and ML-specific infrastructure (feature stores, model serving) • Experience with vector databases (Pinecone, Weaviate, Qdrant, Milvus, pgvector, Opensearch, ElasticSearch) • Demonstrated experience with LLM applications, including RAG architectures and semantic search implementations • Understanding of Kubernetes, cloud-native architectures, and infrastructure-as-code principles • Strong understanding of data requirements for AI/ML systems: training pipelines, feature stores, and inference infrastructure • Hands-on experience building knowledge bases and semantic search systems for technical documentation and code • Experience with embedding models for code and technical documentation • Knowledge of time-series data processing for infrastructure metrics and events • Understanding of graph databases and their application to infrastructure dependency modeling Benefits: • Health insurance • 401(k) matching • Flexible work hours • Paid time off • Remote work options Apply tot his job