Job Title: Data Engineer (Google Cloud Platform)
Job Summary:
We are seeking a skilled Data Engineer with expertise in Google Cloud Platform (GCP) to design, build, and maintain scalable data pipelines. This role involves working with BigQuery, Cloud Storage, Dataflow, and other GCP services to process and analyze large datasets. Additionally, the candidate will support machine learning model deployment and hosting, ensuring seamless integration with data pipelines.
The ideal candidate will have a strong understanding of ETL processes, data modeling, cloud infrastructure, and MLOps, ensuring efficient and reliable data movement, transformation, and model serving.
Key Responsibilities:
- Design, develop, and optimize ETL pipelines to process structured and unstructured data.
- Build and maintain real-time and batch data processing solutions using GCP services like BigQuery, Dataflow, Pub/Sub, and Cloud Functions.
- Ensure data quality, security, and governance across all pipelines.
- Monitor data workflows, troubleshoot failures, and improve performance.
- Work with stakeholders to understand data requirements and implement scalable solutions.
- Automate data processes using Python, SQL, and infrastructure-as-code tools like Terraform and Cloud Deployment Manager.
- Optimize data warehouse solutions for performance and cost efficiency.
- Deploy and manage machine learning models using services like Vertex AI, AI Platform, Cloud Run, and Cloud Functions.
- Support model versioning, monitoring, and retraining pipelines in collaboration with Data Scientists and ML Engineers.
- Implement MLOps best practices for CI/CD pipelines integrating ML model training, deployment, and serving.
Preferred Qualifications:
- Experience integrating BigQuery with third-party tools like Salesforce, Looker, or Power BI.
- Exposure to model performance monitoring and automated retraining pipelines.
- Knowledge of feature engineering, feature stores, and ML pipeline orchestration.
Education Requirements:
- BE/BTech/Masters/MCA in Computer Science, Information Technology, or a related field.