- Kafka, Spark, Scala, EMR and EKS are used to build the data pipelines for the data lake in the AWS environment.
- Experience in Python scripting is highly preferred.
- Experience in IAM, SAML, MFA are nice to haves. Additionally, experience in Fintech is nice to have but not required.
Must Have Qualifications: Experienced Lead with 10+ years of experience, having knowledge of data engineering and pipelines in AWS environment, knowledge of Kafka, EMR, EKS, Scala, Spark.
Qualifications:
- Bachelor's degree in information technology, engineering, computer science, related field, or equivalent experience. Advanced studies/degree is preferred.
- 10+ years of overall information technology experience with an emphasis on integration and delivery of virtual/cloud platforms to enterprise applications, recent public cloud experience preferred.
- At least three years proven experience with AWS cloud formation, Terraform, or similar tools.
- 5+ years of related industry experience in an enterprise environment.
- 3+ years of experience with engineering and supporting containerization technology (Kubernetes, AWS(ECS/EKS) etc.) at scale.
- Experience with Python, Ansible and shell scripting to automate routine operation tasks.
- Experience in identity and access data correlation, normalization and building of cohesive identity and access data models for large enterprises.
- Single Sign On (SSO) integration and session management for multiple web and cloud applications.
- Identity Federation (SAML) configuration and integration across multiple trusted third parties, applications, and systems.
- Multi Factor Authentication (MFA) such as Duo or Microsoft Authenticator security integration into the authentication, authorization, and single sing on process for applications and systems.
- API security and API integration with IAM systems for sharing identity contexts.
- Experience in Jenkins, Git and Artifactory.
- Formal design patterns and industry best-practices.
- Scripting languages (e.g., Groovy / Perl / Python / Ruby).
- 2+ years of experience across a variety of technologies such databases, directory services, application servers, network infrastructures, Linux operating systems, and an understanding of fundamental security and data flows within these components.
- Scaling CI/CD systems and microservices.
- Code coverage analysis / static analysis tools.
- Agile programming processes and methodologies such as Scrum.
- Overseeing / managing production application deployment.
- Excellent verbal and written communication skills.
- Self-motivated, driven, and creative individual.
- Ability to provide 24x7 operational support on periodic basis and triage complex issues to restore availability.
- Documentation of Engineering Solutions and Designs.