SAP Data Engineer & BI Developer
: Job Details :

*PLEASE NOTE: If you DO NOT LIVE in the Houston, TX Metro, (preferably in the North Houston area); PLEASE DO NOT apply for this position; thank you...*

**Due to exposure to sensitive/privileged company data, this position requires US Permanent Resident (Green Card), or US Citizen work status**

***This is a ~ 3 month Contract to Fulltime Hire opportunity; with conversion to fulltime occurring sometime in Jan/Feb 2025. This is a Hybrid work position. The schedule is 3 days/week at the Corp HQ, Tues/Wed/Thurs, located in Houston Galleria, 610 West & San Felipe. Therefore, you must live in the Houston, TX Metro area***

IS&T is helping the CIO & Director of Applications for the largest Salt Mining and Derivatives company in the US find a skilled SAP Data Engineer/BI Developer.

This is a hands on Data Engineering & Developer role, with experience in development and implementation of Data Lake, Azure Data Factory, and Databricks.

Notes from a Teams call with CIO and Dir of Apps:

Quote,

• We need someone who is familiar with the latest version of Data Lake techniques and Databricks, from design, development, to implementation
• The key to this role is ability to interact and engage the business - so business acumen is the key skill set we need, apart from the technical skills.
• We have a need to catch up on our backlog requests relating to SAP as a primary data source.
• We'd prefer, well likely need someone with hands on experience with SAP Transaction Tables, databricks, and master data management
• We think the best candidate would be an expert in mapping SAP & Data Lake and knows the latest technologies and techniques in Python & Spark (PySpark)
• Can deliver results; someone who can show current guys how to do the hands on work. I am currently using a Consulting firm from Europe/UK, and want to end that contract as soon as possible
• Prefer someone with hands on skills using Power Platform Power BI
• Someone with hands on skills using Azure Data stack, would be preferred, as we are a Microsoft environment as well as SAP
• Someone with well-developed interaction skills, drive and problem solver

Skills

• SAP - Transaction Tables
• Expert in mapping SAP and Data Lakes and knows latest techniques in Python and Spark
• Databricks; Azure Databricks
• Data Lakes
• Data Lakehouse
• Build, implement, maintain, and improve performance of data pipelines
• Azure Data Factory Azure Data Services
• ETL/ELT
• DBT
• ADF

Create good development patterns & standards:

• Code standards & Data modeling
• Security

Maintain data engineering and development work product, including:

• Source control & Code deployments
• Dev testing & Code reads
• Documentation

Interview process:

The first interview will be a Teams call, or preferably an invite to Corporate offices, meet with me, my Dir of Applications, and the Manager of SAP.

Our goal is to invite this person to our HQ offices and have them meet everybody, then we can make an informed decision.

Might be need for 2nd interview - only if a key person is unavailable. We'd like to be thorough so we can make a good, and quick decision...

formal job description:

Job Summary

The Data Engineer & Developer will be responsible for supporting the design, development, and maintenance of Databricks, Data Lake/s, data pipelines and data processing systems. They will utilize technologies to include SAP (data source), SAP Data mapping, SAP transaction tables, Python, PySpark, Azure, SQL, Spark, SQL Server, Azure Data Factory, Databricks/Azure Databricks, Azure DevOps, Data build Tool, Git, and Power BI.

Working daily with IT leaders, stakeholders, and other SMEs and will play a crucial role in building and ensuring the availability, reliability, and scalability of the data infrastructure and in supporting the companys data-products within the company.

Job Duties/Roles

Data pipeline development: Collaborate with the data engineering team to develop and maintain data pipelines, ETL/ELT processes, and data integration workflows using Azure Data Factory, Azure Synapse Analytics, Spark SQL, and other relevant technologies. Ensure the timely and accurate movement of data between systems.

Data processing and transformation: Develop & implement / maintain data processing and transformation logic using Azure Data Factory, Databricks, and Spark/Spark SQL. Extract insights from raw data and transform it into meaningful and structured formats in our data products. Also will work on Power BI platform for reporting and data modeling. DBT (Data Build Tool) experience is a plus, for implementation and management of DBT workflows.

Azure service utilization: Work with Azure services such as SQL Server, Azure Data Factory, and Azure Databricks to build and manage data infrastructure components. Leverage the capabilities of these services to ensure efficient and scalable data processing.

Version control and collaboration: Utilize Azure DevOps and Git for version control and collaborate effectively with the team to manage code repositories and ensure proper documentation and knowledge sharing.

Azure DevOps integration: Integrating data engineering workflows with Azure DevOps for continuous integration, continuous deployment, and automated testing. Contribute to the implementation of CI/CD pipelines for data engineering projects.

Data Modelling, visualization and reporting: Demonstrated skills in Power BI reporting tools. Ability to collaborate with cross-functional teams to gather requirements and translate them into optimal data models (dimensional modelling experience required).

Troubleshooting and support: Assist in identifying and resolving data pipeline issues, bottlenecks, and data quality problems. Provide support in investigating and troubleshooting data-related incidents.

Continuous learning and growth: Stay updated with the latest advancements in data engineering technologies and tools. Continuously enhance your knowledge of Azure services and data engineering recommended practices.

Familiarity with data modelling tools (ER Studio | Power Designer) a plus

Required Skills/Abilities:

7 or more years of experience in data engineering or related data lead roles.

7 or more years of direct interaction and engagement with Business users and stakeholders. Business acumen is a required skill set for this role, apart from the technical skills.

Hands-on experience with SAP from Data Engineering/BI Development, Azure services, including Azure Data Factory, and Databricks.

Proficiency in SQL programming, including SQL Server databases.

Familiarity with Spark SQL and Databricks for data processing.

Experience in DevOps practices for CI/CD and testing in data & analytics teams.

Experience with Power BI for data visualization and reporting.

Proven experience in Azure Databricks

Strong problem-solving skills and attention to detail.

Effective communication and teamwork skills.

Passion for using latest techniques in data engineering.

In-depth understanding of data lake, lakehouse design, engineering and data modeling.

Proficiency in data pipeline development and data integration.

Solid understanding of source control (preferably git or Azure DevOps) and branching techniques.

Strong analytical skills for complex data sets and issue troubleshooting.

Preferred Skills/Abilities:

SAP expertise with SAP Transaction tables, databricks, and Master Data Management

Expert in mapping SAP and Data Lake, knowing latest technologies and techniques to build, maintain, and improve performance of Data pipelines, platforms, tools.

Experience with Azure Synapse

Delta Lake and Parquet files, and YMAL a plus

Experience with DBT (Data Build Tool) is a plus

Working experience with Spark, Spark Pool, and Python

Education and Experience:

A minimum of 7 or more years of related experience required.