Job Description: Responsibilities
- Design and implement scalable, high-performance data architectures within the Azure ecosystem, including Azure Data Lake, Azure Synapse Analytics, Azure SQL Database, and Azure Cosmos DB.
- Define data strategies, data flows, and the architecture of data pipelines for both structured and unstructured data.
- rchitect efficient and secure data storage solutions for data lakes, data warehouses, and operational data stores, ensuring alignment with business requirements.
- Create data models, including conceptual, logical, and physical models, to support both transactional and analytical systems.
- Develop and manage end-to-end cloud data architectures using Azure Data Factory, Azure Databricks, Azure Functions, and other services to enable ETL/ELT processes.
- Ensure the architecture supports real-time and batch processing for large datasets.
- Develop and enforce data governance policies, including data quality, lineage, and metadata management using tools like Azure Purview.
- Implement data encryption, masking, and anonymization techniques to protect sensitive data within the architecture.
- Design and manage complex data pipelines and workflows for ingestion, transformation, and loading using Azure Data Factory, Apache Spark, and Databricks.
- Collaborate with data engineers to optimize pipelines for performance, reliability, and scalability.
- Monitor and troubleshoot performance issues in Azure Synapse, Azure SQL Database, and Data Lake Storage to ensure fast data retrieval and processing.
- Collaborate with cross-functional teams, including data engineers, data scientists, software developers, and business stakeholders, to understand their data needs and provide architecture solution.
- Develop detailed architecture documentation, including diagrams, data flows, and technical specifications, to ensure the team and stakeholders understand the data architecture.