Data Engineering Consultant

Documenting Data Lineage and Recommendations

In order to ensure proper data ownership and stewardship, it is crucial to document data lineage effectively and suggest improvements. Below are the qualifications and architecture competencies that can help in this process:

Qualifications

• Programming: Python, SQL, Scala, Java.
• Big Data: Apache Spark, Hadoop, Databricks, Snowflake, etc.
• Cloud: AWS (Glue, Redshift), Azure (Synapse, Data Factory, etc.).
• Modelling & Storage: Relational (PostgreSQL, SQL Server), NoSQL (MongoDB, Cassandra), Dimensional modeling.
• DevOps & Automation: Docker, Kubernetes, Terraform, CI/CD pipelines for data flows.

Architecture Competencies

• Data Modeling: Designing dimensional, relational, and...