Job Title: Data Engineer
Location: Travel required to unanticipated client locations throughout the U.S.
Job Description:
Experience
Required:
5 years
Education
Required:
Bachelor’s degree
Job Duties:
Design
data pipeline architecture using AZURE Databricks, SQL, PYTHON, and
PYSPARK.
Data
Modelling and source system analysis for both structured &
unstructured data.
Create
ML Models to predict the future trends of a definite time frame.
Build
algorithms and prototypes.
Debug
Cluster & Job logs to locate & fix the errors.
Optimizing
the program/code to process Gigabytes of data within a fraction of
minutes.
Modifying
the AZURE Cluster configurations and repartition the data as per
requirement.
Follow
the SLDC methodologies for requirement, analysis, design, development,
and testing.
Design
data quality framework using PYSPARK and PYTHON.
Support
continuing increases in data volume and complexity.
Develop
analytical tools using Databricks SQL Queries and PYSPARK in PALANTIR.
Creating
multiple dashboards/visualizations depicting trends of KPIs.
Interpret
trends and patterns of dynamic & complex datasets.
Program
web based Interactive UI for end customers, consuming the data with a
single API.
Create
& update documentation to maintain the requirements of the project.
Documents
include detailed specifications, implementation guides, architecture
diagrams.
Understand
Business problem statements and help Business take financial decisions
for future months based on the prediction and analysis made on the
available data.
Assists
our client envision, contributing significantly and single handedly to
multiple phases of solution.
Create
interactive dynamic dashboards using Palantir & Databricks SQL,
depicting, multiple monitoring metrics and daily monthly future forecasts
for telecom business used by Finance & Marketing dept for decision
making on Budget.
Create
an automation for Cleaning, Merging, Visualizing & Transforming
datasets from multiple data sources using Azure Databrick and created an
application using Python and H2O, which helped give an overview on the
scope of Tax saving.