What are the responsibilities and job description for the Data Engineer position at Siri InfoSolutions Inc?
Job Title : Data Engineer
Location : Lewisville Texas
Duration : full time
Job Description :
Identification of Data Pipeline softwaredesign along with prerequisites to develop Python Lambda Functions.
Design Patterns with pros and cons in relation with multiple solution approaches using AWS Stack involving Kinesis / EB / SQS / SNS / CW and provided recommendations to implement Data Pipelines in an agile way.
Swagger definition of REST APIs design of Python and Javaspring boot microservices apigee updates with external users using API gateway.
Defining of Data flows related to SnowPipe automation procedure with AWSS3 / SQS / Dynamo DB.
Definition and design involved with KAFKA Connectors SourceConnectorConfiguration SinkConnectorConfiguration WorkerConnectorConfiguration & Connect with RESTAPIs methods
Enhancing of existing DataViews of Data pipeline by optimizing SQL Joins Inner&Outer & CTEs to orchestrate using Python based Lambdas and utilities developed for cron service to perform automation using Boto3 libraries.
Authoring of IaC / Terraform based automation script to support new Data pipeline setup sandbox environments to support multiple PoCs
Design and Development of AWS CloudWatch based metrics dashboard generation to monetize the network Control Flows & Data Flows.
Key Skills
Apache Hive,S3,Hadoop,Redshift,Spark,AWS,Apache Pig,NoSQL,Big Data,Data Warehouse,Kafka,Scala
Employment Type : Full Time
Vacancy : 1
