What are the responsibilities and job description for the Aws Data Engineer (Pyspark) position at NimbusAITech LLC?
Job Details
Data Engineer
Location: Richmond, VA
Job Responsibilities
Design, develop, and maintain scalable data pipelines to support data ingestion and processing.
Collaborate with cross-functional teams to understand data requirements and deliver solutions that meet business needs.
Implement data models and architectures that support analytics and reporting.
Utilize various programming languages and tools such as Python, Java, and SQL to manipulate and analyze data.
Work with cloud technologies, specifically AWS services like Lambda, EMR, and DynamoDB, to build and deploy data solutions.
Integrate data from various sources using tools like Kafka and Spark.
Monitor and optimize data pipelines for performance and reliability.
Ensure data quality and integrity through rigorous testing and validation processes.
Document data processes and workflows for future reference and compliance.
Requirements
Bachelor's degree in Computer Science, Engineering, or a related field.
Proven experience as a Data Engineer or in a similar role within the banking industry.
Strong proficiency in programming languages such as Python and Java.
Experience with web technologies including HTML, CSS, and React.
Familiarity with data processing frameworks like Spark and data orchestration tools such as Airflow or Step Functions.
Hands-on experience with AWS services, particularly Lambda, EMR, and DynamoDB.
Knowledge of data warehousing solutions like Snowflake and Databricks.
Excellent problem-solving skills and the ability to work independently as well as part of a team.
Strong communication skills to effectively collaborate with stakeholders.
