What are the responsibilities and job description for the Big Data Engineer position at CALTEK Solutions Inc.?
JD and responsibilities as below.
Location: Burlingame CA
Work arrangement : Work from Office
Required years of experience: 6-8 years
Responsibilities:
Design and implement scalable data pipelines to extract, transform, and load data from various sources into Meta's data warehouse Develop and maintain data processing and analytics applications using languages such as Python, Java, and SQL Work closely with data scientists and analysts to understand their requirements and develop solutions to meet their needs Optimize data access and retrieval performance by implementing caching, indexing, and other optimization techniques Ensure data quality and integrity by implementing data validation and testing processes Collaborate with cross-functional teams to design and implement data-driven solutions that meet business needs Stay up-to-date with emerging technologies and best practices in data engineering and related fields
Requirements:
- Strong programming skills in languages such as Python, Java, and SQL Experience with data pipeline tools such as Apache Airflow, Luigi, or Dataswarm Experience with big data technologies such as Hadoop, Spark, or Hive Familiarity with cloud computing platforms such as AWS, GCP, or Azure Excellent problem-solving skills and ability to work independently Strong communication and collaboration skills