What are the responsibilities and job description for the Apache NiFi System Administrator position at Starcom Consultant?
Job Details
Key Tasks
Key Tasks
- Design, implement, and manage data flows within Apache NiFi, ensuring efficient and reliable data movement across systems
- Build scalable workflows for data ingestion, transformation, enrichment, and delivery
- Configure processors, connections, and controllers to optimize data throughput and performance
- Regularly monitor and maintain NiFi clusters, ensuring high availability and fault tolerance
- Implement appropriate security measures, including user access control, data encryption, and SSL configurations within NiFi
- Ensure compliance with organizational data governance policies and regulations
- Work on integrating NiFi with other systems such as databases, cloud services (e.g., Google Cloud Platform, AWS, Azure) and other big data or messaging platforms
- Implement monitoring tools and set up alerts to detect failures or performance issues
- Generate reports on data flow metrics and performance for stakeholders
- Stay up to date with the latest NiFi releases and patches, ensuring that the environment is using the latest stable versions
- Optimize the performance of data transfer workflows by configuring NiFi's backpressure settings, load balancing, and ensuring high availability across NiFi clusters
- Monitor file transfer and API performance, troubleshooting bottlenecks and optimizing workflows as needed for both throughput and latency
- Collaborate with cross-functional teams, including data engineers, security teams, and IT infrastructure teams from both entities, to ensure smooth data flow and integration
- Provide detailed documentation on data flow architecture, security protocols, and API configurations to ensure that the integration process is well-understood and maintainable by both organizations post-acquisition
- Ensure that the data transfer mechanisms built are scalable and sustainable for long-term use
- 5 years of experience in Apache NiFi for managing data flows, file transfers, and API integrations
- Knowledge of secure file transfer protocols (SFTP, FTPS, HTTPS) and encryption methods
- Experience integrating systems using APIs and working with open systems (cloud, on-premise, hybrid)
- Strong understanding of data governance, compliance, and security practices, particularly in high-stakes M&A scenarios
- Excellent troubleshooting and optimization skills for large-scale data migration and integration projects
- Excellent communication skills both written and verbal
Salary : $65 - $75
