Gen AI / ML Application Testing Engineer (BA + QA)

Job Title :: Gen AI / ML Application Testing Engineer (BA QA)
Location :: San Jose, CA- Hybrid- Tue,Wed and Thu (ONLY LOCALS)
Duration :: Long Term

Role Summary ::

We are looking for a detail-oriented engineer with experience in Gen AI / ML application testing, business analysis, and product validation. You will help shape the quality of next-gen AI products through systematic testing, prompt validation, and tool-driven evaluation.

Key Responsibilities ::

???Design and execute test cases for Gen AI / ML features and user workflows
???Use GenAI-specific tools to validate prompt consistency, hallucination detection, etc.
???Collaborate with product managers to convert requirements into test cases and test data
???Perform exploratory testing, regression, and prompt-based scenario testing
???Write automation scripts to simulate user behavior and backend interactions
???Track and manage issues using QA platforms and agile tools
???Document test plans, test reports, and AI evaluation metrics

Required Skills ::

???Hands-on testing experience with Gen AI / ML products
???Experience with LLM testing tools like:
???Promptfoo (prompt testing & evaluation)
???LangSmith (LangChain tracing & evals)
???TruLens (feedback tracking for LLMs)
???Rebuff (security and behavior testing)
???Solid understanding of LLM behavior, hallucinations, prompt design
???Scripting: Python, Shell, or JavaScript
???REST APIs, JSON, YAML
???Familiarity with PyTest, Postman, Selenium, or similar tools
Nice-to-Have Skills
???Experience testing RAG, chatbot, or LLM agent systems
???Familiarity with LangChain, LlamaIndex, or Haystack
???Business analysis experience in AI projects
???Knowledge of AI/ML model evaluation metrics
Education
Bachelor's or Master's in CS, Data Science, AI/ML, or related field

Employers have access to artificial intelligence language tools (“AI”) that help generate and enhance job descriptions and AI may have been used to create this description. The position description has been reviewed for accuracy and Dice believes it to correctly reflect the job opportunity.