Chatbot Evaluation through Q&A Test Sets

Open
Main contact
Ample Insight Inc.
Toronto, Ontario, Canada
Associate Partnerships Manager
(1)
3
Project
Academic experience
80 hours of work total
Student
Canada
Intermediate level

Project scope

Categories
Data analysis Machine learning Artificial intelligence Data science
Skills
data science trend analysis large language modeling machine learning chatbot generative artificial intelligence data visualization chatgpt
Details

Join our team to help advance the evaluation of a cutting-edge data analysis chatbot. This is a hands-on opportunity to work in a fast-paced environment, contributing to a project that spans multiple industries and enhances our chatbot's ability to provide accurate, relevant insights across diverse data domains.


In this role, you’ll develop a collection of Question & Answer (Q&A) sets designed to test and benchmark the performance of our AI-driven chatbot. You’ll work directly with structured datasets across specific industries to create targeted Q&A sets, helping us improve and refine our chatbot’s data analysis capabilities.


Who Should Apply:

  • Experience using ChatGPT, Llama, and/or other LLMs.
  • Students in fields such as Data Science, Computer Science, Business Analytics, or similar.
  • Strong analytical and problem-solving skills, with an interest in data science and machine learning.
  • Experience with large datasets and familiarity with data visualization and trend analysis.
  • Working experience in industries relating to healthcare, environment, marketing, and others.


Deliverables
  • Craft Comprehensive Q&A Test Sets: Create detailed question sets and reference answers, working with various Large Language Models (LLMs) and verifying responses to ensure accuracy.
  • Benchmark and Validate: Ensure consistency in answers through manual validation, helping us establish a ground truth for effective evaluation.



Mentorship
Domain expertise and knowledge

Providing specialized, in-depth knowledge and general industry insights for a comprehensive understanding.

Skills, knowledge and expertise

Sharing knowledge in specific technical skills, techniques, methodologies required for the project.

Hands-on support

Direct involvement in project tasks, offering guidance, and demonstrating techniques.

Tools and/or resources

Providing access to necessary tools, software, and resources required for project completion.

Regular meetings

Scheduled check-ins to discuss progress, address challenges, and provide feedback.

About the company

Company
Toronto, Ontario, Canada
11 - 50 employees
Education, Entertainment, Environment, It & computing, Science

Ample Insight is a leading Canadian data consulting firm helping organizations across diverse industries build and scale AI and data solutions. We have a proven track record of solving some of the most complex technical and business problems for leading startups and Fortune 500 companies.