New Quality Assessment With annotation (QAA) Job

The project manager must fill out the Quality Assessment job type form.

New QAA job form fields

FIELD NAME	DESCRIPTION
Name	A unique and meaningful name for internal tracking.	Required
Min. evaluators	A job must have a minimum of one evaluator. More evaluators can be added later in the project, if necessary.	Required
End Date	Deadline for the evaluators. The end date can be modified later on.	Required
Scoring System	The Scoring System dropdown will only show scoring systems that are compatible with the selected job type. The scoring system can be changed until one or more evaluators has accepted the job. See more information about HEVAL Input File Naming Convention	Required
Annotation Set	The Annotation Set dropdown will only show annotation sets that are compatible with the selected job type or scoring system. The annotation set can be changed until one or more evaluators has accepted the job. See more information about HEVAL Input File Naming Convention	Optional
Hide Source Panel	For QAA job types only when this box is ticked, the source is hidden to allow the evaluator to focus on the MT output only. The source is then shown to the evaluator, allowing them to adjust their evaluation, if needed.	Optional
Randomize Record Order	When this box is ticked, the segments are shown to the evaluators out of order to avoid any bias in the evaluation.	Optional
Pre-filled Scores	When this box is ticked, the scores must be included in the input CSV file. It will be pre-filled on the Evaluation screen.	Optional
Allow Showing Additional Info	When this box is ticked, Additional info can be displayed on Evaluation screen.	Optional
Description	This optional field allows the project manager to add some additional meta data about the project.	Optional

QAA input data format

The input data format requirements for Quality Assessment jobs are as follows:

QAA job input data

Column Name	Description	Mandatory	Data Validation
Source	The source text	Yes	Non empty text
MT	The MT output	Yes	Non empty text
Score	The Score for MT	Yes if the job has “Pre-filled Scores Required” option enabled.	No
MT_model_name	Name of MT model	No	No
metadata	Internal metadata for tracking and data analysis	No	No
Additional_information	Internal metadata for tracking and data analysis	No	No
Original_Id	Internal metadata for tracking Original data Id.	No	Maximal length 64 characters
Domain	Internal metadata for tracking domain category.	No	Maximal length 64 characters