Reasons for the Ads Assessor/Arrow project termination

It can happen to anyone at any time. You could end up getting a short email from the company you work for at any time saying that your contract will not be renewed or that your service is no longer required on the project. No tasks are available, all task options are not available anymore except the QR code. This is a sign that the system automatically cuts your account from receiving tasks. Unfortunately, this is an indication that your account has been deactivated due to unacceptable quality, such raters will be removed from the program shortly. We hope this doesn’t happen to you, but if it does it could be from some of the following issues: quality, unacceptable rating behavior patterns, low productivity, or administrative reasons. Let’s go through all these issues. 

What is the quality control system in Ads Assessor(Telus), Arrow (Appen), Ads Quality project (Welocalize)?

Quality of service is really the big one here as the client needs each rater to strictly follow the guidelines and the processes of doing each rating. There are levels of quality and you should be sure to try and get the highest quality rating possible. Also, it’s important to remember that just because you are at an acceptable level doesn’t guarantee job security. If there are a lot of raters for a project and a group of raters is providing much higher quality, then the ones at the bottom are likely to go first in the event of downsizing or, the client just needs the best quality possible. It is advisable to keep striving for perfection when it comes to rating. The quality control of the project is an automated algorithm on EWOQ that constantly monitors rater activity and the quality of the rating service provided. The quality algorithm monitors the following KPIs:

Quality and the blind tests 

Blind sets as they are called, are rating tasks that have the correct rating answers already correctly completed as they should be as per the guide. You will have no idea when these blinds sets come or are able to identify them, but generally, the blind quality tasks are uploaded to the system throughout the month, and you are graded on these to see if you are following the guidelines correctly, and applying the correct ratings. This is why it’s important to follow the guidelines at all times. In fact, it might be even better to not think about the blind sets at all since you will have no idea when you are doing them. We just thought it would be important for you to know that they are there. In total during a month’s working cycle there will be about 150-200 blind test tasks of different types of tasks, raters are expected to obtain at least 75% of correct answers in the pool of blind test tasks, raters with lower quality scores will be removed from the program.

1. The passing score for the monthly blind test is 75%.

2. If the blind test score is above 85%, the next month’s blind test will most likely be skipped, the next blind test will come only after a 1 month period. 

3. If the blind test score is below 85%, the blind test will be next month again.

4. If the score is slightly below 75%, most likely you will be asked to do requalification.

Rating behavior flags 

Apart from quality, the system also monitors rater behavior patterns, some rater’s behavior patterns are considered as bad practices which may compromise the quality of the service. If the rater receives several flags of this type, the account might be terminated or assigned with requalification. 

Repetitive rating behavior flag

An example of bad rating behavior patterns can be repetitive rating, this flag is assigned when the rater is flagged for using continuously some of the same rating buckets during tasks, rather than differentiating and assigning your answers appropriately. Although the client only disclosed the rating-related problem associated with your account without providing any examples or more detailed feedback, we know that there is normally a 3-day window when such quality checks are performed and any issue flagged during that time frame (i.e. a rater using the same ratings either continuously or almost never) will lead to an automatic account suspension. Continuously selecting the same type of answer during a task, or never selecting it, unfortunately, leads to poor quality output and it’s important to take action to correct it, making sure to utilize the whole range of ratings and not always the same ones.

Not reading tasks instructions flag

Another common rater behavior flag is related to spending not sufficient time reading task instructions. Make sure you spend sufficient time reading and understanding task instructions. 

Administrative flags 

This flag is assigned when the rater violates the general policy of the project such as using VPNs, multiple accounts, etc. IP address conflict/Internet Network sharing

In some cases, people may share their Internet network while working as a search engine evaluator to make sure that you have your personal unique IP address and no one uses the same IP to go online for rating projects. Once you are in the Project, you may change your IP address many times, it will not affect your project status. Make sure that you do your project tasks in a private place as stated in the contract. Multi-accounts Be informed that it is prohibited to have multiple accounts, without special arrangement. The customer can find duplicate accounts very fast (even if you are indifferent to Google projects which are administered by different vendors), and as a result, you will lose both accounts.

Productivity flags 

In August 2020, Lionbridge (now Telus) introduced the TCR metric, shortly after introducing this new productivity requirement lots of raters were fired due to low productivity.  What is TCR and how is it calculated?​ TCR stands for “task completion rate” and represents the average number of tasks you completed per hour during a given period. The acceptable TCR rate is 45-50 tasks per hour or 1-1,3 minutes per task on average.

Appen has its own internal productivity metrics called RPH. It is assumed these metrics are based on the average raters’ performance in the project which is calculated as the ratio between the number of tasks and the time spent within an invoice period completed by all raters. In my view this metric is not useful, because in many cases raters get different types of tasks that require a different amount of time to complete, therefore the RPH matrix is not accurate at all. This can lead to a situation where if your productivity doesn’t fall within their RPH metrics, your invoice will be returned for correction and they will ask you to align your RPH to average even though you spent a reasonable amount of time completing more time-consuming tasks. In the case of raters demonstrating multiple low RPH, such raters will be removed from the project.

How to work on a project for a long time and not get fired?

In order to avoid account termination, I recommend constantly focusing on the quality of your service. If you want to learn how to identify blind tests and complete blind test tasks correctly, contact me in a private message to receive more information on this topic.