Common issues of new raters (Yukon Raterlabs/Leapforce and Lionbridge Google Rating Projects)

In this post I will discuss some common issues that new raters face. If you have other issues, feel free to share by leaving a comment below.

What kind of tasks to expect while providing search engine evaluation service for Leapforce/Lionbridge? Rater Tasks Review

If you get a position as a Search Engine Rater with Leapforce or lionbridge (Internet Assessor), you may encounter many different tasks. Some of these tasks are your normal rating tasks, and some are side by side, and you may even run into experimental tasks. This article was written to explain the different tasks that you’ll face as a rater. Before I go on, I should note that not all these tasks will be available at one time. All tasks are given out and are on a first-come first-served basis.

Some of the tasks are easier than others, like experimental tasks. Each task has a different set of instructions, so it’s best to make sure you know exactly what you are doing, and read very carefully each of the task instructions.

Side By Side

Side by Side rating tasks are very interesting and have everything to do with Google algorithm changes. The human role or ‘rating’ in Side By Side tasks is to determine which side, left or right, is better than the other. In this task you are shown blocks on the left and right side. Each block has a slider from Fails to Meet to Fully Meets, and you should choose which one is best for each block depending on the user intent and query. The same guidelines apply for the blocks as they would for a page rating. Meaning that if block meets user intent, it should get the appropriate rating. Once you have chosen the ratings for each block, you will then be asked to choose on a scale which side is better than the other. A comment may be required to give your own opinion on why you’ve chosen which side, and why it is the best over the other side. It’s very important that you base all your ratings on user intent and taking their location in mind. You will also have a quality review on Side by Side tasks  almost each month.

Page Quality

This is the main rating task that a rater is hired to rate. You will be given a website URL along with the first 5 questions which check things like Foreign Language, which if you’re doing English/USA ratings, would need to be in English to be understood from your location. Pages that don’t load need to be checked, as well as flagging for porn.

The second question is asking you to describe the purpose of the page. The purpose of the page and the user intent are entirely different aspects. Make sure you’re checking the website out to see if it matches user intent, and by looking at the website, you should be able to understand what its main purpose is. If the page is malicious, harmful, or just lacks its purpose for what it is searched for, then you should mark that here. Pages that are simply existing to show ads, without any help to the user, should be flagged. And lastly, it is important to make sure that you understand if the page is a YMYL (Your money or your life) webpage. Understanding this is important because it will change the overall rating of the website.

Once you have completed this, you will then rate the different aspects of the page including its Main Content Quality, and the amount of content. Expertise, Authority, and Trustworthiness, who is responsible for the website, the supplementary content, page design, layout, and website care and maintenance are all graded on a scale from lowest to highest. The last thing to do is to give the website its overall rating. Usually these types of tasks are allowed 9 minutes to complete. A quality review is done each month to see how you rate. So it’s important to make sure that you are taking the time to evaluate each website properly.

Needs Met

This type of tasks you have already saw while doing part 3 of the qualification exam. Each task contains 8-12 result blocks. In each task you will be given a certain query, and a list of search results blocks with links where you have to give a usefulness rating to search results using a scale from “Fails to Meet” to “Fully Meets”


Sometimes during rating of Side by Side, you will see Experimental tasks pop up. In most cases Experemental tasks are standard Page quality or Need Met tasks, but sometimes you will get new types of tasks. Experimental tasks have a wide range of instructions. It’s important to read the instructions for each task as they will be different each time. Experimental tasks can be very simple, or take a few minutes to complete. They are not always available, and each task can be completely different than the other.

Quality of your service

New raters do live tasks immediately after getting access to the task portal. It is highly advised that you hold off on these if you do not want to be restricted for quality in the next month. Typically tasks which are available for new raters are quality tasks on which raters are accessed for their quality. Instead, new agents should focus on doing simulation tasks and self-education. Time spent doing this will really pay off, and you wont be restricted for quality in the future. Also, by doing a certain number of simulation tasks correctly you will upgrade your status in the system, and will be allowed to work more hours.

During your participation in the project your service will be evaluated for its quality by means of quality review tasks. In the beginning of every month, a certain number of tasks are uploaded into the system. Normally there are 10-12 tasks of Needs Met or Side by Side, and once in 3 months PQ tasks to be released as quality review tasks. These tasks are quality review tasks up on which your quality is measured every month, in the end of the month you will get your quality score feedback.

If your ratings often stand out of general range of ratings provided by other raters in the project, most likely your account will be flagged by the system automatically and your work will be checked manuallyfor quality .

Read more about this topic here:

How to Stay with Leapforce, Lionbridge and Appen For a Long Time and not get fired?

Time Management and the Leapforce Extension

Time management of the work you complete can become a problem if you aren’t managing how much time you are spending on tasks. You should not sacrifice quality for time. In the beginning you may spend more time than the average estimated time that Leapforce allocates to the task, but make sure that in the invoice you put average estimated time that they give you. If you spend less time than average estimated time per task or set of tasks, be informed that your invoice will be returned for correction, Leapforce will ask you to put actual time spent, that is why there is no sense to complete tasks faster than average estimated time. Take your time, do everything correctly and watch your time.

Do not forget to log on to the Leapforce extension each time you work in the system.

According to the Leapforce policy: any time spent logging on/off, reading, studying, and break times are not billable. Any tasks that are not submitted or are released are also not billable. Please make sure you are not including any of this time on your invoice.

Also please keep in mind that consecutive invoices submitted with productivity below 100% may result in removal from the project or you may lose your contract with Leapforce.

Invoice Returned for Missing or Incorrect Data

Your invoice has been rejected for missing or incorrect data. Please make the necessary corrections or provide specific details on the reason for each discrepancy.

Time Discrepancy: For the following line item(s) there is a discrepancy between the hours you have submitted and the activity and work recorded on our servers.

Reported: x hours, xx mins
Unaccounted Variance: -xx mins

What raters should do in this situation?

Raters have to deduct xx nnaccounted varianceminutes from the invoice, in the future make sure only AET is entered  into invoice and the Leapforce exstension has to be always to be on while active rating.

Working some tasks much faster than the AET followed by holding on to a task for much longer than the AET and billing for the full amount on your invoice

This email is a reminder that raters are allowed to bill for the AET of a task OR the actual time spent working, whichever is LESS of the two amounts. It has been discovered that you are working some tasks much faster than the AET followed by holding on to a task for much longer than the AET and billing for the full amount on your invoice, which is a violation of official policy and will not be tolerated.

For example: You work 5 tasks, each with an AET of 5 minutes. The first 4 tasks you complete in 1 minute each, and the 5th task you hold on to for 30 minutes.

This is an official warning. If this pattern continues, your invoice will be rejected and you will be removed from the rating program. Be sure that all future invoices are in compliance with policy.


Be informed that it is prohibited to have multiple accounts, without special arrangement. Leapforce and Lionbridge can find duplicate accounts very fast, and as a result, you will lose both accounts.

Leapforce IP change/Network sharing

In some cases people may share their Internet network, while working as search engine evaluator make sure that you have your personal unique IP address and no one uses the same IP to go online for rating project.Once you are in the Project, you may change your IP address many times, it will not affect your project status. Make sure that you do your project tasks in a private place as it stated in the contract.

Non-completion of a minimum required number of tasks

This issue is very common among new Leapforce agents and may lead to termination of your contract, and happens when new agents show no activity in the first month of their service. In particular this can happen if new agents enter the project in the second half of month, and for whatever reasons, they did not complete a minimum required number of tasks. In order to avoid this issue, make sure to show the minimum required activity in the system as per your contract.

Tasks availability and time restriction

In general Leapforce and Lionbridge guarantee at least 2 hours of work per day but task availability may depend on your country. There might be some days when tasks are available for more than 4 hours.

You should understand that the system of task distribution is based on a “first come first served” basis and you need to be ready to catch your portion of tasks when they are uploaded into the system. To identify time slots when task availability is the highest please look at task completion graph, usually tasks are uploaded every day into the system at the same time slots.

Lionbridge does not provide raters with a task availability instrument, so raters have to identify time slots when tasks are uploaded into the system by themselves.

Leapforce also has a system that restricts access to tasks for new raters, and those who are undergoing their quality review. To lift this restriction, you have to complete a certain number of simulation tasks or wait till the next quality review, if your quality score is good, the restriction will be lifted automatically.

Locked for quality audit or restricted for quality?

If you receive an email notifying you that your account has been placed under quality review,this is a good time to identify your weak points and improve your rating skills. Below you will find some tips on how to deal with a “Locked for quality audit” situation.

  1. To speed up the process and to get feedback on your mistakes as quickly as possible, I recommend writing to LP/LB and ask why exactly your account has been placed under review.Sometimesthe automatic system locks raters accounts without specific reasons. Usually such cases are being resolved within 1-2 days.
  1. While waiting for detailed feedback for LP team, start doing simulation tasks in order to improve your rating skills. In most cases the issue is related to Side by Side tasks. Make sure that you understand how to deal with this type of task and that you provide well written comments where required. In many cases, raters are placed under quality review due to poorly written comments.
  1. Once you receive the feedback email with your mistakes/quality issues, read it carefully and reply back with an explanation on how you will improve your weak points.
  1. Once your status is moved to “Restricted for quality”, keep in mind that your service is under inspection, thus making sure that you do your best to provide the best possible service.

Why is my Leapforce status restricted with a meets expectations rating?

Sometimes some raters do not get their restricted status changed to unrestricted with getting a meets expectations or above quality score. This happens because they were restricted for Side by Side or PQ tasks quality issues and feedback was released for needs met tasks. In order to lift up all restrictions a rater will have to wait for the next quality review cycle for type of tasks a rater was initially restricted for.

And finally, at the end of month, if you get meets expectations or an above quality score your account will be unrestricted again.

Here are two videos and link to the article which will help you to understand how you will contribute to improving Google search engine.

Leapforce US raters will become part-time employees: no more full-time work

In  April, US raters received an e-mail from  Leapforce, which states the following “Effective June 1, 2017, [Raters in the US] can work up to 26 hours in a calendar week (Sunday to Saturday).” All raters will be moved to a new platform called raterlab and will not be able to work on another projects at Leapforce. Read more about this change here:

How Does Google Search Work?

Survey:Leapforce/Lionbridge working hours spent per month

Have you ever wondered what hours raters work ?

Knowing these statistics can be useful for raters in terms of estimating  possible productivity .

The survey is completely anonymous and once completed will show the overall statistics.

Thanks for participating!