The Evolution Of Human Data Collection Companies In The Last 5 Years

A lot of interesting things have happened over the last five years when you take a look at the three big companies that offer work at home opportunities with jobs in social media evaluation. Those big three companies are of course Appen Global, Leapforce, and Lionbridge.

To get a good idea of the geographical presence of these three companies, you need to look no further than Google trends as far as interest over time along with the search different search terms.

The confusing part of how and when Raterlabs came into play leaving many people wondering if they were the same company as Leapforce. The website looked the same except the logo was changed, and this time when someone went to apply for the search engine rater job, you were redirected to the Raterlabs website. This also had people wondering if the two companies were the same or not.At one point, one of the clients for RaterLabs decided to only work with companies that hire raters as employees and not independent contractors. It’s not known who this is exactly, but considering what happened with the legal situation with Leapforce and Google, it was quite obviously the search engine giant Google. In order for LeapForce to keep their client they created the company RaterLabs and hired people as actual employees, but this is only for the USA.

This brings up two things. You might remember that Leapforce was recently acquired by Appen Global and that there was a situation in where there was confusion on whether independent contractors were considered full-time employees of Google if they were simply independent contractors. One of the biggest mysteries for raters at Leapforce is who they actually work for. After all, paychecks come from Leapforce, but their work all seems to take place via Google’s Raterhub service. Despite all this stuff going on, the press release that Appen was going to acquire Leapforce came out on November 29th, 2017.

From the Google trends charts, you can see a steady rise in interest for Appen Global while the other two companies Leapforce and Lionbridge took a sharp decline. Leapforce as a company still exists, but all hiring is done by Appen Global now since the acquisition. The main reason for this is because it’s easier to simply acquire a company and use their already hired staff. In this way Appen is able to completely control the company and while at the same time, grow much bigger. And while it sounds like company growth would be a good thing for everyone involved, it’s not. As Appen becomes bigger, it can hire more people and pay them less money, which is what is currently happening. The pay rate at Appen has decreased for raters, and because they have so many people internally from their ‘crowdsourcing’ pool, it’s harder to apply and get a job for this position. By the way, Lionbridge also has changed the way they do business in USA (Google rating project) and also made it is raters as part time employees. With this in mind, we wonder why, Appen continues hiring raters as independent contractors for the same project (Inca also known as Yukon) for less pay at $9.00 per hour as opposed to $12.50 per hour at Raterlabs simply because they haven’t had any legal cases so far and they tend to hire people in states with minimum payments rates at the $9.00 per hour rating. This comes even as most states in the USA have a minimum pay rate of $10 per hour and some states even signed a law to increase the min. wage to $15 an hour. It’s almost a guarantee that if you live in these states, you will not likely be hired when they can pay cheaper wages in another state.

Update:Project Inca at Appen, Part-Time team members

On July 1, 2018, independent contractors will move from Appen to RaterLabs. RaterLabs is a separate legal entity owned and operated by Appen. The contractors will become part-time employees amd most likely their hourly rate will be increased. This decision most likely is forced by the customer (Google) and US legislation.

However, it is worthwhile to mention that Leapforce had its own problems in the past when it came to pay as they claimed you would be making a certain amount of money per hour, but that wasn’t accurate. You were actually paid for the time it should have taken you to do the work rather than how long it actually took you to do the work. In other words, they calculate how long they think each task will take you, and pay you for each task no matter how long it took. Work wasn’t consistent enough to make the hourly pay rate either. And the biggest thing, of course, is that training that takes a long time and you’re not paid for it.
This is something that Appen hasn’t changed either. They’ve basically adopted the system which Leapforce uses, but instead, they pay far less than Leaforce did or even as Lionbridge pay. Speaking of Lionbridge, we’ve been talking about Appen and Leapforce. But what about Lionbridge?

If you look at the 5-year chart on Google trends and compare the three companies together, you can see that Lionbridge has just barely managed to stay above the interest level of Leapforce which of course is old news now because of Appen. At the same time, you can see that interest skyrocketed for Appen in January right after it acquired Leapforce. It was the big story at the time, and people that didn’t know about Appen finally found out about them and the work at home jobs they offered.

However, it might be worth mentioning that Lionbridge is probably still the best place to work for when it comes to being a search engine rater. They are known to have a higher pay rate that Appen offers the same job. There are however a couple things that seem to be problems still and that is the 20-24 hourly cap they have for each week, and the fact that there appear to be full-time jobs but part-time employees have no idea how to get into these jobs. With that said, these have been issues for quite some time. Otherwise, Lionbridge is more stable and more rater oriented than Appen which offers a ton of different jobs and doesn’t focus on search engine raters. That is just what you would logically experience from such a big company whose main goal is to increase their client list while utilizing their crowdsourcing at low pay rates.

If you were looking at these companies from a market share perspective, anyone knows that Appen is taking a huge share of the market when it comes to data gathering, translation services, and human-annotated datasets for machine learning and artificial intelligence. But Appen seems to be getting too big for its own good and the bottom line is that the people who work for this company are the ones who miss out on the benefits. Appen seems to also be getting a lot of negative reviews lately and even some people are calling them a ‘scam’ company. Leapforce, since it was acquired, will likely end up being broken down and dismantled and ‘enveloped’ if you will, as more time passes. By the way process has already started as we got confirmation from lots of markets regarding decrease in payment rates in Yukon project at Leapforce. As you may remember we predicted this when Appen acquired Leapforce.

Appen Acquires Leapforce in Multimillion Dollar Deal: What Are Potential Risks for Current Leapforce/Raterlabs Contractors?

The future for Lionbridge lies in its ability to separate itself by doing things that Appen doesn’t do. If this company is going to survive, they should make a few changes, such as hiring full time employees with benefits, pay for the long training that one needs to go through as a search engine evaluator, and make the pay competitive enough so that people will want to work there instead of the big competition, who doesn’t really have much to offer in the terms of well-payed employment.

We just might get lucky and see Lionbridge make these changes in the future.
Have you had experience with Appen, Leapforce, or Lionbridge? Let me know in the comments and as always, if you have a question, let me know and I’ll answer as soon as possible.


Leapforce (Raterlabs)/Lionbridge Third Party Extension Ban:What Are Alternatives?

Leapforce/Lionbridge Third Party Extension Ban

 banned-1726366_960_720If you are a search engine rater for either Leapforce or Lionbridge, you probably got an email from both of these companies talking about how the use of extensions to track time and your tasks would end up being banned, and they explained their decision in a couple of emails with this one being their last warning to remove them.  

“As a follow-up to our communication sent out on March 12th, we wanted to remind you that as of Monday, April 9th, all 3rd party browser extensions must be uninstalled when working on Yukon Search tasks. While 3rd party extensions were developed to aid in rater work, some actually pose security risks and compromise the integrity and the quality of work.” 

While it’s not known exactly how and when these security issues were found, or what types of exact security problems were identified, it does put raters back to a problem of keeping track of tasks and time while doing their jobs. It was a lot easier to use some of the extensions that you could add to Chrome such as Rateraid, Greater Acceptor, and LBTimer which were all great extensions for keeping track of your time and tasks.  

So now what do you do when it comes to keeping track of your tasks as a Search Engine Rater? Well you could resort to using a pen and paper like they used to do back in the good old days, or you could find some alternatives to these tools that can’t be tracked. And before we get into that, we really have to mention that although we are here to help everyone in any way possible to make their job easier, we are not responsible if you start using some tool that ends up on the banned list and you get caught for it. And for anyone wondering if they can detect these tools being used, they can, and if you get caught using them after the fact, you can lose your job.  

Of course we don’t want that to happen to you so we will take a look at some things out there that can help pick up where these tools that were banned left off.  

What are alternative and approved way of tracking your time in effective way? Apparently there are some tools out there that can keep the integrity of the browser intact without opening any security holes. And while it’s not fully understood how using these tools compromises the quality of work, it is very possible that the makers of these tools are being used to scrape data as you are using them as time keepers. Anything is possible and the data you are collecting is an invaluable asset. We are not sure if that is the case with these tools, but you might be aware that RaterAide was released in July 2012 and has since become the de facto time tracking tool for raters around the world. It is produced by an independent developer and fellow rater which is probably the reason that both companies decided to ban use of the tool.  

But this was not the only tool that was banned. Greater Acceptor is also banned and cannot be used as well as the well-designed LBTimer. All of these tools are designed by third-parties and was likely the main reason behind the decision to ban these from being used in connection with tracking your time and your tasks. If someone really wanted to steal something, it wouldn’t be the time you have spent, or even the tasks that you are working on. However, these tools like any other type of software could be used to track everything else and even spy on your activities and compiling a log of results with your data input. We’re not saying that this has been done, but if you are the companies that offer search evaluation jobs, then you probably should tighten things up a little bit when it comes to data collection.  And aside from this, no one knows if these tools were used to exploit or farm data. It’s not likely that either Lionbridge or Leapforce would speak of the specifics about such a topic. And as far as Leapforce is concerned, many of you know that this company was acquired by Appen Global who probably also decided it’s time to tighten things up a little bit when it comes to security.  

So now you are a rater and you’re stuck with no tools to use to track tasks and time. Well there are some extensions that you might be able to use and be OK with. Some of these things are tools that were not designed with search engine rating in mind, and work offline. Something that keeps track of your time while being offline is probably key in that the program works as a standalone application and doesn’t need the internet to function.  

One of those tools is called Task Timer which can be found here 

Task Timer

This tool allows you to create a list of tasks along with time goals represented in hours. It does keep track of each task that you perform and it has a lot of options so that you can customize it. You can input any amount of tasks and include goals, which could be the time you are allowed on each task which is handy for Yukon or other search engine tasks, and you’re able to start, pause, reset, and edit tasks on the fly. This makes it a perfect replacement for the stuff that these companies have banned. Task Timer is available on Chrome Web Store.

Manual recoding of time spent and task done in excel file  

There are some people who love Excel, and some people who loathe it. However, if you need something right now that is completely standalone whether you have internet or not, and you need to keep track of things, Microsoft Excel can fit the bill. It is also worth mentioning that you can customize Excel beyond your typical applications and even create a template for adding time and tasks.  

Auto Page Refresh

As you all probably know, refreshing the page when you are logged into the backend of Lionbridge or Leapforce to check if new tasks are available is quite a pain. There is a tool that is available that can help with this called Auto Refresh Plus. It is a simple add-on for Chrome that acts as a page refresher which sends notifications to your email about tasks availability/ and new content on the page, such as when tasks show up. Auto Refresh Plus is available on Chrome Web Store.

 These are just a few tools that can help you out while Lionbridge and Leapforce have banned the use of these tools. It could be possible that both companies are coming up with their own solutions for helping track time and the amount of tasks that you do, but for now, this is what we have as Search Engine Raters. Of course the moment that we find something new that will help your job go by a bit easier, we will let you know. As always, if you need help with something pertaining to Search Engine Rating, leave a comment or drop us an email and we reply as soon as possible.  

Multiple Accounts – Is There a Safe Solution? 

Multiple accounts – is there a safe solution? 

There might be a reason that you would want to have two accounts, and some contractors do use more than two accounts. There might even be a possibility that you may want to work on this job while someone else in your household is also working for the same company or on one project but at different vendors (Leapforce and Lionbridge).  

However, according to the Leapforce policy you can only have one account per IP address, while Lionbridge allows to have one account per household. Those pretty much mean the same thing. There can’t be two accounts coming from the same internet service and this requirement come from Google system requirements.  

When it comes to Google rating projects or Facebook related projects, Leapforce, Appen and Lionbridge can find duplicate accounts very fast, and as a result, you will end up losing your accounts and they will be terminated. You certainly don’t want that to happen.  

Luckily, there are a few things you can do to get around this problem, and arrange them in way that Leapforce, Lionbridge, and Appen won’t be able to detect.  

Note: One thing before we go on that should be totally understood. There isn’t anything illegal about what you are doing, but if there are two of you that want to work for the same company, or if you want to work on multiple accounts from different vendors, you need to do things the right way. And let’s face it, most of these jobs are part time and give very little hours with most projects allowing a maximum of 20 hours per week. This restriction is due to the fact that vendors do not want pay for insurance and extra expenses which are associated with full time employment.

So how can you work on multiple accounts without violating the company’s rules?  

  1.  You will need two different laptops/PCs and smartphones/one set of devices per account. The reason for this is because of the devices MAC Address which can be detected per device;
  2. Use different internet or mobile networks which guarantees different IP addresses per network. Make sure that each PC is connected only to its designated network. Your internet service provider can provide you with one or more IP addresses, or use SIM card (cellular data connection) for this purpose. SIM card internet has dynamic IP addresses for each SIM card;
  3. Do not use your Wi-Fi network to connect your devices because it will increase the chance of your device accidently connecting to another network by accident and as a result, may lead to an IP address conflict with your client; 
  4. In the project where smartphone devices are required, you will need a separate smartphone for each account; Smartphones must be connected to Internet via sim card only. If you use your WiFi network for your mobile devices, they will all connect to the same IP address, and you will be flagged as duplicate account on the same IP address. For this instance, it might be a good idea to switch off WiFi so that you don’t run the risk of this problem;  
  5. Don’t use the same Gmail addresses for the accounts. This is already a given, but we thought that we should mention it. As an example, if you use one Gmail account for Appen, create another account exclusively for Lionbridge and make sure you reply using only those emails;  
  6. Once you have everything setup, and you’ve checked your IP addresses, only then should you try to apply for other accounts at other vendors. 
  7. You will need to apply on behalf of two different persons as normal.

Thanks to this approach, vendors’ detection system will see your accounts as two separate accounts with different IP addresses which are registered on different people, it means all contract conditions are met.

You also might be interested in this article:

Projects and Vendors Compatibility: How to Work on Several Projects and Do Not Violate Vendors Conditions? How Much Can You Earn With Leapforce, Lionbridge and Appen When Combining Several Projects?

What Should You Do If You Fail Leapforce or Lionbridge Exam Twice and How to Re-Apply?

If you fail the exam the second time, Leapforce will not allow you to retake it a third time. In this case I would advise that you apply to another vendor like Lionbridge (or vice versa). 

Option 1: re-apply from your own PC 

Before proceeding with application you will need to clean your devices from the data left from unsuccessful exams.

  1. Sign out all Gmails associated with your unsuccessful attempts (make a complete sign out on a laptop, PC and your smartphone) and to be on the safe side, re-install your chrome browser on your PC;
  2. A new IP address will be required. You may use a residential VPN to change your IP address. Do not use normal VPN if you are going to apply for Lionbridge, use only a residential VPN. It is VERY easy to know if you are using a regular VPN. As an option you may change your Internet provider or if you have a dynamic IP address then no action is required at this stage;
  3. Create new gmail;
  4. Proceed with application as normal.Once you have started the application and qualification process, I recommend you keep the same IP address during the whole process. If you need to, later on you may change your IP as many times you want.

Note: I would like to point out that Lionbridge is very strict in terms of IP address. The vendor prefers to deal with applicants who have a residential IP address, but it depends on your country. 

Option 2: re-application on behalf of another person the same vendor is possible

Use someone else’s PC to apply and move the account to your PC once you complete the qualification process. 

Note: Before proceeding with registration, make sure that you use a new Gmail account which previously was not used for application process.

Note: It is very important that when you filling out the application and answering the application questions, you must indicate that you have never applied/worked the search engine evaluator position before with another company. Answering yes to this question will automatically disqualify you.  

Lionbridge Unsuccessful Application

There might be many reasons for the rejection of your application. These are some of the reasons: 

It might be possible that your CV does not meet Lionbridge standards. Lionbridge is very selective in terms of quality of CVs of applicants in India and other developing countries.Make sure that you CV looks professional and accurate and there are no gaps in your experience.  

Your IP address might be suspicious, or does not meet Lionbridge requirement. If you have a dynamic IP address (an IP which changes each time you connect to the Internet) – you will be rejected for sure. Make sure that you have static IP address. One way to check is to ask your internet service provider.  

Someone is using the same IP address for a rating project (in the case of sharing an Internet network connection), make sure that your IP address is unique. 

And finally, try to arrange new IP addresses, update your CV, create new Gmails, and apply one more time. If possible you might need to find a person you can trust, like a friend or family member, and apply on behalf of them. It could be a possibility that your name is blacklisted in the vendor database.  

Unable to enter IP address into the Lionbridge exam. What are the reasons?

Some candidates receive from Lionbridge email saying that “we were unable to enter you into our Online exam. According to our checks it appears that you have the same IP Address as another person in our system”.

It can be possible that the vendor was unable to enter your IP into exam because you have dynamic IP address. It might be very risky to have this type of IP address because it means that many people have the same shared IP address. Some internet service providers put a group of customers on the same IP address behind a NAT. This could be a problem and may lead to getting flagged for multiple accounts being used on the same platform such as raterhub.  

Violating terms of contract regarding multiple accounts including IP/Network sharing and using public networks for work

It is prohibited to have multiple accounts with Leapforce and Lionbridge. These companies can find duplicate accounts very fast, and as a result, you will lose both accounts. The good news is that you can combine work at Leapforce or Lionbridge and Appen Butler Hill.

While working as a search engine evaluator, make sure that you have your personal unique IP address and no one uses the same IP to go online and work on for these companies. If you have two or more people in your household, it is advised to get two unique IP addresses, and you can do this for an extra charge with most internet service providers.

These posts also might be interesting to read:

How to pass the Leapforce (Raterlabs), Appen (Inca) and Lionbridge exams? (Exam Guidelines 2018)

You Failed the Leapforce (Raterlabs) or Lionbridge Exam – Now What?

Leapforce and Lionbridge US raters will become part-time employees: no more full-time work


In April, US raters received an e-mail from Leapforce, which states the following “Effective June 1, 2017, [Raters in the US] can work up to 26 hours in a calendar week (Sunday to Saturday).” All raters will be moved to a new platform called raterlab and will not be able to work on another projects at Leapforce, the same situation will happen with Lionbridge raters in USA.

The Phantoms of the Internet – Google Ad Raters

If you are visiting my website, then you probably have an idea of what a Google Rater is. But if you have no idea, this is your first time visiting, then grab a tea or a cold one, and lets take a journey behind the scenes, down the rabbit hole, choose the red pill, but understand that there is no turning back once you start. Many people think that Google is all about a robot running its secret recipe algorithms, or that the spiders (bots) that crawl webpages are the only things that are responsible for a web pages’ rankings. It isn’t true. And you might understand by now there is a human factor involved. Enter the Google Rater.

First of all, we need to understand that there weren’t always humans involved in the ranking of web pages. The changes came about when Googles advertising algorithms were no longer able to discern certain types of content, or mainly, keeping bad content out of places that it shouldn’t be. The purpose of the Google algorithm is responsible for placing ads in appropriate places or context, for example, someone searching for restaurants might want to see some ads for pizza, or sushi. Or someone searching for music would see some ads based on some famous artists. But something went wrong and in the UK, something disturbing happened that caused Google some big problems. Uk Government ads for the Home Office, Royal Navy, and Transport for London ended up being paired with Youtube videos that featured a certain Steven Anderson who is a Holocaust-denying pastor, and who actually praised and enthusiastically commended the guy that killed 49 people in the Florida nightclub Pulse. And that is not all, according to the UK Government, its taxpayer-sponsored ads ended up running alongside videos from so called ‘rape apologists’ and some speeches from David Duke the white supremacist.

So what happened? Well you guessed it. Google’s ad business suddenly lost a lot of business as big European ad agencies had to cut ties with Google, and to top it off, Verizon and AT&T cut all their video ad buys. When you lose a big chunk of money suddenly, you have got to do something to fix the situation, and fast. Google stepped up and assured its clients that it would make sure that no ads ran alongside with bad, upsetting, or offensive content. Google came out and said it was going to unleash an army of 10,000 humans to rate ads, that would work around the clock to make sure that their algorithms don’t return bad results, results that are unhelpful, offensive, or even horrific like the previously mentioned stuff.

So who are these raters behind the scenes? They are carefully trained and tested people that can spend 40 hours a week checking and rating tasks, and use a special website system called Raterhub, which is exclusively owned by Google. The tasks vary, and new tasks or ‘experimental’ tasks are available on a continued basis. Every day thousands of tasks are completed by the army of raters that produce invaluable data about how Googles always changing algorithms are performing. Raters produce significant results to several Google and Android projects from your regular search feature, to voice recognition, and photo tasks in Google/Android.

Not many people realize just how much the rater contribution adds to the smooth functioning machine of Google search. And no one knows who these people are. Not even Google engineers who take the resulting data have any idea. But it seems now that some raters want this to change. The main reason is that recently, an email went out to all raters telling them that their hours would be cut in half, and that is was partly due to Googles staffing policies.

And although Google brags about its army of raters, the raters aren’t even Google employees. Instead these raters are contracted out by different firms. Now raters believe that Google has reaped significant benefits from their rating results without ensuring their jobs are stable and secure. So who are these companies?


There are a handful of companies that supply raters for Googles Raterhub evaluation on a contractual basis. Leapforce, located in Pleasanton, California is one of the biggest. Their offices lie just outside Silicon Valley. But it doesn’t matter where their offices are located, because no raters work from the office. All raters work remotely from their home and there is a chat room in the backend where raters can ask each other questions and meetings are held in this chat room weekly. The meetings are unpaid, but they are optional and you don’t need to attend. Sometimes the meetings can help if you have questions about certain tasks that are seemingly hard to rate for whatever reasons. Managers never go by their names and also raters all have a nickname. Not even in Raterhub do other raters know who each other is. Ghosts…doing important tasks, hidden behind the scenes, and providing invaluable data while remaining anonymous even to each other. You can go years without knowing anyone at all, the real names of managers, or the people you work with.

All the work is task based somewhat like MTurk. To get a task you log into Raterhub and you will see usually 2-3 buttons that say Acquire if available and the name of the task which most now say experimental for some reason or another. Acquire if available means what it says. If there are a lot of raters working, you might not have any tasks to do. Some days there are plenty of tasks available and some days you might work 12 minutes and be given the message ‘Sorry there are no tasks available…. please check back later’. This can be frustrating, but if you figure out the times when tasks are loaded and are right there to get work, you can be OK. You would think that this job might seem like you could work any hours you want. But basically you are at mercy of the Raterhub system and its tasks availability.

Typical tasks can take anywhere from 30 seconds like the voice or speech recognition tasks, and others can take 15 minutes where a rater is looking at a webpage and judging the contents and relevancy. The time a rater can bill is really limited because the tasks have a time limit and in Raterhub there is a timer showing you how much time you are allowed, and how much time you are taking. You can take 5 minutes to do a 30 second task. If you try to bill for extra minutes compared to your tasks, then your invoice will likely get kicked back to you. But this isn’t an only problem, sometimes Raterhub can be really slow and can take a lot of time to load. If you have a 5-minute task, but it takes 2 minutes to load, you’ve only got 3 minutes to complete the task because you can only bill 5 minutes. Sometimes the loading of the task can eat up all your time. Raterhub can be slow and it’s a common occurrence.

So what about the tasks themselves?

Tasks can vary, and even some new tasks will pop up with the term ‘Experimental’ which means they are new tasks rolled out by the engineers. The raters job is to evaluate if the search result is useful to the searcher or not. Audio recognition is becoming more prevalent on Android, and raters sometimes must check and listen to the audio to see if its been transcribed correctly. And according to some raters some tasks can feel a bit creepy. The tasks are related to personalization services, which requires raters to first give Google access to their emails, photos, chats, and even allow Google to record their own search behavior and other Google services that the rater uses. Google then turns the personal data into tasks that allow the rater to give them feedback on how well their algorithm is working. As you can imagine, this can and has created some discomfort giving Google access to their personal accounts.

A rater’s job isn’t easy. Before a rater can even think about starting work, each rater must pass a 3-part exam that is extremely difficult. You are given a 160 page guidelines that you need to more than study, you need to absorb these guidelines, you need to eat, drink, and sleep with them under your pillow. If you do not, you will fail the exams. In fact, the exams are so difficult to pass, that you are even told in the first email with the guidelines that the majority of people who take the exam fail it. And they stress over and over how important it is to know these guidelines. In fact, it’s a good idea to even copy and print them out because you will be going back to them often, and even if you pass the exam, you will want to reference them. So how hard is the exam to pass? One rater had said they referred nine people to the job, and every one of them failed the exam. It is worthwhile to mention that if you do pass the exam, they give you a second chance to pass it. There are three parts. After you pass each part starting with the first, you are sent an email saying you passed, and only then are given access to the second part. If you fail the exam, no matter which part, you must take the entire exam over again.

For those who pass the exam, the training isn’t over as one might expect since Google is ever changing in its algorithms. At the beginning of each month there are evaluation tasks that have to be taken. The right answers for these tasks are already recorded, so if you fail these tasks, you then become limited in what tasks are available until your score improves.

Sometimes a bot will spot check a raters work which can cause problems because it can automatically lock you out of the tasks area. Raters call this being ‘botted’ and the bot can be buggy and occasionally lock you out for no reason. Once this happens, its often difficult to get a hold of a management person to fix the problem which results in a loss of work because it can take days to get a reply.

Despite all these frustrations, raters generally like their jobs and the pay can reach up to $17.40 per hour for those who achieve excellent scores. Regular raters start out with and get $13.50 per hour which is still above the min wage for most parts of the USA at the current moment. The work isn’t fun by all means. But you do have the sense that you are doing something meaningful and helpful.

But now the big problem is there is no more full-time work. On April 3rd thousands of US Leapforce raters received an email from ‘The Leapforce Team’ saying that effective June 1, 2017, each rater can only work 26 hours in a calendar week which is Sunday to Saturday. With that being said, 20 percent of the US raters just took a giant pay cut. Almost half their pay now gone for full time raters. Many raters were greeted by LFAdmin who for the first time told everyone that he was indeed the CEO Daren Jackson and said this was not a decision that they were in control of. It was Googles decision. He added the change was due to risk mitigation and some other regulations but didn’t specify. In the e-mail, however, the Leapforce Team claimed the change was driven by “circumstances that are somewhat out of our control, but will ensure Leapforce is compliant with federal and state regulations including the FLSA [Fair Labor Standards Act] and the ACA [Affordable Care Act].

Its interesting to note however that key provisions of FLSA and ACA don’t apply to contract workers, which is what Leapforce raters are. If raters are independent contractors, Leapforce has no ACA obligations. This makes the changes and loss of hours a confusing subject. A manager in the main rater chat room had told that the pay cut had come from Google themselves. Can they change this? It’s a chance less than zero when Google makes a decision based on this.

What really upset a lot of raters was that this sudden reduction in their hours came a couple of weeks after a Google engineer named Paul Haahr had raved in the media about how terrifically helpful raters are to the company. “We’ve only been able to improve ranking as much as we have over the years because we have this really strong rater program that gives us real feedback on what we’re doing”. So why would they cut the raters hours nearly in half if they were doing such a good job for the company?

One of the mysteries for raters at Leapforce now is, ‘who do I actually work for, Google? Or Leapforce’. Daren Jackson actually left Google and created Leapforce in 2008. Jackson had been working on the project EWOQ which was the precursor to Raterhub. Rumor has it that Google actually had a rater tool as early as 2004.

Google purchased Raterhub in 2012, and at that time, all raters were coming from companies such as Leapforce, Lionbridge, Appen, and ZeroChaos.

Google’s policies have a direct effect on Leapforce raters’ employment status, and its been known that Leapforce CEO has started to reclassify its US Raters from independent contractors to actual employees. It appears as though a change in Google contracts is behind the switch:

“They had a contract that was up for bid and it stipulated that they wanted to work with employee-based workforces. It wasn’t Google’s attorneys [who forced us to do this], but the process was they wanted vendors to work with an employee-based workforce. To continue with the contract, to work on that project, we’re doing [the employee] conversion.”

So is this a joint employment situation? The problem in the Leapforce situation is that the company isn’t depriving employees of benefits they already had, since the raters have been contract workers. But Google does provide its employees with benefits. If a court found that these raters were joint employees of Google, the search company might find itself in a tough spot.

Ghosts in the machine

Many of these raters actually believe they are working for Google and not Leapforce, and simply contracted by these 3rd party companies. But raters at other staffing firms such as Appen, ZeroChaos, and others face similar problems. In fact, ZeroChaos has put is rater system on hold until all of this is worked out. An email went out on May 22nd asking for patience as the program has been put on hold. It didn’t say why, it simply said it was on hold. And its absolutely certain that they are facing the same issues and triyng to figure out the best course on how to proceed.

The Leapforce raters’ current situation appears to be the result of Google insisting that its staffing agencies work with employees rather than independent contractors. Unfortunately, the change has had unintended consequences. In a month, Leapforce raters will be employees, but without full-time work and benefits.

This doesn’t please any of the raters, who had a range of opinions about what would be a good outcome to the situation. Many raters would like to join Google and work directly for the company, and who wouldn’t Google boasts and brags that it likes to take care of its employees, but if you are an independent contractor, you are basically a phantom or a ghost. An unknown force behind the curtain so to speak.

It will be an interesting thing to see what happens now. It is certain that the Google Rater is a very important part of its algorithms, and its not likely that a robot is going to take away the important aspect of a human making sure that things don’t go wrong, like your child accidently seeing some bad websites or ads that aren’t relevant to a search. No robot could likely take away this type of job. In any event, we will keep you updated on the situation.

Please comment below if you liked this story and share.

How to Stay with Leapforce, Lionbridge and Appen For a Long Time and not get fired?

Leapforce and Lionbridge are typical crowd sourcing companies; this type of business supposes that the main communication is done via email, and raters are responsible for self-education and providing high quality service.

Based on my personal experience both companies treat their freelancers well. Although it is true that you can find stories online of freelancers complaining they were fired without cause or notification. But this just isn’t true. My own experience proves that if you provide good quality service/work and meet your contract requirements, your contract will end up being renewed. Contracts usually run on a 6 month basis and I have had mine renewed for several years now.

If you really want to know what happens then read ahead.  I have compiled a list the of most common reasons for these companies terminating a contract with raters/independent agents:

  1. Delivering unacceptable level of service

Multiple substandard quality scores might lead to non-prolongation or termination of your contract (in case of receiving an unacceptable quality score). In most cases a vendor will give you a few chances to improve your quality, but your access to tasks will be restricted. If you want to know what to do if you are restricted for quality read here:

  1. Violating terms of contract regarding multiple accounts including IP/Network sharing and using public networks for work

It is prohibited to have multiple accounts with Leapforce and Lionbridge. These companies can find duplicate accounts very fast, and as a result, you will lose both accounts. The good news is that you can combine work at Leapforce or Lionbridge and Appen Butler Hill.

While working as a search engine evaluator, make sure that you have your personal unique IP address and no one uses the same IP to go online and work on for these companies. If you have two or more people in your household, it is advised to get two unique IP addresses, and you can do this for an extra charge with most internet service providers.

  1. Non-completion of a minimum required number of tasks

This issue is very common among new Leapforce agents and may lead to termination of your contract. This mostly happens when new agents show no activity in the first month of their service. In particular this can also happen if new agents enter the project in the second half of month, and for whatever reasons, they did not complete a minimum required number of tasks. In order to avoid this issue, make sure to show the minimum required activity in the system as per your contract.

  1. Working on the project from abroad

According to the contract you can’t work from abroad. The reason for this is that your contract states the location in which you would work. If you are rating websites in the USA market, you must reside in that market to give accurate data and results. If you work outside of your contracted location, you are breaking the contract. If you do happen to go abroad, make sure you do not complete any tasks on raterhub otherwise your contract will be terminated.

  1. Duplicating/coping/sharing comments from/with other raters

You should not copy and share your comments with other raters, as an automatic system can easily detect duplication, and as a result, your account might be terminated for sharing materials with other raters.

Finally if your account got terminated read this article (How to re-apply section)


5 tips to increase your productivity when working as search engine evaluator

Leapforce and Lionbridge Productivity (Yukon and Internet Assossor projects)

When you work as an evaluator for Leapforce or Lionbridge you are given a lot freedom when it comes to performing tasks. You are not required to work at a certain time like most other jobs. With Leapforce or Lionbridge, you can work at any time, and any day that you wish, but as you may already noticed tasks are available at random time slots during the day, and in some cases the limit of tasks in some countries is just 2-3 hours per day…That is why it is highly important to be able to identify those time slots and to be one of those raters who complete maximum number of tasks. Here are step by step tips regarding how to increase your productivity as a search engine evaluator:

  1. Identify time slots when tasks are uploaded into the system and reserve this time for work on daily basis. You should understand that the system of task distribution is based on a “first come first served” basis and you need to be ready to catch your portion of tasks when they are uploaded into the system. To identify time slots when task availability is the highest, please look at task completion graph. Usually tasks are uploaded every day into the system at the same time slots.
  1. Do simulation tasks and focus on quality to get unrestricted status. Leapforce also has a system that restricts access to tasks for new raters, and those who are undergoing their quality review. To lift this restriction, you have to complete a certain number of simulation tasks or wait till the next quality review, if your quality score is good, the restriction will be lifted automatically.
  1. Do not forget to log on to the Leapforce extension each time you work in the system. If you start doing tasks and you’re not logged into the extension, then you will not be able to prove the time you worked regardless if tasks were done. So it’s very important to make sure this is on. Using RaterAide will ensure that you’re tracking everything correctly.
  1. Tracking time spent for completing tasks. Time management of the work you complete can become a problem if you aren’t managing how much time you are spending on tasks. You should not sacrifice quality for time. In the beginning you may spend more time than the average estimated time that Leapforce allocates to the task, but make sure that in the invoice you put average estimated time that they give you. If you spend less time than average estimated time per task or set of tasks, be informed that your invoice will be returned for correction due to not meeting productivity targets.

Leapforce will ask you to put actual time spent, which is why there is no sense to complete tasks faster than average estimated time. Take your time, do everything correctly and watch your time.

  1. Use apps

There are a couple of tools out there that are very useful when it comes to productivity concerning the Search Engine Evaluator job.


heroThe first one is called RaterAide which is a task tracker and timer which keeps track of all tasks and shows the time you are spending on each task. It keeps an accurate record of how much time you worked so you can submit accurate time on your invoice. Your data can be stored on Dropbox meaning that everything can be synced across all your mobile devices.

Features include:

  • Automatic task tracking
  • Seamless syncing between devices
  • Speed, surplus and earnings tracking
  • Invoice and timesheet displays
  • Multiple currencies and time zones
  • Separate large timer window

Link to app website:

The Greater Acceptor

unnamedThe next tool that can really help productivity is called The Great Acceptor. This is an extension for Chrome which checks for and can optionally accept tasks. The reason this tool is so good is that tasks are given and taken by raters on a first come – first served basis. Sometimes there can be no tasks, so therefore you would need to sit and refresh your browser over and over until you would see tasks that are available to you. The Great Acceptor automatically refreshes the browser and checks for tasks. You can set the refresh interval from 60 seconds and on up from there. From here you have the option to have the tool automatically accept the tasks that it finds. This is completely optional. There is also an email notifications option so that you can be notified if the extension does find tasks. It’s a quite handy tool for grabbing tasks and will be certain to increase your productivity.

Here is the link to the app:


LBTimer is a Chrome extension for Google search engine evaluators. It keeps track of your time and completed rating tasks, saving the data in your Google Drive™. Automatic, lightweight and full of features, it makes the rating work a breeze.

Here is the link to the app:



By following these tips, not only can you increase your productivity, but you will be sure to meet the minimum requirements set by Leapforce. Using the RaterAide and The Great Acceptor applications will help you keep track of everything so you know where you need to improve. When you first do tasks, it can take some time to get up to speed and complete the tasks in the time allotted. As was previously mentioned, it is quality that matters the most. Time will come naturally once you have a good understanding. As always, if you have any questions, then please leave me a comment below and I will be happy to answer them.

Common issues of new raters (Yukon Raterlabs/Leapforce and Lionbridge Google Rating Projects)

In this post I will discuss some common issues that new raters face. If you have other issues, feel free to share by leaving a comment below.

What kind of tasks to expect while providing search engine evaluation service for Leapforce/Lionbridge? Rater Tasks Review

If you get a position as a Search Engine Rater with Leapforce or lionbridge (Internet Assessor), you may encounter many different tasks. Some of these tasks are your normal rating tasks, and some are side by side, and you may even run into experimental tasks. This article was written to explain the different tasks that you’ll face as a rater. Before I go on, I should note that not all these tasks will be available at one time. All tasks are given out and are on a first-come first-served basis.

Some of the tasks are easier than others, like experimental tasks. Each task has a different set of instructions, so it’s best to make sure you know exactly what you are doing, and read very carefully each of the task instructions.

Side By Side

Side by Side rating tasks are very interesting and have everything to do with Google algorithm changes. The human role or ‘rating’ in Side By Side tasks is to determine which side, left or right, is better than the other. In this task you are shown blocks on the left and right side. Each block has a slider from Fails to Meet to Fully Meets, and you should choose which one is best for each block depending on the user intent and query. The same guidelines apply for the blocks as they would for a page rating. Meaning that if block meets user intent, it should get the appropriate rating. Once you have chosen the ratings for each block, you will then be asked to choose on a scale which side is better than the other. A comment may be required to give your own opinion on why you’ve chosen which side, and why it is the best over the other side. It’s very important that you base all your ratings on user intent and taking their location in mind. You will also have a quality review on Side by Side tasks  almost each month.

Page Quality

This is the main rating task that a rater is hired to rate. You will be given a website URL along with the first 5 questions which check things like Foreign Language, which if you’re doing English/USA ratings, would need to be in English to be understood from your location. Pages that don’t load need to be checked, as well as flagging for porn.

The second question is asking you to describe the purpose of the page. The purpose of the page and the user intent are entirely different aspects. Make sure you’re checking the website out to see if it matches user intent, and by looking at the website, you should be able to understand what its main purpose is. If the page is malicious, harmful, or just lacks its purpose for what it is searched for, then you should mark that here. Pages that are simply existing to show ads, without any help to the user, should be flagged. And lastly, it is important to make sure that you understand if the page is a YMYL (Your money or your life) webpage. Understanding this is important because it will change the overall rating of the website.

Once you have completed this, you will then rate the different aspects of the page including its Main Content Quality, and the amount of content. Expertise, Authority, and Trustworthiness, who is responsible for the website, the supplementary content, page design, layout, and website care and maintenance are all graded on a scale from lowest to highest. The last thing to do is to give the website its overall rating. Usually these types of tasks are allowed 9 minutes to complete. A quality review is done each month to see how you rate. So it’s important to make sure that you are taking the time to evaluate each website properly.

Needs Met

This type of tasks you have already saw while doing part 3 of the qualification exam. Each task contains 8-12 result blocks. In each task you will be given a certain query, and a list of search results blocks with links where you have to give a usefulness rating to search results using a scale from “Fails to Meet” to “Fully Meets”


Sometimes during rating of Side by Side, you will see Experimental tasks pop up. In most cases Experemental tasks are standard Page quality or Need Met tasks, but sometimes you will get new types of tasks. Experimental tasks have a wide range of instructions. It’s important to read the instructions for each task as they will be different each time. Experimental tasks can be very simple, or take a few minutes to complete. They are not always available, and each task can be completely different than the other.

Quality of your service

New raters do live tasks immediately after getting access to the task portal. It is highly advised that you hold off on these if you do not want to be restricted for quality in the next month. Typically tasks which are available for new raters are quality tasks on which raters are accessed for their quality. Instead, new agents should focus on doing simulation tasks and self-education. Time spent doing this will really pay off, and you wont be restricted for quality in the future. Also, by doing a certain number of simulation tasks correctly you will upgrade your status in the system, and will be allowed to work more hours.

During your participation in the project your service will be evaluated for its quality by means of quality review tasks. In the beginning of every month, a certain number of tasks are uploaded into the system. Normally there are 10-12 tasks of Needs Met or Side by Side, and once in 3 months PQ tasks to be released as quality review tasks. These tasks are quality review tasks up on which your quality is measured every month, in the end of the month you will get your quality score feedback.

If your ratings often stand out of general range of ratings provided by other raters in the project, most likely your account will be flagged by the system automatically and your work will be checked manuallyfor quality .

Read more about this topic here:

How to Stay with Leapforce, Lionbridge and Appen For a Long Time and not get fired?

Time Management and the Leapforce Extension

Time management of the work you complete can become a problem if you aren’t managing how much time you are spending on tasks. You should not sacrifice quality for time. In the beginning you may spend more time than the average estimated time that Leapforce allocates to the task, but make sure that in the invoice you put average estimated time that they give you. If you spend less time than average estimated time per task or set of tasks, be informed that your invoice will be returned for correction, Leapforce will ask you to put actual time spent, that is why there is no sense to complete tasks faster than average estimated time. Take your time, do everything correctly and watch your time.

Do not forget to log on to the Leapforce extension each time you work in the system.

According to the Leapforce policy: any time spent logging on/off, reading, studying, and break times are not billable. Any tasks that are not submitted or are released are also not billable. Please make sure you are not including any of this time on your invoice.

Also please keep in mind that consecutive invoices submitted with productivity below 100% may result in removal from the project or you may lose your contract with Leapforce.

Invoice Returned for Missing or Incorrect Data

Your invoice has been rejected for missing or incorrect data. Please make the necessary corrections or provide specific details on the reason for each discrepancy.

Time Discrepancy: For the following line item(s) there is a discrepancy between the hours you have submitted and the activity and work recorded on our servers.

Reported: x hours, xx mins
Unaccounted Variance: -xx mins

What raters should do in this situation?

Raters have to deduct xx nnaccounted varianceminutes from the invoice, in the future make sure only AET is entered  into invoice and the Leapforce exstension has to be always to be on while active rating.

Working some tasks much faster than the AET followed by holding on to a task for much longer than the AET and billing for the full amount on your invoice

This email is a reminder that raters are allowed to bill for the AET of a task OR the actual time spent working, whichever is LESS of the two amounts. It has been discovered that you are working some tasks much faster than the AET followed by holding on to a task for much longer than the AET and billing for the full amount on your invoice, which is a violation of official policy and will not be tolerated.

For example: You work 5 tasks, each with an AET of 5 minutes. The first 4 tasks you complete in 1 minute each, and the 5th task you hold on to for 30 minutes.

This is an official warning. If this pattern continues, your invoice will be rejected and you will be removed from the rating program. Be sure that all future invoices are in compliance with policy.


Be informed that it is prohibited to have multiple accounts, without special arrangement. Leapforce and Lionbridge can find duplicate accounts very fast, and as a result, you will lose both accounts.

Leapforce IP change/Network sharing

In some cases people may share their Internet network, while working as search engine evaluator make sure that you have your personal unique IP address and no one uses the same IP to go online for rating project.Once you are in the Project, you may change your IP address many times, it will not affect your project status. Make sure that you do your project tasks in a private place as it stated in the contract.

Non-completion of a minimum required number of tasks

This issue is very common among new Leapforce agents and may lead to termination of your contract, and happens when new agents show no activity in the first month of their service. In particular this can happen if new agents enter the project in the second half of month, and for whatever reasons, they did not complete a minimum required number of tasks. In order to avoid this issue, make sure to show the minimum required activity in the system as per your contract.

Tasks availability and time restriction

In general Leapforce and Lionbridge guarantee at least 2 hours of work per day but task availability may depend on your country. There might be some days when tasks are available for more than 4 hours.

You should understand that the system of task distribution is based on a “first come first served” basis and you need to be ready to catch your portion of tasks when they are uploaded into the system. To identify time slots when task availability is the highest please look at task completion graph, usually tasks are uploaded every day into the system at the same time slots.

Lionbridge does not provide raters with a task availability instrument, so raters have to identify time slots when tasks are uploaded into the system by themselves.

Leapforce also has a system that restricts access to tasks for new raters, and those who are undergoing their quality review. To lift this restriction, you have to complete a certain number of simulation tasks or wait till the next quality review, if your quality score is good, the restriction will be lifted automatically.

Locked for quality audit or restricted for quality?

If you receive an email notifying you that your account has been placed under quality review,this is a good time to identify your weak points and improve your rating skills. Below you will find some tips on how to deal with a “Locked for quality audit” situation.

  1. To speed up the process and to get feedback on your mistakes as quickly as possible, I recommend writing to LP/LB and ask why exactly your account has been placed under review.Sometimesthe automatic system locks raters accounts without specific reasons. Usually such cases are being resolved within 1-2 days.
  1. While waiting for detailed feedback for LP team, start doing simulation tasks in order to improve your rating skills. In most cases the issue is related to Side by Side tasks. Make sure that you understand how to deal with this type of task and that you provide well written comments where required. In many cases, raters are placed under quality review due to poorly written comments.
  1. Once you receive the feedback email with your mistakes/quality issues, read it carefully and reply back with an explanation on how you will improve your weak points.
  1. Once your status is moved to “Restricted for quality”, keep in mind that your service is under inspection, thus making sure that you do your best to provide the best possible service.

Why is my Leapforce status restricted with a meets expectations rating?

Sometimes some raters do not get their restricted status changed to unrestricted with getting a meets expectations or above quality score. This happens because they were restricted for Side by Side or PQ tasks quality issues and feedback was released for needs met tasks. In order to lift up all restrictions a rater will have to wait for the next quality review cycle for type of tasks a rater was initially restricted for.

And finally, at the end of month, if you get meets expectations or an above quality score your account will be unrestricted again.

Here are two videos and link to the article which will help you to understand how you will contribute to improving Google search engine.

Leapforce US raters will become part-time employees: no more full-time work

In  April, US raters received an e-mail from  Leapforce, which states the following “Effective June 1, 2017, [Raters in the US] can work up to 26 hours in a calendar week (Sunday to Saturday).” All raters will be moved to a new platform called raterlab and will not be able to work on another projects at Leapforce. Read more about this change here:

How Does Google Search Work?

Survey:Leapforce/Lionbridge working hours spent per month

Have you ever wondered what hours raters work ?

Knowing these statistics can be useful for raters in terms of estimating  possible productivity .

The survey is completely anonymous and once completed will show the overall statistics.

Thanks for participating!

You Failed the Leapforce (Raterlabs) or Lionbridge Exam – Now What?

Many people fail the Yukon exam at their first attempt because they don’t fully understand the Leapforce way of thinking, and you’ve only got a week to study the 164 page guide. Many people fail the first time because this is not an easy exam.  Leapforce and Lionbridge are aware of this fact, and even mention in the welcome email that many people fail the test. In one week after an unsuccessful attempt,  you will get the second and last chance to re-take the exam. You will have to go through all parts of the exam again, practical tasks in part 2 (Page Quality) and part 3 (Send to Device) will be different from those you had on your first exam, although there may be a few tasks that come from previous exam.

Why do people fail the Leapforce and Lionbridge qualification exams?

Lack of effort and sufficient preparation

The most common reason that people fail the Leapforce exam, is that some candidates think that they may get lucky and pass the exam without proper studying, or you may think you know enough about the internet or have good search skills. Be aware that all parts of exam are designed in a way to assess your ability to understand and apply rating concepts and rules based on a 164 page guide. Its not possible to demonstrate the knowledge of these guidelines without investing a significant amount of effort within the one week they give you to study for the exam. If you don’t at least study the guide, or go over it a couple times, you will most likely fail the theoretical part (part 1) of the exam.

Poor time management

One of the common reasons for unsuccessful attempt to pass the exam is poor time management. People do exam preparation and taking all 3 parts within 1-2 days. This strategy in many cases leads to exam failure, because the exam requires lots of concentration and attention to details…Thus plan your exam carefully so you have enough time for studying the material and taking the exam without rushing. Basically, don’t wait till the last minute.

Inappropriate understanding and/or implementation of rating rules and concepts

Have you failed the Leapforce or Lionbridge exams after carefully studying for them? Don’t worry, a lot of people have. Even people with experience in Search Engine Optimization, or even experience in taking exams have failed this test. In fact it is the first thing that Leapforce mentions before you get the exam. That it is very difficult and many fail. If you have failed, it is most likely you misunderstood the rating concepts and/or applied them in a wrong way; e.g. used personal judgment/common sense, or did not pay attention to details where it was required (e.g. user location for local queries). In this case I recommend you get assistance from an expert, such as myself.

Leapforce & Lionbridge exam tutoring and assistance

If you are not confident about going through the Leapforce, Raterlabs or Lionbridge qualification exam , you may always get my expert assistance, support and advice. For detailed information please write to

Note: I am not giving out Leapforce and Lionbridge exam answers, so please do not ask. The reasons for this is because part 2 (Page Quality) and Part 3 (Send to Device) tasks are unique practical tasks, tasks in those parts are updated on a regular basis. The best way is to understand what you are doing and get training on the subject before and after you pass the exam. If you are not sure you understand rating concepts well and and you are not sure how to rate exam tasks correctly, then that is when its best to get my assistance which will give you confidence and lead to success in passing the exam.

You also might be interested in this post:


Useful links

Bellow you can find the list of links to useful forums where people discuss Leapforce and Lionbridge related topics:

Yukon project exam help Leapforce exam help Lionbridge exam help lionbridge exam answers project yukon qualification exam answers appen exam ads

How to pass the Leapforce (Raterlabs), Appen (Inca) and Lionbridge exams? (Exam Guidelines 2018)

How to crack Leapforce exam? How to pass Yukon exam? How to pass Lionbridge exam? How to pass Inca exam?

Before you can start working on various tasks, you must pass the qualification exam. The qualification exam you will take covers two types of rating tasks which will make up 70% of all tasks in the project: Needs Met and Page Quality.

Once your candidature is approved for certification, you will get an exam email with detailed instructions. Within seven days you have to study 160 pages of the General Guideline which consists of rating rules and concepts which teaches you how to do the rating of search results and assessment of quality of web pages. You should follow these rules and not use your personal judgment. What they want is for you to represent a typical user in your country. Before you proceed with the exam you need to study the Guideline at least twice paying particular attention to practical task examples. Keep in mind that it’s 160 pages long. As I always advise, don’t wait till the last minute.

Note: At the stage of examination, Lionbridge provides candidates with very good exam preparation support. You will get access as an exam guest to the Lionbridge learning portal where you can listen to webinars and even try to do a few practical tasks before the exam!

Important update: Lionbridge is very selective in terms of recruiting new raters, up on revising exam results they will check  your activity on their exam preparation portal, with this in mind, I strongly recommend you go through all the webinars and tasks simulation on their exam preparation portal before proceeding with the exam itself. Make sure that you get a pass for Needs Met simulation tasks. You may do them several times until getting a pass.

There are three parts in the exam: you can proceed with the next part only if you pass the previous one. Once you complete any part of exam, within 3-10 minutes you will receive an email with information about results.

Google updates the way the exam is done from time to time. The main feature that has been changed recently is that you must rate pages from your smartphone (Send to Device tasks). If you do not have smartphone, they say you cannot apply for this job. This is due to the fact that the number of mobile users is constantly growing and the company wants to assess websites from the perspective of on-the-go mobile users.

Note:There is a recommended schedule that will give you plenty of time so that you’re not rushing to complete the exam at the last minute:

  • Day 1 – Study the Guidelines – Make sure to setup your smartphone so that you are ready to take the final portion of the exam.
  • Day 2 – Start the Theory part of the exam making sure you are taking time to CAREFULLY read each question. Some of them can be tricky and that’s meant to make sure you fully understand the concepts.
  • Day 3 – Go back and study the guidelines once again starting with the Page Quality section.
  • Day 3/4 You can start the second part of the exam which is the Page Quality section if you feel comfortable on day 3. You may want to study this entire day, and you’ve got plenty of time if you decide this to take the exam on day 4.
  • Day 5 – Read the Needs Met section of the guidelines again and make sure to take your time. You’ve still got two more days to finish. And as it has been mentioned before, have your guidelines open in case you need to refer back to them.
  • Days 5/6/7 – Get started on the last part of the exam ‘Needs Met Rating’

Part one Theory (Leapforce exam part one)

In this part of the exam you will be assessed on how well you understand the theoretical part of rating concepts, as well as on your ability to interpret questions and search for information in the General Guidelines. It is an open book exam, so you can use the General Guidelines to find answers to the test questions. This part has 32 question sections; each section has 4 parts, so in fact you have 132 true/false questions to answer. Read questions carefully, some of questions can be tricky in terms of their wording. You can consult with the guideline book while passing the exam and it’s strongly recommended to use this opportunity.

Note: Do not leave incomplete tasks more than for 1 hour, otherwise they will expire and you will fail the exam automatically. If you need to take a break,complete tasks and press “Submit and Stop rating”.

In most cases an hour would be enough to complete the theoretical part of Leapforce exam. The following message will be shown to you once you have submitted the last page of questions:

No tasks are currently available. Please try again later. 

Within 10 minutes a result notification will come to your gmail, and in the case you are successful with part 1, the next part will be available in 10 minutes.

Part two Page Quality (Leapforce exam part two)

In this part of the exam you will be asked to assess 7 web pages for their quality. My advice for this part is to check all quality aspects, and fill in the online form very carefully so you can give a correct overall quality score (from lowest to highest) to a page. You should expect 2-3 low quality web pages, 2 high quality web pages and 2-3 medium quality web pages. Keep this in mind. In order to give correct ratings, examine web pages for ETA (Expertise, Authority and Trustworthiness) criteria very carefully.

Also, keep in mind that in this part of exam you need to give a PQ rating mainly to a web page and not to a website, although some quality aspects of a website have to be taken into consideration when assigning overall PQ rating, in particular in case of YMYL (Your Money Or Your Life) content which Google takes very serious.

Do not forget to write useful comments where required, otherwise the system may show an error.

You don’t need to finish this part at once. If you want, you can take a rest. The tasks are still available when you sign in. Again,  if you pass this part, the company will send you an e-mail saying that you can proceed with part 3.

Note: The url/web page may not open on your PC, in this case do not mark these as “Did Not Load”. During the exam you should not expect to give a website this flag. If you do so, your answer will be incorrect. To solve this issue try to open link using incognito mode on your browser or/and give rating using other sources of information (website reputation, etc).

Снимок экрана 2016-01-10 в 17.25.24
Page Quality Task

Part three Send to device (Leapforce exam part three)

This is probably the most difficult part of the exam, and many candidates fail the exam at this stage. There are 7 tasks, and each task contains 8-12 result blocks. In each task you will be given a certain query, and a list of search results blocks with links where you have to give a usefulness rating to search results using a scale from “Fails to Meet” to “Fully Meets”. Lionbridge  and Leapforce exam passing score: in order to pass this part you have to get an overall score of at least 80%.

Note: while doing Leapforce and Lionbridge exam part 3, keep in mind that all links have to be opened on your phone because Google analyses web pages from the point of view of mobile users. If you do not have a smartphone, you may use any android emulator for this purpose, in some cases it is even more convenient than using a smartphone for opening links.In order to open the link on your mobile phone, click the  link on the result block on your desktop and then go to  on your mobile browser  (Google Chrome) to visit the link. You must be logged in your gmail account in order to have access to raterhub platform on your phone. The same rule works if you use android emulator.

Note: Some candidates when accessing the link ( ), got informed that Gmail account was unauthorized to visit the link. Most likely reason for this error is the fact of being logged on multiple Gmail accounts on a device. The solution for this problem is to go to Android setting -> Users & accounts -> add project associated Gmail account, then restart your device. This setting should fix the issue. 

When you are evaluating landing pages in the exam, don’t place too much importance on the Page Quality slider. The focus needs to be on Needs Met Ratings. Do not worry about using flags in the exam such as the ‘Hard to Use’ flags.

One of the most important things is to absolutely know when to use Fully Meets. Fully Meets ratings are very rare compared to Highly Meets ratings. Avoid being too generous on landing pages. If you have any doubts at all, you should not be rating a page ‘Fully Meets’.

Additionally I would like to give general tips on how to proceed with Leapforce (Lionbridge) exam part 3:

1. Make sure that you fully understand the query and user intent. If you do not fully understand user intent, make additional research on the Internet.

2. Determine what type of query (user intent) is:

o Single entity (business) query with local or non local or both intent/to visit in person/navigate to a website

o Business chain query or query which is looking for business within one category (e.g. car repair service near me)

o Simple know query (where the user requests a specific piece of information, do not assign the Fully Meets rating to any particular result, the correct answer must be present in an SCRB).Examples of simple know queries:paris weather, delta airlines customer service number, current time in Amsterdam)

o Information query seeking a long answer/piece of information (what cases stress)

Types of SCRBs (Special Content Result Blocks): ny weather (simple know query), how old is universe (simple know query), does god exist (long answer seeking), berlin airport (single entity), burger king amsterdam (chain), coffee shops amsterdam (categorical).

o URL/website query

3. Consider the following aspects while giving ratings to result blocks: user location, for local intent queries far away results are useless or slightly relevant. This aspect is very important for the following types of queries: tire shops near me, where to eat near me, Walmart near me, etc. When you get a task with this type of query do use in built map for researching user location/view port and results.

4. Make sure you give correct ratings to special content result blocks for single entity and business category queries. In case of business chain/category query, the result block with a list of businesses is always better than just result for a single business within one category.

5. Consider expertise and trustworthiness of search results for financial, legal, and health related queries; low expertise and trustworthiness automatically reduce needs met rating of the results.

6. Think of dominant, possible common, and minor interpretations for the query and take this aspect into account while giving ratings to a result block. Minor interpretation results should get a low usefulness (Needs Met) rating.

7. Take into account how well the web page/website is optimized for mobile users.

8. Once you have analyzed and considered all listed above aspects, you should give a rating to result blocks one by one.

Note: In some cases there might be some results (links) which are not accessible from outside U.S. (usually websites of well known retailers such as Target, Walmart, etc. They ban access from foreign IPs to prevent hacker attacks). Raterlabs or Lionbridge do not take into account this factor and therefore rating such results as fails to met may lead to overall fail.

To make sure you do not miss anything before proceeding with part 3, follow pre-exam instructions to pass this part. There will be a link to a video. Watch it and follow the instructions.

Examination Need Met Task

Note: In part 3 of the exam (Needs Met), most of results blocks have Needs Met and E-A-T (previously PQ) sliders. Main focus should be on providing correct Needs Met rating, which in some cases might be affected by E-T-A (PQ) rating when it comes to YMYL queries and results. Leapforce/Google re-named PQ to E-T-A in part 3 to make it clear that the most important aspects in PQ are expertise, trustworthiness and authority. In general it is similar to PQ concept. While giving E-T-A rating you should evaluate MC under these 3 criteria. In most cases you may do it fast at a first glance approach, but be sure to provide good evaluation for E-T-A when you have YMYL query.

Note: If the user location map does not show user location correctly, and you have a blank page, hit the black arrow button. The task will be closed, but you will see “Incomplete tasks” on the evaluation platform. Click continue and come back to your task with the updated user location field. Remember that it is very important to take into account the user location while giving ratings to block results. Also keep in mind that there is no need to finish the task in one sitting. You can take a break if you need to.

Once you complete 7 tasks you will get this message:

No tasks are currently available. Please try again later.

Leapforce exam results and Lionbriage exam results

You will get overall exam results on the exam due date from Leapforce and exam due date plus one day  if your vendor is Lionbridge. At Leapforce you will be  asked to sign a few electronic forms and in 2-4 days you will be given access to live tasks. At Lionbridge you will have to complete educational course with no pass/no fail quiz before you are granted full access to the tasks.

US Raterlabs and Lionbridge raters

Once you pass the exam you will then go through the employment process which might take up to 3 weeks. You will be asked to complete an online version of the IRS Form 8850 (Pre-Screening Notice and Certification Request for the Work Opportunity Credit), and a tax credit questionnaire from a vendor payroll provider. After you have completed these two forms, you will be presented with a formal employment offer letter. After acknowledging your offer, you will still need to complete the required activities in a timely manner. Try not to postpone the completion of these important steps: Providing bank details for ACH payments on Paycom website. Providing your I9 Employee Section 1 and I9 Employment Eligibility Documents (section 2) must be completed by specified deadline. It is very important that candidates complete all required steps as soon as possible to avoid termination of your employment.

The following posts are recommended for reading:

You Failed the Leapforce (Raterlabs) or Lionbridge Exam – Now What?

Common issues of new raters (Yukon Raterlabs/Leapforce and Lionbridge Google Rating Projects)

What Should You Do If You Fail Leapforce or Lionbridge Exam Twice and How to Re-Apply?