List of Projects And Code Names For Lionbridge & Appen Connect

If you are looking for a list of projects out there from companies like Lionbridge and Appen, we have them. As you may know Appen acquired Leapforce in 2017, since then the company has decided to close it is own project management platform known as Appen Global and move all projects to Appen connect. The group of projects known as Falcon (mainly Facebook-related projects) now is no longer actual, these projects also have been moved to Appen connect platform.

Taking into account the fact that all projects have code names and little description is provided, many candidates might be lost in terms of what project to go for. 

Bellow you will find a list of most popular projects and links to more info about each project:

Appen connect
Project code nameDescription Reference for more info
YukonGoogle Search Engine Evaluation Project Click here

Exam Simulation Tool

Get Personalized Assistance or Advise
ArrowGoogle Ads Evaluation ProjectClick here

Exam Simulation Tool

Get Personalized Assistance or Advise
ShastaMaps Evaluation Project

Provide relevance and accuracy ratings for one of the world's largest and most popular mapping applications.
The job has an ABOVE average volume of work.
Click here

Get Personalized Assistance or Advise
POI ProjectMaps Evaluation Project
Provide relevance and accuracy ratings for one of the world's largest and most popular mapping applications.
The job has an ABOVE average volume of work.
Click here
Echo Apple iTunes Music Search Results Evaluation Project

Improving the relevance of search results for music-related content.
Raters must have access to an iOS device capable of running iOS 12.x. and higher. The job has an average volume of work.
Click here

Exam Simulation Tool

Get Personalized Assistance or Advise
CableApple TV Search Result Evaluation.

Search Relevance task focused on improving the relevance of search results, based on the most likely intent of user searches. Raters must have access to an iOS device capable of running iOS 12.1.4. The job has an average volume of work.
Similar to Echo

Get Personalized Assistance or Advise
JavaApple Store Search Relevance project focused on improving the relevance of search results based on user search intent.
Raters must have access to an iOS device capable of running iOS 12.1.4
Similar to Echo
SharpApple Search Relevance task focused on improving the relevance of search results within different media categories.
Raters must have access to an iOS device capable of running iOS 12.1.4
Similar to Echo
IndexApple App Store Search Results Evaluation Project

Search Relevance task focused on improving the relevance of search results based on user search intent.
Raters must have access to an iOS device capable of running iOS 12.1.4. The job has an average volume of work.
Similar to Echo
SlickrockThis project focuses on Facebook Stories and rating these personalized recommended stories. Click here
Wouri Apple Safari Search Result Evaluation project. The goal of the project is to understand the user intention as represented by the query and determine how well or how completely the result satisfies the user's need. Click here
Pecos/Ipsoot (also known as Pearl and Falcon PT)Post categorization on Facebook. You have to decide what topic that post fits under. It is basically a categorizing system job which aims to put the posts that are made into the correct categories of broad topics. Click here
TruckeeInstagram Ads Evaluation Project

Help improve ads quality for one of the world’s largest image sharing sites.
Required: An active Facebook and Instagram account – The Instagram account must be connected to your Facebook account. You also need a smartphone with Android or iOS. The job has an Above Average work volume.
Click here
NileFacebook Ads Evaluation Project

Help improve ads quality for one of the world’s largest social networking platforms.
Active Facebook account required.
Click here
TahoeThere is a Facebook project called Tahoe that uses a team that checks events that come up in Facebook results. The main mission of this program is to bring people together through real-world activities by showing real events.Click here

Get Personalized Assistance or Advise
BristolFacebook project to help identify duplicates that come up in search results. This helps improve the user experience for people searching for something on Facebook.Click here
RattleThe goal of this project is to improve the relevance of search results for one of the world’s largest search engines. Click here
ParadiseThe goal of this project is to improve the accuracy of search results for businesses and entities on one of the world’s leading search engines. Click here
Mohawk Help improve ad relevancy on a popular social media platform.
Similar to FB
Falcon projects
Prospect MI Facebook Falcon Projects. Help improve ad relevancy on a popular social media platform.
Click here
Zeya Facebook Falcon Projects. In this task, you will be asked to research and evaluate claims stated in various pieces of content (e.g., articles, links, photos, or videos). 20 hrs per week, work is done via Facebook rating tool.
Click here
Copper The aim of this project is to help the client evaluate and improve the quality of the Ads that they present to their users.
Requirements are a personal computer running Windows 8 or better and a desire to look at different ads.
Click here
Alkali The aim of this project is to help the client evaluate and improve the quality of the Ads that they present to their users.
Requirements are a personal computer running Windows 8 or better and a desire to look at different ads.
Average volume of work.
No info
TigrisThe project consists of rating tasks focusing on improving the relevancy of advertisement results from a major social media platform. Consultants will rate advertisements to help the client understand which advertisements (also known as ads) people want to see, and why. The daily task of the consultant consists of rating a batch of ads in a client-owned rating tool. FB project?
CarmelUtterance annotation and evaluation.Microsoft Bing project aimed to learn search engine to recognize voice and human speech. No info

*group of projects known as Falcon (mainly Facebook-related projects) now is no longer actual, these projects also have been moved to Appen connect platform. Newly added projects such as Mohalwk, Prospect MI, Zeya, Copper, and Alkali are most likely related to small Facebook testing projects. If you have been participating in one of the above-listed projects, feel free to share your experience with us via leaving a comment option!

Lionbridge project code nameDescription Reference for more info
Internet Assessor, Personalized Internet Assessor, RaterGoogle Search Engine Evaluation Project Click here

Exam Simulation Tool

Get Personalized Assistance or Advise
Internet Safety EvaluatorGoogle Youtube Evaluation Project Click here

Exam Simulation Tool

Get Personalized Assistance or Advise
Multimedia JudgeBing Search Engine Evaluation project Click here
Personalized Ads Assessor Google Ads Evaluation Project Click here

Exam Simulation Tool

Get Personalized Assistance or Advise
Online Map Quality AnalystMaps Evaluation Project Click here

Get Personalized Assistance or Advise
iSoftStone project code namesDescription Reference for more info
Online Ad EvaluatorBing Ad Evaluation project Click here
Search Engine EvaluatorBing Search Engine Evaluation project Click here

Step-by-Step Guide How to Pass Appen Yukon Project Exam and Lionbridge Internet Assessor, Internet Safety Evaluator Exams

Looking to pass aPPEN Yukon project/lIONBRIDGE iNTERNET Assessor & Internet Safety Evaluator Exams? THEN YOU HAVE COME TO THE RIGHT PLACE FOR all THE INFORMATION THAT YOU NEED
(Updated Jan 2020) 

Before you can start working on various tasks, you must pass the qualification exam. The qualification exam you will take covers two types of rating tasks which will make up 70% of all tasks in the project: Needs Met and Page Quality.

Once your candidature is approved for certification, you will get an exam email with detailed instructions. Within seven days you have to study 160 pages of the general guideline which consists of rating rules and concepts. The guide explains you how to do the rating of search results and assessment of quality of web pages. You should follow these rules and not use your personal judgment. What they want is for you to represent a typical user in your country. Before you proceed with the exam you need to study the guideline at least twice paying particular attention to practical task examples. Keep in mind that it’s 160 pages long. As I always advise, don’t wait till the last minute.

Note: At the stage of examination, Lionbridge provides candidates with very helpful exam preparation support. You will get access as an exam guest to the Lionbridge learning portal where you can listen to webinars and even try to do a few practical tasks before proceeding with actual exam.

Important update: Lionbridge is very selective in terms of recruiting new raters, up on revising exam results they will check  your activity on their exam preparation portal, with this in mind, I strongly recommend you go through all the webinars  on exam preparation portal before proceeding with the exam itself. 

There are three parts in the exam: you can proceed with the next part only if you pass the previous one. Once you complete any part of exam, within 3-10 minutes you will receive an email with information about results.

Google updates the way the exam is done from time to time. The main feature that has been changed is that you must rate pages from your smartphone (Send to Device tasks). If you do not have smartphone, they say you cannot apply for this job. This is due to the fact that the number of mobile users is constantly growing and the company wants to assess websites from the perspective of on-the-go mobile users.

Note: There is a recommended schedule that will give you plenty of time so that you’re not rushing to complete the exam at the last minute:

  • Day 1 – Study the Guidelines – Make sure to set up your smartphone so that you are ready to take the final portion of the exam.
  • Day 2 – Start the Theory part of the exam making sure you are taking time to CAREFULLY read each question. Some of them can be tricky and that’s meant to make sure you fully understand the concepts.
  • Day 3 – Go back and study the guidelines once again starting with the Page Quality section.
  • Day 3/4 You can start the second part of the exam which is the Page Quality section if you feel comfortable on day 3. You may want to study this entire day, and you’ve got plenty of time if you decide this to take the exam on day 4.
  • Day 5 – Read the Needs Met section of the guidelines again and make sure to take your time. You’ve still got two more days to finish. And as it has been mentioned before, have your guidelines open in case you need to refer back to them.
  • Days 5/6/7 – Get started on the last part of the exam ‘Needs Met Rating’
Internet Assessor and Yukon project exam part 1 (theory)

In this part of the exam you will be assessed on how well you understand the theoretical part of rating concepts, as well as on your ability to interpret questions and search for information in the general guidelines. It is an open book exam, so you can use the general guidelines to find answers to the test questions. This part has 32 question sections; each section has 4 parts, so in fact you have 132 true/false questions to answer. Read questions carefully, some of questions can be tricky in terms of their wording. You can consult with the guideline book while passing the exam and it’s strongly recommended to use this opportunity.

Note: Do not leave incomplete tasks more than for 1 hour, otherwise they will expire and you will fail the exam automatically. If you need to take a break, complete tasks and press “Submit and Stop rating”.

In most cases an hour would be enough to complete the theoretical part of the exam. The following message will be shown to you once you have submitted the last page of questions:

No tasks are currently available. Please try again later. 

Within a few minutes a result notification email will come to your Gmail, and in the case you are successful with part 1, the next part will be available in a few minutes.

Lionbridge Internet Assessor and Appen Yukon project exam part 2 (Page Quality)

In this part of the exam you will be asked to assess 7 web pages for their quality. My advice for this part is to check all quality aspects, and fill in the online form very carefully so you can give a correct overall quality score (from lowest to highest) to a page. You should expect 2-3 low-quality web pages, 2 high-quality web pages, and 2-3 medium quality web pages. However this is not a rule, sometimes candidates get 4-5 pages of high or medium quality pages and only 1 or 2 of low-quality pages to rate. Keep this in mind. In order to give correct ratings, examine web pages for ETA (Expertise, Authority, and Trustworthiness) criteria very carefully.

Also, keep in mind that in this part of the exam you need to give a PQ rating mainly to a web page and not to a website, although some quality aspects of a website have to be taken into consideration when assigning overall PQ rating, in particular in case of YMYL (Your Money Or Your Life) content which Google takes very seriously.

Do not forget to write useful comments where required, otherwise, the system may show an error.

You don’t need to finish this part at once. If you want, you can take a rest. The tasks are still available when you sign in. Again, if you pass this part, the company will send you an e-mail saying that you can proceed with part 3.

Note: The URL/web page may not open on your smartphone, in this case, do not mark these as “Did Not Load”. During the exam you should not expect to give a website this flag. If you do so, your answer will be incorrect. To solve this issue try to open links using incognito mode on your browser or/and give rating using other sources of information (website reputation, etc).

Снимок экрана 2016-01-10 в 17.25.24
Page Quality Task
Internet Assessor and Yukon project exam part 3 (needs met)

This is probably the most difficult part of the exam, and many candidates fail the exam at this stage. There are 7 tasks, and each task contains 8-12 result blocks. In each task, you will be given a certain query, and a list of search results blocks with links where you have to give a usefulness rating to search results using a scale from “Fails to Met” to “Fully Met”.

Internet Assessor and Yukon project exam Passing score

In order to pass this part you have to get an overall score of at least 80%.


Note: while doing Yukon (Appen) and Internet Assessor (Lionbridge) exam part 3, keep in mind that all links have to be opened on your phone because Google analyses web pages from the point of view of mobile users. If you do not have a smartphone, you may use any android emulator for this purpose, in some cases it is even more convenient than using a smartphone for opening links. In order to open the link on your mobile phone, click the link on the result block on your desktop and then go to  on your mobile browser  (Google Chrome) to visit the link. You must be logged in your Gmail account in order to have access to raterhub platform on your phone. The same rule works if you use an android emulator.

Note: Some candidates when accessing the link ( ), got informed that Gmail account was unauthorized to visit the link. The most likely reason for this error is the fact of being logged on multiple Gmail accounts on a device. The solution for this problem is to go to android setting -> users & accounts -> add project associated Gmail account, then restart your device. This setting should fix the issue. 

Lionbridge Internet Assessor and Appen Yukon project exam part 3 (Needs Met) Expert Tips and recommendations 

Additionally, I would like to give general tips on how to proceed with Yukon (Appen) and Internet Assessor (Lionbridge) exam part 3:

1. Make sure that you fully understand the query and user intent. If you do not fully understand user intent, make additional research on the Internet.

2. When you are evaluating landing pages in the exam, don’t place too much importance on the Page Quality slider. The focus needs to be on Needs Met Ratings. Do not worry about using flags in the exam such as the ‘Upsetting-Offensive Content’ flags.

3. One of the most important things is to absolutely know when to use Fully Meets. Fully Meets ratings are very rare compared to Highly Meets ratings. Avoid being too generous on landing pages. If you have any doubts at all, you should not be rating a page ‘Fully Meets’.

4. Determine what type of query (user intent) is:

o Single entity (business) query with local or non local or both intent/to visit in person/navigate to a website

o Business chain query or query which is looking for business within one category (e.g. car repair service near me)

o Simple know query (where the user requests a specific piece of information, do not assign the Fully Meets rating to any particular result, the correct answer must be present in an SCRB).Examples of simple know queries:paris weather, delta airlines customer service number, current time in Amsterdam)

o Information query seeking a long answer/piece of information (what cases stress)

Types of SCRBs (Special Content Result Blocks): ny weather (simple know query), how old is universe (simple know query), does god exist (long answer seeking), berlin airport (single entity), burger king amsterdam (chain), coffee shops amsterdam (categorical).

o URL/website query

5. Consider the following aspects while giving ratings to result blocks: user location, for local intent queries far away results are useless or slightly relevant. This aspect is very important for the following types of queries: tire shops near me, where to eat near me, Walmart near me, etc. When you get a task with this type of query do use in built map for researching user location/view port and results.

6. Make sure you give correct ratings to special content result blocks for single entity and business category queries. In the case of a business chain/category query, the result block with a list of businesses is always better than just results for a single business within one category.

7. Consider expertise and trustworthiness of search results for financial, legal, and health-related queries; low expertise and trustworthiness automatically reduce needs met rating of the results.

6. Think of dominant, possible common, and minor interpretations for the query and take this aspect into account while giving ratings to a result block. Minor interpretation results should get a low usefulness (Needs Met) rating.

8. Take into account how well the web page/website is optimized for mobile users.

9. Once you have analyzed and considered all listed above aspects, you should give a rating to result blocks one by one.

Note: In some cases there might be some results (links) that are not accessible from outside the U.S. (usually websites of well-known retailers such as Target, Walmart, etc. They ban access from foreign IPs to prevent hacker attacks). Appen, Raterlabs, or Lionbridge do not take into account this factor, and therefore rating such results as fails to met may lead to overall fail.

To make sure you do not miss anything before proceeding with part 3, follow pre-exam instructions to pass this part. There will be a link to a video. Watch it and follow the instructions.

Examination Needs Met Task

Note: In part 3 of the exam (Needs Met), most of the results blocks have Needs Met and E-A-T (previously PQ) sliders. The main focus should be on providing correct Needs Met rating, which in some cases might be affected by E-T-A (PQ) rating when it comes to YMYL queries and results. Leapforce/Google re-named PQ to E-T-A in part 3 to make it clear that the most important aspects in PQ are expertise, trustworthiness, and authority. In general it is similar to PQ concept. While giving E-T-A rating you should evaluate MC under these 3 criteria. In most cases you may do it fast at a first glance approach, but be sure to provide good evaluation for E-T-A when you have YMYL query.

Note: If the user location map does not show user location correctly, and you have a blank page, hit the black arrow button. The task will be closed, but you will see “Incomplete tasks” on the evaluation platform. Click to continue and come back to your task with the updated user location field. Remember that it is very important to take into account the user location while giving ratings to block results. Also keep in mind that there is no need to finish the task in one sitting. You can take a break if you need to.

Once you complete 7 tasks you will get this message:

No tasks are currently available. Please try again later.

Yukon (Appen) exam results and Internet Assessor (Lionbridge) exam results

You will get overall exam results on the exam due date from Appen and exam due date plus one day if your vendor is Lionbridge. Appen will ask you to sign a few electronic forms and in 2-4 days you will be given access to live tasks. At Lionbridge you will have to complete an educational course with no pass/no fail quiz before you are granted full access to the tasks.

US YUKON Raterlabs and Lionbridge raters

Once you pass the exam you will then go through the employment process which might take up to 3 weeks. You will be asked to complete an online version of the IRS Form 8850 (Pre-Screening Notice and Certification Request for the Work Opportunity Credit), and a tax credit questionnaire from a vendor payroll provider. After you have completed these two forms, you will be presented with a formal employment offer letter. After acknowledging your offer, you will still need to complete the required activities in a timely manner. Try not to postpone the completion of these important steps: Providing bank details for ACH payments on the Paycom website. Providing your I9 Employee Section 1 and I9 Employment Eligibility Documents (section 2) must be completed by the specified deadline. It is very important that candidates complete all required steps as soon as possible to avoid the termination of your employment.

NEW!!! Welcome to the Search Engine Evaluation Exam (Yukon, Internet Assessor) Simulation Tool!
What is The Exam Simulation Tool? 

I have created a special tool that will help anyone who is getting ready or considering taking the exam. This tool was designed to assist candidates with exam preparation with the main focus on practical rating tasks that include Page Quality, and Search Results Satisfaction ratings. The tool has 5 Page Quality tasks and 5 Search Results Satisfaction (Needs Met) tasks of all the major query types. (If you’re not sure what a query type is, that is the keyword that a user searches Google for). These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/task types and skills which will be tested during a real exam.

How can this tool help you pass the exam?

The way the tool is designed will show you exactly your problem area and what you need to focus on. It helps you identify your level of understanding of the rating concepts, and your ability to apply them by practicing first and then identifying your weak points. You will receive your overall score (pass or fail), detailed feedback on your mistakes, and know exactly what you need to go back and study in the guide. Of course we need to mention that there are no real actual exam tasks because that isn’t going to help you understand anyway. Each one of these is hand-designed from the most typical exam tasks and is also designed to be the best simulation possible. If you pass this, you will be ready to pass the actual exam. 

How do I get my hands on the exam simulation tool?

Exam Simulation Demo

If you’re still skeptical, we have a free demo version which you can try FREE right now! It contains a simplified Needs Met and one Page Quality task so you can see how it works. Once again, the demo is 100% free and remain completely anonymous.

Exam Simulation Pro

The tool has 5 Page Quality tasks and 5 Search Results Satisfaction (Needs Met)  of all the major query types. These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/task types and skills which will be tested during a real exam.

Explore the tool here:

Yukon and Internet Assessor Expert Support

If you are not confident about going through the Appen Raterlabs or Lionbridge qualification exam, you may always get my expert support and advice. For detailed information contact me here.

Payoneer Payment method for Appen 

Leapforce/Appen/Raterlabs candidate? If you get a job at one of these places, you’re going to need a Payoneer account because that is the only payment option available for these companies. Simply sign up for Payoneer under my referral link and you’ll get the simulation tool subscription fee back as Payoneer subscription 25$ bonus once your account is loaded with 1000$! Normally it takes 1-2 months to make 1000$ for EU/USA/Canada raters and 2-4 months for raters from other countries.

The following posts are recommended for reading:

Common Issues of New Raters: Appen Yukon and Lionbridge Internet Safety Evaluator Google Rating Projects

You Failed Appen Yukon Project Lionbridge Internet Assessor Exam Now What?

Exam Simulation Tool For Appen Yukon Project Exam/Lionbridge Internet Assessor & Internet Safety Evaluator Exams

What is The Exam Simulation Tool? 

I have created an exam simulation tool that will help anyone who is getting ready or considering taking the exam. This tool was designed to assist candidates with exam preparation with the main focus on practical rating tasks that include Page Quality, and Search Results Satisfaction ratings. The tool has 5 Page Quality tasks, and 5 Search Results Satisfaction (Needs Met) tasks of all the major query types. (If you’re not sure what a query type is, that is the keyword that a user searches Google for). These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/tasks types and skills which will be tested during a real exam.

How can this tool can help you pass the exam?

The way the tool is designed will show you exactly your problem area and what you need to focus on. It helps you identify your level of understanding of the rating concepts, and your ability to apply them by practicing first, and then identifying your weak points. You will receive your overall score (pass or fail), detailed feedback on your mistakes and know exactly what you need to go back and study in the guide. Of course we need to mention that there are no real actual exam tasks because that isn’t going to help you understand anyway. Each one of these is hand designed from the most typical exam tasks, and is also designed to be the best simulation possible. If you pass this, you will be ready to pass the actual exam. 

How do I get my hands on the exam simulation tool?

Exam Simulation Demo

If you’re still skeptical, we have a free demo version which you can try FREE right now! It contains a simplified Needs Met and one Page Quality task so you can see how it works. Once again, the demo is 100% free and remain completely anonymous.

Exam Simulation Pro

The tool has 5 Page Quality tasks, and 5 Search Results Satisfaction (Needs Met)  of all the major query types. These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/tasks types and skills which will be tested during a real exam.

Explore the tool here:

Step-by-Step Guide How to Pass Appen Yukon Project Exam and Lionbridge Internet Assessor, Internet Safety Evaluator Exams

Apply for Search Engine Rater and Social Media Evaluation Jobs at Appen/Lionbridge

Want to Apply For Search Engine Rater and Social Media Evaluations jobs at Lionbridge and Appen?

Sometimes finding these types of jobs online takes a bit of research. Most of the big companies that offer these types of jobs don’t usually advertise in common places or online job boards. You usually have to go directly to the company and apply on their website. I have seen that Appen Global advertises their jobs on, but that website charges a monthly fee. I never quite understood how a company can charge you to find a job, but they are the exception.

Finding these jobs and applying for them are different for each company. Going to each individual website is the best way, as long as you know where to go, and how to apply. With that being said, this article, and the videos we have provided, will show you how to find the work from home jobs.

One of the other important things to consider when looking for these types of jobs is your location. Some of the contracts that are offered through companies like Appen and Lionbridge require you to live in the demographic area that the project requires itself. The reason for this is because the data from different regions will be different depending on location and the language.

Before you begin to apply for these jobs, or really any job for that matter, it’s probably a good idea to go over your resume and make sure it’s up to date, and that it includes only the important information that is necessary. It might also be to your benefit to have your resume checked by a professional. The resume experts at is an awesome service that I have used in the past, and having your resume edited or corrected by a professional is a sure-fire way to get noticed and get you the job you want. Many of our readers have had very good results with them.

To visit their website Click Here:

Another important note is that if you have worked previously on other social media projects with different companies, don’t mention this on your CV/resume. The two most important factors about this are that you do not need previous experience for these jobs. And second, if you have worked on these types of projects before you will most certainly be looked over and turned down for similar projects. The reason for this is uncertain, but we have seen other people turned down because of this.

You can work on two different projects for more than one company. However, you need to use a different IP address for one or the other. So if you work for Appen and want to work on another project with Lionbridge, you will likely need a different IP address. Appen seems to be the only company that records an IP address and this is shown in the backend or the portal of the work dashboard. This is why it’s a good idea to use a separate IP address for another project. It might also be a good idea to use a different email between companies when applying for and working on these jobs.

Projects and Vendors Compatibility: How to Work on Several Projects and Do Not Violate Vendors Conditions? How Much Can You Earn With Lionbridge and Appen When Combining Several Projects?

Some advice when working on different projects so that you don’t overwhelm yourself is to get some experience working on one project with a company before trying to get another contract with a separate company. Most of these contracts are part-time. So it’s beneficial to have more than one project at a time if you can handle it.

A really important thing to note here is that you can only have one contract per IP address when it comes to these companies. So if more than one person is working for any of these companies, it is important to have a separate IP address. You can usually order another IP address from your internet service provider for about $4-5 per month extra. Check with your ISP to find out if you require more than 1 IP address per household. You can read more about this in FAQ

Application process at Lionbridge – Ads Assessor, Internet Safety Evaluator, Ads Assessor 

Lionbridge search rater project is called the Google rating project (ads assessor, internet safety evaluator, ads assessor) which is the same as Yukon at Appen (Raterlabs). They also have the positions of Personalized Social Media Assessor, Personalized Internet Assessor, and Personalized Web Assessor.

Application process at Appen – Search Engine Evaluator, Rater, Internet Analyst 

At Appen/Raterlabs you would apply for search engine evaluator position first and then based on project availability, you would get invites for Yukon, Nile, (Facebook social media evaluator) Truckee, or Shasta projects. You can work on more than one project at a time.

Appen Global

Many of my readers will be interested in how to get invited to other projects at Appen Global. Appen Global offers a lot of different types of projects that you are able to work on. Unfortunately, it is not likely that you would be contracted to start on projects such as the Thunderbird music project from the beginning. You would be more likely to start on a larger project first such FalconInstaAds or the Arrow project. Once you have started working at Appen, and you are on a Falcon project, like any of the social media projects that they offer, you will likely get an invitation to start on other projects as well. The more you work, and the better your performance is, the more offers you will get from Appen.

Those projects include:

Arrow (ads assessor);

Yukon (search engine evaluator);

Nile/Truckee Facebook/Instagram ads assessor projects (social media evaluator).

application process video instructions -Lionbridge, Appen, Appen Global

We thought it would be a good idea to create some quick walk-through videos to show you how to apply for these jobs and the different companies. Some of these:

Lionbridge, go to

Appen, go  to

Appen Global go here:

How long does IT take to hear back from Lionbridge and Appen?

It depends on many factors, sometimes immediately, and sometimes it can take 3-4 months.

It might be that at the moment of your application submission, the quota of raters for your country was reached and therefore Appen or Lionbridge does not hire a new rater at the moment. As soon as they fire unsuccessful raters, they continue hiring new people to maintain the required number of raters in your country.

Low Tasks Availability in Ads Assessor/Arrow Project – What Are the Reasons?

As being a rater you might noticed that starting from March 2020 the volume of tasks in the project significantly decreased, and many raters might be wondering when the situation to be improved or back to normal.

Pandemic affects Google Ads Business

Lionbridge notified raters that the number of tasks might be low or unstable due to slow down of the ads business of Google due to pandemic. As you may know tasks are generated from real ads. The vendor confirmed that this a temporary issue, and the customer expects the volume of tasks to be increased in the future. The good thing is that the number of blind tests was decreased due to this fact and there is a very small chance of being fired.  

Seasonal factor

Tasks are not distributed evenly throughout the year, based on 5 years observation I can confirm that the seasonal factor takes place as well, the low season is spring. The vendor informed raters that the customer usually revises and updates rating algorithms in spring and this process affects task volume in the system.

New task distribution Algorithms

Another point that I wanted to clarify relates to the number of tasks that the raters receive, the number of tasks given is not the same for everyone during the month, and some raters can receive more tasks than others. This is due to the task’s distribution algorithms. Usually in the following month, the unevenness is compensated by an increase in the number of tasks for those raters who received fewer tasks in the current month. 

Tips for getting more tasks

Logon raterhub more frequently to check for tasks in particular between 9AM – 3 PM PST  

Currently, working less than 20 hours per week will not affect your status in the system.  

Useful links

Bristol Project For Facebook By Appen Global

There’s a new project being offered by Appen for Facebook called Bristol and agents hired for the project will help identify duplicates that come up in search results. This helps improve the user experience for people searching for something on Facebook. 

You will also be providing the most relevant results to users along with improving product capabilities. 

The one thing Appen stresses is that agents should avoid any personal political or social bias when rating a search result. Facebook has separate teams that judge controversial topics and misinformation. Your job as an agent is to simply judge if something is a duplicate, and choose the one with the best or more up to date information. 

Rating process

The interface is fairly straightforward and easy to understand.

On the top left hand corner you will see the reference details of the Facebook entity, and then below this you will see candidate details. It may be that you see several candidates here. 

The process is to review each one of the candidates and see if they are duplicates of each other. The workflow can be executed as a one-to-one or one-to-many relationship structure. Basically…you just check each one to see if they are the same or not. 

It can be tricky to know if some places are the same or not because they might not have the same name. You need to be careful in this instance and do enough investigation to see if the places are indeed the same, or if the information in the ad is the same. 

You do this by opening each link and comparing the contents to the reference sources. Thankfully, to help sort through some steps, there is a decision tree to help you. 

If you do find other things in common between the ‘Reference Source’ and the candidate, then you would select ‘Match’ to indicate that the entities are duplicates of one another. If there aren’t enough commonalities then you would select ‘No Match’ and move on. 

The decision tree will help walk you through the steps if you’re not sure what to do. 

As you complete the job, you will notice that above the candidates link section there are some questions you will need to answer before moving on through the job. 

The first question is ‘Translation Tools’ and asks:

  • I will need to use translation tools to complete this job
  • The page link is broken and I can’t complete this job
  • None of the above, I can complete this job

Notice that if you need translation tools or the page link is broken, you will be unable to complete the job and you should choose either one when it is applicable. However, in most cases you will use the last one ‘None of the above, I can complete this job.

If the candidate link is broken then you would also choose ‘Unable to investigate’ in Question 2

Question 2 is the search result pair review and at this point, if you chose the last option in question 1, you would review the Reference Source and each one of the candidates…which are supposed duplicates. 

Question 2 offers you 3 options:

  • Match
  • No Match 
  • Unable to investigate

There are some attributes in the guide that need to be evaluated when deciding if two entities are the same or not. 

It’s really easy to decide if the names of both search results are the same if the names match and are exactly the same. We say EXACTLY because they must be word for word and not missing part of the name which would cause you to further investigate to see if they are indeed the same entity. 

There can be what is called a Soft Match such as FB, Inc. for Facebook Inc. where there is some naming connection between the Reference Source and the Candidate. 

There is something you will notice called an ‘Alias Handle’ which is preceded by an at sign @ and you’re probably familiar with this if you have an Instagram account. You can also check to see if these names match or not in order to make a determination. 

Another thing to mention is ‘Chain Locations’ like McDonalds who will sometimes put the location in the name itself. Pay extra attention to this because if those names differ, then they are not the same entity. 

So we are clear on when you should choose ‘No Match’ we will give you the most important information. No match means there is ZERO similarity between the names and the online aliases. Just be careful and check things like the name, alias handle, business name, biography, photos, and posts. 


If you do need to check photos, then it is important to review the photos from both entities in search of matching or unique photos in posts made by the entity. Sometimes it takes a little research to see if you can match them, but you shouldn’t spend a lot of time looking to match photos. Checking the entities Instagram is a quick and surefire way to see if photos match the entity or not and should not take more than 30-45 seconds or so. 

Profile photos do not need to match, but they should resemble the same business or logo of the entity or person as the reference source. 


If it’s not clear and you don’t see matching photos or it’s clear that the photos between the two entities are totally different and not related, or if you don’t have pictures to compare, then consider other signals on the page to check if they are the same. 

No Match

As we mentioned, if you see two different photos that show a different business and they are not related at all, then you don’t have a solid match. Watch for logos to be entirely different. Some companies out there can have the same name or a name that is similar. Check to see if the logos are the same…if not, then you have no match.  

Bio/Descriptions can be another place to look, but if description is the same it doesn’t mean that the two identities are the same. The text needs to contain some unique identifier such as the location information or other unique reference. 

Websites can sometimes be found in the reference and if the website is the same, it is a strong signal that it’s the same person or business. The one thing to remember here is to make sure it’s not a local chain business like McDonalds where you must identify the location addresses are the same. 

It’s important to note that even if some website URLs are not the same, they can take you to the same page, or the link can redirect you to the same website. This is why it is important to click on each link to see where they take you. 

Regardless of whether they match or not, you should click on all website links to check and see if they are working and are valid links to the website. 

Bio Descriptions can sometimes tell you a page’s intent or purpose, and if those same descriptors match with location or numbers, then you are usually looking at a match.

Business addresses and phone numbers are a crucial thing to check to make sure they are the same. It is important to check and see where the addresses appear, and if they are showing the same pin or address on a map, chances are you’ve found a duplicate. 

Check the phone numbers in each candidate to see if they match the reference.

An Exact Match will be if the address or the phone number EXACTLY match on both the search results and the reference. 

It’s quite obvious at this point that if the addresses or phone number are NOT the same then you’re looking at No Match. 

It is important to understand that there may be more than one business in a city…like say a coffee shop. Therefore, you must do more investigating to see which location is which. Many times these will be confused as the same business simply because they are in the same city. Verify exact locations and phone numbers along with addresses to see if you have a match or not. 

One of the important things to mention is Business Presence which is the representation of an entity on Facebook. It is important to understand the business presence of the entity because it can be a strong indicator of whether or not a location is a duplicate. Therefore, you can use the following ways to think of a business presence. 

  • Check to see if it’s an actual place and if it has the same location and offers the same type of products and services?
  • Do both accounts serve the same purpose?
  • If you were to merge together these two accounts, would the content from one integrate well with the content from the other page?
  • Is the content the same on both accounts
  • Who is the intended audience?

There are also some other guidelines in identifying whether or not two accounts share the same business presence. For example

Brand pages that promote the same brand without a location specified. 

Businesses promoting the same brand with the same location. 

An artist or photographer promoting art or photos on the same account.

Sometimes a business owner can promote the same business on more than one account. 

We’d like to mention that if you do happen to get on this project, you will be given the guidelines, which you want to read over very carefully, and you will also be given supplemental guidelines which show some of the common mistakes that new raters run into.

And while we’re on the subject, you might find yourself wanting to apply for this position. It is important that before you do that you understand the job and the requirements and go over all the guidelines several times. That’s where we come in. We can help you prepare for any type of exam or job that we feature and review. Just send me an email or leave a comment and we’ll help you in any way possible. 

And as always, if you are working on this project, we’d like to hear from you about your experience so far. 

Wishing you luck in your work at home endeavors!

Point of Interest Project (POI Mapping Project)

Point of Interest Project (POI Mapping Project)

This is a review of a new project out there called Point Of Interest, or for the sake of simplicity, we’ll call it the POI Project. 

As with all projects, this one is different in the kinds of tasks being performed. The main goal is to evaluate the quality of POI data as a whole, and we’ll explain what that means later on.

The term POI generally refers to any entity that you would find in the Yellow Pages directory like a mall, a movie theatre, or some kind of department store. These are places that users want to find on a map and potentially navigate to it, so you’ll have to check and make sure that they are correct and accurate as possible. We probably don’t need to go over ALL the examples of POIs but it can be anything from a business to a farmers market. 

However, there can also be a POI in a POI such as a department store in a mall. Examples of some other Parent POIs include, hospitals, universities, parks and amusement parks, hotels, golf courses, beaches, and even boardwalks can be a point of interest. 

Each point of interest is associated with additional information that helps a user locate or contact a point of interest. This is called the POI listing information and what a user sees on a POI info card when interacting with the POI on the map. The accuracy of the listing information must be verified and we will go over how you would do this.

Each POI has the additional information that you’re going to need to accurately verify the listing including:

• Name

• Category

• Address

• Phone number

• Hours

• Official URL/website

• Pin location

• Entry points

When doing an evaluation, you are presented with three different types of questions. 

POI Validity – This is pretty straightforward and asks: Does the POI represent an actual entity? So if you are shown a place on a map where an entity should be, but there is nothing there, you would answer NO to this question. However, chances are that most of the time you will answer YES to this question. 

Data Accuracy – Is the POI Information Correct? You’ll want to take a look at the info card that we just went over and determine the accuracy of each opponent of the POI listing information. If the POI doesn’t meet the requirements to be considered ‘valid’ then you do not need to verify the listing. 

Comments – Are there any issues with the POI?: This is not really a question but you are presented with a comments box that must be filled out whenever conflicting information for a listing is found. Your comments should include links to any websites that you used to verify this information or other information that will be helpful for future visits for that POI. 

And that is where some good research skills are going to come in handy. You are EXPECTED to do online research to verify the validity of the POI and the listing information. You can do this by using the official web pages, social media, street imagery, and postal authorities.

If you cannot find official resources then you should see if there are other reliable sources such as newspaper sites or crowdsourced user review sites. Use your search engine skills to locate official resources and try to find some local knowledge when relevant to understand POI details. 

The rating interface is not bad looking and you shouldn’t have any problem quickly marking what you need to using the drop-down menus and selecting the appropriate result. 

Most of the time you will be responding Yes to this question unless the entity doesn’t appear at all on the map. If you do answer Yes then additional Data Accuracy Questions will appear. 

If for some reason the Address Accuracy Question is rated as ‘Incorrect’ then you will see a checkbox list of address components appear and it will look like this:

We should point out that the POIs evaluation requires research to verify the POIs name, category, address, phone number, hours of operation, the official website location including the latitude and longitude (which is displayed as a PIN on the map) and entry points to the location or entity.

We’ve already gone over what types of resources you should use to verify this information, but we will quickly go over it one more time. Make sure you perform a regular web search of the POI by the name and the town, city, or village to see if an official resource, such as a website, can be found. Only by doing this can you judge if a POI is valid or not. 

Official websites will definitely be the first place to start and here we have provided a few examples.

There can be other official resources as well even if the entity doesn’t have its own website. Make sure to check social media sites that are claimed by the business and have some updates within the last six months. Street imagery for the business showing streets or buildings where the entity is. Remember, you are trying to see if this is a real place that still exists or not. 

Postal authorities can also help with postal codes or locality names if the entity exists. 

If the POI lacks an official website, then officially managed social media pages can also be used to validate the information. 

However, when doing this job, there are some online research exceptions. These include EV Chargers such as PlugShare or Alternative Fuels Data Center, Bike Share POIs which use at least three different street images, bus stops which may or may not be obvious, and in-store kiosks. 

POI Validity

There are times when you will not be able to validate a POI listing. These could be that the entity has closed down, it could operate from a residence and be registered to a residential address. There are also Seasonal POIs such as a farmers market or a seasonal amusement park. There are many examples of these in the official guidelines which should be kept handy at all times while validating. Here we provide a few examples to give you an idea.

If a POI has closed or shut down, it might be really hard to identify but there are some things you can do. Chain POIs that have different branches will usually list those locations on their official site and you can also check social media to see if people are saying the entity closed down. Check the street imagery of where the entity should be. If you don’t see it, you’ll mark No – Closed. 

There are some other exceptions when it comes to research on this that can be found in the guidelines including tons of exhaustive examples of POI Types and how they should be rated including an explanation of why it was rated the way it was. 

As we always say with every project we review, make sure you read the guidelines over and over until you are certain that you understand what you are doing and how to do it. 

Name Accuracy

Obviously, it’s important that the name of the entity is correct and straightforward. If you see any slight alteration to the name, then it will be rated as partially correct. 

There are three different options in rating the name as follows:


This is easy to understand. Simply look at the URL of the website and see if it’s correct or not. If it’s not, then you will rate it depending on which option fits it best. 

The guidelines have many exhaustive examples when it comes to URL Rating examples and shows you how to decide if the URL is correct, or if it’s not. 

Address Accuracy

Every entity should have an address associated with it so that users can navigate to the business or entity. There are two different address types: Formatted address for displaying to users and the Structured Address is the ‘raw’ version of the address used in the database. 

POIs within POIs

We touched on this a little bit at the beginning of our review. These are businesses that are inside of a parent entity such as a mall, hospitals, universities, train stations, and department stores. Keep in mind that a phone number of a POI within a POI that is listed on the info card must meet two things before it can be called ‘Correct’. Check the guidelines for more information on this. 


POI hours need to be verified using official resources only and you’ll see the following Hours Rating Options.

Remember that there will also be hours that are different for the child and parent entity. For example, a mall may stay open while other shops inside close early. You’ll need to make sure that the hours listed are the child entity for which you are researching and not the parent entity. 

Some POIs also have seasonal hours and the guidelines go over all these with a lot of examples. 


Categories for POIs got to be correct as well and you need to determine if the category is accurate and the best fit available in the category list. Some examples of category ratings are as follows:

There are many different examples in the guidelines to help you decide which category the entity should be rated at.

 If you have ever looked at a Google Map for an entity, then you know that there is a Pin that is supposed to mark that location with a certain accuracy. So for each POI, you would check to see if the pin is exactly over the POI or if it’s not accurate and placed somewhere else. The following are the Pin Rating Options:

There are literally 11 or so pages on this topic alone in the guidelines as there can be many different options or exceptions when it comes to checking the accuracy of the pin. 

Entry Point Accuracy

It’s quite obvious what an entry point is, this is when you would be told by Google Maps that you have arrived. That said, there is more than just one entry point including Vehicular Entry Points, Pedestrian Drop Off Zones, and Pedestrian Entry Points such as the front doors to a large mall or hospital. The following are the Entry Point Rating Options

Again, the guidelines give some exhaustive examples of this and how to rate the entity including when there are no entry points available at all including a helpful flowchart that helps to make a decision on Entry Points. 


The last thing is the comments. Comments are incredibly important because here you will be able to list all the issues you might have had in researching the entity. It is important to understand that comments are MANDATORY anytime you find conflicting information and it needs to be corrected. Some of those issues might be as follows:

Make sure that when you comment that you refer to the 150-page guidelines to help you understand how to reference those issues that might arise. Make sure that you are concise, and skip being too elaborate stating only the problem. Also, English is the only language that comments can be written in. 

We hope you found this information useful, and we would like to hear from anyone who is working on this project. As always, if you need help passing an exam, have some questions about one of these work at home opportunities, or if you need any help at all, please leave a comment or email us and we will reply as soon as possible. 

Appen Connect Slickrock – Rating Personalized Recommended Stories

Slickrock – Rating Personalized Recommended Stories

Welcome to this informational review of the Slickrock which is a project that focuses on Facebook Stories and rating these personalized recommended stories. 

Stories are defined as temporary content that is created by a page or a user. Stories on Facebook automatically disappear after 24 hours and can only be viewed during this time and it disappears after this. The important thing to know is that a story can have several pieces of content and include multiple photos and videos. 

Stories can be found in the ‘Stories Tray’ which is located at the top of the newsfeed for those who are not familiar with how to view them and what you’ll be looking at if you decide to try for a job as a rater in this project. 

The reason that human data collection is needed for the Stories Recommendations product is so that some users will see stories from people they don’t follow, but may be interesting and enjoyable. 

Feedback from raters is used to determine whether stories that are recommended to some users are actually relevant to them. Each rater is given a set of personalized recommended stories that Facebook thinks will be interesting to the user. You would then rate the relevance on each story for that particular user. 

Fecebook Rating Tool and the rating process

There is an online user interface that each rater logs into that is provided by Workplace by Facebook so the tools are directly made by them. You simply log into the system and then choose either Assign 20 jobs or Just Go which will do the same thing and give you random stories to rate. These will give YOU stories to rate as how you feel about them. More on that later…

 Once you are logged in this is what you would see when you begin rating:

When you see a story that has multiple pieces of content, you will need to answer some questions based on what you see in the stories. 

So now you have you a sneak peek at what the interface looks like, we’ll tell you a little about how you would rate recommended stories. As we said previously, these are stories that are presented to you that are likely based on your likes and preferences including your interactions on Facebook. The goal of the project is to help FB understand as best as possible how you felt about each individual story that was recommended to you. 

You need to ask yourself the following questions:

Question 1: Did the recommended story make you feel uncomfortable or was it unacceptable? Some of those could be from these examples below:

Nudity/partial nudity
Political views
Feeling spammy or inauthentic like an ad
Being in a language that you are not familiar with

In these instances you would click the ‘Yes’ option and then explain in your own words why the content made you feel the way it did. 

There will come a time when you will see stories that are based on public figures and you will be asked how familiar you are with them and asked if you recognize them in the thumbnail or the story itself. There are different levels of familiarity which range from Very Familiar where you know a public figure reasonably well such as a performer like Lady Gaga, or someone else equally popular, and you are interested in them and their activities. 

Mildly Familiar is when you have heard of the person but are not deeply into their activities or what they do such as knowing the name of a sports player like Lebron James but not knowing what sport they play. 

And at last there is the Not Familiar and you have no idea who the public figure is and you have never seen them before. 

Therefore, knowing these you would select one of these three options depending on how well you know the public figure. 

The third question asks if the author of this recommended story is from your area of interest. It is perfectly fine if you don’t know the person as long as they fall under your areas of interest. If you do not recognize the author, then you would go to their page and look to see if what they are about is relevant to you or not. If the author falls under your area of interest, then you select YES, but if not then NO.

The fourth question wants to know if the content of the story is interesting to you or not. Don’t forget that a story can contain more than one piece of content or video. It is VERY important to understand that you need to define the content interesting to you even if there is ONLY ONE piece of content in the recommended story. If just ONE story is interesting to you then you would mark yes. However, if none of the recommended stories interest you, then you should select no, and then you need to explain in your own words why none of the stories caught your interest. It is important to explain as simply as possible, and give as much useful information as possible and not answers like ‘I just didn’t like it.’. There is an option to Skip or Reroute but these should never be used unless there is a problem with the content not rendering. 

Once you are done answering all four questions, you would then click on ‘Submit Decision’ which will then take you to the next story. Each session to rate is 30 minutes long. 

If you are working on this project, we’d like to hear from you in the comments. If you have any questions on this project, or if you need help passing any exams related to getting any jobs we have listed here, leave a comment or email me and we’ll get back to you as soon as possible!

Appen Connect Tahoe Event Realness 3.0 Facebook Project Review

Tahoe Event Realness 3.0 Facebook Project Review

There is a Facebook project called Tahoe that uses a team that checks events that come up in Facebook results. The main mission of this program is to bring people together through real-world activities by showing real events. This is done by working with tools and resources that allow users to discover interesting real events that they might be interested in attending. 


To qualify for this project, you must take a quiz that has 60 questions. 53 of the questions are practical, and 7 are theoretical. The exam looks like it might be difficult to pass, and you must have a score of 90% in order to pass it. Be sure that you follow the guidelines and the instructions in order to make sure you are giving the correct ratings. If you fail the exam, it can be retaken. However, if you fail the exam the second time, chances are they will not give you another opportunity. Get Expert Personalized Assistance with Any Project Qualification

Tahoe project rating process and logic

Improving the discovery of events is done by making a distinction between real events that are happening in the real world at a specific time, and Non-Real Events which are obviously the opposite and not happening in the real world. The job of the individual working on this team is to label events as either Real or Non-Real events by reading the event and basic information. If needed, the Facebook page is opened, and the page is rated either a real or non-real event. This data will be used to help train an algorithm to be more accurate when it comes to events that are shown on Facebook. Non-real events are not removed from Facebook, so this work is important to help people find real events that are happening in their area. 

So what is a real event? 

When looking at an event, the main question is to ask if the event is an activity in the real world, and if it is, is there sufficient information for the user to attend the event like date and time, and directions or a phone number to reach someone. Chances are if it has all this information on an event, then it’s a real event and can be tagged as such. 

The location for the real event should have the city, state, or a venue where the event is listed and usually found in the description box. 

Time is important of course and there are some rules as far as the time is concerned.

Events that last up to 3 days from start to end, such as weekend festivals, are acceptable. Events that have a duration of more than 4 days are only considered an event if there is enough activity within each day to justify having one event created for the entire duration such as a retreat event. 

When looking at these event ads, it is important to watch for phrases such as “time to be determined at a later date” which tells you that the event creator doesn’t have a set date, therefore the event should be tagged as Non-real. 

Reoccurring events and long-term exhibitions need to have event pages for each day those events are going on. Some examples of this are a movie theatre that has midnight showings every Friday. This needs an event page for every Friday. An art exhibit that is being displayed at a studio for a month needs an event page for every day it is being displayed. This also goes for several performances on the same day. Each performance is considered a separate event and should have a new page. 

The activity of the event doesn’t need to be explicitly stated in the description. Keep in mind that the main intent of the event post does not need to be about the activity itself, such as a page selling tickets to an event. This is still considered a real event because its selling tickets to a real event. 

The key here is to watch out for events that closely resemble an advertisement rather than a real activity for the user. This includes things like promoting sales, tour packages, causes, etc. These are not considered real events. 

What is considered a Non-Real event? 

Depending on the location, many events will be encountered that initially seem like they are real events, but in reality, they do not satisfy the event’s realness requirements. When evaluating these types of posts, remember to always use the main frame of thinking as with every post: Is this an event I can attend, and if it is, is there enough information for me to attend the event?

Below are some examples of some common Non-events that will be encountered. 

Promotional events make up the majority of Non-Real cases that the rater will encounter during this experiment. Most of the time these events will more closely resemble advertisements more than actual activities or events such as a company using the event creation tool to advertise a discount or a sale. Travel agencies creating an event for the purpose of selling some tour package. However, if it is an experience type of event or an educational event such as an ecological walk, even if its called a tour, if its still a real event that holds interest for locals then it is considered real. One key thing to note is that these posts are usually made by non-profit organizations, government agencies, or educators and are a good indicator that they are real. 

Watch out for insufficient times such as an event that has a start and end time that doesn’t align with the actual activity for the event, although these events are very rare. 

Time details to be determined is another factor in determining if something is Non-real. If there is no strict date, then its nothing you can plan for, and therefore not a real event. 

If you see a post that has a schedule of events, it needs to be tagged as non-real as every event should have a single post of its own. 

Call to action events are created for the purpose of receiving donations, votes, or any other type of contribution from a user without the need for in-person activity. With that said, if the event is a fundraiser that occurs at a real-world location, this would be acceptable as Real. 

If the event post is gathering support to see a certain performer or artist in a specific city or country, then the post should be considered non-real because there is no confirmed real-world event. 

Online events are considered Non-Real because they do not happen in the real world and that includes live streams, radio, and TV shows. Watch out for memes, just for fun, and spam that pretends to show real events happening in the world, but upon inspection, you realize that it’s evident that it can’t be real such as bands performing in unrealistic situations, or events that are centered around popular internet memes. 

Another thing that event creators like to do is use the event creation tool to announce a large-scale celebration or holiday. If it’s just some announcement that doesn’t lead to a specific activity, then it gets labeled Non-real. 

Below are some more examples:

Jobs should be rejected if you do not understand the language that the post is in. Any post with solicitation of Child Exploitation or Child Nudity should be escalated immediately with the JOB ID to the manager. If you are uncomfortable reviewing the content, then the job can be skipped after it is escalated. Canceled events are still marked real as long as they fulfill the Real event requirements. 

The language of the experiment should be present in the event title or description. If these are missing then you can immediately reject the job. Make sure that time details listed for an event make sense for the actual activity, and also make sure that the city and state, or venue name is listed somewhere in the event info. 

If you work on this project or know someone that does, we would like to hear from you in the comments about your experience on this project. If you have any more questions, then shoot me an email or just leave a comment and I will answer as quickly as possible. 

Sykes Search Quality Evaluator – New Vendor For Google Search Engine Evaluation Project (US only)

Sykes Search Quality Evaluator

There’s a new job position that is available through Sykes for a Search Quality Evaluator working from home part-time.

It isn’t known exactly what vendor this job could be for since Sykes is an outsourcing provider, but it could be Sykes, Appen or even Lionbridge. Another thing to note is that this job was posted on December 21, 2019 and is still listed. So, if you are looking for a job as a Search Quality Evaluator (known as Google Search Engine Evaluation project: Yukon, Internet Assessor, Rater) this could be your big work at home opportunity.

Judging from the job description about searching for online topics, and how frustrating it can be to get bad results, this is most certainly the same type of job as Search Engine Evaluator and you would be responsible for performing simple web searches on specific topics and then rating the results that are returned within the time frame they give you. A Windows PC is required for this work at home job, and no MACS are being accepted in this project. You must also have an activated smartphone with Wi-Fi capability. Applicants need to have strong computer skills or they will not be considered.  

The time you want to work on this project is flexible and you can work on your own time as you wish. There is a schedule provided after training, although it is uncertain what this training might be since there is no previous training for these types of jobs. Only the guide is supplied to you and you must learn on your own. After this training period you must work a minimum of 20 hours per week. 

One thing to keep in mind for this project is that you would be reviewing offensive and mature content. If this type of stuff offends you, then you may want to skip this one. 

Also, if you live in these states, Sykes is unable to hire you: Alaska, Arizona, California, Colorado, District of Columbia, Hawaii, Illinois, Iowa, Maine, Massachusetts, Montana, New Hampshire, New Mexico, New York, North Dakota, Ohio, Oregon, Rhode Island, South Dakota, Vermont, Washington and West Virginia.

You may apply for Search Quality Evaluator role here:

There is no mention of what the pay is at this time. If you happen to be working on this program, we’d like to hear from you about the training procedure and what the rate of pay might be. As always, if you have questions about this project, send me an email or leave a comment and I will reply as quickly as possible. 

Appen and Lionbridge Projects – Hourly Rates Across Different Markets

Hourly rates across different countries: Lionbridge, Appen

Important Note: Some Appen Global projects are not currently available in many states in the U.S.A. because the payment rate Appen Global offers is below the minimum wage in some of these areas. The company Sykes is also experiencing this same problem. Currently, the only alternative is Lionbridge. Our sources show that in many countries Appen Global pays 2 times less than Lionbridge, and to maximize profit, they cut wages. For example: In India Lionbridge pays $7 per hour, Appen $3 dollars an hour compared to the U.S.A. where Appen Pays $10 per hour and Lionbridge Pays $14.


Echo, Index, Sharp, Cable 12$-13,5$ 14$ 7$
Arrow (Google) 10$ 14$ was 8$/now 5$ was 8$/now 5$ 15$-17$ was 8$/now 5$
Truckee (Insta) was 13,5$/now 10$ 14$ was 9,45$/now 7$ was 9,45$/now 7$ was 8$/now 5$
Nile (Facebook) was 13,5$/now 10$ 14$ was 9,45$/now 7$ was 9,45$/now 7$ 15$
Shasta per task per task per task per task per task per task
Yukon (Google) was 12,5$ (net)/now 10$ (net) 14$ was 10,5$/now 5$ was 10,5$/now 7$ 14$ was 8$/now 5$
Internet Assessor, Rater, Internet Safety Evaluator (Google) 14,5$ (gross)
14$ 10$ 12$ 7,3$
Ads Assessor (Google) 13,5$ 14$ 10$ 12$ 7,3$
Maps Analyst (Apple) per task per task
Why Appen decreased hourly rates almost in all markets after acquiring Leapforce?

Feel free to share your information regarding payment rates in your specific country, based on your input I will update information on this post.

Internet Safety Evaluator Project For Lionbridge Review

Internet Safety Evaluator project review

This project is about a year old and was created and started when Google decided to exclude some YouTube video evaluation tasks from its main search engine evaluation projects: You are probably familiar with these projects: Yukon (Appen/Raterlabs) and Internet Assessor/Rater at Lionbridge. The Internet Safety Evaluator project picks up where search engine evaluation leaves off and the job has you looking at video search results to improve the search result quality for content. As an Internet Safety Evaluator, you are also required to provide feedback and analysis on video content found in the search results and provide ratings on their relevance with the search terms that were used. 

So where does the Internet Safety Evaluator part come in? Well, that’s the most important part and aspect of this role. While reviewing content, you will flag potential upsetting or offensive material to make the internet safer and more secure for all users. You will be able to express your opinions in comments about the video concerning the content and the quality of content, and at the same time, you will be protecting users from viewing unsuitable material. 

So what do you need to have to qualify for the position of Internet Safety Evaluator? Appen and Lionbridge are looking for dynamic and creative people who have a strong interest in improving online search results. One of the main things that you should feel comfortable about is working on sensitive or adult content. Candidates must use a smartphone for this position, and either an Android or iPhone device is suitable. The Internet Safety Evaluator should also have some familiarity with social media and Google products. Be prepared to be flexible concerning working hours and you should have the ability to understand and follow the established guidelines. 

The Internet Safety Evaluator position only allows you to have 20-hours max work time per week. 

The qualification process is identical to the Appen/Yukon Project or the Internet Assessor for Lionbridge. Read more about how to pass that exam here:

Also, don’t forget to check out the really cool tool that we’ve created for helping to pass these tough exams. See Our Exam Simulation Tool Here:

Rating Process And Quality Metrics

As an Internet Safety Rater working on the project, you will have access to a variety of different task types. This is important because it helps you become more familiar with the core concepts of rating and it will help you build experience and knowledge concerning the program. During the beginning, the number of tasks you will receive will be low until you gain enough experience in the program. Once you feel that you have a good understanding of the program and become familiar with all the task types, you will start to see a switch in the task types that you receive. But keep in mind, this will only happen if the managers feel you have a strong grasp on the concept of what you are doing. From that point forward, you will only be required to focus on evaluating more video type tasks from an Internet Safety Rater perspective. You will also see a higher number of tasks given to you. 

As an Internet Safety Rater, you cannot work more than 1 hour per day and this limitation can last several months. The purpose of this limitation is to evaluate the overall quality of ratings that you are providing. During this time you will be given standard Yukon/Internet Assessor tasks (Needs Met and Side by Side and other types of tasks), additionally, you will be evaluated on your quality by means of standard Needs Met and Side by Side quality review tasks. 

If you want to know more about this process, read here:

The quality requirements are high and you need to achieve and keep a quality level higher than 65% or above. This is the very minimum and work is monitored on a regular basis. Feedback will be shared with you so that you can learn from your mistakes. If an Internet Safety Evaluator constantly performs below that minimum quality level or performs significantly lower than the minimum level, that person will be removed from the program. This is why it is very important to use the months long limitations and make sure to learn as much as possible. 

The things closely monitored to evaluation an Internet Safety Evaluator performance is the Needs Met, Page Quality (E.A.T.) and SXS rating. However, other things such as rating accuracy and the quality of the comments that you leave are another thing that is closely looked at. 

Once the Internet Safety Evaluator has shown a stable level of quality, the probation period ends and you will get a notification that the Internet Safety Evaluation transition period ended and all restrictions are then lifted and you see a higher volume of tasks. This is when you will start receiving ONLY video evaluation tasks and you’ll get a higher volume of them. There will be no more review tasks after the completion of the transition period. 

Internet Safety Evaluator probation and transition period

The Internet Safety Evaluator probation and transition period length all depends on the quality performance of the candidate who should demonstrate positive quality review scores. This period usually takes about 3-4 months but could possibly last up to 8 months depending on the performance of the individual. 

In terms of the complexity of this project and how easy it is to participate, the Internet Safety Evaluator project seems to be easier in particular after the transition period has ended. Working on video tasks is a lot easier than doing the Needs Met and Side By Side tasks. However, the big drawback to the project is online being able to work 1-2 hours per day during the transition period. 

You can work on other projects at the same time to make up for these small hours. You can provide similar rating services to other providers. However, due to project restrictions and customer requirements, you cannot work on Appen Yukon and the Lionbridge Internet Assessor at the same time due to an IP address conflict. 

If you want more information on other unrelated projects, you can find that information here:

If you have any questions about the Internet Safety Evaluator position or any other questions, let me know in the comments or simply send me an email and I will reply. If you are working as an Internet Safety Evaluator we’d like to hear about your experience. 

Preferred Raters Program For Lionbridge/Raterlabs

Preferred Raters Program

If you’ve landed here, you might have heard about the Preferred Raters Program…and It looks like there is finally some great benefits coming out for Lionbridge/Raterlabs for raters who achieve high scores and have consistently high-quality ratings and comments for all task types. This is for raters who work on the Yukon/Raterlabs project. Through this program, agents are identified and rewarded based on high-quality review scores and high-quality work they do on the Yukon project. 

It sounds really great, but it is highly selective and prestigious, and only available to a limited amount of raters. The requirements to get into this program are very high, as well as inclusion and retention…meaning it will be tough to get in and also stay there. Participants are held to strict standards of quality and professionalism and the quality team reviews candidates quarterly to determine if they qualify for the program for admission or if you’re already in the program if you stay or someone comes along and takes your place. 

So, what about the rewards and are they really worth it? 

If you do get into the Preferred Raters Program, you’ll get a boost in pay rate. You’ll go from $10 to $15.00 dollars for Raterlabs and from $14 to $18 at Lionbridge. Not a bad little pay raise at all even if it is temporary. 

As another added benefit, your username will be in green in chatrooms showing that you’re a highly skilled and very experienced rater. 

How Do I Qualify? 

As we mentioned earlier, raters are regularly reviewed to see who meets the requirements of the Preferred Agent Program. 

Note: Sending them emails might be the quickest way to NOT get into the program. It is not recommended to email them about your status in the program. 

Only if you have met the program’s requirements will you hear from the project’s Quality Team. If you don’t hear from them, then it may mean that you need to step up your quality or performance, or there just may be too many candidates and not enough room to fill them all. We do believe it might be the opposite since requirements are so high for the program. 

The most important thing to focus on is quality review scores for all task types. Having high scores in this area will get you looked at for eligibility for the Preferred Raters Program. But it’s not all as there are several other additional criteria used to determine an agent’s final acceptance into the program and we’ll go over all the things they use to select someone. 

Primary Selection Criteria

The very first thing to point out is only agents who have worked on the Yukon project for at least one year are eligible. 

And agent needs to complete enough of the following tasks to receive the minimum Quality Review feedback:

Needs Met: 6 sets of feedback

Side by Side: 3 sets of feedback

Page Quality: 2 sets of feedback

The quality reviews all have to be extremely positive for all the task types, not just one, and high adherence and demonstrated comprehension of the principles detailed in all the guidelines and project instructions. 

Comments are also a part of this and Preferred Raters will show excellent commenting skills for all tasks. 

An agent will have a consistent history of good interactions with the community that should always be positive. This includes all areas of communication with admins in chat, social posts, and emails. 

While we’re on the subject, we shouldn’t have to tell you that candidates that have a history of unprofessional, nasty, or obnoxious behavior in chat or other social channels won’t be considered for this program. 

Agents in the program will always meet the minimum hours’ requirement and complete all the tasks in a responsible manner and not releasing them in an excessive or malicious way because maybe the agent doesn’t want to rate the produced results. 

Did we mention that it’s hard to get into the Preferred Raters Program? We should mention it again and also state that it’s not an easy thing to achieve or maintain. But if you give hard work, perseverance, and show a very thorough understanding of the concepts from the General Guidelines, then chances are good that you’ll make it as a candidate. 

I’ve made it as a candidate, now how do I keep my status?

We mentioned that the Preferred Raters Program is considered prestigious, highly selective, and it is limited to a small number of raters. That just means it’s going to be a lot more difficult for you to get in and stay there. But after you are in, a rater must maintain the high-quality standards across all task types and have high-quality reviews after acceptance. 

If a rater fails to maintain high standards of quality, then they are put into a probational period and given the opportunity to improve. That’s good news since if you are in this program, you’ve worked hard to get where you are. However, if an agent continues to fail meeting the program criteria, they will lose their preferred status and the rewards associated. Also, once you are in the program you can be removed at any time at the Quality Teams’ discretion even without a probational period. 

So what if you are put on probation?

If a rater fails to meet requirements they are placed on probation. This probation is actually two more reviews and if your reviews are bad, then you’re out. One other thing:  If you are placed on probation and receive a substandard or unacceptable review score in ANY review type during your probationary period, your Preferred Agent status will be immediately removed.

So what triggers probation? 

When an agent gets three bad scores within the most recent 6-month period below the Preferred Agent standard, this will trigger a probationary period. Also, if the majority of your recent scores are all bad and not meeting the high standard, you are placed on probation. The bottom line is that your scores should be primarily Exceeds Expectations or higher. Meets expectations do not cut the cake. While they are respectable, they are not high enough scores to be in the program. Multiple scores of Meets Expectations can result in the loss of your Preferred Agent status. 

Another thing that should be mentioned is that two Substandard and Unacceptable review scores within a recent year’s time period will trigger a probation period. Any additional scores of the same type will result in the immediate removal from the program. Preferred Raters cannot have such low scores. 

The most important requirement of a Preferred Agent is that they consistently provide exceptionally high-quality work and if your scores fluctuate, it might suggest to the Quality Team that you are unable to maintain the high standards of the program, and you will be placed on probation. While it is a disheartening thing to be placed on probation, this does mean that you are being given the chance to improve your quality of rating. Remember that Meets Expectations is not good enough. 

Some other things that will provoke the Quality Team are poor commenting habits, excessive releasing of tasks, and complaints about unprofessional behavior in chat or other social channels could lead to probation or it could just see you leaving the program altogether. Remember that we said they have total discretion and can release you from the program at any time for any reason. 

If you are put on probation then it’s lucky you and you’ll have to use the opportunity to correct the problems and retain your Preferred Agent status. 

As we touched on a little earlier, Quality Teams review agents on a quarterly basis and if there is room to promote new agents, they will do so. If the program is full and all those raters are keeping their high standards up, then there won’t be any new raters accepted into the program. When a rater is removed from a program not meeting high-quality standards, then another rater who is consistently doing this is taken into the Preferred Agent program. 

You might be wondering if one low score will put you on probation. The answer is no and the Quality Agents recognize that sometimes bad things happen. However, if you continue to get low scores, then you will be put on probation until you fix them and they only give you two reviews after this and you must score high on BOTH reviews, not just one. 

Just so we are sure we covered this, while you are in the program, three low scores will get you noticed by the system which will trigger a probation period. 

A rater must meet high-quality standards in all areas and all tasks, not just one area. Because the program is considered prestigious, it is limited to a certain amount of raters. That said, if you are in the program and have some problem in one area, it will show and you will likely end up on probation. 

You can earn the status back if you lose it by demonstrating that the quality of your work meets the program’s qualification standards on a consistent basis. Keep in mind that just because a rater has been in the program before, preference will not be given over them when determining new rater eligibility. There are only two times you can be put on probation meaning that the third time and you are out of the program. 

We hope that we were able to answer your questions about the Preferred Agent program. Are you in this program? Tell us your experience in the comments and if you need help you can always reach us by email. 

Appen Connect Wouri Project Review And Information

Wouri project review

There’s another project to work on over at Appen called the Appen Connect Wouri Project. We’ll go over the project a little and briefly touch on the main points, what type of job it is and what requirements there are. We will also touch on the grading process and give a few examples. 

The goal of the Appen Connect Wouri Project is to understand the user’s intention as represented by the query, and then determine how good the result is and if it satisfies the user’s needs. Users are using Safari and Spotlight to search for different information needs. Some examples include wanting to find out knowledge or information on something or somebody, wanting to know if there is a certain store nearby, the latest news related to a topic, or since they are MAC users, getting an application on iTunes. This project is a bit like the average search engine evaluator, but it does have a different grading system. However, it uses some of the same ideas from those programs on search engine results. 

A desktop or laptop computer is recommended to work on the Appen Connect Wouri Project over a mobile phone and all graders should be checking search results with Safari. Safari can be used on Windows operating systems, but it should be mentioned that there is no support from Apple when using Safari on MS Windows. You might be better off with a MAC computer, but it isn’t required. 

To participate in the Appen Connect Wouri Project you have to pass an initial assessment and achieve a minimum score of 80%. If you don’t pass the first time, you’ll get another attempt. After you pass the initial assessment you will be given a final assessment in which you must also pass with a minimum score of 80%. 

Wouri Project rating Logic

When looking at a user query, there are three steps that you will take in the grading process. 

  • The first thing is understanding the query. You should make sure it’s not too vague, inappropriate, or in a foreign language. 
  • The second thing is understanding the result. Determine if it is GEO or Time Sensitive. There is also the option to choose both or choose neither GEO or Time Sensitive. 
  • Once you understand the query and the result, you will then decide on a relevance rating for the result based on the query intent. 

It is important to note that Appen expects local graders working on the Appen Connect Wouri Project to judge the search results of the same locale. What that means is you must in the same location as the project requires. In this case, you may need to configure your computer and your search engine to be in the language of the project that you are working on. 

A web relevant search is one that satisfies the user’s information need. Of course, a search result can be more or less relevant based on how well or how completely it satisfies the users’ needs. It is important, and we cannot stress this enough, that you have the guidelines open and you have thoroughly read them. 

In step number one, you need to determine what it is that the user is searching for and decide if it’s valid. You should read the query, and review the third party results and think about the possible user intent that it represents. This is called ‘Understanding the Query.

For example, the query ‘Samsung’ could have many different user intents such as shopping for Samsung products, getting Samsung technical support, or the user may be looking for the Samsung homepage. However, if the user simply types in Samsung, then the query can be too vague. 

If you have no idea what the user might be looking for, then the next step would be to go to a popular search engine and study the term. Some common terms could have some special meaning. However, if it doesn’t have some special meaning, or it has too many special meanings with each one having a small portion of intent, then you would select Query is Too Vague. 

You might also run into a situation where a query is inappropriate and it encourages illegal behavior, includes profane words, using derogatory language, racial slurs or hate speech. It also includes graphic violence or the query might like to retrieve grisly content. You are not expected to do anything about these types of queries except to mark it as inappropriate.  

When in doubt about any type of query, you should check results from other search engines. It’s absolutely crucial to understand what the user is searching for, and you may need to check results from other search engines. The good thing about this project is that top results from other search engines are provided for you. There is a small link that says ‘Web Search Link’ that will automatically do this for you and show you the top results in order to better help you understand the query. 

It is VITAL to point out that opening the webpages is a MUST when you are working on this project. If the results match the first result of the other search engine, then the result is likely an excellent match. Make sure to check the results that are returned by the Web Search Link. Each result has a clickable URL, a small description, and a photo where applicable. 

We touched a little on GEO and Time Sensitive in the beginning, and since you might not understand what that is, we will go over it to help you understand it better.

If a query is about a store or some location that exists in other locations aside from the searcher, let’s say a Starbucks, or a query like ‘movie theatres near me’ then you know its geo sensitive meaning that it matters where and how close the result to the searcher is. If the searcher is from California and looking for a Starbucks and the first result is the homepage or another location, then it doesn’t satisfy the user’s needs. 

Anything that deals with a query that involves some kind of time frame, such as ‘Buy Superbowl Tickets’, age of Harrison Ford, or checking on the weather are results that should be marked as ‘Time Sensitive’. 

It is important to understand that a query can be both Geo sensitive and Time sensitive, and a good example is a query such as ‘tickets to the local basketball game’, local movie showtimes, etc. 

Once you understand the query and you judge the results, then you will assign a relevant grade for each result. There are 9-grade levels:

● Excellent

● Good

● Fair

● Bad

● Unavailable

● Query In Foreign Language

● Result Page In Foreign Language

● Vague

● Inappropriate

There are many examples in the guidelines that go over difficult queries in order to help you evaluate what results would satisfy the informational need for the searcher. It is a must that you go over the guidelines and read them and understand them so you know how to grade something. 

The guidelines go over when you should choose an excellent result, and the biggest point to keep in mind here is to believe that the result will be everything that the user will need, and they will not have to continue searching. If you believe that the result isn’t good enough, and you would also do an additional search, then you might choose ‘Good’ or ‘Fair’ as the grade. 

A ‘Good Result is one that the user wants to see above the fold on the device screen. It is typically the first 4-5 results. However, it is not the best possible result, which would be excellent, but it still provides good information. The guidelines have many examples of what should be chosen as a ‘Good Result’. 

When choosing a ‘Fair Result’ for the query, the result would only be somewhat relevant, but not be of interest to all users. An example of this would be a query for a celebrity name, but the result is an obscure person that has the same name. 

Choosing  ‘Bad Result’ would pertain to a result that does not satisfy user intent and is something that should not be shown to users at all. An example of this would be something that we previously mentioned before if a user was looking for a Starbucks Coffee and lives in San Francisco, but the map result is for a store in San Diego. There are other examples in the guidelines. 

It should be straight-forward of when you should choose ‘Foreign Language’. If the given query is in a different language from the specified one, or the target language that you are assigned to work, then you would mark this as “Query in Foreign Language”. If you are not sure if the query is in a foreign language, then just search that query to understand the word or words. 

Sometimes in the given search results for a query, the URL will be inaccessible, or website link will not be available or a dead link. If this happens then you would grade it ‘Unavailable’ This also includes landing pages that take longer than 10 seconds to load, or you may see that the domain is up for sale by being sent to a redirection page. This grade also pertains to when video content is unavailable or blocked. 

There are some other specific result types that the guidelines go over including ‘Ambiguous Queries’ or when the interpretation is much more popular than others. For example, the query is ‘Golden Retriever’ but the result is a song titled the same. This should be graded as fair because the dominant interpretation would be for the dog breed itself and not a song title. There are some instanced when multiple interpretations can be as equally popular though. In order to assign something to the excellent category, it’s a good idea to take into consideration what geo-location the seacher is looking from. Example:

Query is “UM football”, and the result is the official website for the University of Miami football team. If user’s location is in San Francisco, grade the result as Good, since UM could refer to either U of Miami or U of

Michigan and we don’t know which is best. But if the user location is in Miami, grade as Excellent.

As always, if you have any questions about the program, please send an email or reply in the comments. If you are working on this program, then we would love to hear about your experience in the comments.