As being a rater you might noticed that starting from March 2020 the volume of tasks in the project significantly decreased, and many raters might be wondering when the situation to be improved or back to normal.
Pandemic affects Google Ads Business
Lionbridge notified raters that the number of tasks might be low or unstable due to slow down of the ads business of Google due to pandemic. As you may know tasks are generated from real ads. The vendor confirmed that this a temporary issue, and the customer expects the volume of tasks to be increased in the future. The good thing is that the number of blind tests was decreased due to this fact and there is a very small chance of being fired.
Tasks are not distributed evenly throughout the year, based on 5 years observation I can confirm that the seasonal factor takes place as well, the low season is spring. The vendor informed raters that the customer usually revises and updates rating algorithms in spring and this process affects task volume in the system.
New task distribution Algorithms
Another point that I wanted to clarify relates to the number of tasks that the raters receive, the number of tasks given is not the same for everyone during the month, and some raters can receive more tasks than others. This is due to the task’s distribution algorithms. Usually in the following month, the unevenness is compensated by an increase in the number of tasks for those raters who received fewer tasks in the current month.
Tips for getting more tasks
Logon raterhub more frequently to check for tasks in particular between 9AM – 3 PM PST
Currently, working less than 20 hours per week will not affect your status in the system.
There’s a new project being offered by Appen for Facebook called Bristol and agents hired for the project will help identify duplicates that come up in search results. This helps improve the user experience for people searching for something on Facebook.
You will also be providing the most relevant results to users along with improving product capabilities.
The one thing Appen stresses is that agents should avoid any personal political or social bias when rating a search result. Facebook has separate teams that judge controversial topics and misinformation. Your job as an agent is to simply judge if something is a duplicate, and choose the one with the best or more up to date information.
The interface is fairly straightforward and easy to understand.
On the top left hand corner you will see the reference details of the Facebook entity, and then below this you will see candidate details. It may be that you see several candidates here.
The process is to review each one of the candidates and see if they are duplicates of each other. The workflow can be executed as a one-to-one or one-to-many relationship structure. Basically…you just check each one to see if they are the same or not.
It can be tricky to know if some places are the same or not because they might not have the same name. You need to be careful in this instance and do enough investigation to see if the places are indeed the same, or if the information in the ad is the same.
You do this by opening each link and comparing the contents to the reference sources. Thankfully, to help sort through some steps, there is a decision tree to help you.
If you do find other things in common between the ‘Reference Source’ and the candidate, then you would select ‘Match’ to indicate that the entities are duplicates of one another. If there aren’t enough commonalities then you would select ‘No Match’ and move on.
The decision tree will help walk you through the steps if you’re not sure what to do.
As you complete the job, you will notice that above the candidates link section there are some questions you will need to answer before moving on through the job.
The first question is ‘Translation Tools’ and asks:
I will need to use translation tools to complete this job
The page link is broken and I can’t complete this job
None of the above, I can complete this job
Notice that if you need translation tools or the page link is broken, you will be unable to complete the job and you should choose either one when it is applicable. However, in most cases you will use the last one ‘None of the above, I can complete this job.
If the candidate link is broken then you would also choose ‘Unable to investigate’ in Question 2
Question 2 is the search result pair review and at this point, if you chose the last option in question 1, you would review the Reference Source and each one of the candidates…which are supposed duplicates.
Question 2 offers you 3 options:
Unable to investigate
There are some attributes in the guide that need to be evaluated when deciding if two entities are the same or not.
It’s really easy to decide if the names of both search results are the same if the names match and are exactly the same. We say EXACTLY because they must be word for word and not missing part of the name which would cause you to further investigate to see if they are indeed the same entity.
There can be what is called a Soft Match such as FB, Inc. for Facebook Inc. where there is some naming connection between the Reference Source and the Candidate.
There is something you will notice called an ‘Alias Handle’ which is preceded by an at sign @ and you’re probably familiar with this if you have an Instagram account. You can also check to see if these names match or not in order to make a determination.
Another thing to mention is ‘Chain Locations’ like McDonalds who will sometimes put the location in the name itself. Pay extra attention to this because if those names differ, then they are not the same entity.
So we are clear on when you should choose ‘No Match’ we will give you the most important information. No match means there is ZERO similarity between the names and the online aliases. Just be careful and check things like the name, alias handle, business name, biography, photos, and posts.
If you do need to check photos, then it is important to review the photos from both entities in search of matching or unique photos in posts made by the entity. Sometimes it takes a little research to see if you can match them, but you shouldn’t spend a lot of time looking to match photos. Checking the entities Instagram is a quick and surefire way to see if photos match the entity or not and should not take more than 30-45 seconds or so.
Profile photos do not need to match, but they should resemble the same business or logo of the entity or person as the reference source.
If it’s not clear and you don’t see matching photos or it’s clear that the photos between the two entities are totally different and not related, or if you don’t have pictures to compare, then consider other signals on the page to check if they are the same.
As we mentioned, if you see two different photos that show a different business and they are not related at all, then you don’t have a solid match. Watch for logos to be entirely different. Some companies out there can have the same name or a name that is similar. Check to see if the logos are the same…if not, then you have no match.
Bio/Descriptions can be another place to look, but if description is the same it doesn’t mean that the two identities are the same. The text needs to contain some unique identifier such as the location information or other unique reference.
Websites can sometimes be found in the reference and if the website is the same, it is a strong signal that it’s the same person or business. The one thing to remember here is to make sure it’s not a local chain business like McDonalds where you must identify the location addresses are the same.
It’s important to note that even if some website URLs are not the same, they can take you to the same page, or the link can redirect you to the same website. This is why it is important to click on each link to see where they take you.
Regardless of whether they match or not, you should click on all website links to check and see if they are working and are valid links to the website.
Bio Descriptions can sometimes tell you a page’s intent or purpose, and if those same descriptors match with location or numbers, then you are usually looking at a match.
Business addresses and phone numbers are a crucial thing to check to make sure they are the same. It is important to check and see where the addresses appear, and if they are showing the same pin or address on a map, chances are you’ve found a duplicate.
Check the phone numbers in each candidate to see if they match the reference.
An Exact Match will be if the address or the phone number EXACTLY match on both the search results and the reference.
It’s quite obvious at this point that if the addresses or phone number are NOT the same then you’re looking at No Match.
It is important to understand that there may be more than one business in a city…like say a coffee shop. Therefore, you must do more investigating to see which location is which. Many times these will be confused as the same business simply because they are in the same city. Verify exact locations and phone numbers along with addresses to see if you have a match or not.
One of the important things to mention is Business Presence which is the representation of an entity on Facebook. It is important to understand the business presence of the entity because it can be a strong indicator of whether or not a location is a duplicate. Therefore, you can use the following ways to think of a business presence.
Check to see if it’s an actual place and if it has the same location and offers the same type of products and services?
Do both accounts serve the same purpose?
If you were to merge together these two accounts, would the content from one integrate well with the content from the other page?
Is the content the same on both accounts
Who is the intended audience?
There are also some other guidelines in identifying whether or not two accounts share the same business presence. For example
Brand pages that promote the same brand without a location specified.
Businesses promoting the same brand with the same location.
An artist or photographer promoting art or photos on the same account.
Sometimes a business owner can promote the same business on more than one account.
We’d like to mention that if you do happen to get on this project, you will be given the guidelines, which you want to read over very carefully, and you will also be given supplemental guidelines which show some of the common mistakes that new raters run into.
And while we’re on the subject, you might find yourself wanting to apply for this position. It is important that before you do that you understand the job and the requirements and go over all the guidelines several times. That’s where we come in. We can help you prepare for any type of exam or job that we feature and review. Just send me an email or leave a comment and we’ll help you in any way possible.
And as always, if you are working on this project, we’d like to hear from you about your experience so far.
This is a review of a new project out there called Point Of Interest, or for the sake of simplicity, we’ll call it the POI Project.
As with all projects, this one is different in the kinds of tasks being performed. The main goal is to evaluate the quality of POI data as a whole, and we’ll explain what that means later on.
The term POI generally refers to any entity that you would find in the Yellow Pages directory like a mall, a movie theatre, or some kind of department store. These are places that users want to find on a map and potentially navigate to it, so you’ll have to check and make sure that they are correct and accurate as possible. We probably don’t need to go over ALL the examples of POIs but it can be anything from a business to a farmers market.
However, there can also be a POI in a POI such as a department store in a mall. Examples of some other Parent POIs include, hospitals, universities, parks and amusement parks, hotels, golf courses, beaches, and even boardwalks can be a point of interest.
Each point of interest is associated with additional information that helps a user locate or contact a point of interest. This is called the POI listing information and what a user sees on a POI info card when interacting with the POI on the map. The accuracy of the listing information must be verified and we will go over how you would do this.
Each POI has the additional information that you’re going to need to accurately verify the listing including:
• Phone number
• Official URL/website
• Pin location
• Entry points
When doing an evaluation, you are presented with three different types of questions.
POI Validity – This is pretty straightforward and asks: Does the POI represent an actual entity? So if you are shown a place on a map where an entity should be, but there is nothing there, you would answer NO to this question. However, chances are that most of the time you will answer YES to this question.
Data Accuracy – Is the POI Information Correct? You’ll want to take a look at the info card that we just went over and determine the accuracy of each opponent of the POI listing information. If the POI doesn’t meet the requirements to be considered ‘valid’ then you do not need to verify the listing.
Comments – Are there any issues with the POI?: This is not really a question but you are presented with a comments box that must be filled out whenever conflicting information for a listing is found. Your comments should include links to any websites that you used to verify this information or other information that will be helpful for future visits for that POI.
And that is where some good research skills are going to come in handy. You are EXPECTED to do online research to verify the validity of the POI and the listing information. You can do this by using the official web pages, social media, street imagery, and postal authorities.
If you cannot find official resources then you should see if there are other reliable sources such as newspaper sites or crowdsourced user review sites. Use your search engine skills to locate official resources and try to find some local knowledge when relevant to understand POI details.
The rating interface is not bad looking and you shouldn’t have any problem quickly marking what you need to using the drop-down menus and selecting the appropriate result.
Most of the time you will be responding Yes to this question unless the entity doesn’t appear at all on the map. If you do answer Yes then additional Data Accuracy Questions will appear.
If for some reason the Address Accuracy Question is rated as ‘Incorrect’ then you will see a checkbox list of address components appear and it will look like this:
We should point out that the POIs evaluation requires research to verify the POIs name, category, address, phone number, hours of operation, the official website location including the latitude and longitude (which is displayed as a PIN on the map) and entry points to the location or entity.
We’ve already gone over what types of resources you should use to verify this information, but we will quickly go over it one more time. Make sure you perform a regular web search of the POI by the name and the town, city, or village to see if an official resource, such as a website, can be found. Only by doing this can you judge if a POI is valid or not.
Official websites will definitely be the first place to start and here we have provided a few examples.
There can be other official resources as well even if the entity doesn’t have its own website. Make sure to check social media sites that are claimed by the business and have some updates within the last six months. Street imagery for the business showing streets or buildings where the entity is. Remember, you are trying to see if this is a real place that still exists or not.
Postal authorities can also help with postal codes or locality names if the entity exists.
If the POI lacks an official website, then officially managed social media pages can also be used to validate the information.
However, when doing this job, there are some online research exceptions. These include EV Chargers such as PlugShare or Alternative Fuels Data Center, Bike Share POIs which use at least three different street images, bus stops which may or may not be obvious, and in-store kiosks.
There are times when you will not be able to validate a POI listing. These could be that the entity has closed down, it could operate from a residence and be registered to a residential address. There are also Seasonal POIs such as a farmers market or a seasonal amusement park. There are many examples of these in the official guidelines which should be kept handy at all times while validating. Here we provide a few examples to give you an idea.
If a POI has closed or shut down, it might be really hard to identify but there are some things you can do. Chain POIs that have different branches will usually list those locations on their official site and you can also check social media to see if people are saying the entity closed down. Check the street imagery of where the entity should be. If you don’t see it, you’ll mark No – Closed.
There are some other exceptions when it comes to research on this that can be found in the guidelines including tons of exhaustive examples of POI Types and how they should be rated including an explanation of why it was rated the way it was.
As we always say with every project we review, make sure you read the guidelines over and over until you are certain that you understand what you are doing and how to do it.
Obviously, it’s important that the name of the entity is correct and straightforward. If you see any slight alteration to the name, then it will be rated as partially correct.
There are three different options in rating the name as follows:
This is easy to understand. Simply look at the URL of the website and see if it’s correct or not. If it’s not, then you will rate it depending on which option fits it best.
The guidelines have many exhaustive examples when it comes to URL Rating examples and shows you how to decide if the URL is correct, or if it’s not.
Every entity should have an address associated with it so that users can navigate to the business or entity. There are two different address types: Formatted address for displaying to users and the Structured Address is the ‘raw’ version of the address used in the database.
POIs within POIs
We touched on this a little bit at the beginning of our review. These are businesses that are inside of a parent entity such as a mall, hospitals, universities, train stations, and department stores. Keep in mind that a phone number of a POI within a POI that is listed on the info card must meet two things before it can be called ‘Correct’. Check the guidelines for more information on this.
POI hours need to be verified using official resources only and you’ll see the following Hours Rating Options.
Remember that there will also be hours that are different for the child and parent entity. For example, a mall may stay open while other shops inside close early. You’ll need to make sure that the hours listed are the child entity for which you are researching and not the parent entity.
Some POIs also have seasonal hours and the guidelines go over all these with a lot of examples.
Categories for POIs got to be correct as well and you need to determine if the category is accurate and the best fit available in the category list. Some examples of category ratings are as follows:
There are many different examples in the guidelines to help you decide which category the entity should be rated at.
If you have ever looked at a Google Map for an entity, then you know that there is a Pin that is supposed to mark that location with a certain accuracy. So for each POI, you would check to see if the pin is exactly over the POI or if it’s not accurate and placed somewhere else. The following are the Pin Rating Options:
There are literally 11 or so pages on this topic alone in the guidelines as there can be many different options or exceptions when it comes to checking the accuracy of the pin.
Entry Point Accuracy
It’s quite obvious what an entry point is, this is when you would be told by Google Maps that you have arrived. That said, there is more than just one entry point including Vehicular Entry Points, Pedestrian Drop Off Zones, and Pedestrian Entry Points such as the front doors to a large mall or hospital. The following are the Entry Point Rating Options
Again, the guidelines give some exhaustive examples of this and how to rate the entity including when there are no entry points available at all including a helpful flowchart that helps to make a decision on Entry Points.
The last thing is the comments. Comments are incredibly important because here you will be able to list all the issues you might have had in researching the entity. It is important to understand that comments are MANDATORY anytime you find conflicting information and it needs to be corrected. Some of those issues might be as follows:
Make sure that when you comment that you refer to the 150-page guidelines to help you understand how to reference those issues that might arise. Make sure that you are concise, and skip being too elaborate stating only the problem. Also, English is the only language that comments can be written in.
We hope you found this information useful, and we would like to hear from anyone who is working on this project. As always, if you need help passing an exam, have some questions about one of these work at home opportunities, or if you need any help at all, please leave a comment or email us and we will reply as soon as possible.
Welcome to this informational review of the Slickrock which is a project that focuses on Facebook Stories and rating these personalized recommended stories.
Stories are defined as temporary content that is created by a page or a user. Stories on Facebook automatically disappear after 24 hours and can only be viewed during this time and it disappears after this. The important thing to know is that a story can have several pieces of content and include multiple photos and videos.
Stories can be found in the ‘Stories Tray’ which is located at the top of the newsfeed for those who are not familiar with how to view them and what you’ll be looking at if you decide to try for a job as a rater in this project.
The reason that human data collection is needed for the Stories Recommendations product is so that some users will see stories from people they don’t follow, but may be interesting and enjoyable.
Feedback from raters is used to determine whether stories that are recommended to some users are actually relevant to them. Each rater is given a set of personalized recommended stories that Facebook thinks will be interesting to the user. You would then rate the relevance on each story for that particular user.
Fecebook Rating Tool and the rating process
There is an online user interface that each rater logs into that is provided by Workplace by Facebook so the tools are directly made by them. You simply log into the system and then choose either Assign 20 jobs or Just Go which will do the same thing and give you random stories to rate. These will give YOU stories to rate as how you feel about them. More on that later…
Once you are logged in this is what you would see when you begin rating:
When you see a story that has multiple pieces of content, you will need to answer some questions based on what you see in the stories.
So now you have you a sneak peek at what the interface looks like, we’ll tell you a little about how you would rate recommended stories. As we said previously, these are stories that are presented to you that are likely based on your likes and preferences including your interactions on Facebook. The goal of the project is to help FB understand as best as possible how you felt about each individual story that was recommended to you.
You need to ask yourself the following questions:
Question 1: Did the recommended story make you feel uncomfortable or was it unacceptable? Some of those could be from these examples below:
Feeling spammy or inauthentic like an ad
Being in a language that you are not familiar with
In these instances you would click the ‘Yes’ option and then explain in your own words why the content made you feel the way it did.
There will come a time when you will see stories that are based on public figures and you will be asked how familiar you are with them and asked if you recognize them in the thumbnail or the story itself. There are different levels of familiarity which range from Very Familiar where you know a public figure reasonably well such as a performer like Lady Gaga, or someone else equally popular, and you are interested in them and their activities.
Mildly Familiar is when you have heard of the person but are not deeply into their activities or what they do such as knowing the name of a sports player like Lebron James but not knowing what sport they play.
And at last there is the Not Familiar and you have no idea who the public figure is and you have never seen them before.
Therefore, knowing these you would select one of these three options depending on how well you know the public figure.
The third question asks if the author of this recommended story is from your area of interest. It is perfectly fine if you don’t know the person as long as they fall under your areas of interest. If you do not recognize the author, then you would go to their page and look to see if what they are about is relevant to you or not. If the author falls under your area of interest, then you select YES, but if not then NO.
The fourth question wants to know if the content of the story is interesting to you or not. Don’t forget that a story can contain more than one piece of content or video. It is VERY important to understand that you need to define the content interesting to you even if there is ONLY ONE piece of content in the recommended story. If just ONE story is interesting to you then you would mark yes. However, if none of the recommended stories interest you, then you should select no, and then you need to explain in your own words why none of the stories caught your interest. It is important to explain as simply as possible, and give as much useful information as possible and not answers like ‘I just didn’t like it.’. There is an option to Skip or Reroute but these should never be used unless there is a problem with the content not rendering.
Once you are done answering all four questions, you would then click on ‘Submit Decision’ which will then take you to the next story. Each session to rate is 30 minutes long.
If you are working on this project, we’d like to hear from you in the comments. If you have any questions on this project, or if you need help passing any exams related to getting any jobs we have listed here, leave a comment or email me and we’ll get back to you as soon as possible!
There is a Facebook project called Tahoe that uses a team that checks events that come up in Facebook results. The main mission of this program is to bring people together through real-world activities by showing real events. This is done by working with tools and resources that allow users to discover interesting real events that they might be interested in attending.
TAHOE PROJECT QUALIFICATION PROCESS
To qualify for this project, you must take a quiz that has 60 questions. 53 of the questions are practical, and 7 are theoretical. The exam looks like it might be difficult to pass, and you must have a score of 90% in order to pass it. Be sure that you follow the guidelines and the instructions in order to make sure you are giving the correct ratings. If you fail the exam, it can be retaken. However, if you fail the exam the second time, chances are they will not give you another opportunity. Get Expert Personalized Assistance with Any Project Qualification
Tahoe project rating process and logic
Improving the discovery of events is done by making a distinction between real events that are happening in the real world at a specific time, and Non-Real Events which are obviously the opposite and not happening in the real world. The job of the individual working on this team is to label events as either Real or Non-Real events by reading the event and basic information. If needed, the Facebook page is opened, and the page is rated either a real or non-real event. This data will be used to help train an algorithm to be more accurate when it comes to events that are shown on Facebook. Non-real events are not removed from Facebook, so this work is important to help people find real events that are happening in their area.
So what is a real event?
When looking at an event, the main question is to ask if the event is an activity in the real world, and if it is, is there sufficient information for the user to attend the event like date and time, and directions or a phone number to reach someone. Chances are if it has all this information on an event, then it’s a real event and can be tagged as such.
The location for the real event should have the city, state, or a venue where the event is listed and usually found in the description box.
Time is important of course and there are some rules as far as the time is concerned.
Events that last up to 3 days from start to end, such as weekend festivals, are acceptable. Events that have a duration of more than 4 days are only considered an event if there is enough activity within each day to justify having one event created for the entire duration such as a retreat event.
When looking at these event ads, it is important to watch for phrases such as “time to be determined at a later date” which tells you that the event creator doesn’t have a set date, therefore the event should be tagged as Non-real.
Reoccurring events and long-term exhibitions need to have event pages for each day those events are going on. Some examples of this are a movie theatre that has midnight showings every Friday. This needs an event page for every Friday. An art exhibit that is being displayed at a studio for a month needs an event page for every day it is being displayed. This also goes for several performances on the same day. Each performance is considered a separate event and should have a new page.
The activity of the event doesn’t need to be explicitly stated in the description. Keep in mind that the main intent of the event post does not need to be about the activity itself, such as a page selling tickets to an event. This is still considered a real event because its selling tickets to a real event.
The key here is to watch out for events that closely resemble an advertisement rather than a real activity for the user. This includes things like promoting sales, tour packages, causes, etc. These are not considered real events.
What is considered a Non-Real event?
Depending on the location, many events will be encountered that initially seem like they are real events, but in reality, they do not satisfy the event’s realness requirements. When evaluating these types of posts, remember to always use the main frame of thinking as with every post: Is this an event I can attend, and if it is, is there enough information for me to attend the event?
Below are some examples of some common Non-events that will be encountered.
Promotional events make up the majority of Non-Real cases that the rater will encounter during this experiment. Most of the time these events will more closely resemble advertisements more than actual activities or events such as a company using the event creation tool to advertise a discount or a sale. Travel agencies creating an event for the purpose of selling some tour package. However, if it is an experience type of event or an educational event such as an ecological walk, even if its called a tour, if its still a real event that holds interest for locals then it is considered real. One key thing to note is that these posts are usually made by non-profit organizations, government agencies, or educators and are a good indicator that they are real.
Watch out for insufficient times such as an event that has a start and end time that doesn’t align with the actual activity for the event, although these events are very rare.
Time details to be determined is another factor in determining if something is Non-real. If there is no strict date, then its nothing you can plan for, and therefore not a real event.
If you see a post that has a schedule of events, it needs to be tagged as non-real as every event should have a single post of its own.
Call to action events are created for the purpose of receiving donations, votes, or any other type of contribution from a user without the need for in-person activity. With that said, if the event is a fundraiser that occurs at a real-world location, this would be acceptable as Real.
If the event post is gathering support to see a certain performer or artist in a specific city or country, then the post should be considered non-real because there is no confirmed real-world event.
Online events are considered Non-Real because they do not happen in the real world and that includes live streams, radio, and TV shows. Watch out for memes, just for fun, and spam that pretends to show real events happening in the world, but upon inspection, you realize that it’s evident that it can’t be real such as bands performing in unrealistic situations, or events that are centered around popular internet memes.
Another thing that event creators like to do is use the event creation tool to announce a large-scale celebration or holiday. If it’s just some announcement that doesn’t lead to a specific activity, then it gets labeled Non-real.
Below are some more examples:
Jobs should be rejected if you do not understand the language that the post is in. Any post with solicitation of Child Exploitation or Child Nudity should be escalated immediately with the JOB ID to the manager. If you are uncomfortable reviewing the content, then the job can be skipped after it is escalated. Canceled events are still marked real as long as they fulfill the Real event requirements.
The language of the experiment should be present in the event title or description. If these are missing then you can immediately reject the job. Make sure that time details listed for an event make sense for the actual activity, and also make sure that the city and state, or venue name is listed somewhere in the event info.
If you work on this project or know someone that does, we would like to hear from you in the comments about your experience on this project. If you have any more questions, then shoot me an email or just leave a comment and I will answer as quickly as possible.
There’s a new job position that is available through Sykes for a Search Quality Evaluator working from home part-time.
It isn’t known exactly what vendor this job could be for since Sykes is an outsourcing provider, but it could be Sykes, Appen or even Lionbridge. Another thing to note is that this job was posted on December 21, 2019 and is still listed. So, if you are looking for a job as a Search Quality Evaluator (known as Google Search Engine Evaluation project: Yukon, Internet Assessor, Rater) this could be your big work at home opportunity.
Judging from the job description about searching for online topics, and how frustrating it can be to get bad results, this is most certainly the same type of job as Search Engine Evaluator and you would be responsible for performing simple web searches on specific topics and then rating the results that are returned within the time frame they give you. A Windows PC is required for this work at home job, and no MACS are being accepted in this project. You must also have an activated smartphone with Wi-Fi capability. Applicants need to have strong computer skills or they will not be considered.
The time you want to work on this project is flexible and you can work on your own time as you wish. There is a schedule provided after training, although it is uncertain what this training might be since there is no previous training for these types of jobs. Only the guide is supplied to you and you must learn on your own. After this training period you must work a minimum of 20 hours per week.
One thing to keep in mind for this project is that you would be reviewing offensive and mature content. If this type of stuff offends you, then you may want to skip this one.
Also, if you live in these states, Sykes is unable to hire you: Alaska, Arizona, California, Colorado, District of Columbia, Hawaii, Illinois, Iowa, Maine, Massachusetts, Montana, New Hampshire, New Mexico, New York, North Dakota, Ohio, Oregon, Rhode Island, South Dakota, Vermont, Washington and West Virginia.
There is no mention of what the pay is at this time. If you happen to be working on this program, we’d like to hear from you about the training procedure and what the rate of pay might be. As always, if you have questions about this project, send me an email or leave a comment and I will reply as quickly as possible.
Hourly rates across different countries: Lionbridge, Appen
Important Note: Some Appen Global projects are not currently available in many states in the U.S.A. because the payment rate Appen Global offers is below the minimum wage in some of these areas. The company Sykes is also experiencing this same problem. Currently, the only alternative is Lionbridge. Our sources show that in many countries Appen Global pays 2 times less than Lionbridge, and to maximize profit, they cut wages. For example: In India Lionbridge pays $7 per hour, Appen $3 dollars an hour compared to the U.S.A. where Appen Pays $10 per hour and Lionbridge Pays $14.
Echo, Index, Sharp, Cable
was 8$/now 5$
was 8$/now 5$
was 8$/now 5$
was 13,5$/now 10$
was 9,45$/now 7$
was 9,45$/now 7$
was 8$/now 5$
was 13,5$/now 10$
was 9,45$/now 7$
was 9,45$/now 7$
was 12,5$ (net)/now 10$ (net)
was 10,5$/now 5$
was 10,5$/now 7$
was 8$/now 5$
Internet Assessor, Rater, Internet Safety Evaluator (Google)
Ads Assessor (Google)
Maps Analyst (Apple)
Why Appen decreased hourly rates almost in all markets after acquiring Leapforce?
Feel free to share your information regarding payment rates in your specific country, based on your input I will update information on this post.
This project is about a year old and was created and started when Google decided to exclude some YouTube video evaluation tasks from its main search engine evaluation projects: You are probably familiar with these projects: Yukon (Appen/Raterlabs) and Internet Assessor/Rater at Lionbridge. The Internet Safety Evaluator project picks up where search engine evaluation leaves off and the job has you looking at video search results to improve the search result quality for content. As an Internet Safety Evaluator, you are also required to provide feedback and analysis on video content found in the search results and provide ratings on their relevance with the search terms that were used.
So where does the Internet Safety Evaluator part come in? Well, that’s the most important part and aspect of this role. While reviewing content, you will flag potential upsetting or offensive material to make the internet safer and more secure for all users. You will be able to express your opinions in comments about the video concerning the content and the quality of content, and at the same time, you will be protecting users from viewing unsuitable material.
So what do you need to have to qualify for the position of Internet Safety Evaluator? Appen and Lionbridge are looking for dynamic and creative people who have a strong interest in improving online search results. One of the main things that you should feel comfortable about is working on sensitive or adult content. Candidates must use a smartphone for this position, and either an Android or iPhone device is suitable. The Internet Safety Evaluator should also have some familiarity with social media and Google products. Be prepared to be flexible concerning working hours and you should have the ability to understand and follow the established guidelines.
The Internet Safety Evaluator position only allows you to have 20-hours max work time per week.
The qualification process is identical to the Appen/Yukon Project or the Internet Assessor for Lionbridge. Read more about how to pass that exam here:
As an Internet Safety Rater working on the project, you will have access to a variety of different task types. This is important because it helps you become more familiar with the core concepts of rating and it will help you build experience and knowledge concerning the program. During the beginning, the number of tasks you will receive will be low until you gain enough experience in the program. Once you feel that you have a good understanding of the program and become familiar with all the task types, you will start to see a switch in the task types that you receive. But keep in mind, this will only happen if the managers feel you have a strong grasp on the concept of what you are doing. From that point forward, you will only be required to focus on evaluating more video type tasks from an Internet Safety Rater perspective. You will also see a higher number of tasks given to you.
As an Internet Safety Rater, you cannot work more than 1 hour per day and this limitation can last several months. The purpose of this limitation is to evaluate the overall quality of ratings that you are providing. During this time you will be given standard Yukon/Internet Assessor tasks (Needs Met and Side by Side and other types of tasks), additionally, you will be evaluated on your quality by means of standard Needs Met and Side by Side quality review tasks.
If you want to know more about this process, read here:
The quality requirements are high and you need to achieve and keep a quality level higher than 65% or above. This is the very minimum and work is monitored on a regular basis. Feedback will be shared with you so that you can learn from your mistakes. If an Internet Safety Evaluator constantly performs below that minimum quality level or performs significantly lower than the minimum level, that person will be removed from the program. This is why it is very important to use the months long limitations and make sure to learn as much as possible.
The things closely monitored to evaluation an Internet Safety Evaluator performance is the Needs Met, Page Quality (E.A.T.) and SXS rating. However, other things such as rating accuracy and the quality of the comments that you leave are another thing that is closely looked at.
Once the Internet Safety Evaluator has shown a stable level of quality, the probation period ends and you will get a notification that the Internet Safety Evaluation transition period ended and all restrictions are then lifted and you see a higher volume of tasks. This is when you will start receiving ONLY video evaluation tasks and you’ll get a higher volume of them. There will be no more review tasks after the completion of the transition period.
Internet Safety Evaluator probation and transition period
The Internet Safety Evaluator probation and transition period length all depends on the quality performance of the candidate who should demonstrate positive quality review scores. This period usually takes about 3-4 months but could possibly last up to 8 months depending on the performance of the individual.
In terms of the complexity of this project and how easy it is to participate, the Internet Safety Evaluator project seems to be easier in particular after the transition period has ended. Working on video tasks is a lot easier than doing the Needs Met and Side By Side tasks. However, the big drawback to the project is online being able to work 1-2 hours per day during the transition period.
You can work on other projects at the same time to make up for these small hours. You can provide similar rating services to other providers. However, due to project restrictions and customer requirements, you cannot work on Appen Yukon and the Lionbridge Internet Assessor at the same time due to an IP address conflict.
If you want more information on other unrelated projects, you can find that information here:
If you have any questions about the Internet Safety Evaluator position or any other questions, let me know in the comments or simply send me an email and I will reply. If you are working as an Internet Safety Evaluator we’d like to hear about your experience.
If you’ve landed here, you might have heard about the Preferred Raters Program…and It looks like there is finally some great benefits coming out for Lionbridge/Raterlabs for raters who achieve high scores and have consistently high-quality ratings and comments for all task types. This is for raters who work on the Yukon/Raterlabs project. Through this program, agents are identified and rewarded based on high-quality review scores and high-quality work they do on the Yukon project.
It sounds really great, but it is highly selective and prestigious, and only available to a limited amount of raters. The requirements to get into this program are very high, as well as inclusion and retention…meaning it will be tough to get in and also stay there. Participants are held to strict standards of quality and professionalism and the quality team reviews candidates quarterly to determine if they qualify for the program for admission or if you’re already in the program if you stay or someone comes along and takes your place.
So, what about the rewards and are they really worth it?
If you do get into the Preferred Raters Program, you’ll get a boost in pay rate. You’ll go from $10 to $15.00 dollars for Raterlabs and from $14 to $18 at Lionbridge. Not a bad little pay raise at all even if it is temporary.
As another added benefit, your username will be in green in chatrooms showing that you’re a highly skilled and very experienced rater.
How Do I Qualify?
As we mentioned earlier, raters are regularly reviewed to see who meets the requirements of the Preferred Agent Program.
Note: Sending them emails might be the quickest way to NOT get into the program. It is not recommended to email them about your status in the program.
Only if you have met the program’s requirements will you hear from the project’s Quality Team. If you don’t hear from them, then it may mean that you need to step up your quality or performance, or there just may be too many candidates and not enough room to fill them all. We do believe it might be the opposite since requirements are so high for the program.
The most important thing to focus on is quality review scores for all task types. Having high scores in this area will get you looked at for eligibility for the Preferred Raters Program. But it’s not all as there are several other additional criteria used to determine an agent’s final acceptance into the program and we’ll go over all the things they use to select someone.
Primary Selection Criteria
The very first thing to point out is only agents who have worked on the Yukon project for at least one year are eligible.
And agent needs to complete enough of the following tasks to receive the minimum Quality Review feedback:
Needs Met: 6 sets of feedback
Side by Side: 3 sets of feedback
Page Quality: 2 sets of feedback
The quality reviews all have to be extremely positive for all the task types, not just one, and high adherence and demonstrated comprehension of the principles detailed in all the guidelines and project instructions.
Comments are also a part of this and Preferred Raters will show excellent commenting skills for all tasks.
An agent will have a consistent history of good interactions with the community that should always be positive. This includes all areas of communication with admins in chat, social posts, and emails.
While we’re on the subject, we shouldn’t have to tell you that candidates that have a history of unprofessional, nasty, or obnoxious behavior in chat or other social channels won’t be considered for this program.
Agents in the program will always meet the minimum hours’ requirement and complete all the tasks in a responsible manner and not releasing them in an excessive or malicious way because maybe the agent doesn’t want to rate the produced results.
Did we mention that it’s hard to get into the Preferred Raters Program? We should mention it again and also state that it’s not an easy thing to achieve or maintain. But if you give hard work, perseverance, and show a very thorough understanding of the concepts from the General Guidelines, then chances are good that you’ll make it as a candidate.
I’ve made it as a candidate, now how do I keep my status?
We mentioned that the Preferred Raters Program is considered prestigious, highly selective, and it is limited to a small number of raters. That just means it’s going to be a lot more difficult for you to get in and stay there. But after you are in, a rater must maintain the high-quality standards across all task types and have high-quality reviews after acceptance.
If a rater fails to maintain high standards of quality, then they are put into a probational period and given the opportunity to improve. That’s good news since if you are in this program, you’ve worked hard to get where you are. However, if an agent continues to fail meeting the program criteria, they will lose their preferred status and the rewards associated. Also, once you are in the program you can be removed at any time at the Quality Teams’ discretion even without a probational period.
So what if you are put on probation?
If a rater fails to meet requirements they are placed on probation. This probation is actually two more reviews and if your reviews are bad, then you’re out. One other thing: If you are placed on probation and receive a substandard or unacceptable review score in ANY review type during your probationary period, your Preferred Agent status will be immediately removed.
So what triggers probation?
When an agent gets three bad scores within the most recent 6-month period below the Preferred Agent standard, this will trigger a probationary period. Also, if the majority of your recent scores are all bad and not meeting the high standard, you are placed on probation. The bottom line is that your scores should be primarily Exceeds Expectations or higher. Meets expectations do not cut the cake. While they are respectable, they are not high enough scores to be in the program. Multiple scores of Meets Expectations can result in the loss of your Preferred Agent status.
Another thing that should be mentioned is that two Substandard and Unacceptable review scores within a recent year’s time period will trigger a probation period. Any additional scores of the same type will result in the immediate removal from the program. Preferred Raters cannot have such low scores.
The most important requirement of a Preferred Agent is that they consistently provide exceptionally high-quality work and if your scores fluctuate, it might suggest to the Quality Team that you are unable to maintain the high standards of the program, and you will be placed on probation. While it is a disheartening thing to be placed on probation, this does mean that you are being given the chance to improve your quality of rating. Remember that Meets Expectations is not good enough.
Some other things that will provoke the Quality Team are poor commenting habits, excessive releasing of tasks, and complaints about unprofessional behavior in chat or other social channels could lead to probation or it could just see you leaving the program altogether. Remember that we said they have total discretion and can release you from the program at any time for any reason.
If you are put on probation then it’s lucky you and you’ll have to use the opportunity to correct the problems and retain your Preferred Agent status.
As we touched on a little earlier, Quality Teams review agents on a quarterly basis and if there is room to promote new agents, they will do so. If the program is full and all those raters are keeping their high standards up, then there won’t be any new raters accepted into the program. When a rater is removed from a program not meeting high-quality standards, then another rater who is consistently doing this is taken into the Preferred Agent program.
You might be wondering if one low score will put you on probation. The answer is no and the Quality Agents recognize that sometimes bad things happen. However, if you continue to get low scores, then you will be put on probation until you fix them and they only give you two reviews after this and you must score high on BOTH reviews, not just one.
Just so we are sure we covered this, while you are in the program, three low scores will get you noticed by the system which will trigger a probation period.
A rater must meet high-quality standards in all areas and all tasks, not just one area. Because the program is considered prestigious, it is limited to a certain amount of raters. That said, if you are in the program and have some problem in one area, it will show and you will likely end up on probation.
You can earn the status back if you lose it by demonstrating that the quality of your work meets the program’s qualification standards on a consistent basis. Keep in mind that just because a rater has been in the program before, preference will not be given over them when determining new rater eligibility. There are only two times you can be put on probation meaning that the third time and you are out of the program.
We hope that we were able to answer your questions about the Preferred Agent program. Are you in this program? Tell us your experience in the comments and if you need help you can always reach us by email.
There’s another project to work on over at Appen called the Appen Connect Wouri Project. We’ll go over the project a little and briefly touch on the main points, what type of job it is and what requirements there are. We will also touch on the grading process and give a few examples.
The goal of the Appen Connect Wouri Project is to understand the user’s intention as represented by the query, and then determine how good the result is and if it satisfies the user’s needs. Users are using Safari and Spotlight to search for different information needs. Some examples include wanting to find out knowledge or information on something or somebody, wanting to know if there is a certain store nearby, the latest news related to a topic, or since they are MAC users, getting an application on iTunes. This project is a bit like the average search engine evaluator, but it does have a different grading system. However, it uses some of the same ideas from those programs on search engine results.
A desktop or laptop computer is recommended to work on the Appen Connect Wouri Project over a mobile phone and all graders should be checking search results with Safari. Safari can be used on Windows operating systems, but it should be mentioned that there is no support from Apple when using Safari on MS Windows. You might be better off with a MAC computer, but it isn’t required.
To participate in the Appen Connect Wouri Project you have to pass an initial assessment and achieve a minimum score of 80%. If you don’t pass the first time, you’ll get another attempt. After you pass the initial assessment you will be given a final assessment in which you must also pass with a minimum score of 80%.
Wouri Project rating Logic
When looking at a user query, there are three steps that you will take in the grading process.
The first thing is understanding the query. You should make sure it’s not too vague, inappropriate, or in a foreign language.
The second thing is understanding the result. Determine if it is GEO or Time Sensitive. There is also the option to choose both or choose neither GEO or Time Sensitive.
Once you understand the query and the result, you will then decide on a relevance rating for the result based on the query intent.
It is important to note that Appen expects local graders working on the Appen Connect Wouri Project to judge the search results of the same locale. What that means is you must in the same location as the project requires. In this case, you may need to configure your computer and your search engine to be in the language of the project that you are working on.
A web relevant search is one that satisfies the user’s information need. Of course, a search result can be more or less relevant based on how well or how completely it satisfies the users’ needs. It is important, and we cannot stress this enough, that you have the guidelines open and you have thoroughly read them.
In step number one, you need to determine what it is that the user is searching for and decide if it’s valid. You should read the query, and review the third party results and think about the possible user intent that it represents. This is called ‘Understanding the Query.
For example, the query ‘Samsung’ could have many different user intents such as shopping for Samsung products, getting Samsung technical support, or the user may be looking for the Samsung homepage. However, if the user simply types in Samsung, then the query can be too vague.
If you have no idea what the user might be looking for, then the next step would be to go to a popular search engine and study the term. Some common terms could have some special meaning. However, if it doesn’t have some special meaning, or it has too many special meanings with each one having a small portion of intent, then you would select Query is Too Vague.
You might also run into a situation where a query is inappropriate and it encourages illegal behavior, includes profane words, using derogatory language, racial slurs or hate speech. It also includes graphic violence or the query might like to retrieve grisly content. You are not expected to do anything about these types of queries except to mark it as inappropriate.
When in doubt about any type of query, you should check results from other search engines. It’s absolutely crucial to understand what the user is searching for, and you may need to check results from other search engines. The good thing about this project is that top results from other search engines are provided for you. There is a small link that says ‘Web Search Link’ that will automatically do this for you and show you the top results in order to better help you understand the query.
It is VITAL to point out that opening the webpages is a MUST when you are working on this project. If the results match the first result of the other search engine, then the result is likely an excellent match. Make sure to check the results that are returned by the Web Search Link. Each result has a clickable URL, a small description, and a photo where applicable.
We touched a little on GEO and Time Sensitive in the beginning, and since you might not understand what that is, we will go over it to help you understand it better.
If a query is about a store or some location that exists in other locations aside from the searcher, let’s say a Starbucks, or a query like ‘movie theatres near me’ then you know its geo sensitive meaning that it matters where and how close the result to the searcher is. If the searcher is from California and looking for a Starbucks and the first result is the homepage or another location, then it doesn’t satisfy the user’s needs.
Anything that deals with a query that involves some kind of time frame, such as ‘Buy Superbowl Tickets’, age of Harrison Ford, or checking on the weather are results that should be marked as ‘Time Sensitive’.
It is important to understand that a query can be both Geo sensitive and Time sensitive, and a good example is a query such as ‘tickets to the local basketball game’, local movie showtimes, etc.
Once you understand the query and you judge the results, then you will assign a relevant grade for each result. There are 9-grade levels:
● Query In Foreign Language
● Result Page In Foreign Language
There are many examples in the guidelines that go over difficult queries in order to help you evaluate what results would satisfy the informational need for the searcher. It is a must that you go over the guidelines and read them and understand them so you know how to grade something.
The guidelines go over when you should choose an excellent result, and the biggest point to keep in mind here is to believe that the result will be everything that the user will need, and they will not have to continue searching. If you believe that the result isn’t good enough, and you would also do an additional search, then you might choose ‘Good’ or ‘Fair’ as the grade.
A ‘Good Result is one that the user wants to see above the fold on the device screen. It is typically the first 4-5 results. However, it is not the best possible result, which would be excellent, but it still provides good information. The guidelines have many examples of what should be chosen as a ‘Good Result’.
When choosing a ‘Fair Result’ for the query, the result would only be somewhat relevant, but not be of interest to all users. An example of this would be a query for a celebrity name, but the result is an obscure person that has the same name.
Choosing ‘Bad Result’ would pertain to a result that does not satisfy user intent and is something that should not be shown to users at all. An example of this would be something that we previously mentioned before if a user was looking for a Starbucks Coffee and lives in San Francisco, but the map result is for a store in San Diego. There are other examples in the guidelines.
It should be straight-forward of when you should choose ‘Foreign Language’. If the given query is in a different language from the specified one, or the target language that you are assigned to work, then you would mark this as “Query in Foreign Language”. If you are not sure if the query is in a foreign language, then just search that query to understand the word or words.
Sometimes in the given search results for a query, the URL will be inaccessible, or website link will not be available or a dead link. If this happens then you would grade it ‘Unavailable’ This also includes landing pages that take longer than 10 seconds to load, or you may see that the domain is up for sale by being sent to a redirection page. This grade also pertains to when video content is unavailable or blocked.
There are some other specific result types that the guidelines go over including ‘Ambiguous Queries’ or when the interpretation is much more popular than others. For example, the query is ‘Golden Retriever’ but the result is a song titled the same. This should be graded as fair because the dominant interpretation would be for the dog breed itself and not a song title. There are some instanced when multiple interpretations can be as equally popular though. In order to assign something to the excellent category, it’s a good idea to take into consideration what geo-location the seacher is looking from. Example:
Query is “UM football”, and the result is the official website for the University of Miami football team. If user’s location is in San Francisco, grade the result as Good, since UM could refer to either U of Miami or U of
Michigan and we don’t know which is best. But if the user location is in Miami, grade as Excellent.
As always, if you have any questions about the program, please send an email or reply in the comments. If you are working on this program, then we would love to hear about your experience in the comments.