Sykes Search Quality Evaluator – New Vendor For Google Search Engine Evaluation Project (US only)

Sykes Search Quality Evaluator

There’s a new job position that is available through Sykes for a Search Quality Evaluator working from home part-time.

It isn’t known exactly what vendor this job could be for since Sykes is an outsourcing provider, but it could be Sykes, Appen or even Lionbridge. Another thing to note is that this job was posted on December 21, 2019 and is still listed. So, if you are looking for a job as a Search Quality Evaluator (known as Google Search Engine Evaluation project: Yukon, Internet Assessor, Rater) this could be your big work at home opportunity.

Judging from the job description about searching for online topics, and how frustrating it can be to get bad results, this is most certainly the same type of job as Search Engine Evaluator and you would be responsible for performing simple web searches on specific topics and then rating the results that are returned within the time frame they give you. A Windows PC is required for this work at home job, and no MACS are being accepted in this project. You must also have an activated smartphone with Wi-Fi capability. Applicants need to have strong computer skills or they will not be considered.  

The time you want to work on this project is flexible and you can work on your own time as you wish. There is a schedule provided after training, although it is uncertain what this training might be since there is no previous training for these types of jobs. Only the guide is supplied to you and you must learn on your own. After this training period you must work a minimum of 20 hours per week. 

One thing to keep in mind for this project is that you would be reviewing offensive and mature content. If this type of stuff offends you, then you may want to skip this one. 

Also, if you live in these states, Sykes is unable to hire you: Alaska, Arizona, California, Colorado, District of Columbia, Hawaii, Illinois, Iowa, Maine, Massachusetts, Montana, New Hampshire, New Mexico, New York, North Dakota, Ohio, Oregon, Rhode Island, South Dakota, Vermont, Washington and West Virginia.

You may apply for Search Quality Evaluator role here:https://jobs.sykes.com/en-US/search?keywords=Search%20Quality%20Evaluator&location=

There is no mention of what the pay is at this time. If you happen to be working on this program, we’d like to hear from you about the training procedure and what the rate of pay might be. As always, if you have questions about this project, send me an email or leave a comment and I will reply as quickly as possible. 

Advertisements

Appen and Lionbridge projects – Hourly Rates Across Different Markets

Hourly rates across different countries: Lionbridge, Appen
Country

Vendor/Project
USAEUTurkeyBrazilJapanRussia
Appen
Echo, Index, Sharp, Cable 12$-13,5$ 14$ 7$
Arrow (Google) 10$ 14$ was 8$/now 5$ was 8$/now 5$ 15$-17$ was 8$/now 5$
Truckee (Insta) was 13,5$/now 10$ 14$ was 9,45$/now 7$ was 9,45$/now 7$ was 8$/now 5$
Nile (Facebook) was 13,5$/now 10$ 14$ was 9,45$/now 7$ was 9,45$/now 7$ 15$
Shasta per task per task per task per task per task per task
Yukon (Google) was 12,5$ (net)/now 10$ (net) 14$ was 10,5$/now 5$ was 10,5$/now 7$ 14$ was 8$/now 5$
Lionbridge
Internet Assessor, Rater, Internet Safety Evaluator (Google) 14,5$ (gross)
14$ 10$ 12$ 7,3$
Ads Assessor (Google) 13,5$ 14$ 10$ 12$ 7,3$
Maps Analyst (Apple) per task per task
Why Appen decreased hourly rates almost in all markets after acquiring Leapforce?

Feel free to share your information regarding payment rates in your specific country, based on your input I will update information on this post.

Internet Safety Evaluator Project For Lionbridge Review

Internet Safety Evaluator project review

This project is about a year old and was created and started when Google decided to exclude some YouTube video evaluation tasks from its main search engine evaluation projects: You are probably familiar with these projects: Yukon (Appen/Raterlabs) and Internet Assessor/Rater at Lionbridge. The Internet Safety Evaluator project picks up where search engine evaluation leaves off and the job has you looking at video search results to improve the search result quality for content. As an Internet Safety Evaluator, you are also required to provide feedback and analysis on video content found in the search results and provide ratings on their relevance with the search terms that were used. 

So where does the Internet Safety Evaluator part come in? Well, that’s the most important part and aspect of this role. While reviewing content, you will flag potential upsetting or offensive material to make the internet safer and more secure for all users. You will be able to express your opinions in comments about the video concerning the content and the quality of content, and at the same time, you will be protecting users from viewing unsuitable material. 

So what do you need to have to qualify for the position of Internet Safety Evaluator? Appen and Lionbridge are looking for dynamic and creative people who have a strong interest in improving online search results. One of the main things that you should feel comfortable about is working on sensitive or adult content. Candidates must use a smartphone for this position, and either an Android or iPhone device is suitable. The Internet Safety Evaluator should also have some familiarity with social media and Google products. Be prepared to be flexible concerning working hours and you should have the ability to understand and follow the established guidelines. 

The Internet Safety Evaluator position only allows you to have 20-hours max work time per week. 

The qualification process is identical to the Appen/Yukon Project or the Internet Assessor for Lionbridge. Read more about how to pass that exam here:

Also, don’t forget to check out the really cool tool that we’ve created for helping to pass these tough exams. See Our Exam Simulation Tool Here: https://examsimulation.passleapforceexam.com

Rating Process And Quality Metrics

As an Internet Safety Rater working on the project, you will have access to a variety of different task types. This is important because it helps you become more familiar with the core concepts of rating and it will help you build experience and knowledge concerning the program. During the beginning, the number of tasks you will receive will be low until you gain enough experience in the program. Once you feel that you have a good understanding of the program and become familiar with all the task types, you will start to see a switch in the task types that you receive. But keep in mind, this will only happen if the managers feel you have a strong grasp on the concept of what you are doing. From that point forward, you will only be required to focus on evaluating more video type tasks from an Internet Safety Rater perspective. You will also see a higher number of tasks given to you. 

As an Internet Safety Rater, you cannot work more than 1 hour per day and this limitation can last several months. The purpose of this limitation is to evaluate the overall quality of ratings that you are providing. During this time you will be given standard Yukon/Internet Assessor tasks (Needs Met and Side by Side and other types of tasks), additionally, you will be evaluated on your quality by means of standard Needs Met and Side by Side quality review tasks. 

If you want to know more about this process, read here:

The quality requirements are high and you need to achieve and keep a quality level higher than 65% or above. This is the very minimum and work is monitored on a regular basis. Feedback will be shared with you so that you can learn from your mistakes. If an Internet Safety Evaluator constantly performs below that minimum quality level or performs significantly lower than the minimum level, that person will be removed from the program. This is why it is very important to use the months long limitations and make sure to learn as much as possible. 

The things closely monitored to evaluation an Internet Safety Evaluator performance is the Needs Met, Page Quality (E.A.T.) and SXS rating. However, other things such as rating accuracy and the quality of the comments that you leave are another thing that is closely looked at. 

Once the Internet Safety Evaluator has shown a stable level of quality, the probation period ends and you will get a notification that the Internet Safety Evaluation transition period ended and all restrictions are then lifted and you see a higher volume of tasks. This is when you will start receiving ONLY video evaluation tasks and you’ll get a higher volume of them. There will be no more review tasks after the completion of the transition period. 

Internet Safety Evaluator probation and transition period

The Internet Safety Evaluator probation and transition period length all depends on the quality performance of the candidate who should demonstrate positive quality review scores. This period usually takes about 3-4 months but could possibly last up to 8 months depending on the performance of the individual. 

In terms of the complexity of this project and how easy it is to participate, the Internet Safety Evaluator project seems to be easier in particular after the transition period has ended. Working on video tasks is a lot easier than doing the Needs Met and Side By Side tasks. However, the big drawback to the project is online being able to work 1-2 hours per day during the transition period. 

You can work on other projects at the same time to make up for these small hours. You can provide similar rating services to other providers. However, due to project restrictions and customer requirements, you cannot work on Appen Yukon and the Lionbridge Internet Assessor at the same time due to an IP address conflict. 

If you want more information on other unrelated projects, you can find that information here:

If you have any questions about the Internet Safety Evaluator position or any other questions, let me know in the comments or simply send me an email and I will reply. If you are working as an Internet Safety Evaluator we’d like to hear about your experience. 

Preferred Raters Program For Lionbridge/Raterlabs

Preferred Raters Program

If you’ve landed here, you might have heard about the Preferred Raters Program…and It looks like there is finally some great benefits coming out for Lionbridge/Raterlabs for raters who achieve high scores and have consistently high-quality ratings and comments for all task types. This is for raters who work on the Yukon/Raterlabs project. Through this program, agents are identified and rewarded based on high-quality review scores and high-quality work they do on the Yukon project. 

It sounds really great, but it is highly selective and prestigious, and only available to a limited amount of raters. The requirements to get into this program are very high, as well as inclusion and retention…meaning it will be tough to get in and also stay there. Participants are held to strict standards of quality and professionalism and the quality team reviews candidates quarterly to determine if they qualify for the program for admission or if you’re already in the program if you stay or someone comes along and takes your place. 


So, what about the rewards and are they really worth it? 

If you do get into the Preferred Raters Program, you’ll get a boost in pay rate. You’ll go from $10 to $15.00 dollars for Raterlabs and from $14 to $18 at Lionbridge. Not a bad little pay raise at all even if it is temporary. 

As another added benefit, your username will be in green in chatrooms showing that you’re a highly skilled and very experienced rater. 

How Do I Qualify? 

As we mentioned earlier, raters are regularly reviewed to see who meets the requirements of the Preferred Agent Program. 

Note: Sending them emails might be the quickest way to NOT get into the program. It is not recommended to email them about your status in the program. 

Only if you have met the program’s requirements will you hear from the project’s Quality Team. If you don’t hear from them, then it may mean that you need to step up your quality or performance, or there just may be too many candidates and not enough room to fill them all. We do believe it might be the opposite since requirements are so high for the program. 

The most important thing to focus on is quality review scores for all task types. Having high scores in this area will get you looked at for eligibility for the Preferred Raters Program. But it’s not all as there are several other additional criteria used to determine an agent’s final acceptance into the program and we’ll go over all the things they use to select someone. 

Primary Selection Criteria

The very first thing to point out is only agents who have worked on the Yukon project for at least one year are eligible. 

And agent needs to complete enough of the following tasks to receive the minimum Quality Review feedback:

Needs Met: 6 sets of feedback

Side by Side: 3 sets of feedback

Page Quality: 2 sets of feedback

The quality reviews all have to be extremely positive for all the task types, not just one, and high adherence and demonstrated comprehension of the principles detailed in all the guidelines and project instructions. 

Comments are also a part of this and Preferred Raters will show excellent commenting skills for all tasks. 

An agent will have a consistent history of good interactions with the community that should always be positive. This includes all areas of communication with admins in chat, social posts, and emails. 

While we’re on the subject, we shouldn’t have to tell you that candidates that have a history of unprofessional, nasty, or obnoxious behavior in chat or other social channels won’t be considered for this program. 

Agents in the program will always meet the minimum hours’ requirement and complete all the tasks in a responsible manner and not releasing them in an excessive or malicious way because maybe the agent doesn’t want to rate the produced results. 

Did we mention that it’s hard to get into the Preferred Raters Program? We should mention it again and also state that it’s not an easy thing to achieve or maintain. But if you give hard work, perseverance, and show a very thorough understanding of the concepts from the General Guidelines, then chances are good that you’ll make it as a candidate. 

I’ve made it as a candidate, now how do I keep my status?

We mentioned that the Preferred Raters Program is considered prestigious, highly selective, and it is limited to a small number of raters. That just means it’s going to be a lot more difficult for you to get in and stay there. But after you are in, a rater must maintain the high-quality standards across all task types and have high-quality reviews after acceptance. 

If a rater fails to maintain high standards of quality, then they are put into a probational period and given the opportunity to improve. That’s good news since if you are in this program, you’ve worked hard to get where you are. However, if an agent continues to fail meeting the program criteria, they will lose their preferred status and the rewards associated. Also, once you are in the program you can be removed at any time at the Quality Teams’ discretion even without a probational period. 

So what if you are put on probation?

If a rater fails to meet requirements they are placed on probation. This probation is actually two more reviews and if your reviews are bad, then you’re out. One other thing:  If you are placed on probation and receive a substandard or unacceptable review score in ANY review type during your probationary period, your Preferred Agent status will be immediately removed.

So what triggers probation? 

When an agent gets three bad scores within the most recent 6-month period below the Preferred Agent standard, this will trigger a probationary period. Also, if the majority of your recent scores are all bad and not meeting the high standard, you are placed on probation. The bottom line is that your scores should be primarily Exceeds Expectations or higher. Meets expectations do not cut the cake. While they are respectable, they are not high enough scores to be in the program. Multiple scores of Meets Expectations can result in the loss of your Preferred Agent status. 

Another thing that should be mentioned is that two Substandard and Unacceptable review scores within a recent year’s time period will trigger a probation period. Any additional scores of the same type will result in the immediate removal from the program. Preferred Raters cannot have such low scores. 

The most important requirement of a Preferred Agent is that they consistently provide exceptionally high-quality work and if your scores fluctuate, it might suggest to the Quality Team that you are unable to maintain the high standards of the program, and you will be placed on probation. While it is a disheartening thing to be placed on probation, this does mean that you are being given the chance to improve your quality of rating. Remember that Meets Expectations is not good enough. 

Some other things that will provoke the Quality Team are poor commenting habits, excessive releasing of tasks, and complaints about unprofessional behavior in chat or other social channels could lead to probation or it could just see you leaving the program altogether. Remember that we said they have total discretion and can release you from the program at any time for any reason. 

If you are put on probation then it’s lucky you and you’ll have to use the opportunity to correct the problems and retain your Preferred Agent status. 

As we touched on a little earlier, Quality Teams review agents on a quarterly basis and if there is room to promote new agents, they will do so. If the program is full and all those raters are keeping their high standards up, then there won’t be any new raters accepted into the program. When a rater is removed from a program not meeting high-quality standards, then another rater who is consistently doing this is taken into the Preferred Agent program. 

You might be wondering if one low score will put you on probation. The answer is no and the Quality Agents recognize that sometimes bad things happen. However, if you continue to get low scores, then you will be put on probation until you fix them and they only give you two reviews after this and you must score high on BOTH reviews, not just one. 

Just so we are sure we covered this, while you are in the program, three low scores will get you noticed by the system which will trigger a probation period. 

A rater must meet high-quality standards in all areas and all tasks, not just one area. Because the program is considered prestigious, it is limited to a certain amount of raters. That said, if you are in the program and have some problem in one area, it will show and you will likely end up on probation. 

You can earn the status back if you lose it by demonstrating that the quality of your work meets the program’s qualification standards on a consistent basis. Keep in mind that just because a rater has been in the program before, preference will not be given over them when determining new rater eligibility. There are only two times you can be put on probation meaning that the third time and you are out of the program. 

We hope that we were able to answer your questions about the Preferred Agent program. Are you in this program? Tell us your experience in the comments and if you need help you can always reach us by email. 

Appen Connect Wouri Project Review And Information

Wouri project review

There’s another project to work on over at Appen called the Appen Connect Wouri Project. We’ll go over the project a little and briefly touch on the main points, what type of job it is and what requirements there are. We will also touch on the grading process and give a few examples. 

The goal of the Appen Connect Wouri Project is to understand the user’s intention as represented by the query, and then determine how good the result is and if it satisfies the user’s needs. Users are using Safari and Spotlight to search for different information needs. Some examples include wanting to find out knowledge or information on something or somebody, wanting to know if there is a certain store nearby, the latest news related to a topic, or since they are MAC users, getting an application on iTunes. This project is a bit like the average search engine evaluator, but it does have a different grading system. However, it uses some of the same ideas from those programs on search engine results. 

A desktop or laptop computer is recommended to work on the Appen Connect Wouri Project over a mobile phone and all graders should be checking search results with Safari. Safari can be used on Windows operating systems, but it should be mentioned that there is no support from Apple when using Safari on MS Windows. You might be better off with a MAC computer, but it isn’t required. 

To participate in the Appen Connect Wouri Project you have to pass an initial assessment and achieve a minimum score of 80%. If you don’t pass the first time, you’ll get another attempt. After you pass the initial assessment you will be given a final assessment in which you must also pass with a minimum score of 80%. 

Wouri Project rating Logic

When looking at a user query, there are three steps that you will take in the grading process. 

  • The first thing is understanding the query. You should make sure it’s not too vague, inappropriate, or in a foreign language. 
  • The second thing is understanding the result. Determine if it is GEO or Time Sensitive. There is also the option to choose both or choose neither GEO or Time Sensitive. 
  • Once you understand the query and the result, you will then decide on a relevance rating for the result based on the query intent. 

It is important to note that Appen expects local graders working on the Appen Connect Wouri Project to judge the search results of the same locale. What that means is you must in the same location as the project requires. In this case, you may need to configure your computer and your search engine to be in the language of the project that you are working on. 

A web relevant search is one that satisfies the user’s information need. Of course, a search result can be more or less relevant based on how well or how completely it satisfies the users’ needs. It is important, and we cannot stress this enough, that you have the guidelines open and you have thoroughly read them. 

In step number one, you need to determine what it is that the user is searching for and decide if it’s valid. You should read the query, and review the third party results and think about the possible user intent that it represents. This is called ‘Understanding the Query.

For example, the query ‘Samsung’ could have many different user intents such as shopping for Samsung products, getting Samsung technical support, or the user may be looking for the Samsung homepage. However, if the user simply types in Samsung, then the query can be too vague. 

If you have no idea what the user might be looking for, then the next step would be to go to a popular search engine and study the term. Some common terms could have some special meaning. However, if it doesn’t have some special meaning, or it has too many special meanings with each one having a small portion of intent, then you would select Query is Too Vague. 

You might also run into a situation where a query is inappropriate and it encourages illegal behavior, includes profane words, using derogatory language, racial slurs or hate speech. It also includes graphic violence or the query might like to retrieve grisly content. You are not expected to do anything about these types of queries except to mark it as inappropriate.  

When in doubt about any type of query, you should check results from other search engines. It’s absolutely crucial to understand what the user is searching for, and you may need to check results from other search engines. The good thing about this project is that top results from other search engines are provided for you. There is a small link that says ‘Web Search Link’ that will automatically do this for you and show you the top results in order to better help you understand the query. 

It is VITAL to point out that opening the webpages is a MUST when you are working on this project. If the results match the first result of the other search engine, then the result is likely an excellent match. Make sure to check the results that are returned by the Web Search Link. Each result has a clickable URL, a small description, and a photo where applicable. 

We touched a little on GEO and Time Sensitive in the beginning, and since you might not understand what that is, we will go over it to help you understand it better.

If a query is about a store or some location that exists in other locations aside from the searcher, let’s say a Starbucks, or a query like ‘movie theatres near me’ then you know its geo sensitive meaning that it matters where and how close the result to the searcher is. If the searcher is from California and looking for a Starbucks and the first result is the homepage or another location, then it doesn’t satisfy the user’s needs. 

Anything that deals with a query that involves some kind of time frame, such as ‘Buy Superbowl Tickets’, age of Harrison Ford, or checking on the weather are results that should be marked as ‘Time Sensitive’. 

It is important to understand that a query can be both Geo sensitive and Time sensitive, and a good example is a query such as ‘tickets to the local basketball game’, local movie showtimes, etc. 

Once you understand the query and you judge the results, then you will assign a relevant grade for each result. There are 9-grade levels:

● Excellent

● Good

● Fair

● Bad

● Unavailable

● Query In Foreign Language

● Result Page In Foreign Language

● Vague

● Inappropriate

There are many examples in the guidelines that go over difficult queries in order to help you evaluate what results would satisfy the informational need for the searcher. It is a must that you go over the guidelines and read them and understand them so you know how to grade something. 

The guidelines go over when you should choose an excellent result, and the biggest point to keep in mind here is to believe that the result will be everything that the user will need, and they will not have to continue searching. If you believe that the result isn’t good enough, and you would also do an additional search, then you might choose ‘Good’ or ‘Fair’ as the grade. 

A ‘Good Result is one that the user wants to see above the fold on the device screen. It is typically the first 4-5 results. However, it is not the best possible result, which would be excellent, but it still provides good information. The guidelines have many examples of what should be chosen as a ‘Good Result’. 

When choosing a ‘Fair Result’ for the query, the result would only be somewhat relevant, but not be of interest to all users. An example of this would be a query for a celebrity name, but the result is an obscure person that has the same name. 

Choosing  ‘Bad Result’ would pertain to a result that does not satisfy user intent and is something that should not be shown to users at all. An example of this would be something that we previously mentioned before if a user was looking for a Starbucks Coffee and lives in San Francisco, but the map result is for a store in San Diego. There are other examples in the guidelines. 

It should be straight-forward of when you should choose ‘Foreign Language’. If the given query is in a different language from the specified one, or the target language that you are assigned to work, then you would mark this as “Query in Foreign Language”. If you are not sure if the query is in a foreign language, then just search that query to understand the word or words. 

Sometimes in the given search results for a query, the URL will be inaccessible, or website link will not be available or a dead link. If this happens then you would grade it ‘Unavailable’ This also includes landing pages that take longer than 10 seconds to load, or you may see that the domain is up for sale by being sent to a redirection page. This grade also pertains to when video content is unavailable or blocked. 

There are some other specific result types that the guidelines go over including ‘Ambiguous Queries’ or when the interpretation is much more popular than others. For example, the query is ‘Golden Retriever’ but the result is a song titled the same. This should be graded as fair because the dominant interpretation would be for the dog breed itself and not a song title. There are some instanced when multiple interpretations can be as equally popular though. In order to assign something to the excellent category, it’s a good idea to take into consideration what geo-location the seacher is looking from. Example:

Query is “UM football”, and the result is the official website for the University of Miami football team. If user’s location is in San Francisco, grade the result as Good, since UM could refer to either U of Miami or U of

Michigan and we don’t know which is best. But if the user location is in Miami, grade as Excellent.

As always, if you have any questions about the program, please send an email or reply in the comments. If you are working on this program, then we would love to hear about your experience in the comments. 

List of Projects And Code Names For Lionbridge & Appen Connect

If you are looking for a list of projects out there from companies like Lionbridge and Appen, we have them. As you may know Appen acquired Leapforce in 2017, since then the company has decided to close it is own project management platform known as Appen Global and move all projects to Appen connect. The group of projects known as Falcon (mainly Facebook-related projects) now is no longer actual, these projects also have been moved to Appen connect platform.

Taking into account the fact that all projects have code names and little description is provided, many candidates might be lost in terms what project to go for. 

Bellow you will find a list of most popular projects and links to more info about each project:

Appen connect
Project code nameDescription Reference for more info
YukonGoogle Search Engine Evaluation Project Click here

Exam Simulation Tool

Get Personalized Assistance or Advise
ArrowGoogle Ads Evaluation ProjectClick here

Exam Simulation Tool

Get Personalized Assistance or Advise
ShastaMaps Evaluation Project

Provide relevance and accuracy ratings for one of the world's largest and most popular mapping applications.
The job has an ABOVE average volume of work.
Click here

Get Personalized Assistance or Advise
Echo Apple iTunes Music Search Results Evaluation Project

Improving the relevance of search results for music-related content.
Raters must have access to an iOS device capable of running iOS 12.x. and higher. The job has an average volume of work.
Click here

Exam Simulation Tool

Get Personalized Assistance or Advise
CableApple TV Search Result Evaluation.

Search Relevance task focused on improving the relevance of search results, based on the most likely intent of user searches. Raters must have access to an iOS device capable of running iOS 12.1.4. The job has an average volume of work.
Similar to Echo

Get Personalized Assistance or Advise
JavaApple Store Search Relevance project focused on improving the relevance of search results based on user search intent.
Raters must have access to an iOS device capable of running iOS 12.1.4
Similar to Echo
SharpApple Search Relevance task focused on improving the relevance of search results within different media categories.
Raters must have access to an iOS device capable of running iOS 12.1.4
Similar to Echo
IndexApple App Store Search Results Evaluation Project

Search Relevance task focused on improving the relevance of search results based on user search intent.
Raters must have access to an iOS device capable of running iOS 12.1.4. The job has an average volume of work.
Similar to Echo
Wouri Apple Safari Search Result Evaluation project. The goal of the project is to understand the user intention as represented by the query and determine how well or how completely the result satisfies the user's need. Click here
Pecos/Ipsoot (also known as Pearl and Falcon PT)Post categorization on Facebook. You have to decide what topic that post fits under. It is basically a categorizing system job which aims to put the posts that are made into the correct categories of broad topics. Click here
TruckeeInstagram Ads Evaluation Project

Help improve ads quality for one of the world’s largest image sharing sites.
Required: An active Facebook and Instagram account – The Instagram account must be connected to your Facebook account. You also need a smartphone with Android or iOS. The job has an Above Average work volume.
Click here
NileFacebook Ads Evaluation Project

Help improve ads quality for one of the world’s largest social networking platforms.
Active Facebook account required.
Click here
RattleThe goal of this project is to improve the relevance of search results for one of the world’s largest search engines. Click here
ParadiseThe goal of this project is to improve the accuracy of search results for businesses and entities on one of the world’s leading search engines. Click here
Mohawk Help improve ad relevancy on a popular social media platform.
Similar to FB
Falcon projects
Prospect MI Facebook Falcon Projects. Help improve ad relevancy on a popular social media platform.
Click here
Zeya Facebook Falcon Projects. In this task, you will be asked to research and evaluate claims stated in various pieces of content (e.g., articles, links, photos, or videos). 20 hrs per week, work is done via Facebook rating tool.
Click here
Copper The aim of this project is to help the client evaluate and improve the quality of the Ads that they present to their users.
Requirements are a personal computer running Windows 8 or better and a desire to look at different ads.
Click here
Alkali The aim of this project is to help the client evaluate and improve the quality of the Ads that they present to their users.
Requirements are a personal computer running Windows 8 or better and a desire to look at different ads.
Average volume of work.
No info
TigrisThe project consists of rating tasks focusing on improving the relevancy of advertisement results from a major social media platform. Consultants will rate advertisements to help the client understand which advertisements (also known as ads) people want to see, and why. The daily task of the consultant consists of rating a batch of ads in a client-owned rating tool. FB project?
CarmelUtterance annotation and evaluation.Microsoft Bing project aimed to learn search engine to recognize voice and human speech. No info

*group of projects known as Falcon (mainly Facebook related projects) now is no longer actual, these projects also have been moved to Appen connect platform. Newly added projects such as Mohalwk, Prospect MI, Zeya, Copper and Alkali are most likely related to small Facebook testing projects. If you have been participating in one of the above listed projects, feel free to share your experience with us via leave a comment option!

Lionbridge
Lionbridge project code nameDescription Reference for more info
Internet Assessor, Personalized Internet Assessor, RaterGoogle Search Engine Evaluation Project Click here

Exam Simulation Tool

Get Personalized Assistance or Advise
Internet Safety EvaluatorGoogle Youtube Evaluation Project Click here

Exam Simulation Tool

Get Personalized Assistance or Advise
Multimedia JudgeBing Search Engine Evaluation project Click here
Personalized Ads Assessor Google Ads Evaluation Project Click here


Exam Simulation Tool

Get Personalized Assistance or Advise
Online Map Quality AnalystMaps Evaluation Project Click here

Get Personalized Assistance or Advise
ISoftstone
iSoftStone project code namesDescription Reference for more info
Online Ad EvaluatorBing Ad Evaluation project Click here
Search Engine EvaluatorBing Search Engine Evaluation project Click here

Timesheet Disapproval On Lionbridge Ads Rating Project

Ads Rating Project massive timesheet disapproval: on purpose or unprofessional approach? 

It appears that lately there are a lot of problems with the Lionbridge payment systems that has resulted in a timesheet disapproval and it’s not actually a new thing. According to comments of my readers and discussions on https://www.reddit.com/r/WorkOnline/comments/9b0n2n/lionbridge_scammers/  (2018), https://www.reddit.com/r/WorkOnline/comments/cvol9y/lionbridge_scammers_2/ (2019, August),https://www.reddit.com/r/WorkOnline/comments/d95mwc/lionbridge_scammingasking_to_remove_hours_again/ 

The problem with the timesheet disaproval started to happen in May-June 2018. It just seems that now it’s becoming worse and there are more and more raters complaining that the time sheets that are being submitted are being returned. Lionbridge is asking people to carve a lot of hours from their time sheets and it’s obvious that there is a big problem somewhere, but the company isn’t acknowledging that there is an issue, or perhaps they are just doing this to save themselves a lot of money and get away with it. And the raters who are affected by this timesheet disapproval payment problem aren’t being asked to take an hour or two off, in fact, in some cases, people have been asked to cut 30-40 hours from their time sheets, some of them are asked to remove all hours from July timesheet. This is huge and it’s a lot of money for someone. In fact, it appears that it’s happening to a lot of people as one rater who has been working for the company says that he has 6 friends who all got the same notifications. In one case, one of our readers PrestigiousExplorer says that after working one year for Lionbridge the same thing has now happened to him and the company wants him to deduct 68 hours!  

The problem on their end which results in the timesheet disaproval appears to be that the hours that are being submitted don’t match their data. What it means is that the amount of tasks completed isn’t matching up with what your submitted amount is. But here’s the thing; if suddenly the Lionbridge payment department starts seeing a lot of discrepancies in time sheets and the data that they have, shouldn’t they realize that there is a problem somewhere? Now suddenly here you have a bunch of people being asked to shave a lot of hours off their time sheets, and even people who have been working for the company for a long time and then suddenly are having issues. It seems that Lionbridge doesn’t care about any of this and are sticking to their guns and won’t budge on time sheets. If you were subject to a timesheet disapproval then you might be stuck with it.  

The big question is does no one at Lionbridge see this as a problem? Or is there something else going on?  

As you may know raterhub automatically records the time and the number of tasks completed by each rater on the platform. It is very unlikely that the problem takes place there, however many long-term raters remember that raterhub used to have built-in task counter which was very helpful in terms of recording completed tasks, but for some reason this useful tool was removed from the raterhub functionality.

It looks like the Lionbridge system is just an Excel spreadsheet which corresponds timesheet data with data from raterhub (time recorded/the number of tasks completed) and the problem is there. Many raters worked lots of hours in July and got an email with 0 hours/0 tasks recorded by the system. Lionbridge seems to be very unprofessional in the handling of this problem. It undermines the project and the company’s reputation.  

But the fact that no one has caught the problem with timesheet disapproval or made any statement about this timesheet disaproval problem since it started happening in July leads one to believe in some kind of conspiracy here. Some people are calling Lionbridge scammers because of what is going on. If you’re suddenly asking a lot of raters to cut hours off their paychecks, there’s an obvious problem somewhere.

It does appear the company had the same kind of problem with timesheet disaproval in May in which they sent out incorrect data. It’s quite possible that this is the same issue, but instead of trying to figure out what is going on, it looks like the raters are getting the short end of the deal.

A company can’t just suddenly take away your pay when you put in the hours and did the required tasks to the best of your ability and as per your contract. There are some laws that protect freelancers and independent contractors, and if you are doing the work, then you should be paid. But for freelancers and independent contractors, your options are limited depending on what country you live in.  

Real timesheet correction required due to low productivity vs. Lionbridge error 

In some case raters might be asked to correct heir timesheet due to low productivity and it is OK, keep in mind that your average productivity rate  should be 1.5 – 2 minutes/per task.  

In this project, most of the tasks do not have any recommended time limits. Raters have to manage their own productivity and their own time. The customer does not indicate average estimated time because the priority is quality in rating the tasks rather than productivity, or how many tasks you rate. Raters have to keep in mind that the productivity and time invoiced should be within reasonable metrics. It is OK to spend more time in the beginning and increase your productivity when becoming a more experienced rater. 

Appen and Lionbridge have productivity metrics called RPH. It is assumed these metrics are based on the average raters’ performance in the project which is calculated as the ratio between the number of tasks and the time spent within an invoice period completed by all raters. In my view this metric is not useful, because in many cases raters get different types of tasks which require a different amount of time to complete, therefore the RPH metric is not accurate at all. This can lead to a situation where if your productivity doesn’t fall within their RGP metrics, your invoice will be returned for correction and they will ask you to align your RHP to average even though you spent a reasonable amount of time for completing more time-consuming tasks. The customer understands this issue. That is why they do not indicate the average estimated time. But it would seem that Appen and Lionbridge have again invented their own metrics to pay less to its raters. 

What raters can do in this situation?  
  1. To calculate the number of tasks completed weekly and entered into time sheet vs. Lionbridge records, if you have a huge discrepancy (like most of raters have) this is a clear indication of wrong data sent by Lionbridge. 
  2. Send an email to Lionbridge explaining the discrepancies.  
  3. If necessary as a proof of working hours/number of tasks a browsing history can be sent, it can be found, filtered by dates and website here: https://myactivity.google.com/myactivity
  4. Additionally I recommend sending an email directly to the Ads rating project admin (Google Ads Eval: ads_eval@google.com) to make them aware that this is some kind of issue that really needs to be resolved. 
  5. Do not correct time if you clearly identified discrepancies until Lionbridge responds to your email and provides you with correct data 

Perhaps if the right people receive a large response on the issue, someone will take a look and fix the problem. But as of right now things aren’t looking too good. For those of you who have lost wages, hang in there and hopefully, someone will get this straightened out.  

If you have questions or comments then leave them below.  

Ipsoot Project Review From Appen Connect – Pearl and Falcon PT

Ipsoot project 

There is a new project that we’re going to go over today called Ipsoot Project (also known as Pearl and Falcon PT) where you are shown a random post on Facebook, and then you have to decide what topic that post fits under. It is basically a categorizing system job which aims to put the posts that are made into the correct categories of broad topics. 

Ipsoot project qualification 

The qualification process for this project is straight-forward and not difficult provided candidates study and understand project guidelines. The qualification consists of 22 questions, candidates will be asked to apply categorization and assign relevant topic to randomly selected images. 

Ipsoot project Rating process

The posts you receive are random and in no order and appear on the left, including the ad and page information under it such as the page name, who the author is, etc. You will also notice on the top left a Post ID and a Job ID number. 

экрана 2019-08-05 в 20.44.33

The objective when working on the Ipsoot Project is to look at the post and spend 30-60 seconds making sure that you understand it. Of course, it is important to consider the text in the ad along with the media. Basically, you want to think about how a photo might be understood or interpreted in combination with the text in the ad. 

It’s important to pay special attention to everything that is in the ad, including the photo and any text that may be overlaid on the photo as well as any text appearing above the photo. 

If the ad is text-only it’s important to understand that text and including any emojis that might be used in the post to understand what the ad is trying to accomplish and how it should be grouped. 

Sometimes you will see a link in the ad, and at this point, you should click on the link and quickly scan the content of the page it takes you to and consider if the ad itself or the text relates to that landing page. 

Sometimes an ad will contain a video, and you should watch that video for up to 60 seconds. However, if the video is too long then watch the first 10 to 15 seconds, and then click through five spots and watch another 5 to 10 seconds. You should also consider any accompanying text that is inside of the video. 

On the right-hand of the ad, you will see a box that says ‘To which topic(s) is this post related. Select all that apply.’ You should select every topic to which the post is related on the other hand if the post isn’t related to any of the topics then select the no topic checkbox underneath the box of topics.  

Underneath of this, you will see a panel that says ‘Additional Attributes’ and you should select every attribute which applies to the ad. Of course, if there aren’t any attributes that apply to the ad then you should select the no attribute checkbox. And finally, the last thing to do is to click submit which will serve you up another random ad to check. 

Next, to the submit button, you’ll see some skip buttons. You will use these if you encounter any error that prevents you from evaluating the post and they are as follows:

Skip: Tool Error – You should use this if for some reason the tool doesn’t load. This can mean that the ad area will appear blank, or you may not see all the buttons or functions. 

Skip Other Language – the majority of the time you’ll receive ads in the language which you are assigned to so if you work in the United States your ADS will be either English or Spanish depending on what type of position that you apply for and if your multilingual. However, if you do not understand the language in the ad then you should do nothing else but click on Skip: Other Language button.

That’s pretty much it for the tool and it’s quite straight forward. 

The biggest challenge to this type of job is trying to understand Topic Relevance. To give an example, if it would make sense to see the post in an Instagram feed about a certain topic, then the post is probably related to the topic itself. 

However, it is not sufficient for a post to merely mention the topic. While it’s true that some posts explicitly mention the topic, some do not address the topic as it would show up in a search result. 

A good example of this would be an ad about a school competition winner who is pictured with his parents, his sister, and a dog. Simply looking at the ad isn’t going to help you unless you take a look at the text. So you wouldn’t tag this as “animals and pets” simply because there is a picture of a boy and his dog in the photo. However, if that same photo mentioned something about ‘man’s best friend’ you could tag it as animals and pets.

One of the questions that you may have is ‘What if the post is about more than 1 topic?’ Well, you should pay particular attention to a post that can be related to multiple topics because many posts can be coded with more than one category. 

We’ll give you some examples. 

For instance, let’s say that you have a post about a new economic policy. In this case, you would tag ‘politics’ + ‘business finance and economics’.

If a post was about someone using a new fitness routine to improve their health, then you would tag ‘health and medical’ + ‘fitness and workouts’.

A post about a politicians stance and welfare programs you would select ‘politics + social issues’ 

If the post is about the best restaurant three friends found while traveling through Italy then you would tag it ‘food and drink + travel and leisure + relationships friends and family’ 

As you can see, it’s quite possible many posts can have more than 1 topic. 

There are a few other rules when it comes to tagging topics about a post when it comes to regard for quality, professionalism, or the size of the audience for the post. 

We’ll show you some examples of how a topic doesn’t change regardless of these attributes. 

If the post is a professional video of a Justin Timberlake concert then you would tag this under ‘music and audio’.

But let’s say that the post is of a band performing in a bar. You would still tag it ‘music and audio’. Even though that band may not be as popular as Justin Timberlake.

Even if it is a post of a cell phone video of a 3-year old singing in a car, you would still tag this is as ‘music and audio’.

However, it is important to pay special attention to whether the post is related to any entity such as unique people, places or things. For example

The post is about President Trump – You would tag ‘politics’

If the post was about a McDonald’s restaurant, then you would tag ‘Food and drink’

And if the post was about The Beatles, then you would tag ‘Music and audio’

However, you should be careful to not assume that every post that mentions an entity is it about that entity itself. For instance, a video about the Olympics might show a logo of its sponsor, McDonald’s, but that is not enough to consider the post to be related to McDonald’s itself and you wouldn’t tag the post ‘food and drink’. 

When the author of the post is a well-known individual like Barack Obama, and he’s posted something saying that the Chicago Bulls are the best team ever, you would tag this as politics because it’s just a personal opinion by an individual strongly associated with politics. 

If you use Facebook, you probably understand that memes are something that you see quite often and its important to understand how to tag a meme. Memes often used content from one topic to make a statement about another topic. In the case of memes, you would actually tag both topics. For example, if you see a meme of a TV character looking sad with the caption ‘how I feel when my pet walks away from me’ you would tag this TV & Movies + Animals and Pets.

There’s going to be a lot of times when you’re going to have to do research because you’ll find names or phrases in a post that you do not understand. You should always do a search to find out if you’re not sure when referencing a topic. 

It’s also a good idea to identify some of the definitions in the guidelines. 

Topics describe the subject of the content. It is important to determine if the content in a Facebook feed is relevant to the topic. This means that it’s not sufficient to call something ‘related’ to a piece of content that merely mentions the topic. There are some examples of topics and how they should be grouped. And although the guidelines go over each one and give inclusions and exclusions, it is not an exhaustive list but merely examples.

https://passleapforceexam.com/2019/03/30/appen-connect-platform-list-of-projects-and-code-names/

If you are working on the Ipsoot Project then we would like to hear from you and your experiences. If you need help with any information on this page, then email me and I’ll get back to you as soon as possible. 

Exam Preparation Tool For Appen Arrow and Lionbridge Exams

WHY CANDIDATES FAIL ARROW/ADS ASSESSOR EXAM?

An Exam Preparation Tool has just been created to help you pass these difficult exams you are given when you apply for ads evaluation jobs, you have to pass a qualification exam that is quite difficult. Each company has a name for their ads assessor project. Appen Ads Assessor project is called Arrow if you work for Lionbridge you work on the Ads Assessor project.

Despite all the fancy names, the exam and rating process are the same for these companies because the customer is one, only vendors are different. The biggest problem with this exam is that it is difficult to pass this qualification with only one attempt. In fact, even Appen and Lionbridge explain in a welcome mail that there is a high chance of failing it. With that said, you’re going to need all the help you can get. With the Exam Preparation Tool you can be sure you’re ready.

Keep in mind that unlike Yukon project/Personalized Assessor and Maps Analyst (Shasta) projects’ qualification, the Ads Assessor/Arrow project qualification allows only one attempt to pass the exam.

STUDYING THE GUIDELINES AND DOING THE EXAM SIMULATION PRACTICE ARE ESSENTIAL IN SUCCESS IN PASSING THE ACTUAL EXAM

If you think this is an exaggeration, then just search for comments on passing the exam and see what you will find. You’ll find in a lot of cases people who claim that they studied for several days before the exam began. With that said, it’s just not possible to demonstrate the knowledge of these guidelines without investing a significant amount of effort in the one-week time frame they give you to study the guide. The guide itself is designed in a way that assesses your ability to understand and apply rating concepts and the rules based on instructions. There is a lot of material to go over, and it can be a bit confusing. But once again, you’re not going to get anywhere near passing if you don’t start going through the guide.

WHAT IS ADS EVALUATION EXAM SIMULATION TOOL?

The tool will help anyone who is getting ready or considering taking the exam. This tool was designed to assist candidates with exam preparation with the main focus on practical rating tasks that include: Search Ads Evaluation, User to Keywords, User Interests and Close Match Evaluation tasks. The tool has 10 Search Ads Evaluation tasks, 7 User to Keywords tasks, 5 User Interests Evaluation tasks and and 5 Close Match Evaluation tasks, these tasks are the main exam tasks types which cover basically all rating concepts, which will help you to develop rating skills which will be tested during a real exam.

HOW USING THIS TOOL CAN HELP YOU PASS THE EXAM?

You might be wondering how using this tool can help you pass the exam. The way that it is designed will show you exactly your problem area that you need to focus on. It helps you identify your level of understanding of the rating concepts, and your ability to apply them by practicing first, and then identifying your weak points. You will receive your overall score (pass or fail), detailed feedback on your mistakes and know exactly what you need to go back and study in the guide. Of course we need to disclaim that there are no real actual exam tasks because that isn’t going to help you understand anyway. Each one of these is hand designed from most typical exam tasks, and is designed to be the best simulation possible. If you pass this, you will be ready to pass the actual exam.

HOW TO PROCEED WITH EXAM SIMULATION TOOL?

Exam Simulation Demo

If you’re still skeptical, we have a free demo version which you can try FREE right now! It contains 1 Search Ads Evaluation task, 1 User to Keywords task, 1 User Interests Evaluation task, and 1 Close Match Evaluation tasks so you can see how it works. Once again, the demo is 100% free and you’re completely anonymous.

Exam Simulation Full Version

The tool has 10 Search Ads Evaluation tasks, 7 User to Keywords tasks, 5 User Interests Evaluation tasks and and 5 Close Match Evaluation tasks, these tasks are the main exam tasks types which cover basically all rating concepts, will allow to develop rating skills which will be tested during a real exam.

EXPLORE ADS ASSESSOR/ARROW EXAM SIMULATION TOOL HERE:

https://adsevaluatorexamsimulation.passleapforceexam.com/

iSoftStone Ad Copy Bing Ad Evaluation Project Review

Bing Ad Evaluator Project Purpose

The iSoftStone Ad Copy project is a Microsoft Bing analog of the Google ads rating project for Appen called the Arrow project (Ads Assessor at Lionbridge). An evaluator gets queries and your job is to evaluate how satisfied users would be with the ads proposed by Bing search engine. The purpose of the project is to help Bing to decide which ads are the most helpful and the most relevant to a
searcher. This, of course, improves the overall quality of ads that Bing presents to its users. It sounds like a daunting task, and it can be confusing at first.

iSoftstone is another vendor and is regarded as the main vendor for Microsoft Bing. They also have the 3 test qualification for the iSoftStone Ad Copy and that test seems to be the same no matter who you apply for. The first test goes over the guidelines, the second test is an English Grammar test, and the third exam tests your computer and internet skills. All work is done on the HRS platform and you have to complete a qualification for each task. 

Rating Logic In the iSoftStone Ad Copy Project

There are however basically only five steps when it comes to determining if the ad
is going to make the user happy by satisfying their needs.

  1. Consider what the user had in their mind when making the query, or their intent when they typed the query. You need to take into consideration that
    people also sometimes misspell something when searching and in most cases you
    will understand what they meant.
  2. Do some quick research to understand the meaning of the query unless you
    understand it immediately.
  3. When presented with the ads, determine whether the Ad covers any of the
    user intent and how well it satisfies them
  4. The ad is then given a rating based on a seven-point scale to indicate how
    useful the ad would be to users who type in the query.
  5. Sometimes you need to explain additional information on a rating and explain
    it.

If all that still sounds tough, there are some tips that can help as you perform the
rating job:

 When considering the ad, make sure you are reviewing the content that is displayed. No, you don’t have time to read over the content of an entire website, but a quick glance at the top-fold of any website will tell you right away if the ad is relevant or not.

 Often you will need to research possible intents with a query but you should never base your rating on how the ad is treated in the search engines.

 The rating you provide is based on the query which is the keyword or string of keywords that a user types into the search engine. You should consider if it’s a perfect match or not. Ask if the query is broader than the ad or if the query is narrower than the ad.

 Don’t base your rating on a particular item like the title, description, or the
display URL in the ad.

 Examples are in the guidelines and the guidelines should always be handy in case you get stuck on something and aren’t sure what to do. Of course,
the queries in the guidelines aren’t exhaustive but they do give you a basic idea on many types of ads that you will run into.

Bing Ad Evaluation project Qualification process

The qualification process for getting into the Ad Copy project for online ad
evaluator is just like most other applications for search engine rating projects.
When you apply at isoftstone for the iSoftStone Ad Copy project and are selected to go
through the training process, you are sent an email telling you that you’ve been
added to the training group on the system with a login.
It is important to understand what you need to do in order to pass training:

  1. The most important thing that cannot be stressed enough is to read
    through the guidelines as many times as you need to fully understand
    them.
  2. Once you feel confident enough to try out what you know, there is a
    system that gives you what is called training hits where you can start to judge some ads. Hits are refreshed every so often on the system and you need to aim for 70% correct accuracy because this is the requirement to continue on and qualify for the project. If you’re not getting 70% then go back to the guidelines and read them again.
  3. If you are confirmed by the trainer to be abler to achieve 70% accuracy or
    better, then you are added to the qualification group to take the exam.

IF YOU FAIL THE EXAM MORE THAN TWICE THEN YOU WILL NOT BE ABLE TO
CONTINUE ON…SO MAKE SURE YOU ARE READY!

Once you pass the training the company will issue you a contract and you can begin working.

type of tasks and rating scale

When doing the job, the most important thing that we mentioned before was that you should understand the query and the intent of the user when they were typing it into the search engine. So you must evaluate it by first understanding the query meaning and if not then you must do quick research to understand it before you can determine the query intent. To determine the intent of the query,
you need to consider why the user entered the query and what the results the user would have liked to see. There are three basic intents; Navigation,
Information, and Transactional like websites with a product or service that can be purchased.

The next thing, of course, would be to evaluate the level of specifying and ask yourself; do the query and the ad match perfectly or not. With that said, you
should always disregard marketing terms like “cheap”, “best”, “highest quality”,
“superior”, “great”, and “greatest”. The meaning of these words is arbitrary because what may seem “cheap” to one user may not be cheap to another.
The guidelines go over everything from how to understand the ad, how to best research the ad, and then give you the correct options for rating it. Ads are given
a rating from perfect to bad and plenty of examples are given in the guidelines
that cover just about everything you are likely to face.

Where to apply & requirements

If you are interested in applying for the job, then iSoftstone is currently hiring for
ad evaluators and you can do apply here: https://issworld.isoftstone.com/JobDescription/tabid/104/jobid/9630/title/Onlin
e%20Ad%20Evaluator%20-%20United%20States/Default.aspx?language=en-US

The requirements for getting this job should be a good computer running a
Windows operating system (sorry MAC users) with a high-speed internet connection. You should have some strong computer skills for this job and strong communication skills when you are evaluating ad copy and have to provide additional information. You also need to make sure you are fluent in the language that you applied for and live in the country where you applied as well. The current
job openings want a native proficiency in English and must reside permanently in
the USA.

If given a contract, you are an independent contractor and can usually get 10-25
hours a week if you stay in the accuracy range you need to be in. Currently, the company doesn’t pay for the training and instead says that it’s a ‘free training’ to make it seem more attractive. However, the time involved in getting ready for this does take some hours that should be paid by the company. It could be that training is not paid because many people fail the exam. So make sure you read the guidelines like they were the bible.

As always, if you have any questions, then please send me an email or leave a
comment and I will always reply right away and help you with anything you might
need.