I have created a special tool that will help anyone who is getting ready or considering taking the exam. This tool was designed to assist candidates with exam preparation with the main focus on practical rating tasks that include Page Quality, and Search Results Satisfaction ratings. The tool has 5 Page Quality tasks, and 5 Search Results Satisfaction (Needs Met) tasks of all the major query types. (If you’re not sure what a query type is, that is the keyword that a user searches Google for). These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/tasks types and skills which will be tested during a real exam.
How can this tool can help you pass the exam?
The way the tool is designed will show you exactly your problem area and what you need to focus on. It helps you identify your level of understanding of the rating concepts, and your ability to apply them by practicing first, and then identifying your weak points. You will receive your overall score (pass or fail), detailed feedback on your mistakes and know exactly what you need to go back and study in the guide. Of course we need to mention that there are no real actual exam tasks because that isn’t going to help you understand anyway. Each one of these is hand designed from the most typical exam tasks, and is also designed to be the best simulation possible. If you pass this, you will be ready to pass the actual exam.
How do I get my hands on the exam simulation tool?
If you’re still skeptical, we have a free demo version which you can try FREE right now! It contains a simplified Needs Met and one Page Quality task so you can see how it works. Once again, the demo is 100% free and remain completely anonymous.
The tool has 5 Page Quality tasks, and 5 Search Results Satisfaction (Needs Met) of all the major query types. These include; simple-know, local search, information search, website/navigation, visit-in-person/business which cover basically all rating concepts/tasks types and skills which will be tested during a real exam.
If you’ve ever wondered how it is that Google is able to give searchers the results they were looking for, it all began back in the early 2000’s when people allowed Google to have their data in exchange for Google products and the possibility of seeing more relevant ads. The whole idea is to provide marketers with data that can be exploited. And by the way, if you were ever curious about whether applications can listen to your conversations or see things through your camera, there really wouldn’t be any need because the data that Google gets would tell so much more about you and your habits. It’s really not that hard. With that said, there really isn’t any reason for some company to hijack your camera or microphone.
This is due in part to your search habits; I.E. the things you do with your Google accounts, whether that be Google Play or Google Email, and this also includes searching using the Google search engine for whatever it is your after. And Google gets a LOT of searches. There are about 40,000 search queries on Googles search engine every second. That comes out to around 3.5 billion searches per day, which equals around 1.2 trillion searches per year.
Every time you search for something using their search engine, your query travels to a data center where an army of up to 1000 computers work together to retrieve the best results based on your query and deliver them back to you. It takes just one-fifth of a second to get those results back to you, but you probably already understood this knowing that it doesn’t take too long to get back search engine results.
Google has several methods for you to see what data it collects :
You might be able to figure out what you were doing 3 years ago on this exact date, including what video you watched on Youtube and what searches you did.
However, what most searchers don’t realize is that there is even a faster and more mysterious process that is happening.
User Profile & Google Ad Systems
As many of you know, when you search something you are entering in keywords, and those keywords are fought over by advertisers, because they are likely offering a product or service that are related to your keywords and they want their ad to be seen and clicked on by you. These are generally the first four search results at the top before having to scroll down. You might recall that before this, Google had some ad boxes on the right of their screen. Those were done away with and instead now you simply see top links which are paid ads. Actually, if you didn’t know this, then you’re not alone, and unless you did, you wouldn’t know those top search results from paid to organic except there being a little tiny box under the link that says ad. Now you can probably see why Google got rid of the ad boxes on the right hand side that people generally wouldn’t click on. Once you click on one of these ads, your information is passed right to search engine marketers where its stored forever in an AdWords account which is never erased.
What is Google user profile?
And if your curiosity goes even further than what you’ve already read, there is a huge checklist of everything that Google knows about you. In fact, it would be much simpler to tell you what Googles DOESN’T know about you than what they do. The thing is that, there really isn’t much to add to this list since everything you do online builds what is called a ‘citizen profile’ on every person who uses Google search or any Google products. Everything from your age, income, gender, sexual orientation, parental status, relationship status, your location, the device you are using, your long and short term browsing status, the products you buy, and even the contents of your email and how close you are to a cell tower is known. Talk about Big Brother.
But things are probably about to get even more interesting. In 2019 there will be what some people have called the Holy Grail of search engine marketing: multi-device attribution. When this technology is realized, as will follow searchers seamlessly across all channels and all devices meaning; social, organic, email, and from mobile to laptop, and laptop to television, etc.
Imagine watching a football game, and an ad comes on for Nike shoes. Your television will emit a super high frequency undetectable by human ears, but can be picked up by a nearby smartphone. So, picking up your smartphone, you suddenly see an ad for Nike shoes the first place with ads you visit.
The thing is that internet marketers know so much about you, and if you’re a daily commuter, they show you ads that people on buses and trains would be interested in buying like some cool new headphones or a nice backpack to carry your stuff in. By the way, if you were curious how marketers could possibly know if you are a daily commuter or not, keep in mind that the smartphone you carry pings passing cell towers. If you are passing the towers uninterrupted at a fast speed, then you are most likely on a train. Stopping once in a while and then moving? Bus or a car. But that’s not all, search for some product on your phone and then physically walk into some department store and chances are good that Google used your phones GPS data to connect your ad click to your in-store purchase. And if you have ever searched for anything non-product related, such as some health problem such as high blood pressure, be ready to see ads based on fixing this problem across all your devices. Google is getting smarter all the time.
Today, 3 out of 4 smartphone owners use Google first above anything else to find what they are looking for or buy some product. You are then presented with a whole bunch of ads or results. These ads speak to what you seek, play on the emotions that are unlike you, and fit your age, income, gender, location, and browsing history, not to mention all the other things that were previously mentioned.
But something changed a while back that helped Google get the most relevant search engine results to you. Google might actually sound like the bad guys, the ones who are snooping in everyone’s business to find out what you like and what new toy or service you are looking to buy.
How Raters Improve the Google Algorithms?
Google is simply a business trying to provide the best service they possibly can. It’s their business model: Provide the best and most relevant results to anything you might be looking for or whatever may solve a certain problem you are having. Their idea is you involve as many people as possible by providing an excellent search experience and related online services as well, such as YouTube and other internet giants.
And unknown to a lot of people, this whole gathering and fine-tuning of information isn’t completely automated….
This is where the Google search engine rater comes in. Google hires company’s like Lionbridge, Leapforce, Appen, etc. to help them sort through all the search results and give feedback on the websites that meet or match the needs of a search engine query. Sometimes we don’t always find what we are looking for in the first search. But search engine raters sift through the stuff that isn’t so good, and give good marks to the websites that are actually providing the stuff you need. In fact, most rating projects focus on content improvement of these two business streams of Google which are the Search Engine evaluation projects to improve search system, and the ads rating projects to improve ads targeting. Search engine raters are the ones who actually improve the results. Sure you can game the system as an internet marketer and use a bunch of black-hat techniques to improve your chances of being at the top of search results, but when a search engine rater gets a website up for review, that website better fit the Google guidelines.
Google is constantly evolving and will continue to do so. They are always changing their algorithms, and doing their best to weed out the bad from the good. However, the search engine rater’s job is very important in giving Google the feedback it needs. There are two major Google rating projects: Yukon/Internet Assessor/Inca which focus on Search Engine Improvement and Ads Rating/Arrow/Aztec which focus on improving Google ads system.
How Raters Improve the Google Algorithms (video)
It would be very interesting to see Google stop using these virtual assistants one day, but the chances of it happening are fairly slim. Search engine raters have done a fantastic job of helping fine-tune the search engine giant over the years, and look to continue doing so.
In this post I will discuss some common issues that new raters face. If you have other issues, feel free to share by leaving a comment below.
What kind of tasks to expect while providing search engine evaluation service for Appen/Lionbridge?
If you get a position as a search engine rater with Appen or Lionbridge (Internet Assessor), you may encounter many different tasks. Some of these tasks are your normal rating tasks, and some are side by side, and you may even run into all kinds of experimental tasks. Some of the tasks are easier than others, like experimental tasks. Each task has a different set of instructions, so it’s best to make sure you know exactly what you are doing, and read very carefully each of the task instructions.
Yukon Project Side By Side tasks
Side by Side rating tasks are very interesting and have everything to do with Google algorithm changes. The human role or ‘rating’ in Side By Side tasks is to determine which side, left or right, is better than the other. In this task you are shown blocks on the left and right side. Each block has a slider from Fails to Meet to Fully Meets, and you should choose which one is best for each block depending on the user intent and query. The same guidelines apply for the blocks as they would for a page rating. Meaning that if block meets user intent, it should get the appropriate rating. Once you have chosen the ratings for each block, you will then be asked to choose on a scale which side is better than the other. A comment may be required to give your own opinion on why you’ve chosen which side, and why it is the best over the other side. It’s very important that you base all your ratings on user intent and taking their location in mind. You will also have a quality review on Side by Side tasks once in two months.
Yukon Project Page Quality tasks
This is the main rating task that a rater is hired to rate. You will be given a website URL along with the first 5 questions which check things like foreign language, which if you’re doing English/USA ratings, would need to be in English to be understood from your location. Pages that don’t load need to be checked, as well as flagging for porn.
The second question is asking you to describe the purpose of the page. The purpose of the page and the user intent are entirely different aspects. Make sure you’re checking the website out to see if it matches user intent, and by looking at the website, you should be able to understand what its main purpose is. If the page is malicious, harmful, or just lacks its purpose for what it is searched for, then you should mark that here. Pages that are simply existing to show ads, without any help to the user, should be flagged. And lastly, it is important to make sure that you understand if the page is a YMYL (Your money or your life) webpage. Understanding this is important because it will change the overall rating of the website.
Once you have completed this, you will then rate the different aspects of the page including its main content quality, and the amount of content. Expertise, authority, and trustworthiness, who is responsible for the website, the supplementary content, page design, layout, and website care and maintenance are all graded on a scale from lowest to highest. The last thing to do is to give the website its overall rating. Usually these types of tasks are allowed 9 minutes to complete. A quality review is done each month to see how you rate. So it’s important to make sure that you are taking the time to evaluate each website properly.
Appen Yukon and Lionbridge Internet Assessor Needs Met tasks
This type of tasks you have already saw while doing part 3 of the qualification exam. Each task contains 8-12 result blocks. In each task you will be given a certain query, and a list of search results blocks with links where you have to give a usefulness rating to search results using a scale from ‘Fails to Met’ to ‘Fully Met’
Other Experimental tasks
Sometimes during rating of side by side, you will see experimental tasks pop up. In most cases experimental tasks are standard page quality or need met tasks, but sometimes you will get new types of tasks. Experimental tasks have a wide range of instructions. It’s important to read the instructions for each task as they will be different each time. Experimental tasks can be very simple, or take a few minutes to complete. They are not always available, and each task can be completely different than the other.
A wide variety of tasks require comments; others do not require comments but giveyou the option to leave them. While it may seem like comments just put time pressure on you and end up disappearing into a black hole. Your commentshelp Google analysts understand how you interpret queries and the logic behind your ratings.
Additionally, while block-level comments are not required, they can make Google awareof different experiences raters may have had with a result.
The more information we have the better, so please make sure your commentsprovide an appropriate amount of information. Here are some basic guidelines forcommon types of required comments:
SXS comments always need to include what you think the user intent is, whichresults are good/bad and why that is, and what led you to prefer one side overanother.
PQ comments should not just repeat grid contents (“All the characteristics in thePQ grid are high, so the overall page quality is High too”) – rather, they shouldexplain why a page’s reputation/EAT/MC/SC/etc. are good/bad and why, giving a morestep-by-stepwalkthroughofwhyyougave a particular PQ rating.
New raters should not do live tasks immediately after getting access to the task portal. It is highly advised that you hold off on rating live tasks if you do not want to be restricted for quality in the next month. Typically tasks which are available for new raters are quality review tasks on which raters are accessed for their quality. Instead, new raters should focus on doing simulation tasks and self-education. Time spent doing this will really pay off, and you wont be restricted for quality in the future. Also, by doing a certain number of simulation tasks correctly you will upgrade your status in the system, and will be allowed to work more hours.
During your participation in the project your service will be evaluated for its quality by means of quality review tasks. In the beginning of every month, a certain number of tasks are uploaded into the system. Normally there are 10-12 tasks of Needs Met or Side by Side, and once in 3 months PQ tasks to be released as quality review tasks. These tasks are quality review tasks up on which your quality is measured every month, in the end of the month you will get your quality score feedback. In case of raters get substandard quality for several consecutive months, most likely such raters will be removed from the rating program.
If your ratings often stand out of general range of ratings provided by other raters in the project, most likely your account will be flagged by the system automatically and your work will be checked manually for quality .
recording and invoicing Working time Yukon and Lionbridge internet assessor project
Time management (recording and invoicing working time) of the work you complete can become a problem if you aren’t managing how much time you are spending on tasks. First of all, you should not sacrifice quality for time. In the beginning you may spend more time than the average estimated time (AET) that Appen or Lionbridge allocate to a specific task, but make sure that in the invoice you put average estimated time that they give you. If you spend less time than average estimated time per a task or set of tasks, be informed that your invoice will be returned for correction, Appen and Lionbridge will ask you to put actual time spent, that is why there is no sense to complete tasks faster than average estimated time. Take your time, do everything correctly and watch your time.
the Yukon Project Extension
Do not forget to log on to the Yukon extension each time you work in the system, otherwise your time spent on doing tasks will not be paid.
According to the vendor policy: any time spent logging on/off, reading, studying, and break times are not billable. Any tasks that are not submitted or are released are also not billable. Please make sure you are not including any of this time on your invoice.
Also keep in mind that consecutive invoices submitted with productivity below 100% may result in removal from the project or you may lose your contract with Appen.
Productivity and Average Estimated Time (AET)
When it comes to the time spent working on a task, the listed AET is meant torepresent the average amount of time that a task is expected to take. In general, it isbest to complete tasks near the given AET, but we understand that some tasks maytake a bit longer than others. If more time is needed to complete a task, weencourage you to take it within reason. The overall goal is to complete tasks within10% of the given AET. When productivity falls below 90%, a Low Productivity Noticeis typically sent out.
Appen Yukon project Invoice Returned for Missing or Incorrect Data
Lots of new raters receive the following emails from vendors:
Your invoice has been rejected for missing or incorrect data. Please make the necessary corrections or provide specific details on the reason for each discrepancy.
Time Discrepancy: For the following line item(s) there is a discrepancy between the hours you have submitted and the activity and work recorded on our servers.
Reported: x hours, xx mins Unaccounted Variance: -xx mins
What raters should do in this situation?
In order to get invoice approved, raters have to deduct xx unaccounted variance minutes from the invoice, in the future make sure only AET is entered into invoice and the Yukon extension has to be on during active rating.
Payment method for Appen, Lionbridge and Raterlabs
HOW to add Already existing Payoneer account at appen connect personal project page
How to earn additional 25$ by signing up to Payoneer via RefeRral Link
Appen Yukon get locked or restricted for Productivity
Working on some tasks much faster than the AET followed by holding on to a task for much longer than the AET and billing for the full amount on your invoice
Raters are allowed to bill for the AET of a task OR the actual time spent working, whichever is less of the two amounts. It has been discovered that some raters working some tasks much faster than the AET followed by holding on to a task for much longer than the AET and billing for the full amount on invoice, which is a violation of official policy and will not be tolerated.
For example: Rater works 5 tasks, each with an AET of 5 minutes. The first 4 tasks rater completes in 1 minute each, and the 5th task rater holds on to for 30 minutes.
If this pattern continues, invoices will be rejected and raters will be removed from the rating program. Be sure that all invoices are in compliance with policy.
Appen Yukon & Lionbridge project multi-accounts
According to Appen Yukon project policy you can only have one account per IP address, while Lionbridge allows to have one account per household. Those pretty much mean the same thing. There can’t be two accounts coming from the same internet service and this requirement come from Google system requirements.
When it comes to Google rating projects or Facebook related projects, Appen and Lionbridge can find duplicate accounts very fast, and as a result, you will end up losing your accounts and they will be terminated. You certainly don’t want that to happen.
Be informed that it is prohibited to have multiple accounts, without special arrangement. Appen and Lionbridge can find duplicate accounts very fast, and as a result, you will lose both accounts.
There are a few things you can do to get around this problem, and arrange them in way that Leapforce, Lionbridge, and Appen won’t be able to detect.
IP address change and network sharing while working in Appen Yukon and Lionbridge projects
In some cases people may share their network, while working as search engine evaluator, this case leads to IP address conflict and removal from the project. Make sure that you have your personal unique IP address and no one uses the same IP to go online for rating project.
When rating tasks on raterhub platform you may change your IP address many times, it will not affect your project status. Make sure that you do your project tasks in a private place and in a country of your project locale.
Non-completion of a minimum required number of tasks
This issue is very common among new Appen Yukon project agents and may lead to termination of contract. This happens when new agents show no activity in the first month of service. In particular, this can happen if new agents enter the project in the second half of month, and for whatever reasons, they did not complete a minimum required number of tasks. In order to avoid this issue, make sure to show the minimum required activity in the raterhub system.
Yukon and Lionbridge project tasks availability and new rater time restriction
In general, Appen and Lionbridge guarantee at least 2 hours of work per day but task availability may depend on project locale. There might be some days when tasks are available for more than 4 hours. Some locales, like the US project provides raters with 24 hrs/7 days tasks availability.
For the rest market task availability may be limited. You should understand that the system of task distribution is based on a ‘first come first served’ basis and you need to be ready to catch your portion of tasks when they are uploaded into the system. In order to identify time slots when task availability is the highest please look at task completion graph, usually tasks are uploaded every day into the system at the same time slots.
Lionbridge does not provide raters with a task availability instrument, so raters have to identify time slots when tasks are uploaded into the system by themselves.
Appen also has a system that restricts access to tasks for new raters, and those who are undergoing their quality review. To lift this restriction, you have to complete a certain number of simulation tasks or wait till the next quality review, if your quality score is good, the restriction will be lifted automatically.
Appen Yukon and Lionbridge raters get locked for quality audit or restricted for quality
If you receive an email notifying you that your account has been placed under quality review, this is a good time to identify your weak points and improve your rating skills. Below you will find some tips on how to deal with a “Locked for quality audit” situation.
To speed up the process and to get feedback on your mistakes as quickly as possible, I recommend writing to your vendor and ask why exactly your account has been placed under review. Sometimes the automatic system locks raters accounts without specific reasons. Usually such cases are being resolved within 1-2 days.
While waiting for detailed feedback from your vendor, proceed with doing simulation tasks in order to improve your rating skills. In most cases the issue is related to Side by Side tasks. Make sure that you understand how to deal with this type of task and that you provide well written comments where required. In many cases, raters are placed under quality review due to poorly written comments.
Once you receive the feedback email with your mistakes/quality issues, read it carefully and reply back with an explanation on how you will improve your weak points.
Once your status is moved to “Restricted for quality”, keep in mind that your service is under inspection, thus making sure that you do your best to provide the best possible service.
Appen yukon raters hold status restricted for quality after receiving meets expectations quality feedback
Sometimes some raters do not get their restricted status changed to unrestricted with getting a meets expectations or above quality score. This happens because they were restricted for side by side or PQ tasks quality issues and feedback was released for needs met tasks. In order to lift up all restrictions a rater will have to wait for the next quality review cycle for type of tasks a rater was initially restricted for. If you get meets expectations or an above quality score your account will be unrestricted again.
Have you ever wondered how many hours raters work?
Knowing these statistics can be useful for raters in terms of estimating possible productivity .
The survey is completely anonymous and once completed will show the overall statistics.
If you would like to obtain a job as a tester, you are probably wondering what sorts of skills does it take to become one. I would definitely recommend starting with some basic tutorial on “how to test”, you can find those on official uTest Youtube channel, and there are couple of good books for beginners: “Software Testing” by Ron Patton; “Agile Testing: A Practical Guide for Testers and Agile Teams” by Lisa Crispin and many others. But to become a good tester these kind of skills you need to have for starters:
– Great English skills (written);
– Analytical Thinking;
– Attention to Detail;
– Thinking Outside the Box;
There are no restrictions on amount of work, but it really depends on the platform and your rating as a tester. There are projects that I get invited daily. The project really depends on the hardware you have. Mostly there it’s a web or app based projects. For instance – some hotel website, or booking service. There is so much to test, starting with a web forms, ending with validation of every input field. They are easy for beginners to start finding bugs right away. I worked from US, and got what the platform offered me for the region, but I do know that project can be different depending on country you are working from.
Types of tasks
Usually is’s a whole functionality testing of some elements of the site. For instance: the customer wants a tester to be precise in Log-in page. So the tester focuses on those bugs which occur only on a login page, not considering the other parts of the website. Sometimes there are localization task. Let say you know Chinese, and customer requires the testers to check the spelling and grammatical errors of specific language.Most of the tasks are ad-hoc (exploratory testing) for functional issues. (App crashes, buttons not working, etc).
How to become a successful rater at uTest and make decent money
But there is no deep inside mentioning of a rating system and how it works. Let me explain why is it so important on uTest. This will be a more “in depth” article on earning on uTest. There is this system where it takes your results (amount of submitted bugs) and amount of rejected bugs and counts the successful percentage rate.
Based on this rate there are classifications of testers: Rated; Proven; Bronze; Silver; Gold
And there are different Testing Types, for each of which there is a rating: Functional; Usability; Load; Security; Localization; Automation
But don’t worry, most of the tests are functional and Usability, and polishing skills on these Testing Types can set you with a good additional income for each month.Getting more in those ratings, as I mentioned earlier it is the key filter for uTest to invite you to the project. To become a Gold rated testers it takes quite some time and dedication. Not to mention the quality of your bug reports should be on very high level.
Once you get a couple of rejections it is really hard to achieve the Gold rating. The highest rating gets you not only the more amount of offers, but you also get more paid, and those rare and exclusive projects available only for best testers out there.
Apart from this rating there is also a very cool and interesting opportunity on becoming a “Favorite tester”. Once you have basic rating of Silver or Gold you might end up becoming the one.
Let me explain simply how it works. There is certain person which manages a test cycle a TTL or Testing Team Lead. This person is responsible for other testers to follow the rules and code of conduct of Utest platform, and the project in particular. Sometimes there are even NDA’s which are needed to be signed. TTL is responsible for all this and more.
Let’s assume you have submitted 10 bugs in 1 test cycle, and all of those issues are either very valuable or exceptionally valuable bugs for the customer, Testing Team Lead can easily forward your account to the customer, so that they can give you a “Favorite Tester” status.
What does it mean? Well, you get to be automatically invited to each testing cycle this customer has and many more. Some customers always slot a test case for a favorite tester, which leads to more earnings.
I’ll go into more detail with the favorite tester part and test cases. Why is its important?
How much really you can make of uTest?
Once you become a favorite tester, usually it’s a big project with a big company, and the pay is really good. Some test cases can be up to 50$ per/case.
Once you start doing those cases, simultaneously it is easy and very practical to look for the bugs, (the inconsistencies between an actual and expected results). It really pops in the eye, once you get going through a test case.In that moment when you have found an issue and reported it to the system, apart from getting those 50$ for the test case itself, you can earn a lot more for reporting bugs additionally.
Talking about rates paid for bugs filled… Well, it absolutely depends on the person, the amount of details he/she is giving, the amount of devices the person holds, and the overall work ethics a freelancer sticks up to.
Bugs may vary from 3$ for “Somewhat valuable” to 30-40$ (per bug) for “Exceptionally valuable” issue, depending on the customer.
For sure you are willing to know the real numbers, and the sacramental question needs to be answered.“How much really you can make of Utest”? I will repeat myself stating that everything depends from the person. But, you can crank out a fair 1000-1500$ per/month of income depending on your dedication.
Everything is done via PayPal and Payoneer. Sometimes the bugs are not approved by customer for as long as 35 days. After that period, they are approved automatically by the system.
Use this link for applying for signing up to Payoneer and we will both benefit by earning 25$, once your account is loaded with payments.
There are many factors on becoming successful on this platform. Speed is one of them.
Sometimes it’s critical to start testing immediately, as other (hungry) testers will beat you to that critical issues, while you’re yawning.
The same rule is applied for the test cases.
Test Cases are usually limited, and not everyone can claim them, so it’s lethal to be fast.
To make the numbers I mentioned earlier, it really takes time to pump your account to a certain rating level. Usually it will take around month or two to get going, but hey, Rome wasn’t built in a day either.Once you become an experienced tester and acquainted with a platform like a “Pro”, you actually might get invited easily on becoming a Testing Team Lead.
Who are Testing Team Leads?
The roles of this person are really dependent of the customer you are working for.
My roles were:
Writing test cases;
Participating in meetings with other TTL’s, Project Managers and Software Delivery Managers;
Create and control a test cycle;
Check and forward bugs for the customer (yes, TTL has the power to approve or decline bugs).
The income story is completely different, as the TTL starts working on per hour rate, and once again it really depends from the customer on the rate. (It is near to a very entry level junior position in software testing working for an onsite company in the States).
The benefit of becoming TTL on one project is that very rapidly you become noticed by others, and easily can get invited to manage some other cycle.In addition, TTL’s are still allowed to do the functional testing as other participants, though some customers forbid this.
In conclusion, now that I completely moved to an onsite job I can say with a share amount of confidence – Utest is a great place to start. It takes that will power, but sure does pay off. Good luck fellow freelancers!
Bing Search Engine Rating Project (Side by Side Judge, Multimedia Judge) vs Google projects (Yukon/Internet Safety/Internet Assessor)
When getting into jobs as a search engine evaluator, you will find that one company will produce ratings for one search engine, such as Google, while another company will offer jobs as a rater in Bing which is Microsoft’s Search Engine. Like Google, Microsoft also needs research and pages rated just like Google.
As far as the job is concerned, it is basically the same job except you are rating results in different search engines that turn up different results. The pay generally varies depending what company you work for with some reporting earnings in the $12 to $15 dollar per hour range.
Google projects (Yukon/Internet Safety/Internet Assessor)
Some of these jobs are changing and they are being called different things such as ‘internet analyst’ which puts a twist on the tasks you are given. Previously when working as a search engine rater, you were given results that were based on generic search results. However, as time went by, Google started asking raters for permission to use their personal data. Basically, if you were surfing for stereo’s or smartphones, and you started doing this job, you’d get back results based on your surfing habits as linked to your Gmail address that you use.
The job that Raterlabs offers is a Google Search Engine and Google Playstore project where the raters are now called ‘Internet Analyst’ where the tasks you get are based solely on your personal Google use. It’s a part-time position that currently looks to be for the USA region only, but this does change from time to time. The requirements and skills to get this job are nothing that most people don’t already have if they are good at searching or surfing the internet. With this job they want raters to possess a computer with high speed internet, and also an Android phone with version 4.1 or higher, or an iPhone version 4 or higher. The job most likely involves doing ratings on both the computer and the smartphone. Sometimes tasks are available on only one device or the other. So it’s good to have them both because you will likely get smartphone tasks. It is also noted that in the description that they want raters who use Google Play often. There is a possibility that tasks involving Google Play will be available. This job’s testing and qualification is no different from the previous exams for search engine related jobs. There are three parts to the exam leading us to believe it’s the same exact material for the regular rater. The job is basically the same after all, the only difference is now its personalized to you. It is not known what the pay is for this job as that information is confidential and you sign a non-disclosure agreement about not discussing your pay. But it’s likely in the $12 dollar per hour range if its Raterlabs.
Appen Global also offers this type of position even though they also own Leapforce and likely are using them as a source of experience and also taking advantage of the workers who were already employed by Leapforce along with their experience. Appen calls their position Web Search Evaluator which likely has the same test. It is unknown if Appen supplies raters for both Google and Bing but it’s possible that they do both. All the requirements and testing are the same for Appen since they are providing raters for Googles Search Engine. But it is rumored that the pay for this position is much lower than you would get if you worked at Lionbridge or Raterlabs. This is because Appen is a much larger company with a much larger talent pool to take people from. This allows them to pay much lower wages. So in this case it’s actually better to get these types of jobs at other companies like Leapforce/Raterlabs, and Lionbridge. It’s interesting to hear that Appen is doing some interviews from Skype now.
Speaking of Lionbridge, we are getting to them next. This company seems to have a good reputation for paying people better than the other companies for the same type of job. They also offer opportunities for the Search Engine Rater but they likely offer rating jobs for both Google and Bing. Microsoft does use Lionbridge to hire human search evaluators, but Microsoft calls you ‘judges’ instead of raters. Lionbridge was known to pay raters $14 an hour for these positions although that wage has probably fluctuated over time.
Miscrosoft’s HRS project (Multimedia Judge, Side by Side Judge)
You’re probably wondering what the difference is in rating for Google or the Bing search engine. Bing has a project called Human Relevance System or HRS and it’s the system that they use to have human raters judge search results with. The HRS project is similar to the Quality Rater program that Google employs. However, it’s interesting to know that Microsoft’s version has been around since 2004 since MSN Search began generating its own results.
Not much has been written about Miscrosoft’s HRS project, but leaked copies of the guidelines are floating around that go over the process of rating or ‘judging’ websites. The difference between Googles rating guidelines and the HRS is that HRS uses what is called a rating matrix. This matrix combines the searcher intent with how well the returned document satisfies the searchers intent.
You can see the five rating options that judges can use from perfect to bad. This is the absolute heart of how you would rate returned search results. This might actually be a little easier to understand for some raters by judging just the intent and how well the intent is satisfied.
But there is one other aspect of the HRS system and the guidelines go over it in a pretty detailed manner. This is called Freshness and basically you will look at the content to see how up to date it is. Some information can be relevant but be a couple years old when some things have changed and the information isn’t updated or fresh enough to contain those changes. The Freshness system is based on three categories; Fresh Not Important, Very Likely Fresh, and Most Likely Fresh. The Fresh Not Important will be for things where the facts will never change no matter how fresh the article or the page is, while the other two will be based on the articles ability to receive fresh content.
When it comes to the difference between Google and Bing as far as their rating systems, they aren’t really all that different from each other. But if you will be working as a search engine rater, and you are looking for stable tasks, you might want to try and get Google search rating jobs since there is more volume when it comes to this search engine over Bing. Still, both jobs are decent, but it really depends on who you are employed by. Appen seems to be one of those companies that will pay people as little money as possible to make more money themselves. This was their whole point of buying Leaforce; to squeeze out the competition. So this company will pay less. Lionbridge or Raterlabs will be the places to try and get these jobs.
When it comes down to Bing, it’s not a very big project in terms of the number of raters who are hired for this as compared to the raters that are hired for Google. That shouldn’t be a big surprise because we all know that Google has a much higher search volume and a lot more users. The judging platform for HRS is a bit outdated, not convenient to use, and it only works on Internet Explorer of course. Also, the test has a part in where you have to judge the search results for relevance of around 100 items and they give you very little time to do this. Many candidates often fail the test and are unable to pass the exam.
iSoftstone is another vendor and is regarded as the main vendor for Microsoft Bing. They also have the 3 test qualification and that test seems to be the same no matter who you apply for. The first test goes over the guidelines, the second test is an English Grammar test, and the third exam tests your computer and internet skills. All work is done in the HRS platform and you have to complete a qualification for each task. There is a guideline for each one of the tasks, but you don’t need to worry because everything is explained in videos and emails. Tasks are categorized as hits or pages and for each hit or webpage that is shown you get 0.01 penny to 0.20 cents US. If you do tasks/hits quickly, then you can make easily $10-$15 dollars US per hour, and some people even more. You might be tempted to go through some of the tasks as quickly as possible because the more you complete, the more money you obviously make. However, sometimes during these tasks you are sometimes tested so you have to be very careful. Too many mistakes and they will remove you from the project and you won’t receive any more hits. ISoftstone is a bit easier to work for in that they don’t have a huge amount of work compared to Appen/Raterlabs.
Basically, what this whole thing boils down to is the individual. Some people might like working for Bing because the workload is lower, but you’re going to make less money because of this. If you need more stable work and higher than average work, then working on a project that involves Google will probably be a better alternative.
Do you work for any of these companies? We would like it if you shared your experience here in the comments. And as always, if you have any questions, let me know by email or in the comments and I will answer them as soon as possible.
Many people fail the Yukon exam at their first attempt because they don’t fully understand Google algorithms’ way of thinking, and you’ve only got a week to study the 164 page guide. Many people fail the first time because this is not an easy exam. Appen and Lionbridge are aware of this fact, and even mention in the welcome email that many people fail the test. In one week after an unsuccessful attempt, you will get the second and last chance to re-take the exam. You will have to go through all parts of the exam again, practical tasks in part 2 (Page Quality) and part 3 (Send to Device) will be different from those you had on your first exam, although there may be a few tasks that come from previous exam.
Why do candidates fail the Appen Yukon and Lionbridge qualification exams?
Lack of effort and sufficient preparation
The most common reason that people fail the Yukon project exam, is that some candidates think that they may get lucky and pass the exam without proper studying, or you may think you know enough about the internet or have good search skills. Be aware that all parts of exam are designed in a way to assess your ability to understand and apply rating concepts and rules based on a 164 page guide. Its not possible to demonstrate the knowledge of these guidelines without investing a significant amount of effort within the one week they give you to study for the exam. If you don’t at least study the guide, or go over it a couple times, you will most likely fail the theoretical part (part 1) of the exam.
Poor time management
One of the common reasons for unsuccessful attempt to pass the exam is poor time management. People do exam preparation and taking all 3 parts within 1-2 days. This strategy in many cases leads to exam failure, because the exam requires lots of concentration and attention to details…Thus plan your exam carefully so you have enough time for studying the material and taking the exam without rushing. Basically, don’t wait till the last minute.
Inappropriate understanding and/or implementation of rating rules and concepts
Have you failed the Appen or Lionbridge exams after carefully studying for them? Don’t worry, a lot of people have. Even people with experience in Search Engine Optimization, or even experience in taking exams have failed this test. In fact it is the first thing that Appen mentions before you get the exam. That it is very difficult and many fail. If you have failed, it is most likely you misunderstood the rating concepts and/or applied them in a wrong way; e.g. used personal judgment/common sense, or did not pay attention to details where it was required (e.g. user location for local queries). In this case I recommend you get assistance from an expert, such as myself
Yukon exam and Lionbridge Internet Assessor tutoring and assistance
If you are not confident about going through the Appen or Lionbridge qualification exam , you may always get my expert assistance, support and advice. For detailed information please write to email@example.com
Note: I am not giving out Leapforce and Lionbridge exam answers, so please do not ask. The reasons for this is because part 2 (Page Quality) and Part 3 (Send to Device) tasks are unique practical tasks, tasks in those parts are updated on a regular basis. The best way is to understand what you are doing and get training on the subject before and after you pass the exam. If you are not sure you understand rating concepts well and and you are not sure how to rate exam tasks correctly, then that is when its best to get my assistance which will give you confidence and lead to success in passing the exam.
Welcome to Yukon/Internet Assessor/Internet Safety Evaluator/Rater Exam Simulation Tool
As you may know Appen acquired Leapforce in 2017, since then the company has decided to close it is own project management platform known as Appen Global and move all projects to Appen connect. The group of projects known as Falcon (mainly Facebook related projects) now is no longer actual, these projects also have been moved to Appen connect platform.
Appen usually has a lot of new and different projects, and a new project has recently come out called Paradise (previously known as Megataxon) which is an Facebook ad review job. This is an interesting project because most of the Falcon Facebook projects usually are rating social network stuff like actual videos from pages or the actual content on Facebook or Instagram. This project focuses on the types of ads that you would see on Facebook. The objective on the project is to decide if the core purpose of the ad is presenting a product, service, or advertising content such as getting you to go to a Facebookpage and subscribing to that pages feed.
Once you understand what the ad is about, you categorize the ad by using a tool that is specially designed by and for Facebook.For each ad there are basically 3 steps to putting each one in the proper category. For the first step, you have to choose the core purpose of the ad and decide if it presents a product, a service, or content.
Along with the tool, you get the project's guidelines that go over all the categories, and it does a very good job of explaining the categories and how you should categorize them.As previously mentioned, the object of the job is to categorize ads as you see them, and you are asked to spend no more than 60 seconds on each ad. This is likely due to the sheer volume of ads that are currently cycling on Facebook. However, there is no minimum view time to apply the tags. With that said, if you are on this project it might take some time to get used to deciding what ad should be categorized where, but after time it is likely that you will pick up speed. It should be noted, however, that if you rush through the ads and categorize them wrong, you could be taken off the project. Once again, they want you to spend no more than 1 minute on each ad with no minimum view time, but you also shouldn’t spend more than a few minutes on each ad. Doing this will surely get the attention of a project manager asking you if there is a problem.
There are times when you can automatically skip the ad if you run into errors, the ad won’t load, the ad is not in the target language for your region, for example the ad is not in English, or there are other issues where you immediately see that the ad is illegal, offensive, or needs to be immediately escalated such as in a case of child exploitation, child nudity, hate speech, acts of terrorism, etc. If you see this bad stuff you have the ability to escalate the Job ID to a manager who handles escalations. If you have to skip the ad for any of these reasons, you just indicate the correct option at the bottom of the Facebook tool.
After you select the purpose of the ad, the next step is to choose a top-level category for the ad. Some of the top level categories have some sub-categories and you will see these appear below the top level categories if there are sub-categories available. This is an extra categorization for some top-level categories. If you select a top level and see a sub-level category appear, you simply choose the best one that fits.
Defining the purpose of the ad simply asks you to choose whether the ad is a Product, a Service, or Content which is defined as the sharing of information that doesn’t provide a specific physical product, service that the reader can buy. Content includes all digital content, articles about an experience with a product, photos of products with no specific call to action, or if the ad uses the ‘Learn More’ button which more than likely leads to content.
Choosing the purpose of the ad should become second nature and you should simply ask yourself ‘What is the core purpose of this ad?’ Ads might reference many different products, services, or content, but your goal is to consider and categorize what is most important for the reader.
The tool itself isn’t difficult and is fairly straightforward. Under the ad, you simply choose the appropriate categories by clicking the radio buttons. Each one of the radio buttons opens with different categories or sub-categories. After you have selected the categories, then on the bottom under that is the submit button. You want to be sure that you have chosen the most correct categories for each ad. Also at the bottom next to the “Submit’ button are the other buttons that we previously mentioned. Those are that the ad has some error, the ad won’t load in the tool, or the ad is not in the English language. You will also see the ‘Other Issue’ button here where you would report an ad that contains something that it shouldn’t that may be offensive or illegal. In this case, the ad would go to one of the managers of the project where its escalated and they take action to remove the ad altogether. You should understand that you have to be careful using this and that the ad should really be offensive and not just something you disagree with or don’t like.
One of the things that is not mentioned in the guidelines is that you might see the ad appear more than once during the week or the month. You should try to remember what you have categorized this ad as and not change the results on the next day if you see the ad again. It’s important that Appen collects precise data. An example of this could be categorizing an ad as something different one day, as opposed to the day before. This will surely get the attention of a product manager. Try to fully understand the categories and email your project manager if you have a question. This will ensure you are able to stay on the project if there is something you don’t understand. Admittedly, there are a lot of products and services on Facebook and categorizing them all correctly might be confusing at first.
The guidelines are really concise and fully detailed and go over the majority of products and services in order to help you categorize them. The guidelines are also very helpful when it comes to deciding what is content or not, and discusses what should fall into those categories.
In conclusion, the Megataxon Falcon project looks to be a very easy job. It might be interesting for some people who want to make a difference in what ads are seen on Facebook, however, after 4 hours of doing this job, it might become tedious on a day to day basis. But for those who don’t mind doing this kind of job, it’s definitely not something incredibly vague or difficult such as some search engine rating jobs.
If you are working on this project, we would like to hear from you and your experience with it. And if you have any questions, you can email me or leave a comment and I will reply as quickly as possible.
As Payoneer is now the only way to get paid by Appen, and if you do not have a Payoneer account yet I invite you to apply for Payoneer account via referral link:
After you have passed either the Lionbridge Ads or the Arrow ads exams, you will start getting access to different tasks. There are some things to keep in mind here. You should understand that if you are in a different market that the USA market, the training tasks are going to be in English. And as a side note, it’s going to be a good idea to have someone that is fluent in English ready in case you don’t understand something. Once you have passed all the initial training tasks, you will start to see tasks appear in the language, or region, that you applied for. There is a high chance that you will still see some English tasks come through. In fact, 70% of all tasks will be in English language.
RECORDING AND INVOICING WORKING TIME Appen Arrow and Lionbridge Ads Assessor project
Time management is one of the most important things you need to keep track of when you are doing your tasks. It’s recommended to track your time manually including the amount of tasks that you have completed. There are some free apps out there you can download for your computer, or you can use your timer app on any smartphone. I keep a small notepad and a pen handy and just cross down a line on paper for each task. Depending on how organized you are, it might take you some time to get used to this. Times sheets are reviewed every weeks starting on Monday and ending Sunday. Make sure you are entering the actual time it takes you from getting the task, to submitting it.
One of the tough things that a rater has when they first start is getting the quality up high and improving the quality of your rated tasks and keeping them there. That is likely because of the overwhelming amount of information to swallow at the beginning. The guidelines that you get need to become your bible so to speak. You should have them open always or easily accessible when you are rating so you can check some of the examples if you are confused about something. If you have some extra time to study the guide while not rating, then it’s a good idea to do this before rating. Don’t spend too much time going over the guide while rating. You are only allotted a certain amount of time for each task. So you can’t spend 15-20 minutes on one task and then bill for that if the task should only take you 5 minutes. Bookmark the examples in the guide for quick reference. Your tasks are reviewed and you will get some feedback concerning your rated tasks and where you can improve. You need to be prepared at all times so that when your quality is personally examined, you get good scores. All of this is very important because the client has a strict process that must be followed. By not following the process, you won’t supply the valuable data that is needed. Read the guidelines even after you passed the exam, and read them again and always refer to them when rating. Even the most experienced raters refer to the guide for help.
Appen Arrow blind test and Lionbridge Ads Assessor quality review tasks
Blind sets as they are called, are rating tasks that have the correct rating answers already correctly completed as they should be as per the guide. You will have no idea when these blinds sets come, or be able to identify them, but generally the blind quality tasks are uploaded to the system throughout the month, and you are graded on these to see if you are following the guidelines correctly, and applying the correct ratings. This is why it’s important to follow the guidelines at all times. In fact, it might be even better to not think about the blind sets at all since you will have no idea when you are doing them. We just thought it would be important for you to know that they are there. In total during a month working cycle there are will be about 150 blind test tasks of different types of tasks, raters are expected to obtain at least 70% of correct answers in the pool of blind test task, raters with lower quality score will be removed from the program.
Reasons for losing your service as anAds Quality Rater
It can happen to anyone at any time. You could end up getting a short email from the company you work for at any time saying that your contract will not be renewed or that your service is no longer required on the project. We hope this doesn’t happen to you, but if it does it could be from some of the following issues.
Quality of service is really the big one here as the client needs each rater to strictly follow the guidelines and the processes of doing each rating. There are levels of quality and you should be sure to try and get the highest quality rating possible. Also its important to remember that just because you are at an acceptable level doesn’t guarantee job security. If there are a lot of raters for a project and a group of raters are providing much higher quality, then the ones at the bottom are likely to go first in the event of a downsizing or, the client just needs the best quality possible. It is advisable to keep striving for perfection when it comes to rating.
Participation in the program means that you must put in a required amount of hours or ‘minimum hours required’ which for most of these programs is 19-20 hours per week. Some programs require you to work at least one weekend day as well. If you miss a day or two on some projects, it will be OK but be careful because if you don’t participate on a regular basis then you end up losing access and are removed from the system. Regular participation in the projects also guarantees that you see a larger portion of different types of ads, and will certainly help you build up your quality levels.
IP address conflict/Internet Network sharing
In some cases people may share their Internet network, while working as search engine evaluator make sure that you have your personal unique IP address and no one uses the same IP to go online for rating project.Once you are in the Project, you may change your IP address many times, it will not affect your project status. Make sure that you do your project tasks in a private place as it stated in the contract.
Be informed that it is prohibited to have multiple accounts, without special arrangement. The customer can find duplicate accounts very fast (even if you are in different Google projects which administrated by different vendors), and as a result, you will lose both accounts.
No tasks are available, all task options are not available anymore except the QR code
This is a sign that system automatically cut your account from receiving tasks. Unfortunately this is an indication that your account has been deactivated due to unacceptable quality, such raters will be removed from the program shortly.
Lionbridge (Ads rating project) vs Appen (Arrow)
In this project, most of the tasks do not have any recommended time limits. Raters have to manage their own productivity and their own time. The customer does not indicate average estimated time because the priority is quality in rating the tasks rather than productivity, or how many tasks you rate. In this respect, Lionbridge is very loyal, and most invoices are approved without any issues. Of course, raters have to keep in mind that the productivity and time invoiced should be within reasonable metrics. It is OK to spend more time in the beginning and increase your productivity when becoming a more experienced rater.
Appen has it is own internal productivity metrics called RPH. It is assumed these metrics are based on the average raters’ performance in the project which is calculated as the ratio between the number of tasks and the time spent within an invoice period completed by all raters. In my view this metric is not useful, because in many cases raters get different types of tasks which require a different amount of time to complete, therefore the RPH metric is not accurate at all. This can lead to a situation where if your productivity doesn’t fall within their RGP metrics, your invoice will be returned for correction and they will ask you to align your RHP to average even though you spent a reasonable amount of time for completing more time-consuming tasks. The customer understands this issue. That is why they do not indicate the average estimated time. But it would seem that Appen(Leapforce) has again invented their own metrics to pay less to its raters.
Working hours and flexibility
At Lionbridge, there is no minimum requirement for working hours. However, the maximum normally is 20 hours and in some cases, the max might be increased.
At Appen there is a strict requirement to work a required number of hours as indicated in the consulting agreement (15-20 hours), not meeting this requirement may lead to removal from the project.
Lionbridge normally pays via direct deposit for US raters or wire transfer for raters in other countries. Non-US raters will receive their first payment only on the second month of work this due to their specific payment cycle set up.
Appen(Leapfroce) pays on a monthly basis via the Payoneer payment system, normally on 14th of every month.
In most markets, the Lionbridge payment hourly rate is 15-30% higher than what Appen pays for the same work. The customer sets the basic pay rate which is the same for all vendors (Lionbridge, Appen), and part of this rate goes to the vendor for project administration services and the other part is paid to raters. As some of you may know this project was previously administrated by Zerochaos and at that time the payment rate was 2 times higher than it is now. After Google outsourced this project to Appen and Lionbridge the payment rate was significantly reduced including US market.
The project itself is fully automated and was designed in a way that raters should manage their training and education, and work with minimum support. Those who fail are automatically removed from the project due to low quality etc. In general terms Appen provides better support by means of a monthly practice test and detailed feedback on those practice tasks, Lionbridge does not provide this type of support.
It might be beneficial to know that there is an optional agreement that you can sign that gives the vendor your personal browsing history that personalized tasks are generated by. Basically what that means is that you would get tasks to rate that would be based on your own personal browsing habits. A lot of people don’t want to do this for fear of sharing personal data with an employer. However, your data still remains private and the vendor will not see what websites you are visiting. It does add a little more personalization to your tasks though. This history is associated with the Google Account that you are signed into at the time of browsing, like if you were logged into Chrome and browsing the internet. It’s important to understand that Google has access to this data anyway, but it’s an optional thing whether you want your own personal history used for generating tasks.
The Final Word
If you can get in on this project, it can be very interesting while you are contributing to search engine or ad algorithms. Your direct contributions help to train specially written algorithms that help ads and search engine results become more relevant to the user. So it’s a very important job, and one that you should take seriously. However, this job is not for everyone. Unless you find looking at ads and webpages 4 hours a day, you might get bored very quickly. But for some this is the perfect part-time job because as long as you meet your productivity, deliver high accuracy and quality, then you could be on the project for as long as you want. As with all other Appen projects, you may even be invited to work on other projects too.
Appen Arrow invoice correction required due to low RHP
what is mystical RHP and is this practice legal?
In this project, most of the tasks do not have any recommended time limits. Raters have to manage their own productivity and their own time. The customer does not indicate average estimated time because the priority is quality in rating the tasks rather than productivity, or how many tasks you rate. Raters have to keep in mind that the productivity and time invoiced should be within reasonable metrics. It is OK to spend more time in the beginning and increase your productivity when becoming a more experienced rater.
Appen has productivity metrics called RPH. It is assumed these metrics are based on the average raters’ performance in the project which is calculated as the ratio between the number of tasks and the time spent within an invoice period completed by all raters.
In my view this metric is not useful, because in many cases raters get different types of tasks which require a different amount of time to complete, therefore the RPH metric is not accurate at all. This can lead to a situation where if your productivity doesn’t fall within their RPH metrics, your invoice will be returned for correction and they will ask you to align your RHP to average even though you spent a reasonable amount of time for completing more time-consuming tasks. The customer (Google) understands this issue. That is why they do not indicate the average estimated time. But it would seem that Appen invented their own metrics to pay less to its raters and make more money by invoicing actual time spent on doing task by all raters to Google. Good business, in fact. In this respect Lionbridge plays a fair game and most of raters get their time sheets approved without any issue.
Our records show a significant discrepancy between the hours you’ve written in your invoice and the amount of work completed. You over-billed more than xx hours for the week of x/x/201X – X/X/201X.
Please check our Chat feature, review our project information page and Intl FAQ. Do not write your missed hours in your invoice time card as the entry will be considered billable.
In the meantime, these hours will not be approved until clarification has been received.
Please let us know if there’s ever an issue outside your control that is impacting your overall productivity such as tool errors.
The productivity rate/ rate per hour (RPH) is calculated by looking at the average of team who worked on a particular task. When you have a low RPH it negatively impacts the RPH average of the entire team and project. It is our expectation that raters meet the average rate per hour on a consistent basis. Listed below are some tips to try if you have not done so already, to hopefully push your rate per hour to the average:
• Time Management: Refrain from multi-tasking while working on EWOQ. Watching videos or listening to music not associated to a task may impact your focus and performance. • Timing Devices:
In order to get a better feel for how fast or slow you are working, you might consider using a desktop timer, Toggl, or stopwatch. Always exit EWOQ when taking a break, no matter how short. Do not count the time spent on breaks towards your billable hours.
A company can’t just suddenly take away your pay when you put in the hours and did the required tasks to the best of your ability and as per your contract. There are some laws that protect freelancers and independent contractors, and if you are doing the work, then you should be paid. But for freelancers and independent contractors, your options are limited depending on what country you live in.
What raters can do in this situation?
Before making any invoice correction, ask Appen for detailed clarification about RHP: what is the RPH ratio for all raters; what is your own RHP ratio, what is recommended RPH ratio, what is average recommended time for completion each type of tasks, ask them to provide list of average estimated time for completion for each type of tasks, ask why customer does not indicate recommended time for completion for many types of tasks. Be ready that they will not provide any info, because there is no official RHP approved by customer (Google).
Point out that fact there is neither any policy regarding RHP can be found on Appen connect project platform nor customer (Google) does not provide any recommended time for most of tasks.
Notify Google about the issue by sending email to the ads rating project admin (Google Ads Eval: firstname.lastname@example.org) to make them aware that this is some kind of issue that really needs to be investigated. The message should be the following: actual time spent on doing live tasks was not approved due to RHP, what is the RHP ratio for all raters; what is your own RHP ratio, what is recommended RHP ratio, what is average recommended time for completion each type of tasks, ask them to provide list of average estimated time for completion for each type of tasks, ask why Google does not indicate recommended time for completion for many types of task. FINALLY ask them – is it legal that Appen cuts hours based on mystical RHP which is not trasparant?
Perhaps if the right people receive a large response on the issue, someone will take a look and fix the problem. But as of right now things aren’t looking too good. For those of you who have lost wages, hang in there and hopefully, someone will get this straightened out.
If you have questions or comments then leave them below.
A lot of interesting things have happened over the last five years when you take a look at the three big companies that offer work at home opportunities with jobs in social media evaluation. Those big three companies are of course Appen Global, Leapforce, and Lionbridge.
To get a good idea of the geographical presence of these three companies, you need to look no further than Google trends as far as interest over time along with the search different search terms.
The confusing part of how and when Raterlabs came into play leaving many people wondering if they were the same company as Leapforce. The website looked the same except the logo was changed, and this time when someone went to apply for the search engine rater job, you were redirected to the Raterlabs website. This also had people wondering if the two companies were the same or not.At one point, one of the clients for RaterLabs decided to only work with companies that hire raters as employees and not independent contractors. It’s not known who this is exactly, but considering what happened with the legal situation with Leapforce and Google, it was quite obviously the search engine giant Google. In order for LeapForce to keep their client they created the company RaterLabs and hired people as actual employees, but this is only for the USA.
This brings up two things. You might remember that Leapforce was recently acquired by Appen Global and that there was a situation in where there was confusion on whether independent contractors were considered full-time employees of Google if they were simply independent contractors. One of the biggest mysteries for raters at Leapforce is who they actually work for. After all, paychecks come from Leapforce, but their work all seems to take place via Google’s Raterhub service. Despite all this stuff going on, the press release that Appen was going to acquire Leapforce came out on November 29th, 2017.
From the Google trends charts, you can see a steady rise in interest for Appen Global while the other two companies Leapforce and Lionbridge took a sharp decline. Leapforce as a company still exists, but all hiring is done by Appen Global now since the acquisition. The main reason for this is because it’s easier to simply acquire a company and use their already hired staff. In this way Appen is able to completely control the company and while at the same time, grow much bigger. And while it sounds like company growth would be a good thing for everyone involved, it’s not. As Appen becomes bigger, it can hire more people and pay them less money, which is what is currently happening. The pay rate at Appen has decreased for raters, and because they have so many people internally from their ‘crowdsourcing’ pool, it’s harder to apply and get a job for this position. By the way, Lionbridge also has changed the way they do business in USA (Google rating project) and also made it is raters as part time employees. With this in mind, we wonder why, Appen continues hiring raters as independent contractors for the same project (Inca also known as Yukon) for less pay at $9.00 per hour as opposed to $12.50 per hour at Raterlabs simply because they haven’t had any legal cases so far and they tend to hire people in states with minimum payments rates at the $9.00 per hour rating. This comes even as most states in the USA have a minimum pay rate of $10 per hour and some states even signed a law to increase the min. wage to $15 an hour. It’s almost a guarantee that if you live in these states, you will not likely be hired when they can pay cheaper wages in another state.
Update:Project Inca at Appen, Part-Time team members
On July 1, 2018, independent contractors will move from Appen to RaterLabs. RaterLabs is a separate legal entity owned and operated by Appen. The contractors will become part-time employees amd most likely their hourly rate will be increased. This decision most likely is forced by the customer (Google) and US legislation.
However, it is worthwhile to mention that Leapforce had its own problems in the past when it came to pay as they claimed you would be making a certain amount of money per hour, but that wasn’t accurate. You were actually paid for the time it should have taken you to do the work rather than how long it actually took you to do the work. In other words, they calculate how long they think each task will take you, and pay you for each task no matter how long it took. Work wasn’t consistent enough to make the hourly pay rate either. And the biggest thing, of course, is that training that takes a long time and you’re not paid for it. This is something that Appen hasn’t changed either. They’ve basically adopted the system which Leapforce uses, but instead, they pay far less than Leaforce did or even as Lionbridge pay. Speaking of Lionbridge, we’ve been talking about Appen and Leapforce. But what about Lionbridge?
If you look at the 5-year chart on Google trends and compare the three companies together, you can see that Lionbridge has just barely managed to stay above the interest level of Leapforce which of course is old news now because of Appen. At the same time, you can see that interest skyrocketed for Appen in January right after it acquired Leapforce. It was the big story at the time, and people that didn’t know about Appen finally found out about them and the work at home jobs they offered.
However, it might be worth mentioning that Lionbridge is probably still the best place to work for when it comes to being a search engine rater. They are known to have a higher pay rate that Appen offers the same job. There are however a couple things that seem to be problems still and that is the 20-24 hourly cap they have for each week, and the fact that there appear to be full-time jobs but part-time employees have no idea how to get into these jobs. With that said, these have been issues for quite some time. Otherwise, Lionbridge is more stable and more rater oriented than Appen which offers a ton of different jobs and doesn’t focus on search engine raters. That is just what you would logically experience from such a big company whose main goal is to increase their client list while utilizing their crowdsourcing at low pay rates.
If you were looking at these companies from a market share perspective, anyone knows that Appen is taking a huge share of the market when it comes to data gathering, translation services, and human-annotated datasets for machine learning and artificial intelligence. But Appen seems to be getting too big for its own good and the bottom line is that the people who work for this company are the ones who miss out on the benefits. Appen seems to also be getting a lot of negative reviews lately and even some people are calling them a ‘scam’ company. Leapforce, since it was acquired, will likely end up being broken down and dismantled and ‘enveloped’ if you will, as more time passes. By the way process has already started as we got confirmation from lots of markets regarding decrease in payment rates in Yukon project at Leapforce. As you may remember we predicted this when Appen acquired Leapforce.
The future for Lionbridge lies in its ability to separate itself by doing things that Appen doesn’t do. If this company is going to survive, they should make a few changes, such as hiring full time employees with benefits, pay for the long training that one needs to go through as a search engine evaluator, and make the pay competitive enough so that people will want to work there instead of the big competition, who doesn’t really have much to offer in the terms of well-payed employment.
We just might get lucky and see Lionbridge make these changes in the future. Have you had experience with Appen, Leapforce, or Lionbridge? Let me know in the comments and as always, if you have a question, let me know and I’ll answer as soon as possible.