Bing Search Engine Rating Projects vs. Google Search Engine Evaluation Project

Bing Search Engine Rating Project (Side by Side Judge, Multimedia Judge) vs Google projects (Yukon/Internet Safety/Internet Assessor)

When getting into Search Engine Rating Projects, you will find that one company will produce ratings for one search engine, such as Google, while another company will offer jobs as a rater in Bing which is Microsoft’s Search Engine. Like Google, Microsoft also needs research and pages rated just like Google.

As far as the job is concerned, it is basically the same job except you are rating results in different search engines that turn up different results. The pay generally varies depending what company you work for with some reporting earnings in the $12 to $15 dollar per hour range.

Google projects (Yukon/Internet Safety/Internet Assessor)

Some of these Search Engine Rating Projects are changing and they are being called different things such as ‘internet analyst’ which puts a twist on the tasks you are given. Previously when working as a search engine rater, you were given results that were based on generic search results. However, as time went by, Google started asking raters for permission to use their personal data. Basically, if you were surfing for stereo’s or smartphones, and you started doing this job, you’d get back results based on your surfing habits as linked to your Gmail address that you use.

The job that Raterlabs offers is a Google Search Engine and Google Playstore project where the raters are now called ‘Internet Analyst’ where the tasks you get are based solely on your personal Google use. It’s a part-time position that currently looks to be for the USA region only, but this does change from time to time. The requirements and skills to get this job are nothing that most people don’t already have if they are good at searching or surfing the internet. With this job they want raters to possess a computer with high speed internet, and also an Android phone with version 4.1 or higher, or an iPhone version 4 or higher. The job most likely involves doing ratings on both the computer and the smartphone. Sometimes tasks are available on only one device or the other. So it’s good to have them both because you will likely get smartphone tasks. It is also noted that in the description that they want raters who use Google Play often. There is a possibility that tasks involving Google Play will be available. This job’s testing and qualification is no different from the previous exams for search engine related jobs. There are three parts to the exam leading us to believe it’s the same exact material for the regular rater. The job is basically the same after all, the only difference is now its personalized to you. It is not known what the pay is for this job as that information is confidential and you sign a non-disclosure agreement about not discussing your pay. But it’s likely in the $12 dollar per hour range if its Raterlabs.

Appen Global also offers Search Engine Rating Projects even though they also own Leapforce and likely are using them as a source of experience and also taking advantage of the workers who were already employed by Leapforce along with their experience. Appen calls their position Web Search Evaluator which likely has the same test. It is unknown if Appen supplies raters for both Google and Bing but it’s possible that they do both. All the requirements and testing are the same for Appen since they are providing raters for Googles Search Engine. But it is rumored that the pay for this position is much lower than you would get if you worked at Lionbridge or Raterlabs. This is because Appen is a much larger company with a much larger talent pool to take people from. This allows them to pay much lower wages. So in this case it’s actually better to get these types of jobs at other companies like Leapforce/Raterlabs, and Lionbridge. It’s interesting to hear that Appen is doing some interviews from Skype now.

Speaking of Lionbridge, we are getting to them next. This company seems to have a good reputation for paying people better than the other companies for the same type of Search Engine Rating Projects. They also offer opportunities for the Search Engine Rater but they likely offer rating jobs for both Google and Bing. Microsoft does use Lionbridge to hire human search evaluators, but Microsoft calls you ‘judges’ instead of raters. Lionbridge was known to pay raters $14 an hour for these positions although that wage has probably fluctuated over time.

Miscrosoft’s HRS project (Multimedia Judge, Side by Side Judge)

You’re probably wondering what the difference is in rating for Google or the Bing Search Engine Rating Projects. Bing has a project called Human Relevance System or HRS and it’s the system that they use to have human raters judge search results with. The HRS project is similar to the Quality Rater program that Google employs. However, it’s interesting to know that Microsoft’s version has been around since 2004 since MSN Search began generating its own results.

Not much has been written about Miscrosoft’s HRS project, but leaked copies of the guidelines are floating around that go over the process of rating or ‘judging’ websites. The difference between Googles rating guidelines and the HRS is that HRS uses what is called a rating matrix. This matrix combines the searcher intent with how well the returned document satisfies the searchers intent.

You can see the five rating options that judges can use from perfect to bad. This is the absolute heart of how you would rate returned search results. This might actually be a little easier to understand for some raters by judging just the intent and how well the intent is satisfied.

But there is one other aspect of the HRS system and the guidelines go over it in a pretty detailed manner. This is called Freshness and basically you will look at the content to see how up to date it is. Some information can be relevant but be a couple years old when some things have changed and the information isn’t updated or fresh enough to contain those changes. The Freshness system is based on three categories; Fresh Not Important, Very Likely Fresh, and Most Likely Fresh. The Fresh Not Important will be for things where the facts will never change no matter how fresh the article or the page is, while the other two will be based on the articles ability to receive fresh content.

When it comes to the difference between Google and Bing Search Engine Rating Projects as far as their rating systems, they aren’t really all that different from each other. But if you will be working as a search engine rater, and you are looking for stable tasks, you might want to try and get Google search rating jobs since there is more volume when it comes to this search engine over Bing. Still, both jobs are decent, but it really depends on who you are employed by. Appen seems to be one of those companies that will pay people as little money as possible to make more money themselves. This was their whole point of buying Leaforce; to squeeze out the competition. So this company will pay less. Lionbridge or Raterlabs will be the places to try and get these jobs.

When it comes down to Bing, it’s not a very big project in terms of the number of raters who are hired for this as compared to the raters that are hired for Google. That shouldn’t be a big surprise because we all know that Google has a much higher search volume and a lot more users. The judging platform for HRS is a bit outdated, not convenient to use, and it only works on Internet Explorer of course. Also, the test has a part in where you have to judge the search results for relevance of around 100 items and they give you very little time to do this. Many candidates often fail the test and are unable to pass the exam.

iSoftstone is another vendor offering Search Engine Rating Projects and is regarded as the main vendor for Microsoft Bing. They also have the 3 test qualification and that test seems to be the same no matter who you apply for. The first test goes over the guidelines, the second test is an English Grammar test, and the third exam tests your computer and internet skills. All work is done in the HRS platform and you have to complete a qualification for each task. There is a guideline for each one of the tasks, but you don’t need to worry because everything is explained in videos and emails. Tasks are categorized as hits or pages and for each hit or webpage that is shown you get 0.01 penny to 0.20 cents US. If you do tasks/hits quickly, then you can make easily $10-$15 dollars US per hour, and some people even more. You might be tempted to go through some of the tasks as quickly as possible because the more you complete, the more money you obviously make. However, sometimes during these tasks you are sometimes tested so you have to be very careful. Too many mistakes and they will remove you from the project and you won’t receive any more hits. ISoftstone is a bit easier to work for in that they don’t have a huge amount of work compared to Appen/Raterlabs.

Basically, what this whole thing boils down to is the individual. Some people might like working for Bing because the workload is lower, but you’re going to make less money because of this. If you need more stable work and higher than average work, then working on a project that involves Google will probably be a better alternative.

Do you work for any of these companies on any Search Engine Rating Projects? We would like it if you shared your experience here in the comments. And as always, if you have any questions, let me know by email or in the comments and I will answer them as soon as possible.



iSoftStone and Appen Butler Hill: Search Engine Evaluation Service For Microsoft Bing

Microsoft Bing Search Engine Battle V3  


If you have been searching around online for jobs that you can do from home, you might have run into Appen Hill. Appen Hill is a technology company that hires talent pools, in over 140 different languages, for many different types of online jobs for big companies such as Microsoft, Instagram, and other online companies that need different types of support.

Appen has been offering flexible work from home opportunities since 1996 when the company was founded. In 2010, Appen merged with the Butler Hill group, hence the name “Appen Butler Hill”. The now expanded business scope includes language resources, search, and text services. Appen is a legitimate organization that is well known and the company even works with the United States and United Kingdom government agencies. If you are looking for an ideal income as a freelancer, stay at home parent, Appen Butler Hill just might fit the bill.

Another vendor for Microsoft – iSoftStone (Search Engine Battle V3 project)

iSoftStone is a Chinese company who believes that talent is the most important workforce element.

iSoftStone is a company that hires different types of information technology qualified candidates. One of the job positions that they fill is for the Search Engine Evaluator. When you go to their website, you will see a search function or you can just take a look at all the jobs they offer. The Search Engine Evaluator position is called a couple of things on their website. You can search for Online Search Evaluator, or Search Engine Analyst as well as Search Engine Evaluator. They are the same position.

You are an independent contractor making $13 an hour in the Search Engine Evaluator position. Pay is once a month via PayPal.

iSoftStone requires that you have a Microsoft Live or Outlook email account in order to finish and submit your application. So it might be a good idea to go ahead and get a free live or outlook account before you do the application.

The good news is that you can combine work at Leapforce and Appen Butler Hill (iSoftStone), so potentially you can increase your income providing more or less the same service for Google and Microsoft!

Web Search Evaluator Position for Microsoft Bing

One of the jobs that Appen recruits for is a position called Web Search Evaluator. So what does a Web Search Evaluator do? Most people think that when it comes to search engine results, everything is done by robots or web crawlers. While the big search engines do use complex algorithms, you would be surprised to know that there is a human element when it comes to search engine rankings and the pages that you see when doing a search online using Bing or Google. As a search engine evaluator, it is your job to make sure that search results are accurate, comprehensive, and spam-free without all kinds of ads blocking the main content of a website. As a search engine evaluator, you should have a familiar understanding of the language and culture of the local internet user. If you want to be a search engine rater, you need to be fluent in both written and spoken English.


  • As mentioned before you need to be fluent in written and verbal English.
  • You should have some interest in internet and be capable understanding how web browsers work and be able to navigate through website content.
  • You would be working this job on your own without a boss looking over your shoulder, so you should have the ability to follow instructions.
  • You should have a good working computer and a stable internet connection. Having research skills, problem solving, critical thinking, and decision making skills are vital.
  • A good grasp on basic computer skills is also a plus as you might be installing applications and troubleshooting software issues with limited support.
  • You will also need to pass the online exam for this position.
  • 5 years of residence in your country


  • $13-18 per hour depending on your country of residence and quality of your service
  • You will be hired as a contractor for Appen so you’ll need to bear the tax responsibilities.
  • You’ll be paid once a month via PayPal.

Pre Exam (Appen)

The online test is a 3 part exam that gauges your search and analytical skills along with your English literacy and competence. It is important that if you do get the opportunity to take the exam, that you study the guidelines very carefully. This is very important when it comes to passing this exam. You will get plenty of time to study and do the three parts of the exam. So it’s best to plan out everything and take your time.

The most important part of the exam is the ‘Search’ section. It is this portion of the exam that will determine if you are a good fit for the position and you understand the basics. In this section, you will be asked to rate websites based on their relevance to the search term. You are always given a search term and sometimes a location that will help you determine the user’s intent. This part of the test demonstrates exactly what you would be doing as a Search Engine Evaluator.

You should be objective in rating websites and base the rating that you give off of the guidelines. The guidelines are very important and considered your ‘Holy Grail’. You should always refer to the guidelines when you have a question. In fact, some raters still use the guidelines daily in helping them evaluate a website. You should do your best and take your time when answering the exam. You are only going to get one shot at taking the exam, so its best that you answer each question carefully and try to not run through it as fast as you can.

The main goal is to rate how satisfied a web searcher is going to be with the results for what they are searching for. You are given their search term and the results. It is up to you to decide if the results are going to help the user and give them a good experience, or if the results are no good to the searcher. You should put yourself in the shoes of the searcher and ask yourself ‘If I was looking for this and these are the results, would I be satisfied?” It’s important to understand that the content returned should match the search term or the user’s intent. Other things will be measured as well such as is the website professional, easy to read, easy to navigate, and is it full of ads that are blocking the main content.

It’s important to understand that the searcher is looking for something specific. No one wants to waste their time online searching for something and getting bad results. That is where you come in.

This might sound easy, but it’s not so simple. The reason being is that a single search term could have more than one possible intent. There could be secondary reasons for the user search aside from the primary.

There are multiple questions that you want to ask yourself when trying to evaluate search result such as:

  • Does the returned content closely match the content, are the results too broad or narrow.
  • Can you easily navigate the website and can the content be easily seen or read? If it doesn’t meet these requirements then it should receive a lower rating as it’s not convenient to deal with a site that is hard to get around or find the information that you are looking for.
  • You need to take into consideration why the user has searched for the term and what was their main reason or goal. Why did they search for the term and were the results returned satisfactorily.
  • There can be one or more reasons for a user doing a search for a term. That user could be looking for something specific or visit exact websites using such terms like ‘Facebook’, ‘Wikipedia’, or ‘’.
  • The user could be looking for information or learn about something specific. Some examples might be: ‘cake recipe’, ‘Kansas weather’, ‘Star Wars movies’.

Remember that in almost all cases a single search term could have multiple possible reasons. You should try to figure out what the main or primary reason is and which are secondary reasons.


Search term: (USA) Quizno’s website

Web Page:

The primary reason of this search term is to find the website for this company’s website in USA. This webpage is the Quizno’s home page and is most likely where the user wants to go to and is the most relevant  result

Very Relevant


Search term: Sean Connery

Web Page:

A primary reason users will search for a popular figure such as Sean Connery will be to find out about his life. This Wikipedia article about him is well laid out with detailed information and is a very relevant  result


Search term: Johnny Depp

Web Page:

Taking part in Johnny Depp themed social network activities such as this would be a secondary reason for the search term Johnny Depp (most users would be more specific in their search term wording if this is the action they wanted to take). There is a connection between the search term and this webpage, which is a Johnny Depp fan page on Pinterest, and so it is a slightly relevant  result.

iSoftStone qualification steps: Search Engine Battle V3 project

The qualification process is straight-forward and is similar to the process at Appen: 

Log in to: UHRS Prod WebEntry, click the “Marketplace” tab, and locate “Search Engine Battle v3” from  the list. You have several options:    

Preview: get a feeling for how Search Engine Battle V3 works before you get started   Training: learn judging basics in Training to gain access to the Qualification   Qualification: take the Qualification to qualify for judging (you have 3 attempts)  Start Judging: once you pass the Qualification, select this to begin judging!  

Once the task opens, you’ll be presented with a page containing a query, a six-point rating scale, and two side-by-side desktop search results pages. Ensure that you can view the entire width of both pages and scroll all the way to the bottom while assessing the results.   

And finally, if you pass this exam successfully you will be notified about results within 2 weeks.

Payoneer is now the only way to get paid by iSoftStone.

If you do not have a Payoneer account yet you can get yours here:

If you use this link for applying for a Payoneer account/card we will both benefit by earning 25$, once your account is loaded with payments.