iSoftStone and Appen Butler Hill: Search Engine Evaluation Service For Microsoft Bing

Microsoft Bing Search Engine Battle V3  


If you have been searching around online for jobs that you can do from home, you might have run into Appen Hill. Appen Hill is a technology company that hires talent pools, in over 140 different languages, for many different types of online jobs for big companies such as Microsoft, Instagram, and other online companies that need different types of support.

Appen has been offering flexible work from home opportunities since 1996 when the company was founded. In 2010, Appen merged with the Butler Hill group, hence the name “Appen Butler Hill”. The now expanded business scope includes language resources, search, and text services. Appen is a legitimate organization that is well known and the company even works with the United States and United Kingdom government agencies. If you are looking for an ideal income as a freelancer, stay at home parent, Appen Butler Hill just might fit the bill.

Another vendor for Microsoft – iSoftStone (Search Engine Battle V3 project)

iSoftStone is a Chinese company who believes that talent is the most important workforce element.

iSoftStone is a company that hires different types of information technology qualified candidates. One of the job positions that they fill are for Search Engine Evaluator. When you go to their website, you will see a search function or you can just take a look at all the jobs they offer. The Search Engine Evaluator position is called a couple things on their website. You can search for Online Search Evaluator, or Search Engine Analysist as well as Search Engine Evaluator. They are the same position.

You are an independent contractor making $13 an hour in the Search Engine Evaluator position. Pay is once a month via PayPal.

iSoftStone requires that you have a Microsoft Live or Outlook email account in order to finish and submit your application. So it might be a good idea to go ahead and get a free live or outlook account before you do the application.

The good news is that you can combine work at Leapforce and Appen Butler Hill (iSoftStone), so potentially you can increase your income providing more or less the same service for Google and Microsoft!

Web Search Evaluator Position for Microsoft Bing

One of the jobs that Appen recruits for is a position called Web Search Evaluator. So what does a Web Search Evaluator do? Most people think that when it comes to search engine results, everything is done by robots or web crawlers. While the big search engines do use complex algorithms, you would be surprised to know that there is a human element when it comes to search engine rankings and the pages that you see when doing a search online using Bing or Google. As a search engine evaluator, it is your job to make sure that search results are accurate, comprehensive, and spam free without all kinds of ads blocking the main content of a website. As a search engine evaluator, you should have a familiar understanding of the language and culture of the local internet user. If you want to be a search engine rater, you need to be fluent in both written and spoken English.


  • As mentioned before you need to be fluent in written and verbal English.
  • You should have some interest in internet and be capable understanding how web browsers work and be able to navigate through website content.
  • You would be working this job on your own without a boss looking over your shoulder, so you should have the ability to follow instructions.
  • You should have a good working computer and a stable internet connection. Having research skills, problem solving, critical thinking, and decision making skills are vital.
  • A good grasp on basic computer skills is also a plus as you might be installing applications and troubleshooting software issues with limited support.
  • You will also need to pass the online exam for this position.
  • 5 years of residence in your country


  • $13-18 per hour depending on your country of residence and quality of your service
  • You will be hired as a contractor for Appen so you’ll need to bear the tax responsibilities.
  • You’ll be paid once a month via PayPal.

Pre Exam (Appen)

The online test is a 3 part exam which gauges your search and analytical skills along with your English literacy and competence. It is important that if you do get the opportunity to take the exam, that you study the guidelines very carefully. This is very important when it comes to passing this exam. You will get plenty of time to study and do the three parts of the exam. So it’s best to plan out everything and take your time.

The most important part of the exam is the ‘Search’ section. It is this portion of the exam that will determine if you are a good fit for the position and you understand the basics. In this section you will be asked to rate websites based on their relevance to the search term. You are always given a search term and sometimes a location which will help you determine the user’s intent. This part of the test demonstrates exactly what you would be doing as a Search Engine Evaluator.

You should be objective in rating websites and base the rating that you give off of the guidelines. The guidelines are very important and considered your ‘Holy Grail’. You should always refer to the guidelines when you have a question. In fact, some raters still use the guidelines daily in helping them evaluate a website. You should do your best and take your time when answering the exam. You are only going to get one shot at taking the exam, so its best that you answer each question carefully and try to not run through it as fast as you can.

The main goal is to rate how satisfied a web searcher is going to be with the results for what they are searching for. You are given their search term and the results. It is up to you to decide if the results are going to help the user and give them a good experience, or if the results are no good to the searcher. You should put yourself in the shoes of the searcher and ask yourself ‘If I was looking for this and these are the results, would I be satisfied?” It’s important to understand that the content returned should match the search term or the user’s intent. Other things will be measured as well such as is the website professional, easy to read, easy to navigate, and is it full of ads that are blocking the main content.

It’s important to understand that the searcher is looking for something specific. No one wants to waste their time online searching for something and getting bad results. That is where you come in.

This might sound easy, but it’s not so simple. The reason being is that a single search term could have more than one possible intent. There could be secondary reasons for the user search aside from the primary.

There are multiple questions that you want to ask yourself when trying to evaluate search result such as:

  • Does the returned content closely match the content, are the results too broad or narrow.
  • Can you easily navigate the website and can the content be easily seen or read? If it doesn’t meet these requirements then it should receive a lower rating as it’s not convenient to deal with a site that is hard to get around or find the information that you are looking for.
  • You need to take into consideration why the user has searched for the term and what was their main reason or goal. Why did they search for the term and were the results returned satisfactory.
  • There can be a one or more reasons for a user doing a search for a term. That user could be looking for something specific or visit exact websites using such terms like ‘Facebook’, ‘Wikipedia’, or ‘’.
  • The user could be looking for information or learn about something specific. Some examples might be: ‘cake recipe’, ‘Kansas weather’, ‘Star Wars movies’.

Remember that in almost all cases a single search term could have multiple possible reasons. You should try to figure out what the main or primary reason is and which are secondary reasons.


Search term: (USA) Quizno’s website

Web Page:

The primary reason of this search term is to find the website for this company’s website in USA. This webpage is the Quizno’s home page, and is most likely where the user wants to go to and is the most relevant  result

Very Relevant


Search term: Sean Connery

Web Page:

A primary reason users will search for a popular figure such as Sean Connery will be to find out about his life. This Wikipedia article about him is well laid out with detailed information and is a very relevant  result


Search term: Johnny Depp

Web Page:

Taking part in Johnny Depp themed social network activities such as this would be a secondary reason for the search term Johnny Depp (most users would be more specific in their search term wording if this is the action they wanted to take). There is a connection between the search term and this webpage, which is a Johnny Depp fan page on Pinterest, and so it is a slightly relevant  result.

iSoftStone qualification steps: Search Engine Battle V3 project

The qualification process is straight-forward and is similar to the process at Appen: 

Log in to: UHRS Prod WebEntry, click the “Marketplace” tab, and locate “Search Engine Battle v3” from  the list. You have several options:    

Preview: get a feeling for how Search Engine Battle V3 works before you get started   Training: learn judging basics in Training to gain access to the Qualification   Qualification: take the Qualification to qualify for judging (you have 3 attempts)  Start Judging: once you pass the Qualification, select this to begin judging!  

Once the task opens, you’ll be presented with a page containing a query, a six-point rating scale, and two side-by-side desktop search results pages. Ensure that you can view the entire width of both pages and scroll all the way to the bottom while assessing the results.   

And finally if you pass this exam successfully you will be notified about results within 2 weeks.

Payoneer is now the only way to get paid by iSoftStone.

If you do not have a Payoneer account yet you can get yours here:

If you use this link for applying for a Payoneer account/card we will both benefit by earning 25$, once your account is loaded with payments.