iSoftStone and Appen Butler Hill: Search Engine Evaluation Service For Microsoft Bing

Microsoft Bing Search Engine Battle V3  


If you have been searching around online for jobs that you can do from home, you might have run into Appen Hill. Appen Hill is a technology company that hires talent pools, in over 140 different languages, for many different types of online jobs for big companies such as Microsoft, Instagram, and other online companies that need different types of support.

Appen has been offering flexible work from home opportunities since 1996 when the company was founded. In 2010, Appen merged with the Butler Hill group, hence the name “Appen Butler Hill”. The now expanded business scope includes language resources, search, and text services. Appen is a legitimate organization that is well known and the company even works with the United States and United Kingdom government agencies. If you are looking for an ideal income as a freelancer, stay at home parent, Appen Butler Hill just might fit the bill.

Another vendor for Microsoft – iSoftStone (Search Engine Battle V3 project)

iSoftStone is a Chinese company who believes that talent is the most important workforce element.

iSoftStone is a company that hires different types of information technology qualified candidates. One of the job positions that they fill is for the Search Engine Evaluator. When you go to their website, you will see a search function or you can just take a look at all the jobs they offer. The Search Engine Evaluator position is called a couple of things on their website. You can search for Online Search Evaluator, or Search Engine Analyst as well as Search Engine Evaluator. They are the same position.

You are an independent contractor making $13 an hour in the Search Engine Evaluator position. Pay is once a month via PayPal.

iSoftStone requires that you have a Microsoft Live or Outlook email account in order to finish and submit your application. So it might be a good idea to go ahead and get a free live or outlook account before you do the application.

The good news is that you can combine work at Leapforce and Appen Butler Hill (iSoftStone), so potentially you can increase your income providing more or less the same service for Google and Microsoft!

Web Search Evaluator Position for Microsoft Bing

One of the jobs that Appen recruits for is a position called Web Search Evaluator. So what does a Web Search Evaluator do? Most people think that when it comes to search engine results, everything is done by robots or web crawlers. While the big search engines do use complex algorithms, you would be surprised to know that there is a human element when it comes to search engine rankings and the pages that you see when doing a search online using Bing or Google. As a search engine evaluator, it is your job to make sure that search results are accurate, comprehensive, and spam-free without all kinds of ads blocking the main content of a website. As a search engine evaluator, you should have a familiar understanding of the language and culture of the local internet user. If you want to be a search engine rater, you need to be fluent in both written and spoken English.


  • As mentioned before you need to be fluent in written and verbal English.
  • You should have some interest in internet and be capable understanding how web browsers work and be able to navigate through website content.
  • You would be working this job on your own without a boss looking over your shoulder, so you should have the ability to follow instructions.
  • You should have a good working computer and a stable internet connection. Having research skills, problem solving, critical thinking, and decision making skills are vital.
  • A good grasp on basic computer skills is also a plus as you might be installing applications and troubleshooting software issues with limited support.
  • You will also need to pass the online exam for this position.
  • 5 years of residence in your country


  • $13-18 per hour depending on your country of residence and quality of your service
  • You will be hired as a contractor for Appen so you’ll need to bear the tax responsibilities.
  • You’ll be paid once a month via PayPal.

Pre Exam (Appen)

The online test is a 3 part exam that gauges your search and analytical skills along with your English literacy and competence. It is important that if you do get the opportunity to take the exam, that you study the guidelines very carefully. This is very important when it comes to passing this exam. You will get plenty of time to study and do the three parts of the exam. So it’s best to plan out everything and take your time.

The most important part of the exam is the ‘Search’ section. It is this portion of the exam that will determine if you are a good fit for the position and you understand the basics. In this section, you will be asked to rate websites based on their relevance to the search term. You are always given a search term and sometimes a location that will help you determine the user’s intent. This part of the test demonstrates exactly what you would be doing as a Search Engine Evaluator.

You should be objective in rating websites and base the rating that you give off of the guidelines. The guidelines are very important and considered your ‘Holy Grail’. You should always refer to the guidelines when you have a question. In fact, some raters still use the guidelines daily in helping them evaluate a website. You should do your best and take your time when answering the exam. You are only going to get one shot at taking the exam, so its best that you answer each question carefully and try to not run through it as fast as you can.

The main goal is to rate how satisfied a web searcher is going to be with the results for what they are searching for. You are given their search term and the results. It is up to you to decide if the results are going to help the user and give them a good experience, or if the results are no good to the searcher. You should put yourself in the shoes of the searcher and ask yourself ‘If I was looking for this and these are the results, would I be satisfied?” It’s important to understand that the content returned should match the search term or the user’s intent. Other things will be measured as well such as is the website professional, easy to read, easy to navigate, and is it full of ads that are blocking the main content.

It’s important to understand that the searcher is looking for something specific. No one wants to waste their time online searching for something and getting bad results. That is where you come in.

This might sound easy, but it’s not so simple. The reason being is that a single search term could have more than one possible intent. There could be secondary reasons for the user search aside from the primary.

There are multiple questions that you want to ask yourself when trying to evaluate search result such as:

  • Does the returned content closely match the content, are the results too broad or narrow.
  • Can you easily navigate the website and can the content be easily seen or read? If it doesn’t meet these requirements then it should receive a lower rating as it’s not convenient to deal with a site that is hard to get around or find the information that you are looking for.
  • You need to take into consideration why the user has searched for the term and what was their main reason or goal. Why did they search for the term and were the results returned satisfactorily.
  • There can be one or more reasons for a user doing a search for a term. That user could be looking for something specific or visit exact websites using such terms like ‘Facebook’, ‘Wikipedia’, or ‘’.
  • The user could be looking for information or learn about something specific. Some examples might be: ‘cake recipe’, ‘Kansas weather’, ‘Star Wars movies’.

Remember that in almost all cases a single search term could have multiple possible reasons. You should try to figure out what the main or primary reason is and which are secondary reasons.


Search term: (USA) Quizno’s website

Web Page:

The primary reason of this search term is to find the website for this company’s website in USA. This webpage is the Quizno’s home page and is most likely where the user wants to go to and is the most relevant  result

Very Relevant


Search term: Sean Connery

Web Page:

A primary reason users will search for a popular figure such as Sean Connery will be to find out about his life. This Wikipedia article about him is well laid out with detailed information and is a very relevant  result


Search term: Johnny Depp

Web Page:

Taking part in Johnny Depp themed social network activities such as this would be a secondary reason for the search term Johnny Depp (most users would be more specific in their search term wording if this is the action they wanted to take). There is a connection between the search term and this webpage, which is a Johnny Depp fan page on Pinterest, and so it is a slightly relevant  result.

iSoftStone qualification steps: Search Engine Battle V3 project

The qualification process is straight-forward and is similar to the process at Appen: 

Log in to: UHRS Prod WebEntry, click the “Marketplace” tab, and locate “Search Engine Battle v3” from  the list. You have several options:    

Preview: get a feeling for how Search Engine Battle V3 works before you get started   Training: learn judging basics in Training to gain access to the Qualification   Qualification: take the Qualification to qualify for judging (you have 3 attempts)  Start Judging: once you pass the Qualification, select this to begin judging!  

Once the task opens, you’ll be presented with a page containing a query, a six-point rating scale, and two side-by-side desktop search results pages. Ensure that you can view the entire width of both pages and scroll all the way to the bottom while assessing the results.   

And finally, if you pass this exam successfully you will be notified about results within 2 weeks.

Payoneer is now the only way to get paid by iSoftStone.

If you do not have a Payoneer account yet you can get yours here:

If you use this link for applying for a Payoneer account/card we will both benefit by earning 25$, once your account is loaded with payments.


  1. Hi,
    I work now for ISoftStone after I found informations about on your site, so thank you very much for writting about it!

      1. Hi again!
        Yes, of course!

        I applied for both ISoftStone and Appen. I am living in France, so I think there’s a different situation comparing the US online workers.

        I send my CV and I completed the forms for Appen. The answer came in about one week. I read online that there’s a skype interview, if you’re accepted, but unfortunately they don’t have more work for users from France, but they said they would contact me when they will have more work.
        About ISoftStone, I signed some online forms and then they invited me to pass 3 tests. The first one was on Guidelines, the second was a Grammar English test and the third had some technical questions about using internet and the computer. I was contacted after 3 -4 days. They told me that I am accepted to work and gave me link to the UHRS site (After I researched it on internet, I found that the UHRS platform is used by Clickworker and by ISoftStone).
        All the work is made only on this platform. It contains different tasks like: comparing results from different pages, giving ratings to pages. Those are small tasks.In order to complete the tasks you have to pass a qualification for each task. There’s also a Guideline for each task ( with about 20 pages). By the way, everything is explained in their videos and mails.
        Each task contains more hits(pages). For each hit completed you are paid with 0.01 US – 0.20 US. If you are doing the hits/tasks fast, then you can gain up to 10-15 US dollars by hour, or even more.
        While completing tasks you are sometimes tested, So you have to work very carefully. If there’re too many mistakes, you can get unassigned from the task, so you get no more hits.
        You can see you report on the work in a special section : My Report. There you can find information about your productivity, time spent and also potential money gained.
        I have tasks in English and French(France and Canada), because I’m working in France.
        I started in the middle of May and I was already paid on 1st June. That was very fast.
        Sometimes I am contacted to make some different tasks, that don’t appear on the Platform.
        It’s simple to work for ISoftStone, but the amount of work is not huge(comparing to Leapforce). Still, there’re some issues: you can’t know the amount of money transferred to your account ( paypal) and the date of the transfer. You can find information about the money gained from each task , but not the whole “salary”.

        PS Sorry for my English xD Hope you will understand xD xD

    1. Can you tell me how you generate an invoice to a us / international company? Because I’m from Germany and I’ve never done this in my life. Are there any templates they give you?

      1. Hi,

        Could you specify what kind of invoice do you have in mind? Do you mean how to invoice Leapforce for your service as an independent contactor?

  2. Hello your blog is very helpful. I have a question for I’m from romania but I lived in Ireland from the age of 18 years. I want to apply for a Online Crowd Worker job but my passport is not irish does is it make any diffrences. Can you please let me know what documents they need if I aplly for this job. Please also let me know how much they pay me.

    1. Hi Marian,

      As many services of Google are localized for each country of its presence, applying to this position is only possible if person knows the language, culture peculiarity of users of local searching system.Thus, the main condition is that a candidate must be living in a country of application at least 5 consectuvie years in order to be eligeble for this job.Another requiremnt is that a candidate has a bank account in a country of application.

  3. I have passed qualification test for social media evaluator with Appen but they sent an email saying that the project is post-dated, they don’t know when it will begin 🙁 Somebody has happened?

  4. Hi! I am planning to apply to isoftstone for search engine evaluation position, It is really helpful for me to get insight into the details and requirements for applying for this post, thanks a lot for this!!!
    Hopefully looking forward to your review of iSoftStone exam and qualification process soon!!

    1. I think you answered that question yourself. We should all check out details for any job, just to get some insight on what to expect. So do your own research and then u can decide if it is good for you. That’s what I did. Good luck to u and everyone else looking to work from home. I juss recently applied to be a genius with I’m now waiting for a response. Check it out good luck

  5. Hello, can I apply for Leapforce even if I am already working with Lionbridge? In case I will not be recontracted by Lionbridge, can I apply for Leapforce? Thanks

    1. Hello,


      Be informed that it is prohibited to have multiple accounts, without special arrangement. Leapforce and Lionbridge can find duplicate accounts very fast, and as a result, you will lose both accounts.

      You can combine work at Leapforce and Appen Butler Hill (iSoftStone), so potentially you can increase your income providing more or less the same service for Google and Microsoft!

  6. Hi!

    I taked exam for lionbridge and i’m wating for results, Now i received an offer from isoftstone as triscriber. I don’t understand if i can work for both. Liobridge don’t allow for example to work with leapforce, clickworker and other companies that i don’t remember now. You can hel me to know i i can work for both without any kind of risk to be kicked from both?

    thank you a lot

  7. Tim can you help me? Also I finished the last part for lionbridge 4 days ago and they still do not respond…

    1. Exam results:

      You will get overall exam results on the exam due date from Leapforce and exam due date plus one day with Lionbridge. At Leapforce you will be asked to sign a few electronic forms and in 2-4 days you will be given access to live tasks. With Lionbridge you will have to complete educational course with no pass/no fail quiz before you are granted full access to the tasks.

  8. Thank you for your answer! I finished tha last part of lionbridge exam (part 3) monday, usually they send results within 24h, i don’t still receive nothing…

  9. Hello Allein,

    Hope you are fine

    Me and my sister live in the same home and we want to work in Leapforce, I know that it is not allowed that 2 persons live in the same house work in leapforce but If we will use different PCs and different internet providers (2 different IPs), in this case, is it ok that we work together or they will catch us and close the 2 accounts?

    If we will use different PCs and different internet providers (2 different IPs), can still leapforce catch our location and close the 2 accounts or it is impossible?

    Can you advise me?

    I am looking forward to hearing from you.


  10. Hi, I just started the Nile project. When I go to “all projects” I see: Nile (to work) and Yukon (to qualify). Can I qualify for Yukon while working in Nile? What I mean; Can I work on both projects at once?


  11. Hi Tim.
    Thank you for the useful article.

    I just passed the Lionbridge exam.
    In this article, I thought I can work both Lionbridge and Apen(or iSoft).
    I just only avoid applying for LeapForce. Is this correct?

    Or is it possible that I can work for 3 companies if I pick a different type of job?
    For example, at LionBridge, I will do Ad Assessor.
    at Apen, Social Media Evaluator, then at iSoft,there is “Local Search Pilot”.(Maybe it is same as Lionbridge position)


    1. Hello, Ads Assessor (Aztec) is the project which is administrated from the same platform (raterhub) as Yukon (Internet Assessor) project, thus these two projects are not compatible.

  12. Hi Tim, I currently work as a Social Media Evaluator at Appen and was invited to do the Search Engine Battle V3 project by iSoftStone. Would I be able to do both of these since they use different tools technically? I want to make sure before committing. Thank you so much!

    1. Hello, yes you can, iSoftStone’s customer is Microsoft (Bing), so there is no any risk of IP address conflict.

  13. hello I was wondering if I can work for appen and Isoftstone at the same time. May I ?

  14. Hello Tim,

    I have been working for Lionbridge as a multimedia judge for a couple of months now, and just got an email today for iSoftstone’s Online ad evaluator position. Are these two position’s compatible or not?

    Thanks 🙂

    1. Hello,

      These two project belongs to one customer which is Microsoft Bing, in this connection I assume that it is not possible to work on both projects using the same/one live ID.

  15. Hi-

    How do I become a transcriber for isoftstone or for lionbridge?

    I know that these companies have this type of job?

    Will this affect my participation with Appen’s current work?

    I would love to be able to find out how I can sign up or the direct link for U.S Markets if anyone currently works as this or knows of someone.

    Thanks in advance


    1. Hello, why do you consider transcriber role? I would not recommend you this type of work (project aimed to learn search engine to recognize voice and human speech. I tried to work on this project a few years ago I found it difficult due to high requirements in terms of hearing, sound recognition and ability to transcribe sounds and words in to written labels). Instead I recommend you to consider search engine evalution jobs such as yukon project at Appen or Internet Safety Evaluator at Lionbridge. Check for more info here:

  16. Hello, I applied for IsoftStone about 10 days ago. Do you know how long it takes for them to reach out? Do they reach out through email or there dashboard?

    1. Hello,

      The reaching out time depends on current local needs for rasters, typically they reply with 2 weeks…

Leave a Reply