Maps Analyst Projects at Lionbridge and Shasta Project at Appen

Apple-Maps-IconThere’s new stuff out called Maps Analyst Projects and we’ve got some new information to share about the projects for maps analysts at Lionbridge (Codenamed: Shasta project at Appen. It should be mentioned that the recruitment process for this project normally lasts a very limited period of time (at Lionbridge), and usually 1-2 months during different months depending on the country. However, the Shasta project could be available at any time. 

 The project involves evaluating online mapping and routing related tasks in order to improve the content and accuracy of map service: Apple maps.

Raters are required to conduct research and use online tools to investigate the geographic accuracy of tasks. Work is provided to you as a list of tasks through an online work platform, which is somewhat like raterhub. Most tasks will be in the language related to the position that you applied for. For example, if you applied for the French role, then you will be rating in French, etc. You will also be required to provide comments/feedback in English. 

Map Analyst & Shasta project payment and working hours

This project is paid on a per-task basis. You need to keep track of the number of tasks, as well as the type of tasks completed on a day-to-day basis. This information is used to generate your invoice on the billing dates. The payment rate per task depends on your market, including the complexity of each task, and payment may vary from, $0.22 to $1.00 per task. Since you are working in a freelance independent capacity, you can manage your working time as you like and work whatever times you want. But this is a part-time job, and the number of hours you can accrue is set at a maximum of 20 per week. You must not exceed the maximum of 20 hours per week unless authorized. Lionbridge expects new Map Quality Analysts to be active and to participate each week. 

 Project compatibility

This project is fully compatible with any other projects at other vendors. You are not allowed to work on another project at Lionbridge, although you are allowed to qualify on another project if your vendor is Appen. 

Maps Analyst/Shasta project qualification 

In my opinion, the qualification process for this particular project is quite difficult if you compare it with other projects. First, you need to study and understand 200 pages of guidelines and instructions within a limited period of time, and then complete a 2 stage exam, which consists of theoretical and practical parts, within 10 days. As with any project, you should read the Guidelines as many times as you can before you start working. There is a lot of information in the guidelines so you may want to read them multiple times. You should also use them while working, and refer to them to ensure you are applying the correct rating. There is no chance to pass this exam without getting a profound understanding of the rating rules and concepts. 

Exam Preparation Approach (Maps Analyst and Shasta project exam)

Lionbridge has very good exam support so candidates should use this to their advantage as much as possible. It is highly recommended to use the below approach in order to increase your chances of successfully passing the exam: 

  1. Read the Guidelines thoroughly and understand them
  2. Read/watch the content on the Pre-Exam Support Portal
  3. Complete the Sample Exam Questions

Completing sample exam tasks will help you understand what kind of exam tasks to expect. A typical maps analyst task is a query; address, chain/organization, point of interest, and a list of results generated by the map service based on user location, user viewport, and type of query you will need to provide:

Before proceeding with the exam simulation tool read the project guidelines thoroughly and understand them.
Evaluate and rate each rating drop-down independently of the others. Rate result Relevance based on whether the result fits the query intent, even if the result is closed or does not exist or has incorrect Name, Address, or Pin data.

1. Determine what type of query and user intent is, look for examples on the guidelines:
Single address query, Category query, Chain query, Transit query, etc.

2. Research user location and viewport and possible rating scenarios related to distance and user intent. Keep in mind that distance ratings are variable ratings that depend on query type (a type of business), user location, location modifier, type of area, etc. Each case should be evaluated individually in terms of proximity and distance, that is why there are no specific criteria for distance.

3. Once you understand user intent, rate each results for relevance, one by one.

4. Name accuracy (you always need to check name accuracy on the official website of an organization).

5. Address accuracy (check all components of address on the official website/UPS address verification website).

6. Pin accuracy (apply rules as per guidelines).

The pin location, name, and address accuracy do not affect the relevance rating. If the result meets the requirements for a navigational rating then you should go for it. Relevance and the pin location, name, and address accuracy are independent ratings.

Finally, you will provide comments if your relevance rating is below good. This just a general overview of typical task/algorithm however it gives a good understanding of what kind of tasks to expect in this project. 

Maps Analyst Exam Task

 It is very important that you thoroughly research the query, and the result in every task you are rating. In order to establish the correct rating, you need to understand the query, and you also need to verify that the information returned in a result is correct and accurate including the name, address, location, existence, popularity etc. In order to determine if the information is true, use official sources wherever possible. Include links in your comments whenever you find out that the task returned incorrect data. To determine popularity, you will need to check reviews on reputable sites to decide on the most accurate rating to apply. You do not need to include a comment for every rating, but comments are mandatory for any rating below Good for relevance or below Correct/Perfect for data accuracy. 

  • Reference Guidelines – If you are demoting because of a guideline intricacy, it is useful to reference the guideline/rule that influenced your decision.
  • Be Concise – Don’t waste time writing elaborate comments, be specific and efficient with your comments.
  • Comment in English Only- Your comments will sometimes be reviewed or used by those who are not very familiar with your locale or its language.

As mentioned before you will be given 7 days to complete the exam. The theoretical part will consist of 20 questions which are entirely based on the guidelines. I think this part won’t create any problem because answers to most of the questions can be found in the guidelines, the average estimated time for completing this part of the exam is 1 – 2 hours. 

 Part 2 is the most difficult part of the qualification, some of the tasks are really difficult, and will require a high level of attention, the ability to apply rating concepts, and analytical thinking. In total, you will be asked to rate 20 tasks, but in fact they’re more tasks because some of them are complex tasks that have a list of results that have to be rated. You will be given almost all types of results to rate (chain, address, transit, POI, etc.) So be careful to research and examine each case step-by-step. Keep in mind that only one attempt to pass this exam is allowed. If you need any tips or further information, do not hesitate to contact me.

Shasta (Maps Analyst) Project Exam Personalized Assistance

If you are not confident about going through the Shasta (Appen) or Maps Analyst (Lionbridge) qualification exam, you may always get my expert assistance, support, and advice. For detailed information please write to

Shasta Maps Analyst Exam Simulation Tool
What is The Shasta Maps Analyst Exam Simulation Tool?

This tool will help anyone who is getting ready or considering taking the exam. This tool was designed to assist candidates with exam preparation with the main focus on practical rating tasks that include major maps rating cases and scenarios like single address query, category query, chain query, transit query, variable ratings which depend on query type (a type of business), user location, location modifier, type of area. Maps exam simulation tasks cover basically all rating concepts/task types and skills which will be tested during a real exam.

How can this tool help you pass the exam?

The way the tool is designed will show you exactly your problem area and what you need to focus on. It helps you identify your level of understanding of the rating concepts, and your ability to apply them by practicing first and then identifying your weak points. You will receive your overall score (pass or fail), detailed feedback on your mistakes, and know exactly what you need to go back and study in the guide. Of course, we need to mention that there are no real actual exam tasks because that isn’t going to help you understand anyway. Each one of these is hand-designed from the most typical exam tasks and is also designed to be the best simulation possible. If you pass this, you will be ready to pass the actual exam.

How do I get my hands on the exam simulation tool?

Exam Simulation Demo 

If you’re still skeptical, we have a free demo version which you can try FREE right now! It contains 2 tasks so you can see how it works. Once again, the demo is 100% free.

Exam Simulation Pro 

The tool has 10 tasks of all the major query types and rating scenarios. You will be able to practice your rating skills on practical rating tasks that include major maps rating cases and scenarios like single address query, category query, chain query, transit query, variable ratings which depend on query type (a type of business), user location, location modifier, type of area. Maps exam simulation tasks cover basically all rating concepts/task types and skills which will be tested during a real exam.

Maps Analyst Exam results 

Once you complete this exam within 2-3 days you should expect exam results and a welcome email containing information about access to the supplier and project portal. You also will be given a login and password to the rating tool where you will do your production. 

Project workflow

When you start working, the process will be more smooth because you have already gained a conceptual understanding of the work during testing, plus most of the tasks will be in your local language. It is recommended that 2 minutes be taken to rate each task in order to complete your research accurately. Please note this is an average. Some tasks may require more time, while others may require less time. Tasks can be made available any day of the week, at any time of the day. There is no specific time when tasks are launched. If you don’t see tasks, then you will need to check more frequently to see if new tasks have been added to your queue.

Useful links:

Maps Analyst — Lionbridge — Overwhelmed from WorkOnline


  1. is Name accuracy rating only for business or is it applicable to any type of address ( ex. Henderson, TX or 2468 Radio Park Drive ) if not what should I select from the drop down list ( N/A or Can’t Verify )

  2. Hello,

    As per guidelines, If query is an address query then name accuracy should be selected as N/A.

  3. Hi Tim thanks for the information. I have been working for lion bridge as social media elevator and want to apply for this position. I think i have a chance for applying this position at leap force . If please provide me the link. Thank You.

  4. Hello Tim thank you for the information , i will like to ask you if we have a time limit for the 2 parts of the shasta project tests and if we can take the exam test multiple times?

  5. Hello Tim, if there is a query with specific address and the result is that address with the pin incorrectly dropped 200 meters away but telling that this is the correct address even it is not. Should i rate it acceptable/good – user intent issue or navigational and disregard the pin?

    1. Hello,

      The pin location (exact, approximate or incorrect) does not affect the main rating. If the result meets the requirements for a navigational rating then you should go for it. Relevance and the pin location are independent ratings.

      1. Thank you man this is what I did (for the exam) and the confirmation from you, is really what I was needed. Btw I passed the exam and I’m waiting to start. Any other advises (Is the work similar to the exam, do they expect a high level of accuracy for newbies)? Keep up the good work!

    2. hey man I couldnt help but notice that you passed the exams and congratulations btw. Do you mind if I asked you a few questions?

  6. I passed the test and finished document signning. But I got another mail said they couldn’t proceed this time and my profile will be kept for future. And the most interesting is that I couldn’t log into my profile with my applied account anymore.

    1. What happened to U ? I passed the exam also >> and they sent the same email to me ? Whats next ?

      1. It might be that at the moment of your qualification, quota of raters for your locate was reached and therefore Leapforce does not proceed further. As soon as they fire unsuccessful raters, they continue hiring new people to maintain the required number of raters in your country.

    2. I whish I read this comments before I spent an entire week studing and taking the test. I passed, but I received that same email. Now I’m supposedly in a waitlist.

    3. I wish I had read this comment before I spent one entire week studing and giving the exam. I passed it, but the next day I received the same email. Now I’m supposedly in a waitlist.

    4. Hey, did you eventually get to work for the project at some point? I’m in a similar situation and wondering about what happened to all of you guys who got that email after the exam.

  7. Hello everybody
    Can anyone who passed leapforce exam for shasta project give me some tips to pass it

  8. I also received the message below from Leapforce after passing the exam and process all the documents and now my account is locked at the screen with this message. I have emailed them but received no reply. I am wondering if I should cancel the account and reapplied.

    Thank you for your interest. Unfortunately, we are unable to proceed at this time. We will keep your qualification information on file, should other opportunities become available.

    1. Hello,

      I think your situation has to do with overcapacity of raters. See email bellow:

      We would like to take this opportunity to thank you for your commitment to the Map Analyst Program.

      We appreciate the efforts to join the program in which we had a high demand for the position.

      At the moment we have sufficient capacity, however we have placed you on a shortlisted panel. We hope to be in contact with you in the near future when our capacity needs increase for your market.

  9. Hi Tim.

    What great job you are doing here! Helping people without asking something in return. I thank you on behalf of all people that you have been helping.

    I wish you all the best.

  10. Hi Tim,

    Thank you for your generous efforts to answer newbies’ qustions.
    I am concerned about the tools which are/are not allowed to validate the user’s search result.
    I want to do my exam for Leapforce Shasta project. In the MapSearch Evaluation Guidelines it is written that all user searches should be re-performed by testers. Does it mean the tester should use an independent map and perform there the user’s original search? If it is so, which websites are accepted by Leapforce?
    2. Assuming no tools are allowed outside of Leapforce tools. In this case will they provide an alternative map (website or tool) where the control can be performed?
    3. How long time is given for taking the exam part 1 and part 2? (ie. 30 or 60 minues for start)

    Thank you for your clarification.

    With appreciation,


    1. Hi

      3:: part 1 os theoretical take around 1-2 hr its easy.
      Part 2 very crucial take almost 8- 10 hrs , depends. 20 question agin many questions carry series of questions.

    2. Sorry for a late reply.
      1. You may use any tools you wish for your research.
      2. Part 1 will take approximately 2 hours to complete, part 3 is a complex test which will require lots of time to complete, when I did this part I spent almost 5 hours to complete it.

  11. Hi Tim,
    Very useful post and thanks for valuable information. I have a doubt,
    1) Is there a Navigational Result for a Query.
    When to opt yes/no
    If result showing multiple outcome ans is no and when showing only one result ans is yes??
    Could you pls clarify.
    2) when the address showing abbreviations eg W/ave instead of West/Avenue as per official website accuracy should be marked Correct?
    Appreciate your advice

    1. I am sorry I cannot answer your questions I need more info in order to answer your questions.

  12. What should i write in the comment section of the second part of leapforce’s test? Also since im not from the states and the addresses do not work that way , do I rate the address as wrong if there is the W(est) symbol on the real address and not on the test’s address and vice versa?

    1. The address must be verified from official webpage and if anything missing quote as wrong address street name( whatever wrong). And vuce versa . In comment provide correct address.
      Hope it clarify.
      Could you clarify one of my doubt
      1) Is there a Navigational Result for a Query.
      When to opt yes/no
      If result showing multiple outcome ans is no and when showing only one result ans is yes??
      Could you pls clarify.

      1. Your question is missing additional context in order to be answered. There are a few specific cases when the query should be assigned with navigational rating, please refer to instructions for more information.

      2. what do i have to put in the comment section if the rating is excellent or navigational and everything else correct and perfect? The problem I had the first time I tried the test was that I did not know exactly what I had to write at the comment section . Does it have be specific like a command or can it be a more simple answer?

    2. The exam is designed in a way to test your understanding of the guidelines, most of tasks are US cases, before proceeding with the exam I recommend you to familiarize yourself with US address formats.

      1. Could really use your help about the question above mate . My exam is in 2 days !

  13. Hi Tim, I will work for LB, just got an email and waiting for the exam. Can you inform what rate per hours job. thanks.

  14. Hi Tim,
    Good day, I’ve some questions for you related to the exam, and I hope you help me.
    -When will we consider pin accuracy incorrect or approximate, Is it depending on sense or distance or what?
    -If the pin is incorrect, should we write the correct coordinates in the comment?
    -Is the correct coordinates mean any point above right result area?
    -For relevance, which result we can consider it navigational?
    -For comments should we write the exact address from official website and write the website which we get our correct address?

    1. Hello,
      I strongly recommend you to look up for answers and explainations to your question s in the project guidelines.

  15. Hi Tim, I didnt understand again same question:

    Shasta part 2: on every question this is 1st part

    1) IS their a navigational result for query: yes or No
    how to choose the correct answer.
    EX. if the query is about Walmart and showing 4 results, so ANSWER should be yes or NO
    or if address query and showing only 1 result, what should be answered.
    2) On results if W is written while official website says WEST and vice versa, then on address accuracy is it partial correct or correct
    Please Your quick advice is appreciated.


    1. Yes, you need to select user intent or distance issue as a reason for downgrading the rating bellow excellent.

  16. Hi Tim!

    I work at Leapforce for Nile Project but I wanna work a Shasta Project too. I emailed the support team and I know we can work on 2 projects but I can’t see other projects in my all projects page. They told me that I had to Project page regularly. Is it help when I check? Because I never saw there was another project or is it a city legend?

    Do you know someone working in two projects? What should I do? I’ll be glad, if you help me. Thanks your interest!

    1. Hello, you may find all available projects to you on your personal Leapforce page, if you have only one project available it means that at the moment only one project is running in your locale. You also may check another vendors for a similar projects in your country (Lionbridge, Appen).

      1. Is it legal to work for 2 companies? I read something on the internet that you can not work for two companies at the same time and I can not be sure of that.

  17. Hi!
    Does anyone know how tasks are counted? Does the system automatically record or do we have to enter manually? thanks

    1. You should count completed tasks and time spent on completing each task by yourself.Raters are supposed to enter time/number of tasks completed on every day basis (in case of Lionbridge Map Analyst), and enter only time spent on completing tasks in case of Leapforce (Shasta project).

  18. I failed Part2 on the Leapforce Shasta project exam and I am about to take it again until 13.nov. But there is one issue which I can’t fully understand: If there is no possible real world navigational result (due to the distance between two continents) but the result is the most relevant and the only possible should I still rate it as navigational even though I answer the question:” Is there a navigational result for the query?” with No? 

    Thank you in advance and hope to get your answer

    1. Hello,

      For some types of queries (e.g. country or address) the distance has no influence on navigational rating, the navigational rating has specific criteria. Queries like POI, countries, cities, addresses, in most cases they have navigational results in spite of any distance from user location.

  19. Hi Tim,
    I have passed the map analyst exam and the last one they sent me somethings as the tasks but not really according to the update guideline and I had done with their paid 60$ for 5 hours , but the answer from them that the final assessment is not meet metric quality system but they will contact me again after new opportunity coming.
    So what does it mean?

    1. Hello,
      I am sorry to say, but most likely you are removed from the project due to quality issues. I recommend you to apply for Yukon project if this project is available to you.

  20. May I ask the people who already work there., normally, how many tasks per 1 or 2 hours can one do? Thanks in advance

    1. Hello,

      Productivity depends on the complexity of tasks you do, some tasks might take a longer time to complete some of the might take a few seconds. Normally each task have recommended average estimated time for completion and raters should be within those recommended productivity…

  21. Hello Tim! thanks for all this great advice! I passed the exam, i ve started working in Shasta for a couple of days (4days/15 hours) but since that I have the “non survey available message” like trhee days in a row. Do you think that i ve been sacked?

    1. hello,

      It might be indication of your account being placed under quality review. You should wait for more info from your vendor.

  22. Thank you for the good tips! I never found the pre-exam portal, but I acdidentally just started the exam a bit sooner I was supposed to. Luckilly I had studied the manual pretty well before that, and I passed. I was pretty tired after I finished, because I wasn’t supposed to take the test after a workday, and in the evening! I finished at night. Oops. So here’s a tip for others: dont’t click the exam link before you are actually ready to take the test. 😀

    1. There are strict criteria for contract termination due to quality. Normally new raters are allowed to improve their quality after receiving substandard quality score, however if no improvement is demonstrated, such raters are in the top list for contract termination.

  23. Hi Tim I just have a few questions if you would be kind enough to answer:

    You mentioned the pay varies on the task, could you give an estimate of how much an average weekly or monthly salary would be.

    I’m currently working as an web search evaluator for Appen is it acceptable to work on this as well, I know you say it’s compatible with other vendors but just making sure.

    How active is this project in the UK market, are there usually tasks available or will there be long periods without work.

    Thanks for your help.

  24. The pay is pretty poor at first and I rarely hit minimum wage levels. The amount of tasks required will take more than 14 hours a week and not 10. Of course you will be much slower to start with. (£115 pw for this).Training is in-depth on the basic principles, but perhaps they could help more with specific and harder tasks. LB are very polite and friendly and I feel sad that I had to resign from the project.

  25. Hey Tim,

    Thanks for all the information.
    I’ve just received an email with all pre-exam logins etc.. but none of the browsers I’ve tried (safari, chrome, firefox, opera) allow me to access, just loops me back to Authorisation Required/login details.
    Do you have any ideas what browser I could use?
    I’m using MacOS High Sierra

    Many thanks

    1. Hello, I may assume that the problem is not related to a browser, it might be that the vendor has server error, in this case you should notify your vendor about the issue and wait for the solution form their side…

  26. for those of you have been doing this work for a while, what is the average hourly pay ?

    1. Ollie says: 07.01.2018 at 7:34 PM Edit
      The pay is pretty poor at first and I rarely hit minimum wage levels. The amount of tasks required will take more than 14 hours a week and not 10. Of course you will be much slower to start with. (£115 pw for this).Training is in-depth on the basic principles, but perhaps they could help more with specific and harder tasks. LB are very polite and friendly and I feel sad that I had to resign from the project.

      The payment rate per tasks depends on your market, including the complexity of each task, and payment may vary from, $0.22 to $1.00 per task. Since you are working in a freelance independent capacity, you can manage your working time as you like and work whatever times you want. But this is a part-time job, and the number of hours you can accrue is set at a maximum of 20 per week. You must not exceed the maximum of 20 hours per week unless authorized. Lionbridge expects new Map Quality Analysts to be active and to participate each week. You should expect to make 8-15$ per hour depending on your productivity and market.

  27. Hi Tim,
    I got two questions
    the first one is i applied for 2 projects the shasta project and the youkon project
    youkon didn’t reply for nearly 2 months now how ever shasta replied after 2 days, i passed the exam for shasta now and i’m wondering since i qualified for my first project now will it be any quicker for them to reply for youkon project now if i applied again or will it take the same amount of time.
    second question in my country if u visit an official website they mostly provide addresses in the form of intersections and landmarks how am i supposed to evaluate addresses in that case.

  28. Hi Tim, I’m due to take the exams in the next few days, Do you have to do the exam in one go ? or can you stop and return to it ? and i’m guessing there is no time limit ?

    1. Yes, you may stop and return to the stage where you stopped at any time before your exam due date.

  29. Dear tim

    I had succeeded to pass the final exam for Shasta project since more than 4 months
    Then recevid the below message

    Dear Yehia Ahmed Mohamed Abdelrafeaa,

    Thank you for your interest. Unfortunately, we are unable to proceed at this time. We will keep your qualification information on file, should other opportunities become available.

    Best Regards,
    The Leapforce at Home team

    As well as my account has locked with below message

    You are currently in the qualification process. Please refer to the latest status email sent to you for instructions on what you need to do to complete your qualification process.

    Sent many emails, but i didn’t get a reply
    Could you please advise what shall I do?
    Do I lost the chance to work ?

    Thank in advance

    Yehia AbdelRafeaa

    1. Hello,

      I think your situation has to do with overcapacity of raters. See email bellow:

      We would like to take this opportunity to thank you for your commitment to the Map Analyst Program.

      We appreciate the efforts to join the program in which we had a high demand for the position.

      At the moment we have sufficient capacity, however we have placed you on a shortlisted panel. We hope to be in contact with you in the near future when our capacity needs increase for your market.

  30. Hi Tim,

    This is really helpful page. I attempted and failed Lionbridge Map analyst Part 2. I did not complete reading the Guide twice and attempted exam. They are giving me another chance now. I remember few questions that I was struggling with, could you help explaining these:

    If the full correct address is given, and the result returns correct full address. But when i did research, i found that this is actually a business address, so business name is missing.

    there was a point in guideline, which said if address has Business name, it improves result quality.

    So there were many questions like this and I answered excellent (If business name was there it would be navigational).

    Could you also tell me what to do with YES/NO (for navigational query in the start of each question). If I press YES, can I give excellent in the relevancy?


    1. Hi Ali!
      I took the exam today and failed 🙁 I’m very sad, and I really don’t understand why, I read all the material several times, but there are many circumstances to take in consideration, however, I went to the guidelines to help me with the questions, so I don’t know where I failed.
      You’re are saying they are letting you take the test again. Did they contacted you or did you asked for? Because they say it can be taken only once. I’m from Brazil. I appreciate any tip you can give me. If you want we can discuss these kind of questions you have. Best regards!

    2. Wrong, putting the business name provides additional correct information but if the query was the full address even if the result doesn t display the business name is navigational…. address query.

  31. Hello everyone. I applied for the position and they have sent me the study matterial + a link for a 20 question (multiple choice) exam.

    1) Am I going to have only those 20 questions?
    2) I have heard that I am going to have a 2nd part with the “practical” questions that is going to be many hours? Am I supposed to take the second exam in one go? or Can I do it during 2-3 days?
    3) leapforce hasn’t informed me that it is going to be a 2nd part and I have a week for the 1st part. Am I going to have any additional time, in order to stady and take the 2nd part too? Or am I supposed to take both exams in 7 seven days?

    Kind ragards

    1. As far as I know you have to complete all parts of the exam within 7 days before exam due date.

  32. Just failed second part of map analyst exam for lionbridge. Thoroughly studied material and used it to justify every answer I gave, I really now this was a waste of time because no human actually reads them. I don’t think I could have done any better and feel disadvantaged that the questions were all American based, (I live in uk) is there any point in asking to retake, I do not think I would answer differently. Should I apply for different role at another company? I really like doing the map queries, but I am obviously not good enough at them.

    1. You may apply for another positions like Ads Assessor or Search Engine Evaluator, they are similar in terms of nature of work. Check Leapforce, Appen and Lionbridge for open positions in your country.

  33. I have come across a query where the viewport is small and the user is several blocks outside of it. The query is a chain business with several locations only a mile from the viewport. The guidelines state to use the user location if no locations are in or near the viewport. Using the viewport or the user location will significantly alter the ratings, but I cant decide if these locations are close enough to the tiny viewport to justify using it as the location. Any thoughts? Thank you.

    1. In this case you should base your rating based on user location and disregard the viewport.

      1. Hello Tim
        Where do you get your knowledge from? 🙂
        The guidelines indeed state to use the user location if no results are in or near the viewport but there is not a single example of how this should actually be applied. How near is near? Is there some kind of rule?
        (E.g. when the result is closer to users location than to viewport?)

  34. How to respond when the query is for a business (or chain business) e.g, [walmart], and the result is showing the correct location, but is an address type result with no business name or category shown? I saw this scenario twice, once in Shasta exam part 1, and once in part 2. I can’t find this scenario anywhere in the guidelines. How should the relevancy and/or name accuracy be rated?

  35. As soon as I finished the exam, I received an email that I did not pass the exam. It is not fair. It is impossible to read all the questions in a second. The second part of the exam has narratives. It is impossible to read and evaluate it. They stole my time. I had studied hard to pass the exam. I do not trust them.

  36. Hey Tim, thank you very much for your post.
    Are you sure there has the possibility to apply for Leapforce projects along with this one?

    1. You may apply to another vendor (Appen) for a similar search engine evaluator project and combine two project without any risk of IP address conflict.

  37. Hello Tim. I have heard from LB as an Online Map Quality Analyst and LP as a search engine elevator. While both require passing their test, is there one you would recommend over the other? Thank you,

    1. I have no preference for one over another projects, you should decide by yourself what project fit you better.

  38. Hi, Tim. Thanks so much for this article, it was very enlightening.
    I’ve been recruited to the Maps analyst project at Lionbridge but I already work for Appen on the Aztec Ads project, that is a search engine evaluating project. Are you 100% sure that those two projects are compatible? I mean, it seems like they have the same client, but with different task intents. I really wanted to try the assessment test for Lion, but I’m afraid to lose my actual job at Appen for data or IP adress conflit.
    Do you have any thoughts on that?

  39. Hi Tim, Could you please help? I always rate every result against the real world. But do I take the results in the rating tool also in account when rating one result in the real world? I know I have to rate the results in the rating tool individually but I’m a bit confused due to a sentence “Distance evaluations should be based on chain business locations in the real world, not just the returned results”. Thank you!

  40. Hi Tim, due to my use of a dynamic IP address, I was unable to take the lionbridge exam. Can I reapply using the same name? Thank you.

  41. Hello, With the pay per task payment is there still a time set that the task must be completed in? I remember years ago it was stressful to complete some tasks in the timeframe alloted.

  42. Hello,

    thanks for sharing your knowledge with us. I read above that after the test the reply comes in about 2-3 days. I took the test and I failed, Leapforce gave me a second change within 14 days to take the test again and so I will do. My question is that the fail email-response came in a second after I saved the test, so is it something I did wrong while taking the test and I got back a quick answer? Or is it something has happened to others also?

    1. Instant ‘fail’ response happened to me also. However, interestingly the ‘successful’ response was also instant on my second attempt.

  43. Thanks for the info. Can you work as a Rater (employee) for LB and also the Map Analyst contractor at the same time?

    1. At Lionbridge – NO, only one project is possible to work on, but you may apply for Maps project at Leapforce (Shasta) in this case you may combine two projects.

      1. Thanks for the response. I’m wondering where do you get your information from? I was told from Lionbridge last week you CANNOT work for anyone else. :-/

      2. LOL of course they tell you that you CANNOT work, but if you do not tell them about the fact of working on several unrelated projects they never ever will know about this fact.

  44. Hi

    Thank you for your excellent site.

    Can you tell me how it works if I take a vacation during my work with Lionbridge?

    Do I just forgo the opportunities on offer that week?
    Will I be penalised for taking time off?
    Can I continue to work from my laptop or will there be IP issues?
    Do you have any tips or advice in this situation?

    Many thanks for your assistance.

    Kind regards

    1. Hello,

      At Lionbridge you may fill out so called ‘my black out period’ form which is available on the Lionbrdge project specific portal.
      You may continue working if you travel with your country but you cannot work from abroad otherwise it will lead to contract termination.

  45. Hi Tim

    It’s definitely not easy taking these exams as a non-American, since all the questions are based on US scenarios and American ppls intentions. For instance searching for a bank. In the guidelines it’s quite clear that if you search “bank” it’s a category search and you’d want results that fulfill that level of service (and not just ATMs). But if you then on the other hand search for a specific bank name (“Citibank”) then I’m not sure of which level of service you’re searching for. Could an ATM fulfill that intent – or maybe work as secondary intent? Where I come from searches on bank names could just as easily be to find the nearest ATM from that bank but I don’t know if Americans think the same way or if you seperate those two entities very clearly?

    Secondly I am confused about the comment field being mandatory in the test. When working you shouldn’t have to fill out the comment field if the rating is Excellent or Navigational, right? But in the test you can’t get to the next question unless you’ve filled them out :/ I am afraid that me writing something in these fields will make the answer wrong when I know that I’m not expected to write anything there :/

    Thanks for a very informational post!! 😀

  46. First of all , an awesome resource about the Map analyst. Great work out there. You mentioned that Map Analyst position is compatible with projects of other vendor. As far as I know, Lionbridge is very strict when it comes to accepting people from companies like Leapforce, Appen, iSoftstone,etc. I am currently with Leapforce (yukon). You absolutely sure that neither Leapforce nor Lionbridge got any problem with somebody working for Yukon as well as Map Analyst position.

  47. In MapsSearchGuidelines on page 48, example is;

    Query: bank


    1 – Bank of America (EXCELLENT)
    10900 Orangewood Blvd, Orlando, FL 32821

    2- Bank of America ATM (BAD)
    10900 Orangewood Blvd, Orlando, FL 32821

    But if ;

    Query: Bank of America ATM


    1 – Bank of America (?)
    10900 Orangewood Blvd, Orlando, FL 32821

    2- Bank of America ATM (?)
    10900 Orangewood Blvd, Orlando, FL 32821



    1. Relevance depends on many factors such as distance from user location and/or viewport so there is not enough entry data to answer your question.

  48. Hello. Just would like to share as well.. the pins are not color coded on the test.. contrary to what was said on the guidelines.. its so tedious.. liike there would be 9 pins on the map and you dont know which result each pin is for.. and when u click on it only coordinates will be shown.. so i had to check in google maps what the address of the coordinates are.. maybe there was a way to work around it that is easier, maybe i just dont know.. it would have been a lot easier if it was color coded like its supposed to be.

    unfortunately i failed the test.. 😦 i spent so much time and effort on it.. 😦

  49. Thinking same as Hanne. I dont think that someone can pass the test outside usa. same situation with a query which has the search term “station”. would a gas station satisfy the user intent??? or should we only consider bus-train stations and give a bad rating to a gas station??? the guideline is very long but not clear!

    1. Exactly my thoughts about the gas station! I could not find any example in guidelines with any resemblance and make up my desicion.I think there are not enough examples and also no clear instructions about how the comments supposed to be.Agree its hard for a non USA pass the exam.

      1. it’s all commonsense dear. when someone says station, obviously he is referring to either bus or train station. If his intention was gas station, he could have said gas.

    2. You are wrong. I passed. Just be careful about the guidelines.

      Also, don’t worry about not passing exam because I got ”
      Thank you for your interest. Unfortunately, we are unable to proceed at this time. We will keep your qualification information on file, should other opportunities become available.”

      Doesn’t matter pass or fail. It is rare to get a job.

      1. Hi Davut,
        How much do they pay for evaluaters from Turkey approximately per hour?

      2. Hello, on average the payment hourly rate in your country ranges from 5$ to 7$ depending on the type of project.

    3. Think in the real world someone looking for gas what the chances the user types only station… very unlikely….

  50. Hello all,
    once again I took the test (Leapforce), and I failed, again. This time I did it in 3 days,not in just 1 like the first time but I think something is definetely wrong with the company. Maybe they do not want anymore participants because the rejection email came in seconds,literaly in seconds, just like the first time.How is this even possible?Is there anyone that actually succeed in this exam recently?Waste of time for me.

    1. Hello! I succeeded. I failed at first and got a second chance. The result comes in seconds because the entry exam is automatized, tthe tasks have previous rights and wrongs.

      1. I can understand that there is an automated formula , but what about the comments? Noone actual read those, such a waste of time.

    2. Hi Vivi

      Sorry to hear your frustrations at not passing on your second attempt. It is so disappointing to get that instant response after all that study time. I found the lack of feedback particularly difficult. I certainly felt like I had wasted time after my first unsuccessful attempt.

      You did ask if anyone has succeeded recently. I don’t wish to boast but I have just received the email
      (also instant) informing me that I have passed on my second attempt. All I can say was that I stuck rigidly to the guidelines and avoided using my intuition, common sense or gut feeling. The information in this blog was extremely helpful in changing my mindset to this approach.

      All the questions in the test have equivalents in the guidelines and therefore the answers are contained therein. I used the Find on Page feature and searched meticulously through the document for all the keywords I felt related to a particular question. It took me ten hours over two days.

      Not sure if I will be able to make any money but I do feel confident that I am on track!

      1. hi. i’m having difficulty with pin accuracy. If i may ask, when you’re on street view, where is the pin exactly? because it seem to disappear.. 🙁 tia

      2. also, if the result name are all the same (ex. Bank of America) and there are 9 pins on the map.. how do u know which pin is for a certain result? What I did was, I get the coordinates, then check google whats the location etc.. for each and every pin.. Im really new to this, Im sure there’s an easier way. Help pls. thank you.

      3. Hi single mommy,

        You can just click on the business name or street and it will take you to the pin automatically. There is no need to check google for each pin.

      4. Hi I m going to take exam in this week. I m facing it first time . May I know d whole procedure like after reading question do we have to open google map. N search for tht query n then rate it? I may sound foolish but literally not getting how would be the questions n what’s d approach to answer them, going thru d guide lines though. Plz answer it .thanks a lot

  51. I worked in Shasta project about one year. Passing the exam was easy, but the actual Job/guidelines are confusing and changes all the time. Shasta management is bit weird; Sometimes you don’t have any tasks on days/weeks and you never know why. I sent few emails to management and it took few weeks before they replied to me. Not professonial at all..

    I liked rating those tasks, but the volume of tasks was too low in my locale (outside USA)

  52. i failed the 2nd part of the test and after a few days I was invited to retake the test. I’m not sure if I should. Im having difficulty checking pin accuracy.. 🙁 when I check street view, I don’t know where is the pin at exactly.. I want to get the job but I feel I’d be wasting time if I’m not sure how to do one of the most important aspect of the job. If anyone can help me, it’d be much appreciated. To be honest, I don’t use online maps so I’m really new to th

  53. and I think there’s no need to dwell on the comments as the result of the test comes literally within seconds. Just put something short and simple to get you to the next task.

  54. This site is an amazing resource and I wish I knew about it last week. I have just failed Part 2 of the Map Analyst exam at Lionbridge, after 10 1/2 hours (which included meals). I tried to read the guidelines twice, but with having unusual family chaos around, I only managed one complete read. I found the exam a good learning experience as I re-read through most parts of them again whilst doing the exam. I now know I spent time on comments that wasn’t needed and this added to the time. It was disappointing to read after my fail result that I should have input the words “No Comment” where there is no need for a comment to proceed and that I should have entered almost anything at all. I should have kept comments very short, they don’t want detail.

    I’m in the UK and the US location of the exam mapping added some interest.

    This forum has given many other answers that I wish I had last week! I found the whole map analyst role quite interesting and if I get and opportunity to take Part 2 again, I might consider it.

    The exam assessment is interesting. They must have a clear set of answers but I wonder if they are also looking at time to complete the exam. With the almost instantaneous exam result, it must be machine marked.

  55. Hi,

    Is the Shasta project a timed job? For example, when completing a task does one have to set a timer and it cannot be paused until the task is completed? I am a stay at home mother (SAHM) of a toddler and an infant and am wondering if this is a doable project to work with two littles. I will need the flexibility of pausing from my work from time to time (some times more frequently than others) and am wondering if this will be possible. I remember with the Yukon project I could not pause in the middle of a task, and that was quite difficult with little kiddies to tend to.

    Thank you!


    1. You may stop qualification process/task at any time and resume the process at any time before your exam due date.

  56. Hi, thanks for your site. Very good infos here. Small question, how would you rate the relevance of a query for a chain business, like ”McDonald’s” if the result only show a near real McDo, but only the address without the name ”McDonald’s” in the result ? Thanks

    1. Relevance bad for user intent. Even though result shows correct address of the nearest McDonald’s, it only showed the address, which is not what the user wanted to see. If result actually showed the name McDonald’s then it would be Excellent relevance.

  57. Hello,

    Great article. Really gave me that boost of confidence before taking the exam.

    Just passed both parts of the exam. I didn’t really need to read the whole guideline (Took 2-3 days to skim through it to gain a basic understanding). I just used it as a reference while taking the exam. It wasn’t really that hard, the main thing is to understand user intent, and finding data errors in each result (use guideline for this purpose).

    Anyway, question for Tim or for those who passed. How long does it take Leapforce to allow you to start working after you finished signing all the electronic documents? I only received an email stating that they are reviewing said documents and that they will finalize my registration. This was 3 days after passing the exam.

    Thanks in advance.

  58. Hello,

    Thanks for the great article. Gave me that boost of confidence to take the exam.

    Recently passed the Shasta exam on the first try. I was intimidated by a lot of the comments about the difficulty of the exam, but it actually wasn’t as difficult as a lot of people made it out to be.

    I didn’t have to read the whole guideline. I just took 2-3 days to skim through it to gain a basic understanding of the whole map search process. The main thing is to just understand what the user’s intention was when they typed their query. Once you understand the intent of the query, then it makes everything else easy.

    There are A LOT of result data errors that must be corrected in the exam. Use the guidelines as a reference on how to find these mistakes. The errors you may encounter are wrong addresses, pin locations, incorrect name/classification and many more. Again, use the guidelines as a reference to find these errors, and how to comment on them accordingly.

    Anyway, a question for Tim or for those who have passed the exam. How long did it take for Leapforce to finish reviewing your signed electronic documents and allow you to start work? It’s been 3 days since passing the exam and the last thing I heard from them is that they were reviewing said documents and that they will finalize my registration.

    I would really appreciate an answer, and I thank you in advance.

    1. Hello,

      Normally the paper finalization process takes 3-5 working days.

  59. I completed 3rd Part of lionbridge exam on 3rd april 2018. But i didn’t get any mail from lionbridge that fail/pass. when can i expect mail from them. As my due date of exam was April 7th 2018.

    1. Leapforce exam results and Lionbriage exam results

      You will get overall exam results on the exam due date from Leapforce and exam due date plus one day if your vendor is Lionbridge. At Leapforce you will be asked to sign a few electronic forms and in 2-4 days you will be given access to live tasks. At Lionbridge you will have to complete educational course with no pass/no fail quiz before you are granted full access to the tasks.

    2. Hello sunil, are you from India? And what is this 3rd part? in emails and during webinar they said, we will have only theory part and practicals, what about this 3rd part which no one informed me. btw, have you started working on this project and if yes, what the pay scale?

  60. Hi!
    Thank for this precious post.
    I am reading now the guidelines and I have a question from the 10.7.4 section.
    What’s the meaning of soft categories?
    There is an example, but I can’t understand.
    Is “ski shop” a soft category?


  61. Hi Tim, Great job with this post.
    I have one query that came up during the exam and i’m curious to get another take on it, as it is really subject to interpretation.
    The question was for a chain business and the view-port was fresh and user outside. The view-port was very small though and so no results fell inside. In the guidelines it tells us to use the view-port location unless there are no results “within or near the fresh view-port”.
    Now how do you interpret “near” in this case? it’s so very subjective that it did my head in, and i’m sure they did it just to test us and see how we would see it. Also user is not far away but if we do switch to the user as the location intent the results change quite a lot so it is very important to the results.
    In the end I decided to use the view-port and consider the results near, but I can’t shake the feeling I was wrong.

  62. Are people still getting offers for UK work through Lionbridge? I failed the part 2 exam and had the impression that the quota of places in the UK has been filled. I enjoyed the work and felt that I was getting to understand it and do it well, but have been not been given a re-test opportunity.

  63. Hi Tim, great job with this post! I stumbled upon your article when I failed my first try because I wasn’t able to answer 6 items on part 2.
    I am from the Philippines and Lionbridge is hiring a Map Analyst for my country. First of all, even though the rating guidelines and concepts are universal, it is much harder for me to do the evaluation of places in the US. When they invited me to re-take the qualification again, I told them this concern of mine, and I hoped they will revise the items in the test to adapt to my country. Unfortunately, the contents of the test are similar as the first one. On my second take, I failed to answer just 1 item, and they told me I failed the exam. I replied back and told them that it would be fairer if they will evaluate the results of the test based on my answers to the items that I was able to answer. They should have just marked the last item as wrong, and give me a score for the totality of the test. What’s your take on this and can this be appealed?
    Thanks and more power!

    1. Hi, I do not think that you may appeal test results, simply because nobody will read your appeal because the qualification process is fully automated, your test answers are checked against correct one on a serve by algorithm.

  64. Hi Tim, Lionbridge gave me 15 hours of reading material and another 3-4 hours of practice tests, videos etc. 24 hours before the start day of the exam.
    I worked my butt off after they sent me a reminder email, passed the first test at 3am then over the next few days failed the prac. very disappointed to hear that you had more time to prepare. Anyway they can keep the f’ing 22 cents.

  65. HI
    I took the test and passed part 1.failed part 2. I NEED to undestand a few things will really apprieciate it if some one can help. 1.the query is for a particular station…its not a gas station so I would rate any other “station” as bad, user intent?

  66. Hi and thank you so much for this informative post! It was just what I was looking for. Can anyone tell me how much one can expect to make, say, per week? I see the rate for the tasks, but just wondering if there was some kind of ballpark range to refer to. Thanks!

  67. Hi, (from India)

    I started to work for falcon ads facebook project in appen one month ago. Now i got a mail from leapforce for nile project but i declined it because of same email address and ip address. They removed Even i applied for online map analyst in lionbridge i got rejected. Now what are the things should i need or to follow in order to get a job at lionbridge.
    Thanks in advance Tim.

  68. Hi, thank you for your blog, I find it very useful.
    I’m doubtful about how to rate relevance of a result: the query is a station (name and address) and the result is just the correct address without the name of the station. Should I rate it bad? I suppose I should consider it as a business, therefore it doesn’t satisfy the user intent

  69. Thanks for a really good overview and detailed explanation of the mapping job.

    I just did my exam and failed . I’m not in the u.s. and it was hard trying to understand the examples and results given with strange street naming etc so this was extra work to learn this.

    I’m not sure why I failed! 15 questions and I understood from the webinar you need 80 percent, so that’s 3 could be wrong, but with so many results in each wisdom for a query and with all the parts to each result- it could just be if any are incorrect it’s wrong for the whole question.

    Very annoying they just tell you failed.. No feedback as to percentage or what you did wrong. I spent 5 days reading the 210 page pdf and about 15 hours on both exams, youd think a bit more of an email thanking us for our free time given for this.

    Strangely I now have another chance to do the exam as they say they are still actively recruiting for this position. Perhaps they haven’t had much interest in my country with this recruitment so far. fingers crossed it goes better this time.

    Reading the comments below it seems many haven’t read the basic information they send you about the test format, and haven’t even read through the guidelines completely.. It’s a lot to read and it could be much simpler to understand it they’d done some video clips showing through examples.. Anyway they obviously want a certain person that will spend time to really try to understand how the results are rated and not waste time with others who just don’t bother to research.

    My too. The more you read and do the examples and reread examples at the end of the guidelines PDF, then it really starts to make sense. Ok I failed it but I felt at least that by the end of it I’d really interestin what the mapping role requires you to do..

    Good luck everyone.. I’ll let you know how I get on 2nd and final time!

    1. Managed to pass second time. Still was no detail about percentage or where i went wrong now into the actual work..and…into another test lol (100% but can redo until get it right).

  70. Hi Tim, I took part 1 and part 2 in one sitting yesterday and got an immediate “failed” reply. I want to do the Practice Exam again to review but I can’t seem to access it anymore. Have emailed Leapforce but no reply till now. We can’t access the Practice Exam again once we are done with it so only have the guidelines (and video) to review?

  71. Hi Tim. Thank you for your blog.
    I’m from Poland and I failed Lionbridge part 2
    I hope that someone here will be able to help me.
    The majority is understandable to me but…:)
    I remember one question from the exam.

    The query was for IRVINE STATION.

    All results were located in Irvine:
    Irvine Station (Railways)
    Tea Station – business
    Strain Station – Business ( Cannabis Dispensary)
    gas station

    My rating was
    Irvine Station (Railway) – Navigational
    Tea Station – Acceptable
    Strain Station – Acceptable
    gas station – Bad

    After rereading the Guidelines again I am still not sure how to rate the business with the word “station” in the name.

  72. Relevance is bad due to user intent. User searched for a specific transport station. The results givene are not secondary intent, as the station in the name isn’t related to the business type.

    Reread the guidelines on secondary intent

  73. Hi Allen,

    About the ‘Project computability’ mentioned above , are we allowed to work for leapforce’s any other project and should we mention this to lionbridge or even to leapforce while submitting the application.

    1. Of course you should not mention about this, they do not like the fact of working for several vendors.

      1. So Tim, now i am working for lionbridge for Maps analyst.

        Am i eligible for all the other project with leapforce? or should i be avoiding their shasta project maybe, i am not sure?

  74. Hi Tim, hi all.
    I noticed ppl have been asking about scenario that is bothering me and nobody answered nor explained. So if anyone passed the exam or knows hhow to apply guidlines rutes in this particular case please leave a comment. I will appreciate any help.
    I am talking about case when there are No results in or near fresh viewport. I know that i am supposed to disregard viewport and use users location but it was only mentioned twice in Guidlines without further explanation and no examples at all. Im doing exam now and there is a query for Burger king in Tampa -9 results and very small viewport. Maybe someone did it and has some piece of advice. Thanks. C.

  75. I wrote Lionbridge requesting information how to apply this rule correctly.
    I am very interested in the answer, because I wonder if it is difficult only for me to understand?

    It was mentioned only twice in guidelines and there is no example to match such a scenario:

    2.3.2. Implicit Location (page 18)
    “Results are expected in or near the viewport area. If no results
    can be found in or near the viewport, use the user location as
    a secondary location intent.”

    5.3. Distance (page 33)
    “Note: If no results can be found in or near the
    viewport, use the user location as a secondary
    location intent.”

    Tim, you wrote:
    “18.02.2018 at 8:51 PM
    In this case you should base your rating based on user location and disregard the viewport.” – How come you are so sure about it? Is it really so simple and I alone cannot understand it?

      1. I do wish this straightforward question that’s been asked multiple times could be answered in this forum. What defines near?

        2.3.2. Implicit Location (page 18)
        “Results are expected in or near the viewport area. If no results
        can be found in or near the viewport, use the user location as
        a secondary location intent.”

        5.3. Distance (page 33)
        “Note: If no results can be found in or near the
        viewport, use the user location as a secondary
        location intent.”

        Tim, you wrote:
        “18.02.2018 at 8:51 PM
        In this case you should base your rating based on user location and disregard the viewport.” – How come you are so sure about it? Is it really so simple and I alone cannot understand it?

  76. Hey Tim,

    I have submitted my Application to LeapForce and havent heard from them, been two months now.

    I submitted my Application under my yahoomail, i read somewhere that they only except if its an gmail account.
    Should i resubmit the application now with my gmail account and its been just two months now.

    Please, Help!


    1. How long does take to hear back from Lionbridge/Leapforce/Appen?

      It depends on many factors, sometimes immediately, and sometimes it can take 3-4 months.

      It might be that at the moment of your application submission, quota of raters for your country was reached and therefore Leapforce or Lionbridge does not hire new rater at the moment. As soon as they fire unsuccessful raters, they continue hiring new people to maintain the required number of raters in your country.

      1. And is it true they reject your application if you submit it under yahoo mail?

        if someone fails the leapforce Exam are they allowed to resubmit the application again maybe after 6months?

  77. Hi Tim, your work on this page is much appreciated. do you know what’s the pay rate for Indian market? I just got an invitation email so I was just kind of curios to know the rate & how difficult it will be to pass the test?

    1. Hello, Unfortunately I do not have information about payment rater in India.

    2. $7 with leapforce. it is sure very difficult if you dont study your PDF guideline well enough! Study Those Well !

      1. Hi Ky, thank you for your reply. Are you currently working on it? I just got another email stating I will be having 7 days to take up the exam and tomorrow they will send me reading stuff. I’m currently working full time so do you think, I can read all that stuff while being on another job on full time basis or should i take off from office?

  78. try reading it twice , understand it well.

    if you give time reading it twice with your office going, it should be fine.

    understanding it is the KEY!

    All the best.!

  79. leapforce, after being acquired by Appen, has reduced the pay of new raters from $7/per hour to $3.5/h for Asian countries.

    1. Wow that’s is a lot, I predicted the decrease early in my article about Leapforce acquisition by Appen.

  80. it feels barely like anything now after $3.5! And For the Social Media Evaluator you are paid $3.5 with just an hour of work everyday, better with SEE where you are at least have 20hours of work weekly.

    So, TIM it shouldn’t make any difference submitting application to LEAPFORCE or APPEN now either them are the same.?

    1. thank you for your replies. LB emailed me with exam guidelines and other related stuff and they have given 7 days time to finish exam. I think it’s it’s very difficult to get prepare in such a short notice since I’m a regular employee. could you pls help us in knowing what is the passing percentage? they have 20 theory questions and 46 practicals. after going through guidelines partially i tried sample theory exam and I got 15 corrects and 5 wrongs. What is the main tip that you experienced guys can give? I also attended a webinar yesterday and main focus was on Relevance criteria.

  81. Dont fool yourself bro for getting decent score of 15 in Sample theory , The real exam is a lot tougher. read guidelines twice atleast. Relevance is bit complicated, understand the section study well.

  82. Hello Ky, thank you for your suggestions but yesterday while going back to my home from workplace early morning (since I work in nightshift) I planned to change my password and once I logged in, i saw one theory question for which I knew the answer, i clicked on answer and submitted. somehow within 45 mins I answered all the questions. As soon as I submitted my 20th questions, I got the practical exam question. I don’t think questions were that tough in theory but yes I agree practical will be very tough.

    1. Just wanted to add that I passed the exam. It was quite tough & an advise is to use commonsense since lot of questions are very confusing. Waiting to get my account activated.

  83. Hi Tim
    Great post. I have a question on how to zoom more or open the map on a new window on the exam port as it is small.

    Also, I have the same question regards viewport vs user location where no result is in the viewpoint but it is so close. The user is outside the viewpoint. In this case what should be used?

    What if the viewport is just overa shared parking lot of the business but the business building is not in the viewport. Will the business be considered as if it is in the viewport?

    Seqrch is for location based station and the result has gas station. Of course the intent won’t be for gas station but provided this is in the location … will this be rated acceptable or bad?

    1. If the view port is too small, use a user location for rating results. The rest of your questions I cannot answer here because they are referring to a specific exam tasks.

  84. Hello Tim do they give third chance for Map Analyst position? I have failed twice and too desperate to get this job.

  85. Hey Tim, great work on helping us all. I am confused with the reading material. if the query is for one specific school (A), and the result is that school (A), and also another school (B) in same viewport, do we rate the second school (B) as bad relevance, due to intent or rate it as acceptable considering the associated entity rule, thinking that it is in viewport and also may be of use to the person asking the query.

  86. Hi Tim,

    I’m working as a Map Analyst for Lionbridge a month ago.I’m a resident in Saudi Arabia and I’ll take my annual vaccation for 3 months to my home country.Can I do my work from my home country during this period?

  87. Hi Tim,
    I had applied for the position of map analyst and got an email back with the test link but failed in 1st attempt and again i got chance of re-attempt but due to a family emergency could not give the test.
    Is there any other way to apply and get this job ??I tried re-appling by changing email id and ip address but did not receive reply.I am completely eligible for the job.
    Please help asap

    1. I recommend you to write an email to agent-qualification support and ask if one more attempt is possible, in many cases they allow candidates to attempt the exam up on request.

  88. Hi Tim,

    I am try to complete my application a fresh with leapforce after not getting any reply from them for six months , and while verifying my mobile phone, while filling my application its say ‘this mobile phone is already register’.

    So, i have register it with my friends mobile now. Can i switch back to my phone after clearing the test. but its say my mobile is already registered with other application.


  89. Hi Tim,

    When the POI-name is correct, the address is incorrect (street number) and the pin drops on the address in the result, should the PIN be rated correct?


    1. Yes, the pin should be rater as correct, because pin rating is an independent rating.

  90. Hi again,

    When the query is for an address (street + street number that exists), what should be the relevancy rating when the result is that same street and street number, plus a street number extention?

    Bad, due to user intent?


  91. Hello,
    which job gives more money,map analyst or personalized social media evaluator in Lionbridge?

  92. I just did the qualification exam with appen and immediately learned that I failed. I was thoroughly studying the guidelines before and during and the exam and was certain I was right for the majority of my answers. I put official website links in the comments and explained my reasoning for every result that wasn’t excellent or navigational. What could be the glaring issue that made me fail ? When trying the exam again is it necessary to fill the comments that much like I did ? I am lost..

  93. I failed the exam but for the life of me I don’t understand of why. I studied the guidelines thoroughly before and during every problem, put useful website links in almost every comment, justified all errors. I was certain of 95 % of my answers… I need to know where I went wrong. Do I need to write comments on every result, or was I doing too much ? Also, is every result faulty in some way ? Cause I found that a lot of them were fine except for distance problems causing relevance demotion and maybe approximate pin placements…
    What to do ??

  94. Leapforce acquired by Appen, is conflict interest still there. i work in yukon project now , can i submit my application to lionbridge mapping programme.

  95. Hi Tim,

    I attempted the Yukon test with Appen and failed at Part 3 both times. I applied to lionbridge now for the maps analyst position and have been given a date for the exam. However reading about how hard everyone is saying the exam is, I am reconsidering taking it. Would you say that the maps analyst exam is harder than yukon? Or just as hard?

  96. Hi,

    I am gonna take the exam as a second opportunity, I remember some questions. What I am guessing now is regarding the main question “is this a navigational result”? When should I select it as “yes”. Hope you guys help me.

  97. Is Lionbridge internet assessor compatible with Appen maps? Thanks. I’ve been with LB for 2 years and now have offer to do test for Appen maps.

  98. Hi Tim,

    Two weeks ago I was offered to qualify for Shasta project. I opened the preparation documents and spent these last two weeks making notes on the guidelines in preparation for beginning the study guide. Today I opened the projects page on Appen Connect and the project has disappeared. Have I been disqualified for taking too long to begin the exam? I did not receive any email letting me know I had been disqualified.

    1. Hello, most likely Appen hired enough raters, for this reason the project has been removed from your list. I recommend you to write to agent support and ask if they still can offer you qualification for this project.

    2. I recommend you to write a message to agent support and ask about project availability.

      1. Thanks Tim. It has reappeared now. I think I took too long last time in making notes on the guidelines.

        I’m at the end of the study guide now and it says that the practice exam is coming but not yet available. I only have maybe four more days to complete the qualifying exam. Should I wait for the practice exam to appear? Will it appear if I press the button on the project page that says ‘I wish to proceed with the qualification process’?

      2. Yes you need to press “I wish to proceed with the qualification process” in order to begin qualification.

  99. Glad to find your post, I am studying for the Map Analyst exams and have a few questions I am hoping you can help with: 1.) First one;
    Is there a navigational result? I am getting confused on the difference between. Is there a navigational result And Relevance Rating. I see in the GL lots of example queries that answer it as “YES” but fewer examples for what a NO is. I assume if I answer NO, no other rating will be done for that TASK….? In the GL they give a query example of 2 locations and that is a NO and coordinate points to an intersection and that is a NO. But then later in the GL I see a task with ratings for 2 localities in Europe (as if user is creating a route) and they rate it.

    Also can you tell me which source is superior for validating correct address: USPS or corp. website? If they differ is there a rule to always go with USPS or is it just which ever matches the result as to be more lieutenant, or maybe there has to be more than one source to validate the correct address? I used property tax records for a practice question and go it wrong. I as was told USPS trumps property records! But then in another question I went with USPS and missed it because the answer used website to validate address…. very frustrating trying to understand what is a consistent way to analyze the data.

    1. Hello,

      Navigational result is the only one implied by a user, normally the following queries have the navigational results: specific address or business entity, locality, city, etc.; when query is a chain business query and there is only one result near to user location/viewport.If only one result is present in the explicit area, that particular result can be eligible for Navigational (in case of chain query).

      You may use different sources for address verification, the best ones are USPS, official websites of entities, Google maps.

  100. Hi, If a relevance rating is bad, how should i rate the name and address accuracy. The Query is for station ‘X’ and result is station ‘Y’. Shoul i rate accuracy of station Y even though bad relevance?

    1. Relevance rating and name/address accuracy are independent ratings it means you rate name/address accuracy as normal.

  101. Hi, if you don’t mind me asking and happen to know, what is the pay for the initial task type for the Polish market? I have been considering applying 🙂

    1. the average payment rate for EU raters ranges from10EUR to 14EUR per hour depending on productivity rate.

  102. Hello I wanted to ask what I do in this situation
    You are currently in the qualification process. Please refer to the latest status email sent to you for instructions on what you need to do to complete your qualification process.

    thank you

  103. Hi Tim. Intense exam but I passed first try and passed the new rater simulator. This is Appen Shasta. But I’m concerned by the service level reviews they seem to put raters under and how random they seem to be. I’ve been lurking in the project chat and in the last 10 days several long term raters have reported been suddenly and randomly been out under service level review. Someone was put under review for getting like 79.4%, I mean come on. It seems it’s all automated and no human consideration or respect for your history or overall quality throughout your months/years of employment. The mood on the rater chat is constantly fearful and afraid to attract a review or bad audit. I think it’s 2 SLR and you’re out so is there realistically any security to this project? I’m also afraid to ask in chat as everyone seems to scared on there and probably wouldn’t be honest. Is this just a “make hay when the sun shines” and expect to be dropped at any moment project? It’s so difficult I’m amazed they don’t try to retain workers and offer better support as I imagine many fail or are put off by the lengthy guidelines. I’m starting my first day as a real rater tomorrow and not really looking forward to it with all the insecurity and nervousness I perceive.

  104. Hello,

    This project is long-term, as all projects of this kind you should expect monthly quality review (blind test) and in case rater do not demonstrate sufficient score, such raters will removed from the project due to not meeting quality expectations.

  105. I’ve been a map analyst for for about 6 months. I passed the first time. They’ve sent me a few saying I need to do better and a few saying I’m doing better. Also, they keep adding more and more tasks. Then suddenly they send me an email saying if I keep making the same relevance mistakes, they’re going to remove me from the program. But, they’re not really clear as to what my mistakes are. When I’ve asked them for examples in the past, they’ll send me one example that doesn’t seem like they copied the information correctly. They did tell me that Distance is part of what I need to work on, but it’s so subjective. Do you have any advice on how to work on my mistakes?

    1. Probably the best way to improve your distance rating skills is to revise an appropriate section of the guidelines, paying attention to rating examples.

  106. Hi, can i work on Project Shasta at Appen, while working as an Ad Quality Evaluator at LionBridge?

  107. Hi every one,

    I received this email from Appen, can someone explain to me what it means?

    I worked very hard on this project and spent a lot of time and my rating score was up to 85%.


    “Dear Search Engine Evaluator,

    You have been permanently blocked from acquiring tasks in the Shasta project due to administrative reasons. The decision to end your contract is based on our proprietary internal business processes and further detail is not available.

    Please visit the Projects page on the Appen Connect website for other projects that may be available to you.

    We would like to remind you that you are obligated to maintain the confidentiality of the Shasta project and any information related to it. Please destroy or securely discard all instructional material related to this project that you might have downloaded or printed from TryRating or the Appen Connect website.

    We appreciate your participation in our projects and wish you all the best.

    Kind Regards,
    Your support team”

    1. It looks like you violated vendor or project agreement such as creating multiple accounts, working on another project locale using VPN, etc.

  108. Thank you Tim,

    I only have one account, I don’t working on another project locale and if I use a VPN, I should have been banned from the company but here they offer me to work for other projects.

    I don’t understand, I spent a lot of energy on this project and I respected all the rules.

    I sent an email but no response.

  109. Thanks Tim

    But if I use a VPN, they should ban me from the company, and in their email, they suggest that I visit other projects that may be available to me.
    I have only one account with this company and I worked in my local project.

    It’s very frustrating, I spent a lot of energy on this project, I only ask for honesty

  110. Hi Tim,
    I’ve done my study and about to take the Shasta test, however when I click the link, it took me to this rater qualification page. I can reset my password, but problem is i don’t think i have a username. when I join the APPEN, only use the email address. i tried to use email address as username, it doesn;t work. any suggestion? thank you

    1. Appen will send you the login details in a separate email. If you didn’t receive it you should reach out to them.

  111. Hi Tim.. Please help…

    1.3.1. Query-Level ‘Navigational Result’ Question
    “You will answer a top-level question for each query to indicate whether there are any results in the
    real world that would completely satisfy the user intent.” There is a top level question on each question on the exam. “Is there a navigational result for this query? Yes No”.

    This question has nothing to do with relevance? The question is asking about the result. The GL is not clear for me. Can u elaborate? And Give an example?

    “Is there a navigational RESULT for this query? Yes No”.

    Thanks! I really need this job! 🙂

Leave a Reply