Run your experiment

Once you have completed testing and setting up your experiment, you can run your LIONESS experiment online. Here we describe the steps to collect data with participants recruited from Amazon Mechanical Turk. Procedures for Prolific are very similar. Before you run your experiment online, it is useful to take a look at this paper discussing best practices and methodological details of conducting interactive experiments online.

Note

LIONESS experiments can of course also be used in the physical decision making laboratory. As dropouts are very unlikely to occur, please make sure that you set the parameter dropoutHandling to disable exclusion.

Recruit participants

If you have access to an established laboratory participant pool (e.g. through your research institute), you may be able to recruit your participants for your LIONESS experiment from there. Alternatively, there are several crowd-sourcing platforms available to recruit participants for online experiments. Here we describe how to recruit participants from Amazon Mechanical Turk (AMT). Prolific has very similar procedures.

Participants will enter your experiment through a URL link, which you can find in the Control Panel, under ‘link for participants’.

Note

You can send parameters from MTurk (or Prolific) to your LIONESS experiment using the URL parameter ‘tic’. For example, the ‘link for participants’ you post on MTurk or Prolific could read [your_ip]/[your_experiment]/_beginParticipant.php?tic={{%PROLIFIC_PID%}}.

Setting up a HIT on Amazon Mechanical Turk

Once you logged into your AMT account, click on the tab Create and then choose New project. Among the options displayed, you might want to use Survey Link. This type of project will allow you to request a code for the task to be paid. Hence, participants in your study will complete their task, see a unique code at the end of your LIONESS experiment and then enter it as a code in this type of survey.

Select Create Project and fill in the required details for the tab Properties as you like (title, description, reward per assignment, etc). Select Design Layout (shown below). In this page edit the content that you want your participants to see, usually the title and description you already used in the previous tab will be enough.

Once you are done with the edition, press Source and search for the two instances where http://www.linktomysurvey.com appears. Replace these with the link to your LIONESS experiment. You can find the link in the control panel under address for participants. Press Source again, and finally click on Preview. If you are happy with the way your task looks, press Finish.

General pointers for writing a HIT description and setting up a can be found in a paper by Arechar et al..

Warning

Make sure you have the Control panel of your interactive experiment open. The Control panel regulates the synchronisation of participants throughout the experiment and regulates their progress through any loops. When running your experiment, <b>make sure that only a single LIONESS experiment is active in your browser</b>. Having open multiple experiments simultaneously in the same browser leads to conflicts which may stop synchronisation. Also, before you launch your experiment, make sure that the number of allowed players is high enough when you publish your HIT.

Monitor your experiment

During a session you can monitor your experiment using the control panel. This allow you to track the session’ progress, and browse through all data associated with the session (parameter settings, participants’ decisions, etc).

End of a session

At the end of a session, you can download all data as an Excel file by clicking the button Export database. This will download the database of the experiment in Excel format. The first five tabs correspond to the five tables underlying your experiment. The most informative table for the data analysis will often be the decisions table, as it stores the responses of the participants in the experiment.

Pay your participants

Warning

As of June 2019, Amazon stopped supporting Command Line Tools. The description below is based on the new (and more user friendly) Command Line Interface.

The participation fees of your experiment can be paid painlessly via the MTurk website. Bonuses (which are often performance-specific) can be assigned manually in the AMT user interface. However, especially when sessions are large, it is often handy to pay participants with the help of a script. One useful software for this is Amazon’s Command Line Interface (CLI). Click here to go to the installation webpage for CLI. When you install, you can stick to all default settings. Here we assume that you use Windows. For other platforms (e.g., Mac), the procedures are very similar.

Once the CLI has been installed, open your command line shell (in Windows type cmd in the search bar of your Start menu, in Mac OS and Linux it is called Terminal) and locate CLI on your hard drive (the default is C:/Program Files/Amazon/AWSCLI/bin so your command line needs to point to that location). Connect CLI to your Amazon account by typing the command aws configure and hit enter. This opens an interface for entering the credentials of your MTurk account (the access key and the secret key, which can be found in your AWS account). The region code needs to match the one you set when you created your AWS account (the default is us-east-1). Set the output format to text.

You can check whether the connection to your MTurk account has been successful by typing aws mturk get-account-balance. This should return the current balance in your account.

Once your CLI has been successfully set up, you can pay participants of your session by following these steps:

  • On MTurk, download and open Batch results file

  • Copy all its contents to the clipboard

  • Open the LIONESS results file in Excel and paste the copied cells to cell A1 of the tab ‘batchResults’. The Excel file will automatically link the LIONESS code and its earnings to the MTurk worker ID of the participant. NB: make sure that when pasting the copied cells, Excel parses the csv into about 30 separate columns (this parsing is the default for the English version of Excel, but might be different for other language versions, e.g. German; please also make sure you use periods (‘.’) rather than commas to separate digits). In case you used multiple smaller batches on MTurk to recruit your participants, you can paste the data from the csv files right below each other.

  • The tab paymentsMTurk then contains the ready-made codes you can use in MTurk Command Line Tools.

  • Double-check if the bonus amounts in the column bonus are correct

  • Add a description explaining participants why they earned this bonus and copy that into all rows of that column

  • The column MTurkPaymentToolsCode will contain a list of codes that can you can paste into CLT (without the column header)

Note

In case you run your experiment on Prolific, you can use the column called ‘ProlificPaymentCode’ from the payments tab to make bulk payments. You can find instructions for how to do this here: https://researcher-help.prolific.co/hc/en-gb/articles/360009222573-How-do-I-send-bonus-payments-#heading-0. You can simply paste the codes from the column in the form ‘Bonus payments’.

Note

In case you run your experiment in the decision making laboratory, you can refer to the session table to calculate the payments for each participant. If your laboratory uses a system with numbered cubicles, you might consider recording the cubicle number in a screen of your experiment; right in the first screen typically makes sense. Then, at the end of your experiment (e.g. in the final screen) you can calculate the participants’ total earnings and write that to the database, for example, by using the JS command setBonus. This procedure will allow you to match the LIONESS-internal playerNr to the cubicle number in your laboratory.