Run your experiment

Once you have completed testing and setting up your experiment, you can run your LIONESS experiment online. Here we describe the steps to collect data with participants recruited from Amazon Mechanical Turk. Before you run your experiment, it is useful to take a look at this paper discussing best practices and methodological details of conducting interactive experiments online.

Recruit participants

If you have access to an established laboratory participant pool (e.g. through your research institute), you may be able to recruit your participants for your LIONESS experiment from there. Alternatively, there are several crowd-sourcing platforms available to recruit participants for online experiments. Here we describe how to recruit participants from Amazon Mechanical Turk (AMT). A description for Prolific Academic will be added soon.

Setting up a HIT on Amazon Mechanical Turk

Once you logged into your AMT account, click on the tab Create and then choose New project. Among the options displayed, you might want to use Survey Link. This type of project will allow you to request a code for the task to be paid. Hence, participants in your study will complete their task, see a unique code at the end of your LIONESS experiment and then enter it as a code in this type of survey.

Select Create Project and fill in the required details for the tab Properties as you like (title, description, reward per assignment, etc). Select Design Layout (shown below). In this page edit the content that you want your participants to see, usually the title and description you already used in the previous tab will be enough.

Once you are done with the edition, press Source and search for the two instances where http://www.linktomysurvey.com appears. Replace these with the link to your LIONESS experiment. You can find the link in the control panel under address for participants. Press Source again, and finally click on Preview. If you are happy with the way your task looks, press Finish.

General pointers for writing a HIT description and setting up a can be found in a paper by Arechar et al..

Make sure you have the control panel of your experiment open when you publish your HIT.

Monitor your experiment

During a session you can monitor your experiment using the control panel. This allow you to track the session’ progress, and browse through all data associated with the session (parameter settings, participants’ decisions, etc).

End of a session

At the end of a session, you can download all data as an Excel file by clicking the button Export database. This will download the database of the experiment in Excel format. The first five tabs correspond to the five tables underlying your experiment. The most informative table for the data analysis will often be the decisions table, as it stores the responses of the participants in the experiment.

Pay your participants

One can assign bonuses manually in the AMT user interface. However, especially when sessions are large, it is often handy to pay participants with the help of a script. One useful software for this is Amazon’s Command Line Tools. Click here to get the CLT running on your system. To do this, follow these steps:

  • On MTurk, download and open Batch results file
  • Copy all its contents to the clipboard
  • Open the LIONESS results file in Excel and paste the copied cells to cell A1 of the tab ‘batchResults’. The Excel file will automatically link the LIONESS code and its earnings to the MTurk worker ID of the participant.
  • The tab paymentsMTurk then contains the ready-made codes you can use in MTurk Command Line Tools.
  • Double-check if the bonus amounts in the column bonus are correct
  • Add a description explaining participants why they earned this bonus and copy that into all rows of that column
  • The column MTurkPaymentToolsCode will contain a list of codes that can you can paste into CLT