Create and Rate

Create and Rate is an experimental template supported by PsyNet. It allows matching any creation (microphone recording, GSP, text input) with ratings in the same experiment. For example, it can be used to validate stimuli during the experiment or it can be used in a chain, in which creations are passed to the next iteration if they are rated highest by the majority (selection) or obtain the highest average score (rating).

To write your own Create and Rate experiment, you need to implement three classes:

  • The creator class,

  • the rater class,

  • and the trial maker

Let’s present an image of an animal to the creators and ask them to come up with a description of that animal:

from markupsafe import Markup
from psynet.modular_page import ImagePrompt, ModularPage, TextControl, PushButtonControl
from psynet.trial.create_and_rate import (
    CreateAndRateNode,
    CreateAndRateTrialMakerMixin,
    CreateTrialMixin,
    RateTrialMixin,
)
from psynet.trial.imitation_chain import ImitationChainTrial


def animal_prompt(text, img_url):
    return ImagePrompt(
        url=img_url,
        text=Markup(text),
        width="300px",
        height="300px",
    )

class CreateTrial(CreateTrialMixin, ImitationChainTrial):
    time_estimate = 5

    def show_trial(self, experiment, participant):
        return ModularPage(
            "create_trial",
            animal_prompt(text="Describe the animal", img_url=self.context["img_url"]),
            TextControl(),
            time_estimate=self.time_estimate,
        )

We can now specify a rater which will see the description and the image of the animal and has to rate how well the description matches the image:

class SingleRateTrial(RateTrialMixin, ImitationChainTrial):
    time_estimate = 5

    def show_trial(self, experiment, participant):
        assert self.trial_maker.target_selection_method == "load_balanced"
        assert len(self.targets) == 1
        target = self.targets[0]
        creation = self.get_target_answer(target)
        return ModularPage(
            "rate_trial",
            animal_prompt(
                text=f"How well does this description match the animal?<br><strong>{creation}</strong>",
                img_url=self.context["img_url"],
            ),
            PushButtonControl(
                choices=[1, 2, 3, 4, 5],
                labels=["not at all", "a little", "somewhat", "very", "perfectly"],
                arrange_vertically=False,
            ),
        )

The last thing we need to implement is the trial maker. You need to decide how the ratings are made, whether the raters select or rate (here they do the latter). If they rate, do they validate one stimulus or all at once? Should they rate the creations at the current iteration or also the creation which was passed on from the previous iteration?

The trial maker needs the following parameters:

  • num_creators, which sets the number of creators e.g. two creators,

  • num_raters, number of raters; if people only rate 1 stimulus, the number of raters needs to be an integer multiple of the number of rated stimuli, i.e. num_creators (and optionally the previous iteration),

  • node_class, the class of the Node; in most use-cases CreateAndRateNode is fine,

  • creator_class=CreateTrial, set this to your creator class

  • rater_class=RateClass, set this to your rater class

Optionally, you can set

  • include_previous_iteration (default False) which indicates if the previous iteration is rated. If this is the case you need to specify a seed in the start_nodes, e.g.:

start_nodes = [
    CreateAndRateNode(context={"img_url": "static/dog.jpg"}, seed=seed_definition)
]

  • rate_mode which can be set to "rate" if people can give integer ratings to the stimuli or "select" if raters are faced with all creations at once and have to pick one

  • target_selection_method, can be set to "all" (required if rate_mode=="rate") indicating that raters rate all creations or set to "one" which randomly selects one target (internally it prioritizes creations that obtained least ratings)

  • randomize_target_order (default True) which indicates if the presentation order of the targets is randomized. In most cases this should be set to True.

  • verbose can be set to True to print the Create and Rate decisions to the experiment log

The TrialMaker class just needs to inherit from CreateAndRateTrialMakerMixin and some TrialMaker class, e.g. ImitationChainTrialMaker:

class CreateAndRateTrialMaker(CreateAndRateTrialMakerMixin, ImitationChainTrialMaker):
    pass

It is also possible to customize the behaviour. For example, say we want to separate raters and creators into two different groups which is set in participant.var.is_creator. We can then implement the following:

class CreateAndRateTrialMaker(CreateAndRateTrialMakerMixin, ImitationChainTrialMaker):
   def get_trial_class(self, node, participant, experiment):
         proposed_role_class = self.get_trial_class(node, participant, experiment)
         if participant.var.is_creator:
             if proposed_role_class == self.creator_class:
                 return self.creator_class
         else:
             if proposed_role_class == self.rater_class:
                 return self.rater_class
         return None

Let’s now put all pieces together:

from markupsafe import Markup
import psynet.experiment
from psynet.consent import NoConsent
from psynet.modular_page import ImagePrompt, ModularPage, PushButtonControl, TextControl
from psynet.page import SuccessfulEndPage
from psynet.timeline import Timeline
from psynet.trial.create_and_rate import (
    CreateAndRateNode,
    CreateAndRateTrialMakerMixin,
    CreateTrialMixin,
    RateTrialMixin,
)
from psynet.trial.imitation_chain import ImitationChainTrial, ImitationChainTrialMaker


def animal_prompt(text, img_url):
    return ImagePrompt(
        url=img_url,
        text=Markup(text),
        width="300px",
        height="300px",
    )


class CreateTrial(CreateTrialMixin, ImitationChainTrial):
    time_estimate = 5

    def show_trial(self, experiment, participant):
        return ModularPage(
            "create_trial",
            animal_prompt(text="Describe the animal", img_url=self.context["img_url"]),
            TextControl(),
            time_estimate=self.time_estimate,
        )


class SingleRateTrial(RateTrialMixin, ImitationChainTrial):
    time_estimate = 5

    def show_trial(self, experiment, participant):
        assert len(self.targets) == 1
        target = self.targets[0]
        creation = self.get_target_answer(target)
        return ModularPage(
            "rate_trial",
            animal_prompt(
                text=f"How well does this description match the animal?<br><strong>{creation}</strong>",
                img_url=self.context["img_url"],
            ),
            PushButtonControl(
                choices=[1, 2, 3, 4, 5],
                labels=["not at all", "a little", "somewhat", "very", "perfectly"],
                arrange_vertically=False,
            ),
        )


class CreateAndRateTrialMaker(CreateAndRateTrialMakerMixin, ImitationChainTrialMaker):
    pass


start_nodes = [
    CreateAndRateNode(context={"img_url": "static/dog.jpg"})
]


class Exp(psynet.experiment.Experiment):
    label = "Basic Create and Rate Experiment"
    initial_recruitment_size = 1

    timeline = Timeline(
        NoConsent(),
        CreateAndRateTrialMaker(
            num_creators=2,
            num_raters=2,
            node_class=CreateAndRateNode,
            creator_class=CreateTrial,
            rater_class=SingleRateTrial,
            include_previous_iteration=False,
            rate_mode="rate",
            target_selection_method="one",
            verbose=True,
            # trial_maker params
            id_="create_and_rate_trial_maker",
            chain_type="across",
            expected_trials_per_participant=len(start_nodes),
            max_trials_per_participant=len(start_nodes),
            start_nodes=start_nodes,
            chains_per_experiment=len(start_nodes),
            balance_across_chains=False,
            check_performance_at_end=True,
            check_performance_every_trial=False,
            propagate_failure=False,
            recruit_mode="n_trials",
            target_n_participants=None,
            wait_for_networks=False,
            max_nodes_per_chain=10,
        ),
        SuccessfulEndPage(),
    )

This gives you a simple Create and Rate experiment in just 120 lines 😉