ORA2 Team Submissions#
Contents
Workflows#
There’s an AssessmentWorkflow model that has a workflow app label, which means that the table name is prefixed with workflow_ and not assessment_ like everything else (I’m sure it’s to help resolve a name collision). See for reference:. In fact, each of the AssessmentWorkflow, AssessmentWorkflowStep, and AssessmentWorkflowCancellation models live under the workflow app label.
The
SubmissionMixin.create_submission()
method calls theworkflow_api.create_workflow()
method (it stops atWorkflowMixin.create_workflow()
along the way). This is the entry point to workflow management in ORA.create_workflow()
callsAssessmentWorkflow.start_workflow()
…which calls the
on_init()
method of each specific type of step (e.g. self, peer, staff).The
on_init()
method of e.g. StaffWorkflow is what creates a StaffWorkflow record associated with a given course, item, and submission (and eventually, a scorer and an assessment).
Staff Workflows#
These should be the only workflow type that we care about for team submissions.
Python API for managing StaffWorkflow data::
on_init(submission_uuid)
Looks up submission record from Submissions API. Then creates a StaffWorkflow associated with it.
For teams:
We will have one workflow per team - there’s no value in creating e.g. 5 workflows for a team of 5 learners when we only want 1 of those 5 workflows to be acted on by course staff.
From the perspective of grading, it doesn’t matter, as long as we end up in a state where there is one Score per learner.
Data Models Design#
Motivation#
For course staff to grade learners’ team-based submissions, we have to allow staff to “pull” from a pool of team-based submissions in a different manner from how submissions are pulled for individual submissions.
The
StaffWorkflow
model contains aget_submission_for_review()
method that returns a new submission for staff to review.At the completion of the assessment, we need each team member’s submission record to have a corresponding
Score
(and each score within the team-assignment should be identical).
New Models, sublcass StaffWorkflow#
Create a new
TeamStaffWorkflow
model that inherits fromStaffWorkflow
.Create a new
TeamSubmission
model with a many to many relationship toSubmission
. TheTeamStaffWorkflow
now references aTeamSubmission
.The TeamSubmission model also records who submitted on behalf of the team.
We’ll actually also need a new, team-centric version of the
workflow.AssessmentWorkflow
model, so that we can pass ateam_submission_uuid
to thestart_workflow()
method. It will follow the same flow of logic of creating a workflow record and associated steps.AssessmentWorkflow.start_workflow()
calls theon_init()
function of each assessment type’s API, e.g. In fact, pretty much all of the functions provided by assessment API’s require asubmission_uuid
parameter. So we’ll likely want some new API likeassessment.api.team
that provides the same functionality, but takes ateam_submission_uuid
as the first parameter.
No new team-based AssessmentWorkflowStep model. All that the steps care about are their types and the corresponding workflow instance.
Diagrams#
Behavior on Submit#
For team ORA submissions, only staff assessment types are allowed (or no assessment types at all, in the case of practice assignments). - Peers should not see Team Submissions in their workflows. - “Selves” should not see Team Submissions in their workflows.
A
Submission
record should be created for each member of the team.One
TeamSubmission
record should be created for the team.One
TeamAssessmentWorkflow
should be created for the team.Because this inherits from
AssessmentWorkflow
, we can useAssessmentWorkflowCancellation
for team workflows, too, without any model changes.There will be one “content reference” submission that points to the
AssessmentWorkflow
that’s associated with theTeamAssessmentWorkflow
- this is the submission that’s used to show text/file content to instructors while the team submission is being assessed.
If the staff assessment step (which is the only valid assessment type for team-based ORAs) is selected for the block, one
TeamStaffWorkflow
record is created for the team.One
Assessment
record and (eventually, after grading) oneScore
record per team member is/are created.