# MasteryPath AI Assessment Integration Analysis

## Overview
MasteryPath (centriclearnosity) is an internal assessment engine that replaces the external Learnosity API. The system manages:
- **Activities**: Collections of items (assessment containers)
- **Items**: Question containers with metadata
- **Questions**: Individual assessment items with various question types
- **Responses**: Student submissions and scores
- **Sessions**: Assessment session tracking

---

## Current System Architecture

### Data Structure
```
Activity (curric.activities)
  ├── id, reference, title, description
  ├── definition (JSON) - metadata about assessment
  ├── metadata (JSON) - additional configuration
  ├── data (JSON) - activity content
  └── Items → Questions → Responses
         └── Sessions (curric.sessions)
```

### Question Types Supported
- **Auto-gradable**: mcq, mcq_multiple, true_false, cloze_text, cloze_dropdown, short_text
- **Manual-gradable**: essay_rich, essay_plain, drawing, audio_recorder, video_recorder, file_upload

### Response Processing Pipeline
1. Student submits via `submit_assessment.php`
2. System stores responses in `curric.responses` table
3. Auto-grading runs for eligible question types
4. Score is calculated and stored in `curric.sessions` metadata
5. Feedback is provided based on score thresholds

---

## Current Auto-Grading Logic

### Supported Question Types

#### MCQ / Multiple Choice
```php
// Line 100-122 in submit_assessment.php
- Validates against stored correct answer
- Supports single and multiple response options
- Awards full/zero points
```

#### Cloze (Fill-in-the-blank)
```php
// Line 124-149
- Multiple blanks per question
- Partial credit: (correct_count / total_count) * points
- Case-insensitive matching with trimmed whitespace
```

#### Short Text
```php
// Line 151-166
- Exact match comparison (case-insensitive)
- Full or zero points
- Currently no fuzzy matching
```

### Questions Requiring Manual Grading
- Essay responses (free-form text)
- Audio/video submissions
- Drawing submissions
- File uploads
- Any question without validation rules

---

## AI Integration Opportunities

### 1. **Enhanced Automatic Grading** ⭐⭐⭐ (High Value)
**Current Gap**: Short text, essay, and drawing submissions require manual grading

**AI Enhancement**:
```php
// Pseudocode for essay grading
case 'essay_rich':
case 'essay_plain':
    $rubric = $content['grading_rubric'] ?? null;
    $sampleAnswers = $content['sample_answers'] ?? [];

    $aiGradingPrompt = buildEssayGradingPrompt(
        studentResponse: $response['value'],
        question: $content['stimulus'],
        rubric: $rubric,
        samples: $sampleAnswers,
        model: 'claude-3-sonnet' // AWS Bedrock
    );

    $aiScore = callClaudeWithBedrock($aiGradingPrompt);
    // Returns: { score: number, feedback: string, reasoning: string }
```

**Implementation Point**: `submit_assessment.php` line 168-176

### 2. **Intelligent Feedback Generation** ⭐⭐⭐ (High Value)
**Current State**: Generic score-based feedback (lines 262-274)

**AI Enhancement**:
```php
// Current (generic):
if ($percentageScore >= 90) {
    $feedback = 'Excellent work! You demonstrated strong understanding...';
}

// Enhanced (AI-powered):
$feedbackPrompt = buildFeedbackPrompt(
    responses: $responses,
    questions: $questions,
    correctAnswers: $correctAnswers,
    score: $percentageScore,
    questionTypes: array_unique($questionTypes)
);

$aiFeedback = callClaudeWithBedrock($feedbackPrompt);
// Returns detailed feedback on:
// - Strengths shown
// - Specific areas for improvement
// - Resources to review
// - Next steps
```

**Implementation Point**: `submit_assessment.php` line 261-274

### 3. **Question Analysis & Difficulty Metrics** ⭐⭐ (Medium Value)
**Use Case**: Help educators understand which questions are confusing

```php
// Periodic analysis of assessment data
$analysisPrompt = buildQuestionAnalysisPrompt(
    allResponses: $responses, // Historical data
    correctResponseRate: calculateAccuracy(),
    commonIncorrectAnswers: getFrequentMistakes(),
    timePerQuestion: getAvgTimeSpent()
);

$aiAnalysis = callClaudeWithBedrock($analysisPrompt);
// Returns:
// - Question clarity issues
// - Suggested improvements
// - Misconceptions to address
// - Difficulty classification
```

### 4. **Plagiarism Detection** ⭐⭐ (Medium Value)
**For**: Essay, short answer, and drawing submissions

```php
case 'essay_rich':
    $plagiarismCheck = callClaudeWithBedrock(
        prompt: "Check for plagiarism in: " . $response['value'],
        model: 'claude-3-haiku' // Faster, cheaper
    );
    // Returns: { isPlagiarized: bool, confidence: float, details: string }
```

### 5. **Adaptive Assessment Recommendations** ⭐ (Lower Priority)
**Future Enhancement**: Guide students to next questions based on understanding

```php
// After grading each response
if ($score < MASTERY_THRESHOLD && $autoGradable) {
    $recommendationPrompt = buildAdaptivePrompt(
        currentQuestion: $question,
        studentResponse: $response,
        learningObjectives: $activity['learning_objectives']
    );

    $nextSteps = callClaudeWithBedrock($recommendationPrompt);
    // Returns: recommended review materials, practice problems, etc.
}
```

### 6. **Answer Key Validation** ⭐ (Low Priority)
**Use Case**: Validate/improve stored correct answers

```php
// During assessment creation/editing
$validationPrompt = buildAnswerKeyPrompt(
    question: $question['content'],
    providedAnswer: $question['validation']['valid_response'],
    questionType: $question['type']
);

$validation = callClaudeWithBedrock($validationPrompt);
// Returns: { isValid: bool, improvements: string, alternatives: array }
```

---

## Database Tables Involved

### curric.activities
```sql
id, reference, title, description, definition (JSON), metadata (JSON),
data (JSON), status, created_at, updated_at
```

### curric.items
```sql
id, reference, title, description, workflow, definition (JSON),
metadata (JSON), status, created_at, updated_at
```

### curric.questions
```sql
id, uuid, item_id, type, content (JSON), status, created_at
```

### curric.responses
```sql
id, user_id, activity_id, item_reference, question_reference,
response (JSON), score (decimal), created_at, updated_at
```

### curric.sessions
```sql
id, session_id, activity_id, user_id, started_at, completed_at,
time_spent, status, metadata (JSON)
```

---

## Integration Approach

### Option A: Synchronous (Current Pattern)
**Advantages**: Immediate feedback, simpler implementation
**Disadvantages**: Slower for essays/videos, poor UX with delays

```php
// In submit_assessment.php during response processing
if (!in_array($questionType, AUTO_GRADABLE_TYPES)) {
    $score = callClaudeGrading($response, $question); // Blocking call
}
```

**Recommended for**: Short text, initial essay analysis

### Option B: Asynchronous (Queue-Based) ⭐ Recommended
**Advantages**: Responsive UX, scalable, handles complex cases
**Disadvantages**: Requires job queue infrastructure

```php
// In submit_assessment.php - quick response
$response = [
    'success' => true,
    'grading_status' => 'pending_ai_review' // for essays/videos
];

// Queue async grading job
queueAsyncJob('grade_essay', [
    'response_id' => $responseId,
    'question_id' => $questionId,
    'student_response' => $response['value']
]);

// Separate worker process periodically calls Claude
// Updates curric.responses.score and sends feedback email
```

### Option C: Hybrid (Intelligent Routing)
- Auto-grade MCQ, True/False, Cloze immediately
- Queue essays/uploads for AI grading
- Show preliminary results immediately
- Update with AI scores asynchronously

---

## AWS Bedrock Integration Points

### Existing Pattern in Codebase
Location: `/awsAI/stream.php`

```php
class BedrockStreamingRuntime {
    public function streamClaude3($prompt,
        $model = 'anthropic.claude-3-sonnet-20240229-v1:0') {
        // Calls AWS Bedrock runtime
        // Streams response for real-time display
    }
}
```

### Available Models
- `anthropic.claude-3-sonnet-20240229-v1:0` - Balanced (recommended for assessment)
- `anthropic.claude-3-haiku-20240307-v1:0` - Fast/cheap (good for validation)

### Usage in Assessment Context
```php
require_once __DIR__ . '/../../awsAI/stream.php';

class AssessmentAIGrader {
    private $bedrockRuntime;

    public function __construct($accessKey, $secretKey) {
        $this->bedrockRuntime = new BedrockStreamingRuntime(
            $accessKey,
            $secretKey
        );
    }

    public function gradeEssay($prompt) {
        // Can stream for immediate feedback
        // Or collect full response for parsing
        return $this->bedrockRuntime->streamClaude3($prompt);
    }
}
```

---

## Recommended Implementation Path

### Phase 1: Quick Wins (1-2 weeks)
1. **Enhanced Feedback Generation**
   - Modify `submit_assessment.php` lines 261-274
   - Call Claude to generate personalized feedback
   - Store feedback in `curric.responses.metadata` as JSON

2. **Short Text Answer Fuzzy Matching**
   - Enhance line 152-166 with semantic similarity
   - Use Claude to check meaning, not just exact match
   - Reduces false negatives

### Phase 2: Essay Grading (2-4 weeks)
1. **Create Rubric Storage**
   - Add `grading_rubric` field to `curric.questions` content JSON
   - Educators define rubric during question creation

2. **Async Essay Grading**
   - Implement job queue (PHP background worker)
   - Call Claude for essay scoring
   - Store scores and update UI

3. **Quality Control**
   - First 30 days: AI suggestions only (educator reviews)
   - After validation: Full automation with spot-check

### Phase 3: Advanced Features (3+ months)
1. Plagiarism detection
2. Drawing assessment
3. Question difficulty analysis
4. Adaptive recommendations

---

## Data Requirements for AI

### Minimal (for immediate use)
- Student response text
- Question prompt/stimulus
- Question type
- Correct answer(s) for comparison

### Optimal (for quality)
- Grading rubric/criteria
- Sample good/poor answers
- Learning objectives
- Question difficulty level
- Student background/previous performance

---

## Security & Privacy Considerations

### PII Protection
```php
// Before sending to AI API
$sanitizedResponse = [
    'response_text' => $studentResponse, // Only educational content
    'question_id' => $questionId,         // Not user ID
    // DO NOT SEND: user_id, email, names in prompt
];

// Prompt construction - anonymize
$prompt = "Grade this student response to: " . $question['stimulus'] .
          "\nResponse: " . $studentResponse;
// Student identity should NOT be in prompt
```

### Compliance
- Check FERPA/state education laws about AI use
- Document AI grading in course syllabus
- Provide option to opt-out of AI grading
- Maintain audit trail of AI decisions

---

## Cost Estimation

### Bedrock Pricing (as of 2025)
- **Sonnet**: Input ~$3/1M tokens, Output ~$15/1M tokens
- **Haiku**: Input ~$0.25/1M tokens, Output ~$1.25/1M tokens

### Volume Examples
- 100 students × 10 assessments = 1,000 assessments/semester
- Essay grading: ~500 tokens per response = $0.01-0.05 per essay
- Feedback: ~200 tokens per response = $0.001-0.005 per question

**Estimated annual cost**: $100-300 for typical institution

---

## Testing Strategy

### Unit Tests
```php
// Test essay grading prompt generation
public function testEssayGradingPromptFormat() {
    $prompt = buildEssayGradingPrompt(...);
    $this->assertStringContains('rubric');
    $this->assertStringContains('student response');
}

// Test score extraction from AI response
public function testScoreExtraction() {
    $aiResponse = "Score: 8/10...";
    $score = extractScoreFromResponse($aiResponse);
    $this->assertEquals(8, $score);
}
```

### Integration Tests
```php
// Test end-to-end essay submission
public function testEssaySubmissionWithAIGrading() {
    $response = submitAssessment([
        'questions' => [
            ['type' => 'essay_rich', 'response' => 'Full essay text...']
        ]
    ]);

    $this->assertEquals('pending_ai_review', $response['grading_status']);
    // Later: Check that async job queued and score updated
}
```

---

## Questions for Stakeholder Review

1. **Grading Authority**: Should AI grades be final or educator-reviewed only?
2. **Rubric Creation**: Will rubrics be provided during question creation or added later?
3. **Feedback Style**: Detailed rubric-based feedback or conversational coaching feedback?
4. **Compliance**: Any institutional policies on AI assessment?
5. **User Notification**: Should students know their response was AI-graded?
6. **Performance**: Acceptable response time for essay grading feedback (immediate vs. overnight)?

---

## Files to Modify/Create

### Modify
- `centriclearnosity/app/player/submit_assessment.php` - Add AI grading logic
- `centriclearnosity/api/models/CentricResponse.php` - Enhance response model

### Create New
- `centriclearnosity/api/models/AIGrader.php` - Central AI grading class
- `centriclearnosity/api/grading/essay_grader.php` - Essay-specific logic
- `centriclearnosity/api/queue/async_grader_worker.php` - Background job processor
- `centriclearnosity/api/prompts/assessment_prompts.php` - Prompt templates

### Update Database
```sql
-- Add rubric storage to questions content JSON
-- Add metadata field to responses for AI feedback storage
ALTER TABLE curric.responses
ADD COLUMN metadata JSON DEFAULT NULL;

-- Track grading method used
ALTER TABLE curric.responses
ADD COLUMN graded_by ENUM('auto', 'ai', 'human') DEFAULT 'auto';

-- Store AI reasoning
ALTER TABLE curric.responses
ADD COLUMN grading_notes TEXT DEFAULT NULL;
```

---

## Conclusion

AI can dramatically enhance MasteryPath by:
1. **Enabling immediate feedback** on essays and complex responses
2. **Personalized guidance** for student improvement
3. **Reducing educator workload** for repetitive grading
4. **Data insights** on question clarity and student understanding
5. **Scalability** for large enrollment courses

The **recommended first implementation** is personalized feedback generation + async essay grading, targeting Phase 1-2 timeline.