AI Assessment three case studies

AI in Assessment - Three Case Studies 

Video recording of online session

AI generated summary: AI in Assessment — Three Case Studies

• 00:20–05:22 — Human teachers must remain the masters of AI. The inaugural remarks caution that, just as language labs and ICT tools were once seen as solutions to classroom problems, AI should not be treated as a panacea. Teachers should evaluate its usefulness critically and adapt it to Indian classroom contexts.
• 20:51–30:45 — The central problem: assessment workload and AI-era assignments. The speaker argues that conventional written assignments have lost some assessment value because students can readily use generative AI. This creates a need to move from merely assessing written products toward performance-based assessment and digital portfolios, while using AI to manage the resulting workload.
• 31:08–42:40 — Pre-session survey establishes teachers' assessment challenges. Among 103 respondents, MCQs were the most common assessment method (77%), followed by descriptive essays and presentations. Teachers identified lengthy answers, meaningful feedback, identifying strengths/weaknesses, comparing performances, and the time required for assessment as major difficulties. Most participants were comfortable with AI assisting in preliminary evaluation of descriptive answers.
• 43:04–54:55 — The key conceptual shift: AI as assessment assistant, not assessor. Assessment involves reading, interpreting, applying criteria, judging, scoring, and providing feedback—not merely assigning marks. The proposed model is Student Work → AI-Assisted Analysis → Rubric-Based Evidence/Patterns → Teacher Validation → Final Judgment & Feedback. The speaker emphasizes that the rubric is the backbone: the better the assessment design and criteria, the more useful AI assistance becomes.
• 55:21–1:09:40 — Case Study 1: AI-assisted assessment of essay answers. Handwritten literature answers were evaluated using teacher-designed prompts, the BAWE corpus, and CEFR guidelines. AI identified strengths and weaknesses in textual understanding, interpretation, argumentation, evidence, academic language, organization, and critical analysis. Students could then ask AI to show how their own answer might be improved rather than simply receiving an ideal model answer. The teacher remained responsible for validating the AI's evaluation and final score.
• 1:10:11–1:29:25 — Case Study 2: Video essays and performance assessment. Students produced video essays, presentations, literary performances, and short videos. AI-assisted analysis helped examine content, argument, language, delivery, visuals, creativity, coherence, and critical thinking. Tools such as Gemini, Adobe Enhance, and NotebookLM were explored. An important unexpected outcome was that students sometimes challenged AI feedback, encouraging critical engagement with assessment itself. The speaker concludes that AI can observe evidence, but humans must interpret its significance, especially regarding creativity, authenticity, cultural context, and nuance.
• 1:29:25–1:35:47 — Case Study 3: AI-assisted digital portfolio assessment. Portfolios containing essays, videos, presentations, reflections, research, projects, blogs, and creative work were analysed for evidence, reflection, growth, and competency development. AI helped identify patterns in students' learning journeys and possible future competencies/trajectories, while the teacher provided holistic judgment.
• 1:35:47–1:39:44 — The major pedagogical shift: from assessing answers to assessing learning. The teacher's role changes from marker → assessment designer → rubric designer → evidence interpreter → feedback designer → AI validator → ethical decision-maker. The proposed AI Assessment Triangle has three essential components: Assessment Design + Rubric + AI Assistance, resting on Human Judgment. If any component is weak, assessment quality suffers.
• 1:39:44–1:55:51 — Q&A reinforces validity, reliability, bias and human oversight. Questions address free/subscription tools, AI bias, validity and reliability, rubric constraints, language learning, and accessibility. The speaker stresses that AI-generated scores should never be accepted blindly and that anonymizing students as “Student 1, Student 2…” may help reduce potential name-, gender-, or caste-associated bias. AI should be AI-assisted, not AI-determined.

Core takeaway

The session's central message can be reduced to one principle:

The future of assessment is not “Human vs AI” but “Human + AI.”

AI can automate reading at scale, pattern identification, rubric application, evidence extraction, comparison, summarization, and preliminary feedback. The teacher must retain responsibility for context, interpretation, fairness, validity, ethics, final judgment, and academic decisions. This aligns closely with recent Glasp discussions emphasizing that AI can generate signals and support evaluation, but human judgment remains essential for deciding what those signals mean and what action should follow.

The most important five questions before using AI for assessment are:

1. What exactly am I assessing?
2. What evidence demonstrates it?
3. What rubric will I use?
4. What can AI reliably assist with?
5. What must remain with human judgment?

This framework also resonates with current thinking on AI evaluation: criteria should be observable, AI judgments should be validated against human judgment, and there should always be a mechanism for human disagreement and override. 

Humans in the Loop - Film Review: Exploring AI Bias through Indigenous Perspectives

 Introduction

In an era where artificial intelligence increasingly shapes our world, the film "Humans in the Loop" emerges as a powerful meditation on the hidden human labor behind AI systems. Directed by Aranya Sahay, this 2024 independent Indian drama explores the complex intersection of Adivasi culture, technology, and the invisible work of data labeling that trains AI systems. The film not only tells the story of Nehma, an indigenous mother, but also raises crucial questions about bias, representation, and the value of traditional knowledge in the digital age.

Film Overview

"Humans in the Loop" is a 2024 independent Indian drama that explores the intersection of Adivasi culture and artificial intelligence. The plot follows Nehma, an indigenous mother who works as a data labeller in Jharkhand, where she discovers how machine learning often ignores or misrepresents her community's traditional knowledge. 

The film is distinguished by its visual poetry and sharp social commentary on the gendered biases embedded within modern technology. Despite its modest box office performance, the film has gained significant momentum through a micro-community rollout and received executive producer support from Kiran Rao. It has achieved international recognition by winning several Best Film awards and securing the prestigious Sloan Distribution Grant.

Thematic Exploration

The film brilliantly uses Nehma's story to reveal the invisible human labour embedded in AI systems. It portrays AI not as a neutral technological force, but as a system shaped by the biases, assumptions, and cultural perspectives of those who train and develop it. This is a poignant reminder that behind every algorithm lies human judgment and labor, often performed by workers from marginalized communities.

The narrative also highlights how indigenous worldviews challenge the cultural biases of modern technology. The film argues that true expertise comes from lived experience and that the wisdom of indigenous communities should be recognized and valued in the development of AI systems.

Key Visual and Narrative Elements

The film employs visual poetry as a means to communicate the complex relationship between human identity and technology. Through striking cinematography, Sahay captures the landscape of Jharkhand while interweaving the protagonist's internal world. The director effectively uses metaphor and visual symbolism to illustrate concepts that could otherwise be abstract, making the philosophical questions about AI bias accessible and emotionally resonant.

Distribution and Global Reach

What makes "Humans in the Loop" particularly significant is its distribution strategy. Despite being an independent Indian film, it has achieved remarkable global reach through: film festival circuits, the Sloan Distribution Grant which positioned it as eligible for Academy Awards consideration, and ultimately, a global release on Netflix. This trajectory demonstrates how niche, socially conscious cinema can find global audiences when backed by strategic support and critical acclaim.

Critical Reception

Critics have consistently lauded the film for its visual stunning execution combined with thematic importance. The film has been recognized for its emotional resonance and social commentary, winning awards at international film festivals. The recognition from institutions like the Sloan Foundation underscores the film's value as both an artistic work and an important cultural document exploring critical contemporary issues.

Conclusion

"Humans in the Loop" is more than a film about AI training data. It is a meditation on labor, dignity, cultural representation, and the voices that shape technology. By centering an Adivasi woman's experience, Aranya Sahay creates a space for conversations about whose perspectives matter in the development of artificial intelligence. The film stands as a powerful reminder that technology is never neutral—it reflects the values, biases, and wisdom of those who create it.

For educators, technologists, and anyone concerned with ethical AI development, this film offers invaluable insights into the human side of machine learning and the importance of including diverse voices in technological progress.

Related Videos

Watch these videos to deepen your understanding of the themes in 'Humans in the Loop':

AI-Assisted Self-Assessment of Essay Type Descriptive Answers

 

[Video recording of the live session]

Transcript for the "Essay Type Answer Writing | Error Analysis & AI-assisted Self-assessment | Academic Writing Workshop" by Merlin and organised by Chat GPT

Good afternoon, everyone, and welcome to another installment of our academic writing workshop series. These workshops have become a tradition, occurring at the close of each semester following our internal tests. It's a time when teachers meticulously review your answer scripts, pinpointing common errors for discussion. But beyond this, it's a chance for you to engage in a crucial aspect of academic growth: self-assessment.

Self-assessment involves more than just glancing over your work; it requires a deep dive into your writing, identifying both obvious mistakes and subtler nuances that often escape notice. Consider creating PDFs of your answer books, allowing for ongoing reflection and improvement, whether you're at home or in the hostel. While AI tools can assist in this process, it's essential to remember that they're just that—tools. Our ultimate goal is to maintain and enhance our own writing abilities, not to become overly reliant on technology.

As we transition into discussing the role of AI in our workshops, it's important to strike a balance. While AI offers valuable insights and can even serve as a personalized tutor, it's not without its limitations. Over-reliance on AI runs the risk of diminishing our own linguistic capabilities. Our workshops aim to harness the benefits of AI while ensuring that human judgment and creativity remain at the forefront.

In analyzing sample answers, we aim to bridge the gap between proficiency and excellence. By leveraging both AI insights and human discernment, we can identify areas for improvement and chart a course towards advanced levels of writing proficiency. Practical exercises are integral to this process, allowing you to apply feedback and refine your approach iteratively.

As we conclude today's workshop, I encourage you to continue honing your writing skills beyond these sessions. The journey towards academic excellence is ongoing, and your commitment to self-assessment and improvement will undoubtedly pay dividends in the long run. Thank you for your participation, and don't forget to complete the assigned activities.

The highlights of the session:

Mastering Essay Writing: Error Analysis & AI-powered Self-Assessment

This blog post summarizes an academic writing workshop focused on improving essay writing through error analysis and AI-assisted self-assessment.

Introduction

The workshop addresses a common challenge: the gap between internal assessments (where improvement is possible) and final exams (where it's not). It emphasizes the importance of strong writing skills and explores how AI tools can enhance self-assessment without diminishing human writing ability.

Challenges of Traditional Error Analysis

Personalized feedback can be time-consuming for teachers.

Replicating student answers with high-quality variations is difficult for humans.

AI as a Solution

Generative AI offers one-on-one tutoring through feedback suggestions.

AI excels at analyzing student writing and suggesting improvements.

Cautions and Best Practices

Overreliance on AI can hinder independent writing development.

Critical thinking skills remain essential, and AI should not replace them.

University exams typically require writing without AI assistance.

Benefits of AI-assisted Self-Assessment

Identifies areas for improvement (e.g., grammar, mechanics, clarity).

Elevates writing quality from B1/B2 to C1/C2 levels (CEFR framework).

Provides suggestions for stronger introductions, conclusions, and stances.

Workshop Activity

1. Analyze Past Errors: Review feedback from teachers on past exams.
2. Select an Answer: Choose an answer you wrote for a previous internal assessment.
3. AI Feedback: Upload a photocopy/PDF of your answer to a designated AI tool.
4. Prompt Selection: Clearly state you are a postgraduate student seeking a high academic level response.
5. Self-Assessment: Analyze the AI's suggestions and compare them to your original writing.
6. Continuous Improvement: Practice writing and self-assessment throughout the semester, aiming for excellence without AI dependence.

Conclusion

This workshop equips students with the tools and strategies to become self-sufficient, confident essay writers. By combining traditional error analysis with AI-powered feedback, students can elevate their writing skills and achieve academic success.

Cyberfeminism - AI and Gender Biases

Cyberfeminism: Artificial Intelligence and the Unconscious Biases

Cyberfeminism appeared in the 1980s and founded on the ideas post-humanist feminist thinker Donna Haraway expresses in her A Cyborg Manifesto. In this manifesto, she lays the groundwork for the concept of the internet being a revolutionary tool to overthrow patriarchy, destroy the existing gender binary and achieve feminist liberation. She sees the internet as a new neutral space women need to ally with and that needs to be shaped by women in a way that will allow them to overthrow the existing social order.

Cyberfeminism had grand ambitions for the internet; however, it failed to acknowledge that the internet does not necessarily represent a fresh start or a free space in which gender does not matter, but is a new space that is very much embedded in society, and that sexist, racist etc. assumptions are imported into the cyberspace. Online spaces and innovative technologies are human creations and therefore biased from their very creation. Nonetheless, although the internet and online technologies are an extension of society, replicating the same problems therein, and even if the platforms are somehow biased, it still represents a separate space for expression, which “negotiates the border” between our public and private lives (Harris, 2008, p.491). It presents opportunities for self-creation and reinvention of identity. This separate space, of course, also offers new opportunities for harassment, exacerbating certain types of behaviours because of the possibility for the perpetrator to hide behind the anonymity of the internet (Evans, 2015). All this leads us to the necessity of questioning the idea of space, safe space, and online versus offline identities and more importantly, to understanding the importance feminist activism online plays in shaping those safe spaces and identities. (Paula Ranzel)

Mia Consalvo defines cyberfeminism as:

1. a label for women—especially young women who might not even want to align with feminism's history—not just to consume new technologies but to actively participate in their making;
2. a critical engagement with new technologies and their entanglement with power structures and systemic oppression. (in "Cyberfeminism", Encyclopedia of New Media, SAGE Publications)

Bruce Grenville in The Uncanny: Experiments in Cyborg Culture mentions: "The dominant cyberfeminist perspective takes a utopian view of cyberspace and the Internet as a means of freedom from social constructs such as gender, sex difference and race. For instance, a description of the concept described it as a struggle to be aware of the impact of new technologies on the lives of women as well as the so-called insidious gendering of technoculture in everyday life.".

It has been proved in several researches that the unconscious biases are creeping in the coding of Artificial Intelligence also. Virtual world is nothing but mirror image of real world. The AI coders are also human beings. If these coders are unconsciously biased or are not made about their unconscious gender biases, the aritificial intelligence / machines / robots / algorithm made by them is bound to have similar biases. If this is not given serious consideration then the hope that people dreamt of, the world free of gender bias, will be lost, even in this digital era.

Here are some interesting observations made by these researchers:

1. Kirti Sharma: How to keep human bias out of AI?

Transcript

2. Robin Hauser: Can we protect AI from our biases?

Additional resources:

• Sarah Kamber - Cyberfeminism and Artificial Life - Routledge
• Kira Hall - Cyberfeminism - in Susan Herring edited 'Computer Mediated Communication'.
• Francesca Ferrando - 

  Is the post-human a post-woman? Cyborgs, robots, artificial intelligence and the futures of gender: a case study

• Josh Feast: 4 Ways to Address Gender Bias in AI
• Paula Ranzel: 

  How does online feminist activism offer the possibility for a truly intersectional feminist resistance?

• Carol Manton and Birgitte Aga: 

  Putting feminism into Artificial Intelligence

Why are We so Scared of Robots / AIs?

Why are we afraid of robots?

Why are we scared of Artificial Intelligence?

Even though robots and AI are products of human imagination & intelligence, the human creative imagination loves to tell stories where in we are warned against Robots / AIs. Why? Why our stories about robots and AI scare us? Is there some deep truths in these narratives? or are they useless apprehensions?

If we look at the stories told to us, we find that we were always warned or scared or told to be afraid of some monsters. The fear of wolves was always narrated to the sheep. The form of wolf keeps on changing. The wolves of the Aesop's fables turns into human-monsters in mythology. The great epics like Ramayana and Mahabharata scared us against crossing the 'Laxman-Rekha' or else be ready to face the Ravanas. The Kansas, the Duryodhanas, the Dushashanas - are remembered time and again so that we remain perpetually scared. In the industrial era, the machines run on electricity were monsters. In the digital era, robots run on artificial intelligence are here to scare us.

Why?

Is it so that the fear of monster strengthen and alerted against impending danger and so we survive today? Is there any direct correlation between fear and survival instinct? Does survival instinct strengthen with fear mechanics?

If so, we should keep on telling stories to be scared of, the stories of monsters. Let the form of monster keep on changing with the advancement in science and technology. Let us move away from the monster of the hell and face the monster of the laboratory. Let us tell the stories to scare us to find a way out.

After all, there is nothing more human than the will to survive.

Let us see these sci-fi short films on AI - Robots to scare ourselves.

 
1. Ghost Machine: The first one is about babysitter robot who becomes so obsessed of the child that murders the murder. Director: Kim GokCountry & year: South-Korea, 2016

2.  The iMOM: The second one is on the iMom - Mom robot. Dir. Ariel Martin

3. Anukul: The third is on Satyajit Ray's short story 'Anukul' (1976) - directed by Sujoy Ghosh